git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: dynamic_snitch=false, prioritisation/order or reads from replicas


Thank you for replying, Alain!


Better use of cache for 'pinned' requests explains good the case when CL=ONE.


But in case of CL=QUORUM/LOCAL_QUORUM, if I'm not wrong, read request is sent to all replicas waiting for first 2 to reply.

When dynamic snitching is turned on, "data" request is sent to "the fastest replica", and "digest" requests - to the rest of replicas.

But anyway digest is the same read operation [from SSTables through filesystem cache] + calculating and sending hash to coordinator. Looks like the only change for dynamic_snitch=false is that "data" request is sent to a determined node instead of "currently the fastest one".

So, if there are no mistakes in above description, improvement shouldn't be much visible for CL=*QUORUM...


Did you get improved response for CL=ONE only or for higher CL's as well?


Indeed an interesting thread in Jira.


Thanks,

Kyrill


From: Alain RODRIGUEZ <arodrime@xxxxxxxxx>
Sent: Monday, August 6, 2018 8:26:43 PM
To: user cassandra.apache.org
Subject: Re: dynamic_snitch=false, prioritisation/order or reads from replicas
 
Hello,
 
There are reports (in this ML too) that disabling dynamic snitching decreases response time.

I confirm that I have seen this improvement on clusters under pressure. 

What effects stand behind this improvement?

My understanding is that this is due to the fact that the clients are then 'pinned', more sticking to specific nodes when the dynamic snitching is off. I guess there is a better use of caches and in-memory structures, reducing the amount of disk read needed, which can lead to way more performances than switching from node to node as soon as the score of some node is not good enough. 
I am also not sure that the score calculation is always relevant, thus increasing the threshold before switching reads to another node is still often worst than disabling it completely. I am not sure if the score calculation was fixed, but in most cases, I think it's safer to run with 'dynamic_snitch: false'. Anyway, it's possible to test it on a canary node (or entire rack) and look at the p99 for read latencies for example :).

This ticket is old, but was precisely on that topic: https://issues.apache.org/jira/browse/CASSANDRA-6908

C*heers
-----------------------
Alain Rodriguez - @arodream - alain@xxxxxxxxxxxxxxxxx
France / Spain

The Last Pickle - Apache Cassandra Consulting

2018-08-04 15:37 GMT+02:00 Kyrylo Lebediev <Kyrylo_Lebediev@xxxxxxxx.invalid>:

Hello!


In case when dynamic snitching is enabled data is read from 'the fastest replica' and other replicas send digests for CL=QUORUM/LOCAL_QUORUM .

When dynamic snitching is disabled, as the concept of the fastest replica disappears, which rules are used to choose from which replica to read actual data (not digests):

 1) when all replicas are online

 2) when the node primarily responsible for the token range is offline.


There are reports (in this ML too) that disabling dynamic snitching decreases response time.

What effects stand behind this improvement?


Regards,

Kyrill