git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: system_auth keyspace replication factor


Attaching the runner log snippet, where we can see that "Rebuilding token map" took most of the time.
getAllroles is using quorum, don't if it is used during login
https://github.com/apache/cassandra/blob/cc12665bb7645d17ba70edcf952ee6a1ea63127b/src/java/org/apache/cassandra/auth/CassandraRoleManager.java#L260

Vitali Djatsuk,
On Fri, Nov 23, 2018 at 8:32 PM Jeff Jirsa <jjirsa@xxxxxxxxx> wrote:
I suspect some of the intermediate queries (determining role, etc) happen at quorum in 2.2+, but I don’t have time to go read the code and prove it. 

In any case, RF > 10 per DC is probably excessive

Also want to crank up the validity times so it uses cached info longer


-- 
Jeff Jirsa


On Nov 23, 2018, at 10:18 AM, Vitali Dyachuk <vdjatsuk@xxxxxxxxx> wrote:

no its not a cassandra user and as i understood all other users login local_one.

On Fri, 23 Nov 2018, 19:30 Jonathan Haddad <jon@xxxxxxxxxxxxx wrote:
Any chance you’re logging in with the Cassandra user? It uses quorum reads. 


On Fri, Nov 23, 2018 at 11:38 AM Vitali Dyachuk <vdjatsuk@xxxxxxxxx> wrote:
Hi,
We have recently met a problem when we added 60 nodes in 1 region to the cluster
and set an RF=60 for the system_auth ks, following this documentation https://docs.datastax.com/en/cql/3.3/cql/cql_using/useUpdateKeyspaceRF.html
However we've started to see increased login latencies in the cluster 5x bigger than before changing RF of system_auth ks.
We have casandra runner written is csharp, running against the cluster, when analyzing the logs we notices that   Rebuilding token map  is taking most of the time ~20s.
When we changed RF to 3 the issue has resolved.
We are using C* 3.0.17 , 4 DC, system_auth RF=3, "CassandraCSharpDriver" version="3.2.1"   
I've found somehow related to my problem ticket https://datastax-oss.atlassian.net/browse/CSHARP-436 but it says in the related tickets, that the issue with the token map rebuild time has been fixed in the previous versions of the driver.
So my question is what is the best recommendation of the setting system_auth ks RF?

Regards,
Vitali Djatsuk.


--
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade
ControlConnection: 11/22/2018 10:30:32.170 +00:00 : Trying to connect the ControlConnection
TcpSocket: 11/22/2018 10:30:32.170 +00:00 Socket connected, starting SSL client authentication
TcpSocket: 11/22/2018 10:30:32.170 +00:00 Starting SSL authentication
TcpSocket: 11/22/2018 10:30:32.217 +00:00 SSL authentication successful
Connection: 11/22/2018 10:30:32.217 +00:00 Sending #0 for StartupRequest to node1:9042
Connection: 11/22/2018 10:30:32.217 +00:00 Received #0 from node1:9042
Connection: 11/22/2018 10:30:32.217 +00:00 Sending #0 for AuthResponseRequest to node1:9042
Connection: 11/22/2018 10:30:32.329 +00:00 Received #0 from node1:9042
ControlConnection: 11/22/2018 10:30:32.329 +00:00 : Connection established to node1:9042
Connection: 11/22/2018 10:30:32.329 +00:00 Sending #0 for RegisterForEventRequest to node1:9042
Connection: 11/22/2018 10:30:32.329 +00:00 Received #0 from node1:9042
ControlConnection: 11/22/2018 10:30:32.329 +00:00 : Refreshing node list
Connection: 11/22/2018 10:30:32.329 +00:00 Sending #0 for QueryRequest to node1:9042
Connection: 11/22/2018 10:30:32.342 +00:00 Received #0 from node1:9042
Connection: 11/22/2018 10:30:32.342 +00:00 Sending #0 for QueryRequest to node1:9042
Connection: 11/22/2018 10:30:32.373 +00:00 Received #0 from node1:9042
ControlConnection: 11/22/2018 10:30:32.389 +00:00 : Node list retrieved successfully
ControlConnection: 11/22/2018 10:30:32.389 +00:00 : Retrieving keyspaces metadata
Connection: 11/22/2018 10:30:32.389 +00:00 Sending #0 for QueryRequest to node1:9042
Connection: 11/22/2018 10:30:32.389 +00:00 Received #0 from node1:9042
ControlConnection: 11/22/2018 10:30:32.389 +00:00 : Rebuilding token map
Cluster: 11/22/2018 10:30:55.233 +00:00 : Cluster Connected using binary protocol version: [4]
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@xxxxxxxxxxxxxxxxxxxx
For additional commands, e-mail: user-help@xxxxxxxxxxxxxxxxxxxx