git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: cassandro nodes restarts


Need to review java gc, system , network, disk, memory, node, and table statistics. A lot can be discerned from visually examining the charts. Eg. if the nodes with the most local reads is failing or is it the one with the most writes or is it completely unrelated.

Since it’s a distributed system you need to review the data points together for all nodes. Data is the only way to see what’s going on. Either connect Prometheus / Grafana , get Datadog , New Relic, or something else to see the patterns across the cluster.

https://blog.anant.us/resources-for-monitoring-datastax-cassandra-spark-solr-performance/

I assembled that list recently — I would even add that getting system logs into ELK or Splunk could also show some patterns otherwise not detected tailing and gripping.

Rahul
On Jul 26, 2018, 10:20 AM -0400, R1 J1 <rjsoft100@xxxxxxxxx>, wrote:
Thanks for your prompt replies. No the same node is not bouncing over. When you say it is about to tip over: What can we do to stop that ?

Also about that error : you guys are correct: it is  a warning and might not be contributing to the node bounce issue and it can be removed by changing batch_size_warn_threshold_in_kb: 5

R1J1

On Wed, Jul 25, 2018 at 10:32 PM, R1 J1 <rjsoft100@xxxxxxxxx> wrote:
cassandro nodes restarts



we see errors typically like these


WARN  [Native-Transport-Requests-3] 2018-07-25 20:51:38,520 BatchStatement.java:301 - Batch for "keyspace.table"
 is of size 19.386KiB, exceeding specified threshold of 5.000KiB by 14.386KiB.


Regards
R1J1



( ! ) Warning: include(msgfooter.php): failed to open stream: No such file or directory in /var/www/git/apache-cassandra-users/msg06611.html on line 115
Call Stack
#TimeMemoryFunctionLocation
10.0012364552{main}( ).../msg06611.html:0

( ! ) Warning: include(): Failed opening 'msgfooter.php' for inclusion (include_path='.:/var/www/git') in /var/www/git/apache-cassandra-users/msg06611.html on line 115
Call Stack
#TimeMemoryFunctionLocation
10.0012364552{main}( ).../msg06611.html:0