git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Too many Cassandra threads waiting!!!


Renoy,

Thanks. That kernel is new enough to have the patch for the infamous Linux kernel futex bug detailed here: https://groups.google.com/d/topic/mechanical-sympathy/QbmpZxp6C64

To answer your questions above:

What you're seeing is likely just normal behavior for Cassandra and is an artifact of its staged event driven architecture (SEDA). You can read more about that if you follow the links in the post above. There is work to move from SEDA to a thread-per-core (TPC) architecture, which you can read about in https://issues.apache.org/jira/browse/CASSANDRA-10989.

There are a number of tuning parameters you can tune to adjust the number of threads working on a few of the various stages within Cassandra (e.g. memtable_flush_writers, native_transport_max_threads, and max_hints_delivery_threads).

There will of course be performance impacts for tuning these parameters and the right values will depend on your data model, hardware, and workload (among other things).

On Thu, Aug 2, 2018 at 10:44 PM, nokia ceph <nokiacephusers@xxxxxxxxx> wrote:
Hi Joshua,

# uname -a
Linux cn6.chn6us1c1.cdn 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core)

On Fri, Aug 3, 2018 at 8:27 AM, Joshua Galbraith <jgalbraith@xxxxxxxxxxxx.invalid> wrote:
Renoy,

Out of curiosity, which kernel version are your nodes running?

You may find this old message on the mailing list helpful:
http://mail-archives.apache.org/mod_mbox/cassandra-user/201602.mbox/%3CCAA=6J0-0VabfAn3DJfatOxyJwwEHpdiE67v2wm_u5KAqOrOqnA@xxxxxxxxxxxxxx%3E

On Wed, Aug 1, 2018 at 5:38 PM, Elliott Sims <elliott@xxxxxxxxxxxxx> wrote:
You might have more luck trying to analyze at the Java level, either via a (Java) stack dump and the "ttop" tool from Swiss Java Knife, or Cassandra tools like "nodetool tpstats"

On Wed, Aug 1, 2018 at 2:08 AM, nokia ceph <nokiacephusers@xxxxxxxxx> wrote:
Hi,

i'm having a 5 node cluster with cassandra 3.0.13.

i could see the cassandra process has too many threads.

# pstree -p `pgrep java` | wc -l
453

And almost all of those threads are in sleeping state and wait at
# cat  /proc/166022/task/1698913/wchan
futex_wait_queue_me

Some more info:
# strace -e trace=all -p 166022
strace: Process 166022 attached
futex(0x7efc24aeb9d0, FUTEX_WAIT, 166023, NULL

# cat /proc/166022/stack
[<ffffffff810f5126>] futex_wait_queue_me+0xc6/0x130
[<ffffffff810f5deb>] futex_wait+0x17b/0x280
[<ffffffff810f7b26>] do_futex+0x106/0x5a0
[<ffffffff810f8040>] SyS_futex+0x80/0x180
[<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff


What is the reason cassandra is having these many threads? is it the normal behavior of cassandra?  Is there a way to reduce this thread count? will there be any performance impact because of this (our platform experts suspects so)?

Regards,
Renoy  Paulose





--
Joshua Galbraith | Lead Software Engineer | New Relic




--
Joshua Galbraith | Lead Software Engineer | New Relic


( ! ) Warning: include(msgfooter.php): failed to open stream: No such file or directory in /var/www/git/apache-cassandra-users/msg06656.html on line 75
Call Stack
#TimeMemoryFunctionLocation
10.0008358376{main}( ).../msg06656.html:0

( ! ) Warning: include(): Failed opening 'msgfooter.php' for inclusion (include_path='.:/var/www/git') in /var/www/git/apache-cassandra-users/msg06656.html on line 75
Call Stack
#TimeMemoryFunctionLocation
10.0008358376{main}( ).../msg06656.html:0