[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Too many Cassandra threads waiting!!!


Thanks. That kernel is new enough to have the patch for the infamous Linux kernel futex bug detailed here:

To answer your questions above:

What you're seeing is likely just normal behavior for Cassandra and is an artifact of its staged event driven architecture (SEDA). You can read more about that if you follow the links in the post above. There is work to move from SEDA to a thread-per-core (TPC) architecture, which you can read about in

There are a number of tuning parameters you can tune to adjust the number of threads working on a few of the various stages within Cassandra (e.g. memtable_flush_writers, native_transport_max_threads, and max_hints_delivery_threads).

There will of course be performance impacts for tuning these parameters and the right values will depend on your data model, hardware, and workload (among other things).

On Thu, Aug 2, 2018 at 10:44 PM, nokia ceph <nokiacephusers@xxxxxxxxx> wrote:
Hi Joshua,

# uname -a
Linux cn6.chn6us1c1.cdn 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core)

On Fri, Aug 3, 2018 at 8:27 AM, Joshua Galbraith <jgalbraith@xxxxxxxxxxxx.invalid> wrote:

Out of curiosity, which kernel version are your nodes running?

You may find this old message on the mailing list helpful:

On Wed, Aug 1, 2018 at 5:38 PM, Elliott Sims <elliott@xxxxxxxxxxxxx> wrote:
You might have more luck trying to analyze at the Java level, either via a (Java) stack dump and the "ttop" tool from Swiss Java Knife, or Cassandra tools like "nodetool tpstats"

On Wed, Aug 1, 2018 at 2:08 AM, nokia ceph <nokiacephusers@xxxxxxxxx> wrote:

i'm having a 5 node cluster with cassandra 3.0.13.

i could see the cassandra process has too many threads.

# pstree -p `pgrep java` | wc -l

And almost all of those threads are in sleeping state and wait at
# cat  /proc/166022/task/1698913/wchan

Some more info:
# strace -e trace=all -p 166022
strace: Process 166022 attached
futex(0x7efc24aeb9d0, FUTEX_WAIT, 166023, NULL

# cat /proc/166022/stack
[<ffffffff810f5126>] futex_wait_queue_me+0xc6/0x130
[<ffffffff810f5deb>] futex_wait+0x17b/0x280
[<ffffffff810f7b26>] do_futex+0x106/0x5a0
[<ffffffff810f8040>] SyS_futex+0x80/0x180
[<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff

What is the reason cassandra is having these many threads? is it the normal behavior of cassandra?  Is there a way to reduce this thread count? will there be any performance impact because of this (our platform experts suspects so)?

Renoy  Paulose

Joshua Galbraith | Lead Software Engineer | New Relic

Joshua Galbraith | Lead Software Engineer | New Relic