[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Too many Cassandra threads waiting!!!

Hi Joshua,

# uname -a
Linux cn6.chn6us1c1.cdn 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core)

On Fri, Aug 3, 2018 at 8:27 AM, Joshua Galbraith <jgalbraith@xxxxxxxxxxxx.invalid> wrote:

Out of curiosity, which kernel version are your nodes running?

You may find this old message on the mailing list helpful:

On Wed, Aug 1, 2018 at 5:38 PM, Elliott Sims <elliott@xxxxxxxxxxxxx> wrote:
You might have more luck trying to analyze at the Java level, either via a (Java) stack dump and the "ttop" tool from Swiss Java Knife, or Cassandra tools like "nodetool tpstats"

On Wed, Aug 1, 2018 at 2:08 AM, nokia ceph <nokiacephusers@xxxxxxxxx> wrote:

i'm having a 5 node cluster with cassandra 3.0.13.

i could see the cassandra process has too many threads.

# pstree -p `pgrep java` | wc -l

And almost all of those threads are in sleeping state and wait at
# cat  /proc/166022/task/1698913/wchan

Some more info:
# strace -e trace=all -p 166022
strace: Process 166022 attached
futex(0x7efc24aeb9d0, FUTEX_WAIT, 166023, NULL

# cat /proc/166022/stack
[<ffffffff810f5126>] futex_wait_queue_me+0xc6/0x130
[<ffffffff810f5deb>] futex_wait+0x17b/0x280
[<ffffffff810f7b26>] do_futex+0x106/0x5a0
[<ffffffff810f8040>] SyS_futex+0x80/0x180
[<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff

What is the reason cassandra is having these many threads? is it the normal behavior of cassandra?  Is there a way to reduce this thread count? will there be any performance impact because of this (our platform experts suspects so)?

Renoy  Paulose

Joshua Galbraith | Lead Software Engineer | New Relic