Out of curiosity, which kernel version are your nodes running?
You may find this old message on the mailing list helpful:
org/mod_mbox/cassandra-user/ 201602.mbox/%3CCAA=6J0- 0VabfAn3DJfatOxyJwwEHpdiE67v2w m_u5KAqOrOqnA@xxxxxxxxxxxxxx% 3E--On Wed, Aug 1, 2018 at 5:38 PM, Elliott Sims <elliott@xxxxxxxxxxxxx> wrote:You might have more luck trying to analyze at the Java level, either via a (Java) stack dump and the "ttop" tool from Swiss Java Knife, or Cassandra tools like "nodetool tpstats"On Wed, Aug 1, 2018 at 2:08 AM, nokia ceph <nokiacephusers@xxxxxxxxx> wrote:Hi,i'm having a 5 node cluster with cassandra 3.0.13.i could see the cassandra process has too many threads.# pstree -p `pgrep java` | wc -l453And almost all of those threads are in sleeping state and wait at# cat /proc/166022/task/1698913/wcha nfutex_wait_queue_meSome more info:# strace -e trace=all -p 166022strace: Process 166022 attachedfutex(0x7efc24aeb9d0, FUTEX_WAIT, 166023, NULL# cat /proc/166022/stack[<ffffffff810f5126>] futex_wait_queue_me+0xc6/0x130[<ffffffff810f5deb>] futex_wait+0x17b/0x280[<ffffffff810f7b26>] do_futex+0x106/0x5a0[<ffffffff810f8040>] SyS_futex+0x80/0x180[<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b[<ffffffffffffffff>] 0xffffffffffffffffWhat is the reason cassandra is having these many threads? is it the normal behavior of cassandra? Is there a way to reduce this thread count? will there be any performance impact because of this (our platform experts suspects so)?Regards,Renoy PauloseJoshua Galbraith | Lead Software Engineer | New Relic