[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

TWCS Compaction backed up

Hey guys, quick question: 
I've got a v2.1 cassandra cluster, 12 nodes on aws i3.2xl, commit log on one drive, data on nvme.  That was working very well, it's a ts db and has been accumulating data for about 4weeks.  

The nodes have increased in load and compaction seems to be falling behind.  I used to get about 1 file per day for this column family, about ~30GB Data.db file per day.  I am now getting hundreds per day at  1mb - 50mb.

How to recover from this? 

I can scale out to give some breathing room but will it go back and compact the old days into nicely packed files for the day?    

I tried setting compaction throughput to 1000 from 256 and it seemed to make things worse for the CPU, it's configured on i3.2xl with 8 compaction threads. 


Lastly, I have mixed TTLs in this CF and need to run a repair (I think) to get rid of old tombstones, however running repairs in 2.1 on TWCS column families causes a very large spike in sstable counts due to anti-compaction which causes a lot of disruption, is there any other way?