git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

TWCS Compaction backed up


Hey guys, quick question: 
 
I've got a v2.1 cassandra cluster, 12 nodes on aws i3.2xl, commit log on one drive, data on nvme.  That was working very well, it's a ts db and has been accumulating data for about 4weeks.  

The nodes have increased in load and compaction seems to be falling behind.  I used to get about 1 file per day for this column family, about ~30GB Data.db file per day.  I am now getting hundreds per day at  1mb - 50mb.

How to recover from this? 

I can scale out to give some breathing room but will it go back and compact the old days into nicely packed files for the day?    

I tried setting compaction throughput to 1000 from 256 and it seemed to make things worse for the CPU, it's configured on i3.2xl with 8 compaction threads. 

-B

Lastly, I have mixed TTLs in this CF and need to run a repair (I think) to get rid of old tombstones, however running repairs in 2.1 on TWCS column families causes a very large spike in sstable counts due to anti-compaction which causes a lot of disruption, is there any other way?  




( ! ) Warning: include(msgfooter.php): failed to open stream: No such file or directory in /var/www/git/apache-cassandra-users/msg06706.html on line 69
Call Stack
#TimeMemoryFunctionLocation
10.0008358384{main}( ).../msg06706.html:0

( ! ) Warning: include(): Failed opening 'msgfooter.php' for inclusion (include_path='.:/var/www/git') in /var/www/git/apache-cassandra-users/msg06706.html on line 69
Call Stack
#TimeMemoryFunctionLocation
10.0008358384{main}( ).../msg06706.html:0