We feed a Cassandra v3.7 DB from a streaming application. From time to time we need to update a boolean column for an entire table based on some logic . For this we wrote a Spark job but this always crashed after a while with tombstone threshold error ("Scanned over 100001 tombstones during query..").
When we compacted the table and set spark.cassandra.input.split.size_in_mb to a lower size the job became really slow but after few retries it was able to finish.
Here we remarked exceptions like:
java.io.IOException: Failed to write statements to xxx. The latest exception was
Cassandra failure during write query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded, 2 failed) I guess because we put too much load on the cluster.
We're thinking about to take a snapshot of a table, read and manipulate the sstables with a MR job and bulk load it into Cassandra, but in this case we most probably can't do this on the same table without affecting the consumers accessing that table.
What is the recommended way to do such a live-update on an entire table?