git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: C* as fluent data storage, 10MB/sec/node?


Thanks for the excellent advice, this was extremely helpful! Did not know about TWCS... curing a lot of headache.

Adam

Am Mi., 28. Nov. 2018 um 20:47 Uhr schrieb Jeff Jirsa <jjirsa@xxxxxxxxx>:
Probably fine as long as there’s some concept of time in the partition key to keep them from growing unbounded.

Use TWCS, TTLs and something like 5-10 minute buckets. Don’t use RF=1, but you can write at CL ONE. TWCS will largely just drop whole sstables as they expire (especially with 3.11 and the more aggressive expiration logic there)



--
Jeff Jirsa


> On Nov 28, 2018, at 11:24 AM, Adam Smith <adamsmith8745@xxxxxxxxx> wrote:
>
> Hi All,
>
> I need to use C* somehow as fluent data storage - maybe this is different to the queue antipattern? Lots of data come in (10MB/sec/node), remains for e.g. 1 hour and should then be evicted. It is somehow not critical when data would occasionally disappear/get lost.
>
> Thankful for any advice!
>
> Is this nowadays possible without suffering too much from compactation? I would not have ranged tombstones, and depending on a possible solution only using point deletes (PK+CK). There is only one CK, could also be empty.
>
> 1) The data is usually 1 MB. Can I just update with empty data? PK + CK would remain, but I would not carry about that. Would this create tombstones or is equivalent to a DELETE?
>
> 2) Like 1) and later then set a TTL == small amount of data to be deleted then? And hopefully small compactation?
>
> 3) Simply setting TTL 1h and hoping the best, because I am wrong with my worries?
>
> 4) Any optimization strategies like setting the RF to 1? Which compactation strategy is advised?
>
> 5) Are there any recent performance benchmarks for one of the scenarios?
>
> What else could I do?
>
> Thanks a lot!
> Adam

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@xxxxxxxxxxxxxxxxxxxx
For additional commands, e-mail: user-help@xxxxxxxxxxxxxxxxxxxx