git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Bootstrap OOM issues with Cassandra 3.11.1


Hello All,

I'm having JVM unstable / OOM errors when attempting to auto bootstrap a 9th node to an existing 8 node cluster (256 tokens).  Each machine has 24 cores 148GB RAM and 10TB (2TB used).  Under normal operation the 8 nodes have JVM memory configured with Xms35G and Xmx35G, and handle 2-4 billion inserts per day.  There are never updates, deletes, or sparsely populated rows.  

For the bootstrap node, I've tried memory values from 35GB to 135GB in 10GB increments. I've tried using both memtable_allocation_types (heap_buffers and offheap_buffers).  I've not tried modifying the memtable_cleanup_threshold but instead have tried memtable_flush_writers from 2 to 8.  I've tried memtable_(off)heap_space_in_mb from 20000 to 60000.  I've tried both CMS and G1 garbage collection with various settings.  

Typically, after streaming about ~2TB of data, CPU load will hit a maximum, and the "nodetool info" heap memory will, over the course of an hour, approach the maximum.  At that point, CPU load will drop to a single thread with minimal activity until the system becomes unstable and eventually the OOM error occurs.

Excerpt of the system log is below, and what I consistently see is the MemtableFlushWriter and the MemtableReclaimMemory pending queues grow as the memory becomes depleted, but the number of completed seems to stop changing a few minutes after the CPU load spikes.

One other data point is there seems to be a huge number of mutations that occur after most of the stream has occured.  Concurrent_writes is set at 256 with the queue getting as high as 200K before dropping down.  

Any suggestions for yaml changes or jvm changes?  JVM.options is currently the default with the memory set to the max, the current YAML file is below.

Thanks!


INFO  [ScheduledTasks:1] 2018-08-06 17:49:26,329 StatusLogger.java:51 - MutationStage                     1         2      191498052         0                 0
INFO  [ScheduledTasks:1] 2018-08-06 17:49:26,331 StatusLogger.java:51 - ViewMutationStage                 0         0              0         0                 0
INFO  [Service Thread] 2018-08-06 17:49:26,338 StatusLogger.java:51 - PerDiskMemtableFlushWriter_0         0         0           5865         0                 0
INFO  [ScheduledTasks:1] 2018-08-06 17:49:26,343 StatusLogger.java:51 - ReadStage                         0         0              0         0                 0
INFO  [Service Thread] 2018-08-06 17:49:26,347 StatusLogger.java:51 - ValidationExecutor                0         0              0         0                 0
INFO  [ScheduledTasks:1] 2018-08-06 17:49:26,360 StatusLogger.java:51 - RequestResponseStage              0         0              8         0                 0
INFO  [Service Thread] 2018-08-06 17:49:26,380 StatusLogger.java:51 - Sampler                           0         0              0         0                 0
INFO  [Service Thread] 2018-08-06 17:49:26,382 StatusLogger.java:51 - MemtableFlushWriter               8     74293           4716         0                 0
INFO  [ScheduledTasks:1] 2018-08-06 17:49:26,388 StatusLogger.java:51 - ReadRepairStage                   0         0              0         0                 0
INFO  [ScheduledTasks:1] 2018-08-06 17:49:26,389 StatusLogger.java:51 - CounterMutationStage              0         0              0         0                 0
INFO  [ScheduledTasks:1] 2018-08-06 17:49:26,404 StatusLogger.java:51 - MiscStage                         0         0              0         0                 0
INFO  [ScheduledTasks:1] 2018-08-06 17:49:26,407 StatusLogger.java:51 - CompactionExecutor                8        13            493         0                 0
INFO  [Service Thread] 2018-08-06 17:49:26,410 StatusLogger.java:51 - InternalResponseStage             0         0             16         0                 0
INFO  [ScheduledTasks:1] 2018-08-06 17:49:26,413 StatusLogger.java:51 - MemtableReclaimMemory             1      6066            356         0                 0
INFO  [Service Thread] 2018-08-06 17:49:26,421 StatusLogger.java:51 - AntiEntropyStage                  0         0              0         0                 0
INFO  [Service Thread] 2018-08-06 17:49:26,430 StatusLogger.java:51 - CacheCleanupExecutor              0         0              0         0                 0
INFO  [ScheduledTasks:1] 2018-08-06 17:49:26,431 StatusLogger.java:51 - PendingRangeCalculator            0         0              9         0                 0
INFO  [Service Thread] 2018-08-06 17:49:26,436 StatusLogger.java:61 - CompactionManager                 8        19



 Current Yaml
num_tokens: 256
hinted_handoff_enabled: true
hinted_handoff_throttle_in_kb: 10240 
max_hints_delivery_threads: 8
hints_flush_period_in_ms: 10000
max_hints_file_size_in_mb: 128
batchlog_replay_throttle_in_kb: 10240
authenticator: AllowAllAuthenticator
authorizer: AllowAllAuthorizer
role_manager: CassandraRoleManager
roles_validity_in_ms: 2000
permissions_validity_in_ms: 2000
credentials_validity_in_ms: 2000
partitioner: org.apache.cassandra.dht.Murmur3Partitioner
data_file_directories:
    - /data/cassandra/data
commitlog_directory: /data/cassandra/commitlog
cdc_enabled: false
disk_failure_policy: stop
commit_failure_policy: stop
prepared_statements_cache_size_mb:
thrift_prepared_statements_cache_size_mb:
key_cache_size_in_mb:
key_cache_save_period: 14400
row_cache_size_in_mb: 0
row_cache_save_period: 0
counter_cache_size_in_mb:
counter_cache_save_period: 7200
saved_caches_directory: /data/cassandra/saved_caches
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000
commitlog_segment_size_in_mb: 32
seed_provider:
    - class_name: org.apache.cassandra.locator.SimpleSeedProvider
      parameters:
          - seeds: "10.1.1.11,10.1.1.12,10.1.1.13"
concurrent_reads: 128
concurrent_writes: 256
concurrent_counter_writes: 96
concurrent_materialized_view_writes: 32
disk_optimization_strategy: spinning
memtable_heap_space_in_mb: 61440
memtable_offheap_space_in_mb: 61440
memtable_allocation_type: heap_buffers
commitlog_total_space_in_mb: 81920
memtable_flush_writers: 8



( ! ) Warning: include(msgfooter.php): failed to open stream: No such file or directory in /var/www/git/apache-cassandra-users/msg06681.html on line 73
Call Stack
#TimeMemoryFunctionLocation
10.0009358376{main}( ).../msg06681.html:0

( ! ) Warning: include(): Failed opening 'msgfooter.php' for inclusion (include_path='.:/var/www/git') in /var/www/git/apache-cassandra-users/msg06681.html on line 73
Call Stack
#TimeMemoryFunctionLocation
10.0009358376{main}( ).../msg06681.html:0