[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

How to avoid ActiveMQ 5.x OutOfMemory in certain high-pressure conditions?

Hi everyone,

Some time I posted about ActiveMQ OOM that kept occurring on our staging
environment. At that point, I was missing reproducible test scenarios which
I would like to post here. 

The original problem is about ActiveMQ starting to spend too much time with
Full GC, and eventually fails with OOM: GC Overhead limit exceeded. The root
cause is slow consumer on non-persistent topic, which makes ActiveMQ flow to
disk (temp store). Producer flow control was switched off, and we were
hoping that producer will fill temp store (limit 10 Gb) before it gets
blocked. Surprisingly, it turned out that the system was most stable when
memory limit was set to a *smaller number*, 300Mb. Having 4Gb or 8Gb of heap
and 70% of heap memory set as memory limit makes system to crash faster.

I realized that PFC off means broker has no means to provide backpressure,
and it can eventually flood the broker memory with incoming messages.

However, it turned out that even Producer flow control ON (and even having
reasonable producer window size) can provoke OOM on a broker side, depending
on conditions. I can reproduce it with a simple example I want to share with
you. I'm running broker with 3Gb heap, having memory usage limit set as 1Gb;
producer window size: 10Mb. Per-topic policy memoryLimit is 10Mb. With a
demo producer, I'm sending 100Kb messages with no delay (which is up to
10,000 msgs/sec until consumer started!), my slow consumer delays 50ms after
each message.

I can see PFC kicks in, but soon after I get OOM:

2018-08-07 15:28:33 INFO  BrokerService:773 - Apache ActiveMQ 5.15.0
(localhost, ID:a-thinkpad-46421-1533644913567-0:1) started
2018-08-07 15:28:33 INFO  BrokerService:774 - For help or more information
please see:
2018-08-07 15:28:33 INFO  TransportConnector:263 - Connector vm://localhost
2018-08-07 15:29:01 INFO  Topic:386 - topic://amq.topic, Usage Manager
memory limit reached 10485760. Producers will be throttled to the rate at
which messages are removed from this destination to prevent flooding it. See for more info.
2018-08-07 15:29:17 WARN  Transport:237 - Transport Connection to:
tcp:// failed: Unexpected error
occurred: java.lang.OutOfMemoryError: GC overhead limit exceeded

For convenience, I wrapped all the scenario on GitHub using the embedded
broker, and demo producer/consumer in the same entry class. However, during
my tests I was running producer/consumer in a separate JVMs, and still
getting the same OOM result. You might object that such a 'heavy' message
flow is disproportional to heap ActiveMQ gets, but I'd want to see the
stable backpressure we would have during peaks.

Thank you,

Sent from: