git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: dlqDeliveryFailureCause Message Expired and deadLetterStrategy


Thank you Tim!

https://issues.apache.org/jira/browse/AMQ-7061

BR,
- Simon

On Fri, 2018-09-21 at 14:55:14 +0200, Tim Bain wrote:
Please submit this as a bug in JIRA, and provide the link here.

My first guess would be that the message is dispatched to the consumer
before it expires, but expires before the consumer pulls it from the
dispatch queue (so this is consumer-side expiration rather than broker-side
expiration), and that the broker code isn't detecting that this particular
Throwable counts as expiration. But I could be wrong about those guesses,
and whoever investigates will have to see if it's that or something else.

One thing you can do to help the investigation is to allow a backlog of
messages to build up on the queue (e.g. by pointing the consumer at a
different queue for a little while) and then see if all the built-up
messages are properly discarded once the consumer reconnects. That would
help test whether broker-side expiration is working.

For your setup, I suspect that fixing your 60ms problem will reduce the
frequency at which you see this to nearly zero, so I expect that this won't
affect you in any meaningful way, but it's still a bug that should be
fixed.

Thanks,
Tim

On Fri, Sep 21, 2018, 3:03 AM Simon Lundström <simlu@xxxxx> wrote:

Hey all!

We're running 5.15.3 and we're using
<https://github.com/ARGOeu/nagios-plugins-activemq/> for monitoring
(SSL'ed) OpenWire. Healthchecks fail and to not get our aliveness-check
queue filled with old messages we added TTL via
<
https://github.com/ARGOeu/nagios-plugins-activemq/blob/3d6d43c/lib/OpenWireProbe/src/org/activemq/probes/OpenWireProbe.java#L124-L126
>.
We got one DLQ'd OpenWire check message and just discarded it but when
we got the second one I started to investigate.

The message which got put into the DLQ has the propery:
dlqDeliveryFailureCause         java.lang.Throwable: Message Expired.
Expiration:1537468027378

Our configuration related to the queue aliveness-test in activemq.xml
is:

[...]
              <authorizationEntry
                queue="aliveness-test.>"
                read="monitoring"
                write="monitoring"
                admin="admins"
              />
[...]
      <queue physicalName="aliveness-test"/>
[...]
          <policyEntry queue="aliveness-test">
            <!--
              Tell the dead letter strategy not to process expired messages
              so that they will just be discarded instead of being sent to
              the DLQ
            -->
            <deadLetterStrategy>
              <sharedDeadLetterStrategy processExpired="false" />
            </deadLetterStrategy>
          </policyEntry>
[...]

I just noticed a bug that our TTL is set to 60ms (instead of seconds)
but that's just how fast ActiveMQ is ; ) So I don't think that's
related.

So what we expect is for those messages to just get deleted but instead
we they get DLQ'd into ActiveMQ.DLQ (since no DLQ is configured).

Thanks and have a great weekend!

BR,
- Simon

____________________________________

Simon Lundström
Section for Infrastructure

IT Services
Stockholm University
SE-106 91 Stockholm, Sweden

www.su.se/english/staff-info/it
https://www.su.se/english/about-this-website/privacy-policy


On Sep 21, 2018 3:03 AM, "Simon Lundström" <simlu@xxxxx> wrote:

Hey all!

We're running 5.15.3 and we're using
<https://github.com/ARGOeu/nagios-plugins-activemq/> for monitoring
(SSL'ed) OpenWire. Healthchecks fail and to not get our aliveness-check
queue filled with old messages we added TTL via
<
https://github.com/ARGOeu/nagios-plugins-activemq/blob/3d6d43c/lib/OpenWireProbe/src/org/activemq/probes/OpenWireProbe.java#L124-L126
.
We got one DLQ'd OpenWire check message and just discarded it but when
we got the second one I started to investigate.

The message which got put into the DLQ has the propery:
dlqDeliveryFailureCause         java.lang.Throwable: Message Expired.
Expiration:1537468027378

Our configuration related to the queue aliveness-test in activemq.xml
is:

[...]
             <authorizationEntry
               queue="aliveness-test.>"
               read="monitoring"
               write="monitoring"
               admin="admins"
             />
[...]
     <queue physicalName="aliveness-test"/>
[...]
         <policyEntry queue="aliveness-test">
           <!--
             Tell the dead letter strategy not to process expired messages
             so that they will just be discarded instead of being sent to
             the DLQ
           -->
           <deadLetterStrategy>
             <sharedDeadLetterStrategy processExpired="false" />
           </deadLetterStrategy>
         </policyEntry>
[...]

I just noticed a bug that our TTL is set to 60ms (instead of seconds)
but that's just how fast ActiveMQ is ; ) So I don't think that's
related.

So what we expect is for those messages to just get deleted but instead
we they get DLQ'd into ActiveMQ.DLQ (since no DLQ is configured).

Thanks and have a great weekend!

BR,
- Simon

____________________________________

Simon Lundström
Section for Infrastructure

IT Services
Stockholm University
SE-106 91 Stockholm, Sweden

www.su.se/english/staff-info/it
https://www.su.se/english/about-this-website/privacy-policy