git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: dlqDeliveryFailureCause Message Expired and deadLetterStrategy


No problem. Hopefully whoever picks up the JIRA issue will be able to
reproduce the problem.

Tim

On Mon, Sep 24, 2018, 11:14 PM Simon Lundström <simlu@xxxxx> wrote:

> Hey Tim!
>
> Sorry, but I can't actually reproduce it. I tried sending persistent
> messages with 60ms TTL from the ActiveMQ console and then starting the
> producer/consumer monitoring check client. All messages are eaten and
> done are DLQ'd = (
>
> You're welcome for nothing ; )
>
> BR,
> - Simon
>
> On Mon, 2018-09-24 at 14:41:14 +0200, Tim Bain wrote:
> >Thank you.
> >
> >Did you get a chance to test the behavior if you allowed messages to back
> >up and then expire on the broker? That would be valuable information for
> >the JIRA issue you submitted, so if you have the ability to do that test
> >(since you already have a system that's known to do this), that would be
> >helpful for the investigation..
> >
> >Tim
> >
> >On Mon, Sep 24, 2018, 5:24 AM Simon Lundström <simlu@xxxxx> wrote:
> >
> >> Thank you Tim!
> >>
> >> https://issues.apache.org/jira/browse/AMQ-7061
> >>
> >> BR,
> >> - Simon
> >>
> >> On Fri, 2018-09-21 at 14:55:14 +0200, Tim Bain wrote:
> >> >Please submit this as a bug in JIRA, and provide the link here.
> >> >
> >> >My first guess would be that the message is dispatched to the consumer
> >> >before it expires, but expires before the consumer pulls it from the
> >> >dispatch queue (so this is consumer-side expiration rather than
> >> broker-side
> >> >expiration), and that the broker code isn't detecting that this
> particular
> >> >Throwable counts as expiration. But I could be wrong about those
> guesses,
> >> >and whoever investigates will have to see if it's that or something
> else.
> >> >
> >> >One thing you can do to help the investigation is to allow a backlog of
> >> >messages to build up on the queue (e.g. by pointing the consumer at a
> >> >different queue for a little while) and then see if all the built-up
> >> >messages are properly discarded once the consumer reconnects. That
> would
> >> >help test whether broker-side expiration is working.
> >> >
> >> >For your setup, I suspect that fixing your 60ms problem will reduce the
> >> >frequency at which you see this to nearly zero, so I expect that this
> >> won't
> >> >affect you in any meaningful way, but it's still a bug that should be
> >> >fixed.
> >> >
> >> >Thanks,
> >> >Tim
> >> >
> >> >On Fri, Sep 21, 2018, 3:03 AM Simon Lundström <simlu@xxxxx> wrote:
> >> >
> >> >> Hey all!
> >> >>
> >> >> We're running 5.15.3 and we're using
> >> >> <https://github.com/ARGOeu/nagios-plugins-activemq/> for monitoring
> >> >> (SSL'ed) OpenWire. Healthchecks fail and to not get our
> aliveness-check
> >> >> queue filled with old messages we added TTL via
> >> >> <
> >> >>
> >>
> https://github.com/ARGOeu/nagios-plugins-activemq/blob/3d6d43c/lib/OpenWireProbe/src/org/activemq/probes/OpenWireProbe.java#L124-L126
> >> >> >.
> >> >> We got one DLQ'd OpenWire check message and just discarded it but
> when
> >> >> we got the second one I started to investigate.
> >> >>
> >> >> The message which got put into the DLQ has the propery:
> >> >> dlqDeliveryFailureCause         java.lang.Throwable: Message Expired.
> >> >> Expiration:1537468027378
> >> >>
> >> >> Our configuration related to the queue aliveness-test in activemq.xml
> >> >> is:
> >> >>
> >> >> [...]
> >> >>               <authorizationEntry
> >> >>                 queue="aliveness-test.>"
> >> >>                 read="monitoring"
> >> >>                 write="monitoring"
> >> >>                 admin="admins"
> >> >>               />
> >> >> [...]
> >> >>       <queue physicalName="aliveness-test"/>
> >> >> [...]
> >> >>           <policyEntry queue="aliveness-test">
> >> >>             <!--
> >> >>               Tell the dead letter strategy not to process expired
> >> messages
> >> >>               so that they will just be discarded instead of being
> sent
> >> to
> >> >>               the DLQ
> >> >>             -->
> >> >>             <deadLetterStrategy>
> >> >>               <sharedDeadLetterStrategy processExpired="false" />
> >> >>             </deadLetterStrategy>
> >> >>           </policyEntry>
> >> >> [...]
> >> >>
> >> >> I just noticed a bug that our TTL is set to 60ms (instead of seconds)
> >> >> but that's just how fast ActiveMQ is ; ) So I don't think that's
> >> >> related.
> >> >>
> >> >> So what we expect is for those messages to just get deleted but
> instead
> >> >> we they get DLQ'd into ActiveMQ.DLQ (since no DLQ is configured).
> >> >>
> >> >> Thanks and have a great weekend!
> >> >>
> >> >> BR,
> >> >> - Simon
> >> >>
> >> >> ____________________________________
> >> >>
> >> >> Simon Lundström
> >> >> Section for Infrastructure
> >> >>
> >> >> IT Services
> >> >> Stockholm University
> >> >> SE-106 91 Stockholm, Sweden
> >> >>
> >> >> www.su.se/english/staff-info/it
> >> >> https://www.su.se/english/about-this-website/privacy-policy
> >> >>
> >> >
> >> >On Sep 21, 2018 3:03 AM, "Simon Lundström" <simlu@xxxxx> wrote:
> >> >
> >> >Hey all!
> >> >
> >> >We're running 5.15.3 and we're using
> >> ><https://github.com/ARGOeu/nagios-plugins-activemq/> for monitoring
> >> >(SSL'ed) OpenWire. Healthchecks fail and to not get our aliveness-check
> >> >queue filled with old messages we added TTL via
> >> ><
> >> >
> >>
> https://github.com/ARGOeu/nagios-plugins-activemq/blob/3d6d43c/lib/OpenWireProbe/src/org/activemq/probes/OpenWireProbe.java#L124-L126
> >> >>.
> >> >We got one DLQ'd OpenWire check message and just discarded it but when
> >> >we got the second one I started to investigate.
> >> >
> >> >The message which got put into the DLQ has the propery:
> >> >dlqDeliveryFailureCause         java.lang.Throwable: Message Expired.
> >> >Expiration:1537468027378
> >> >
> >> >Our configuration related to the queue aliveness-test in activemq.xml
> >> >is:
> >> >
> >> >[...]
> >> >              <authorizationEntry
> >> >                queue="aliveness-test.>"
> >> >                read="monitoring"
> >> >                write="monitoring"
> >> >                admin="admins"
> >> >              />
> >> >[...]
> >> >      <queue physicalName="aliveness-test"/>
> >> >[...]
> >> >          <policyEntry queue="aliveness-test">
> >> >            <!--
> >> >              Tell the dead letter strategy not to process expired
> >> messages
> >> >              so that they will just be discarded instead of being
> sent to
> >> >              the DLQ
> >> >            -->
> >> >            <deadLetterStrategy>
> >> >              <sharedDeadLetterStrategy processExpired="false" />
> >> >            </deadLetterStrategy>
> >> >          </policyEntry>
> >> >[...]
> >> >
> >> >I just noticed a bug that our TTL is set to 60ms (instead of seconds)
> >> >but that's just how fast ActiveMQ is ; ) So I don't think that's
> >> >related.
> >> >
> >> >So what we expect is for those messages to just get deleted but instead
> >> >we they get DLQ'd into ActiveMQ.DLQ (since no DLQ is configured).
> >> >
> >> >Thanks and have a great weekend!
> >> >
> >> >BR,
> >> >- Simon
> >> >
> >> >____________________________________
> >> >
> >> >Simon Lundström
> >> >Section for Infrastructure
> >> >
> >> >IT Services
> >> >Stockholm University
> >> >SE-106 91 Stockholm, Sweden
> >> >
> >> >www.su.se/english/staff-info/it
> >> >https://www.su.se/english/about-this-website/privacy-policy
> >>
>