Service Bus messages are mysteriously abandoned

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Service Bus messages are mysteriously abandoned

jxz024000
Hello,

We encountered an production outage last Wednesday related to Azure Service
Bus. We have a microservice developed in Java. We use spring-jms and qpid
for Azure Service Bus and use two Topics in the service. The service is
deployed in AKS on 3 pods. On Wednesday, start from a certain time, all
three pods no longer showing receive any messages from Service Bus. But they
can still send messages to Service Bus correctly. When we checked Service
Bus side, the messages sent are redelivered multiple times and then moved to
Deadletter queue. Once we restart the service, the issue is gone.

Our first suspect is some outage on Service Bus side. But we have contact
Microsoft support. When they checked the Service Bus log, they found there
are receivers continue receive the messages but instead of call Complete to
ack the message, they call Abandon. This causes the Service Bus redelivery
the message again and in the end move them to deadletter queue after 10
redelivery.

The mystery part is that we don't find any errors in our service logs
related to the message receiving. There is no log at all indicating a
message is received and then abandoned. We use MODIFIED_FAILED as
disposition policy. Below is the full connection factory config.
connectionfactory.ServiceBus:
amqps://xxxx.servicebus.windows.net?amqp.idleTimeout=24000&jms.redeliveryPolicy.maxRedeliveries=0&jms.redeliveryPolicy.outcome=MODIFIED_FAILED&jms.prefetchPolicy.all=50&jms.receiveLocalOnly=true&transport.enabledProtocols=TLSv1.2

My question is that if it's possible for qpid to actually receive messages
and then abandon them without demarshal them to application layer?

Thanks,
Jia



--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Service Bus messages are mysteriously abandoned

Robbie Gemmell
Administrator
I dont know what 'call abandon' means in terms of AMQP protocol
behaviour, so it's hard to be sure what that description actually
implies.

I would note that you have specifically configured your client to
apply the modified-failed outcome to any message arriving with any
non-zero delivery-count set. If the server only DLQs things after 10,
then that certainly allows any messages arriving with non-zero
delivery count to be resent (either to the same or different consumer)
and 'returned' repeatedly until the server side DLQ occurs.

The client can 'return' messages to the server without ever handing
them to the application for a variety of reasons yes, such as: being
unable to decode them to determine what they say they are, being
unable to deliver them to an async consumer after determining what
they say they are, them being expired already before arrival, and in
this particular case, them having a non zero delivery count as you
have specifically configured it to do that. In all cases I can think
of, there is either logging (error, or trace/debug in 'expected'
cases), or the connection ExceptionListener is called.


On Tue, 1 Dec 2020 at 15:57, jxz024000 <[hidden email]> wrote:

>
> Hello,
>
> We encountered an production outage last Wednesday related to Azure Service
> Bus. We have a microservice developed in Java. We use spring-jms and qpid
> for Azure Service Bus and use two Topics in the service. The service is
> deployed in AKS on 3 pods. On Wednesday, start from a certain time, all
> three pods no longer showing receive any messages from Service Bus. But they
> can still send messages to Service Bus correctly. When we checked Service
> Bus side, the messages sent are redelivered multiple times and then moved to
> Deadletter queue. Once we restart the service, the issue is gone.
>
> Our first suspect is some outage on Service Bus side. But we have contact
> Microsoft support. When they checked the Service Bus log, they found there
> are receivers continue receive the messages but instead of call Complete to
> ack the message, they call Abandon. This causes the Service Bus redelivery
> the message again and in the end move them to deadletter queue after 10
> redelivery.
>
> The mystery part is that we don't find any errors in our service logs
> related to the message receiving. There is no log at all indicating a
> message is received and then abandoned. We use MODIFIED_FAILED as
> disposition policy. Below is the full connection factory config.
> connectionfactory.ServiceBus:
> amqps://xxxx.servicebus.windows.net?amqp.idleTimeout=24000&jms.redeliveryPolicy.maxRedeliveries=0&jms.redeliveryPolicy.outcome=MODIFIED_FAILED&jms.prefetchPolicy.all=50&jms.receiveLocalOnly=true&transport.enabledProtocols=TLSv1.2
>
> My question is that if it's possible for qpid to actually receive messages
> and then abandon them without demarshal them to application layer?
>
> Thanks,
> Jia
>
>
>
> --
> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]