[Proton 0.16.0] [Dispatch Router 0.7.0] [Java Broker 6.0.4] [JMS 0.11.1] JMS sender's connection is being closed by the dispatch router

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Proton 0.16.0] [Dispatch Router 0.7.0] [Java Broker 6.0.4] [JMS 0.11.1] JMS sender's connection is being closed by the dispatch router

Adel Boutros

Hello,


As stated in a separate email, we are trying to compile Proton/Dispatch router on Solaris using GCC4.9.2. The code compiles fine and all the unit tests are green.


However, we have noticed a weird failure in our of our feature tests. The test is composed of a JMS producer (0.11.1) connected to a dispatch router which is backed by 2 Qpid Java Broker (6.0.4).

The sender is sending around 500 messages in asynchronous mode (jms.forceAsyncSend=true).

It seems that while the JMS producer is sending the message, at some random stage, the connection is dropped. The error seems to be coming from Proton which seems to be used by the Dispatch Router for AMQP connections if I am not mistaken.


I have attached the logs of the Producer(Logging in debug mode on org.apache.qpid) and Dispatch Router (Log level set to Trace + PN_TRACE_FRM=1 + PN_TRACE_RAW=1)


Could you please provide us hints or help to analyze this problem?


Regards,

Adel



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

dispatch.txt (481K) Download Attachment
jms.producer.txt (155K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Proton 0.16.0] [Dispatch Router 0.7.0] [Java Broker 6.0.4] [JMS 0.11.1] JMS sender's connection is being closed by the dispatch router

Ganesh Murthy
Hi Adel,
    Looking at the trace, there seems to be a framing error (ERROR
amqp:connection:framing-error connection aborted) but it is not immediately
clear what caused it. Sometimes, some other problem might manifest as a
framing error, so I am not really sure. Are you able to replicate this in a
non-Solaris environment?

Can you please share your qpid-jms producer source code?

Thanks.

On Wed, Aug 2, 2017 at 12:42 PM, Adel Boutros <[hidden email]> wrote:

> Hello,
>
>
> As stated in a separate email, we are trying to compile Proton/Dispatch
> router on Solaris using GCC4.9.2. The code compiles fine and all the unit
> tests are green.
>
>
> However, we have noticed a weird failure in our of our feature tests. The
> test is composed of a JMS producer (0.11.1) connected to a dispatch router
> which is backed by 2 Qpid Java Broker (6.0.4).
>
> The sender is sending around 500 messages in asynchronous mode (
> jms.forceAsyncSend=true).
>
> It seems that while the JMS producer is sending the message, at some
> random stage, the connection is dropped. The error seems to be coming from
> Proton which seems to be used by the Dispatch Router for AMQP connections
> if I am not mistaken.
>
>
> I have attached the logs of the Producer(Logging in debug mode on
> org.apache.qpid) and Dispatch Router (Log level set to Trace +
> PN_TRACE_FRM=1 + PN_TRACE_RAW=1)
>
>
> Could you please provide us hints or help to analyze this problem?
>
>
> Regards,
>
> Adel
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Proton 0.16.0] [Dispatch Router 0.7.0] [Java Broker 6.0.4] [JMS 0.11.1] JMS sender's connection is being closed by the dispatch router

Ganesh Murthy
Hi Adel,
   It seems that your error might be coming from this piece of proton code
- https://github.com/apache/qpid-proton/blob/master/proton-c/
src/core/transport.c#L2538

This means that the socket was closed while in the middle of AMQP
communication.

Thanks.

On Wed, Aug 2, 2017 at 2:57 PM, Ganesh Murthy <[hidden email]> wrote:

> Hi Adel,
>     Looking at the trace, there seems to be a framing error (ERROR
> amqp:connection:framing-error connection aborted) but it is not immediately
> clear what caused it. Sometimes, some other problem might manifest as a
> framing error, so I am not really sure. Are you able to replicate this in a
> non-Solaris environment?
>
> Can you please share your qpid-jms producer source code?
>
> Thanks.
>
> On Wed, Aug 2, 2017 at 12:42 PM, Adel Boutros <[hidden email]>
> wrote:
>
>> Hello,
>>
>>
>> As stated in a separate email, we are trying to compile Proton/Dispatch
>> router on Solaris using GCC4.9.2. The code compiles fine and all the unit
>> tests are green.
>>
>>
>> However, we have noticed a weird failure in our of our feature tests. The
>> test is composed of a JMS producer (0.11.1) connected to a dispatch router
>> which is backed by 2 Qpid Java Broker (6.0.4).
>>
>> The sender is sending around 500 messages in asynchronous mode (
>> jms.forceAsyncSend=true).
>>
>> It seems that while the JMS producer is sending the message, at some
>> random stage, the connection is dropped. The error seems to be coming from
>> Proton which seems to be used by the Dispatch Router for AMQP connections
>> if I am not mistaken.
>>
>>
>> I have attached the logs of the Producer(Logging in debug mode on
>> org.apache.qpid) and Dispatch Router (Log level set to Trace +
>> PN_TRACE_FRM=1 + PN_TRACE_RAW=1)
>>
>>
>> Could you please provide us hints or help to analyze this problem?
>>
>>
>> Regards,
>>
>> Adel
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Proton 0.16.0] [Dispatch Router 0.7.0] [Java Broker 6.0.4] [JMS 0.11.1] JMS sender's connection is being closed by the dispatch router

Adel Boutros

Indeed, we had checked that it was coming from Proton code.


I was able to reduce test case by eliminating the brokers and I am still able to reproduce the error.

So I have 1 JMS producer and 1 JMS consumer connected to a local queue of the Dispatch Router and the producer is sending asynchronously 50 000 messages (repeated 5 times). Also, in this case, the connection is being closed during the transfer.


However, I still am not able to reproduce on Linux. I tested with Proton 0.17.0 and have the same issue.


One last thing I tried was reducing the number of workerThreads from 4 to 1 and it seems I am no longer reproducing on Solaris Sparc but I still have the issue on Solaris x86. I am suspecting a multi-threaded problem here.


The last thing I tried was putting back the workerThread to 2 instead of one. Here I had a yet different behavior on Solarix x86:

    * JMS producer is receiving empty frames in an infinite loop (No sign of a connection aborted anymore)

    * A connection error is logged but the dispatch router continues working for a while before getting the empty frames as well.

I have attached the new logs for this last behavior.


Regards,

Adel


From: Ganesh Murthy <[hidden email]>
Sent: Wednesday, August 2, 2017 10:56:09 PM
To: [hidden email]
Subject: Re: [Proton 0.16.0] [Dispatch Router 0.7.0] [Java Broker 6.0.4] [JMS 0.11.1] JMS sender's connection is being closed by the dispatch router
 
Hi Adel,
   It seems that your error might be coming from this piece of proton code
- https://github.com/apache/qpid-proton/blob/master/proton-c/
src/core/transport.c#L2538

This means that the socket was closed while in the middle of AMQP
communication.

Thanks.

On Wed, Aug 2, 2017 at 2:57 PM, Ganesh Murthy <[hidden email]> wrote:

> Hi Adel,
>     Looking at the trace, there seems to be a framing error (ERROR
> amqp:connection:framing-error connection aborted) but it is not immediately
> clear what caused it. Sometimes, some other problem might manifest as a
> framing error, so I am not really sure. Are you able to replicate this in a
> non-Solaris environment?
>
> Can you please share your qpid-jms producer source code?
>
> Thanks.
>
> On Wed, Aug 2, 2017 at 12:42 PM, Adel Boutros <[hidden email]>
> wrote:
>
>> Hello,
>>
>>
>> As stated in a separate email, we are trying to compile Proton/Dispatch
>> router on Solaris using GCC4.9.2. The code compiles fine and all the unit
>> tests are green.
>>
>>
>> However, we have noticed a weird failure in our of our feature tests. The
>> test is composed of a JMS producer (0.11.1) connected to a dispatch router
>> which is backed by 2 Qpid Java Broker (6.0.4).
>>
>> The sender is sending around 500 messages in asynchronous mode (
>> jms.forceAsyncSend=true).
>>
>> It seems that while the JMS producer is sending the message, at some
>> random stage, the connection is dropped. The error seems to be coming from
>> Proton which seems to be used by the Dispatch Router for AMQP connections
>> if I am not mistaken.
>>
>>
>> I have attached the logs of the Producer(Logging in debug mode on
>> org.apache.qpid) and Dispatch Router (Log level set to Trace +
>> PN_TRACE_FRM=1 + PN_TRACE_RAW=1)
>>
>>
>> Could you please provide us hints or help to analyze this problem?
>>
>>
>> Regards,
>>
>> Adel
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

dispatch.txt (504K) Download Attachment
jms.producer.txt (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Proton 0.16.0] [Dispatch Router 0.7.0] [Java Broker 6.0.4] [JMS 0.11.1] JMS sender's connection is being closed by the dispatch router

Ganesh Murthy
In reply to this post by Adel Boutros
On Wed, Aug 2, 2017 at 12:42 PM, Adel Boutros <[hidden email]> wrote:

> Hello,
>
>
> As stated in a separate email, we are trying to compile Proton/Dispatch
> router on Solaris using GCC4.9.2. The code compiles fine and all the unit
> tests are green.
>
>
> Hi Adel,
     If the code compiles and all the unit tests are green on Solaris, can
this JIRA be marked as resolved ?
https://issues.apache.org/jira/browse/DISPATCH-738
Thanks.

> However, we have noticed a weird failure in our of our feature tests. The
> test is composed of a JMS producer (0.11.1) connected to a dispatch router
> which is backed by 2 Qpid Java Broker (6.0.4).
>
> The sender is sending around 500 messages in asynchronous mode (
> jms.forceAsyncSend=true).
>
> It seems that while the JMS producer is sending the message, at some
> random stage, the connection is dropped. The error seems to be coming from
> Proton which seems to be used by the Dispatch Router for AMQP connections
> if I am not mistaken.
>
>
> I have attached the logs of the Producer(Logging in debug mode on
> org.apache.qpid) and Dispatch Router (Log level set to Trace +
> PN_TRACE_FRM=1 + PN_TRACE_RAW=1)
>
>
> Could you please provide us hints or help to analyze this problem?
>
>
> Regards,
>
> Adel
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Proton 0.16.0] [Dispatch Router 0.7.0] [Java Broker 6.0.4] [JMS 0.11.1] JMS sender's connection is being closed by the dispatch router

Adel Boutros
Hello Ganesh,

Unfortunately no, the jira issue you are mentioning is about dispatch 0.8.0 whereas here we are talking about dispatch 0.7.0.
We tried compiling the 0.8 version with gcc4.9.2 but still had test failures.
________________________________
From: Ganesh Murthy <[hidden email]>
Sent: Thursday, August 3, 2017 10:32:25 PM
To: [hidden email]
Subject: Re: [Proton 0.16.0] [Dispatch Router 0.7.0] [Java Broker 6.0.4] [JMS 0.11.1] JMS sender's connection is being closed by the dispatch router

On Wed, Aug 2, 2017 at 12:42 PM, Adel Boutros <[hidden email]> wrote:

> Hello,
>
>
> As stated in a separate email, we are trying to compile Proton/Dispatch
> router on Solaris using GCC4.9.2. The code compiles fine and all the unit
> tests are green.
>
>
> Hi Adel,
     If the code compiles and all the unit tests are green on Solaris, can
this JIRA be marked as resolved ?
https://issues.apache.org/jira/browse/DISPATCH-738
Thanks.

> However, we have noticed a weird failure in our of our feature tests. The
> test is composed of a JMS producer (0.11.1) connected to a dispatch router
> which is backed by 2 Qpid Java Broker (6.0.4).
>
> The sender is sending around 500 messages in asynchronous mode (
> jms.forceAsyncSend=true).
>
> It seems that while the JMS producer is sending the message, at some
> random stage, the connection is dropped. The error seems to be coming from
> Proton which seems to be used by the Dispatch Router for AMQP connections
> if I am not mistaken.
>
>
> I have attached the logs of the Producer(Logging in debug mode on
> org.apache.qpid) and Dispatch Router (Log level set to Trace +
> PN_TRACE_FRM=1 + PN_TRACE_RAW=1)
>
>
> Could you please provide us hints or help to analyze this problem?
>
>
> Regards,
>
> Adel
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
Loading...