Uneven distribution of messages

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

Uneven distribution of messages

rabih
Hello,

We are using Qpid dispatch router 1.7, Qpid broker 7.1.3 on redhat linux rhel 6.4.

Use case description:
We have a cluster of 2 dispatch routers and 2 brokers. A sharded queue "myQueue" on the 2 brokers.
Here is an illustration:

Diagram.jpg

The producer produces 2 messages, the messages are load balanced on the 2 brokers. Then, Consumer 1 and consumer 2 ask for a receive().

Our observation is that the consumer 1 consumes the first message and the consumer 2 is never getting the second message.
We are aware that the first dispatch router will do a prefetch for the 2 messages but what is weird is that the prefetched message 2 is never routed to the second dispatch router and consumer 2.
I attached the dispatch routers config.  

Do you have any idea how we can make consumer 2 receive the message?

Thanks for your help,
Rabih


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Uneven distribution of messages

rabih
Re-attached conf files with .txt extension.


On Thu, Jul 11, 2019 at 4:36 PM Rabih M <[hidden email]> wrote:
Hello,

We are using Qpid dispatch router 1.7, Qpid broker 7.1.3 on redhat linux rhel 6.4.

Use case description:
We have a cluster of 2 dispatch routers and 2 brokers. A sharded queue "myQueue" on the 2 brokers.
Here is an illustration:

Diagram.jpg

The producer produces 2 messages, the messages are load balanced on the 2 brokers. Then, Consumer 1 and consumer 2 ask for a receive().

Our observation is that the consumer 1 consumes the first message and the consumer 2 is never getting the second message.
We are aware that the first dispatch router will do a prefetch for the 2 messages but what is weird is that the prefetched message 2 is never routed to the second dispatch router and consumer 2.
I attached the dispatch routers config.  

Do you have any idea how we can make consumer 2 receive the message?

Thanks for your help,
Rabih


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

dispatch2.txt (1K) Download Attachment
dispatch1.txt (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Uneven distribution of messages

Ganesh Murthy
In reply to this post by rabih


On Thu, Jul 11, 2019 at 10:37 AM Rabih M <[hidden email]> wrote:
Hello,

We are using Qpid dispatch router 1.7, Qpid broker 7.1.3 on redhat linux rhel 6.4.

Use case description:
We have a cluster of 2 dispatch routers and 2 brokers. A sharded queue "myQueue" on the 2 brokers.
Here is an illustration:



The producer produces 2 messages, the messages are load balanced on the 2 brokers. Then, Consumer 1 and consumer 2 ask for a receive().
Does your receive() function issue one credit per call?

Our observation is that the consumer 1 consumes the first message and the consumer 2 is never getting the second message.
We are aware that the first dispatch router will do a prefetch for the 2 messages but what is weird is that the prefetched message 2 is never routed to the second dispatch router and consumer 2.
I attached the dispatch routers config.  

This is what is going on in my view -
1. The producer sends two messages - Message 1 goes to Broker 1 and Message 2 goes to Broker 2.
2. Now Consumer 1 attaches to Router 1 and calls receive() issuing one credit. (The Consumer 2 has not yet attached to Router 2). The autolinks from Broker 1 to Router 1 and Broker 2 to Router 2 each have a prefetch of 250 credits each. So,Message 1 and Message 2
both come down to Router 1. Message 1 is sent to Consumer 1. Message 2 is waiting on the outbound queue of Consumer 1 waiting to receive a credit from Consumer 1.
3. Now Consumer 2 shows up and wants to receive Message 2 but that message is already queued up to go to Consumer 1 who is still keeping his connection open
4. If now Consumer 1 drops out, the Message 2 will be RELEASED back to its broker and that message will be sent to Consumer 2.

Have you tried attaching Consumer 1 and Consumer 2 first and then subsequently bringing up the Producer 1 and send two messages ? In that case, I think, both consumers will receive one message.

Thanks.
Do you have any idea how we can make consumer 2 receive the message?

Thanks for your help,
Rabih

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Uneven distribution of messages

rabih
Hello,

Thank you Ganesh for your answer.
To answer your first question yes the consumers we have, issue one credit per call.
For the the second question, yes we are attaching consumers before sending the messages.

After setting the dispatch "link capacity" on the listeners to 1, we managed to have an equal load-balancing of the messages between consumer 1 and 2.
Our understanding is that the problem comes from the prefetch at the level of the "normal" listener, which is preventing the "inter-router" listener to pass messages to the second dispatch.

You can find attached the new config files.

Is this the correct way to resolve this kind of problem? Does it sound reasonable to you?

Best regards,
Rabih and Ali

On Thu, Jul 11, 2019 at 5:50 PM Ganesh Murthy <[hidden email]> wrote:


On Thu, Jul 11, 2019 at 10:37 AM Rabih M <[hidden email]> wrote:
Hello,

We are using Qpid dispatch router 1.7, Qpid broker 7.1.3 on redhat linux rhel 6.4.

Use case description:
We have a cluster of 2 dispatch routers and 2 brokers. A sharded queue "myQueue" on the 2 brokers.
Here is an illustration:

Diagram.jpg

The producer produces 2 messages, the messages are load balanced on the 2 brokers. Then, Consumer 1 and consumer 2 ask for a receive().
Does your receive() function issue one credit per call?

Our observation is that the consumer 1 consumes the first message and the consumer 2 is never getting the second message.
We are aware that the first dispatch router will do a prefetch for the 2 messages but what is weird is that the prefetched message 2 is never routed to the second dispatch router and consumer 2.
I attached the dispatch routers config.  

This is what is going on in my view -
1. The producer sends two messages - Message 1 goes to Broker 1 and Message 2 goes to Broker 2.
2. Now Consumer 1 attaches to Router 1 and calls receive() issuing one credit. (The Consumer 2 has not yet attached to Router 2). The autolinks from Broker 1 to Router 1 and Broker 2 to Router 2 each have a prefetch of 250 credits each. So,Message 1 and Message 2
both come down to Router 1. Message 1 is sent to Consumer 1. Message 2 is waiting on the outbound queue of Consumer 1 waiting to receive a credit from Consumer 1.
3. Now Consumer 2 shows up and wants to receive Message 2 but that message is already queued up to go to Consumer 1 who is still keeping his connection open
4. If now Consumer 1 drops out, the Message 2 will be RELEASED back to its broker and that message will be sent to Consumer 2.

Have you tried attaching Consumer 1 and Consumer 2 first and then subsequently bringing up the Producer 1 and send two messages ? In that case, I think, both consumers will receive one message.

Thanks.
Do you have any idea how we can make consumer 2 receive the message?

Thanks for your help,
Rabih

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

dispatch1.txt (2K) Download Attachment
dispatch2.txt (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Uneven distribution of messages

Ganesh Murthy
On Fri, Jul 12, 2019 at 11:29 AM Rabih M <[hidden email]> wrote:

> Hello,
>
> Thank you Ganesh for your answer.
> To answer your first question yes the consumers we have, issue one credit
> per call.
> For the the second question, yes we are attaching consumers before sending
> the messages.
>
> After setting the dispatch "link capacity" on the listeners to 1, we
> managed to have an equal load-balancing of the messages between consumer 1
> and 2.
> Our understanding is that the problem comes from the prefetch at the level
> of the "normal" listener, which is preventing the "inter-router" listener
> to pass messages to the second dispatch.
>
Just want to note here that the linkCapacity on the inter-router listeners
are ignored. So setting linkCapacity to 1 on that listener has no effect.

This is what I think is happening
1. The Router 1 (R1) and Router 2 (R2) are started and each of them connect
to both Brokers, B1 and B2 and establish the autolinks. As soon as inbound
autoLinks (inbound to the router from the broker) are established on
address myQueue, the broker(s) immediately provides initial credit on both
autoLinks. This initial credit is usually 500 or 1000 depending on the
broker you are using.
2. Now Consumer 1 (C1) connects to R1 and Consumer 2 (C2) connects to R2
providing one credit each respectively to each router.
3. Now the Producer connects to R1 and sends one message M1 (it cannot send
more than one message because the linkCapacity is set to 1). Assuming M1
goes to B1 (it has an equal chance of ending up in B2 as well), B1  sees
that there are 2 consumers (autolinks) for that message.Say B1 sends M1  to
R1, M1 will be sent to C1.
4. Producer produces M2 and it goes to B2. B2 might send it to R1 or R2.
         4a. If B2 sends M2 to R1, the message might go over the
inter-router link  to R2 to C2 because there is no credit to send to C1.
         4b. If B2 sends M2 to R2, then C2 gets M2  because routers prefer
local consumers to remote consumers.
          4c. If C1 is fast enough to consume M1 and replenish the credit
and if B2 ends up sending M2 to R1, C1 might end up getting the second
message as well.
          4d. You need to also find out if the brokers load balances across
autolinks.

In conclusion, you are lucky that the messages are being load balanced
across C1 and C2 when you changed linkCapacity to 1 but it is making it
more likely. This might not always happen.

Turn on trace logging on the routers to see which route the messages are
taking. Try the scenario over and over again.

To turn on trace logging add the following to the router config file -

log {
    module: DEFAULT
    enable: trace+
    output: path/to/qdrouterd.log
}

If you want true load balancing between the consumers, put a router R3 in
front of R1 and R2 and connect both consumers  to R3. The producer can
connect to any router.

>
> You can find attached the new config files.
>
> Is this the correct way to resolve this kind of problem? Does it sound
> reasonable to you?
>
> Best regards,
> Rabih and Ali
>
> On Thu, Jul 11, 2019 at 5:50 PM Ganesh Murthy <[hidden email]> wrote:
>
>>
>>
>> On Thu, Jul 11, 2019 at 10:37 AM Rabih M <[hidden email]> wrote:
>>
>>> Hello,
>>>
>>> We are using Qpid dispatch router 1.7, Qpid broker 7.1.3 on redhat linux
>>> rhel 6.4.
>>>
>>> Use case description:
>>> We have a cluster of 2 dispatch routers and 2 brokers. A sharded queue
>>> "myQueue" on the 2 brokers.
>>> Here is an illustration:
>>>
>>> [image: Diagram.jpg]
>>>
>>> The producer produces 2 messages, the messages are load balanced on the
>>> 2 brokers. Then, Consumer 1 and consumer 2 ask for a receive().
>>>
>> Does your receive() function issue one credit per call?
>>
>>>
>>> Our observation is that the consumer 1 consumes the first message and
>>> the consumer 2 is never getting the second message.
>>> We are aware that the first dispatch router will do a prefetch for the 2
>>> messages but what is weird is that the prefetched message 2 is never routed
>>> to the second dispatch router and consumer 2.
>>> I attached the dispatch routers config.
>>>
>>> This is what is going on in my view -
>> 1. The producer sends two messages - Message 1 goes to Broker 1 and
>> Message 2 goes to Broker 2.
>> 2. Now Consumer 1 attaches to Router 1 and calls receive() issuing one
>> credit. (The Consumer 2 has not yet attached to Router 2). The autolinks
>> from Broker 1 to Router 1 and Broker 2 to Router 2 each have a prefetch of
>> 250 credits each. So,Message 1 and Message 2
>> both come down to Router 1. Message 1 is sent to Consumer 1. Message 2 is
>> waiting on the outbound queue of Consumer 1 waiting to receive a credit
>> from Consumer 1.
>> 3. Now Consumer 2 shows up and wants to receive Message 2 but that
>> message is already queued up to go to Consumer 1 who is still keeping his
>> connection open
>> 4. If now Consumer 1 drops out, the Message 2 will be RELEASED back to
>> its broker and that message will be sent to Consumer 2.
>>
>> Have you tried attaching Consumer 1 and Consumer 2 first and then
>> subsequently bringing up the Producer 1 and send two messages ? In that
>> case, I think, both consumers will receive one message.
>>
>> Thanks.
>>
>>> Do you have any idea how we can make consumer 2 receive the message?
>>>
>>> Thanks for your help,
>>> Rabih
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Uneven distribution of messages

rabih
Hello,

We are testing with the trace+ log. We will brief you of the results when
they are ready.

Our goal is not to load balance equally the messages between the consumers
but we would like the dispatch router to send the message to the free
consumer.
Specially if the consumer does a long calculation, we do not want to block
the message in the outbound queue knowing there is another idle consumer.

Is there any special configuration for this?

Best regards,
Rabih


On Fri, Jul 12, 2019 at 7:26 PM Ganesh Murthy <[hidden email]> wrote:

> On Fri, Jul 12, 2019 at 11:29 AM Rabih M <[hidden email]> wrote:
>
> > Hello,
> >
> > Thank you Ganesh for your answer.
> > To answer your first question yes the consumers we have, issue one credit
> > per call.
> > For the the second question, yes we are attaching consumers before
> sending
> > the messages.
> >
> > After setting the dispatch "link capacity" on the listeners to 1, we
> > managed to have an equal load-balancing of the messages between consumer
> 1
> > and 2.
> > Our understanding is that the problem comes from the prefetch at the
> level
> > of the "normal" listener, which is preventing the "inter-router" listener
> > to pass messages to the second dispatch.
> >
> Just want to note here that the linkCapacity on the inter-router listeners
> are ignored. So setting linkCapacity to 1 on that listener has no effect.
>
> This is what I think is happening
> 1. The Router 1 (R1) and Router 2 (R2) are started and each of them connect
> to both Brokers, B1 and B2 and establish the autolinks. As soon as inbound
> autoLinks (inbound to the router from the broker) are established on
> address myQueue, the broker(s) immediately provides initial credit on both
> autoLinks. This initial credit is usually 500 or 1000 depending on the
> broker you are using.
> 2. Now Consumer 1 (C1) connects to R1 and Consumer 2 (C2) connects to R2
> providing one credit each respectively to each router.
> 3. Now the Producer connects to R1 and sends one message M1 (it cannot send
> more than one message because the linkCapacity is set to 1). Assuming M1
> goes to B1 (it has an equal chance of ending up in B2 as well), B1  sees
> that there are 2 consumers (autolinks) for that message.Say B1 sends M1  to
> R1, M1 will be sent to C1.
> 4. Producer produces M2 and it goes to B2. B2 might send it to R1 or R2.
>          4a. If B2 sends M2 to R1, the message might go over the
> inter-router link  to R2 to C2 because there is no credit to send to C1.
>          4b. If B2 sends M2 to R2, then C2 gets M2  because routers prefer
> local consumers to remote consumers.
>           4c. If C1 is fast enough to consume M1 and replenish the credit
> and if B2 ends up sending M2 to R1, C1 might end up getting the second
> message as well.
>           4d. You need to also find out if the brokers load balances across
> autolinks.
>
> In conclusion, you are lucky that the messages are being load balanced
> across C1 and C2 when you changed linkCapacity to 1 but it is making it
> more likely. This might not always happen.
>
> Turn on trace logging on the routers to see which route the messages are
> taking. Try the scenario over and over again.
>
> To turn on trace logging add the following to the router config file -
>
> log {
>     module: DEFAULT
>     enable: trace+
>     output: path/to/qdrouterd.log
> }
>
> If you want true load balancing between the consumers, put a router R3 in
> front of R1 and R2 and connect both consumers  to R3. The producer can
> connect to any router.
>
> >
> > You can find attached the new config files.
> >
> > Is this the correct way to resolve this kind of problem? Does it sound
> > reasonable to you?
> >
> > Best regards,
> > Rabih and Ali
> >
> > On Thu, Jul 11, 2019 at 5:50 PM Ganesh Murthy <[hidden email]>
> wrote:
> >
> >>
> >>
> >> On Thu, Jul 11, 2019 at 10:37 AM Rabih M <[hidden email]>
> wrote:
> >>
> >>> Hello,
> >>>
> >>> We are using Qpid dispatch router 1.7, Qpid broker 7.1.3 on redhat
> linux
> >>> rhel 6.4.
> >>>
> >>> Use case description:
> >>> We have a cluster of 2 dispatch routers and 2 brokers. A sharded queue
> >>> "myQueue" on the 2 brokers.
> >>> Here is an illustration:
> >>>
> >>> [image: Diagram.jpg]
> >>>
> >>> The producer produces 2 messages, the messages are load balanced on the
> >>> 2 brokers. Then, Consumer 1 and consumer 2 ask for a receive().
> >>>
> >> Does your receive() function issue one credit per call?
> >>
> >>>
> >>> Our observation is that the consumer 1 consumes the first message and
> >>> the consumer 2 is never getting the second message.
> >>> We are aware that the first dispatch router will do a prefetch for the
> 2
> >>> messages but what is weird is that the prefetched message 2 is never
> routed
> >>> to the second dispatch router and consumer 2.
> >>> I attached the dispatch routers config.
> >>>
> >>> This is what is going on in my view -
> >> 1. The producer sends two messages - Message 1 goes to Broker 1 and
> >> Message 2 goes to Broker 2.
> >> 2. Now Consumer 1 attaches to Router 1 and calls receive() issuing one
> >> credit. (The Consumer 2 has not yet attached to Router 2). The autolinks
> >> from Broker 1 to Router 1 and Broker 2 to Router 2 each have a prefetch
> of
> >> 250 credits each. So,Message 1 and Message 2
> >> both come down to Router 1. Message 1 is sent to Consumer 1. Message 2
> is
> >> waiting on the outbound queue of Consumer 1 waiting to receive a credit
> >> from Consumer 1.
> >> 3. Now Consumer 2 shows up and wants to receive Message 2 but that
> >> message is already queued up to go to Consumer 1 who is still keeping
> his
> >> connection open
> >> 4. If now Consumer 1 drops out, the Message 2 will be RELEASED back to
> >> its broker and that message will be sent to Consumer 2.
> >>
> >> Have you tried attaching Consumer 1 and Consumer 2 first and then
> >> subsequently bringing up the Producer 1 and send two messages ? In that
> >> case, I think, both consumers will receive one message.
> >>
> >> Thanks.
> >>
> >>> Do you have any idea how we can make consumer 2 receive the message?
> >>>
> >>> Thanks for your help,
> >>> Rabih
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: [hidden email]
> >>> For additional commands, e-mail: [hidden email]
> >>
> >>
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
>
Reply | Threaded
Open this post in threaded view
|

Re: Uneven distribution of messages

Ganesh Murthy
On Mon, Jul 15, 2019 at 1:26 PM Rabih M <[hidden email]> wrote:

> Hello,
>
> We are testing with the trace+ log. We will brief you of the results when
> they are ready.
>
> Our goal is not to load balance equally the messages between the consumers
> but we would like the dispatch router to send the message to the free
> consumer.
>

What you did (setting the linkCapacity on the producer and consumer
listeners to 1) is the right thing to do if you want the router
to send the message to *any* free consumer.

If Consumer C1 is processing the first message and does not issue the next
credit until it finishes processing the first message and if the second
message arrives to Router 1 when C1 is
still processing the message, then Router 1 will definitely forward the
message to Router 2. BUT if C1 is fast enough and ends up processing the
first message and immediately issues credit, the second message if it
arrives in Router 1 will also
be sent to C1 (because the router prefers local consumers).

Remember that the key here is which Router ends up getting the message
since each broker has two autoLinks to both routers and we don't know which
autoLink the broker will choose.

But overall, with your new configuration, the router will send the message
to a consumer that is not busy no further configuration is necessary.

Specially if the consumer does a long calculation, we do not want to block

> the message in the outbound queue knowing there is another idle consumer.
>
> Is there any special configuration for this?
>
> Best regards,
> Rabih
>
>
> On Fri, Jul 12, 2019 at 7:26 PM Ganesh Murthy <[hidden email]> wrote:
>
> > On Fri, Jul 12, 2019 at 11:29 AM Rabih M <[hidden email]>
> wrote:
> >
> > > Hello,
> > >
> > > Thank you Ganesh for your answer.
> > > To answer your first question yes the consumers we have, issue one
> credit
> > > per call.
> > > For the the second question, yes we are attaching consumers before
> > sending
> > > the messages.
> > >
> > > After setting the dispatch "link capacity" on the listeners to 1, we
> > > managed to have an equal load-balancing of the messages between
> consumer
> > 1
> > > and 2.
> > > Our understanding is that the problem comes from the prefetch at the
> > level
> > > of the "normal" listener, which is preventing the "inter-router"
> listener
> > > to pass messages to the second dispatch.
> > >
> > Just want to note here that the linkCapacity on the inter-router
> listeners
> > are ignored. So setting linkCapacity to 1 on that listener has no effect.
> >
> > This is what I think is happening
> > 1. The Router 1 (R1) and Router 2 (R2) are started and each of them
> connect
> > to both Brokers, B1 and B2 and establish the autolinks. As soon as
> inbound
> > autoLinks (inbound to the router from the broker) are established on
> > address myQueue, the broker(s) immediately provides initial credit on
> both
> > autoLinks. This initial credit is usually 500 or 1000 depending on the
> > broker you are using.
> > 2. Now Consumer 1 (C1) connects to R1 and Consumer 2 (C2) connects to R2
> > providing one credit each respectively to each router.
> > 3. Now the Producer connects to R1 and sends one message M1 (it cannot
> send
> > more than one message because the linkCapacity is set to 1). Assuming M1
> > goes to B1 (it has an equal chance of ending up in B2 as well), B1  sees
> > that there are 2 consumers (autolinks) for that message.Say B1 sends M1
> to
> > R1, M1 will be sent to C1.
> > 4. Producer produces M2 and it goes to B2. B2 might send it to R1 or R2.
> >          4a. If B2 sends M2 to R1, the message might go over the
> > inter-router link  to R2 to C2 because there is no credit to send to C1.
> >          4b. If B2 sends M2 to R2, then C2 gets M2  because routers
> prefer
> > local consumers to remote consumers.
> >           4c. If C1 is fast enough to consume M1 and replenish the credit
> > and if B2 ends up sending M2 to R1, C1 might end up getting the second
> > message as well.
> >           4d. You need to also find out if the brokers load balances
> across
> > autolinks.
> >
> > In conclusion, you are lucky that the messages are being load balanced
> > across C1 and C2 when you changed linkCapacity to 1 but it is making it
> > more likely. This might not always happen.
> >
> > Turn on trace logging on the routers to see which route the messages are
> > taking. Try the scenario over and over again.
> >
> > To turn on trace logging add the following to the router config file -
> >
> > log {
> >     module: DEFAULT
> >     enable: trace+
> >     output: path/to/qdrouterd.log
> > }
> >
> > If you want true load balancing between the consumers, put a router R3 in
> > front of R1 and R2 and connect both consumers  to R3. The producer can
> > connect to any router.
> >
> > >
> > > You can find attached the new config files.
> > >
> > > Is this the correct way to resolve this kind of problem? Does it sound
> > > reasonable to you?
> > >
> > > Best regards,
> > > Rabih and Ali
> > >
> > > On Thu, Jul 11, 2019 at 5:50 PM Ganesh Murthy <[hidden email]>
> > wrote:
> > >
> > >>
> > >>
> > >> On Thu, Jul 11, 2019 at 10:37 AM Rabih M <[hidden email]>
> > wrote:
> > >>
> > >>> Hello,
> > >>>
> > >>> We are using Qpid dispatch router 1.7, Qpid broker 7.1.3 on redhat
> > linux
> > >>> rhel 6.4.
> > >>>
> > >>> Use case description:
> > >>> We have a cluster of 2 dispatch routers and 2 brokers. A sharded
> queue
> > >>> "myQueue" on the 2 brokers.
> > >>> Here is an illustration:
> > >>>
> > >>> [image: Diagram.jpg]
> > >>>
> > >>> The producer produces 2 messages, the messages are load balanced on
> the
> > >>> 2 brokers. Then, Consumer 1 and consumer 2 ask for a receive().
> > >>>
> > >> Does your receive() function issue one credit per call?
> > >>
> > >>>
> > >>> Our observation is that the consumer 1 consumes the first message and
> > >>> the consumer 2 is never getting the second message.
> > >>> We are aware that the first dispatch router will do a prefetch for
> the
> > 2
> > >>> messages but what is weird is that the prefetched message 2 is never
> > routed
> > >>> to the second dispatch router and consumer 2.
> > >>> I attached the dispatch routers config.
> > >>>
> > >>> This is what is going on in my view -
> > >> 1. The producer sends two messages - Message 1 goes to Broker 1 and
> > >> Message 2 goes to Broker 2.
> > >> 2. Now Consumer 1 attaches to Router 1 and calls receive() issuing one
> > >> credit. (The Consumer 2 has not yet attached to Router 2). The
> autolinks
> > >> from Broker 1 to Router 1 and Broker 2 to Router 2 each have a
> prefetch
> > of
> > >> 250 credits each. So,Message 1 and Message 2
> > >> both come down to Router 1. Message 1 is sent to Consumer 1. Message 2
> > is
> > >> waiting on the outbound queue of Consumer 1 waiting to receive a
> credit
> > >> from Consumer 1.
> > >> 3. Now Consumer 2 shows up and wants to receive Message 2 but that
> > >> message is already queued up to go to Consumer 1 who is still keeping
> > his
> > >> connection open
> > >> 4. If now Consumer 1 drops out, the Message 2 will be RELEASED back to
> > >> its broker and that message will be sent to Consumer 2.
> > >>
> > >> Have you tried attaching Consumer 1 and Consumer 2 first and then
> > >> subsequently bringing up the Producer 1 and send two messages ? In
> that
> > >> case, I think, both consumers will receive one message.
> > >>
> > >> Thanks.
> > >>
> > >>> Do you have any idea how we can make consumer 2 receive the message?
> > >>>
> > >>> Thanks for your help,
> > >>> Rabih
> > >>>
> > >>> ---------------------------------------------------------------------
> > >>> To unsubscribe, e-mail: [hidden email]
> > >>> For additional commands, e-mail: [hidden email]
> > >>
> > >>
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: [hidden email]
> > > For additional commands, e-mail: [hidden email]
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Uneven distribution of messages

rabih
Hello,

We tested with LinkCapacity equal to 1 on the "normal" listener with debug level trace+, here are our findings:
Our Cluster:
Diagram.jpg
We are using the broker-j, the consumer are connected to the dispatch routers before we start sending.

For use case 1: 
1- the producer sends a message.
2- Consumer 1 issues one credit, receives the message without acknowledging.
3- the producer sends another message.
4- Consumer 2 in auto-ack mode issues one credit and receives the message.
5- we repeated steps 3 and 4 ten times.
6- Consumer 1 acknowledges.  
The results were correct: all the messages were correctly distributed to the idle consumer.

For use case 2:
1- the producer send 10 messages while no credits were issued yet by the consumers.
2- Consumer 1 issues one credit, receives a message without acknowledging.
3- Consumer 2 in auto-ack mode issues one credit and times out after 5 seconds if nothing is received.
4- we repeated step 3 eight times.
5- Consumer 1 acknowledges.
The results were not as expected: 4 messages were blocked in the outbound queue of the consumer 1 and consumer 2 was able to receive only 5 messages.
We analysed the traces to follow the messages. We found that 4 messages were blocked in the dispatch 1. 
Conclusion: if no consumers are issuing credits (are busy) then the incoming messages will be pre-assigned automatically by the dispatch router to the listeners (in a round robin way?).

Is it an expected behavior in the dispatch router? is it not supposed to wait for a credit to be issued before binding the message to an outbound queue?
If you want to check the paths of the messages, I attached the routers logs of the use case 2.

Best regards,
Rabih

On Mon, Jul 15, 2019 at 8:14 PM Ganesh Murthy <[hidden email]> wrote:
On Mon, Jul 15, 2019 at 1:26 PM Rabih M <[hidden email]> wrote:

> Hello,
>
> We are testing with the trace+ log. We will brief you of the results when
> they are ready.
>
> Our goal is not to load balance equally the messages between the consumers
> but we would like the dispatch router to send the message to the free
> consumer.
>

What you did (setting the linkCapacity on the producer and consumer
listeners to 1) is the right thing to do if you want the router
to send the message to *any* free consumer.

If Consumer C1 is processing the first message and does not issue the next
credit until it finishes processing the first message and if the second
message arrives to Router 1 when C1 is
still processing the message, then Router 1 will definitely forward the
message to Router 2. BUT if C1 is fast enough and ends up processing the
first message and immediately issues credit, the second message if it
arrives in Router 1 will also
be sent to C1 (because the router prefers local consumers).

Remember that the key here is which Router ends up getting the message
since each broker has two autoLinks to both routers and we don't know which
autoLink the broker will choose.

But overall, with your new configuration, the router will send the message
to a consumer that is not busy no further configuration is necessary.

Specially if the consumer does a long calculation, we do not want to block
> the message in the outbound queue knowing there is another idle consumer.
>
> Is there any special configuration for this?
>
> Best regards,
> Rabih
>
>
> On Fri, Jul 12, 2019 at 7:26 PM Ganesh Murthy <[hidden email]> wrote:
>
> > On Fri, Jul 12, 2019 at 11:29 AM Rabih M <[hidden email]>
> wrote:
> >
> > > Hello,
> > >
> > > Thank you Ganesh for your answer.
> > > To answer your first question yes the consumers we have, issue one
> credit
> > > per call.
> > > For the the second question, yes we are attaching consumers before
> > sending
> > > the messages.
> > >
> > > After setting the dispatch "link capacity" on the listeners to 1, we
> > > managed to have an equal load-balancing of the messages between
> consumer
> > 1
> > > and 2.
> > > Our understanding is that the problem comes from the prefetch at the
> > level
> > > of the "normal" listener, which is preventing the "inter-router"
> listener
> > > to pass messages to the second dispatch.
> > >
> > Just want to note here that the linkCapacity on the inter-router
> listeners
> > are ignored. So setting linkCapacity to 1 on that listener has no effect.
> >
> > This is what I think is happening
> > 1. The Router 1 (R1) and Router 2 (R2) are started and each of them
> connect
> > to both Brokers, B1 and B2 and establish the autolinks. As soon as
> inbound
> > autoLinks (inbound to the router from the broker) are established on
> > address myQueue, the broker(s) immediately provides initial credit on
> both
> > autoLinks. This initial credit is usually 500 or 1000 depending on the
> > broker you are using.
> > 2. Now Consumer 1 (C1) connects to R1 and Consumer 2 (C2) connects to R2
> > providing one credit each respectively to each router.
> > 3. Now the Producer connects to R1 and sends one message M1 (it cannot
> send
> > more than one message because the linkCapacity is set to 1). Assuming M1
> > goes to B1 (it has an equal chance of ending up in B2 as well), B1  sees
> > that there are 2 consumers (autolinks) for that message.Say B1 sends M1
> to
> > R1, M1 will be sent to C1.
> > 4. Producer produces M2 and it goes to B2. B2 might send it to R1 or R2.
> >          4a. If B2 sends M2 to R1, the message might go over the
> > inter-router link  to R2 to C2 because there is no credit to send to C1.
> >          4b. If B2 sends M2 to R2, then C2 gets M2  because routers
> prefer
> > local consumers to remote consumers.
> >           4c. If C1 is fast enough to consume M1 and replenish the credit
> > and if B2 ends up sending M2 to R1, C1 might end up getting the second
> > message as well.
> >           4d. You need to also find out if the brokers load balances
> across
> > autolinks.
> >
> > In conclusion, you are lucky that the messages are being load balanced
> > across C1 and C2 when you changed linkCapacity to 1 but it is making it
> > more likely. This might not always happen.
> >
> > Turn on trace logging on the routers to see which route the messages are
> > taking. Try the scenario over and over again.
> >
> > To turn on trace logging add the following to the router config file -
> >
> > log {
> >     module: DEFAULT
> >     enable: trace+
> >     output: path/to/qdrouterd.log
> > }
> >
> > If you want true load balancing between the consumers, put a router R3 in
> > front of R1 and R2 and connect both consumers  to R3. The producer can
> > connect to any router.
> >
> > >
> > > You can find attached the new config files.
> > >
> > > Is this the correct way to resolve this kind of problem? Does it sound
> > > reasonable to you?
> > >
> > > Best regards,
> > > Rabih and Ali
> > >
> > > On Thu, Jul 11, 2019 at 5:50 PM Ganesh Murthy <[hidden email]>
> > wrote:
> > >
> > >>
> > >>
> > >> On Thu, Jul 11, 2019 at 10:37 AM Rabih M <[hidden email]>
> > wrote:
> > >>
> > >>> Hello,
> > >>>
> > >>> We are using Qpid dispatch router 1.7, Qpid broker 7.1.3 on redhat
> > linux
> > >>> rhel 6.4.
> > >>>
> > >>> Use case description:
> > >>> We have a cluster of 2 dispatch routers and 2 brokers. A sharded
> queue
> > >>> "myQueue" on the 2 brokers.
> > >>> Here is an illustration:
> > >>>
> > >>> [image: Diagram.jpg]
> > >>>
> > >>> The producer produces 2 messages, the messages are load balanced on
> the
> > >>> 2 brokers. Then, Consumer 1 and consumer 2 ask for a receive().
> > >>>
> > >> Does your receive() function issue one credit per call?
> > >>
> > >>>
> > >>> Our observation is that the consumer 1 consumes the first message and
> > >>> the consumer 2 is never getting the second message.
> > >>> We are aware that the first dispatch router will do a prefetch for
> the
> > 2
> > >>> messages but what is weird is that the prefetched message 2 is never
> > routed
> > >>> to the second dispatch router and consumer 2.
> > >>> I attached the dispatch routers config.
> > >>>
> > >>> This is what is going on in my view -
> > >> 1. The producer sends two messages - Message 1 goes to Broker 1 and
> > >> Message 2 goes to Broker 2.
> > >> 2. Now Consumer 1 attaches to Router 1 and calls receive() issuing one
> > >> credit. (The Consumer 2 has not yet attached to Router 2). The
> autolinks
> > >> from Broker 1 to Router 1 and Broker 2 to Router 2 each have a
> prefetch
> > of
> > >> 250 credits each. So,Message 1 and Message 2
> > >> both come down to Router 1. Message 1 is sent to Consumer 1. Message 2
> > is
> > >> waiting on the outbound queue of Consumer 1 waiting to receive a
> credit
> > >> from Consumer 1.
> > >> 3. Now Consumer 2 shows up and wants to receive Message 2 but that
> > >> message is already queued up to go to Consumer 1 who is still keeping
> > his
> > >> connection open
> > >> 4. If now Consumer 1 drops out, the Message 2 will be RELEASED back to
> > >> its broker and that message will be sent to Consumer 2.
> > >>
> > >> Have you tried attaching Consumer 1 and Consumer 2 first and then
> > >> subsequently bringing up the Producer 1 and send two messages ? In
> that
> > >> case, I think, both consumers will receive one message.
> > >>
> > >> Thanks.
> > >>
> > >>> Do you have any idea how we can make consumer 2 receive the message?
> > >>>
> > >>> Thanks for your help,
> > >>> Rabih
> > >>>
> > >>> ---------------------------------------------------------------------
> > >>> To unsubscribe, e-mail: [hidden email]
> > >>> For additional commands, e-mail: [hidden email]
> > >>
> > >>
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: [hidden email]
> > > For additional commands, e-mail: [hidden email]
> >
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Uneven distribution of messages

Ted Ross


On Wed, Jul 17, 2019 at 12:00 PM Rabih M <[hidden email]> wrote:
Hello,

We tested with LinkCapacity equal to 1 on the "normal" listener with debug level trace+, here are our findings:
Our Cluster:

We are using the broker-j, the consumer are connected to the dispatch routers before we start sending.

For use case 1: 
1- the producer sends a message.
2- Consumer 1 issues one credit, receives the message without acknowledging.
3- the producer sends another message.
4- Consumer 2 in auto-ack mode issues one credit and receives the message.
5- we repeated steps 3 and 4 ten times.
6- Consumer 1 acknowledges.  
The results were correct: all the messages were correctly distributed to the idle consumer.

For use case 2:
1- the producer send 10 messages while no credits were issued yet by the consumers.
2- Consumer 1 issues one credit, receives a message without acknowledging.
3- Consumer 2 in auto-ack mode issues one credit and times out after 5 seconds if nothing is received.
4- we repeated step 3 eight times.
5- Consumer 1 acknowledges.
The results were not as expected: 4 messages were blocked in the outbound queue of the consumer 1 and consumer 2 was able to receive only 5 messages.
We analysed the traces to follow the messages. We found that 4 messages were blocked in the dispatch 1. 
Conclusion: if no consumers are issuing credits (are busy) then the incoming messages will be pre-assigned automatically by the dispatch router to the listeners (in a round robin way?).

Is it an expected behavior in the dispatch router? is it not supposed to wait for a credit to be issued before binding the message to an outbound queue?

Yes, this is the expected behavior.  The router does not propagate each individual credit from receiver to sender.  It would be impractical to do so, would not scale well, and probably still wouldn't provide the behavior you expect.  What the router does is to use delivery settlement as the way to control credit flow to producers.  If the link capacity is 250, each producer will be limited to 250 unsettled deliveries at any time.  As the deliveries are settled, more credit is issued.  This scales well in large networks, keeps a limit on the memory consumed by deliveries, and allows for high delivery rates.

Is this a real use case or are you experimenting to learn how the router works?

Under steady state flow, the messages will be delivered to the consumers in proportion to the rate at which the consumers acknowledge (settle) deliveries.  If a consumer attaches a receiving link but withholds credit, the router network will route deliveries to that consumer in anticipation of credit being issued.  It is an anti-pattern to attach a receiving link and stop issuing credit.  Credit should be used to control the rate of delivery.  If you want to stop delivery, detach the receiver.

If you really want a completely synchronous transfer across your network, you can set the link capacity on the broker connections to 1.  This will limit the number of in-flight unsettled deliveries on each incoming auto-link to 1.  It will be slow and will be prone to stalling, especially if your consumers withhold acknowledgement.

If you want to check the paths of the messages, I attached the routers logs of the use case 2.

Best regards,
Rabih

On Mon, Jul 15, 2019 at 8:14 PM Ganesh Murthy <[hidden email]> wrote:
On Mon, Jul 15, 2019 at 1:26 PM Rabih M <[hidden email]> wrote:

> Hello,
>
> We are testing with the trace+ log. We will brief you of the results when
> they are ready.
>
> Our goal is not to load balance equally the messages between the consumers
> but we would like the dispatch router to send the message to the free
> consumer.
>

What you did (setting the linkCapacity on the producer and consumer
listeners to 1) is the right thing to do if you want the router
to send the message to *any* free consumer.

If Consumer C1 is processing the first message and does not issue the next
credit until it finishes processing the first message and if the second
message arrives to Router 1 when C1 is
still processing the message, then Router 1 will definitely forward the
message to Router 2. BUT if C1 is fast enough and ends up processing the
first message and immediately issues credit, the second message if it
arrives in Router 1 will also
be sent to C1 (because the router prefers local consumers).

Remember that the key here is which Router ends up getting the message
since each broker has two autoLinks to both routers and we don't know which
autoLink the broker will choose.

But overall, with your new configuration, the router will send the message
to a consumer that is not busy no further configuration is necessary.

Specially if the consumer does a long calculation, we do not want to block
> the message in the outbound queue knowing there is another idle consumer.
>
> Is there any special configuration for this?
>
> Best regards,
> Rabih
>
>
> On Fri, Jul 12, 2019 at 7:26 PM Ganesh Murthy <[hidden email]> wrote:
>
> > On Fri, Jul 12, 2019 at 11:29 AM Rabih M <[hidden email]>
> wrote:
> >
> > > Hello,
> > >
> > > Thank you Ganesh for your answer.
> > > To answer your first question yes the consumers we have, issue one
> credit
> > > per call.
> > > For the the second question, yes we are attaching consumers before
> > sending
> > > the messages.
> > >
> > > After setting the dispatch "link capacity" on the listeners to 1, we
> > > managed to have an equal load-balancing of the messages between
> consumer
> > 1
> > > and 2.
> > > Our understanding is that the problem comes from the prefetch at the
> > level
> > > of the "normal" listener, which is preventing the "inter-router"
> listener
> > > to pass messages to the second dispatch.
> > >
> > Just want to note here that the linkCapacity on the inter-router
> listeners
> > are ignored. So setting linkCapacity to 1 on that listener has no effect.
> >
> > This is what I think is happening
> > 1. The Router 1 (R1) and Router 2 (R2) are started and each of them
> connect
> > to both Brokers, B1 and B2 and establish the autolinks. As soon as
> inbound
> > autoLinks (inbound to the router from the broker) are established on
> > address myQueue, the broker(s) immediately provides initial credit on
> both
> > autoLinks. This initial credit is usually 500 or 1000 depending on the
> > broker you are using.
> > 2. Now Consumer 1 (C1) connects to R1 and Consumer 2 (C2) connects to R2
> > providing one credit each respectively to each router.
> > 3. Now the Producer connects to R1 and sends one message M1 (it cannot
> send
> > more than one message because the linkCapacity is set to 1). Assuming M1
> > goes to B1 (it has an equal chance of ending up in B2 as well), B1  sees
> > that there are 2 consumers (autolinks) for that message.Say B1 sends M1
> to
> > R1, M1 will be sent to C1.
> > 4. Producer produces M2 and it goes to B2. B2 might send it to R1 or R2.
> >          4a. If B2 sends M2 to R1, the message might go over the
> > inter-router link  to R2 to C2 because there is no credit to send to C1.
> >          4b. If B2 sends M2 to R2, then C2 gets M2  because routers
> prefer
> > local consumers to remote consumers.
> >           4c. If C1 is fast enough to consume M1 and replenish the credit
> > and if B2 ends up sending M2 to R1, C1 might end up getting the second
> > message as well.
> >           4d. You need to also find out if the brokers load balances
> across
> > autolinks.
> >
> > In conclusion, you are lucky that the messages are being load balanced
> > across C1 and C2 when you changed linkCapacity to 1 but it is making it
> > more likely. This might not always happen.
> >
> > Turn on trace logging on the routers to see which route the messages are
> > taking. Try the scenario over and over again.
> >
> > To turn on trace logging add the following to the router config file -
> >
> > log {
> >     module: DEFAULT
> >     enable: trace+
> >     output: path/to/qdrouterd.log
> > }
> >
> > If you want true load balancing between the consumers, put a router R3 in
> > front of R1 and R2 and connect both consumers  to R3. The producer can
> > connect to any router.
> >
> > >
> > > You can find attached the new config files.
> > >
> > > Is this the correct way to resolve this kind of problem? Does it sound
> > > reasonable to you?
> > >
> > > Best regards,
> > > Rabih and Ali
> > >
> > > On Thu, Jul 11, 2019 at 5:50 PM Ganesh Murthy <[hidden email]>
> > wrote:
> > >
> > >>
> > >>
> > >> On Thu, Jul 11, 2019 at 10:37 AM Rabih M <[hidden email]>
> > wrote:
> > >>
> > >>> Hello,
> > >>>
> > >>> We are using Qpid dispatch router 1.7, Qpid broker 7.1.3 on redhat
> > linux
> > >>> rhel 6.4.
> > >>>
> > >>> Use case description:
> > >>> We have a cluster of 2 dispatch routers and 2 brokers. A sharded
> queue
> > >>> "myQueue" on the 2 brokers.
> > >>> Here is an illustration:
> > >>>
> > >>> [image: Diagram.jpg]
> > >>>
> > >>> The producer produces 2 messages, the messages are load balanced on
> the
> > >>> 2 brokers. Then, Consumer 1 and consumer 2 ask for a receive().
> > >>>
> > >> Does your receive() function issue one credit per call?
> > >>
> > >>>
> > >>> Our observation is that the consumer 1 consumes the first message and
> > >>> the consumer 2 is never getting the second message.
> > >>> We are aware that the first dispatch router will do a prefetch for
> the
> > 2
> > >>> messages but what is weird is that the prefetched message 2 is never
> > routed
> > >>> to the second dispatch router and consumer 2.
> > >>> I attached the dispatch routers config.
> > >>>
> > >>> This is what is going on in my view -
> > >> 1. The producer sends two messages - Message 1 goes to Broker 1 and
> > >> Message 2 goes to Broker 2.
> > >> 2. Now Consumer 1 attaches to Router 1 and calls receive() issuing one
> > >> credit. (The Consumer 2 has not yet attached to Router 2). The
> autolinks
> > >> from Broker 1 to Router 1 and Broker 2 to Router 2 each have a
> prefetch
> > of
> > >> 250 credits each. So,Message 1 and Message 2
> > >> both come down to Router 1. Message 1 is sent to Consumer 1. Message 2
> > is
> > >> waiting on the outbound queue of Consumer 1 waiting to receive a
> credit
> > >> from Consumer 1.
> > >> 3. Now Consumer 2 shows up and wants to receive Message 2 but that
> > >> message is already queued up to go to Consumer 1 who is still keeping
> > his
> > >> connection open
> > >> 4. If now Consumer 1 drops out, the Message 2 will be RELEASED back to
> > >> its broker and that message will be sent to Consumer 2.
> > >>
> > >> Have you tried attaching Consumer 1 and Consumer 2 first and then
> > >> subsequently bringing up the Producer 1 and send two messages ? In
> that
> > >> case, I think, both consumers will receive one message.
> > >>
> > >> Thanks.
> > >>
> > >>> Do you have any idea how we can make consumer 2 receive the message?
> > >>>
> > >>> Thanks for your help,
> > >>> Rabih
> > >>>
> > >>> ---------------------------------------------------------------------
> > >>> To unsubscribe, e-mail: [hidden email]
> > >>> For additional commands, e-mail: [hidden email]
> > >>
> > >>
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: [hidden email]
> > > For additional commands, e-mail: [hidden email]
> >
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Uneven distribution of messages

rabih
Hello,

Yes this is one of our production use cases, where we have low throughput but the processing of the messages can take from seconds to minutes (financial calculations). This is why it was bothering us to have an idle consumer while another is overloaded.

I understand that for the performance, we need to prefetch messages from broker to make them available in the dispatch, but what i did not understand is why inside a router we need to assign the prefetched messages to a consumer connection and not waiting until the connected consumer issues a credit, knowing that from a performance point of view, the costly action of "prefetching the messages through IO calls" was made. 
Is it because the complexity of the routing algorithm and the communications between the dispatch routers will increase?

Last point, we did the following test: 
A dispatch router have a connector with a LinkCapacity=250 connected to a broker and a listener with a LinkCapacity=1 connected to a consumer.




1- the router 1 prefetches 250 message from the broker
2- the consumer issues a credit
3- the consumer receives a message from the router but does not acknowledge
4- the consumer issues another credit
5- the consumer receives a message from the router but does not acknowledge again
Steps 4 and 5 can be repeated until the 250 msgs are all transferred to the consumer

Is this an expected behavior? consumer 1 should not acknowledge before he can receive another message knowing that the link capacity of the listener is 1?

Thanks for your explanations and help,
Best regards,
Rabih


On Wed, Jul 17, 2019 at 6:46 PM Ted Ross <[hidden email]> wrote:


On Wed, Jul 17, 2019 at 12:00 PM Rabih M <[hidden email]> wrote:
Hello,

We tested with LinkCapacity equal to 1 on the "normal" listener with debug level trace+, here are our findings:
Our Cluster:
Diagram.jpg
We are using the broker-j, the consumer are connected to the dispatch routers before we start sending.

For use case 1: 
1- the producer sends a message.
2- Consumer 1 issues one credit, receives the message without acknowledging.
3- the producer sends another message.
4- Consumer 2 in auto-ack mode issues one credit and receives the message.
5- we repeated steps 3 and 4 ten times.
6- Consumer 1 acknowledges.  
The results were correct: all the messages were correctly distributed to the idle consumer.

For use case 2:
1- the producer send 10 messages while no credits were issued yet by the consumers.
2- Consumer 1 issues one credit, receives a message without acknowledging.
3- Consumer 2 in auto-ack mode issues one credit and times out after 5 seconds if nothing is received.
4- we repeated step 3 eight times.
5- Consumer 1 acknowledges.
The results were not as expected: 4 messages were blocked in the outbound queue of the consumer 1 and consumer 2 was able to receive only 5 messages.
We analysed the traces to follow the messages. We found that 4 messages were blocked in the dispatch 1. 
Conclusion: if no consumers are issuing credits (are busy) then the incoming messages will be pre-assigned automatically by the dispatch router to the listeners (in a round robin way?).

Is it an expected behavior in the dispatch router? is it not supposed to wait for a credit to be issued before binding the message to an outbound queue?

Yes, this is the expected behavior.  The router does not propagate each individual credit from receiver to sender.  It would be impractical to do so, would not scale well, and probably still wouldn't provide the behavior you expect.  What the router does is to use delivery settlement as the way to control credit flow to producers.  If the link capacity is 250, each producer will be limited to 250 unsettled deliveries at any time.  As the deliveries are settled, more credit is issued.  This scales well in large networks, keeps a limit on the memory consumed by deliveries, and allows for high delivery rates.

Is this a real use case or are you experimenting to learn how the router works?

Under steady state flow, the messages will be delivered to the consumers in proportion to the rate at which the consumers acknowledge (settle) deliveries.  If a consumer attaches a receiving link but withholds credit, the router network will route deliveries to that consumer in anticipation of credit being issued.  It is an anti-pattern to attach a receiving link and stop issuing credit.  Credit should be used to control the rate of delivery.  If you want to stop delivery, detach the receiver.

If you really want a completely synchronous transfer across your network, you can set the link capacity on the broker connections to 1.  This will limit the number of in-flight unsettled deliveries on each incoming auto-link to 1.  It will be slow and will be prone to stalling, especially if your consumers withhold acknowledgement.

If you want to check the paths of the messages, I attached the routers logs of the use case 2.

Best regards,
Rabih

On Mon, Jul 15, 2019 at 8:14 PM Ganesh Murthy <[hidden email]> wrote:
On Mon, Jul 15, 2019 at 1:26 PM Rabih M <[hidden email]> wrote:

> Hello,
>
> We are testing with the trace+ log. We will brief you of the results when
> they are ready.
>
> Our goal is not to load balance equally the messages between the consumers
> but we would like the dispatch router to send the message to the free
> consumer.
>

What you did (setting the linkCapacity on the producer and consumer
listeners to 1) is the right thing to do if you want the router
to send the message to *any* free consumer.

If Consumer C1 is processing the first message and does not issue the next
credit until it finishes processing the first message and if the second
message arrives to Router 1 when C1 is
still processing the message, then Router 1 will definitely forward the
message to Router 2. BUT if C1 is fast enough and ends up processing the
first message and immediately issues credit, the second message if it
arrives in Router 1 will also
be sent to C1 (because the router prefers local consumers).

Remember that the key here is which Router ends up getting the message
since each broker has two autoLinks to both routers and we don't know which
autoLink the broker will choose.

But overall, with your new configuration, the router will send the message
to a consumer that is not busy no further configuration is necessary.

Specially if the consumer does a long calculation, we do not want to block
> the message in the outbound queue knowing there is another idle consumer.
>
> Is there any special configuration for this?
>
> Best regards,
> Rabih
>
>
> On Fri, Jul 12, 2019 at 7:26 PM Ganesh Murthy <[hidden email]> wrote:
>
> > On Fri, Jul 12, 2019 at 11:29 AM Rabih M <[hidden email]>
> wrote:
> >
> > > Hello,
> > >
> > > Thank you Ganesh for your answer.
> > > To answer your first question yes the consumers we have, issue one
> credit
> > > per call.
> > > For the the second question, yes we are attaching consumers before
> > sending
> > > the messages.
> > >
> > > After setting the dispatch "link capacity" on the listeners to 1, we
> > > managed to have an equal load-balancing of the messages between
> consumer
> > 1
> > > and 2.
> > > Our understanding is that the problem comes from the prefetch at the
> > level
> > > of the "normal" listener, which is preventing the "inter-router"
> listener
> > > to pass messages to the second dispatch.
> > >
> > Just want to note here that the linkCapacity on the inter-router
> listeners
> > are ignored. So setting linkCapacity to 1 on that listener has no effect.
> >
> > This is what I think is happening
> > 1. The Router 1 (R1) and Router 2 (R2) are started and each of them
> connect
> > to both Brokers, B1 and B2 and establish the autolinks. As soon as
> inbound
> > autoLinks (inbound to the router from the broker) are established on
> > address myQueue, the broker(s) immediately provides initial credit on
> both
> > autoLinks. This initial credit is usually 500 or 1000 depending on the
> > broker you are using.
> > 2. Now Consumer 1 (C1) connects to R1 and Consumer 2 (C2) connects to R2
> > providing one credit each respectively to each router.
> > 3. Now the Producer connects to R1 and sends one message M1 (it cannot
> send
> > more than one message because the linkCapacity is set to 1). Assuming M1
> > goes to B1 (it has an equal chance of ending up in B2 as well), B1  sees
> > that there are 2 consumers (autolinks) for that message.Say B1 sends M1
> to
> > R1, M1 will be sent to C1.
> > 4. Producer produces M2 and it goes to B2. B2 might send it to R1 or R2.
> >          4a. If B2 sends M2 to R1, the message might go over the
> > inter-router link  to R2 to C2 because there is no credit to send to C1.
> >          4b. If B2 sends M2 to R2, then C2 gets M2  because routers
> prefer
> > local consumers to remote consumers.
> >           4c. If C1 is fast enough to consume M1 and replenish the credit
> > and if B2 ends up sending M2 to R1, C1 might end up getting the second
> > message as well.
> >           4d. You need to also find out if the brokers load balances
> across
> > autolinks.
> >
> > In conclusion, you are lucky that the messages are being load balanced
> > across C1 and C2 when you changed linkCapacity to 1 but it is making it
> > more likely. This might not always happen.
> >
> > Turn on trace logging on the routers to see which route the messages are
> > taking. Try the scenario over and over again.
> >
> > To turn on trace logging add the following to the router config file -
> >
> > log {
> >     module: DEFAULT
> >     enable: trace+
> >     output: path/to/qdrouterd.log
> > }
> >
> > If you want true load balancing between the consumers, put a router R3 in
> > front of R1 and R2 and connect both consumers  to R3. The producer can
> > connect to any router.
> >
> > >
> > > You can find attached the new config files.
> > >
> > > Is this the correct way to resolve this kind of problem? Does it sound
> > > reasonable to you?
> > >
> > > Best regards,
> > > Rabih and Ali
> > >
> > > On Thu, Jul 11, 2019 at 5:50 PM Ganesh Murthy <[hidden email]>
> > wrote:
> > >
> > >>
> > >>
> > >> On Thu, Jul 11, 2019 at 10:37 AM Rabih M <[hidden email]>
> > wrote:
> > >>
> > >>> Hello,
> > >>>
> > >>> We are using Qpid dispatch router 1.7, Qpid broker 7.1.3 on redhat
> > linux
> > >>> rhel 6.4.
> > >>>
> > >>> Use case description:
> > >>> We have a cluster of 2 dispatch routers and 2 brokers. A sharded
> queue
> > >>> "myQueue" on the 2 brokers.
> > >>> Here is an illustration:
> > >>>
> > >>> [image: Diagram.jpg]
> > >>>
> > >>> The producer produces 2 messages, the messages are load balanced on
> the
> > >>> 2 brokers. Then, Consumer 1 and consumer 2 ask for a receive().
> > >>>
> > >> Does your receive() function issue one credit per call?
> > >>
> > >>>
> > >>> Our observation is that the consumer 1 consumes the first message and
> > >>> the consumer 2 is never getting the second message.
> > >>> We are aware that the first dispatch router will do a prefetch for
> the
> > 2
> > >>> messages but what is weird is that the prefetched message 2 is never
> > routed
> > >>> to the second dispatch router and consumer 2.
> > >>> I attached the dispatch routers config.
> > >>>
> > >>> This is what is going on in my view -
> > >> 1. The producer sends two messages - Message 1 goes to Broker 1 and
> > >> Message 2 goes to Broker 2.
> > >> 2. Now Consumer 1 attaches to Router 1 and calls receive() issuing one
> > >> credit. (The Consumer 2 has not yet attached to Router 2). The
> autolinks
> > >> from Broker 1 to Router 1 and Broker 2 to Router 2 each have a
> prefetch
> > of
> > >> 250 credits each. So,Message 1 and Message 2
> > >> both come down to Router 1. Message 1 is sent to Consumer 1. Message 2
> > is
> > >> waiting on the outbound queue of Consumer 1 waiting to receive a
> credit
> > >> from Consumer 1.
> > >> 3. Now Consumer 2 shows up and wants to receive Message 2 but that
> > >> message is already queued up to go to Consumer 1 who is still keeping
> > his
> > >> connection open
> > >> 4. If now Consumer 1 drops out, the Message 2 will be RELEASED back to
> > >> its broker and that message will be sent to Consumer 2.
> > >>
> > >> Have you tried attaching Consumer 1 and Consumer 2 first and then
> > >> subsequently bringing up the Producer 1 and send two messages ? In
> that
> > >> case, I think, both consumers will receive one message.
> > >>
> > >> Thanks.
> > >>
> > >>> Do you have any idea how we can make consumer 2 receive the message?
> > >>>
> > >>> Thanks for your help,
> > >>> Rabih
> > >>>
> > >>> ---------------------------------------------------------------------
> > >>> To unsubscribe, e-mail: [hidden email]
> > >>> For additional commands, e-mail: [hidden email]
> > >>
> > >>
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: [hidden email]
> > > For additional commands, e-mail: [hidden email]
> >
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Uneven distribution of messages

Ted Ross
On Fri, Jul 19, 2019 at 7:50 AM Rabih M <[hidden email]> wrote:

> Hello,
>
> Yes this is one of our production use cases, where we have low throughput
> but the processing of the messages can take from seconds to minutes
> (financial calculations). This is why it was bothering us to have an idle
> consumer while another is overloaded.
>

You should consider using link-routes for your consumers.  Link-routes
trade off the load balancing for tight control of credit end-to-end.  If a
consumer is link-routed to a broker, the broker won't deliver a message on
that link until the consumer issues a credit,  and then it will only
delivery as many messages as credits were issued.


>
> I understand that for the performance, we need to prefetch messages from
> broker to make them available in the dispatch, but what i did not
> understand is why inside a router we need to assign the prefetched messages
> to a consumer connection and not waiting until the connected consumer
> issues a credit, knowing that from a performance point of view, the costly
> action of "prefetching the messages through IO calls" was made.
> Is it because the complexity of the routing algorithm and the
> communications between the dispatch routers will increase?
>
>
When messages are sent into the router network, they are immediately routed
to a destination.  The routers don't hold messages for later routing.
Also, the synchronization of real-time credit state across a network for
all addresses is not practical or scalable.


> Last point, we did the following test:
> A dispatch router have a connector with a LinkCapacity=250 connected to a
> broker and a listener with a LinkCapacity=1 connected to a consumer.
>

The important link capacity in this scenario is the 250 as it controls the
router pre-fetch.  The consumer's link capacity of 1 is not relevant to
this case.


>
> [image: Untitled Diagram.jpg]
>

I can't see the diagram, but I think I get the idea.


>
>
> 1- the router 1 prefetches 250 message from the broker
> 2- the consumer issues a credit
> 3- the consumer receives a message from the router but does not acknowledge
> 4- the consumer issues another credit
> 5- the consumer receives a message from the router but does not
> acknowledge again
> Steps 4 and 5 can be repeated until the 250 msgs are all transferred to
> the consumer
>

This is consistent with there being only one consumer for the address on
the network at the time the broker sent the 250 messages.


>
> Is this an expected behavior? consumer 1 should not acknowledge before he
> can receive another message knowing that the link capacity of the listener
> is 1?
>

Best practice for acknowledgement is for the consumer to acknowledge
(settle) immediately after finishing the processing of the message (i.e.
once that message is no longer consuming memory or compute resources on the
host).  This causes the settlement state of deliveries to be directly
related to consumer resources.  Again, the link capacity of 1 is not having
any effect on the behavior of this scenario.


>
> Thanks for your explanations and help,
>

Am I to understand that your case is this?  You have a distributed work
queue in which the time-to-process is highly variable.  Some messages are
processed quickly and others take much longer.  You don't want to incur the
longer latency on messages that can be handled quickly if there are many
more fast messages than slow messages.

Is it possible to know beforehand which messages are going to take long?
Could you put these on a different queue with a different address?


> Best regards,
> Rabih
>
>
> On Wed, Jul 17, 2019 at 6:46 PM Ted Ross <[hidden email]> wrote:
>
>>
>>
>> On Wed, Jul 17, 2019 at 12:00 PM Rabih M <[hidden email]> wrote:
>>
>>> Hello,
>>>
>>> We tested with LinkCapacity equal to 1 on the "normal" listener with
>>> debug level trace+, here are our findings:
>>> Our Cluster:
>>> [image: Diagram.jpg]
>>> We are using the broker-j, the consumer are connected to the dispatch
>>> routers before we start sending.
>>>
>>> For use case 1:
>>> 1- the producer sends a message.
>>> 2- Consumer 1 issues one credit, receives the message without
>>> acknowledging.
>>> 3- the producer sends another message.
>>> 4- Consumer 2 in auto-ack mode issues one credit and receives the
>>> message.
>>> 5- we repeated steps 3 and 4 ten times.
>>> 6- Consumer 1 acknowledges.
>>> The results were correct: all the messages were correctly distributed to
>>> the idle consumer.
>>>
>>> For use case 2:
>>> 1- the producer send 10 messages while no credits were issued yet by the
>>> consumers.
>>> 2- Consumer 1 issues one credit, receives a message without
>>> acknowledging.
>>> 3- Consumer 2 in auto-ack mode issues one credit and times out after 5
>>> seconds if nothing is received.
>>> 4- we repeated step 3 eight times.
>>> 5- Consumer 1 acknowledges.
>>> The results were not as expected: 4 messages were blocked in the
>>> outbound queue of the consumer 1 and consumer 2 was able to receive only 5
>>> messages.
>>> We analysed the traces to follow the messages. We found that 4 messages
>>> were blocked in the dispatch 1.
>>> Conclusion: if no consumers are issuing credits (are busy) then the
>>> incoming messages will be pre-assigned automatically by the dispatch router
>>> to the listeners (in a round robin way?).
>>>
>>> Is it an expected behavior in the dispatch router? is it not supposed to
>>> wait for a credit to be issued before binding the message to an outbound
>>> queue?
>>>
>>
>> Yes, this is the expected behavior.  The router does not propagate each
>> individual credit from receiver to sender.  It would be impractical to do
>> so, would not scale well, and probably still wouldn't provide the behavior
>> you expect.  What the router does is to use delivery settlement as the way
>> to control credit flow to producers.  If the link capacity is 250, each
>> producer will be limited to 250 unsettled deliveries at any time.  As the
>> deliveries are settled, more credit is issued.  This scales well in large
>> networks, keeps a limit on the memory consumed by deliveries, and allows
>> for high delivery rates.
>>
>> Is this a real use case or are you experimenting to learn how the router
>> works?
>>
>> Under steady state flow, the messages will be delivered to the consumers
>> in proportion to the rate at which the consumers acknowledge (settle)
>> deliveries.  If a consumer attaches a receiving link but withholds credit,
>> the router network will route deliveries to that consumer in anticipation
>> of credit being issued.  It is an anti-pattern to attach a receiving link
>> and stop issuing credit.  Credit should be used to control the rate of
>> delivery.  If you want to stop delivery, detach the receiver.
>>
>> If you really want a completely synchronous transfer across your network,
>> you can set the link capacity on the broker connections to 1.  This will
>> limit the number of in-flight unsettled deliveries on each incoming
>> auto-link to 1.  It will be slow and will be prone to stalling, especially
>> if your consumers withhold acknowledgement.
>>
>> If you want to check the paths of the messages, I attached the routers
>>> logs of the use case 2.
>>>
>>> Best regards,
>>> Rabih
>>>
>>> On Mon, Jul 15, 2019 at 8:14 PM Ganesh Murthy <[hidden email]>
>>> wrote:
>>>
>>>> On Mon, Jul 15, 2019 at 1:26 PM Rabih M <[hidden email]>
>>>> wrote:
>>>>
>>>> > Hello,
>>>> >
>>>> > We are testing with the trace+ log. We will brief you of the results
>>>> when
>>>> > they are ready.
>>>> >
>>>> > Our goal is not to load balance equally the messages between the
>>>> consumers
>>>> > but we would like the dispatch router to send the message to the free
>>>> > consumer.
>>>> >
>>>>
>>>> What you did (setting the linkCapacity on the producer and consumer
>>>> listeners to 1) is the right thing to do if you want the router
>>>> to send the message to *any* free consumer.
>>>>
>>>> If Consumer C1 is processing the first message and does not issue the
>>>> next
>>>> credit until it finishes processing the first message and if the second
>>>> message arrives to Router 1 when C1 is
>>>> still processing the message, then Router 1 will definitely forward the
>>>> message to Router 2. BUT if C1 is fast enough and ends up processing the
>>>> first message and immediately issues credit, the second message if it
>>>> arrives in Router 1 will also
>>>> be sent to C1 (because the router prefers local consumers).
>>>>
>>>> Remember that the key here is which Router ends up getting the message
>>>> since each broker has two autoLinks to both routers and we don't know
>>>> which
>>>> autoLink the broker will choose.
>>>>
>>>> But overall, with your new configuration, the router will send the
>>>> message
>>>> to a consumer that is not busy no further configuration is necessary.
>>>>
>>>> Specially if the consumer does a long calculation, we do not want to
>>>> block
>>>> > the message in the outbound queue knowing there is another idle
>>>> consumer.
>>>> >
>>>> > Is there any special configuration for this?
>>>> >
>>>> > Best regards,
>>>> > Rabih
>>>> >
>>>> >
>>>> > On Fri, Jul 12, 2019 at 7:26 PM Ganesh Murthy <[hidden email]>
>>>> wrote:
>>>> >
>>>> > > On Fri, Jul 12, 2019 at 11:29 AM Rabih M <[hidden email]>
>>>> > wrote:
>>>> > >
>>>> > > > Hello,
>>>> > > >
>>>> > > > Thank you Ganesh for your answer.
>>>> > > > To answer your first question yes the consumers we have, issue one
>>>> > credit
>>>> > > > per call.
>>>> > > > For the the second question, yes we are attaching consumers before
>>>> > > sending
>>>> > > > the messages.
>>>> > > >
>>>> > > > After setting the dispatch "link capacity" on the listeners to 1,
>>>> we
>>>> > > > managed to have an equal load-balancing of the messages between
>>>> > consumer
>>>> > > 1
>>>> > > > and 2.
>>>> > > > Our understanding is that the problem comes from the prefetch at
>>>> the
>>>> > > level
>>>> > > > of the "normal" listener, which is preventing the "inter-router"
>>>> > listener
>>>> > > > to pass messages to the second dispatch.
>>>> > > >
>>>> > > Just want to note here that the linkCapacity on the inter-router
>>>> > listeners
>>>> > > are ignored. So setting linkCapacity to 1 on that listener has no
>>>> effect.
>>>> > >
>>>> > > This is what I think is happening
>>>> > > 1. The Router 1 (R1) and Router 2 (R2) are started and each of them
>>>> > connect
>>>> > > to both Brokers, B1 and B2 and establish the autolinks. As soon as
>>>> > inbound
>>>> > > autoLinks (inbound to the router from the broker) are established on
>>>> > > address myQueue, the broker(s) immediately provides initial credit
>>>> on
>>>> > both
>>>> > > autoLinks. This initial credit is usually 500 or 1000 depending on
>>>> the
>>>> > > broker you are using.
>>>> > > 2. Now Consumer 1 (C1) connects to R1 and Consumer 2 (C2) connects
>>>> to R2
>>>> > > providing one credit each respectively to each router.
>>>> > > 3. Now the Producer connects to R1 and sends one message M1 (it
>>>> cannot
>>>> > send
>>>> > > more than one message because the linkCapacity is set to 1).
>>>> Assuming M1
>>>> > > goes to B1 (it has an equal chance of ending up in B2 as well), B1
>>>> sees
>>>> > > that there are 2 consumers (autolinks) for that message.Say B1
>>>> sends M1
>>>> > to
>>>> > > R1, M1 will be sent to C1.
>>>> > > 4. Producer produces M2 and it goes to B2. B2 might send it to R1
>>>> or R2.
>>>> > >          4a. If B2 sends M2 to R1, the message might go over the
>>>> > > inter-router link  to R2 to C2 because there is no credit to send
>>>> to C1.
>>>> > >          4b. If B2 sends M2 to R2, then C2 gets M2  because routers
>>>> > prefer
>>>> > > local consumers to remote consumers.
>>>> > >           4c. If C1 is fast enough to consume M1 and replenish the
>>>> credit
>>>> > > and if B2 ends up sending M2 to R1, C1 might end up getting the
>>>> second
>>>> > > message as well.
>>>> > >           4d. You need to also find out if the brokers load balances
>>>> > across
>>>> > > autolinks.
>>>> > >
>>>> > > In conclusion, you are lucky that the messages are being load
>>>> balanced
>>>> > > across C1 and C2 when you changed linkCapacity to 1 but it is
>>>> making it
>>>> > > more likely. This might not always happen.
>>>> > >
>>>> > > Turn on trace logging on the routers to see which route the
>>>> messages are
>>>> > > taking. Try the scenario over and over again.
>>>> > >
>>>> > > To turn on trace logging add the following to the router config
>>>> file -
>>>> > >
>>>> > > log {
>>>> > >     module: DEFAULT
>>>> > >     enable: trace+
>>>> > >     output: path/to/qdrouterd.log
>>>> > > }
>>>> > >
>>>> > > If you want true load balancing between the consumers, put a router
>>>> R3 in
>>>> > > front of R1 and R2 and connect both consumers  to R3. The producer
>>>> can
>>>> > > connect to any router.
>>>> > >
>>>> > > >
>>>> > > > You can find attached the new config files.
>>>> > > >
>>>> > > > Is this the correct way to resolve this kind of problem? Does it
>>>> sound
>>>> > > > reasonable to you?
>>>> > > >
>>>> > > > Best regards,
>>>> > > > Rabih and Ali
>>>> > > >
>>>> > > > On Thu, Jul 11, 2019 at 5:50 PM Ganesh Murthy <[hidden email]
>>>> >
>>>> > > wrote:
>>>> > > >
>>>> > > >>
>>>> > > >>
>>>> > > >> On Thu, Jul 11, 2019 at 10:37 AM Rabih M <
>>>> [hidden email]>
>>>> > > wrote:
>>>> > > >>
>>>> > > >>> Hello,
>>>> > > >>>
>>>> > > >>> We are using Qpid dispatch router 1.7, Qpid broker 7.1.3 on
>>>> redhat
>>>> > > linux
>>>> > > >>> rhel 6.4.
>>>> > > >>>
>>>> > > >>> Use case description:
>>>> > > >>> We have a cluster of 2 dispatch routers and 2 brokers. A sharded
>>>> > queue
>>>> > > >>> "myQueue" on the 2 brokers.
>>>> > > >>> Here is an illustration:
>>>> > > >>>
>>>> > > >>> [image: Diagram.jpg]
>>>> > > >>>
>>>> > > >>> The producer produces 2 messages, the messages are load
>>>> balanced on
>>>> > the
>>>> > > >>> 2 brokers. Then, Consumer 1 and consumer 2 ask for a receive().
>>>> > > >>>
>>>> > > >> Does your receive() function issue one credit per call?
>>>> > > >>
>>>> > > >>>
>>>> > > >>> Our observation is that the consumer 1 consumes the first
>>>> message and
>>>> > > >>> the consumer 2 is never getting the second message.
>>>> > > >>> We are aware that the first dispatch router will do a prefetch
>>>> for
>>>> > the
>>>> > > 2
>>>> > > >>> messages but what is weird is that the prefetched message 2 is
>>>> never
>>>> > > routed
>>>> > > >>> to the second dispatch router and consumer 2.
>>>> > > >>> I attached the dispatch routers config.
>>>> > > >>>
>>>> > > >>> This is what is going on in my view -
>>>> > > >> 1. The producer sends two messages - Message 1 goes to Broker 1
>>>> and
>>>> > > >> Message 2 goes to Broker 2.
>>>> > > >> 2. Now Consumer 1 attaches to Router 1 and calls receive()
>>>> issuing one
>>>> > > >> credit. (The Consumer 2 has not yet attached to Router 2). The
>>>> > autolinks
>>>> > > >> from Broker 1 to Router 1 and Broker 2 to Router 2 each have a
>>>> > prefetch
>>>> > > of
>>>> > > >> 250 credits each. So,Message 1 and Message 2
>>>> > > >> both come down to Router 1. Message 1 is sent to Consumer 1.
>>>> Message 2
>>>> > > is
>>>> > > >> waiting on the outbound queue of Consumer 1 waiting to receive a
>>>> > credit
>>>> > > >> from Consumer 1.
>>>> > > >> 3. Now Consumer 2 shows up and wants to receive Message 2 but
>>>> that
>>>> > > >> message is already queued up to go to Consumer 1 who is still
>>>> keeping
>>>> > > his
>>>> > > >> connection open
>>>> > > >> 4. If now Consumer 1 drops out, the Message 2 will be RELEASED
>>>> back to
>>>> > > >> its broker and that message will be sent to Consumer 2.
>>>> > > >>
>>>> > > >> Have you tried attaching Consumer 1 and Consumer 2 first and then
>>>> > > >> subsequently bringing up the Producer 1 and send two messages ?
>>>> In
>>>> > that
>>>> > > >> case, I think, both consumers will receive one message.
>>>> > > >>
>>>> > > >> Thanks.
>>>> > > >>
>>>> > > >>> Do you have any idea how we can make consumer 2 receive the
>>>> message?
>>>> > > >>>
>>>> > > >>> Thanks for your help,
>>>> > > >>> Rabih
>>>> > > >>>
>>>> > > >>>
>>>> ---------------------------------------------------------------------
>>>> > > >>> To unsubscribe, e-mail: [hidden email]
>>>> > > >>> For additional commands, e-mail: [hidden email]
>>>> > > >>
>>>> > > >>
>>>> > > >
>>>> ---------------------------------------------------------------------
>>>> > > > To unsubscribe, e-mail: [hidden email]
>>>> > > > For additional commands, e-mail: [hidden email]
>>>> > >
>>>> >
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>
>>
Reply | Threaded
Open this post in threaded view
|

Re: Uneven distribution of messages

rabih
Hello Ted,

We cannot use the link-routes because we need to have multiple consumers
connected to the same queue.

Having multiple queues depending on the task execution time can be a good
solution.
We are thinking about another solution too:
To have consumer that receives all the messages and dispatches them to the
free workers and at the same time it can monitor the load to upscale and
downscale the workers depending on the work received...

Thanks for your help,
Best regards,
Rabih

On Fri, Jul 19, 2019 at 6:02 PM Ted Ross <[hidden email]> wrote:

> On Fri, Jul 19, 2019 at 7:50 AM Rabih M <[hidden email]> wrote:
>
> > Hello,
> >
> > Yes this is one of our production use cases, where we have low throughput
> > but the processing of the messages can take from seconds to minutes
> > (financial calculations). This is why it was bothering us to have an idle
> > consumer while another is overloaded.
> >
>
> You should consider using link-routes for your consumers.  Link-routes
> trade off the load balancing for tight control of credit end-to-end.  If a
> consumer is link-routed to a broker, the broker won't deliver a message on
> that link until the consumer issues a credit,  and then it will only
> delivery as many messages as credits were issued.
>
>
> >
> > I understand that for the performance, we need to prefetch messages from
> > broker to make them available in the dispatch, but what i did not
> > understand is why inside a router we need to assign the prefetched
> messages
> > to a consumer connection and not waiting until the connected consumer
> > issues a credit, knowing that from a performance point of view, the
> costly
> > action of "prefetching the messages through IO calls" was made.
> > Is it because the complexity of the routing algorithm and the
> > communications between the dispatch routers will increase?
> >
> >
> When messages are sent into the router network, they are immediately routed
> to a destination.  The routers don't hold messages for later routing.
> Also, the synchronization of real-time credit state across a network for
> all addresses is not practical or scalable.
>
>
> > Last point, we did the following test:
> > A dispatch router have a connector with a LinkCapacity=250 connected to a
> > broker and a listener with a LinkCapacity=1 connected to a consumer.
> >
>
> The important link capacity in this scenario is the 250 as it controls the
> router pre-fetch.  The consumer's link capacity of 1 is not relevant to
> this case.
>
>
> >
> > [image: Untitled Diagram.jpg]
> >
>
> I can't see the diagram, but I think I get the idea.
>
>
> >
> >
> > 1- the router 1 prefetches 250 message from the broker
> > 2- the consumer issues a credit
> > 3- the consumer receives a message from the router but does not
> acknowledge
> > 4- the consumer issues another credit
> > 5- the consumer receives a message from the router but does not
> > acknowledge again
> > Steps 4 and 5 can be repeated until the 250 msgs are all transferred to
> > the consumer
> >
>
> This is consistent with there being only one consumer for the address on
> the network at the time the broker sent the 250 messages.
>
>
> >
> > Is this an expected behavior? consumer 1 should not acknowledge before he
> > can receive another message knowing that the link capacity of the
> listener
> > is 1?
> >
>
> Best practice for acknowledgement is for the consumer to acknowledge
> (settle) immediately after finishing the processing of the message (i.e.
> once that message is no longer consuming memory or compute resources on the
> host).  This causes the settlement state of deliveries to be directly
> related to consumer resources.  Again, the link capacity of 1 is not having
> any effect on the behavior of this scenario.
>
>
> >
> > Thanks for your explanations and help,
> >
>
> Am I to understand that your case is this?  You have a distributed work
> queue in which the time-to-process is highly variable.  Some messages are
> processed quickly and others take much longer.  You don't want to incur the
> longer latency on messages that can be handled quickly if there are many
> more fast messages than slow messages.
>
> Is it possible to know beforehand which messages are going to take long?
> Could you put these on a different queue with a different address?
>
>
> > Best regards,
> > Rabih
> >
> >
> > On Wed, Jul 17, 2019 at 6:46 PM Ted Ross <[hidden email]> wrote:
> >
> >>
> >>
> >> On Wed, Jul 17, 2019 at 12:00 PM Rabih M <[hidden email]>
> wrote:
> >>
> >>> Hello,
> >>>
> >>> We tested with LinkCapacity equal to 1 on the "normal" listener with
> >>> debug level trace+, here are our findings:
> >>> Our Cluster:
> >>> [image: Diagram.jpg]
> >>> We are using the broker-j, the consumer are connected to the dispatch
> >>> routers before we start sending.
> >>>
> >>> For use case 1:
> >>> 1- the producer sends a message.
> >>> 2- Consumer 1 issues one credit, receives the message without
> >>> acknowledging.
> >>> 3- the producer sends another message.
> >>> 4- Consumer 2 in auto-ack mode issues one credit and receives the
> >>> message.
> >>> 5- we repeated steps 3 and 4 ten times.
> >>> 6- Consumer 1 acknowledges.
> >>> The results were correct: all the messages were correctly distributed
> to
> >>> the idle consumer.
> >>>
> >>> For use case 2:
> >>> 1- the producer send 10 messages while no credits were issued yet by
> the
> >>> consumers.
> >>> 2- Consumer 1 issues one credit, receives a message without
> >>> acknowledging.
> >>> 3- Consumer 2 in auto-ack mode issues one credit and times out after 5
> >>> seconds if nothing is received.
> >>> 4- we repeated step 3 eight times.
> >>> 5- Consumer 1 acknowledges.
> >>> The results were not as expected: 4 messages were blocked in the
> >>> outbound queue of the consumer 1 and consumer 2 was able to receive
> only 5
> >>> messages.
> >>> We analysed the traces to follow the messages. We found that 4 messages
> >>> were blocked in the dispatch 1.
> >>> Conclusion: if no consumers are issuing credits (are busy) then the
> >>> incoming messages will be pre-assigned automatically by the dispatch
> router
> >>> to the listeners (in a round robin way?).
> >>>
> >>> Is it an expected behavior in the dispatch router? is it not supposed
> to
> >>> wait for a credit to be issued before binding the message to an
> outbound
> >>> queue?
> >>>
> >>
> >> Yes, this is the expected behavior.  The router does not propagate each
> >> individual credit from receiver to sender.  It would be impractical to
> do
> >> so, would not scale well, and probably still wouldn't provide the
> behavior
> >> you expect.  What the router does is to use delivery settlement as the
> way
> >> to control credit flow to producers.  If the link capacity is 250, each
> >> producer will be limited to 250 unsettled deliveries at any time.  As
> the
> >> deliveries are settled, more credit is issued.  This scales well in
> large
> >> networks, keeps a limit on the memory consumed by deliveries, and allows
> >> for high delivery rates.
> >>
> >> Is this a real use case or are you experimenting to learn how the router
> >> works?
> >>
> >> Under steady state flow, the messages will be delivered to the consumers
> >> in proportion to the rate at which the consumers acknowledge (settle)
> >> deliveries.  If a consumer attaches a receiving link but withholds
> credit,
> >> the router network will route deliveries to that consumer in
> anticipation
> >> of credit being issued.  It is an anti-pattern to attach a receiving
> link
> >> and stop issuing credit.  Credit should be used to control the rate of
> >> delivery.  If you want to stop delivery, detach the receiver.
> >>
> >> If you really want a completely synchronous transfer across your
> network,
> >> you can set the link capacity on the broker connections to 1.  This will
> >> limit the number of in-flight unsettled deliveries on each incoming
> >> auto-link to 1.  It will be slow and will be prone to stalling,
> especially
> >> if your consumers withhold acknowledgement.
> >>
> >> If you want to check the paths of the messages, I attached the routers
> >>> logs of the use case 2.
> >>>
> >>> Best regards,
> >>> Rabih
> >>>
> >>> On Mon, Jul 15, 2019 at 8:14 PM Ganesh Murthy <[hidden email]>
> >>> wrote:
> >>>
> >>>> On Mon, Jul 15, 2019 at 1:26 PM Rabih M <[hidden email]>
> >>>> wrote:
> >>>>
> >>>> > Hello,
> >>>> >
> >>>> > We are testing with the trace+ log. We will brief you of the results
> >>>> when
> >>>> > they are ready.
> >>>> >
> >>>> > Our goal is not to load balance equally the messages between the
> >>>> consumers
> >>>> > but we would like the dispatch router to send the message to the
> free
> >>>> > consumer.
> >>>> >
> >>>>
> >>>> What you did (setting the linkCapacity on the producer and consumer
> >>>> listeners to 1) is the right thing to do if you want the router
> >>>> to send the message to *any* free consumer.
> >>>>
> >>>> If Consumer C1 is processing the first message and does not issue the
> >>>> next
> >>>> credit until it finishes processing the first message and if the
> second
> >>>> message arrives to Router 1 when C1 is
> >>>> still processing the message, then Router 1 will definitely forward
> the
> >>>> message to Router 2. BUT if C1 is fast enough and ends up processing
> the
> >>>> first message and immediately issues credit, the second message if it
> >>>> arrives in Router 1 will also
> >>>> be sent to C1 (because the router prefers local consumers).
> >>>>
> >>>> Remember that the key here is which Router ends up getting the message
> >>>> since each broker has two autoLinks to both routers and we don't know
> >>>> which
> >>>> autoLink the broker will choose.
> >>>>
> >>>> But overall, with your new configuration, the router will send the
> >>>> message
> >>>> to a consumer that is not busy no further configuration is necessary.
> >>>>
> >>>> Specially if the consumer does a long calculation, we do not want to
> >>>> block
> >>>> > the message in the outbound queue knowing there is another idle
> >>>> consumer.
> >>>> >
> >>>> > Is there any special configuration for this?
> >>>> >
> >>>> > Best regards,
> >>>> > Rabih
> >>>> >
> >>>> >
> >>>> > On Fri, Jul 12, 2019 at 7:26 PM Ganesh Murthy <[hidden email]>
> >>>> wrote:
> >>>> >
> >>>> > > On Fri, Jul 12, 2019 at 11:29 AM Rabih M <[hidden email]
> >
> >>>> > wrote:
> >>>> > >
> >>>> > > > Hello,
> >>>> > > >
> >>>> > > > Thank you Ganesh for your answer.
> >>>> > > > To answer your first question yes the consumers we have, issue
> one
> >>>> > credit
> >>>> > > > per call.
> >>>> > > > For the the second question, yes we are attaching consumers
> before
> >>>> > > sending
> >>>> > > > the messages.
> >>>> > > >
> >>>> > > > After setting the dispatch "link capacity" on the listeners to
> 1,
> >>>> we
> >>>> > > > managed to have an equal load-balancing of the messages between
> >>>> > consumer
> >>>> > > 1
> >>>> > > > and 2.
> >>>> > > > Our understanding is that the problem comes from the prefetch at
> >>>> the
> >>>> > > level
> >>>> > > > of the "normal" listener, which is preventing the "inter-router"
> >>>> > listener
> >>>> > > > to pass messages to the second dispatch.
> >>>> > > >
> >>>> > > Just want to note here that the linkCapacity on the inter-router
> >>>> > listeners
> >>>> > > are ignored. So setting linkCapacity to 1 on that listener has no
> >>>> effect.
> >>>> > >
> >>>> > > This is what I think is happening
> >>>> > > 1. The Router 1 (R1) and Router 2 (R2) are started and each of
> them
> >>>> > connect
> >>>> > > to both Brokers, B1 and B2 and establish the autolinks. As soon as
> >>>> > inbound
> >>>> > > autoLinks (inbound to the router from the broker) are established
> on
> >>>> > > address myQueue, the broker(s) immediately provides initial credit
> >>>> on
> >>>> > both
> >>>> > > autoLinks. This initial credit is usually 500 or 1000 depending on
> >>>> the
> >>>> > > broker you are using.
> >>>> > > 2. Now Consumer 1 (C1) connects to R1 and Consumer 2 (C2) connects
> >>>> to R2
> >>>> > > providing one credit each respectively to each router.
> >>>> > > 3. Now the Producer connects to R1 and sends one message M1 (it
> >>>> cannot
> >>>> > send
> >>>> > > more than one message because the linkCapacity is set to 1).
> >>>> Assuming M1
> >>>> > > goes to B1 (it has an equal chance of ending up in B2 as well), B1
> >>>> sees
> >>>> > > that there are 2 consumers (autolinks) for that message.Say B1
> >>>> sends M1
> >>>> > to
> >>>> > > R1, M1 will be sent to C1.
> >>>> > > 4. Producer produces M2 and it goes to B2. B2 might send it to R1
> >>>> or R2.
> >>>> > >          4a. If B2 sends M2 to R1, the message might go over the
> >>>> > > inter-router link  to R2 to C2 because there is no credit to send
> >>>> to C1.
> >>>> > >          4b. If B2 sends M2 to R2, then C2 gets M2  because
> routers
> >>>> > prefer
> >>>> > > local consumers to remote consumers.
> >>>> > >           4c. If C1 is fast enough to consume M1 and replenish the
> >>>> credit
> >>>> > > and if B2 ends up sending M2 to R1, C1 might end up getting the
> >>>> second
> >>>> > > message as well.
> >>>> > >           4d. You need to also find out if the brokers load
> balances
> >>>> > across
> >>>> > > autolinks.
> >>>> > >
> >>>> > > In conclusion, you are lucky that the messages are being load
> >>>> balanced
> >>>> > > across C1 and C2 when you changed linkCapacity to 1 but it is
> >>>> making it
> >>>> > > more likely. This might not always happen.
> >>>> > >
> >>>> > > Turn on trace logging on the routers to see which route the
> >>>> messages are
> >>>> > > taking. Try the scenario over and over again.
> >>>> > >
> >>>> > > To turn on trace logging add the following to the router config
> >>>> file -
> >>>> > >
> >>>> > > log {
> >>>> > >     module: DEFAULT
> >>>> > >     enable: trace+
> >>>> > >     output: path/to/qdrouterd.log
> >>>> > > }
> >>>> > >
> >>>> > > If you want true load balancing between the consumers, put a
> router
> >>>> R3 in
> >>>> > > front of R1 and R2 and connect both consumers  to R3. The producer
> >>>> can
> >>>> > > connect to any router.
> >>>> > >
> >>>> > > >
> >>>> > > > You can find attached the new config files.
> >>>> > > >
> >>>> > > > Is this the correct way to resolve this kind of problem? Does it
> >>>> sound
> >>>> > > > reasonable to you?
> >>>> > > >
> >>>> > > > Best regards,
> >>>> > > > Rabih and Ali
> >>>> > > >
> >>>> > > > On Thu, Jul 11, 2019 at 5:50 PM Ganesh Murthy <
> [hidden email]
> >>>> >
> >>>> > > wrote:
> >>>> > > >
> >>>> > > >>
> >>>> > > >>
> >>>> > > >> On Thu, Jul 11, 2019 at 10:37 AM Rabih M <
> >>>> [hidden email]>
> >>>> > > wrote:
> >>>> > > >>
> >>>> > > >>> Hello,
> >>>> > > >>>
> >>>> > > >>> We are using Qpid dispatch router 1.7, Qpid broker 7.1.3 on
> >>>> redhat
> >>>> > > linux
> >>>> > > >>> rhel 6.4.
> >>>> > > >>>
> >>>> > > >>> Use case description:
> >>>> > > >>> We have a cluster of 2 dispatch routers and 2 brokers. A
> sharded
> >>>> > queue
> >>>> > > >>> "myQueue" on the 2 brokers.
> >>>> > > >>> Here is an illustration:
> >>>> > > >>>
> >>>> > > >>> [image: Diagram.jpg]
> >>>> > > >>>
> >>>> > > >>> The producer produces 2 messages, the messages are load
> >>>> balanced on
> >>>> > the
> >>>> > > >>> 2 brokers. Then, Consumer 1 and consumer 2 ask for a
> receive().
> >>>> > > >>>
> >>>> > > >> Does your receive() function issue one credit per call?
> >>>> > > >>
> >>>> > > >>>
> >>>> > > >>> Our observation is that the consumer 1 consumes the first
> >>>> message and
> >>>> > > >>> the consumer 2 is never getting the second message.
> >>>> > > >>> We are aware that the first dispatch router will do a prefetch
> >>>> for
> >>>> > the
> >>>> > > 2
> >>>> > > >>> messages but what is weird is that the prefetched message 2 is
> >>>> never
> >>>> > > routed
> >>>> > > >>> to the second dispatch router and consumer 2.
> >>>> > > >>> I attached the dispatch routers config.
> >>>> > > >>>
> >>>> > > >>> This is what is going on in my view -
> >>>> > > >> 1. The producer sends two messages - Message 1 goes to Broker 1
> >>>> and
> >>>> > > >> Message 2 goes to Broker 2.
> >>>> > > >> 2. Now Consumer 1 attaches to Router 1 and calls receive()
> >>>> issuing one
> >>>> > > >> credit. (The Consumer 2 has not yet attached to Router 2). The
> >>>> > autolinks
> >>>> > > >> from Broker 1 to Router 1 and Broker 2 to Router 2 each have a
> >>>> > prefetch
> >>>> > > of
> >>>> > > >> 250 credits each. So,Message 1 and Message 2
> >>>> > > >> both come down to Router 1. Message 1 is sent to Consumer 1.
> >>>> Message 2
> >>>> > > is
> >>>> > > >> waiting on the outbound queue of Consumer 1 waiting to receive
> a
> >>>> > credit
> >>>> > > >> from Consumer 1.
> >>>> > > >> 3. Now Consumer 2 shows up and wants to receive Message 2 but
> >>>> that
> >>>> > > >> message is already queued up to go to Consumer 1 who is still
> >>>> keeping
> >>>> > > his
> >>>> > > >> connection open
> >>>> > > >> 4. If now Consumer 1 drops out, the Message 2 will be RELEASED
> >>>> back to
> >>>> > > >> its broker and that message will be sent to Consumer 2.
> >>>> > > >>
> >>>> > > >> Have you tried attaching Consumer 1 and Consumer 2 first and
> then
> >>>> > > >> subsequently bringing up the Producer 1 and send two messages ?
> >>>> In
> >>>> > that
> >>>> > > >> case, I think, both consumers will receive one message.
> >>>> > > >>
> >>>> > > >> Thanks.
> >>>> > > >>
> >>>> > > >>> Do you have any idea how we can make consumer 2 receive the
> >>>> message?
> >>>> > > >>>
> >>>> > > >>> Thanks for your help,
> >>>> > > >>> Rabih
> >>>> > > >>>
> >>>> > > >>>
> >>>> ---------------------------------------------------------------------
> >>>> > > >>> To unsubscribe, e-mail: [hidden email]
> >>>> > > >>> For additional commands, e-mail: [hidden email]
> >>>> > > >>
> >>>> > > >>
> >>>> > > >
> >>>> ---------------------------------------------------------------------
> >>>> > > > To unsubscribe, e-mail: [hidden email]
> >>>> > > > For additional commands, e-mail: [hidden email]
> >>>> > >
> >>>> >
> >>>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: [hidden email]
> >>> For additional commands, e-mail: [hidden email]
> >>
> >>
>
Reply | Threaded
Open this post in threaded view
|

RE: Uneven distribution of messages

HADI Ali
Hello Ted,

Like already discussed, with linkCapacity = 1 in the normal listener, messages are routed to idle consumers if any, or equally load balanced between busy consumers. This is already solving a big part of our problem.

However with this configuration, we are limiting all our cluster and in particular producer features. We might need to use asynchronous send in other use cases which will not be possible with linkCapacity equals to one.

Having different linkCapacity for incoming and outgoing links might solve the problem. Do you think it's a good practice to do that ? And do you think it is feasible to have that without having different entrypoints (listener) for each dispatch router?

Best Regards,
Ali

-----Original Message-----
From: Rabih M <[hidden email]>
Sent: lundi 22 juillet 2019 17:15
To: [hidden email]
Subject: Re: Uneven distribution of messages

Hello Ted,

We cannot use the link-routes because we need to have multiple consumers connected to the same queue.

Having multiple queues depending on the task execution time can be a good solution.
We are thinking about another solution too:
To have consumer that receives all the messages and dispatches them to the free workers and at the same time it can monitor the load to upscale and downscale the workers depending on the work received...

Thanks for your help,
Best regards,
Rabih

On Fri, Jul 19, 2019 at 6:02 PM Ted Ross <[hidden email]> wrote:

> On Fri, Jul 19, 2019 at 7:50 AM Rabih M <[hidden email]> wrote:
>
> > Hello,
> >
> > Yes this is one of our production use cases, where we have low
> > throughput but the processing of the messages can take from seconds
> > to minutes (financial calculations). This is why it was bothering us
> > to have an idle consumer while another is overloaded.
> >
>
> You should consider using link-routes for your consumers.  Link-routes
> trade off the load balancing for tight control of credit end-to-end.
> If a consumer is link-routed to a broker, the broker won't deliver a
> message on that link until the consumer issues a credit,  and then it
> will only delivery as many messages as credits were issued.
>
>
> >
> > I understand that for the performance, we need to prefetch messages
> > from broker to make them available in the dispatch, but what i did
> > not understand is why inside a router we need to assign the
> > prefetched
> messages
> > to a consumer connection and not waiting until the connected
> > consumer issues a credit, knowing that from a performance point of
> > view, the
> costly
> > action of "prefetching the messages through IO calls" was made.
> > Is it because the complexity of the routing algorithm and the
> > communications between the dispatch routers will increase?
> >
> >
> When messages are sent into the router network, they are immediately
> routed to a destination.  The routers don't hold messages for later routing.
> Also, the synchronization of real-time credit state across a network
> for all addresses is not practical or scalable.
>
>
> > Last point, we did the following test:
> > A dispatch router have a connector with a LinkCapacity=250 connected
> > to a broker and a listener with a LinkCapacity=1 connected to a consumer.
> >
>
> The important link capacity in this scenario is the 250 as it controls
> the router pre-fetch.  The consumer's link capacity of 1 is not
> relevant to this case.
>
>
> >
> > [image: Untitled Diagram.jpg]
> >
>
> I can't see the diagram, but I think I get the idea.
>
>
> >
> >
> > 1- the router 1 prefetches 250 message from the broker
> > 2- the consumer issues a credit
> > 3- the consumer receives a message from the router but does not
> acknowledge
> > 4- the consumer issues another credit
> > 5- the consumer receives a message from the router but does not
> > acknowledge again Steps 4 and 5 can be repeated until the 250 msgs
> > are all transferred to the consumer
> >
>
> This is consistent with there being only one consumer for the address on
> the network at the time the broker sent the 250 messages.
>
>
> >
> > Is this an expected behavior? consumer 1 should not acknowledge before he
> > can receive another message knowing that the link capacity of the
> listener
> > is 1?
> >
>
> Best practice for acknowledgement is for the consumer to acknowledge
> (settle) immediately after finishing the processing of the message (i.e.
> once that message is no longer consuming memory or compute resources on the
> host).  This causes the settlement state of deliveries to be directly
> related to consumer resources.  Again, the link capacity of 1 is not having
> any effect on the behavior of this scenario.
>
>
> >
> > Thanks for your explanations and help,
> >
>
> Am I to understand that your case is this?  You have a distributed work
> queue in which the time-to-process is highly variable.  Some messages are
> processed quickly and others take much longer.  You don't want to incur the
> longer latency on messages that can be handled quickly if there are many
> more fast messages than slow messages.
>
> Is it possible to know beforehand which messages are going to take long?
> Could you put these on a different queue with a different address?
>
>
> > Best regards,
> > Rabih
> >
> >
> > On Wed, Jul 17, 2019 at 6:46 PM Ted Ross <[hidden email]> wrote:
> >
> >>
> >>
> >> On Wed, Jul 17, 2019 at 12:00 PM Rabih M <[hidden email]>
> wrote:
> >>
> >>> Hello,
> >>>
> >>> We tested with LinkCapacity equal to 1 on the "normal" listener with
> >>> debug level trace+, here are our findings:
> >>> Our Cluster:
> >>> [image: Diagram.jpg]
> >>> We are using the broker-j, the consumer are connected to the dispatch
> >>> routers before we start sending.
> >>>
> >>> For use case 1:
> >>> 1- the producer sends a message.
> >>> 2- Consumer 1 issues one credit, receives the message without
> >>> acknowledging.
> >>> 3- the producer sends another message.
> >>> 4- Consumer 2 in auto-ack mode issues one credit and receives the
> >>> message.
> >>> 5- we repeated steps 3 and 4 ten times.
> >>> 6- Consumer 1 acknowledges.
> >>> The results were correct: all the messages were correctly distributed
> to
> >>> the idle consumer.
> >>>
> >>> For use case 2:
> >>> 1- the producer send 10 messages while no credits were issued yet by
> the
> >>> consumers.
> >>> 2- Consumer 1 issues one credit, receives a message without
> >>> acknowledging.
> >>> 3- Consumer 2 in auto-ack mode issues one credit and times out after 5
> >>> seconds if nothing is received.
> >>> 4- we repeated step 3 eight times.
> >>> 5- Consumer 1 acknowledges.
> >>> The results were not as expected: 4 messages were blocked in the
> >>> outbound queue of the consumer 1 and consumer 2 was able to receive
> only 5
> >>> messages.
> >>> We analysed the traces to follow the messages. We found that 4 messages
> >>> were blocked in the dispatch 1.
> >>> Conclusion: if no consumers are issuing credits (are busy) then the
> >>> incoming messages will be pre-assigned automatically by the dispatch
> router
> >>> to the listeners (in a round robin way?).
> >>>
> >>> Is it an expected behavior in the dispatch router? is it not supposed
> to
> >>> wait for a credit to be issued before binding the message to an
> outbound
> >>> queue?
> >>>
> >>
> >> Yes, this is the expected behavior.  The router does not propagate each
> >> individual credit from receiver to sender.  It would be impractical to
> do
> >> so, would not scale well, and probably still wouldn't provide the
> behavior
> >> you expect.  What the router does is to use delivery settlement as the
> way
> >> to control credit flow to producers.  If the link capacity is 250, each
> >> producer will be limited to 250 unsettled deliveries at any time.  As
> the
> >> deliveries are settled, more credit is issued.  This scales well in
> large
> >> networks, keeps a limit on the memory consumed by deliveries, and allows
> >> for high delivery rates.
> >>
> >> Is this a real use case or are you experimenting to learn how the router
> >> works?
> >>
> >> Under steady state flow, the messages will be delivered to the consumers
> >> in proportion to the rate at which the consumers acknowledge (settle)
> >> deliveries.  If a consumer attaches a receiving link but withholds
> credit,
> >> the router network will route deliveries to that consumer in
> anticipation
> >> of credit being issued.  It is an anti-pattern to attach a receiving
> link
> >> and stop issuing credit.  Credit should be used to control the rate of
> >> delivery.  If you want to stop delivery, detach the receiver.
> >>
> >> If you really want a completely synchronous transfer across your
> network,
> >> you can set the link capacity on the broker connections to 1.  This will
> >> limit the number of in-flight unsettled deliveries on each incoming
> >> auto-link to 1.  It will be slow and will be prone to stalling,
> especially
> >> if your consumers withhold acknowledgement.
> >>
> >> If you want to check the paths of the messages, I attached the routers
> >>> logs of the use case 2.
> >>>
> >>> Best regards,
> >>> Rabih
> >>>
> >>> On Mon, Jul 15, 2019 at 8:14 PM Ganesh Murthy <[hidden email]>
> >>> wrote:
> >>>
> >>>> On Mon, Jul 15, 2019 at 1:26 PM Rabih M <[hidden email]>
> >>>> wrote:
> >>>>
> >>>> > Hello,
> >>>> >
> >>>> > We are testing with the trace+ log. We will brief you of the results
> >>>> when
> >>>> > they are ready.
> >>>> >
> >>>> > Our goal is not to load balance equally the messages between the
> >>>> consumers
> >>>> > but we would like the dispatch router to send the message to the
> free
> >>>> > consumer.
> >>>> >
> >>>>
> >>>> What you did (setting the linkCapacity on the producer and consumer
> >>>> listeners to 1) is the right thing to do if you want the router
> >>>> to send the message to *any* free consumer.
> >>>>
> >>>> If Consumer C1 is processing the first message and does not issue the
> >>>> next
> >>>> credit until it finishes processing the first message and if the
> second
> >>>> message arrives to Router 1 when C1 is
> >>>> still processing the message, then Router 1 will definitely forward
> the
> >>>> message to Router 2. BUT if C1 is fast enough and ends up processing
> the
> >>>> first message and immediately issues credit, the second message if it
> >>>> arrives in Router 1 will also
> >>>> be sent to C1 (because the router prefers local consumers).
> >>>>
> >>>> Remember that the key here is which Router ends up getting the message
> >>>> since each broker has two autoLinks to both routers and we don't know
> >>>> which
> >>>> autoLink the broker will choose.
> >>>>
> >>>> But overall, with your new configuration, the router will send the
> >>>> message
> >>>> to a consumer that is not busy no further configuration is necessary.
> >>>>
> >>>> Specially if the consumer does a long calculation, we do not want to
> >>>> block
> >>>> > the message in the outbound queue knowing there is another idle
> >>>> consumer.
> >>>> >
> >>>> > Is there any special configuration for this?
> >>>> >
> >>>> > Best regards,
> >>>> > Rabih
> >>>> >
> >>>> >
> >>>> > On Fri, Jul 12, 2019 at 7:26 PM Ganesh Murthy <[hidden email]>
> >>>> wrote:
> >>>> >
> >>>> > > On Fri, Jul 12, 2019 at 11:29 AM Rabih M <[hidden email]
> >
> >>>> > wrote:
> >>>> > >
> >>>> > > > Hello,
> >>>> > > >
> >>>> > > > Thank you Ganesh for your answer.
> >>>> > > > To answer your first question yes the consumers we have, issue
> one
> >>>> > credit
> >>>> > > > per call.
> >>>> > > > For the the second question, yes we are attaching consumers
> before
> >>>> > > sending
> >>>> > > > the messages.
> >>>> > > >
> >>>> > > > After setting the dispatch "link capacity" on the listeners to
> 1,
> >>>> we
> >>>> > > > managed to have an equal load-balancing of the messages between
> >>>> > consumer
> >>>> > > 1
> >>>> > > > and 2.
> >>>> > > > Our understanding is that the problem comes from the prefetch at
> >>>> the
> >>>> > > level
> >>>> > > > of the "normal" listener, which is preventing the "inter-router"
> >>>> > listener
> >>>> > > > to pass messages to the second dispatch.
> >>>> > > >
> >>>> > > Just want to note here that the linkCapacity on the inter-router
> >>>> > listeners
> >>>> > > are ignored. So setting linkCapacity to 1 on that listener has no
> >>>> effect.
> >>>> > >
> >>>> > > This is what I think is happening
> >>>> > > 1. The Router 1 (R1) and Router 2 (R2) are started and each of
> them
> >>>> > connect
> >>>> > > to both Brokers, B1 and B2 and establish the autolinks. As soon as
> >>>> > inbound
> >>>> > > autoLinks (inbound to the router from the broker) are established
> on
> >>>> > > address myQueue, the broker(s) immediately provides initial credit
> >>>> on
> >>>> > both
> >>>> > > autoLinks. This initial credit is usually 500 or 1000 depending on
> >>>> the
> >>>> > > broker you are using.
> >>>> > > 2. Now Consumer 1 (C1) connects to R1 and Consumer 2 (C2) connects
> >>>> to R2
> >>>> > > providing one credit each respectively to each router.
> >>>> > > 3. Now the Producer connects to R1 and sends one message M1 (it
> >>>> cannot
> >>>> > send
> >>>> > > more than one message because the linkCapacity is set to 1).
> >>>> Assuming M1
> >>>> > > goes to B1 (it has an equal chance of ending up in B2 as well), B1
> >>>> sees
> >>>> > > that there are 2 consumers (autolinks) for that message.Say B1
> >>>> sends M1
> >>>> > to
> >>>> > > R1, M1 will be sent to C1.
> >>>> > > 4. Producer produces M2 and it goes to B2. B2 might send it to R1
> >>>> or R2.
> >>>> > >          4a. If B2 sends M2 to R1, the message might go over the
> >>>> > > inter-router link  to R2 to C2 because there is no credit to send
> >>>> to C1.
> >>>> > >          4b. If B2 sends M2 to R2, then C2 gets M2  because
> routers
> >>>> > prefer
> >>>> > > local consumers to remote consumers.
> >>>> > >           4c. If C1 is fast enough to consume M1 and replenish the
> >>>> credit
> >>>> > > and if B2 ends up sending M2 to R1, C1 might end up getting the
> >>>> second
> >>>> > > message as well.
> >>>> > >           4d. You need to also find out if the brokers load
> balances
> >>>> > across
> >>>> > > autolinks.
> >>>> > >
> >>>> > > In conclusion, you are lucky that the messages are being load
> >>>> balanced
> >>>> > > across C1 and C2 when you changed linkCapacity to 1 but it is
> >>>> making it
> >>>> > > more likely. This might not always happen.
> >>>> > >
> >>>> > > Turn on trace logging on the routers to see which route the
> >>>> messages are
> >>>> > > taking. Try the scenario over and over again.
> >>>> > >
> >>>> > > To turn on trace logging add the following to the router config
> >>>> file -
> >>>> > >
> >>>> > > log {
> >>>> > >     module: DEFAULT
> >>>> > >     enable: trace+
> >>>> > >     output: path/to/qdrouterd.log
> >>>> > > }
> >>>> > >
> >>>> > > If you want true load balancing between the consumers, put a
> router
> >>>> R3 in
> >>>> > > front of R1 and R2 and connect both consumers  to R3. The producer
> >>>> can
> >>>> > > connect to any router.
> >>>> > >
> >>>> > > >
> >>>> > > > You can find attached the new config files.
> >>>> > > >
> >>>> > > > Is this the correct way to resolve this kind of problem? Does it
> >>>> sound
> >>>> > > > reasonable to you?
> >>>> > > >
> >>>> > > > Best regards,
> >>>> > > > Rabih and Ali
> >>>> > > >
> >>>> > > > On Thu, Jul 11, 2019 at 5:50 PM Ganesh Murthy <
> [hidden email]
> >>>> >
> >>>> > > wrote:
> >>>> > > >
> >>>> > > >>
> >>>> > > >>
> >>>> > > >> On Thu, Jul 11, 2019 at 10:37 AM Rabih M <
> >>>> [hidden email]>
> >>>> > > wrote:
> >>>> > > >>
> >>>> > > >>> Hello,
> >>>> > > >>>
> >>>> > > >>> We are using Qpid dispatch router 1.7, Qpid broker 7.1.3 on
> >>>> redhat
> >>>> > > linux
> >>>> > > >>> rhel 6.4.
> >>>> > > >>>
> >>>> > > >>> Use case description:
> >>>> > > >>> We have a cluster of 2 dispatch routers and 2 brokers. A
> sharded
> >>>> > queue
> >>>> > > >>> "myQueue" on the 2 brokers.
> >>>> > > >>> Here is an illustration:
> >>>> > > >>>
> >>>> > > >>> [image: Diagram.jpg]
> >>>> > > >>>
> >>>> > > >>> The producer produces 2 messages, the messages are load
> >>>> balanced on
> >>>> > the
> >>>> > > >>> 2 brokers. Then, Consumer 1 and consumer 2 ask for a
> receive().
> >>>> > > >>>
> >>>> > > >> Does your receive() function issue one credit per call?
> >>>> > > >>
> >>>> > > >>>
> >>>> > > >>> Our observation is that the consumer 1 consumes the first
> >>>> message and
> >>>> > > >>> the consumer 2 is never getting the second message.
> >>>> > > >>> We are aware that the first dispatch router will do a prefetch
> >>>> for
> >>>> > the
> >>>> > > 2
> >>>> > > >>> messages but what is weird is that the prefetched message 2 is
> >>>> never
> >>>> > > routed
> >>>> > > >>> to the second dispatch router and consumer 2.
> >>>> > > >>> I attached the dispatch routers config.
> >>>> > > >>>
> >>>> > > >>> This is what is going on in my view -
> >>>> > > >> 1. The producer sends two messages - Message 1 goes to Broker 1
> >>>> and
> >>>> > > >> Message 2 goes to Broker 2.
> >>>> > > >> 2. Now Consumer 1 attaches to Router 1 and calls receive()
> >>>> issuing one
> >>>> > > >> credit. (The Consumer 2 has not yet attached to Router 2). The
> >>>> > autolinks
> >>>> > > >> from Broker 1 to Router 1 and Broker 2 to Router 2 each have a
> >>>> > prefetch
> >>>> > > of
> >>>> > > >> 250 credits each. So,Message 1 and Message 2
> >>>> > > >> both come down to Router 1. Message 1 is sent to Consumer 1.
> >>>> Message 2
> >>>> > > is
> >>>> > > >> waiting on the outbound queue of Consumer 1 waiting to receive
> a
> >>>> > credit
> >>>> > > >> from Consumer 1.
> >>>> > > >> 3. Now Consumer 2 shows up and wants to receive Message 2 but
> >>>> that
> >>>> > > >> message is already queued up to go to Consumer 1 who is still
> >>>> keeping
> >>>> > > his
> >>>> > > >> connection open
> >>>> > > >> 4. If now Consumer 1 drops out, the Message 2 will be RELEASED
> >>>> back to
> >>>> > > >> its broker and that message will be sent to Consumer 2.
> >>>> > > >>
> >>>> > > >> Have you tried attaching Consumer 1 and Consumer 2 first and
> then
> >>>> > > >> subsequently bringing up the Producer 1 and send two messages ?
> >>>> In
> >>>> > that
> >>>> > > >> case, I think, both consumers will receive one message.
> >>>> > > >>
> >>>> > > >> Thanks.
> >>>> > > >>
> >>>> > > >>> Do you have any idea how we can make consumer 2 receive the
> >>>> message?
> >>>> > > >>>
> >>>> > > >>> Thanks for your help,
> >>>> > > >>> Rabih
> >>>> > > >>>
> >>>> > > >>>
> >>>> ---------------------------------------------------------------------
> >>>> > > >>> To unsubscribe, e-mail: [hidden email]
> >>>> > > >>> For additional commands, e-mail: [hidden email]
> >>>> > > >>
> >>>> > > >>
> >>>> > > >
> >>>> ---------------------------------------------------------------------
> >>>> > > > To unsubscribe, e-mail: [hidden email]
> >>>> > > > For additional commands, e-mail: [hidden email]
> >>>> > >
> >>>> >
> >>>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: [hidden email]
> >>> For additional commands, e-mail: [hidden email]
> >>
> >>
>
*******************************
This e-mail contains information for the intended recipient only. It may contain proprietary material or confidential information. If you are not the intended recipient you are not authorized to distribute, copy or use this e-mail or any attachment to it. Murex cannot guarantee that it is virus free and accepts no responsibility for any loss or damage arising from its use. If you have received this e-mail in error please notify immediately the sender and delete the original email received, any attachments and all copies from your system.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Uneven distribution of messages

Gordon Sim
On 24/07/2019 10:43 am, HADI Ali wrote:
> Having different linkCapacity for incoming and outgoing links might solve the problem. Do you think it's a good practice to do that ? And do you think it is feasible to have that without having different entrypoints (listener) for each dispatch router?

At present the capacity is configured on a listener, so to have
different values you need different listeners.

The link capacity has no effect on consumers who control their own
credit. It is used only to determine how many outstanding
(unacknowledged) messages to allow from a producer.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]