Dispatch Router prefetch

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

Dispatch Router prefetch

ali hadi
Hello,



We are using a cluster with one Dispatch Router version 1.5.0 and one
Broker-J version 7.1.0 on which we created a queue with a TTL of 5 seconds .


We are noticing that the dispatch is prefetching messages from the broker (
messages become in acquired state ) as soon as the consumer establishes a
connection with the dispatch and before starting consuming. This is causing
the messages to not be discarded after the TTL expires and the consumer
receiving the expired message.


We tried changing the linkCapacity on the level of the connector to 1 which
allowed us to prefetch one message only instead of the default 250.


We are trying to find a way to remove completely the prefetch of the
dispatch in order to have the correct behavior from the TTL with our
cluster.

Are there any flags or properties to be set in order for the dispatch to
only fetch a message on consumer demand?



Thank you,

Ali
Reply | Threaded
Open this post in threaded view
|

Re: Dispatch Router prefetch

Gordon Sim
On 06/03/2019 1:20 pm, ali hadi wrote:

> We are using a cluster with one Dispatch Router version 1.5.0 and one
> Broker-J version 7.1.0 on which we created a queue with a TTL of 5 seconds .
>
>
> We are noticing that the dispatch is prefetching messages from the broker (
> messages become in acquired state ) as soon as the consumer establishes a
> connection with the dispatch and before starting consuming. This is causing
> the messages to not be discarded after the TTL expires and the consumer
> receiving the expired message.
>
>
> We tried changing the linkCapacity on the level of the connector to 1 which
> allowed us to prefetch one message only instead of the default 250.
>
>
> We are trying to find a way to remove completely the prefetch of the
> dispatch in order to have the correct behavior from the TTL with our
> cluster.
>
> Are there any flags or properties to be set in order for the dispatch to
> only fetch a message on consumer demand?

When using autolinks, the autolink for messages from the broker to the
router will at present be activate as soon as there is an active
receiver for the messages, whether or not that receiver has credit. In
message routing the credit is not directly propagated from client to broker.

If you use a link route then credit would be propagated directly, i.e.
only when your client issues credit will the link between broker and
router get credit (and it will be the exact same amount).


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Dispatch Router prefetch

HADI Ali
Hello,

We are actually using in our cluster multiple brokers and thus we need to define the same address on multiple brokers.
For this, we cannot use linkroutes as suggested, but we still need to have the correct behavior of the TTL in our cluster.

Is it an option to manage the TTL of the message at the level of the dispatch router since we have all of the information needed in the message headers?
In Internet Protocol, ipv4 for example, the routers manage the TTL and discard any expired messages.

Or make it feasible to have the autolinks propagate the credit directly from consumers? Therefore the dispatch router will only transit messages and the broker will handle the lifecycle of the message.

Thank you,
Ali

-----Original Message-----
From: Gordon Sim <[hidden email]>
Sent: mercredi 6 mars 2019 16:25
To: [hidden email]
Subject: Re: Dispatch Router prefetch

On 06/03/2019 1:20 pm, ali hadi wrote:

> We are using a cluster with one Dispatch Router version 1.5.0 and one
> Broker-J version 7.1.0 on which we created a queue with a TTL of 5 seconds .
>
>
> We are noticing that the dispatch is prefetching messages from the
> broker ( messages become in acquired state ) as soon as the consumer
> establishes a connection with the dispatch and before starting
> consuming. This is causing the messages to not be discarded after the
> TTL expires and the consumer receiving the expired message.
>
>
> We tried changing the linkCapacity on the level of the connector to 1
> which allowed us to prefetch one message only instead of the default 250.
>
>
> We are trying to find a way to remove completely the prefetch of the
> dispatch in order to have the correct behavior from the TTL with our
> cluster.
>
> Are there any flags or properties to be set in order for the dispatch
> to only fetch a message on consumer demand?

When using autolinks, the autolink for messages from the broker to the router will at present be activate as soon as there is an active receiver for the messages, whether or not that receiver has credit. In message routing the credit is not directly propagated from client to broker.

If you use a link route then credit would be propagated directly, i.e.
only when your client issues credit will the link between broker and router get credit (and it will be the exact same amount).


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

*******************************
This e-mail contains information for the intended recipient only. It may contain proprietary material or confidential information. If you are not the intended recipient you are not authorized to distribute, copy or use this e-mail or any attachment to it. Murex cannot guarantee that it is virus free and accepts no responsibility for any loss or damage arising from its use. If you have received this e-mail in error please notify immediately the sender and delete the original email received, any attachments and all copies from your system.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Dispatch Router prefetch

Ganesh Murthy
On Fri, Mar 8, 2019 at 7:59 AM HADI Ali <[hidden email]> wrote:

> Hello,
>
> We are actually using in our cluster multiple brokers and thus we need to
> define the same address on multiple brokers.
> For this, we cannot use linkroutes as suggested, but we still need to have
> the correct behavior of the TTL in our cluster.
>
> Is it an option to manage the TTL of the message at the level of the
> dispatch router since we have all of the information needed in the message
> headers?
>
At present the router does not deal with TTL. We would welcome a pull
request if you would like to have this feature in the router. Thanks.

> In Internet Protocol, ipv4 for example, the routers manage the TTL and
> discard any expired messages.
>
> Or make it feasible to have the autolinks propagate the credit directly
> from consumers? Therefore the dispatch router will only transit messages
> and the broker will handle the lifecycle of the message.
>
> Thank you,
> Ali
>
> -----Original Message-----
> From: Gordon Sim <[hidden email]>
> Sent: mercredi 6 mars 2019 16:25
> To: [hidden email]
> Subject: Re: Dispatch Router prefetch
>
> On 06/03/2019 1:20 pm, ali hadi wrote:
> > We are using a cluster with one Dispatch Router version 1.5.0 and one
> > Broker-J version 7.1.0 on which we created a queue with a TTL of 5
> seconds .
> >
> >
> > We are noticing that the dispatch is prefetching messages from the
> > broker ( messages become in acquired state ) as soon as the consumer
> > establishes a connection with the dispatch and before starting
> > consuming. This is causing the messages to not be discarded after the
> > TTL expires and the consumer receiving the expired message.
> >
> >
> > We tried changing the linkCapacity on the level of the connector to 1
> > which allowed us to prefetch one message only instead of the default 250.
> >
> >
> > We are trying to find a way to remove completely the prefetch of the
> > dispatch in order to have the correct behavior from the TTL with our
> > cluster.
> >
> > Are there any flags or properties to be set in order for the dispatch
> > to only fetch a message on consumer demand?
>
> When using autolinks, the autolink for messages from the broker to the
> router will at present be activate as soon as there is an active receiver
> for the messages, whether or not that receiver has credit. In message
> routing the credit is not directly propagated from client to broker.
>
> If you use a link route then credit would be propagated directly, i.e.
> only when your client issues credit will the link between broker and
> router get credit (and it will be the exact same amount).
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
> *******************************
> This e-mail contains information for the intended recipient only. It may
> contain proprietary material or confidential information. If you are not
> the intended recipient you are not authorized to distribute, copy or use
> this e-mail or any attachment to it. Murex cannot guarantee that it is
> virus free and accepts no responsibility for any loss or damage arising
> from its use. If you have received this e-mail in error please notify
> immediately the sender and delete the original email received, any
> attachments and all copies from your system.
>
Reply | Threaded
Open this post in threaded view
|

Re: Dispatch Router prefetch

Gordon Sim
In reply to this post by HADI Ali
On 08/03/2019 12:59 pm, HADI Ali wrote:
> Hello,
>
> We are actually using in our cluster multiple brokers and thus we need to define the same address on multiple brokers.
> For this, we cannot use linkroutes as suggested, but we still need to have the correct behavior of the TTL in our cluster.
>
> Is it an option to manage the TTL of the message at the level of the dispatch router since we have all of the information needed in the message headers?

It doesn't do that at present, but it doesn't seem like an reasonable
enhancement to me.

> In Internet Protocol, ipv4 for example, the routers manage the TTL and discard any expired messages.
>
> Or make it feasible to have the autolinks propagate the credit directly from consumers?

This isn't really possible when you have autolinks for same address to
multiple brokers. If the consumer gives 10 credits, how do you propagate
that to two brokers?  5 each? What if they don't both have 5 messages?
10 each? Then you are back to the situation where you have more credit
issued at source than the consumer has granted.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Dispatch Router prefetch

Gordon Sim
On 08/03/2019 2:12 pm, Gordon Sim wrote:

> On 08/03/2019 12:59 pm, HADI Ali wrote:
>> Hello,
>>
>> We are actually using in our cluster multiple brokers and thus we need
>> to define the same address on multiple brokers.
>> For this, we cannot use linkroutes as suggested, but we still need to
>> have the correct behavior of the TTL in our cluster.
>>
>> Is it an option to manage the TTL of the message at the level of the
>> dispatch router since we have all of the information needed in the
>> message headers?
>
> It doesn't do that at present, but it doesn't seem like an reasonable
> enhancement to me.

Sorry, meant to say it doesn't seem like an *un*reasonable enhancement!

>> In Internet Protocol, ipv4 for example, the routers manage the TTL and
>> discard any expired messages.
>>
>> Or make it feasible to have the autolinks propagate the credit
>> directly from consumers?
>
> This isn't really possible when you have autolinks for same address to
> multiple brokers. If the consumer gives 10 credits, how do you propagate
> that to two brokers?  5 each? What if they don't both have 5 messages?
> 10 each? Then you are back to the situation where you have more credit
> issued at source than the consumer has granted.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Dispatch Router prefetch

Ted Ross
On Fri, Mar 8, 2019 at 9:19 AM Gordon Sim <[hidden email]> wrote:

> On 08/03/2019 2:12 pm, Gordon Sim wrote:
> > On 08/03/2019 12:59 pm, HADI Ali wrote:
> >> Hello,
> >>
> >> We are actually using in our cluster multiple brokers and thus we need
> >> to define the same address on multiple brokers.
> >> For this, we cannot use linkroutes as suggested, but we still need to
> >> have the correct behavior of the TTL in our cluster.
> >>
> >> Is it an option to manage the TTL of the message at the level of the
> >> dispatch router since we have all of the information needed in the
> >> message headers?
> >
> > It doesn't do that at present, but it doesn't seem like an reasonable
> > enhancement to me.
>
> Sorry, meant to say it doesn't seem like an *un*reasonable enhancement!
>

I'd like to better understand the use case here.  We've avoided adding any
kind of TTL support in Dispatch Router up to this point.

I assume, based on the fact that prefetch-1 didn't solve your problem, that
you have consumers that are attached but don't issue credit for long
periods of time.  Is this accurate?

What is the pattern of your consumers?  Do they attach, then later issue
credit to process a message?  How many messages per second/minute/hour do
your consumers handle?  Do they issue one credit at a time?

What are the typical TTLs in your messages?  How granular does the
expiration need to be (i.e. how accurate of a timer would need to be used
to tag each incoming delivery)?  Would one-second granularity be
sufficient, or do you need milliseconds?

An alternate approach would be to not consider a consumer to be a routable
destination until it issues initial credit.  Would this address your
problem?


>
> >> In Internet Protocol, ipv4 for example, the routers manage the TTL and
> >> discard any expired messages.
> >>
> >> Or make it feasible to have the autolinks propagate the credit
> >> directly from consumers?
> >
> > This isn't really possible when you have autolinks for same address to
> > multiple brokers. If the consumer gives 10 credits, how do you propagate
> > that to two brokers?  5 each? What if they don't both have 5 messages?
> > 10 each? Then you are back to the situation where you have more credit
> > issued at source than the consumer has granted.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Dispatch Router prefetch

Gordon Sim
On 11/03/2019 2:32 pm, Ted Ross wrote:
> We've avoided adding any kind of TTL support in Dispatch Router up to
> this point.

Dropping an expired message is less work than delivering it, even
delivering pre-settled, I suspect.

The ttl is defined in the header of the message, before annotations, so
in the case of message routing at least we will always have read past
that point. Parsing out the ttl will have some cost, but is suspect not
a huge one.

I wouldn't advocate setting up timers to trigger the processing of
expired messages, but wherever it makes sense it could be part of the
processing of deliveries, as a way of saving effort as much as anything.

I probably wouldn't bother too much initially with adjusting the ttl
either, as generally messages flowing into the router network will
pretty quickly reach the egress router (which is where at present they
are 'delayed' until there is credit).

Anyway, not advocating that this is needed, just commenting that I don't
think it needs to have a negative effect on efficiency (indeed it could
even be seen as an optimisation).

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Dispatch Router prefetch

Ted Ross
On Mon, Mar 11, 2019 at 11:26 AM Gordon Sim <[hidden email]> wrote:

> On 11/03/2019 2:32 pm, Ted Ross wrote:
> > We've avoided adding any kind of TTL support in Dispatch Router up to
> > this point.
>
> Dropping an expired message is less work than delivering it, even
> delivering pre-settled, I suspect.
>
> The ttl is defined in the header of the message, before annotations, so
> in the case of message routing at least we will always have read past
> that point. Parsing out the ttl will have some cost, but is suspect not
> a huge one.
>
> I wouldn't advocate setting up timers to trigger the processing of
> expired messages, but wherever it makes sense it could be part of the
> processing of deliveries, as a way of saving effort as much as anything.
>

Agreed.  We would need to store with the delivery an arrival timestamp of
sufficient granularity to satisfy the requirements.  If the granularity is
large (one second), this will have very little impact on the performance of
the router.


>
> I probably wouldn't bother too much initially with adjusting the ttl
> either, as generally messages flowing into the router network will
> pretty quickly reach the egress router (which is where at present they
> are 'delayed' until there is credit).
>
> Anyway, not advocating that this is needed, just commenting that I don't
> think it needs to have a negative effect on efficiency (indeed it could
> even be seen as an optimisation).
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

RE: Dispatch Router prefetch

HADI Ali
In reply to this post by Ted Ross
Hello,

In our use case we have polling consumers with a prefetch policy of zero that issues one credit at a time every few seconds. Between two receive, the consumer will be attached with zero credit.
Thus, not considering a consumer to be a routable destination until it issues initial credit would address the problem only for the first message, because the dispatch will still prefetch possibly expired messages as soon as the destination is considered routable.

In this use case we are consuming a few messages per minutes and TTLs are between 2 to 5 seconds. Concerning the granularity, one second should be sufficient for us.

We also noticed that the broker is not forwarding the TTL set at the level of the queue. Is this an expected behavior?

Thanks,
Ali

-----Original Message-----
From: Ted Ross <[hidden email]>
Sent: lundi 11 mars 2019 15:32
To: [hidden email]
Subject: Re: Dispatch Router prefetch

On Fri, Mar 8, 2019 at 9:19 AM Gordon Sim <[hidden email]> wrote:

> On 08/03/2019 2:12 pm, Gordon Sim wrote:
> > On 08/03/2019 12:59 pm, HADI Ali wrote:
> >> Hello,
> >>
> >> We are actually using in our cluster multiple brokers and thus we
> >> need to define the same address on multiple brokers.
> >> For this, we cannot use linkroutes as suggested, but we still need
> >> to have the correct behavior of the TTL in our cluster.
> >>
> >> Is it an option to manage the TTL of the message at the level of
> >> the dispatch router since we have all of the information needed in
> >> the message headers?
> >
> > It doesn't do that at present, but it doesn't seem like an
> > reasonable enhancement to me.
>
> Sorry, meant to say it doesn't seem like an *un*reasonable enhancement!
>

I'd like to better understand the use case here.  We've avoided adding any kind of TTL support in Dispatch Router up to this point.

I assume, based on the fact that prefetch-1 didn't solve your problem, that you have consumers that are attached but don't issue credit for long periods of time.  Is this accurate?

What is the pattern of your consumers?  Do they attach, then later issue credit to process a message?  How many messages per second/minute/hour do your consumers handle?  Do they issue one credit at a time?

What are the typical TTLs in your messages?  How granular does the expiration need to be (i.e. how accurate of a timer would need to be used to tag each incoming delivery)?  Would one-second granularity be sufficient, or do you need milliseconds?

An alternate approach would be to not consider a consumer to be a routable destination until it issues initial credit.  Would this address your problem?


>
> >> In Internet Protocol, ipv4 for example, the routers manage the TTL
> >> and discard any expired messages.
> >>
> >> Or make it feasible to have the autolinks propagate the credit
> >> directly from consumers?
> >
> > This isn't really possible when you have autolinks for same address
> > to multiple brokers. If the consumer gives 10 credits, how do you
> > propagate that to two brokers?  5 each? What if they don't both have 5 messages?
> > 10 each? Then you are back to the situation where you have more
> > credit issued at source than the consumer has granted.
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: [hidden email] For
> > additional commands, e-mail: [hidden email]
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email] For
> additional commands, e-mail: [hidden email]
>
>
*******************************
This e-mail contains information for the intended recipient only. It may contain proprietary material or confidential information. If you are not the intended recipient you are not authorized to distribute, copy or use this e-mail or any attachment to it. Murex cannot guarantee that it is virus free and accepts no responsibility for any loss or damage arising from its use. If you have received this e-mail in error please notify immediately the sender and delete the original email received, any attachments and all copies from your system.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Dispatch Router prefetch

Robbie Gemmell
Administrator
What acknowledgement mode mode are you using?

On Tue, 12 Mar 2019 at 13:22, HADI Ali <[hidden email]> wrote:

>
> Hello,
>
> In our use case we have polling consumers with a prefetch policy of zero that issues one credit at a time every few seconds. Between two receive, the consumer will be attached with zero credit.
> Thus, not considering a consumer to be a routable destination until it issues initial credit would address the problem only for the first message, because the dispatch will still prefetch possibly expired messages as soon as the destination is considered routable.
>
> In this use case we are consuming a few messages per minutes and TTLs are between 2 to 5 seconds. Concerning the granularity, one second should be sufficient for us.
>
> We also noticed that the broker is not forwarding the TTL set at the level of the queue. Is this an expected behavior?
>
> Thanks,
> Ali
>
> -----Original Message-----
> From: Ted Ross <[hidden email]>
> Sent: lundi 11 mars 2019 15:32
> To: [hidden email]
> Subject: Re: Dispatch Router prefetch
>
> On Fri, Mar 8, 2019 at 9:19 AM Gordon Sim <[hidden email]> wrote:
>
> > On 08/03/2019 2:12 pm, Gordon Sim wrote:
> > > On 08/03/2019 12:59 pm, HADI Ali wrote:
> > >> Hello,
> > >>
> > >> We are actually using in our cluster multiple brokers and thus we
> > >> need to define the same address on multiple brokers.
> > >> For this, we cannot use linkroutes as suggested, but we still need
> > >> to have the correct behavior of the TTL in our cluster.
> > >>
> > >> Is it an option to manage the TTL of the message at the level of
> > >> the dispatch router since we have all of the information needed in
> > >> the message headers?
> > >
> > > It doesn't do that at present, but it doesn't seem like an
> > > reasonable enhancement to me.
> >
> > Sorry, meant to say it doesn't seem like an *un*reasonable enhancement!
> >
>
> I'd like to better understand the use case here.  We've avoided adding any kind of TTL support in Dispatch Router up to this point.
>
> I assume, based on the fact that prefetch-1 didn't solve your problem, that you have consumers that are attached but don't issue credit for long periods of time.  Is this accurate?
>
> What is the pattern of your consumers?  Do they attach, then later issue credit to process a message?  How many messages per second/minute/hour do your consumers handle?  Do they issue one credit at a time?
>
> What are the typical TTLs in your messages?  How granular does the expiration need to be (i.e. how accurate of a timer would need to be used to tag each incoming delivery)?  Would one-second granularity be sufficient, or do you need milliseconds?
>
> An alternate approach would be to not consider a consumer to be a routable destination until it issues initial credit.  Would this address your problem?
>
>
> >
> > >> In Internet Protocol, ipv4 for example, the routers manage the TTL
> > >> and discard any expired messages.
> > >>
> > >> Or make it feasible to have the autolinks propagate the credit
> > >> directly from consumers?
> > >
> > > This isn't really possible when you have autolinks for same address
> > > to multiple brokers. If the consumer gives 10 credits, how do you
> > > propagate that to two brokers?  5 each? What if they don't both have 5 messages?
> > > 10 each? Then you are back to the situation where you have more
> > > credit issued at source than the consumer has granted.
> > >
> > > --------------------------------------------------------------------
> > > - To unsubscribe, e-mail: [hidden email] For
> > > additional commands, e-mail: [hidden email]
> > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email] For
> > additional commands, e-mail: [hidden email]
> >
> >
> *******************************
> This e-mail contains information for the intended recipient only. It may contain proprietary material or confidential information. If you are not the intended recipient you are not authorized to distribute, copy or use this e-mail or any attachment to it. Murex cannot guarantee that it is virus free and accepts no responsibility for any loss or damage arising from its use. If you have received this e-mail in error please notify immediately the sender and delete the original email received, any attachments and all copies from your system.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Dispatch Router prefetch

HADI Ali
We support both, it depends on the use case (we have multiple services using the messaging).

-----Original Message-----
From: Robbie Gemmell <[hidden email]>
Sent: mardi 12 mars 2019 15:15
To: [hidden email]
Subject: Re: Dispatch Router prefetch

What acknowledgement mode mode are you using?

On Tue, 12 Mar 2019 at 13:22, HADI Ali <[hidden email]> wrote:

>
> Hello,
>
> In our use case we have polling consumers with a prefetch policy of zero that issues one credit at a time every few seconds. Between two receive, the consumer will be attached with zero credit.
> Thus, not considering a consumer to be a routable destination until it issues initial credit would address the problem only for the first message, because the dispatch will still prefetch possibly expired messages as soon as the destination is considered routable.
>
> In this use case we are consuming a few messages per minutes and TTLs are between 2 to 5 seconds. Concerning the granularity, one second should be sufficient for us.
>
> We also noticed that the broker is not forwarding the TTL set at the level of the queue. Is this an expected behavior?
>
> Thanks,
> Ali
>
> -----Original Message-----
> From: Ted Ross <[hidden email]>
> Sent: lundi 11 mars 2019 15:32
> To: [hidden email]
> Subject: Re: Dispatch Router prefetch
>
> On Fri, Mar 8, 2019 at 9:19 AM Gordon Sim <[hidden email]> wrote:
>
> > On 08/03/2019 2:12 pm, Gordon Sim wrote:
> > > On 08/03/2019 12:59 pm, HADI Ali wrote:
> > >> Hello,
> > >>
> > >> We are actually using in our cluster multiple brokers and thus we
> > >> need to define the same address on multiple brokers.
> > >> For this, we cannot use linkroutes as suggested, but we still
> > >> need to have the correct behavior of the TTL in our cluster.
> > >>
> > >> Is it an option to manage the TTL of the message at the level of
> > >> the dispatch router since we have all of the information needed
> > >> in the message headers?
> > >
> > > It doesn't do that at present, but it doesn't seem like an
> > > reasonable enhancement to me.
> >
> > Sorry, meant to say it doesn't seem like an *un*reasonable enhancement!
> >
>
> I'd like to better understand the use case here.  We've avoided adding any kind of TTL support in Dispatch Router up to this point.
>
> I assume, based on the fact that prefetch-1 didn't solve your problem, that you have consumers that are attached but don't issue credit for long periods of time.  Is this accurate?
>
> What is the pattern of your consumers?  Do they attach, then later issue credit to process a message?  How many messages per second/minute/hour do your consumers handle?  Do they issue one credit at a time?
>
> What are the typical TTLs in your messages?  How granular does the expiration need to be (i.e. how accurate of a timer would need to be used to tag each incoming delivery)?  Would one-second granularity be sufficient, or do you need milliseconds?
>
> An alternate approach would be to not consider a consumer to be a routable destination until it issues initial credit.  Would this address your problem?
>
>
> >
> > >> In Internet Protocol, ipv4 for example, the routers manage the
> > >> TTL and discard any expired messages.
> > >>
> > >> Or make it feasible to have the autolinks propagate the credit
> > >> directly from consumers?
> > >
> > > This isn't really possible when you have autolinks for same
> > > address to multiple brokers. If the consumer gives 10 credits, how
> > > do you propagate that to two brokers?  5 each? What if they don't both have 5 messages?
> > > 10 each? Then you are back to the situation where you have more
> > > credit issued at source than the consumer has granted.
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail: [hidden email] For
> > > additional commands, e-mail: [hidden email]
> > >
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: [hidden email] For
> > additional commands, e-mail: [hidden email]
> >
> >
> *******************************
> This e-mail contains information for the intended recipient only. It may contain proprietary material or confidential information. If you are not the intended recipient you are not authorized to distribute, copy or use this e-mail or any attachment to it. Murex cannot guarantee that it is virus free and accepts no responsibility for any loss or damage arising from its use. If you have received this e-mail in error please notify immediately the sender and delete the original email received, any attachments and all copies from your system.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email] For
> additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email]

*******************************
This e-mail contains information for the intended recipient only. It may contain proprietary material or confidential information. If you are not the intended recipient you are not authorized to distribute, copy or use this e-mail or any attachment to it. Murex cannot guarantee that it is virus free and accepts no responsibility for any loss or damage arising from its use. If you have received this e-mail in error please notify immediately the sender and delete the original email received, any attachments and all copies from your system.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Dispatch Router prefetch

HADI Ali
Hello,

Concerning handling the TTL at the level of the Dispatch Router, should we open a Jira ticket to track the issue and continue the discussion ?
Depending on the priority of this issue on both sides, we are open to contribute if needed.

Regards,
Ali

-----Original Message-----
From: HADI Ali
Sent: mercredi 13 mars 2019 11:14
To: [hidden email]
Subject: RE: Dispatch Router prefetch

We support both, it depends on the use case (we have multiple services using the messaging).

-----Original Message-----
From: Robbie Gemmell <[hidden email]>
Sent: mardi 12 mars 2019 15:15
To: [hidden email]
Subject: Re: Dispatch Router prefetch

What acknowledgement mode mode are you using?

On Tue, 12 Mar 2019 at 13:22, HADI Ali <[hidden email]> wrote:

>
> Hello,
>
> In our use case we have polling consumers with a prefetch policy of zero that issues one credit at a time every few seconds. Between two receive, the consumer will be attached with zero credit.
> Thus, not considering a consumer to be a routable destination until it issues initial credit would address the problem only for the first message, because the dispatch will still prefetch possibly expired messages as soon as the destination is considered routable.
>
> In this use case we are consuming a few messages per minutes and TTLs are between 2 to 5 seconds. Concerning the granularity, one second should be sufficient for us.
>
> We also noticed that the broker is not forwarding the TTL set at the level of the queue. Is this an expected behavior?
>
> Thanks,
> Ali
>
> -----Original Message-----
> From: Ted Ross <[hidden email]>
> Sent: lundi 11 mars 2019 15:32
> To: [hidden email]
> Subject: Re: Dispatch Router prefetch
>
> On Fri, Mar 8, 2019 at 9:19 AM Gordon Sim <[hidden email]> wrote:
>
> > On 08/03/2019 2:12 pm, Gordon Sim wrote:
> > > On 08/03/2019 12:59 pm, HADI Ali wrote:
> > >> Hello,
> > >>
> > >> We are actually using in our cluster multiple brokers and thus we
> > >> need to define the same address on multiple brokers.
> > >> For this, we cannot use linkroutes as suggested, but we still
> > >> need to have the correct behavior of the TTL in our cluster.
> > >>
> > >> Is it an option to manage the TTL of the message at the level of
> > >> the dispatch router since we have all of the information needed
> > >> in the message headers?
> > >
> > > It doesn't do that at present, but it doesn't seem like an
> > > reasonable enhancement to me.
> >
> > Sorry, meant to say it doesn't seem like an *un*reasonable enhancement!
> >
>
> I'd like to better understand the use case here.  We've avoided adding any kind of TTL support in Dispatch Router up to this point.
>
> I assume, based on the fact that prefetch-1 didn't solve your problem, that you have consumers that are attached but don't issue credit for long periods of time.  Is this accurate?
>
> What is the pattern of your consumers?  Do they attach, then later issue credit to process a message?  How many messages per second/minute/hour do your consumers handle?  Do they issue one credit at a time?
>
> What are the typical TTLs in your messages?  How granular does the expiration need to be (i.e. how accurate of a timer would need to be used to tag each incoming delivery)?  Would one-second granularity be sufficient, or do you need milliseconds?
>
> An alternate approach would be to not consider a consumer to be a routable destination until it issues initial credit.  Would this address your problem?
>
>
> >
> > >> In Internet Protocol, ipv4 for example, the routers manage the
> > >> TTL and discard any expired messages.
> > >>
> > >> Or make it feasible to have the autolinks propagate the credit
> > >> directly from consumers?
> > >
> > > This isn't really possible when you have autolinks for same
> > > address to multiple brokers. If the consumer gives 10 credits, how
> > > do you propagate that to two brokers?  5 each? What if they don't both have 5 messages?
> > > 10 each? Then you are back to the situation where you have more
> > > credit issued at source than the consumer has granted.
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail: [hidden email] For
> > > additional commands, e-mail: [hidden email]
> > >
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: [hidden email] For
> > additional commands, e-mail: [hidden email]
> >
> >
> *******************************
> This e-mail contains information for the intended recipient only. It may contain proprietary material or confidential information. If you are not the intended recipient you are not authorized to distribute, copy or use this e-mail or any attachment to it. Murex cannot guarantee that it is virus free and accepts no responsibility for any loss or damage arising from its use. If you have received this e-mail in error please notify immediately the sender and delete the original email received, any attachments and all copies from your system.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email] For
> additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email]

*******************************
This e-mail contains information for the intended recipient only. It may contain proprietary material or confidential information. If you are not the intended recipient you are not authorized to distribute, copy or use this e-mail or any attachment to it. Murex cannot guarantee that it is virus free and accepts no responsibility for any loss or damage arising from its use. If you have received this e-mail in error please notify immediately the sender and delete the original email received, any attachments and all copies from your system.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Dispatch Router prefetch

Ted Ross
Yes, please do.  Jira is the right place to capture the requirements for
this feature.

-Ted

On Tue, Mar 19, 2019 at 9:27 AM HADI Ali <[hidden email]> wrote:

> Hello,
>
> Concerning handling the TTL at the level of the Dispatch Router, should we
> open a Jira ticket to track the issue and continue the discussion ?
> Depending on the priority of this issue on both sides, we are open to
> contribute if needed.
>
> Regards,
> Ali
>
> -----Original Message-----
> From: HADI Ali
> Sent: mercredi 13 mars 2019 11:14
> To: [hidden email]
> Subject: RE: Dispatch Router prefetch
>
> We support both, it depends on the use case (we have multiple services
> using the messaging).
>
> -----Original Message-----
> From: Robbie Gemmell <[hidden email]>
> Sent: mardi 12 mars 2019 15:15
> To: [hidden email]
> Subject: Re: Dispatch Router prefetch
>
> What acknowledgement mode mode are you using?
>
> On Tue, 12 Mar 2019 at 13:22, HADI Ali <[hidden email]> wrote:
> >
> > Hello,
> >
> > In our use case we have polling consumers with a prefetch policy of zero
> that issues one credit at a time every few seconds. Between two receive,
> the consumer will be attached with zero credit.
> > Thus, not considering a consumer to be a routable destination until it
> issues initial credit would address the problem only for the first message,
> because the dispatch will still prefetch possibly expired messages as soon
> as the destination is considered routable.
> >
> > In this use case we are consuming a few messages per minutes and TTLs
> are between 2 to 5 seconds. Concerning the granularity, one second should
> be sufficient for us.
> >
> > We also noticed that the broker is not forwarding the TTL set at the
> level of the queue. Is this an expected behavior?
> >
> > Thanks,
> > Ali
> >
> > -----Original Message-----
> > From: Ted Ross <[hidden email]>
> > Sent: lundi 11 mars 2019 15:32
> > To: [hidden email]
> > Subject: Re: Dispatch Router prefetch
> >
> > On Fri, Mar 8, 2019 at 9:19 AM Gordon Sim <[hidden email]> wrote:
> >
> > > On 08/03/2019 2:12 pm, Gordon Sim wrote:
> > > > On 08/03/2019 12:59 pm, HADI Ali wrote:
> > > >> Hello,
> > > >>
> > > >> We are actually using in our cluster multiple brokers and thus we
> > > >> need to define the same address on multiple brokers.
> > > >> For this, we cannot use linkroutes as suggested, but we still
> > > >> need to have the correct behavior of the TTL in our cluster.
> > > >>
> > > >> Is it an option to manage the TTL of the message at the level of
> > > >> the dispatch router since we have all of the information needed
> > > >> in the message headers?
> > > >
> > > > It doesn't do that at present, but it doesn't seem like an
> > > > reasonable enhancement to me.
> > >
> > > Sorry, meant to say it doesn't seem like an *un*reasonable enhancement!
> > >
> >
> > I'd like to better understand the use case here.  We've avoided adding
> any kind of TTL support in Dispatch Router up to this point.
> >
> > I assume, based on the fact that prefetch-1 didn't solve your problem,
> that you have consumers that are attached but don't issue credit for long
> periods of time.  Is this accurate?
> >
> > What is the pattern of your consumers?  Do they attach, then later issue
> credit to process a message?  How many messages per second/minute/hour do
> your consumers handle?  Do they issue one credit at a time?
> >
> > What are the typical TTLs in your messages?  How granular does the
> expiration need to be (i.e. how accurate of a timer would need to be used
> to tag each incoming delivery)?  Would one-second granularity be
> sufficient, or do you need milliseconds?
> >
> > An alternate approach would be to not consider a consumer to be a
> routable destination until it issues initial credit.  Would this address
> your problem?
> >
> >
> > >
> > > >> In Internet Protocol, ipv4 for example, the routers manage the
> > > >> TTL and discard any expired messages.
> > > >>
> > > >> Or make it feasible to have the autolinks propagate the credit
> > > >> directly from consumers?
> > > >
> > > > This isn't really possible when you have autolinks for same
> > > > address to multiple brokers. If the consumer gives 10 credits, how
> > > > do you propagate that to two brokers?  5 each? What if they don't
> both have 5 messages?
> > > > 10 each? Then you are back to the situation where you have more
> > > > credit issued at source than the consumer has granted.
> > > >
> > > > ------------------------------------------------------------------
> > > > --
> > > > - To unsubscribe, e-mail: [hidden email] For
> > > > additional commands, e-mail: [hidden email]
> > > >
> > >
> > >
> > > --------------------------------------------------------------------
> > > - To unsubscribe, e-mail: [hidden email] For
> > > additional commands, e-mail: [hidden email]
> > >
> > >
> > *******************************
> > This e-mail contains information for the intended recipient only. It may
> contain proprietary material or confidential information. If you are not
> the intended recipient you are not authorized to distribute, copy or use
> this e-mail or any attachment to it. Murex cannot guarantee that it is
> virus free and accepts no responsibility for any loss or damage arising
> from its use. If you have received this e-mail in error please notify
> immediately the sender and delete the original email received, any
> attachments and all copies from your system.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email] For
> > additional commands, e-mail: [hidden email]
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email] For additional
> commands, e-mail: [hidden email]
>
> *******************************
> This e-mail contains information for the intended recipient only. It may
> contain proprietary material or confidential information. If you are not
> the intended recipient you are not authorized to distribute, copy or use
> this e-mail or any attachment to it. Murex cannot guarantee that it is
> virus free and accepts no responsibility for any loss or damage arising
> from its use. If you have received this e-mail in error please notify
> immediately the sender and delete the original email received, any
> attachments and all copies from your system.
>