Dispatch router 2-phase start

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Dispatch router 2-phase start

Adel Boutros
Hello,

We have noticed a race conditon with the dispatch router's behavior.
If you have a producer and a consumer exchanging messages on a queue configured on a broker and accessible via the router. The consumer and producers are JMS clients configured with Failover options for retry.
If the router goes down, the retry mechanism will kick in until it is up again.

As we are configuring th addresses and connectors on the router dynamically, the producer and consumer might connect to the router before the waypointed address is created on it. In this case, a local address will be created on the router. The clients would exchange the messages directly from the router and the messages on the broker would never be consumed.

I discussed this issue with Justin and Ulf during the RivieraDev conference and we were wondering if it was possible to implement a 2-phase startup on the router to avoid this issue.
In that case, the router would start but not accept any connetion except for management. Once all dynamic configuration is done, we send a management message to allow the router to start accepting connections.

Any thoughts on this?

Regards,
Adel

Get Outlook for Android<https://aka.ms/ghei36>

Reply | Threaded
Open this post in threaded view
|

Re: Dispatch router 2-phase start

Gordon Sim
On 13/05/17 10:29, Adel Boutros wrote:

> Hello,
>
> We have noticed a race conditon with the dispatch router's behavior.
> If you have a producer and a consumer exchanging messages on a queue configured on a broker and accessible via the router. The consumer and producers are JMS clients configured with Failover options for retry.
> If the router goes down, the retry mechanism will kick in until it is up again.
>
> As we are configuring th addresses and connectors on the router dynamically, the producer and consumer might connect to the router before the waypointed address is created on it. In this case, a local address will be created on the router. The clients would exchange the messages directly from the router and the messages on the broker would never be consumed.
>
> I discussed this issue with Justin and Ulf during the RivieraDev conference and we were wondering if it was possible to implement a 2-phase startup on the router to avoid this issue.
> In that case, the router would start but not accept any connetion except for management. Once all dynamic configuration is done, we send a management message to allow the router to start accepting connections.
>
> Any thoughts on this?

You could do something similar already by defining a specific listener
for management only in the static config, and then only define the
'public' listener for real clients once any other configuration has been
done.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Dispatch router 2-phase start

Adel Boutros
Hello Gordon,


With what you are proposing, the order of the configuration becomes critical because if the public listener is configured before the connectors and autolinks, I would have the same issue with the producer/consumer. So my management process would have to send the management commands in a predefined order.


In the case of the 2-phase start, the order of configuration is irrelevant and the management is thus easier.


Regards,

Adel

________________________________
From: Gordon Sim <[hidden email]>
Sent: Monday, May 15, 2017 9:58:50 AM
To: [hidden email]
Subject: Re: Dispatch router 2-phase start

On 13/05/17 10:29, Adel Boutros wrote:

> Hello,
>
> We have noticed a race conditon with the dispatch router's behavior.
> If you have a producer and a consumer exchanging messages on a queue configured on a broker and accessible via the router. The consumer and producers are JMS clients configured with Failover options for retry.
> If the router goes down, the retry mechanism will kick in until it is up again.
>
> As we are configuring th addresses and connectors on the router dynamically, the producer and consumer might connect to the router before the waypointed address is created on it. In this case, a local address will be created on the router. The clients would exchange the messages directly from the router and the messages on the broker would never be consumed.
>
> I discussed this issue with Justin and Ulf during the RivieraDev conference and we were wondering if it was possible to implement a 2-phase startup on the router to avoid this issue.
> In that case, the router would start but not accept any connetion except for management. Once all dynamic configuration is done, we send a management message to allow the router to start accepting connections.
>
> Any thoughts on this?

You could do something similar already by defining a specific listener
for management only in the static config, and then only define the
'public' listener for real clients once any other configuration has been
done.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Dispatch router 2-phase start

Jiri Danek
On Mon, May 15, 2017 at 12:00 PM, Adel Boutros <[hidden email]> wrote:

> Hello Gordon,
>
>
> With what you are proposing, the order of the configuration becomes
> critical because if the public listener is configured before the connectors
> and autolinks, I would have the same issue with the producer/consumer. So
> my management process would have to send the management commands in a
> predefined order.
>
>
> In the case of the 2-phase start, the order of configuration is irrelevant
> and the management is thus easier.
>
>
> Regards,
>
> Adel
>

On Sat, May 13, 2017 at 11:29 AM, Adel Boutros <[hidden email]> wrote:

> Once all dynamic configuration is done, we send a management message to
> allow the router to start accepting connections.


So even with the 2-phase startup you'd have to keep the order of commands
in mind. You'd have to send this special startup command last.
--
Jiří Daněk
Messaging QA
Reply | Threaded
Open this post in threaded view
|

Re: Dispatch router 2-phase start

Adel Boutros
Indeed, but I would not need an extra port in this case.

Also, the extra management port could be used by mistake by any misconfigured consumer/producer.

________________________________
From: Jiri Danek <[hidden email]>
Sent: Monday, May 15, 2017 12:04:50 PM
To: users
Subject: Re: Dispatch router 2-phase start

On Mon, May 15, 2017 at 12:00 PM, Adel Boutros <[hidden email]> wrote:

> Hello Gordon,
>
>
> With what you are proposing, the order of the configuration becomes
> critical because if the public listener is configured before the connectors
> and autolinks, I would have the same issue with the producer/consumer. So
> my management process would have to send the management commands in a
> predefined order.
>
>
> In the case of the 2-phase start, the order of configuration is irrelevant
> and the management is thus easier.
>
>
> Regards,
>
> Adel
>

On Sat, May 13, 2017 at 11:29 AM, Adel Boutros <[hidden email]> wrote:

> Once all dynamic configuration is done, we send a management message to
> allow the router to start accepting connections.


So even with the 2-phase startup you'd have to keep the order of commands
in mind. You'd have to send this special startup command last.
--
Jiří Daněk
Messaging QA
Reply | Threaded
Open this post in threaded view
|

Re: Dispatch router 2-phase start

Jiri Danek
On Mon, May 15, 2017 at 12:14 PM, Adel Boutros <[hidden email]> wrote:

> Indeed, but I would not need an extra port in this case.
>
> Also, the extra management port could be used by mistake by any
> misconfigured consumer/producer.
>

You could setup a policy to allow only management on this extra port...


>
> ________________________________
> From: Jiri Danek <[hidden email]>
> Sent: Monday, May 15, 2017 12:04:50 PM
> To: users
> Subject: Re: Dispatch router 2-phase start
>
> On Mon, May 15, 2017 at 12:00 PM, Adel Boutros <[hidden email]>
> wrote:
>
> > Hello Gordon,
> >
> >
> > With what you are proposing, the order of the configuration becomes
> > critical because if the public listener is configured before the
> connectors
> > and autolinks, I would have the same issue with the producer/consumer. So
> > my management process would have to send the management commands in a
> > predefined order.
> >
> >
> > In the case of the 2-phase start, the order of configuration is
> irrelevant
> > and the management is thus easier.
> >
> >
> > Regards,
> >
> > Adel
> >
>
> On Sat, May 13, 2017 at 11:29 AM, Adel Boutros <[hidden email]>
> wrote:
>
> > Once all dynamic configuration is done, we send a management message to
> > allow the router to start accepting connections.
>
>
> So even with the 2-phase startup you'd have to keep the order of commands
> in mind. You'd have to send this special startup command last.
> --
> Jiří Daněk
> Messaging QA
>



--
Jiří Daněk
Messaging QA
Reply | Threaded
Open this post in threaded view
|

Re: Dispatch router 2-phase start

Ganesh Murthy
Hi Adel,
   Can you please enter an enhancement JIRA for this so we can track this. This problem definitely needs to be addressed. I see this problem happening even in the case of static config files.
If you see here - https://github.com/apache/qpid-dispatch/blob/master/python/qpid_dispatch_internal/management/config.py#L167 - the listener is configured before the waypoint and address entities and there is a possibility that a producer and consumer might start directly exchanging messages if the listeners are setup are setup and the clients attach immediately thereafter and before the router could run thru the rest of the static configuration.

Thanks.

----- Original Message -----

> From: "Jiri Danek" <[hidden email]>
> To: "users" <[hidden email]>
> Sent: Monday, May 15, 2017 6:21:01 AM
> Subject: Re: Dispatch router 2-phase start
>
> On Mon, May 15, 2017 at 12:14 PM, Adel Boutros <[hidden email]> wrote:
>
> > Indeed, but I would not need an extra port in this case.
> >
> > Also, the extra management port could be used by mistake by any
> > misconfigured consumer/producer.
> >
>
> You could setup a policy to allow only management on this extra port...
>
>
> >
> > ________________________________
> > From: Jiri Danek <[hidden email]>
> > Sent: Monday, May 15, 2017 12:04:50 PM
> > To: users
> > Subject: Re: Dispatch router 2-phase start
> >
> > On Mon, May 15, 2017 at 12:00 PM, Adel Boutros <[hidden email]>
> > wrote:
> >
> > > Hello Gordon,
> > >
> > >
> > > With what you are proposing, the order of the configuration becomes
> > > critical because if the public listener is configured before the
> > connectors
> > > and autolinks, I would have the same issue with the producer/consumer. So
> > > my management process would have to send the management commands in a
> > > predefined order.
> > >
> > >
> > > In the case of the 2-phase start, the order of configuration is
> > irrelevant
> > > and the management is thus easier.
> > >
> > >
> > > Regards,
> > >
> > > Adel
> > >
> >
> > On Sat, May 13, 2017 at 11:29 AM, Adel Boutros <[hidden email]>
> > wrote:
> >
> > > Once all dynamic configuration is done, we send a management message to
> > > allow the router to start accepting connections.
> >
> >
> > So even with the 2-phase startup you'd have to keep the order of commands
> > in mind. You'd have to send this special startup command last.
> > --
> > Jiří Daněk
> > Messaging QA
> >
>
>
>
> --
> Jiří Daněk
> Messaging QA
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Dispatch router 2-phase start

Adel Boutros
Hello Ganesh,


Done: https://issues.apache.org/jira/browse/DISPATCH-773


Indeed, I didn't see it could happen even to static config files.


Regards,

Adel

________________________________
From: Ganesh Murthy <[hidden email]>
Sent: Monday, May 15, 2017 4:09:17 PM
To: [hidden email]
Subject: Re: Dispatch router 2-phase start

Hi Adel,
   Can you please enter an enhancement JIRA for this so we can track this. This problem definitely needs to be addressed. I see this problem happening even in the case of static config files.
If you see here - https://github.com/apache/qpid-dispatch/blob/master/python/qpid_dispatch_internal/management/config.py#L167 - the listener is configured before the waypoint and address entities and there is a possibility that a producer and consumer might start directly exchanging messages if the listeners are setup are setup and the clients attach immediately thereafter and before the router could run thru the rest of the static configuration.

Thanks.

----- Original Message -----

> From: "Jiri Danek" <[hidden email]>
> To: "users" <[hidden email]>
> Sent: Monday, May 15, 2017 6:21:01 AM
> Subject: Re: Dispatch router 2-phase start
>
> On Mon, May 15, 2017 at 12:14 PM, Adel Boutros <[hidden email]> wrote:
>
> > Indeed, but I would not need an extra port in this case.
> >
> > Also, the extra management port could be used by mistake by any
> > misconfigured consumer/producer.
> >
>
> You could setup a policy to allow only management on this extra port...
>
>
> >
> > ________________________________
> > From: Jiri Danek <[hidden email]>
> > Sent: Monday, May 15, 2017 12:04:50 PM
> > To: users
> > Subject: Re: Dispatch router 2-phase start
> >
> > On Mon, May 15, 2017 at 12:00 PM, Adel Boutros <[hidden email]>
> > wrote:
> >
> > > Hello Gordon,
> > >
> > >
> > > With what you are proposing, the order of the configuration becomes
> > > critical because if the public listener is configured before the
> > connectors
> > > and autolinks, I would have the same issue with the producer/consumer. So
> > > my management process would have to send the management commands in a
> > > predefined order.
> > >
> > >
> > > In the case of the 2-phase start, the order of configuration is
> > irrelevant
> > > and the management is thus easier.
> > >
> > >
> > > Regards,
> > >
> > > Adel
> > >
> >
> > On Sat, May 13, 2017 at 11:29 AM, Adel Boutros <[hidden email]>
> > wrote:
> >
> > > Once all dynamic configuration is done, we send a management message to
> > > allow the router to start accepting connections.
> >
> >
> > So even with the 2-phase startup you'd have to keep the order of commands
> > in mind. You'd have to send this special startup command last.
> > --
> > Jiří Daněk
> > Messaging QA
> >
>
>
>
> --
> Jiří Daněk
> Messaging QA
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]