Federation and Queue Route Failover

[Fallon, Richard]

All,

I have been successfully using a federated network of Apache Qpid
brokers for some time now, we have had some problems along the way but
generally everything is working very well.

I have a requirement to introduce more resilience to parts of the
network. The (simplified) solution is this

"OPERATIONAL BEHAVIOUR"
1. SOURCE -> DESTINATION A : source sends messages to destination A
2. IF DESTINATION A FAILS THEN
3. SOURCE -> DESTINATION B : source sends messages to destination B

We have a piece of hardware between the brokers so we refer to the
destination with a generic value (e.g. DESTINATION) and this returns the
correct hostname of the destination (i.e. A OR B).

The above scenario works very well if the source is an amqp publisher,
e.g. a JMS Client using AMQP syntax.

However if the source and destinations are qpid brokers and connected
via a queue route this does not work.

Let me try and explain my rather bizarre results...

1. SOURCE BROKER -> DESTINATION BROKER (hostname = A) - connected
successfully
2. STOP DESTINATION BROKER (hostname = A)
3. Load Balancer now returns hostname=B for DESTINATION BROKER
4. SOURCE BROKER shows connection refused to DESTINATION BROKER


So maybe I can understand the above, the SOURCE has created a socket
connection to hostname A, and is not releasing it.

However if I stop and start the SOURCE broker, and re add the queue
route using SOURCE and DESTINATION I still get the connection refused
message, but hostname = B is running. It is like the SOURCE broker is
still trying to connect to the DESTINATION (hostname = A), even after
restart. That suggests to me that Qpid persists some connection
information throughout a restart.

Does this sound plausible? If so where would the connection info be
stored? FYI - I'm using version 0.8.

Thanks in advance

Richard

Richard Fallon
Architect
01928 594109
M:[phone]
E:[hidden email]
Atos.net

---

Atos and Atos Consulting are trading names used by the Atos group. The following trading entities are registered in England and Wales: Atos IT Services UK Limited (registered number 01245534), Atos Consulting Limited (registered number 04312380) and Atos IT Solutions and Services Limited (registered number 01203466) The registered office for each is at 4 Triton Square, Regents Place, London, NW1 3HG.The VAT No. for each is: GB232327983

This e-mail and the documents attached are confidential and intended solely for the addressee, and may contain confidential or privileged information. If you receive this e-mail in error, you are not authorised to copy, disclose, use or retain it. Please notify the sender immediately and delete this email from your systems. As emails may be intercepted, amended or lost, they are not secure. Atos therefore can accept no liability for any errors or their content. Although Atos endeavours to maintain a virus-free network, we do not warrant that this transmission is virus-free and can accept no liability for any damages resulting from any virus transmitted. The risks are deemed to be accepted by everyone who communicates with Atos by email.

[Gordon Sim]

On 04/23/2012 01:56 PM, Fallon, Richard wrote:

>
>
> All,
>
> I have been successfully using a federated network of Apache Qpid
> brokers for some time now, we have had some problems along the way but
> generally everything is working very well.
>
> I have a requirement to introduce more resilience to parts of the
> network. The (simplified) solution is this
>
> "OPERATIONAL BEHAVIOUR"
> 1. SOURCE -> DESTINATION A : source sends messages to destination A
> 2. IF DESTINATION A FAILS THEN
> 3. SOURCE -> DESTINATION B : source sends messages to destination B
>
> We have a piece of hardware between the brokers so we refer to the
> destination with a generic value (e.g. DESTINATION) and this returns the
> correct hostname of the destination (i.e. A OR B).
>
> The above scenario works very well if the source is an amqp publisher,
> e.g. a JMS Client using AMQP syntax.
>
> However if the source and destinations are qpid brokers and connected
> via a queue route this does not work.
>
> Let me try and explain my rather bizarre results...
>
> 1. SOURCE BROKER -> DESTINATION BROKER (hostname = A) - connected
> successfully
> 2. STOP DESTINATION BROKER (hostname = A)
> 3. Load Balancer now returns hostname=B for DESTINATION BROKER
> 4. SOURCE BROKER shows connection refused to DESTINATION BROKER

Are you using the generic DESTINATION value for the route as well?

> So maybe I can understand the above, the SOURCE has created a socket
> connection to hostname A, and is not releasing it.
>
> However if I stop and start the SOURCE broker, and re add the queue
> route using SOURCE and DESTINATION I still get the connection refused
> message, but hostname = B is running. It is like the SOURCE broker is
> still trying to connect to the DESTINATION (hostname = A), even after
> restart. That suggests to me that Qpid persists some connection
> information throughout a restart.

Yes, links and bridges can be durable (though I believe they are not by
default). Their details are then stored in the database along with
details of durable queues, exchanges etc.

You can run qpid-route route list to see if that gives more information.
I'm not entirely sure what all has changed since 0.8.

> Does this sound plausible? If so where would the connection info be
> stored? FYI - I'm using version 0.8.
>
> Thanks in advance
>
> Richard
>
>
>
> Picture (Metafile)
> *Richard Fallon*
> Architect
> 01928 594109
> M:[phone]
> E:[e-mail]_ <mailto:[hidden email]>
> Atos.net
> Picture (Metafile)
>
>
>
>
> ------------------------------------------------------------------------
> Atos and Atos Consulting are trading names used by the Atos group. The
> following trading entities are registered in England and Wales: Atos IT
> Services UK Limited (registered number 01245534), Atos Consulting
> Limited (registered number 04312380) and Atos IT Solutions and Services
> Limited (registered number 01203466) The registered office for each is
> at 4 Triton Square, Regents Place, London, NW1 3HG.The VAT No. for each
> is: GB232327983
>
> This e-mail and the documents attached are confidential and intended
> solely for the addressee, and may contain confidential or privileged
> information. If you receive this e-mail in error, you are not authorised
> to copy, disclose, use or retain it. Please notify the sender
> immediately and delete this email from your systems. As emails may be
> intercepted, amended or lost, they are not secure. Atos therefore can
> accept no liability for any errors or their content. Although Atos
> endeavours to maintain a virus-free network, we do not warrant that this
> transmission is virus-free and can accept no liability for any damages
> resulting from any virus transmitted. The risks are deemed to be
> accepted by everyone who communicates with Atos by email.
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[rfallon]

Thanks for the quick reply.

"Are you using the generic DESTINATION value for the route as well?"

Yes.

"Yes, links and bridges can be durable (though I believe they are not by
default). Their details are then stored in the database along with
details of durable queues, exchanges etc.

You can run qpid-route route list to see if that gives more information.
I'm not entirely sure what all has changed since 0.8."

I'm not actually specifying the route as durable.  I create a queue and a binding as you might expect and the command to create the queue route is...

qpid-config -s queue add DESTINATION SOURCE amq.match queue_name

[Gordon Sim]

On 04/23/2012 03:32 PM, rfallon wrote:
> I'm not actually specifying the route as durable.

I wouldn't expect the information to be persisted in that case... what
is the output for qpid-route link list SOURCE?

Maybe it would be worth turning on debug logging to see if that gives
any other clues (e.g. --log-enable notice+ --log-enable debug+:Link
--log-enable debug+:Bridge). If you do that after restarting the source,
what do you see?

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[fadams]

Richard's issue does seem weird.

I note that he's using source routes, given his comment.

qpid-config -s queue add DESTINATION SOURCE amq.match queue_name


I'm not at all clear how qpid/amqp behaves through a hardware load
balancer. I did wonder if there might be an issue whereby the federating
bridge talks to the balancer with a hostname/servicename that resolves
to an IP address and whether somehow the IP is cached in the broker so
when the connection fails it uses the IP rather than doing another DNS
lookup which ought to resolve to the failover address, however Richard's
other observation seems plain bizarre "However if I stop and start the
SOURCE broker, and re add the queue route using SOURCE and DESTINATION I
still get the connection refused message, but hostname = B is running"

Richard out of curiosity in this scenario can you ping hostname = B? can
you actually connect to broker B using an ordinary client (you mentioned
in your original post The above scenario works very well if the source
is an amqp publisher, e.g. a JMS Client using AMQP syntax.) so to be
clear does hostname = B only seem "invisible" to a queue route (I'm just
trying to narrow if it's a bridge issue or some weirdness with the load
balancer's DNS resolver).

Surely if source routes are being used stopping and starting the source
broker would remove the federation link (Richard mentions he's not using
durable anything).

I'm interested in this sort of thing too, has anyone else had experience
running federated links through a hardware load balancer (we were
contemplating Cisco's Global Site Selector) but I've no experience of
this sort of technology so I'm wondering whether it's actually a viable
approach.

As a related aside as far as I'm aware qpid has a failover mechanism
that uses amqp heartbeats IIRC, from the perspective of amqp client
software I believe that this works by specifying a broker list in the
connection URL, however it's not clear how (or even whether) it's
possible to set up failover on federated links. With source routes I'd
have *assumed* that the bridge code would behave rather like any old
producer client so if a destination broker fails it (logically at any
rate) ought to be able to fail over to an alternate destination broker,
but qpid-route certainly doesn't suggest that broker lists can be used.

So bottom line is that I'm as interested as Richard in strategies for
fail over in a federated architecture (I don't want to use clustering).


On 23/04/12 16:04, Gordon Sim wrote:

> On 04/23/2012 03:32 PM, rfallon wrote:
>> I'm not actually specifying the route as durable.
>
> I wouldn't expect the information to be persisted in that case... what
> is the output for qpid-route link list SOURCE?
>
> Maybe it would be worth turning on debug logging to see if that gives
> any other clues (e.g. --log-enable notice+ --log-enable debug+:Link
> --log-enable debug+:Bridge). If you do that after restarting the
> source, what do you see?
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

[rfallon]

Is it possible to use the failover syntax when creating queue routes via qpid-config?   If so, how?