Debugging delays/lost messages/poor throughput

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Debugging delays/lost messages/poor throughput

Toralf Lund
I'm trying to get back to an old issue with one of our software
installations based the C++ QPid Messaging API with AMPQ 0-10; It's been
discussed here a couple of times before, but unfortunately not a lot has
come out of it.

To make a long story short, I see the following issues in my software:

 1. qpid::messaging::Sender::send() on a relatively small message
    frequently takes several seconds (often 3 or 4, sometimes perhaps
    10) - even though there should be a reasonable sender capacity.
 2. Based on timestamps etc., it may look like there is sometimes a
    similar delay between the broker and receiver - from the the time
    given by "x-amqp-0-10.timestamp to the actual receive time in an
    application that *should* be ready for new data.
 3. Sometimes heartbeats are lost or take too long, so that the
    application reconnects. Heartbeat interval is currently.
 4. On reconnect, the broker often still has an active connection, which
    results in a session already in use message.

The broker and "client" application are connected to the same network
switch, and I don't think the load should be *that* high, so I'd expect
messages to be delivered in a few milliseconds rather than seconds. I
also don't get the long delays all the time - some messages actually
seem to arrive quite fast.

This all happens at a remove work site. I don't see the same problem on
a local "test installation".

My question this time is, how can I check for connectivity issues,
messaging throughput etc.? I'm thinking this should perhaps be done
outside ouf or software, to test at a more fundamental level, so as to
speak. Any suggestions?

- Toralf