Qpid JMS: Message filter validation

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Qpid JMS: Message filter validation

Michael Walser
Hello,

I'm attempting to use Qpid JMS to connect to an AMQP 1.0 server (namely
Azure Event Hubs). To be able to start consuming messages at a certain
offset I have to set a message filter.

The filters reads like this:
    amqp.annotation.x-opt-offset > '100'

When setting the filter using
    session.createConsumer(
      destination,
      "amqp.annotation.x-opt-offset > '100'"
    )
an InvalidSelectorException gets thrown.

This is due to the Qpid library checking [1] the filter against a
grammar defined in SelectorParserImpl.jj [2].

It appears like there is no way around this validation.
I spent the better part of this afternoon trying to find a syntax that
is accepted by both Qpid and the Azure Event Hub.

Other people facing the same problem are resorting to modifying the Qpid
source code [3].


I think it would be enormously helpful if Qpid JMS would support setting
a custom JMS_SELECTOR_SYMBOL and provide an option to disable the
selector validation.

If such a change would have the chance of getting accepted I would
volunteer to implement this.

Is this in the realm of possibility?

~~ Michael

[1]
https://github.com/apache/qpid-jms/blob/0.55.0/qpid-jms-client/src/main/java/org/apache/qpid/jms/JmsSession.java#L476
[2]
https://github.com/apache/qpid-jms/blob/0.55.0/qpid-jms-client/src/main/javacc/SelectorParserImpl.jj
[3] https://stackoverflow.com/q/56194397/2377042

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Qpid JMS: Message filter validation

Robbie Gemmell
Administrator
On Tue, 8 Dec 2020 at 22:21, Michael Walser <[hidden email]> wrote:

>
> Hello,
>
> I'm attempting to use Qpid JMS to connect to an AMQP 1.0 server (namely
> Azure Event Hubs). To be able to start consuming messages at a certain
> offset I have to set a message filter.
>
> The filters reads like this:
>     amqp.annotation.x-opt-offset > '100'
>
> When setting the filter using
>     session.createConsumer(
>       destination,
>       "amqp.annotation.x-opt-offset > '100'"
>     )
> an InvalidSelectorException gets thrown.
>
> This is due to the Qpid library checking [1] the filter against a
> grammar defined in SelectorParserImpl.jj [2].
>
> It appears like there is no way around this validation.

Not at present. This is verifying the JMS selector syntax, giving
behaviours needed/verified by JMS, with e.g. the above selector
example being invalid/unexpected in a couple ways in JMS terms and
thus causing an exception as seen.

That said, Qpid JMS and at least the Qpid brokers have long supported
(along with ActiveMQ Artemis more recently) an escape of "quoting" the
contents of the 'property name' variable for bypassing some of the
restrictions and allowing some of these additional behaviours, e.g. a
quoted variable in a selector string:
"\"my.quoted-var\"='some-value'"

That likely won't help if the server doesn't support it however, I
dont know whether it does in this case.

> I spent the better part of this afternoon trying to find a syntax that
> is accepted by both Qpid and the Azure Event Hub.
>
> Other people facing the same problem are resorting to modifying the Qpid
> source code [3].
>
>
> I think it would be enormously helpful if Qpid JMS would support setting
> a custom JMS_SELECTOR_SYMBOL and provide an option to disable the
> selector validation.

I think adding the option to disable the selector validation is
reasonable, given we do provide option for disabling related property
name validations. Adding one for changing the protocol filter map key,
not so much though.

It simply shouldn't be necessary to change the map key since using a
particular map key is not part of the filters semantic, the server
shouldnt especially care what is used, and so I dont think it should
be encouraged or warrants adding+maintaining an option for this.
Especially with the filter behaviour originating here and having been
around as long as it has now been. The key in use has been so since
the protocol filter in question was originally created and used here
at Qpid in an earlier [prototype] AMQP 1.0 JMS client over 8 years
ago. I also believe at least Azure ServiceBus should already work with
the filter map key the client currently uses, given Microsoft actually
now uses Qpid JMS in their JMS offering.

The client code changes mentioned use the filters symbolic descriptor
as the map key, which isn't actually the intent of the value but
rather that it can be used as part of the filter construction, which
the change does then also go on to do, instead of the equivalent
numeric descriptor code that the existing client impl is already
using. I would be interested to know if either of those was actually
necessary (as it certainly shouldn't be), i.e it was tried otherwise
or just immediately done like shown because it was mentioned on the
referenced page.

Disabling the client-side validation to allow an 'invalid' (for JMS)
selector string through should be the only meaningful client change
'needed' (assuming the mentioned quoting-escape isn't already viable
at the server, in which case no change would really be required).
Anything else really represents an issue with the servers behaviour
(i.e requiring a particular filter key, or not supporting using the
filters numeric descriptor) that can/should be rectified there.


>
> If such a change would have the chance of getting accepted I would
> volunteer to implement this.
>
> Is this in the realm of possibility?
>
> ~~ Michael
>
> [1]
> https://github.com/apache/qpid-jms/blob/0.55.0/qpid-jms-client/src/main/java/org/apache/qpid/jms/JmsSession.java#L476
> [2]
> https://github.com/apache/qpid-jms/blob/0.55.0/qpid-jms-client/src/main/javacc/SelectorParserImpl.jj
> [3] https://stackoverflow.com/q/56194397/2377042
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Qpid JMS: Message filter validation

Robbie Gemmell
Administrator
On Wed, 9 Dec 2020 at 15:39, Robbie Gemmell <[hidden email]> wrote:

>
> On Tue, 8 Dec 2020 at 22:21, Michael Walser <[hidden email]> wrote:
> >
> > Hello,
> >
> > I'm attempting to use Qpid JMS to connect to an AMQP 1.0 server (namely
> > Azure Event Hubs). To be able to start consuming messages at a certain
> > offset I have to set a message filter.
> >
> > The filters reads like this:
> >     amqp.annotation.x-opt-offset > '100'
> >
> > When setting the filter using
> >     session.createConsumer(
> >       destination,
> >       "amqp.annotation.x-opt-offset > '100'"
> >     )
> > an InvalidSelectorException gets thrown.
> >
> > This is due to the Qpid library checking [1] the filter against a
> > grammar defined in SelectorParserImpl.jj [2].
> >
> > It appears like there is no way around this validation.
>
> Not at present. This is verifying the JMS selector syntax, giving
> behaviours needed/verified by JMS, with e.g. the above selector
> example being invalid/unexpected in a couple ways in JMS terms and
> thus causing an exception as seen.
>
> That said, Qpid JMS and at least the Qpid brokers have long supported
> (along with ActiveMQ Artemis more recently) an escape of "quoting" the
> contents of the 'property name' variable for bypassing some of the
> restrictions and allowing some of these additional behaviours, e.g. a
> quoted variable in a selector string:
> "\"my.quoted-var\"='some-value'"
>
> That likely won't help if the server doesn't support it however, I
> dont know whether it does in this case.
>
> > I spent the better part of this afternoon trying to find a syntax that
> > is accepted by both Qpid and the Azure Event Hub.
> >
> > Other people facing the same problem are resorting to modifying the Qpid
> > source code [3].
> >
> >
> > I think it would be enormously helpful if Qpid JMS would support setting
> > a custom JMS_SELECTOR_SYMBOL and provide an option to disable the
> > selector validation.
>
> I think adding the option to disable the selector validation is
> reasonable, given we do provide option for disabling related property
> name validations. Adding one for changing the protocol filter map key,
> not so much though.
>
> It simply shouldn't be necessary to change the map key since using a
> particular map key is not part of the filters semantic, the server
> shouldnt especially care what is used, and so I dont think it should
> be encouraged or warrants adding+maintaining an option for this.
> Especially with the filter behaviour originating here and having been
> around as long as it has now been. The key in use has been so since
> the protocol filter in question was originally created and used here
> at Qpid in an earlier [prototype] AMQP 1.0 JMS client over 8 years
> ago. I also believe at least Azure ServiceBus should already work with
> the filter map key the client currently uses, given Microsoft actually
> now uses Qpid JMS in their JMS offering.
>
> The client code changes mentioned use the filters symbolic descriptor
> as the map key, which isn't actually the intent of the value but
> rather that it can be used as part of the filter construction, which
> the change does then also go on to do, instead of the equivalent
> numeric descriptor code that the existing client impl is already
> using. I would be interested to know if either of those was actually
> necessary (as it certainly shouldn't be), i.e it was tried otherwise
> or just immediately done like shown because it was mentioned on the
> referenced page.
>
> Disabling the client-side validation to allow an 'invalid' (for JMS)
> selector string through should be the only meaningful client change
> 'needed' (assuming the mentioned quoting-escape isn't already viable
> at the server, in which case no change would really be required).
> Anything else really represents an issue with the servers behaviour
> (i.e requiring a particular filter key, or not supporting using the
> filters numeric descriptor) that can/should be rectified there.
>
>

Here is a branch with a change adding support for a
"jms.validateSelector" URI option that could be used (by setting the
value to false) to disable the local JMS syntax validation, resulting
in it simply passing the selector string through to be sent in the
filter. You can see how you get on with that:
https://github.com/gemmellr/qpid-jms/tree/selector-validation-toggle

> >
> > If such a change would have the chance of getting accepted I would
> > volunteer to implement this.
> >
> > Is this in the realm of possibility?
> >
> > ~~ Michael
> >
> > [1]
> > https://github.com/apache/qpid-jms/blob/0.55.0/qpid-jms-client/src/main/java/org/apache/qpid/jms/JmsSession.java#L476
> > [2]
> > https://github.com/apache/qpid-jms/blob/0.55.0/qpid-jms-client/src/main/javacc/SelectorParserImpl.jj
> > [3] https://stackoverflow.com/q/56194397/2377042
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Qpid JMS: Message filter validation

Tom Jordahl
> Date: Wednesday, December 9, 2020 at 10:53 AM
> > It appears like there is no way around this validation.
>
> Not at present.

> Date: Wednesday, December 9, 2020 at 10:39 AM
> Here is a branch with a change adding support for a "jms.validateSelector" URI option..

Wow Robbie – Your enhancement request in 30 minutes or less or your software is free!!
:-) :-) :-)

But seriously, thanks to the Qpid community for the tremendous help and support for their users!

--
Tom Jordahl
Happy Qpid Broker-J & JMS client user

From: Robbie Gemmell <[hidden email]>
Date: Wednesday, December 9, 2020 at 10:53 AM
To: users <[hidden email]>
Subject: Re: Qpid JMS: Message filter validation
On Wed, 9 Dec 2020 at 15:39, Robbie Gemmell <[hidden email]> wrote:

>
> On Tue, 8 Dec 2020 at 22:21, Michael Walser <[hidden email]> wrote:
> >
> > Hello,
> >
> > I'm attempting to use Qpid JMS to connect to an AMQP 1.0 server (namely
> > Azure Event Hubs). To be able to start consuming messages at a certain
> > offset I have to set a message filter.
> >
> > The filters reads like this:
> >     amqp.annotation.x-opt-offset > '100'
> >
> > When setting the filter using
> >     session.createConsumer(
> >       destination,
> >       "amqp.annotation.x-opt-offset > '100'"
> >     )
> > an InvalidSelectorException gets thrown.
> >
> > This is due to the Qpid library checking [1] the filter against a
> > grammar defined in SelectorParserImpl.jj [2].
> >
> > It appears like there is no way around this validation.
>
> Not at present. This is verifying the JMS selector syntax, giving
> behaviours needed/verified by JMS, with e.g. the above selector
> example being invalid/unexpected in a couple ways in JMS terms and
> thus causing an exception as seen.
>
> That said, Qpid JMS and at least the Qpid brokers have long supported
> (along with ActiveMQ Artemis more recently) an escape of "quoting" the
> contents of the 'property name' variable for bypassing some of the
> restrictions and allowing some of these additional behaviours, e.g. a
> quoted variable in a selector string:
> "\"my.quoted-var\"='some-value'"
>
> That likely won't help if the server doesn't support it however, I
> dont know whether it does in this case.
>
> > I spent the better part of this afternoon trying to find a syntax that
> > is accepted by both Qpid and the Azure Event Hub.
> >
> > Other people facing the same problem are resorting to modifying the Qpid
> > source code [3].
> >
> >
> > I think it would be enormously helpful if Qpid JMS would support setting
> > a custom JMS_SELECTOR_SYMBOL and provide an option to disable the
> > selector validation.
>
> I think adding the option to disable the selector validation is
> reasonable, given we do provide option for disabling related property
> name validations. Adding one for changing the protocol filter map key,
> not so much though.
>
> It simply shouldn't be necessary to change the map key since using a
> particular map key is not part of the filters semantic, the server
> shouldnt especially care what is used, and so I dont think it should
> be encouraged or warrants adding+maintaining an option for this.
> Especially with the filter behaviour originating here and having been
> around as long as it has now been. The key in use has been so since
> the protocol filter in question was originally created and used here
> at Qpid in an earlier [prototype] AMQP 1.0 JMS client over 8 years
> ago. I also believe at least Azure ServiceBus should already work with
> the filter map key the client currently uses, given Microsoft actually
> now uses Qpid JMS in their JMS offering.
>
> The client code changes mentioned use the filters symbolic descriptor
> as the map key, which isn't actually the intent of the value but
> rather that it can be used as part of the filter construction, which
> the change does then also go on to do, instead of the equivalent
> numeric descriptor code that the existing client impl is already
> using. I would be interested to know if either of those was actually
> necessary (as it certainly shouldn't be), i.e it was tried otherwise
> or just immediately done like shown because it was mentioned on the
> referenced page.
>
> Disabling the client-side validation to allow an 'invalid' (for JMS)
> selector string through should be the only meaningful client change
> 'needed' (assuming the mentioned quoting-escape isn't already viable
> at the server, in which case no change would really be required).
> Anything else really represents an issue with the servers behaviour
> (i.e requiring a particular filter key, or not supporting using the
> filters numeric descriptor) that can/should be rectified there.
>
>

Here is a branch with a change adding support for a
"jms.validateSelector" URI option that could be used (by setting the
value to false) to disable the local JMS syntax validation, resulting
in it simply passing the selector string through to be sent in the
filter. You can see how you get on with that:
https://github.com/gemmellr/qpid-jms/tree/selector-validation-toggle

> >
> > If such a change would have the chance of getting accepted I would
> > volunteer to implement this.
> >
> > Is this in the realm of possibility?
> >
> > ~~ Michael
> >
> > [1]
> > https://github.com/apache/qpid-jms/blob/0.55.0/qpid-jms-client/src/main/java/org/apache/qpid/jms/JmsSession.java#L476
> > [2]
> > https://github.com/apache/qpid-jms/blob/0.55.0/qpid-jms-client/src/main/javacc/SelectorParserImpl.jj
> > [3] https://stackoverflow.com/q/56194397/2377042
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Qpid JMS: Message filter validation

Michael Walser
In reply to this post by Robbie Gemmell
Thank your for your exhaustive answer.


Am 09.12.20 um 16:53 schrieb Robbie Gemmell:
>> That said, Qpid JMS and at least the Qpid brokers have long supported
>> (along with ActiveMQ Artemis more recently) an escape of "quoting" the
>> contents of the 'property name' variable for bypassing some of the
>> restrictions and allowing some of these additional behaviours, e.g. a
>> quoted variable in a selector string:
>> "\"my.quoted-var\"='some-value'"

Thanks for that suggestion. Unfortunately the server appears to only
accept a very specific syntax. The quoted variable name is unfortunately
not allowed by the server. Also the validation disallows the ">"
operator being used when the right hand side of the expression is a
string, which the server unfortunately appears to require.


>> I think adding the option to disable the selector validation is
>> reasonable, given we do provide option for disabling related property
>> name validations.

That's great to hear!


>> It simply shouldn't be necessary to change the map key since using a
>> particular map key is not part of the filters semantic, the server
>> shouldnt especially care what is used, and so I dont think it should
>> be encouraged or warrants adding+maintaining an option for this.

Yes, you are right. This change is not needed. Sorry, my understanding
of the AMQP 1.0 protocol is still developing.


> Here is a branch with a change adding support for a
> "jms.validateSelector" URI option that could be used (by setting the
> value to false) to disable the local JMS syntax validation, resulting
> in it simply passing the selector string through to be sent in the
> filter. You can see how you get on with that:
> https://github.com/gemmellr/qpid-jms/tree/selector-validation-toggle

I compiled your branch locally and was able to get the filter to work
using the new option. Again, thank you for taking your time and
implementing this so swiftly.


There is already an email regarding your great support and I want to say
that I share the praise expressed there. You not only taking your time
suggesting a possible workaround, clearing up my misunderstanding of the
filter symbols but then also implementing the change required to solve
the problem at hand is really great.
Thank you for that!

~~ Michael

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Qpid JMS: Message filter validation

Robbie Gemmell
Administrator
On Wed, 9 Dec 2020 at 22:19, Michael Walser <[hidden email]> wrote:

>
> Thank your for your exhaustive answer.
>
>
> Am 09.12.20 um 16:53 schrieb Robbie Gemmell:
> >> That said, Qpid JMS and at least the Qpid brokers have long supported
> >> (along with ActiveMQ Artemis more recently) an escape of "quoting" the
> >> contents of the 'property name' variable for bypassing some of the
> >> restrictions and allowing some of these additional behaviours, e.g. a
> >> quoted variable in a selector string:
> >> "\"my.quoted-var\"='some-value'"
>
> Thanks for that suggestion. Unfortunately the server appears to only
> accept a very specific syntax. The quoted variable name is unfortunately
> not allowed by the server. Also the validation disallows the ">"
> operator being used when the right hand side of the expression is a
> string, which the server unfortunately appears to require.
>

Yep, so I actually overlooked that string element in your example
hehe, or I needn't have even mentioned the quoting workaround.

The > operator simply isn't applicable to strings in JMS selectors,
and the quoting workaround escape the client supports is really about
naming, and so unsurprisingly has no effect on that type-validation
aspect of the existing checks.

>
> >> I think adding the option to disable the selector validation is
> >> reasonable, given we do provide option for disabling related property
> >> name validations.
>
> That's great to hear!
>
>
> >> It simply shouldn't be necessary to change the map key since using a
> >> particular map key is not part of the filters semantic, the server
> >> shouldnt especially care what is used, and so I dont think it should
> >> be encouraged or warrants adding+maintaining an option for this.
>
> Yes, you are right. This change is not needed. Sorry, my understanding
> of the AMQP 1.0 protocol is still developing.
>
>
> > Here is a branch with a change adding support for a
> > "jms.validateSelector" URI option that could be used (by setting the
> > value to false) to disable the local JMS syntax validation, resulting
> > in it simply passing the selector string through to be sent in the
> > filter. You can see how you get on with that:
> > https://github.com/gemmellr/qpid-jms/tree/selector-validation-toggle
>
> I compiled your branch locally and was able to get the filter to work
> using the new option. Again, thank you for taking your time and
> implementing this so swiftly.
>

Good to hear. I finished the change off and committed it via
https://issues.apache.org/jira/browse/QPIDJMS-522.

A new 0.56.0-SNAPSHOT can be tested either via a local build from
https://github.com/apache/qpid-jms or there will now be a build in the
snapshots repo
(https://repository.apache.org/content/repositories/snapshots/) at
this point also. It's really about the same as yesterdays branch
though, but with an updated Netty also.

I'll be looking to do a release next week.

>
> There is already an email regarding your great support and I want to say
> that I share the praise expressed there. You not only taking your time
> suggesting a possible workaround, clearing up my misunderstanding of the
> filter symbols but then also implementing the change required to solve
> the problem at hand is really great.
> Thank you for that!
>
> ~~ Michael
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]