Multiple connections with ssl client auth does not work as expected

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Multiple connections with ssl client auth does not work as expected

Chris Richardson
Hi,

I've encountered an issue with the C++ broker (1.36) where it appears
that multiple connections within the same process using SSL client
auth do not use the "ssl-cert-name" property provided when the
connection instance is created. Rather, it appears the second
connection will use auth details of the first connection.

Is this expected behaviour?

Setup to show this happening is rather involved so I've created a jira
issue with some attached material to demonstrate the problem:
https://issues.apache.org/jira/browse/QPID-7894.

Regards

--
Chris Richardson, System Architect
[hidden email]

FourC AS, Vestre Rosten 81, Trekanten, NO-7075 Tiller, Norway
www.fourc.eu

Follow us on LinkedIn, Facebook, Google+ and Twitter!

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Multiple connections with ssl client auth does not work as expected

Jakub Scholz-2
This is definitely not new in 1.36. I think this is present "since the
beginning". :-(

If I remember correctly the problem was that the NSS library which handles
the SSL certificates is initialised only once even when you create multiple
connections. I was quite sure this was already discussed in some issue or
on the mailing list but I have problems finding it now - I will keep
looking.

Regards
Jakub

On Mon, Aug 21, 2017 at 10:46 PM, Chris Richardson <[hidden email]> wrote:

> Hi,
>
> I've encountered an issue with the C++ broker (1.36) where it appears
> that multiple connections within the same process using SSL client
> auth do not use the "ssl-cert-name" property provided when the
> connection instance is created. Rather, it appears the second
> connection will use auth details of the first connection.
>
> Is this expected behaviour?
>
> Setup to show this happening is rather involved so I've created a jira
> issue with some attached material to demonstrate the problem:
> https://issues.apache.org/jira/browse/QPID-7894.
>
> Regards
>
> --
> Chris Richardson, System Architect
> [hidden email]
>
> FourC AS, Vestre Rosten 81, Trekanten, NO-7075 Tiller, Norway
> www.fourc.eu
>
> Follow us on LinkedIn, Facebook, Google+ and Twitter!
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Multiple connections with ssl client auth does not work as expected

Gordon Sim
In reply to this post by Chris Richardson
On 21/08/17 21:46, Chris Richardson wrote:

> Hi,
>
> I've encountered an issue with the C++ broker (1.36) where it appears
> that multiple connections within the same process using SSL client
> auth do not use the "ssl-cert-name" property provided when the
> connection instance is created. Rather, it appears the second
> connection will use auth details of the first connection.
>
> Is this expected behaviour?
>
> Setup to show this happening is rather involved so I've created a jira
> issue with some attached material to demonstrate the problem:
> https://issues.apache.org/jira/browse/QPID-7894.

I don't think I can reproduce your issue. I downloaded your reproducer
and with a couple of minor modifications[1] ran it and it showed no deny
logs.

The broker log has the following:

> 2017-08-22 21:11:23 [Security] debug ACL: Lookup for id:user1@QPID action:access objectType:queue name:user1 with params { durable=false autodelete=false exclusive=false alternate= policytype= maxqueuesize=0 max
> queuecount=0 }
> 2017-08-22 21:11:23 [Security] debug ACL: checking rule [rule 2 ruleMode = allow props{ name=${user} }]
> 2017-08-22 21:11:23 [Security] debug ACL: lookup name 'user1' matched with rule name '${user}'
> 2017-08-22 21:11:23 [Security] debug ACL: Successful match, the decision is:allow
> 2017-08-22 21:11:23 [Security] debug ACL: Lookup for id:user1@QPID action:consume objectType:queue name:user1 with params { }
> 2017-08-22 21:11:23 [Security] debug ACL: checking rule [rule 3 ruleMode = allow props{ name=${user} }]
> 2017-08-22 21:11:23 [Security] debug ACL: lookup name 'user1' matched with rule name '${user}'
> 2017-08-22 21:11:23 [Security] debug ACL: Successful match, the decision is:allow
> 2017-08-22 21:11:23 [Security] debug ACL: Lookup for id:user2@QPID action:access objectType:queue name:user2 with params { durable=false autodelete=false exclusive=false alternate= policytype= maxqueuesize=0 max
> queuecount=0 }
> 2017-08-22 21:11:23 [Security] debug ACL: checking rule [rule 2 ruleMode = allow props{ name=${user} }]
> 2017-08-22 21:11:23 [Security] debug ACL: lookup name 'user2' matched with rule name '${user}'
> 2017-08-22 21:11:23 [Security] debug ACL: Successful match, the decision is:allow
> 2017-08-22 21:11:23 [Security] debug ACL: Lookup for id:user2@QPID action:consume objectType:queue name:user2 with params { }
> 2017-08-22 21:11:23 [Security] debug ACL: checking rule [rule 3 ruleMode = allow props{ name=${user} }]
> 2017-08-22 21:11:23 [Security] debug ACL: lookup name 'user2' matched with rule name '${user}'
> 2017-08-22 21:11:23 [Security] debug ACL: Successful match, the decision is:allow
> 2017-08-22 21:11:23 [Security] trace ACL ConnectionCounter closed: qpid.127.0.0.1:5671-127.0.0.1:56482, userId:user1@QPID
> 2017-08-22 21:11:23 [Security] trace ACL ConnectionCounter closed: qpid.127.0.0.1:5671-127.0.0.1:56484, userId:user2@QPID

Which looks like the connections were correctly identified.

I'm using latest master for qpid-cpp and nss-devel-3.31.0-1.1.fc25.x86_64.

Am I doing something wrong, or do our setups differ?

[1] The changes I made were (a) to not use the qpid-cpp based
qpid-config to create the two queues and (b) I built the test client
without your cmake file, both just for convenience on my part.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Multiple connections with ssl client auth does not work as expected

Jakub Scholz-2
In reply to this post by Jakub Scholz-2
I did some more digging through my old work emails. We had a customer with
this kind of problem in June 2015. At that time I was able to reproduce
this behaviour on Linux (I believe they were Windows users, so it should
have affected also Windows). The problem was exactly the same as described
by Chris.

I was able to find a very primitive workaround at that time ... the
"caching" was done based on hostname. So when two different hostnames were
used the caching did not happen and the connections worked fine. So the
workaround was to 1) create for each connection an artificial /etc/hosts
record to which will route to the same broker, 2) Disable SSL hostname
verification and 3) open each connection through a different hostname. Hey
it was ugly and a bit insecure ... but it worked as a quick fix :-).

Later the customer got back to us with following info:
"After much investigation and experimentation, we have identified a
potential issue with the Qpid library implementation of SSL connections,
where the SSL session id might occasionally be "cached" or "reused". We
believe this is the cause of the ACL exceptions, since when the session Id
is cached and reused in an application that connects with multiple SSL
client certificates, the previously used client certificate information of
that previous SSL session might be reused as well for the next SSL
connection attempt. We have implemented a fix in our Qpid library and
application (by essentially disabling the caching of SSL session id for
each SSL connection attempt), and deployed them in our UAT environment
(user acceptance test). It seems to be working, as our application is able
to run with multiple SSL certificates now."

I asked them to provide the patch to the Qpid community. But I'm not sure
they ever did - at least I cannot find anything. I'm not sure why I didn't
raised a QPID JIRA for it my self. :-(

I probably haven't tried it since 2015. So if it works for Gordon it might
be already fixed in master.

Regards
Jakub

On Mon, Aug 21, 2017 at 11:18 PM, Jakub Scholz <[hidden email]> wrote:

> This is definitely not new in 1.36. I think this is present "since the
> beginning". :-(
>
> If I remember correctly the problem was that the NSS library which handles
> the SSL certificates is initialised only once even when you create multiple
> connections. I was quite sure this was already discussed in some issue or
> on the mailing list but I have problems finding it now - I will keep
> looking.
>
> Regards
> Jakub
>
> On Mon, Aug 21, 2017 at 10:46 PM, Chris Richardson <[hidden email]> wrote:
>
>> Hi,
>>
>> I've encountered an issue with the C++ broker (1.36) where it appears
>> that multiple connections within the same process using SSL client
>> auth do not use the "ssl-cert-name" property provided when the
>> connection instance is created. Rather, it appears the second
>> connection will use auth details of the first connection.
>>
>> Is this expected behaviour?
>>
>> Setup to show this happening is rather involved so I've created a jira
>> issue with some attached material to demonstrate the problem:
>> https://issues.apache.org/jira/browse/QPID-7894.
>>
>> Regards
>>
>> --
>> Chris Richardson, System Architect
>> [hidden email]
>>
>> FourC AS, Vestre Rosten 81, Trekanten, NO-7075 Tiller, Norway
>> www.fourc.eu
>>
>> Follow us on LinkedIn, Facebook, Google+ and Twitter!
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Multiple connections with ssl client auth does not work as expected

Chris Richardson
Thanks to both of you Gordon and Jakub for your feedback.

Following Gordon's comment showing a different NSS version from mine
I've done some testing and it seems it is indeed down to the version
of NSS. In particular it looks like NSS 3.28 does not work and NSS
3.31 does work.

Here's what I did in a bit more detail:
* Started with a fresh Ubuntu 16.04.2 server image
* Installed apt packages 'build-essential cmake-curses-gui python-dev
gyp ninja-build zlib1g-dev ruby libboost-program-options-dev
libboost-system-dev uuid-dev libsasl2-dev libssl-dev pkg-config swig'
* Installed (from source) NSS 3.31
* Installed (from source) qpid-proton 0.16 and qpid-python 1.36
* Installed qpid-cpp master branch (test passed)
* Installed qpid-cpp 1.36.x branch (test passed)
* Downgraded NSS to 3.28 (apt packages 'libnss3-dev libnss3-tools')
* Re-installed the same qpid components (1.36.x branch) - TEST FAILED
* Upgraded back to NSS 3.1, re-installed qpid-cpp (1.36.x) - test passed.

Thanks again!

Chris

On 22 August 2017 at 22:39, Jakub Scholz <[hidden email]> wrote:

> I did some more digging through my old work emails. We had a customer with
> this kind of problem in June 2015. At that time I was able to reproduce
> this behaviour on Linux (I believe they were Windows users, so it should
> have affected also Windows). The problem was exactly the same as described
> by Chris.
>
> I was able to find a very primitive workaround at that time ... the
> "caching" was done based on hostname. So when two different hostnames were
> used the caching did not happen and the connections worked fine. So the
> workaround was to 1) create for each connection an artificial /etc/hosts
> record to which will route to the same broker, 2) Disable SSL hostname
> verification and 3) open each connection through a different hostname. Hey
> it was ugly and a bit insecure ... but it worked as a quick fix :-).
>
> Later the customer got back to us with following info:
> "After much investigation and experimentation, we have identified a
> potential issue with the Qpid library implementation of SSL connections,
> where the SSL session id might occasionally be "cached" or "reused". We
> believe this is the cause of the ACL exceptions, since when the session Id
> is cached and reused in an application that connects with multiple SSL
> client certificates, the previously used client certificate information of
> that previous SSL session might be reused as well for the next SSL
> connection attempt. We have implemented a fix in our Qpid library and
> application (by essentially disabling the caching of SSL session id for
> each SSL connection attempt), and deployed them in our UAT environment
> (user acceptance test). It seems to be working, as our application is able
> to run with multiple SSL certificates now."
>
> I asked them to provide the patch to the Qpid community. But I'm not sure
> they ever did - at least I cannot find anything. I'm not sure why I didn't
> raised a QPID JIRA for it my self. :-(
>
> I probably haven't tried it since 2015. So if it works for Gordon it might
> be already fixed in master.
>
> Regards
> Jakub
>
> On Mon, Aug 21, 2017 at 11:18 PM, Jakub Scholz <[hidden email]> wrote:
>
>> This is definitely not new in 1.36. I think this is present "since the
>> beginning". :-(
>>
>> If I remember correctly the problem was that the NSS library which handles
>> the SSL certificates is initialised only once even when you create multiple
>> connections. I was quite sure this was already discussed in some issue or
>> on the mailing list but I have problems finding it now - I will keep
>> looking.
>>
>> Regards
>> Jakub
>>
>> On Mon, Aug 21, 2017 at 10:46 PM, Chris Richardson <[hidden email]> wrote:
>>
>>> Hi,
>>>
>>> I've encountered an issue with the C++ broker (1.36) where it appears
>>> that multiple connections within the same process using SSL client
>>> auth do not use the "ssl-cert-name" property provided when the
>>> connection instance is created. Rather, it appears the second
>>> connection will use auth details of the first connection.
>>>
>>> Is this expected behaviour?
>>>
>>> Setup to show this happening is rather involved so I've created a jira
>>> issue with some attached material to demonstrate the problem:
>>> https://issues.apache.org/jira/browse/QPID-7894.
>>>
>>> Regards
>>>
>>> --
>>> Chris Richardson, System Architect
>>> [hidden email]
>>>
>>> FourC AS, Vestre Rosten 81, Trekanten, NO-7075 Tiller, Norway
>>> www.fourc.eu
>>>
>>> Follow us on LinkedIn, Facebook, Google+ and Twitter!
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>>
>>



--
Chris Richardson, System Architect
[hidden email]

FourC AS, Vestre Rosten 81, Trekanten, NO-7075 Tiller, Norway
www.fourc.eu

Follow us on LinkedIn, Facebook, Google+ and Twitter!

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]