Python 3 port is 'done'

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

Python 3 port is 'done'

Ken Giusti

Well, done enough to consider merging to master.

While the patch is quite large, most of the changes are simple syntax changes to avoid non-python3 compliant syntax.

The code is available on the kgiusti-python3 branch at the Apache repo.

https://git-wip-us.apache.org/repos/asf?p=qpid-proton.git;a=shortlog;h=refs/heads/kgiusti-python3

I've also made a patch that can be viewed up on reviewboard:

https://reviews.apache.org/r/33691/

I've verified that the unit tests and python examples run under python2.6, 2.7, and python3.3.   I'd appreciate if folks would take this patch for a spin and report back their experience.

Known Issues:

These changes will be incompatible with earlier versions of the python 2.x series.  I know for a fact that python versions <= 2.4 won't even parse this patch, and I suspect getting such older versions of python to work would require lots of effort.   I'm a little unsure of how well python 2.5 will be supported - I have yet to test that far back.  I also didn't test anything earlier than 3.3 in the python3.x stream.

--
-K
Reply | Threaded
Open this post in threaded view
|

Re: Python 3 port is 'done'

Rafael Schloming-3
What happens when I run make test and I have both python2 and python3
installed on my system? Do the tests run once under each version or does
one of the versions 'win'?

--Rafael

On Wed, Apr 29, 2015 at 4:05 PM, Ken Giusti <[hidden email]> wrote:

>
> Well, done enough to consider merging to master.
>
> While the patch is quite large, most of the changes are simple syntax
> changes to avoid non-python3 compliant syntax.
>
> The code is available on the kgiusti-python3 branch at the Apache repo.
>
>
> https://git-wip-us.apache.org/repos/asf?p=qpid-proton.git;a=shortlog;h=refs/heads/kgiusti-python3
>
> I've also made a patch that can be viewed up on reviewboard:
>
> https://reviews.apache.org/r/33691/
>
> I've verified that the unit tests and python examples run under python2.6,
> 2.7, and python3.3.   I'd appreciate if folks would take this patch for a
> spin and report back their experience.
>
> Known Issues:
>
> These changes will be incompatible with earlier versions of the python 2.x
> series.  I know for a fact that python versions <= 2.4 won't even parse
> this patch, and I suspect getting such older versions of python to work
> would require lots of effort.   I'm a little unsure of how well python 2.5
> will be supported - I have yet to test that far back.  I also didn't test
> anything earlier than 3.3 in the python3.x stream.
>
> --
> -K
>
Reply | Threaded
Open this post in threaded view
|

Re: Python 3 port is 'done'

Ken Giusti


----- Original Message -----
> From: "Rafael Schloming" <[hidden email]>
> To: [hidden email]
> Sent: Wednesday, April 29, 2015 4:24:09 PM
> Subject: Re: Python 3 port is 'done'
>
> What happens when I run make test and I have both python2 and python3
> installed on my system? Do the tests run once under each version or does
> one of the versions 'win'?

At this point it only runs on the 'default' version - whatever /usr/bin/python resolves to.

I like the idea of having it run on all installed python versions, but I haven't explored how to do that yet.

I've been using virtualenv [1] to switch between the two versions of python I have installed on my development station.  Tox [2] is probably the best approach to enable testing against multiple python environments.

I'll look into tox a bit and see what I can come up with.

-K

[1] http://docs.python-guide.org/en/latest/dev/virtualenvs/
[2] https://tox.readthedocs.org/en/latest/




>
> --Rafael
>
> On Wed, Apr 29, 2015 at 4:05 PM, Ken Giusti <[hidden email]> wrote:
>
> >
> > Well, done enough to consider merging to master.
> >
> > While the patch is quite large, most of the changes are simple syntax
> > changes to avoid non-python3 compliant syntax.
> >
> > The code is available on the kgiusti-python3 branch at the Apache repo.
> >
> >
> > https://git-wip-us.apache.org/repos/asf?p=qpid-proton.git;a=shortlog;h=refs/heads/kgiusti-python3
> >
> > I've also made a patch that can be viewed up on reviewboard:
> >
> > https://reviews.apache.org/r/33691/
> >
> > I've verified that the unit tests and python examples run under python2.6,
> > 2.7, and python3.3.   I'd appreciate if folks would take this patch for a
> > spin and report back their experience.
> >
> > Known Issues:
> >
> > These changes will be incompatible with earlier versions of the python 2.x
> > series.  I know for a fact that python versions <= 2.4 won't even parse
> > this patch, and I suspect getting such older versions of python to work
> > would require lots of effort.   I'm a little unsure of how well python 2.5
> > will be supported - I have yet to test that far back.  I also didn't test
> > anything earlier than 3.3 in the python3.x stream.
> >
> > --
> > -K
> >
>

--
-K
Reply | Threaded
Open this post in threaded view
|

Re: Python 3 port is 'done'

Rafael Schloming-3
On Thu, Apr 30, 2015 at 8:35 AM, Ken Giusti <[hidden email]> wrote:

>
>
> ----- Original Message -----
> > From: "Rafael Schloming" <[hidden email]>
> > To: [hidden email]
> > Sent: Wednesday, April 29, 2015 4:24:09 PM
> > Subject: Re: Python 3 port is 'done'
> >
> > What happens when I run make test and I have both python2 and python3
> > installed on my system? Do the tests run once under each version or does
> > one of the versions 'win'?
>
> At this point it only runs on the 'default' version - whatever
> /usr/bin/python resolves to.
>
> I like the idea of having it run on all installed python versions, but I
> haven't explored how to do that yet.
>
> I've been using virtualenv [1] to switch between the two versions of
> python I have installed on my development station.  Tox [2] is probably the
> best approach to enable testing against multiple python environments.
>
> I'll look into tox a bit and see what I can come up with.
>

My system comes with both python and python3 on my path. Just running
python3 manually on proton/tests/proton-test will run it with the python3
interpreter. I don't know how standard this setup is (I'm running stock
fedora 20), but it would be pretty easy to do a check in cmake and run the
tests using python3 if present.

I'm also a fan of running both python versions if present, but I also don't
want to double the time it takes to run through the tests. Given that we
are mostly looking for syntactic incompatibilities in the wrapper code
here, I wonder if it would be sufficient to run a subset of the tests that
is likely to give us good coverage on the wrapper code but doesn't bother
trying to exercise all the C code twice. Obviously if this proves
insufficient we could expand the subset.

--Rafael
Reply | Threaded
Open this post in threaded view
|

Re: Python 3 port is 'done'

Ken Giusti


----- Original Message -----

> From: "Rafael Schloming" <[hidden email]>
> To: [hidden email]
> Sent: Thursday, April 30, 2015 9:00:14 AM
> Subject: Re: Python 3 port is 'done'
>
> On Thu, Apr 30, 2015 at 8:35 AM, Ken Giusti <[hidden email]> wrote:
>
> >
> >
> > ----- Original Message -----
> > > From: "Rafael Schloming" <[hidden email]>
> > > To: [hidden email]
> > > Sent: Wednesday, April 29, 2015 4:24:09 PM
> > > Subject: Re: Python 3 port is 'done'
> > >
> > > What happens when I run make test and I have both python2 and python3
> > > installed on my system? Do the tests run once under each version or does
> > > one of the versions 'win'?
> >
> > At this point it only runs on the 'default' version - whatever
> > /usr/bin/python resolves to.
> >
> > I like the idea of having it run on all installed python versions, but I
> > haven't explored how to do that yet.
> >
> > I've been using virtualenv [1] to switch between the two versions of
> > python I have installed on my development station.  Tox [2] is probably the
> > best approach to enable testing against multiple python environments.
> >
> > I'll look into tox a bit and see what I can come up with.
> >
>
> My system comes with both python and python3 on my path. Just running
> python3 manually on proton/tests/proton-test will run it with the python3
> interpreter. I don't know how standard this setup is (I'm running stock
> fedora 20), but it would be pretty easy to do a check in cmake and run the
> tests using python3 if present.
>
> I'm also a fan of running both python versions if present, but I also don't
> want to double the time it takes to run through the tests. Given that we
> are mostly looking for syntactic incompatibilities in the wrapper code
> here, I wonder if it would be sufficient to run a subset of the tests that
> is likely to give us good coverage on the wrapper code but doesn't bother
> trying to exercise all the C code twice. Obviously if this proves
> insufficient we could expand the subset.

Oh yeah - totally agreed.  Just some smaller subset of python-test would make me happy.  I found most problems were covered by the engine, codec, transport test modules btw.  If that's all we need, then simply running python3 directly on the unit tests makes the most sense.

>
> --Rafael
>

--
-K
Reply | Threaded
Open this post in threaded view
|

Re: Python 3 port is 'done'

Robbie Gemmell
Administrator
In reply to this post by Ken Giusti
On 29 April 2015 at 21:05, Ken Giusti <[hidden email]> wrote:

>
> Well, done enough to consider merging to master.
>
> While the patch is quite large, most of the changes are simple syntax changes to avoid non-python3 compliant syntax.
>
> The code is available on the kgiusti-python3 branch at the Apache repo.
>
> https://git-wip-us.apache.org/repos/asf?p=qpid-proton.git;a=shortlog;h=refs/heads/kgiusti-python3
>
> I've also made a patch that can be viewed up on reviewboard:
>
> https://reviews.apache.org/r/33691/
>
> I've verified that the unit tests and python examples run under python2.6, 2.7, and python3.3.   I'd appreciate if folks would take this patch for a spin and report back their experience.
>
> Known Issues:
>
> These changes will be incompatible with earlier versions of the python 2.x series.  I know for a fact that python versions <= 2.4 won't even parse this patch, and I suspect getting such older versions of python to work would require lots of effort.   I'm a little unsure of how well python 2.5 will be supported - I have yet to test that far back.  I also didn't test anything earlier than 3.3 in the python3.x stream.
>
> --
> -K

I gave thigns a kick with Python 2.7, and Jython 2.5.3 without issue.

I also tried the maven build with Jython 2.7 RC3 (there was a new one)
and things exploded similarly to the way they did before.

Robbie
Reply | Threaded
Open this post in threaded view
|

Re: Python 3 port is 'done'

Ken Giusti


----- Original Message -----

> From: "Robbie Gemmell" <[hidden email]>
> To: [hidden email]
> Cc: [hidden email]
> Sent: Thursday, April 30, 2015 10:20:07 AM
> Subject: Re: Python 3 port is 'done'
>
> On 29 April 2015 at 21:05, Ken Giusti <[hidden email]> wrote:
> >
> > Well, done enough to consider merging to master.
> >
> > While the patch is quite large, most of the changes are simple syntax
> > changes to avoid non-python3 compliant syntax.
> >
> > The code is available on the kgiusti-python3 branch at the Apache repo.
> >
> > https://git-wip-us.apache.org/repos/asf?p=qpid-proton.git;a=shortlog;h=refs/heads/kgiusti-python3
> >
> > I've also made a patch that can be viewed up on reviewboard:
> >
> > https://reviews.apache.org/r/33691/
> >
> > I've verified that the unit tests and python examples run under python2.6,
> > 2.7, and python3.3.   I'd appreciate if folks would take this patch for a
> > spin and report back their experience.
> >
> > Known Issues:
> >
> > These changes will be incompatible with earlier versions of the python 2.x
> > series.  I know for a fact that python versions <= 2.4 won't even parse
> > this patch, and I suspect getting such older versions of python to work
> > would require lots of effort.   I'm a little unsure of how well python 2.5
> > will be supported - I have yet to test that far back.  I also didn't test
> > anything earlier than 3.3 in the python3.x stream.
> >
> > --
> > -K
>
> I gave thigns a kick with Python 2.7, and Jython 2.5.3 without issue.
>
> I also tried the maven build with Jython 2.7 RC3 (there was a new one)
> and things exploded similarly to the way they did before.
>

Thanks Robbie.

What kind of issues does Jython 2.7 complain about?  I'll have to install that RC at some point... :(

> Robbie
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

--
-K
Reply | Threaded
Open this post in threaded view
|

Re: Python 3 port is 'done'

Ken Giusti
In reply to this post by Ken Giusti


----- Original Message -----

> From: "Ken Giusti" <[hidden email]>
> To: [hidden email]
> Cc: [hidden email]
> Sent: Thursday, April 30, 2015 9:18:26 AM
> Subject: Re: Python 3 port is 'done'
>
>
>
> ----- Original Message -----
> > From: "Rafael Schloming" <[hidden email]>
> > To: [hidden email]
> > Sent: Thursday, April 30, 2015 9:00:14 AM
> > Subject: Re: Python 3 port is 'done'
> >
> > On Thu, Apr 30, 2015 at 8:35 AM, Ken Giusti <[hidden email]> wrote:
> >
> > >
> > >
> > > ----- Original Message -----
> > > > From: "Rafael Schloming" <[hidden email]>
> > > > To: [hidden email]
> > > > Sent: Wednesday, April 29, 2015 4:24:09 PM
> > > > Subject: Re: Python 3 port is 'done'
> > > >
> > > > What happens when I run make test and I have both python2 and python3
> > > > installed on my system? Do the tests run once under each version or
> > > > does
> > > > one of the versions 'win'?
> > >
> > > At this point it only runs on the 'default' version - whatever
> > > /usr/bin/python resolves to.
> > >
> > > I like the idea of having it run on all installed python versions, but I
> > > haven't explored how to do that yet.
> > >
> > > I've been using virtualenv [1] to switch between the two versions of
> > > python I have installed on my development station.  Tox [2] is probably
> > > the
> > > best approach to enable testing against multiple python environments.
> > >
> > > I'll look into tox a bit and see what I can come up with.
> > >
> >
> > My system comes with both python and python3 on my path. Just running
> > python3 manually on proton/tests/proton-test will run it with the python3
> > interpreter. I don't know how standard this setup is (I'm running stock
> > fedora 20), but it would be pretty easy to do a check in cmake and run the
> > tests using python3 if present.
> >
> > I'm also a fan of running both python versions if present, but I also don't
> > want to double the time it takes to run through the tests. Given that we
> > are mostly looking for syntactic incompatibilities in the wrapper code
> > here, I wonder if it would be sufficient to run a subset of the tests that
> > is likely to give us good coverage on the wrapper code but doesn't bother
> > trying to exercise all the C code twice. Obviously if this proves
> > insufficient we could expand the subset.
>
> Oh yeah - totally agreed.  Just some smaller subset of python-test would make
> me happy.  I found most problems were covered by the engine, codec,
> transport test modules btw.  If that's all we need, then simply running
> python3 directly on the unit tests makes the most sense.
>


Ah, turns out the 'hard problem' is not running the tests, but building both py2 and py3 binaries from the c-wrapper.  Totally different include and link libraries and different install paths.

Cmake doesn't appear to directly support this - it only resolves to one instance of python, though that instance can be configured.  But nowhere does it provide a list of available pythons - we'd have to script that ourselves.

Once we have that list, we should use the setup.py script to build/install the language specific module. setup.py already handles swig, so simply invoking setup.py for each available python interpreter would do all the heavy lifting.

thoughts?




> >
> > --Rafael
> >
>
> --
> -K
>

--
-K
Reply | Threaded
Open this post in threaded view
|

Re: Python 3 port is 'done'

Robbie Gemmell
Administrator
In reply to this post by Ken Giusti
On 30 April 2015 at 15:56, Ken Giusti <[hidden email]> wrote:

>
>
> ----- Original Message -----
>> From: "Robbie Gemmell" <[hidden email]>
>> To: [hidden email]
>> Cc: [hidden email]
>> Sent: Thursday, April 30, 2015 10:20:07 AM
>> Subject: Re: Python 3 port is 'done'
>>
>> On 29 April 2015 at 21:05, Ken Giusti <[hidden email]> wrote:
>> >
>> > Well, done enough to consider merging to master.
>> >
>> > While the patch is quite large, most of the changes are simple syntax
>> > changes to avoid non-python3 compliant syntax.
>> >
>> > The code is available on the kgiusti-python3 branch at the Apache repo.
>> >
>> > https://git-wip-us.apache.org/repos/asf?p=qpid-proton.git;a=shortlog;h=refs/heads/kgiusti-python3
>> >
>> > I've also made a patch that can be viewed up on reviewboard:
>> >
>> > https://reviews.apache.org/r/33691/
>> >
>> > I've verified that the unit tests and python examples run under python2.6,
>> > 2.7, and python3.3.   I'd appreciate if folks would take this patch for a
>> > spin and report back their experience.
>> >
>> > Known Issues:
>> >
>> > These changes will be incompatible with earlier versions of the python 2.x
>> > series.  I know for a fact that python versions <= 2.4 won't even parse
>> > this patch, and I suspect getting such older versions of python to work
>> > would require lots of effort.   I'm a little unsure of how well python 2.5
>> > will be supported - I have yet to test that far back.  I also didn't test
>> > anything earlier than 3.3 in the python3.x stream.
>> >
>> > --
>> > -K
>>
>> I gave thigns a kick with Python 2.7, and Jython 2.5.3 without issue.
>>
>> I also tried the maven build with Jython 2.7 RC3 (there was a new one)
>> and things exploded similarly to the way they did before.
>>
>
> Thanks Robbie.
>
> What kind of issues does Jython 2.7 complain about?  I'll have to install that RC at some point... :(
>
> -K

Lots of the tests fail due to error, most if not all of which seem to
be "TypeError: Type not compatible with array". As I say though, this
isnt to do with your changes since it did that last time I tried too
:)

If you want to try it, below is change needed. Probably worth making
sure you clean out the old .pyc files left by previous test runs (I
use git-clean to nuke things back to fresh state).

diff --git tests/pom.xml tests/pom.xml
index 8367eca..7dcee1e 100644
--- tests/pom.xml
+++ tests/pom.xml
@@ -93,7 +93,7 @@ directory &lt;basedir&gt;/build/proton-c.</description>
     <dependency>
         <groupId>org.python</groupId>
         <artifactId>jython-standalone</artifactId>
-        <version>2.5.3</version>
+        <version>2.7-rc3</version>
       <scope>test</scope>
     </dependency>
   </dependencies>
Reply | Threaded
Open this post in threaded view
|

Re: Python 3 port is 'done'

Chuck Rolke
In reply to this post by Ken Giusti
Failures on Windows (beyond the four SASL failures in PROTON-875) running Python 2.6.1

37 MessengerTests tests (all that aren't skipped) fail with
  Unexpected input while waiting for receiver to initialize: READY


----- Original Message -----

> From: "Ken Giusti" <[hidden email]>
> To: [hidden email], [hidden email]
> Sent: Wednesday, April 29, 2015 4:05:32 PM
> Subject: Python 3 port is 'done'
>
>
> Well, done enough to consider merging to master.
>
> While the patch is quite large, most of the changes are simple syntax changes
> to avoid non-python3 compliant syntax.
>
> The code is available on the kgiusti-python3 branch at the Apache repo.
>
> https://git-wip-us.apache.org/repos/asf?p=qpid-proton.git;a=shortlog;h=refs/heads/kgiusti-python3
>
> I've also made a patch that can be viewed up on reviewboard:
>
> https://reviews.apache.org/r/33691/
>
> I've verified that the unit tests and python examples run under python2.6,
> 2.7, and python3.3.   I'd appreciate if folks would take this patch for a
> spin and report back their experience.
>
> Known Issues:
>
> These changes will be incompatible with earlier versions of the python 2.x
> series.  I know for a fact that python versions <= 2.4 won't even parse this
> patch, and I suspect getting such older versions of python to work would
> require lots of effort.   I'm a little unsure of how well python 2.5 will be
> supported - I have yet to test that far back.  I also didn't test anything
> earlier than 3.3 in the python3.x stream.
>
> --
> -K
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Python 3 port is 'done'

Ken Giusti
Thanks for checking that Chuck.   I think I've fixed it - all tests that pass on master now pass for me on the kgiusti-python3 branch under windows.

I see failures of the proton_tests.sasl.* tests on both master and the py3 branch, but I think that may be related by my cyrus setup.

----- Original Message -----

> From: "Chuck Rolke" <[hidden email]>
> To: [hidden email]
> Cc: [hidden email]
> Sent: Thursday, April 30, 2015 5:15:14 PM
> Subject: Re: Python 3 port is 'done'
>
> Failures on Windows (beyond the four SASL failures in PROTON-875) running
> Python 2.6.1
>
> 37 MessengerTests tests (all that aren't skipped) fail with
>   Unexpected input while waiting for receiver to initialize: READY
>
>
> ----- Original Message -----
> > From: "Ken Giusti" <[hidden email]>
> > To: [hidden email], [hidden email]
> > Sent: Wednesday, April 29, 2015 4:05:32 PM
> > Subject: Python 3 port is 'done'
> >
> >
> > Well, done enough to consider merging to master.
> >
> > While the patch is quite large, most of the changes are simple syntax
> > changes
> > to avoid non-python3 compliant syntax.
> >
> > The code is available on the kgiusti-python3 branch at the Apache repo.
> >
> > https://git-wip-us.apache.org/repos/asf?p=qpid-proton.git;a=shortlog;h=refs/heads/kgiusti-python3
> >
> > I've also made a patch that can be viewed up on reviewboard:
> >
> > https://reviews.apache.org/r/33691/
> >
> > I've verified that the unit tests and python examples run under python2.6,
> > 2.7, and python3.3.   I'd appreciate if folks would take this patch for a
> > spin and report back their experience.
> >
> > Known Issues:
> >
> > These changes will be incompatible with earlier versions of the python 2.x
> > series.  I know for a fact that python versions <= 2.4 won't even parse
> > this
> > patch, and I suspect getting such older versions of python to work would
> > require lots of effort.   I'm a little unsure of how well python 2.5 will
> > be
> > supported - I have yet to test that far back.  I also didn't test anything
> > earlier than 3.3 in the python3.x stream.
> >
> > --
> > -K
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
>

--
-K
Reply | Threaded
Open this post in threaded view
|

Re: Python 3 port is 'done'

Rafael Schloming-3
In reply to this post by Robbie Gemmell
On Thu, Apr 30, 2015 at 11:18 AM, Robbie Gemmell <[hidden email]>
wrote:

> On 30 April 2015 at 15:56, Ken Giusti <[hidden email]> wrote:
> >
> >
> > ----- Original Message -----
> >> From: "Robbie Gemmell" <[hidden email]>
> >> To: [hidden email]
> >> Cc: [hidden email]
> >> Sent: Thursday, April 30, 2015 10:20:07 AM
> >> Subject: Re: Python 3 port is 'done'
> >>
> >> On 29 April 2015 at 21:05, Ken Giusti <[hidden email]> wrote:
> >> >
> >> > Well, done enough to consider merging to master.
> >> >
> >> > While the patch is quite large, most of the changes are simple syntax
> >> > changes to avoid non-python3 compliant syntax.
> >> >
> >> > The code is available on the kgiusti-python3 branch at the Apache
> repo.
> >> >
> >> >
> https://git-wip-us.apache.org/repos/asf?p=qpid-proton.git;a=shortlog;h=refs/heads/kgiusti-python3
> >> >
> >> > I've also made a patch that can be viewed up on reviewboard:
> >> >
> >> > https://reviews.apache.org/r/33691/
> >> >
> >> > I've verified that the unit tests and python examples run under
> python2.6,
> >> > 2.7, and python3.3.   I'd appreciate if folks would take this patch
> for a
> >> > spin and report back their experience.
> >> >
> >> > Known Issues:
> >> >
> >> > These changes will be incompatible with earlier versions of the
> python 2.x
> >> > series.  I know for a fact that python versions <= 2.4 won't even
> parse
> >> > this patch, and I suspect getting such older versions of python to
> work
> >> > would require lots of effort.   I'm a little unsure of how well
> python 2.5
> >> > will be supported - I have yet to test that far back.  I also didn't
> test
> >> > anything earlier than 3.3 in the python3.x stream.
> >> >
> >> > --
> >> > -K
> >>
> >> I gave thigns a kick with Python 2.7, and Jython 2.5.3 without issue.
> >>
> >> I also tried the maven build with Jython 2.7 RC3 (there was a new one)
> >> and things exploded similarly to the way they did before.
> >>
> >
> > Thanks Robbie.
> >
> > What kind of issues does Jython 2.7 complain about?  I'll have to
> install that RC at some point... :(
> >
> > -K
>
> Lots of the tests fail due to error, most if not all of which seem to
> be "TypeError: Type not compatible with array". As I say though, this
> isnt to do with your changes since it did that last time I tried too
> :)
>

I happened to run into a similar error recently and tracked the issue down.
I believe this error is due to an API incompatibility between jython 2.5
and jython 2.7. In jython 2.5 you need to convert python strings into java
byte arrays manually and you use the 'array' constructor from the jarray
module to do this, e.g. something like:
array("string-that-has-bytes-in-it", 'b') will construct a java byte array
from a python string. In jython 2.7 this conversion is done automatically
(which is nice). Sadly though, the array constructor that worked in jython
2.5 now barfs with exactly the TypeError message you are showing above. I
was able to get a simple smoke test working with proton-j and jython 2.7 by
removing the array constructor and just passing the string directly,
however this will not work on 2.5, so we probably either need to write our
own conversion function that behaves differently for different jython
versions or do a full switch over to 2.7.

--Rafael
Reply | Threaded
Open this post in threaded view
|

Re: Python 3 port is 'done'

Robbie Gemmell
Administrator
On 8 May 2015 at 13:34, Rafael Schloming <[hidden email]> wrote:

> On Thu, Apr 30, 2015 at 11:18 AM, Robbie Gemmell <[hidden email]>
> wrote:
>
>> On 30 April 2015 at 15:56, Ken Giusti <[hidden email]> wrote:
>> >
>> >
>> > ----- Original Message -----
>> >> From: "Robbie Gemmell" <[hidden email]>
>> >> To: [hidden email]
>> >> Cc: [hidden email]
>> >> Sent: Thursday, April 30, 2015 10:20:07 AM
>> >> Subject: Re: Python 3 port is 'done'
>> >>
>> >> On 29 April 2015 at 21:05, Ken Giusti <[hidden email]> wrote:
>> >> >
>> >> > Well, done enough to consider merging to master.
>> >> >
>> >> > While the patch is quite large, most of the changes are simple syntax
>> >> > changes to avoid non-python3 compliant syntax.
>> >> >
>> >> > The code is available on the kgiusti-python3 branch at the Apache
>> repo.
>> >> >
>> >> >
>> https://git-wip-us.apache.org/repos/asf?p=qpid-proton.git;a=shortlog;h=refs/heads/kgiusti-python3
>> >> >
>> >> > I've also made a patch that can be viewed up on reviewboard:
>> >> >
>> >> > https://reviews.apache.org/r/33691/
>> >> >
>> >> > I've verified that the unit tests and python examples run under
>> python2.6,
>> >> > 2.7, and python3.3.   I'd appreciate if folks would take this patch
>> for a
>> >> > spin and report back their experience.
>> >> >
>> >> > Known Issues:
>> >> >
>> >> > These changes will be incompatible with earlier versions of the
>> python 2.x
>> >> > series.  I know for a fact that python versions <= 2.4 won't even
>> parse
>> >> > this patch, and I suspect getting such older versions of python to
>> work
>> >> > would require lots of effort.   I'm a little unsure of how well
>> python 2.5
>> >> > will be supported - I have yet to test that far back.  I also didn't
>> test
>> >> > anything earlier than 3.3 in the python3.x stream.
>> >> >
>> >> > --
>> >> > -K
>> >>
>> >> I gave thigns a kick with Python 2.7, and Jython 2.5.3 without issue.
>> >>
>> >> I also tried the maven build with Jython 2.7 RC3 (there was a new one)
>> >> and things exploded similarly to the way they did before.
>> >>
>> >
>> > Thanks Robbie.
>> >
>> > What kind of issues does Jython 2.7 complain about?  I'll have to
>> install that RC at some point... :(
>> >
>> > -K
>>
>> Lots of the tests fail due to error, most if not all of which seem to
>> be "TypeError: Type not compatible with array". As I say though, this
>> isnt to do with your changes since it did that last time I tried too
>> :)
>>
>
> I happened to run into a similar error recently and tracked the issue down.
> I believe this error is due to an API incompatibility between jython 2.5
> and jython 2.7. In jython 2.5 you need to convert python strings into java
> byte arrays manually and you use the 'array' constructor from the jarray
> module to do this, e.g. something like:
> array("string-that-has-bytes-in-it", 'b') will construct a java byte array
> from a python string. In jython 2.7 this conversion is done automatically
> (which is nice). Sadly though, the array constructor that worked in jython
> 2.5 now barfs with exactly the TypeError message you are showing above. I
> was able to get a simple smoke test working with proton-j and jython 2.7 by
> removing the array constructor and just passing the string directly,
> however this will not work on 2.5, so we probably either need to write our
> own conversion function that behaves differently for different jython
> versions or do a full switch over to 2.7.
>
> --Rafael

Oops. I missed your reply at the time, just saw it while looking for
something else.

I think we would just switch entirely to Jython 2.7, which has since
reached GA, wouldnt we? Given those bits are only really intended for
use in the tests, I don't see the same need for wide ranging
compatibility there. I also wouldn't expect people to be configuring a
different major version of Jython (but if I were to try, some
incompatibility wouldn't necessarily be that surprising to me
either).

Robbie
Reply | Threaded
Open this post in threaded view
|

Re: Python 3 port is 'done'

Flavio Percoco
In reply to this post by Ken Giusti
On 30/04/15 11:08 -0400, Ken Giusti wrote:

>
>
>----- Original Message -----
>> From: "Ken Giusti" <[hidden email]>
>> To: [hidden email]
>> Cc: [hidden email]
>> Sent: Thursday, April 30, 2015 9:18:26 AM
>> Subject: Re: Python 3 port is 'done'
>>
>>
>>
>> ----- Original Message -----
>> > From: "Rafael Schloming" <[hidden email]>
>> > To: [hidden email]
>> > Sent: Thursday, April 30, 2015 9:00:14 AM
>> > Subject: Re: Python 3 port is 'done'
>> >
>> > On Thu, Apr 30, 2015 at 8:35 AM, Ken Giusti <[hidden email]> wrote:
>> >
>> > >
>> > >
>> > > ----- Original Message -----
>> > > > From: "Rafael Schloming" <[hidden email]>
>> > > > To: [hidden email]
>> > > > Sent: Wednesday, April 29, 2015 4:24:09 PM
>> > > > Subject: Re: Python 3 port is 'done'
>> > > >
>> > > > What happens when I run make test and I have both python2 and python3
>> > > > installed on my system? Do the tests run once under each version or
>> > > > does
>> > > > one of the versions 'win'?
>> > >
>> > > At this point it only runs on the 'default' version - whatever
>> > > /usr/bin/python resolves to.
>> > >
>> > > I like the idea of having it run on all installed python versions, but I
>> > > haven't explored how to do that yet.
>> > >
>> > > I've been using virtualenv [1] to switch between the two versions of
>> > > python I have installed on my development station.  Tox [2] is probably
>> > > the
>> > > best approach to enable testing against multiple python environments.
>> > >
>> > > I'll look into tox a bit and see what I can come up with.
>> > >
>> >
>> > My system comes with both python and python3 on my path. Just running
>> > python3 manually on proton/tests/proton-test will run it with the python3
>> > interpreter. I don't know how standard this setup is (I'm running stock
>> > fedora 20), but it would be pretty easy to do a check in cmake and run the
>> > tests using python3 if present.
>> >
>> > I'm also a fan of running both python versions if present, but I also don't
>> > want to double the time it takes to run through the tests. Given that we
>> > are mostly looking for syntactic incompatibilities in the wrapper code
>> > here, I wonder if it would be sufficient to run a subset of the tests that
>> > is likely to give us good coverage on the wrapper code but doesn't bother
>> > trying to exercise all the C code twice. Obviously if this proves
>> > insufficient we could expand the subset.
>>
>> Oh yeah - totally agreed.  Just some smaller subset of python-test would make
>> me happy.  I found most problems were covered by the engine, codec,
>> transport test modules btw.  If that's all we need, then simply running
>> python3 directly on the unit tests makes the most sense.
>>
>
>
>Ah, turns out the 'hard problem' is not running the tests, but building both py2 and py3 binaries from the c-wrapper.  Totally different include and link libraries and different install paths.
>
>Cmake doesn't appear to directly support this - it only resolves to one instance of python, though that instance can be configured.  But nowhere does it provide a list of available pythons - we'd have to script that ourselves.
>
>Once we have that list, we should use the setup.py script to build/install the language specific module. setup.py already handles swig, so simply invoking setup.py for each available python interpreter would do all the heavy lifting.
>
>thoughts?
I'd probably have cmake/ctest running things for the default python
and leave the rest of the tests to `tox` as it'll create the required
virtual environments for each supported version. This would be my
preference.

That said, you could also have cmake building the bindings for each
version in different paths. This will complicate the cmake step
further and I presonally think this belongs to the bindings,

Cheers,
Flavio

>
>
>
>
>> >
>> > --Rafael
>> >
>>
>> --
>> -K
>>
>
>--
>-K
--
@flaper87
Flavio Percoco

attachment0 (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Python 3 port is 'done'

Ken Giusti
In reply to this post by Ken Giusti

No, really - this time I'm serious.

I've resynced with master, fixed up a few minor issues, and pushed to the kgiusti-python3 branch:

https://git-wip-us.apache.org/repos/asf?p=qpid-proton.git;a=shortlog;h=refs/heads/kgiusti-python3

Thanks go to Flavio for all the help with getting this done and tested.

I'm planning on merging this into master after the weekend unless I hear of any show stoppers.

thanks,

-K


----- Original Message -----

> From: "Ken Giusti" <[hidden email]>
> To: [hidden email], [hidden email]
> Sent: Wednesday, April 29, 2015 4:05:32 PM
> Subject: Python 3 port is 'done'
>
>
> Well, done enough to consider merging to master.
>
> While the patch is quite large, most of the changes are simple syntax changes
> to avoid non-python3 compliant syntax.
>
> The code is available on the kgiusti-python3 branch at the Apache repo.
>
> https://git-wip-us.apache.org/repos/asf?p=qpid-proton.git;a=shortlog;h=refs/heads/kgiusti-python3
>
> I've also made a patch that can be viewed up on reviewboard:
>
> https://reviews.apache.org/r/33691/
>
> I've verified that the unit tests and python examples run under python2.6,
> 2.7, and python3.3.   I'd appreciate if folks would take this patch for a
> spin and report back their experience.
>
> Known Issues:
>
> These changes will be incompatible with earlier versions of the python 2.x
> series.  I know for a fact that python versions <= 2.4 won't even parse this
> patch, and I suspect getting such older versions of python to work would
> require lots of effort.   I'm a little unsure of how well python 2.5 will be
> supported - I have yet to test that far back.  I also didn't test anything
> earlier than 3.3 in the python3.x stream.
>
> --
> -K
>

Reply | Threaded
Open this post in threaded view
|

Re: Python 3 port is 'done'

Ken Giusti
FYI:

Python 3 support has been merged to qpid-proton master.

Let us know if you hit any problems - thanks.

-K


----- Original Message -----

> From: "Ken Giusti" <[hidden email]>
> To: [hidden email], [hidden email]
> Cc: "Flavio Percoco" <[hidden email]>
> Sent: Thursday, June 18, 2015 4:21:23 PM
> Subject: Re: Python 3 port is 'done'
>
>
> No, really - this time I'm serious.
>
> I've resynced with master, fixed up a few minor issues, and pushed to the
> kgiusti-python3 branch:
>
> https://git-wip-us.apache.org/repos/asf?p=qpid-proton.git;a=shortlog;h=refs/heads/kgiusti-python3
>
> Thanks go to Flavio for all the help with getting this done and tested.
>
> I'm planning on merging this into master after the weekend unless I hear of
> any show stoppers.
>
> thanks,
>
> -K
>
>
> ----- Original Message -----
> > From: "Ken Giusti" <[hidden email]>
> > To: [hidden email], [hidden email]
> > Sent: Wednesday, April 29, 2015 4:05:32 PM
> > Subject: Python 3 port is 'done'
> >
> >
> > Well, done enough to consider merging to master.
> >
> > While the patch is quite large, most of the changes are simple syntax
> > changes
> > to avoid non-python3 compliant syntax.
> >
> > The code is available on the kgiusti-python3 branch at the Apache repo.
> >
> > https://git-wip-us.apache.org/repos/asf?p=qpid-proton.git;a=shortlog;h=refs/heads/kgiusti-python3
> >
> > I've also made a patch that can be viewed up on reviewboard:
> >
> > https://reviews.apache.org/r/33691/
> >
> > I've verified that the unit tests and python examples run under python2.6,
> > 2.7, and python3.3.   I'd appreciate if folks would take this patch for a
> > spin and report back their experience.
> >
> > Known Issues:
> >
> > These changes will be incompatible with earlier versions of the python 2.x
> > series.  I know for a fact that python versions <= 2.4 won't even parse
> > this
> > patch, and I suspect getting such older versions of python to work would
> > require lots of effort.   I'm a little unsure of how well python 2.5 will
> > be
> > supported - I have yet to test that far back.  I also didn't test anything
> > earlier than 3.3 in the python3.x stream.
> >
> > --
> > -K
> >
>
>

--
-K