Re: Proposal to create Docker images for Qpid components.

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Proposal to create Docker images for Qpid components.

Irina Boverman
As many as I can. Currently we do not package java broker as an RPM for
Fedora/CentOS, nor do we have it packaged for Ubuntu.


On Tue, Jun 13, 2017 at 12:03 PM, Lorenz Quack <[hidden email]>
wrote:

> Hello Irina,
>
> Sounds like a good idea.
>
> Do you plan on providing docker images of all Qpid components?
> Would be good to have for example the cpp and the java broker available to
> easily evaluate and compare them.
>
> Kind regards,
> Lorenz
>
>
> On Tue, 2017-06-13 at 10:13 -0400, Irina Boverman wrote:
> > Hi everyone,
> >
> > I would like to propose creating Docker images for Qpid components hosted
> > in Docker Hub, updated upon component release and maintained by the
> > project, and I would like to contribute to doing this.
> >
> > Availability of Qpid images will make it easier to consume/deploy Qpid
> > components and promote Qpid visibility.
> >
> > We can maintain docker scripts creating these images from the base OS
> > images and using Qpid installation methods consistent with the OS
> > distribution. A possible naming convention might be
> qpid/<component>/<OS>.
> > I registered the 'qpid' user on DockerHub to use if this seems
> reasonable.
> > For example, we could create qpid/dispatch/<OS> image, qpid/<broker>/<OS>
> > image, qpid/<client(s)>/<OS> image, etc. Initially I would look to
> support
> > Fedora/CentOS latest images and Qpid components as RPMs for them, then
> aim
> > to expand OS coverage for debian/Ubuntu/etc in the future.
> >
> > The goal would be to update Qpid images within a few days upon component
> > release (either directly or indirectly using yum/dnf from public
> > repositories). We could ask the Docker team to grant Qpid "official"
> status
> > when images have been stabilized.
> > --
> > Regards, Irina.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Proposal to create Docker images for Qpid components.

Irina Boverman
Thank you for excellent remarks. I will take a look at your work and what
can be done to minimise images, although I must admit it is easier to layer
on top of official images produced by the distros.

On Tue, Jun 13, 2017 at 11:36 AM, Fraser Adams <
[hidden email]> wrote:

> Hi Irina,
>
> I think that this is an excellent idea and I totally agree with the
> sentiment "Availability of Qpid images will make it easier to
> consume/deploy Qpid components and promote Qpid visibility." I think that
> in 2017 it's really important for the project to be providing official
> maintained images as otherwise we'll continue to see a fragmented
> miscellany of images of unknown provenance growing up out of necessity. To
> be fair that seems to be a general trend on Docker Hub for many projects,
> but I personally tend to find it a bit frustrating and would prefer a bit
> more control over provenance and versioning for things used on production
> systems.
>
> I'd make a few comments though.
>
> 1. It's probably better to propose this on the users list as that tends to
> have more traffic than the dev list. If you choose to repost on the users
> list feel free to include this response (if it's useful).
>
> 2. I do wonder about the comment "We can maintain docker scripts creating
> these images from the base OS images and using Qpid installation methods
> consistent with the OS distribution" and "I would look to support
> Fedora/CentOS latest images" and "expand OS coverage for debian/Ubuntu/etc".
>
> I can see why this might be the /obvious/ approach, but IMHO I think that
> it's a somewhat poor anti-pattern for containerisation that is propagated
> way too often on Docker Hub and tends to result in large bloaty images with
> larger potential attack surfaces than necessary and in addition reinforces
> a model that containers are somehow "lightweight VMs" rather than
> containerised appliances. I very much prefer a micro-containerised approach
> using an approach similar to that descrived here:
> http://blog.oddbit.com/2015/02/05/creating-minimal-docker-images/ though
> acknowledging that this approach adds some complexity to the build process
> and requires a little care.
>
> Ultimately reducing image size from the 400-500MB range to something like
> 55MB for qpidd feels like a good thing for a number of reasons, the
> following article contains some interesting musings on the subject
> https://opensolitude.com/2015/05/13/docker-images-should-be-small.html.
>
> It's a bit more complex for Python and Java based components, but I think
> that basing components on lighter weight Alpine Linux based Java or Python
> runtimes is probably the best way to go, I note that Docker Hub is (slowly)
> moving towards Alpine and http://www.iron.io/ have built some uber tiny
> Docker images for a wide range of languages see
> https://github.com/iron-io/dockers and https://hub.docker.com/u/iron/
>
> 3. One question to ask is which Qpid versions would be supported by the
> build system. Obviously one advantage of containerisation is the ability to
> easily use multiple versions of an application in order to support
> regression testing and potentially legacy capabilities in an operational
> environment, but of course there is a price to be paid as older versions
> run into difficulties with newer compilers, boost versions etc.
>
>
> I had a bit of a go at creating a build system for qpidd, my intention has
> been to do the same for other components, but I've been stuck for time, see
> https://github.com/fadams/docker-qpid though the qpid-cpp link is the
> only one with any useful content.
>
> The approach that I took was to create a Dockerised build system based on
> Debian so that the only dependency to actually build the component is a
> working Docker install and an Internet connection, the build system
> installs the necessary dev packages then installs the requested qpid-cpp
> source, runs the Qpid build then it exports the relevant shared libraries
> etc. and dynamic linker in order that a separate Dockerfile can use these
> to create a small image containing only what is necessary for qpidd to
> execute.
>
> I created two alternative build systems, one based on Debian wheezy the
> other based on jessie, the former was somewhat simpler as wheezy has
> somewhat older gcc and boost packages so less "wrangling" was necessary to
> build really old Qpid versions, with jessie I had to build gcc and boost
> from source. If you only intent to build recent Qpid versions much of that
> complexity is unnecessary as recent Qpid versions compile with recent gcc
> and boost versions, but I'm a little bit of a glutton for punishment and it
> seemed a bit more of a challenge to try and create something that could
> build every qpidd version from 0.5 to trunk. I suspect that the best
> approach might be to use jessie with the default compiler for recent Qpids
> and only use wheezy for older ones to provide legacy support.
>
>
> I'm no bash expert so the build.sh used by the Dockerfiles is a little bit
> monolithic and could do with something of a refactor into functions, but
> hopefully what it's doing should be fairly clear.
>
> Hope this is useful.
>
> Best regards,
>
> Frase
>
>
>
> On 13/06/17 15:13, Irina Boverman wrote:
>
>> Hi everyone,
>>
>> I would like to propose creating Docker images for Qpid components hosted
>> in Docker Hub, updated upon component release and maintained by the
>> project, and I would like to contribute to doing this.
>>
>> Availability of Qpid images will make it easier to consume/deploy Qpid
>> components and promote Qpid visibility.
>>
>> We can maintain docker scripts creating these images from the base OS
>> images and using Qpid installation methods consistent with the OS
>> distribution. A possible naming convention might be qpid/<component>/<OS>.
>> I registered the 'qpid' user on DockerHub to use if this seems reasonable.
>> For example, we could create qpid/dispatch/<OS> image, qpid/<broker>/<OS>
>> image, qpid/<client(s)>/<OS> image, etc. Initially I would look to support
>> Fedora/CentOS latest images and Qpid components as RPMs for them, then aim
>> to expand OS coverage for debian/Ubuntu/etc in the future.
>>
>> The goal would be to update Qpid images within a few days upon component
>> release (either directly or indirectly using yum/dnf from public
>> repositories). We could ask the Docker team to grant Qpid "official"
>> status
>> when images have been stabilized.
>> --
>> Regards, Irina.
>>
>>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Proposal to create Docker images for Qpid components.

Jakub Scholz-2
In reply to this post by Irina Boverman
Definitely a good idea.

I have my own images for C++ broker (https://hub.docker.com/r/
scholzj/qpid-cpp/) / Dispatch (https://hub.docker.com/r/
scholzj/qpid-dispatch/) / GUI for the C++ broker (
https://hub.docker.com/r/scholzj/qpid-cpp-gui/) and since I built them for
the first time I probably never used anything else. So having something
official and centrally maintained would be great.

What I'm curios about is what would you (and of course also others) expect
in terms of how to configure the images - what env. variables should be
supported etc.

Getting it to the official docker library would be great. What I'm not sure
about is whether we really need the different OS versions. What would be
the benefit of maintaining them?

Regards
Jakub


On Tue, Jun 13, 2017 at 4:13 PM, Irina Boverman <[hidden email]> wrote:

> Hi everyone,
>
> I would like to propose creating Docker images for Qpid components hosted
> in Docker Hub, updated upon component release and maintained by the
> project, and I would like to contribute to doing this.
>
> Availability of Qpid images will make it easier to consume/deploy Qpid
> components and promote Qpid visibility.
>
> We can maintain docker scripts creating these images from the base OS
> images and using Qpid installation methods consistent with the OS
> distribution. A possible naming convention might be qpid/<component>/<OS>.
> I registered the 'qpid' user on DockerHub to use if this seems reasonable.
> For example, we could create qpid/dispatch/<OS> image, qpid/<broker>/<OS>
> image, qpid/<client(s)>/<OS> image, etc. Initially I would look to support
> Fedora/CentOS latest images and Qpid components as RPMs for them, then aim
> to expand OS coverage for debian/Ubuntu/etc in the future.
>
> The goal would be to update Qpid images within a few days upon component
> release (either directly or indirectly using yum/dnf from public
> repositories). We could ask the Docker team to grant Qpid "official" status
> when images have been stabilized.
> --
> Regards, Irina.
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Proposal to create Docker images for Qpid components.

Jakub Scholz-2
In reply to this post by Irina Boverman
I think the small images have their pros and cons. When I run into problems
I find it always great to have something like CentOS, Fedora or Ubuntu as
the base image. You can easily install many tools to debug the problems
etc. And often the size doesn't have to be that big. It depends how minimal
the base image is.

BTW: I have also created my own build system (
https://github.com/scholzj/build-qpid-cpp and
https://github.com/scholzj/build-qpid-dispatch). The great thing is that
with Docker you can run the build anywhere including TravisCI or CircleCI.
I think this is another area which we might improve on as community. To
have also some nightly RPMs etc.

Regards
Jakub

On Tue, Jun 13, 2017 at 5:36 PM, Fraser Adams <[hidden email]
> wrote:

> Hi Irina,
>
> I think that this is an excellent idea and I totally agree with the
> sentiment "Availability of Qpid images will make it easier to
> consume/deploy Qpid components and promote Qpid visibility." I think that
> in 2017 it's really important for the project to be providing official
> maintained images as otherwise we'll continue to see a fragmented
> miscellany of images of unknown provenance growing up out of necessity. To
> be fair that seems to be a general trend on Docker Hub for many projects,
> but I personally tend to find it a bit frustrating and would prefer a bit
> more control over provenance and versioning for things used on production
> systems.
>
> I'd make a few comments though.
>
> 1. It's probably better to propose this on the users list as that tends to
> have more traffic than the dev list. If you choose to repost on the users
> list feel free to include this response (if it's useful).
>
> 2. I do wonder about the comment "We can maintain docker scripts creating
> these images from the base OS images and using Qpid installation methods
> consistent with the OS distribution" and "I would look to support
> Fedora/CentOS latest images" and "expand OS coverage for debian/Ubuntu/etc".
>
> I can see why this might be the /obvious/ approach, but IMHO I think that
> it's a somewhat poor anti-pattern for containerisation that is propagated
> way too often on Docker Hub and tends to result in large bloaty images with
> larger potential attack surfaces than necessary and in addition reinforces
> a model that containers are somehow "lightweight VMs" rather than
> containerised appliances. I very much prefer a micro-containerised approach
> using an approach similar to that descrived here:
> http://blog.oddbit.com/2015/02/05/creating-minimal-docker-images/ though
> acknowledging that this approach adds some complexity to the build process
> and requires a little care.
>
> Ultimately reducing image size from the 400-500MB range to something like
> 55MB for qpidd feels like a good thing for a number of reasons, the
> following article contains some interesting musings on the subject
> https://opensolitude.com/2015/05/13/docker-images-should-be-small.html.
>
> It's a bit more complex for Python and Java based components, but I think
> that basing components on lighter weight Alpine Linux based Java or Python
> runtimes is probably the best way to go, I note that Docker Hub is (slowly)
> moving towards Alpine and http://www.iron.io/ have built some uber tiny
> Docker images for a wide range of languages see
> https://github.com/iron-io/dockers and https://hub.docker.com/u/iron/
>
> 3. One question to ask is which Qpid versions would be supported by the
> build system. Obviously one advantage of containerisation is the ability to
> easily use multiple versions of an application in order to support
> regression testing and potentially legacy capabilities in an operational
> environment, but of course there is a price to be paid as older versions
> run into difficulties with newer compilers, boost versions etc.
>
>
> I had a bit of a go at creating a build system for qpidd, my intention has
> been to do the same for other components, but I've been stuck for time, see
> https://github.com/fadams/docker-qpid though the qpid-cpp link is the
> only one with any useful content.
>
> The approach that I took was to create a Dockerised build system based on
> Debian so that the only dependency to actually build the component is a
> working Docker install and an Internet connection, the build system
> installs the necessary dev packages then installs the requested qpid-cpp
> source, runs the Qpid build then it exports the relevant shared libraries
> etc. and dynamic linker in order that a separate Dockerfile can use these
> to create a small image containing only what is necessary for qpidd to
> execute.
>
> I created two alternative build systems, one based on Debian wheezy the
> other based on jessie, the former was somewhat simpler as wheezy has
> somewhat older gcc and boost packages so less "wrangling" was necessary to
> build really old Qpid versions, with jessie I had to build gcc and boost
> from source. If you only intent to build recent Qpid versions much of that
> complexity is unnecessary as recent Qpid versions compile with recent gcc
> and boost versions, but I'm a little bit of a glutton for punishment and it
> seemed a bit more of a challenge to try and create something that could
> build every qpidd version from 0.5 to trunk. I suspect that the best
> approach might be to use jessie with the default compiler for recent Qpids
> and only use wheezy for older ones to provide legacy support.
>
>
> I'm no bash expert so the build.sh used by the Dockerfiles is a little bit
> monolithic and could do with something of a refactor into functions, but
> hopefully what it's doing should be fairly clear.
>
> Hope this is useful.
>
> Best regards,
>
> Frase
>
>
>
> On 13/06/17 15:13, Irina Boverman wrote:
>
>> Hi everyone,
>>
>> I would like to propose creating Docker images for Qpid components hosted
>> in Docker Hub, updated upon component release and maintained by the
>> project, and I would like to contribute to doing this.
>>
>> Availability of Qpid images will make it easier to consume/deploy Qpid
>> components and promote Qpid visibility.
>>
>> We can maintain docker scripts creating these images from the base OS
>> images and using Qpid installation methods consistent with the OS
>> distribution. A possible naming convention might be qpid/<component>/<OS>.
>> I registered the 'qpid' user on DockerHub to use if this seems reasonable.
>> For example, we could create qpid/dispatch/<OS> image, qpid/<broker>/<OS>
>> image, qpid/<client(s)>/<OS> image, etc. Initially I would look to support
>> Fedora/CentOS latest images and Qpid components as RPMs for them, then aim
>> to expand OS coverage for debian/Ubuntu/etc in the future.
>>
>> The goal would be to update Qpid images within a few days upon component
>> release (either directly or indirectly using yum/dnf from public
>> repositories). We could ask the Docker team to grant Qpid "official"
>> status
>> when images have been stabilized.
>> --
>> Regards, Irina.
>>
>>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Proposal to create Docker images for Qpid components.

fadams
Oops, I've just realised that I accidentally PM'd this reply to Jakub
rather than the group (sorry Jakub, finger trouble!)

Re: "I think the small images have their pros and cons. When I run into
problems I find it always great to have something like CentOS, Fedora or
Ubuntu as the base image. You can easily install many tools to debug the
problems etc."

Sure there are trades. With what I did the build system Dockerfile
actually creates an executable qpidd too so you *could* use that and do
the sort of things you allude to too, but I prefer to create minimal
images as I feel it's a cleaner approach and I'd much rather operational
images expose as minimal a surface as possible.

I'd generally go with Alpine as a base OS if I had to use a base OS, but
I've not got Qpid compiling on Alpine yet and as I managed to get a
minimal OS-less Image for qpidd working I didn't pursue it.

IMHO there are far too many fat, bloaty and "kind-of-random" images
floating around on Docker Hub and have a hunch all those miscellaneous,
and usually unmaintained and unpatched base images sitting atop
ostensibly official application images is an accident waiting to happen.
As I said in my reply to Irina I think that there's a lot to be said for
having well understood provenance and good versioning, which is
something I see precious little of on Docker Hub, call me old fashioned .

Frase


On 13/06/17 20:39, Jakub Scholz wrote:

> I think the small images have their pros and cons. When I run into problems
> I find it always great to have something like CentOS, Fedora or Ubuntu as
> the base image. You can easily install many tools to debug the problems
> etc. And often the size doesn't have to be that big. It depends how minimal
> the base image is.
>
> BTW: I have also created my own build system (
> https://github.com/scholzj/build-qpid-cpp and
> https://github.com/scholzj/build-qpid-dispatch). The great thing is that
> with Docker you can run the build anywhere including TravisCI or CircleCI.
> I think this is another area which we might improve on as community. To
> have also some nightly RPMs etc.
>
> Regards
> Jakub
>
> On Tue, Jun 13, 2017 at 5:36 PM, Fraser Adams <[hidden email]
>> wrote:
>> Hi Irina,
>>
>> I think that this is an excellent idea and I totally agree with the
>> sentiment "Availability of Qpid images will make it easier to
>> consume/deploy Qpid components and promote Qpid visibility." I think that
>> in 2017 it's really important for the project to be providing official
>> maintained images as otherwise we'll continue to see a fragmented
>> miscellany of images of unknown provenance growing up out of necessity. To
>> be fair that seems to be a general trend on Docker Hub for many projects,
>> but I personally tend to find it a bit frustrating and would prefer a bit
>> more control over provenance and versioning for things used on production
>> systems.
>>
>> I'd make a few comments though.
>>
>> 1. It's probably better to propose this on the users list as that tends to
>> have more traffic than the dev list. If you choose to repost on the users
>> list feel free to include this response (if it's useful).
>>
>> 2. I do wonder about the comment "We can maintain docker scripts creating
>> these images from the base OS images and using Qpid installation methods
>> consistent with the OS distribution" and "I would look to support
>> Fedora/CentOS latest images" and "expand OS coverage for debian/Ubuntu/etc".
>>
>> I can see why this might be the /obvious/ approach, but IMHO I think that
>> it's a somewhat poor anti-pattern for containerisation that is propagated
>> way too often on Docker Hub and tends to result in large bloaty images with
>> larger potential attack surfaces than necessary and in addition reinforces
>> a model that containers are somehow "lightweight VMs" rather than
>> containerised appliances. I very much prefer a micro-containerised approach
>> using an approach similar to that descrived here:
>> http://blog.oddbit.com/2015/02/05/creating-minimal-docker-images/ though
>> acknowledging that this approach adds some complexity to the build process
>> and requires a little care.
>>
>> Ultimately reducing image size from the 400-500MB range to something like
>> 55MB for qpidd feels like a good thing for a number of reasons, the
>> following article contains some interesting musings on the subject
>> https://opensolitude.com/2015/05/13/docker-images-should-be-small.html.
>>
>> It's a bit more complex for Python and Java based components, but I think
>> that basing components on lighter weight Alpine Linux based Java or Python
>> runtimes is probably the best way to go, I note that Docker Hub is (slowly)
>> moving towards Alpine and http://www.iron.io/ have built some uber tiny
>> Docker images for a wide range of languages see
>> https://github.com/iron-io/dockers and https://hub.docker.com/u/iron/
>>
>> 3. One question to ask is which Qpid versions would be supported by the
>> build system. Obviously one advantage of containerisation is the ability to
>> easily use multiple versions of an application in order to support
>> regression testing and potentially legacy capabilities in an operational
>> environment, but of course there is a price to be paid as older versions
>> run into difficulties with newer compilers, boost versions etc.
>>
>>
>> I had a bit of a go at creating a build system for qpidd, my intention has
>> been to do the same for other components, but I've been stuck for time, see
>> https://github.com/fadams/docker-qpid though the qpid-cpp link is the
>> only one with any useful content.
>>
>> The approach that I took was to create a Dockerised build system based on
>> Debian so that the only dependency to actually build the component is a
>> working Docker install and an Internet connection, the build system
>> installs the necessary dev packages then installs the requested qpid-cpp
>> source, runs the Qpid build then it exports the relevant shared libraries
>> etc. and dynamic linker in order that a separate Dockerfile can use these
>> to create a small image containing only what is necessary for qpidd to
>> execute.
>>
>> I created two alternative build systems, one based on Debian wheezy the
>> other based on jessie, the former was somewhat simpler as wheezy has
>> somewhat older gcc and boost packages so less "wrangling" was necessary to
>> build really old Qpid versions, with jessie I had to build gcc and boost
>> from source. If you only intent to build recent Qpid versions much of that
>> complexity is unnecessary as recent Qpid versions compile with recent gcc
>> and boost versions, but I'm a little bit of a glutton for punishment and it
>> seemed a bit more of a challenge to try and create something that could
>> build every qpidd version from 0.5 to trunk. I suspect that the best
>> approach might be to use jessie with the default compiler for recent Qpids
>> and only use wheezy for older ones to provide legacy support.
>>
>>
>> I'm no bash expert so the build.sh used by the Dockerfiles is a little bit
>> monolithic and could do with something of a refactor into functions, but
>> hopefully what it's doing should be fairly clear.
>>
>> Hope this is useful.
>>
>> Best regards,
>>
>> Frase
>>
>>
>>
>> On 13/06/17 15:13, Irina Boverman wrote:
>>
>>> Hi everyone,
>>>
>>> I would like to propose creating Docker images for Qpid components hosted
>>> in Docker Hub, updated upon component release and maintained by the
>>> project, and I would like to contribute to doing this.
>>>
>>> Availability of Qpid images will make it easier to consume/deploy Qpid
>>> components and promote Qpid visibility.
>>>
>>> We can maintain docker scripts creating these images from the base OS
>>> images and using Qpid installation methods consistent with the OS
>>> distribution. A possible naming convention might be qpid/<component>/<OS>.
>>> I registered the 'qpid' user on DockerHub to use if this seems reasonable.
>>> For example, we could create qpid/dispatch/<OS> image, qpid/<broker>/<OS>
>>> image, qpid/<client(s)>/<OS> image, etc. Initially I would look to support
>>> Fedora/CentOS latest images and Qpid components as RPMs for them, then aim
>>> to expand OS coverage for debian/Ubuntu/etc in the future.
>>>
>>> The goal would be to update Qpid images within a few days upon component
>>> release (either directly or indirectly using yum/dnf from public
>>> repositories). We could ask the Docker team to grant Qpid "official"
>>> status
>>> when images have been stabilized.
>>> --
>>> Regards, Irina.
>>>
>>>

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Proposal to create Docker images for Qpid components.

rgodfrey
In reply to this post by Irina Boverman
On 19 June 2017 at 14:16, Robbie Gemmell <[hidden email]> wrote:

> I think docker images would be good to have.
>
>
+1


> Rather than reply to lots of little bits from various different mails,
> I'm just going to gather some general comments/questions here based on
> the previous mails.
>
> - If I understand correctly Irina's proposal is essentially to create
> create RPMs/DEBs (for certain components at least, might not be as
> useful/necessary to others, see later) and then create docker images
> using them (configure yum/dnf/apt repo, call install?).



If this is the proposal then does that also mean that these rpms/debs will
also have some sort of official status?  If so would we want to
document/vote that as well?


> To Jakub's
> point around doing things such as nightly RPMs, that could form part
> of it however we still wouldn't especially promote them to end users,

much like the maven repo snapshots arent, as such interim builds are
> only meant to be known to folks tracking/partipating in development as
> we are only allowed to direct regular everyday users at project
> releases.
>
>
Agreed - IIUC it would actually be against Apache policy to make available
any "official" binary that was not just a convenience build of a voted
release.


> - One thing I think worth some thought/discussion is versioning, e.g .
> what would the docker tags offered be. It looks like some projects do
> precise full-version tags to a specific 'major.minor.patch' release,
> as well as more general 'minor version' and 'major version' tags
> netting the latest release in a given stream. There might be some
> related behaviour there to decide on, e.g if using the OS package
> systems, would running an update result in moving to a newer component
> release in the stream. For some of those tag formats it could, while
> for others it should not.
>
> - If we do start offering images based on packages, then those
> packages and the related images essentially become 'convenience
> binaries' for the release that get created+tested by the project as
> part of the components release process. For bug fixes and
> functionality updates to be usable in images, there will need to be
> matching component releases to allow updating their packages and in
> turn images.


It seems to me that discussion on use of OS packaging systems and our
versioning within them is worthy of a separate discussion / proposal (and
may be a pre-requisite for building docker images for some components, if
that is the way we wish to go).


> I think this points to the project doing more [patch]
> releases for some of our components (i.e. ones we only do source
> release archives for), which I think would also be a very good thing.
>

+1


>
> - Its true that having multiple base OS options would take
> maintenance. Only having one option on the other hand probably
> involves some decisions as to which (for a given component? Perhaps
> they might not all be the same?). I think this also probably gets into
> considering what people are looking for when they use a docker image.
> In the case of a server such as a broker some people might just want
> the port(s) listening and doing stuff to use/try-out a particular
> server and dont care what OS etc is under it, while others might care
> that that its on a particular OS variant (e.g one they use for all
> their other bits), and others still might only care that its as small
> an image as possible (somewhat combining the previous behaviours to an
> extent).
>

Agreed - are there any precedents in other Apache projects that we think it
would be useful to follow?


>
> - To the comments around the Java broker, I don't think creating
> packages for it is really necessary? From a quick look at some others
> images it doesnt seem unusual to have a Dockerfile set up to pull the
> existing binary release archive, verify its sigs, and
> extract+configure it in an appropriate location.
>

This also make sense to me.  As a consumer of Java components, the use of
OS package systems would seem unnecessary here.

Do we have a list of the components/combinations which we think it would
make sense to produce/support docker images for?

-- Rob


>
> Robbie
>
> On 13 June 2017 at 15:13, Irina Boverman <[hidden email]> wrote:
> > Hi everyone,
> >
> > I would like to propose creating Docker images for Qpid components hosted
> > in Docker Hub, updated upon component release and maintained by the
> > project, and I would like to contribute to doing this.
> >
> > Availability of Qpid images will make it easier to consume/deploy Qpid
> > components and promote Qpid visibility.
> >
> > We can maintain docker scripts creating these images from the base OS
> > images and using Qpid installation methods consistent with the OS
> > distribution. A possible naming convention might be
> qpid/<component>/<OS>.
> > I registered the 'qpid' user on DockerHub to use if this seems
> reasonable.
> > For example, we could create qpid/dispatch/<OS> image, qpid/<broker>/<OS>
> > image, qpid/<client(s)>/<OS> image, etc. Initially I would look to
> support
> > Fedora/CentOS latest images and Qpid components as RPMs for them, then
> aim
> > to expand OS coverage for debian/Ubuntu/etc in the future.
> >
> > The goal would be to update Qpid images within a few days upon component
> > release (either directly or indirectly using yum/dnf from public
> > repositories). We could ask the Docker team to grant Qpid "official"
> status
> > when images have been stabilized.
> > --
> > Regards, Irina.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Proposal to create Docker images for Qpid components.

Robbie Gemmell
Administrator
On 19 June 2017 at 13:57, Rob Godfrey <[hidden email]> wrote:

> On 19 June 2017 at 14:16, Robbie Gemmell <[hidden email]> wrote:
>
>> I think docker images would be good to have.
>>
>>
> +1
>
>
>> Rather than reply to lots of little bits from various different mails,
>> I'm just going to gather some general comments/questions here based on
>> the previous mails.
>>
>> - If I understand correctly Irina's proposal is essentially to create
>> create RPMs/DEBs (for certain components at least, might not be as
>> useful/necessary to others, see later) and then create docker images
>> using them (configure yum/dnf/apt repo, call install?).
>
>
>
> If this is the proposal then does that also mean that these rpms/debs will
> also have some sort of official status?

I would say so yes, one of my later items was relating to that aspect.

>  If so would we want to
> document/vote that as well?
>
>
>> To Jakub's
>> point around doing things such as nightly RPMs, that could form part
>> of it however we still wouldn't especially promote them to end users,
>
> much like the maven repo snapshots arent, as such interim builds are
>> only meant to be known to folks tracking/partipating in development as
>> we are only allowed to direct regular everyday users at project
>> releases.
>>
>>
> Agreed - IIUC it would actually be against Apache policy to make available
> any "official" binary that was not just a convenience build of a voted
> release.
>

Yep

>
>> - One thing I think worth some thought/discussion is versioning, e.g .
>> what would the docker tags offered be. It looks like some projects do
>> precise full-version tags to a specific 'major.minor.patch' release,
>> as well as more general 'minor version' and 'major version' tags
>> netting the latest release in a given stream. There might be some
>> related behaviour there to decide on, e.g if using the OS package
>> systems, would running an update result in moving to a newer component
>> release in the stream. For some of those tag formats it could, while
>> for others it should not.
>>
>> - If we do start offering images based on packages, then those
>> packages and the related images essentially become 'convenience
>> binaries' for the release that get created+tested by the project as
>> part of the components release process. For bug fixes and
>> functionality updates to be usable in images, there will need to be
>> matching component releases to allow updating their packages and in
>> turn images.
>
>
> It seems to me that discussion on use of OS packaging systems and our
> versioning within them is worthy of a separate discussion / proposal (and
> may be a pre-requisite for building docker images for some components, if
> that is the way we wish to go).
>

Yes the specifics of any particular package should also be discussed
seperately. I'd say we are still at the point of this discusssion that
things can be discussed in tandem with regard to Docker images and
whether they go that way.

>
>> I think this points to the project doing more [patch]
>> releases for some of our components (i.e. ones we only do source
>> release archives for), which I think would also be a very good thing.
>>
>
> +1
>
>
>>
>> - Its true that having multiple base OS options would take
>> maintenance. Only having one option on the other hand probably
>> involves some decisions as to which (for a given component? Perhaps
>> they might not all be the same?). I think this also probably gets into
>> considering what people are looking for when they use a docker image.
>> In the case of a server such as a broker some people might just want
>> the port(s) listening and doing stuff to use/try-out a particular
>> server and dont care what OS etc is under it, while others might care
>> that that its on a particular OS variant (e.g one they use for all
>> their other bits), and others still might only care that its as small
>> an image as possible (somewhat combining the previous behaviours to an
>> extent).
>>
>
> Agreed - are there any precedents in other Apache projects that we think it
> would be useful to follow?
>

I'm not sure there is any consistent precedent, more a variety of
different appaches. Some dont mention OS at all, ones choose one,
others have a couple with perhaps Alpine being there for low size. For
some it also looks like images, though 'Docker official', aren't
necessarily maintained by the projects but rather the Docker community
instead. Some others it seems dont have a direct set of images at all
but perhaps do see popular images for related offerings built on them.

>
>>
>> - To the comments around the Java broker, I don't think creating
>> packages for it is really necessary? From a quick look at some others
>> images it doesnt seem unusual to have a Dockerfile set up to pull the
>> existing binary release archive, verify its sigs, and
>> extract+configure it in an appropriate location.
>>
>
> This also make sense to me.  As a consumer of Java components, the use of
> OS package systems would seem unnecessary here.
>
> Do we have a list of the components/combinations which we think it would
> make sense to produce/support docker images for?
>
> -- Rob
>
>
>>
>> Robbie
>>
>> On 13 June 2017 at 15:13, Irina Boverman <[hidden email]> wrote:
>> > Hi everyone,
>> >
>> > I would like to propose creating Docker images for Qpid components hosted
>> > in Docker Hub, updated upon component release and maintained by the
>> > project, and I would like to contribute to doing this.
>> >
>> > Availability of Qpid images will make it easier to consume/deploy Qpid
>> > components and promote Qpid visibility.
>> >
>> > We can maintain docker scripts creating these images from the base OS
>> > images and using Qpid installation methods consistent with the OS
>> > distribution. A possible naming convention might be
>> qpid/<component>/<OS>.
>> > I registered the 'qpid' user on DockerHub to use if this seems
>> reasonable.
>> > For example, we could create qpid/dispatch/<OS> image, qpid/<broker>/<OS>
>> > image, qpid/<client(s)>/<OS> image, etc. Initially I would look to
>> support
>> > Fedora/CentOS latest images and Qpid components as RPMs for them, then
>> aim
>> > to expand OS coverage for debian/Ubuntu/etc in the future.
>> >
>> > The goal would be to update Qpid images within a few days upon component
>> > release (either directly or indirectly using yum/dnf from public
>> > repositories). We could ask the Docker team to grant Qpid "official"
>> status
>> > when images have been stabilized.
>> > --
>> > Regards, Irina.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Proposal to create Docker images for Qpid components.

Lorenz Quack
In reply to this post by Irina Boverman
On Mon, 2017-06-19 at 13:16 +0100, Robbie Gemmell wrote:
> - To the comments around the Java broker, I don't think creating
> packages for it is really necessary? From a quick look at some others
> images it doesnt seem unusual to have a Dockerfile set up to pull the
> existing binary release archive, verify its sigs, and
> extract+configure it in an appropriate location.
>

Great. That would work for me.
I just thought it would be good to have the entire Qpid project represented
and to provide some choice at the same time.

Kind regards,
Lorenz

> Robbie
>
> On 13 June 2017 at 15:13, Irina Boverman <[hidden email]> wrote:
> >
> > Hi everyone,
> >
> > I would like to propose creating Docker images for Qpid components hosted
> > in Docker Hub, updated upon component release and maintained by the
> > project, and I would like to contribute to doing this.
> >
> > Availability of Qpid images will make it easier to consume/deploy Qpid
> > components and promote Qpid visibility.
> >
> > We can maintain docker scripts creating these images from the base OS
> > images and using Qpid installation methods consistent with the OS
> > distribution. A possible naming convention might be qpid/<component>/<OS>.
> > I registered the 'qpid' user on DockerHub to use if this seems reasonable.
> > For example, we could create qpid/dispatch/<OS> image, qpid/<broker>/<OS>
> > image, qpid/<client(s)>/<OS> image, etc. Initially I would look to support
> > Fedora/CentOS latest images and Qpid components as RPMs for them, then aim
> > to expand OS coverage for debian/Ubuntu/etc in the future.
> >
> > The goal would be to update Qpid images within a few days upon component
> > release (either directly or indirectly using yum/dnf from public
> > repositories). We could ask the Docker team to grant Qpid "official" status
> > when images have been stabilized.
> > --
> > Regards, Irina.
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Proposal to create Docker images for Qpid components.

fadams
Re: "it doesnt seem unusual to have a Dockerfile set up to pull the existing binary release archive, verify its sigs, and extract+configure it in an appropriate location."

Yes, it's certainly not unusual, but my personal view is that it isn't great practice.

As I said in my earlier reply to Irina, IMHO there are far too many instances of really bloaty Docker images containing far more than they need, as well as unnecessarily making images larger than they need to be (which isn't great if you are doing Continuous Deployment on a large system) it also unnecessarily increases the attack surface. Now OK Qpid brokers are probably long-lived services, so the first point might about minimising size may apply less to them than say 12 Factor App business function services, but as a general principle I tend to think that not enough thought is given to the footprint of Docker images.

I may have misunderstood, If the sentence I've quoted is referring to a Dockerfile for a *build system*, which subsequently exports a zip containing only that necessary to build (using a separate Dockerfile) a small, versioned microcontainer based on a minimal distro like Alpine (or from scratch) then that's fine, but having an image intended for use on a production system doing that sort of thing doesn't seem appropriate to me.

F.


On 20/06/17 08:54, Lorenz Quack wrote:

> On Mon, 2017-06-19 at 13:16 +0100, Robbie Gemmell wrote:
>> - To the comments around the Java broker, I don't think creating
>> packages for it is really necessary? From a quick look at some others
>> images it doesnt seem unusual to have a Dockerfile set up to pull the
>> existing binary release archive, verify its sigs, and
>> extract+configure it in an appropriate location.
>>
> Great. That would work for me.
> I just thought it would be good to have the entire Qpid project represented
> and to provide some choice at the same time.
>
> Kind regards,
> Lorenz
>
>> Robbie
>>
>> On 13 June 2017 at 15:13, Irina Boverman <[hidden email]> wrote:
>>> Hi everyone,
>>>
>>> I would like to propose creating Docker images for Qpid components hosted
>>> in Docker Hub, updated upon component release and maintained by the
>>> project, and I would like to contribute to doing this.
>>>
>>> Availability of Qpid images will make it easier to consume/deploy Qpid
>>> components and promote Qpid visibility.
>>>
>>> We can maintain docker scripts creating these images from the base OS
>>> images and using Qpid installation methods consistent with the OS
>>> distribution. A possible naming convention might be qpid/<component>/<OS>.
>>> I registered the 'qpid' user on DockerHub to use if this seems reasonable.
>>> For example, we could create qpid/dispatch/<OS> image, qpid/<broker>/<OS>
>>> image, qpid/<client(s)>/<OS> image, etc. Initially I would look to support
>>> Fedora/CentOS latest images and Qpid components as RPMs for them, then aim
>>> to expand OS coverage for debian/Ubuntu/etc in the future.
>>>
>>> The goal would be to update Qpid images within a few days upon component
>>> release (either directly or indirectly using yum/dnf from public
>>> repositories). We could ask the Docker team to grant Qpid "official" status
>>> when images have been stabilized.
>>> --
>>> Regards, Irina.
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Proposal to create Docker images for Qpid components.

Robbie Gemmell
Administrator
It was talking about downloading the built Java broker binary release
tar.gz, verifying it, and doing something with it. It wasn't saying
anything in particular about the OS, except there is one and Java is
available somehow.

For example, some randomly selected 'docker official' images I looked
at for Apache projects with Java components which all happened to do
this (I'm sure there are others that are different, of course):

https://hub.docker.com/r/_/tomcat/
not-alpine: https://github.com/docker-library/tomcat/blob/5ac222d258dc70c77bb3a9a4fab81ea286c9abd1/8.5/jre8/Dockerfile
alpine: https://github.com/docker-library/tomcat/blob/5ac222d258dc70c77bb3a9a4fab81ea286c9abd1/8.5/jre8-alpine/Dockerfile

https://hub.docker.com/_/maven/
not-alpine: https://github.com/carlossg/docker-maven/blob/0490eff01e529b2d94789511b008d01a7b314953/jdk-8/Dockerfile
alpine: https://github.com/carlossg/docker-maven/blob/2357d3394f19730172ac9c7f4afe7cf052f36b4d/jdk-8/Dockerfile

https://hub.docker.com/_/zookeeper/
alpine: https://github.com/31z4/zookeeper-docker/blob/f12428ab7c6ea263ef037cf258129b83276c009c/3.4.10/Dockerfile

At another try I got one thats doing something different:

https://hub.docker.com/_/cassandra/
https://github.com/docker-library/cassandra/blob/d83b850cd17bc9198876f8686197c730e29c7448/3.10/Dockerfile

Here they seem to be using their own .deb files via
http://www.apache.org/dist/cassandra/debian which actually redirects
to http://dl.bintray.com/apache/cassandra/, a debian repo
(https://bintray.com/apache/cassandra/debian) set up within the ASF
org on bintray (https://bintray.com/apache)

Robbie

On 20 June 2017 at 11:05, Fraser Adams <[hidden email]> wrote:

> Re: "it doesnt seem unusual to have a Dockerfile set up to pull the existing
> binary release archive, verify its sigs, and extract+configure it in an
> appropriate location."
>
> Yes, it's certainly not unusual, but my personal view is that it isn't great
> practice.
>
> As I said in my earlier reply to Irina, IMHO there are far too many
> instances of really bloaty Docker images containing far more than they need,
> as well as unnecessarily making images larger than they need to be (which
> isn't great if you are doing Continuous Deployment on a large system) it
> also unnecessarily increases the attack surface. Now OK Qpid brokers are
> probably long-lived services, so the first point might about minimising size
> may apply less to them than say 12 Factor App business function services,
> but as a general principle I tend to think that not enough thought is given
> to the footprint of Docker images.
>
> I may have misunderstood, If the sentence I've quoted is referring to a
> Dockerfile for a *build system*, which subsequently exports a zip containing
> only that necessary to build (using a separate Dockerfile) a small,
> versioned microcontainer based on a minimal distro like Alpine (or from
> scratch) then that's fine, but having an image intended for use on a
> production system doing that sort of thing doesn't seem appropriate to me.
>
> F.
>
>
>
> On 20/06/17 08:54, Lorenz Quack wrote:
>>
>> On Mon, 2017-06-19 at 13:16 +0100, Robbie Gemmell wrote:
>>>
>>> - To the comments around the Java broker, I don't think creating
>>> packages for it is really necessary? From a quick look at some others
>>> images it doesnt seem unusual to have a Dockerfile set up to pull the
>>> existing binary release archive, verify its sigs, and
>>> extract+configure it in an appropriate location.
>>>
>> Great. That would work for me.
>> I just thought it would be good to have the entire Qpid project
>> represented
>> and to provide some choice at the same time.
>>
>> Kind regards,
>> Lorenz
>>
>>> Robbie
>>>
>>> On 13 June 2017 at 15:13, Irina Boverman <[hidden email]> wrote:
>>>>
>>>> Hi everyone,
>>>>
>>>> I would like to propose creating Docker images for Qpid components
>>>> hosted
>>>> in Docker Hub, updated upon component release and maintained by the
>>>> project, and I would like to contribute to doing this.
>>>>
>>>> Availability of Qpid images will make it easier to consume/deploy Qpid
>>>> components and promote Qpid visibility.
>>>>
>>>> We can maintain docker scripts creating these images from the base OS
>>>> images and using Qpid installation methods consistent with the OS
>>>> distribution. A possible naming convention might be
>>>> qpid/<component>/<OS>.
>>>> I registered the 'qpid' user on DockerHub to use if this seems
>>>> reasonable.
>>>> For example, we could create qpid/dispatch/<OS> image,
>>>> qpid/<broker>/<OS>
>>>> image, qpid/<client(s)>/<OS> image, etc. Initially I would look to
>>>> support
>>>> Fedora/CentOS latest images and Qpid components as RPMs for them, then
>>>> aim
>>>> to expand OS coverage for debian/Ubuntu/etc in the future.
>>>>
>>>> The goal would be to update Qpid images within a few days upon component
>>>> release (either directly or indirectly using yum/dnf from public
>>>> repositories). We could ask the Docker team to grant Qpid "official"
>>>> status
>>>> when images have been stabilized.
>>>> --
>>>> Regards, Irina.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Proposal to create Docker images for Qpid components.

rgodfrey
So, stepping back for a second, which components do we think we should be
releasing docker images for (and once we've agreed on this we can agree on
the number/form of images for each component perhaps :-) )?

-- Rob

On 20 June 2017 at 14:06, Robbie Gemmell <[hidden email]> wrote:

> It was talking about downloading the built Java broker binary release
> tar.gz, verifying it, and doing something with it. It wasn't saying
> anything in particular about the OS, except there is one and Java is
> available somehow.
>
> For example, some randomly selected 'docker official' images I looked
> at for Apache projects with Java components which all happened to do
> this (I'm sure there are others that are different, of course):
>
> https://hub.docker.com/r/_/tomcat/
> not-alpine: https://github.com/docker-library/tomcat/blob/
> 5ac222d258dc70c77bb3a9a4fab81ea286c9abd1/8.5/jre8/Dockerfile
> alpine: https://github.com/docker-library/tomcat/blob/
> 5ac222d258dc70c77bb3a9a4fab81ea286c9abd1/8.5/jre8-alpine/Dockerfile
>
> https://hub.docker.com/_/maven/
> not-alpine: https://github.com/carlossg/docker-maven/blob/
> 0490eff01e529b2d94789511b008d01a7b314953/jdk-8/Dockerfile
> alpine: https://github.com/carlossg/docker-maven/blob/
> 2357d3394f19730172ac9c7f4afe7cf052f36b4d/jdk-8/Dockerfile
>
> https://hub.docker.com/_/zookeeper/
> alpine: https://github.com/31z4/zookeeper-docker/blob/
> f12428ab7c6ea263ef037cf258129b83276c009c/3.4.10/Dockerfile
>
> At another try I got one thats doing something different:
>
> https://hub.docker.com/_/cassandra/
> https://github.com/docker-library/cassandra/blob/
> d83b850cd17bc9198876f8686197c730e29c7448/3.10/Dockerfile
>
> Here they seem to be using their own .deb files via
> http://www.apache.org/dist/cassandra/debian which actually redirects
> to http://dl.bintray.com/apache/cassandra/, a debian repo
> (https://bintray.com/apache/cassandra/debian) set up within the ASF
> org on bintray (https://bintray.com/apache)
>
> Robbie
>
> On 20 June 2017 at 11:05, Fraser Adams <[hidden email]>
> wrote:
> > Re: "it doesnt seem unusual to have a Dockerfile set up to pull the
> existing
> > binary release archive, verify its sigs, and extract+configure it in an
> > appropriate location."
> >
> > Yes, it's certainly not unusual, but my personal view is that it isn't
> great
> > practice.
> >
> > As I said in my earlier reply to Irina, IMHO there are far too many
> > instances of really bloaty Docker images containing far more than they
> need,
> > as well as unnecessarily making images larger than they need to be (which
> > isn't great if you are doing Continuous Deployment on a large system) it
> > also unnecessarily increases the attack surface. Now OK Qpid brokers are
> > probably long-lived services, so the first point might about minimising
> size
> > may apply less to them than say 12 Factor App business function services,
> > but as a general principle I tend to think that not enough thought is
> given
> > to the footprint of Docker images.
> >
> > I may have misunderstood, If the sentence I've quoted is referring to a
> > Dockerfile for a *build system*, which subsequently exports a zip
> containing
> > only that necessary to build (using a separate Dockerfile) a small,
> > versioned microcontainer based on a minimal distro like Alpine (or from
> > scratch) then that's fine, but having an image intended for use on a
> > production system doing that sort of thing doesn't seem appropriate to
> me.
> >
> > F.
> >
> >
> >
> > On 20/06/17 08:54, Lorenz Quack wrote:
> >>
> >> On Mon, 2017-06-19 at 13:16 +0100, Robbie Gemmell wrote:
> >>>
> >>> - To the comments around the Java broker, I don't think creating
> >>> packages for it is really necessary? From a quick look at some others
> >>> images it doesnt seem unusual to have a Dockerfile set up to pull the
> >>> existing binary release archive, verify its sigs, and
> >>> extract+configure it in an appropriate location.
> >>>
> >> Great. That would work for me.
> >> I just thought it would be good to have the entire Qpid project
> >> represented
> >> and to provide some choice at the same time.
> >>
> >> Kind regards,
> >> Lorenz
> >>
> >>> Robbie
> >>>
> >>> On 13 June 2017 at 15:13, Irina Boverman <[hidden email]> wrote:
> >>>>
> >>>> Hi everyone,
> >>>>
> >>>> I would like to propose creating Docker images for Qpid components
> >>>> hosted
> >>>> in Docker Hub, updated upon component release and maintained by the
> >>>> project, and I would like to contribute to doing this.
> >>>>
> >>>> Availability of Qpid images will make it easier to consume/deploy Qpid
> >>>> components and promote Qpid visibility.
> >>>>
> >>>> We can maintain docker scripts creating these images from the base OS
> >>>> images and using Qpid installation methods consistent with the OS
> >>>> distribution. A possible naming convention might be
> >>>> qpid/<component>/<OS>.
> >>>> I registered the 'qpid' user on DockerHub to use if this seems
> >>>> reasonable.
> >>>> For example, we could create qpid/dispatch/<OS> image,
> >>>> qpid/<broker>/<OS>
> >>>> image, qpid/<client(s)>/<OS> image, etc. Initially I would look to
> >>>> support
> >>>> Fedora/CentOS latest images and Qpid components as RPMs for them, then
> >>>> aim
> >>>> to expand OS coverage for debian/Ubuntu/etc in the future.
> >>>>
> >>>> The goal would be to update Qpid images within a few days upon
> component
> >>>> release (either directly or indirectly using yum/dnf from public
> >>>> repositories). We could ask the Docker team to grant Qpid "official"
> >>>> status
> >>>> when images have been stabilized.
> >>>> --
> >>>> Regards, Irina.
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: [hidden email]
> >>> For additional commands, e-mail: [hidden email]
> >>>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: [hidden email]
> >> For additional commands, e-mail: [hidden email]
> >>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Proposal to create Docker images for Qpid components.

Robbie Gemmell
Administrator
I think it makes sense for qpid-broker-j, qpid-cpp broker, and qpid-dispatch.

I believe some folks might also like 'client' images, which is much
less obvious to me..though I can see that for those needing
compilation or with interdependencies on bits that do, perhaps it
could make them slightly easier to get started with. Packages would
also in many cases though.

On 20 June 2017 at 14:17, Rob Godfrey <[hidden email]> wrote:

> So, stepping back for a second, which components do we think we should be
> releasing docker images for (and once we've agreed on this we can agree on
> the number/form of images for each component perhaps :-) )?
>
> -- Rob
>
> On 20 June 2017 at 14:06, Robbie Gemmell <[hidden email]> wrote:
>
>> It was talking about downloading the built Java broker binary release
>> tar.gz, verifying it, and doing something with it. It wasn't saying
>> anything in particular about the OS, except there is one and Java is
>> available somehow.
>>
>> For example, some randomly selected 'docker official' images I looked
>> at for Apache projects with Java components which all happened to do
>> this (I'm sure there are others that are different, of course):
>>
>> https://hub.docker.com/r/_/tomcat/
>> not-alpine: https://github.com/docker-library/tomcat/blob/
>> 5ac222d258dc70c77bb3a9a4fab81ea286c9abd1/8.5/jre8/Dockerfile
>> alpine: https://github.com/docker-library/tomcat/blob/
>> 5ac222d258dc70c77bb3a9a4fab81ea286c9abd1/8.5/jre8-alpine/Dockerfile
>>
>> https://hub.docker.com/_/maven/
>> not-alpine: https://github.com/carlossg/docker-maven/blob/
>> 0490eff01e529b2d94789511b008d01a7b314953/jdk-8/Dockerfile
>> alpine: https://github.com/carlossg/docker-maven/blob/
>> 2357d3394f19730172ac9c7f4afe7cf052f36b4d/jdk-8/Dockerfile
>>
>> https://hub.docker.com/_/zookeeper/
>> alpine: https://github.com/31z4/zookeeper-docker/blob/
>> f12428ab7c6ea263ef037cf258129b83276c009c/3.4.10/Dockerfile
>>
>> At another try I got one thats doing something different:
>>
>> https://hub.docker.com/_/cassandra/
>> https://github.com/docker-library/cassandra/blob/
>> d83b850cd17bc9198876f8686197c730e29c7448/3.10/Dockerfile
>>
>> Here they seem to be using their own .deb files via
>> http://www.apache.org/dist/cassandra/debian which actually redirects
>> to http://dl.bintray.com/apache/cassandra/, a debian repo
>> (https://bintray.com/apache/cassandra/debian) set up within the ASF
>> org on bintray (https://bintray.com/apache)
>>
>> Robbie
>>
>> On 20 June 2017 at 11:05, Fraser Adams <[hidden email]>
>> wrote:
>> > Re: "it doesnt seem unusual to have a Dockerfile set up to pull the
>> existing
>> > binary release archive, verify its sigs, and extract+configure it in an
>> > appropriate location."
>> >
>> > Yes, it's certainly not unusual, but my personal view is that it isn't
>> great
>> > practice.
>> >
>> > As I said in my earlier reply to Irina, IMHO there are far too many
>> > instances of really bloaty Docker images containing far more than they
>> need,
>> > as well as unnecessarily making images larger than they need to be (which
>> > isn't great if you are doing Continuous Deployment on a large system) it
>> > also unnecessarily increases the attack surface. Now OK Qpid brokers are
>> > probably long-lived services, so the first point might about minimising
>> size
>> > may apply less to them than say 12 Factor App business function services,
>> > but as a general principle I tend to think that not enough thought is
>> given
>> > to the footprint of Docker images.
>> >
>> > I may have misunderstood, If the sentence I've quoted is referring to a
>> > Dockerfile for a *build system*, which subsequently exports a zip
>> containing
>> > only that necessary to build (using a separate Dockerfile) a small,
>> > versioned microcontainer based on a minimal distro like Alpine (or from
>> > scratch) then that's fine, but having an image intended for use on a
>> > production system doing that sort of thing doesn't seem appropriate to
>> me.
>> >
>> > F.
>> >
>> >
>> >
>> > On 20/06/17 08:54, Lorenz Quack wrote:
>> >>
>> >> On Mon, 2017-06-19 at 13:16 +0100, Robbie Gemmell wrote:
>> >>>
>> >>> - To the comments around the Java broker, I don't think creating
>> >>> packages for it is really necessary? From a quick look at some others
>> >>> images it doesnt seem unusual to have a Dockerfile set up to pull the
>> >>> existing binary release archive, verify its sigs, and
>> >>> extract+configure it in an appropriate location.
>> >>>
>> >> Great. That would work for me.
>> >> I just thought it would be good to have the entire Qpid project
>> >> represented
>> >> and to provide some choice at the same time.
>> >>
>> >> Kind regards,
>> >> Lorenz
>> >>
>> >>> Robbie
>> >>>
>> >>> On 13 June 2017 at 15:13, Irina Boverman <[hidden email]> wrote:
>> >>>>
>> >>>> Hi everyone,
>> >>>>
>> >>>> I would like to propose creating Docker images for Qpid components
>> >>>> hosted
>> >>>> in Docker Hub, updated upon component release and maintained by the
>> >>>> project, and I would like to contribute to doing this.
>> >>>>
>> >>>> Availability of Qpid images will make it easier to consume/deploy Qpid
>> >>>> components and promote Qpid visibility.
>> >>>>
>> >>>> We can maintain docker scripts creating these images from the base OS
>> >>>> images and using Qpid installation methods consistent with the OS
>> >>>> distribution. A possible naming convention might be
>> >>>> qpid/<component>/<OS>.
>> >>>> I registered the 'qpid' user on DockerHub to use if this seems
>> >>>> reasonable.
>> >>>> For example, we could create qpid/dispatch/<OS> image,
>> >>>> qpid/<broker>/<OS>
>> >>>> image, qpid/<client(s)>/<OS> image, etc. Initially I would look to
>> >>>> support
>> >>>> Fedora/CentOS latest images and Qpid components as RPMs for them, then
>> >>>> aim
>> >>>> to expand OS coverage for debian/Ubuntu/etc in the future.
>> >>>>
>> >>>> The goal would be to update Qpid images within a few days upon
>> component
>> >>>> release (either directly or indirectly using yum/dnf from public
>> >>>> repositories). We could ask the Docker team to grant Qpid "official"
>> >>>> status
>> >>>> when images have been stabilized.
>> >>>> --
>> >>>> Regards, Irina.
>> >>>
>> >>> ---------------------------------------------------------------------
>> >>> To unsubscribe, e-mail: [hidden email]
>> >>> For additional commands, e-mail: [hidden email]
>> >>>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: [hidden email]
>> >> For additional commands, e-mail: [hidden email]
>> >>
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: [hidden email]
>> > For additional commands, e-mail: [hidden email]
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Proposal to create Docker images for Qpid components.

Robbie Gemmell
Administrator
On 29 June 2017 at 17:52, Irina Boverman <[hidden email]> wrote:

> Thank you all who thoughtfully responded to this proposal. This is a
> summary as I understood it and my comments regarding these points.
>
> (1) Qpid community is in favour of creating and maintaining docker images
> for cpp/java brokers
> and dispatch router. OS choice is less important, as long as the user can
> connect to the broker/router port. We only need to support one OS choice.
>
> I think this is a good start. It covers a use case when someone wants to
> test or deploy client apps against broker/router conveniently running as a
> docker service.
>
> I think there might be another use case for those users who want to develop
> client apps in a known environment with client libraries already installed
> for them to work with. I think the OS choice is important to these users
> because they may have a preference/better knowledge of this particular OS.
> Please consider this use case.
>
> (2) Docker images should be of minimal size.
>
> As a concept, I fully agree with it. However, community support for
> official images also makes sense to me. These images go through review and
> testing, as well as, get patches for CVEs. There is a whole infrastructure
> in place in Fedora to build docker images, etc following best practices
> from OCI.
>
> Official images range in size between 120 to 200 mb, and have 10+ mil of
> downloads, so the size does not appear to be an obstacle to adopting them.
> It is my opinion that future official images will evolve to be better
> aligned with how they are used, for example "server" and "workstation", etc.
>
> REPOSITORY                          TAG                 IMAGE ID
> CREATED             SIZE
> docker.io/ubuntu                    latest              ebcd9d4fca80
> 6 weeks ago         117.9 MB
> docker.io/debian                    latest              e5599115b6a6
> 5 months ago        123 MB
> docker.io/centos                    latest              67591570dd29
> 6 months ago        191.8 MB
> docker.io/fedora                    latest              a1e614f0f30e
> 6 months ago        197.1 MB
>
> We can also modify the base image to remove some items if we think they are
> not needed.
>

I think using a base OS image is fine, and your text and Justin's link
suggests there are likely to be improvements in some regards in this
area too. There is also nothing stopping a further image as minimal
size option based on Alpine etc, but obviously components would need
to work and be supported there first and it wasn't clear that is
currently always true.

> My personal preference is to start with fedora image because this
> distribution has latest upstream releases for qpid packages, and I am the
> maintainer for a number of these packages.

I think we possibly need some clarity around the end result here, as
it could strongly influence this particular area.

For images to be considered 'official' Apache Qpid artifacts
maintained directly by the project, they and their Qpid related
contents would really need to be fully managed in concert with the
projects release process. For example, if such an image were utilising
packages to incorporate their relevant Qpid bits then I think it
follows those packages would similarly need to be considered
'official' and handled similarly. Utilising the Qpid packages from
Fedora distributions for example, wouldn't be appropriate in an
'official Apache Qpid' image since they are not directly managed
artifacts of the Apache Qpid project.

There wouldn't be an issue in simply considering such images to be a
community-created convenience binaries based on the projects source
release, as e.g. the existing Fedora packages themselves already are,
but they wouldn't be considered 'official' Apache Qpid artifacts. It
is worth saying this isn't particularly unusual, for example httpd
direct people to their source releases but do reference various third
party binaries for Windows users, and don't seem to be involved with
the docker images for httpd, with members of the docker community
instead handling that.

I think either approach works here, having distribution based packages
obviously has certain advantages for users and the project, so its
probably more of a question around what people are actually expecting
and/or looking to contribute towards here.

> Fedora community is also working on its own docker registry (it will have
> stable and test registries), has koji support for building images and
> guidelines for naming and versioning them.
>
> (3). What upstream releases will be dockerised?
>
> I would like to propose to start with latest released version. We will
> maintain at most 1 version at any time. All code necessary to create these
> images should be included in the upstream source repository starting with
> the next release. We will provide security patches for it as needed (and
> possibly will need patch releases). I am not sure if it will be possible to
> remove older releases from docker registries (TBD).
>
> (4) What will be used to configure docker images?
>
> We can use env to configure brokers/router, similar to what was done here:
> https://hub.docker.com/r/scholzj/qpid-cpp/ (courtesy of Jakub Scholz).
>
> References:
>
> #Changes/Layered Docker Image Build Service
> https://fedoraproject.org/wiki/Changes/Layered_Docker_Image_Build_Service#Policies_and_guidelines
>
> #Fedora Docker Layered Image Build Service by Adam Miller
> https://blog.openshift.com/wp-content/uploads/Introducing-Docker-Layered-Image-Build-Service.pdf
>
> #OpenShift Commons Briefing #57: Fedora Docker Layered Image Build Service
> by Adam Miller
> https://www.youtube.com/watch?v=HHm0L6Fw5nk
>
> #Layered Image Build System
> https://docs.pagure.org/infra-docs/sysadmin-guide/sops/layered-image-buildsys.html
>
> #Atomic Reactor: Python library with command line interface for building
> docker images.
> https://github.com/projectatomic/atomic-reactor
>
> #Fedora Docker Layered image build service now available blog
> https://communityblog.fedoraproject.org/fedora-docker-layered-image-build-service-now-available/
>
> #Example of container build: cockpit
> https://koji.fedoraproject.org/koji/packageinfo?packageID=17959
>
> #Container:Guidelines
> https://fedoraproject.org/wiki/Container:Guidelines
>
> #Fedora container repo
> https://src.fedoraproject.org/cgit/container/
>
> #Building a modular Linux OS
> https://docs.pagure.org/modularity/
>
> #projectatomic/container-best-practices
> https://github.com/projectatomic/container-best-practices
>
> #Container Best Practices
> http://docs.projectatomic.io/container-best-practices/
>
> #Red Hat Image Naming Policy
> https://github.com/projectatomic/ContainerApplicationGenericLabels/blob/master/vendor/redhat/names.md
>
> #Guidelines for Naming Fedora Packages (including docker images)
> https://fedoraproject.org/wiki/Packaging:Naming?rd=Packaging:NamingGuidelines
>
> #OCI: open container initiative
> https://www.opencontainers.org/
>
> #Changes/FedoraDockerRegistry
> https://fedoraproject.org/wiki/Changes/FedoraDockerRegistry
>
> #Creating minimal Docker images from dynamically linked ELF binaries
> http://blog.oddbit.com/2015/02/05/creating-minimal-docker-images/
>
> #Why docker images should be small
> https://opensolitude.com/2015/05/13/docker-images-should-be-small.html
>
> Regards, Irina.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Loading...