[Pulp-dev] Providing Pulp 3 Container Images and Installation Support

Wed Feb 13 01:57:48 UTC 2019

On Tue, Feb 12, 2019 at 1:00 PM Daniel Alley <dalley at redhat.com> wrote:

> There's another thing to think about which is that not all dependencies
> plugins will need need will be available as Python packages without
> substantial effort.
>
> Take the RPM plugin as an example.  It's likely to depend on at least 3
> native libraries once we reach feature parity with Pulp 2.
>
>    - createrepo_c
>    - libsolv
>    - libmodulemd
>
> Of these:
>
>    - createrepo_c can be installed via Python package, but you have to
>    install the build dependencies on your local machine because there's a
>    compilation step that happens behind the scenes.  We could make pre-built
>    wheels available which would eliminate that issue, but we would need to set
>    up (and maintain) a pipeline to do this because upstream isn't likely to be
>    willing to do so themselves.
>    - Libsolv is not available as a Python package, you have to install
>    the RPM packages "libsolv" and "python{2|3}-solv".  "libsolv" is available
>    on Debian/Ubuntu also but I don't think that the Python bindings for it
>    are.  I'm not 100% sure about this.
>    - Libmodulemd is not available as a Python package, nor is it
>    available on Debian/Ubuntu at all.  It's only available in the RPM
>    ecosystem.
>
> If we're going for primarily container-based installation anyways using a
> Fedora base image, it would probably be substantially easier to provide
> these things via RPM than to make every single dependency installable from
> PyPI.  We'd would probably be the only users of those packaging methods,
> and hence, likely need to maintain it ourselves...
>

If these packages are available already as RPMs, from the distributions
repositories what would require us to do more work than simply install the
RPM packages inside the container?

>
>
> On Mon, Jan 21, 2019 at 2:47 PM Brian Bouterse <bbouters at redhat.com>
> wrote:
>
>> Thank you for this. I put some responses inline. Does it make any sense
>> to put this well written proposal (and its updates over time) into a wiki
>> page?  https://pulp.plan.io/projects/pulp/wiki/
>>
>>
>> On Fri, Jan 11, 2019 at 1:27 PM Eric Helms <ehelms at redhat.com> wrote:
>>
>>> Howdy,
>>>
>>> A few months back, the team that handles RPM packaging for Pulp 2 (my
>>> team) and the Pulp team got together to discuss production 'packaging'
>>> support for Pulp 3. Where we landed was to take a container native approach
>>> from the outset and provide installation support via the Ansible installer
>>> for Pulp 3. This RFC aims to outline the general strategy, link to existing
>>> work and aim towards next steps.
>>>
>>> Note: We do not plan to provide RPMs at this time in favor of a
>>> container native solution for deploying Pulp 3. If the user demand becomes
>>> high enough, we'll look to discuss and collaborate on how to move forward.
>>>
>>> Proposal
>>>
>>> The proposal is to package Pulp 3 via containers using pypi assets as
>>> the de-facto input source for building the container image(s). The image(s)
>>> would be stored on https://quay.io and initially provided with support
>>> through the ansible-pulp3 installer. The installer would pull and run the
>>> images via systemd using podman as the default with support for Docker.
>>> This will provide users with a singular interface for deploying and
>>> managing Pulp 3 no matter the choice of installation (e.g. source, pypi,
>>> containers). In the future, existing work to deploy on to Kubernetes would
>>> be brought into the installer to support deploying the containers on top of
>>> Kubeneretes or Openshift.
>>>
>> This sounds consistent with the discussions at PulpCon. +1
>>
>>>
>>> Given Pulp is a plugin based ecosystem, and images are immutable plugins
>>> need to be handled in a user configurable way. In this space, the original
>>> proposal was to provide enough build tooling and documentation to allow
>>> users to re-build Pulp images with any combination of plugins they require.
>>> The Pulp published image(s) would include a base image and at least one
>>> image with a specific combination of plugins provided. This is an area that
>>> has had much discussion, and ideas. Ideas such as images that deploy plugin
>>> code to a shared volume, sidecar containers that provide plugins at
>>> runtime, a single image with all plugins and runtime configuration and the
>>> Discourse model of rebuilding the image locally when a plugin is added via
>>> a commandline.
>>>
>> I think having all the software built into one image would be the
>> simplest. Making it really easy for a user to pick what they want, and to
>> get that rebuilt over time would be valuable.
>>
>>>
>>> Existing Work
>>>
>>> The existing work in this area is around two pull requests that have
>>> provided a working proof of concept. The first is the image itself [1] and
>>> the second is to the installer [2] that makes use of this image.
>>>
>>> The current works' biggest design assumption (up for debate) is around a
>>> single image vs. an image per service. The strategy difference is around a
>>> single image that can, with the right start command, take on the role of
>>> any Pulp 3 service. The multi-image strategy would aim to ensure there is
>>> an image per service that knows how to properly run itself. A single image
>>> provides a simpler build strategy to start with potentially more confusion
>>> to users wanting to use them within their own context.
>>>
>> If a more complicated build system makes for a simpler deployment
>> experience I think that's a win, so I'm +1 for the multi-image approach. If
>> it's too much to do right now I would understand that too.
>>
>>
>>> Open Questions
>>>
>>>  * What are the communities thoughts on this approach?
>>>
>>  * What are the gaps to get both [1] and [2] into the respective code
>>> bases?
>>>
>> I will look at reviewing these today/tomorrow ot give some initial
>> feedback
>>
>>>  * Should we stick with a single image strategy or opt to pivot to
>>> multi-image?
>>>
>> +1 for multi, but that is my opinion
>>
>>  * What changes need to be made to selinux policies?
>>>
>> We have no selinux policy atm so none. When we need one we can do that.
>>
>>  * How is plugin support handled?
>>>
>> This ticket epic describes that each plugin could choose to ship a
>> container w/ just their plugin code on top of core. Each plugin could add
>> that to the "master" pipeline that core maintains via PRs, which sounds
>> sustainable. Another option is to have each plugin "replicate" the code
>> from [1] into their own repos. That sounds unmaintainable. Suggests are
>> welcome.
>>
>>>
>>> Next Steps
>>>
>>> To move forward, I'll be looking for a Pulp maintainer to help sponsor
>>> and shepherd this work into the Pulp ecosystem. Along with that a rough
>>> list:
>>>
>>>  * Resolve open questions
>>>
>>  * Create Pulp account on quay.io
>>>
>> I made  a pulp account for us to ship the images with.
>>
>>  * Add deployment support to Travis configuration
>>>
>> Let's IRC collab so I can make the encrypted Travis secret for your PR.
>> Where in the Travis config does it go?
>>
>>  * Get initial installer work merged
>>>
>> Yes, let's. I'm going to look at it.
>>
>>
>>> [1] https://github.com/pulp/pulp/pull/3740
>>> [2] https://github.com/pulp/ansible-pulp3/pull/45
>>> _______________________________________________
>>> Pulp-dev mailing list
>>> Pulp-dev at redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>> _______________________________________________
>> Pulp-dev mailing list
>> Pulp-dev at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20190212/5257b3f1/attachment.htm>