[Pulp-dev] Providing Pulp 3 Container Images and Installation Support

Wed Feb 13 14:23:58 UTC 2019

Well, that's my point :)

I was just trying to make it clear that we might need to install more than
just PyPI assets, and that might influence the design of the ansible
installer, and also the ansible installer might not be able to install
every plugin on every OS we "support", which I know is something we were
hoping to be able to do.

On Tue, Feb 12, 2019 at 8:58 PM Eric Helms <ehelms at redhat.com> wrote:

>
>
> On Tue, Feb 12, 2019 at 1:00 PM Daniel Alley <dalley at redhat.com> wrote:
>
>> There's another thing to think about which is that not all dependencies
>> plugins will need need will be available as Python packages without
>> substantial effort.
>>
>> Take the RPM plugin as an example.  It's likely to depend on at least 3
>> native libraries once we reach feature parity with Pulp 2.
>>
>>    - createrepo_c
>>    - libsolv
>>    - libmodulemd
>>
>> Of these:
>>
>>    - createrepo_c can be installed via Python package, but you have to
>>    install the build dependencies on your local machine because there's a
>>    compilation step that happens behind the scenes.  We could make pre-built
>>    wheels available which would eliminate that issue, but we would need to set
>>    up (and maintain) a pipeline to do this because upstream isn't likely to be
>>    willing to do so themselves.
>>    - Libsolv is not available as a Python package, you have to install
>>    the RPM packages "libsolv" and "python{2|3}-solv".  "libsolv" is available
>>    on Debian/Ubuntu also but I don't think that the Python bindings for it
>>    are.  I'm not 100% sure about this.
>>    - Libmodulemd is not available as a Python package, nor is it
>>    available on Debian/Ubuntu at all.  It's only available in the RPM
>>    ecosystem.
>>
>> If we're going for primarily container-based installation anyways using a
>> Fedora base image, it would probably be substantially easier to provide
>> these things via RPM than to make every single dependency installable from
>> PyPI.  We'd would probably be the only users of those packaging methods,
>> and hence, likely need to maintain it ourselves...
>>
>
> If these packages are available already as RPMs, from the distributions
> repositories what would require us to do more work than simply install the
> RPM packages inside the container?
>
>
>>
>>
>> On Mon, Jan 21, 2019 at 2:47 PM Brian Bouterse <bbouters at redhat.com>
>> wrote:
>>
>>> Thank you for this. I put some responses inline. Does it make any sense
>>> to put this well written proposal (and its updates over time) into a wiki
>>> page?  https://pulp.plan.io/projects/pulp/wiki/
>>>
>>>
>>> On Fri, Jan 11, 2019 at 1:27 PM Eric Helms <ehelms at redhat.com> wrote:
>>>
>>>> Howdy,
>>>>
>>>> A few months back, the team that handles RPM packaging for Pulp 2 (my
>>>> team) and the Pulp team got together to discuss production 'packaging'
>>>> support for Pulp 3. Where we landed was to take a container native approach
>>>> from the outset and provide installation support via the Ansible installer
>>>> for Pulp 3. This RFC aims to outline the general strategy, link to existing
>>>> work and aim towards next steps.
>>>>
>>>> Note: We do not plan to provide RPMs at this time in favor of a
>>>> container native solution for deploying Pulp 3. If the user demand becomes
>>>> high enough, we'll look to discuss and collaborate on how to move forward.
>>>>
>>>> Proposal
>>>>
>>>> The proposal is to package Pulp 3 via containers using pypi assets as
>>>> the de-facto input source for building the container image(s). The image(s)
>>>> would be stored on https://quay.io and initially provided with support
>>>> through the ansible-pulp3 installer. The installer would pull and run the
>>>> images via systemd using podman as the default with support for Docker.
>>>> This will provide users with a singular interface for deploying and
>>>> managing Pulp 3 no matter the choice of installation (e.g. source, pypi,
>>>> containers). In the future, existing work to deploy on to Kubernetes would
>>>> be brought into the installer to support deploying the containers on top of
>>>> Kubeneretes or Openshift.
>>>>
>>> This sounds consistent with the discussions at PulpCon. +1
>>>
>>>>
>>>> Given Pulp is a plugin based ecosystem, and images are immutable
>>>> plugins need to be handled in a user configurable way. In this space, the
>>>> original proposal was to provide enough build tooling and documentation to
>>>> allow users to re-build Pulp images with any combination of plugins they
>>>> require. The Pulp published image(s) would include a base image and at
>>>> least one image with a specific combination of plugins provided. This is an
>>>> area that has had much discussion, and ideas. Ideas such as images that
>>>> deploy plugin code to a shared volume, sidecar containers that provide
>>>> plugins at runtime, a single image with all plugins and runtime
>>>> configuration and the Discourse model of rebuilding the image locally when
>>>> a plugin is added via a commandline.
>>>>
>>> I think having all the software built into one image would be the
>>> simplest. Making it really easy for a user to pick what they want, and to
>>> get that rebuilt over time would be valuable.
>>>
>>>>
>>>> Existing Work
>>>>
>>>> The existing work in this area is around two pull requests that have
>>>> provided a working proof of concept. The first is the image itself [1] and
>>>> the second is to the installer [2] that makes use of this image.
>>>>
>>>> The current works' biggest design assumption (up for debate) is around
>>>> a single image vs. an image per service. The strategy difference is around
>>>> a single image that can, with the right start command, take on the role of
>>>> any Pulp 3 service. The multi-image strategy would aim to ensure there is
>>>> an image per service that knows how to properly run itself. A single image
>>>> provides a simpler build strategy to start with potentially more confusion
>>>> to users wanting to use them within their own context.
>>>>
>>> If a more complicated build system makes for a simpler deployment
>>> experience I think that's a win, so I'm +1 for the multi-image approach. If
>>> it's too much to do right now I would understand that too.
>>>
>>>
>>>> Open Questions
>>>>
>>>>  * What are the communities thoughts on this approach?
>>>>
>>>  * What are the gaps to get both [1] and [2] into the respective code
>>>> bases?
>>>>
>>> I will look at reviewing these today/tomorrow ot give some initial
>>> feedback
>>>
>>>>  * Should we stick with a single image strategy or opt to pivot to
>>>> multi-image?
>>>>
>>> +1 for multi, but that is my opinion
>>>
>>>  * What changes need to be made to selinux policies?
>>>>
>>> We have no selinux policy atm so none. When we need one we can do that.
>>>
>>>  * How is plugin support handled?
>>>>
>>> This ticket epic describes that each plugin could choose to ship a
>>> container w/ just their plugin code on top of core. Each plugin could add
>>> that to the "master" pipeline that core maintains via PRs, which sounds
>>> sustainable. Another option is to have each plugin "replicate" the code
>>> from [1] into their own repos. That sounds unmaintainable. Suggests are
>>> welcome.
>>>
>>>>
>>>> Next Steps
>>>>
>>>> To move forward, I'll be looking for a Pulp maintainer to help sponsor
>>>> and shepherd this work into the Pulp ecosystem. Along with that a rough
>>>> list:
>>>>
>>>>  * Resolve open questions
>>>>
>>>  * Create Pulp account on quay.io
>>>>
>>> I made  a pulp account for us to ship the images with.
>>>
>>>  * Add deployment support to Travis configuration
>>>>
>>> Let's IRC collab so I can make the encrypted Travis secret for your PR.
>>> Where in the Travis config does it go?
>>>
>>>  * Get initial installer work merged
>>>>
>>> Yes, let's. I'm going to look at it.
>>>
>>>
>>>> [1] https://github.com/pulp/pulp/pull/3740
>>>> [2] https://github.com/pulp/ansible-pulp3/pull/45
>>>> _______________________________________________
>>>> Pulp-dev mailing list
>>>> Pulp-dev at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>
>>> _______________________________________________
>>> Pulp-dev mailing list
>>> Pulp-dev at redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20190213/d7111502/attachment.htm>