<?xml version="1.0" encoding="utf-8" ?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <meta name="generator" content="Docutils 0.14: http://docutils.sourceforge.net/" /> <title>Proposal to Consolidate the Pulp 3 Installers</title> <style type="text/css"> /* :Author: David Goodger (goodger@python.org) :Id: $Id: html4css1.css 7952 2016-07-26 18:15:59Z milde $ :Copyright: This stylesheet has been placed in the public domain. Default cascading style sheet for the HTML output of Docutils. See http://docutils.sf.net/docs/howto/html-stylesheets.html for how to customize this style sheet. */ /* used to remove borders from tables and images */ .borderless, table.borderless td, table.borderless th { border: 0 } table.borderless td, table.borderless th { /* Override padding for "table.docutils td" with "! important". The right padding separates the table cells. */ padding: 0 0.5em 0 0 ! important } .first { /* Override more specific margin styles with "! important". */ margin-top: 0 ! important } .last, .with-subtitle { margin-bottom: 0 ! important } .hidden { display: none } .subscript { vertical-align: sub; font-size: smaller } .superscript { vertical-align: super; font-size: smaller } a.toc-backref { text-decoration: none ; color: black } blockquote.epigraph { margin: 2em 5em ; } dl.docutils dd { margin-bottom: 0.5em } object[type="image/svg+xml"], object[type="application/x-shockwave-flash"] { overflow: hidden; } /* Uncomment (and remove this text!) to get bold-faced definition list terms dl.docutils dt { font-weight: bold } */ div.abstract { margin: 2em 5em } div.abstract p.topic-title { font-weight: bold ; text-align: center } div.admonition, div.attention, div.caution, div.danger, div.error, div.hint, div.important, div.note, div.tip, div.warning { margin: 2em ; border: medium outset ; padding: 1em } div.admonition p.admonition-title, div.hint p.admonition-title, div.important p.admonition-title, div.note p.admonition-title, div.tip p.admonition-title { font-weight: bold ; font-family: sans-serif } div.attention p.admonition-title, div.caution p.admonition-title, div.danger p.admonition-title, div.error p.admonition-title, div.warning p.admonition-title, .code .error { color: red ; font-weight: bold ; font-family: sans-serif } /* Uncomment (and remove this text!) to get reduced vertical space in compound paragraphs. div.compound .compound-first, div.compound .compound-middle { margin-bottom: 0.5em } div.compound .compound-last, div.compound .compound-middle { margin-top: 0.5em } */ div.dedication { margin: 2em 5em ; text-align: center ; font-style: italic } div.dedication p.topic-title { font-weight: bold ; font-style: normal } div.figure { margin-left: 2em ; margin-right: 2em } div.footer, div.header { clear: both; font-size: smaller } div.line-block { display: block ; margin-top: 1em ; margin-bottom: 1em } div.line-block div.line-block { margin-top: 0 ; margin-bottom: 0 ; margin-left: 1.5em } div.sidebar { margin: 0 0 0.5em 1em ; border: medium outset ; padding: 1em ; background-color: #ffffee ; width: 40% ; float: right ; clear: right } div.sidebar p.rubric { font-family: sans-serif ; font-size: medium } div.system-messages { margin: 5em } div.system-messages h1 { color: red } div.system-message { border: medium outset ; padding: 1em } div.system-message p.system-message-title { color: red ; font-weight: bold } div.topic { margin: 2em } h1.section-subtitle, h2.section-subtitle, h3.section-subtitle, h4.section-subtitle, h5.section-subtitle, h6.section-subtitle { margin-top: 0.4em } h1.title { text-align: center } h2.subtitle { text-align: center } hr.docutils { width: 75% } img.align-left, .figure.align-left, object.align-left, table.align-left { clear: left ; float: left ; margin-right: 1em } img.align-right, .figure.align-right, object.align-right, table.align-right { clear: right ; float: right ; margin-left: 1em } img.align-center, .figure.align-center, object.align-center { display: block; margin-left: auto; margin-right: auto; } table.align-center { margin-left: auto; margin-right: auto; } .align-left { text-align: left } .align-center { clear: both ; text-align: center } .align-right { text-align: right } /* reset inner alignment in figures */ div.align-right { text-align: inherit } /* div.align-center * { */ /* text-align: left } */ .align-top { vertical-align: top } .align-middle { vertical-align: middle } .align-bottom { vertical-align: bottom } ol.simple, ul.simple { margin-bottom: 1em } ol.arabic { list-style: decimal } ol.loweralpha { list-style: lower-alpha } ol.upperalpha { list-style: upper-alpha } ol.lowerroman { list-style: lower-roman } ol.upperroman { list-style: upper-roman } p.attribution { text-align: right ; margin-left: 50% } p.caption { font-style: italic } p.credits { font-style: italic ; font-size: smaller } p.label { white-space: nowrap } p.rubric { font-weight: bold ; font-size: larger ; color: maroon ; text-align: center } p.sidebar-title { font-family: sans-serif ; font-weight: bold ; font-size: larger } p.sidebar-subtitle { font-family: sans-serif ; font-weight: bold } p.topic-title { font-weight: bold } pre.address { margin-bottom: 0 ; margin-top: 0 ; font: inherit } pre.literal-block, pre.doctest-block, pre.math, pre.code { margin-left: 2em ; margin-right: 2em } pre.code .ln { color: grey; } /* line numbers */ pre.code, code { background-color: #eeeeee } pre.code .comment, code .comment { color: #5C6576 } pre.code .keyword, code .keyword { color: #3B0D06; font-weight: bold } pre.code .literal.string, code .literal.string { color: #0C5404 } pre.code .name.builtin, code .name.builtin { color: #352B84 } pre.code .deleted, code .deleted { background-color: #DEB0A1} pre.code .inserted, code .inserted { background-color: #A3D289} span.classifier { font-family: sans-serif ; font-style: oblique } span.classifier-delimiter { font-family: sans-serif ; font-weight: bold } span.interpreted { font-family: sans-serif } span.option { white-space: nowrap } span.pre { white-space: pre } span.problematic { color: red } span.section-subtitle { /* font-size relative to parent (h1..h6 element) */ font-size: 80% } table.citation { border-left: solid 1px gray; margin-left: 1px } table.docinfo { margin: 2em 4em } table.docutils { margin-top: 0.5em ; margin-bottom: 0.5em } table.footnote { border-left: solid 1px black; margin-left: 1px } table.docutils td, table.docutils th, table.docinfo td, table.docinfo th { padding-left: 0.5em ; padding-right: 0.5em ; vertical-align: top } table.docutils th.field-name, table.docinfo th.docinfo-name { font-weight: bold ; text-align: left ; white-space: nowrap ; padding-left: 0 } /* "booktabs" style (no vertical lines) */ table.docutils.booktabs { border: 0px; border-top: 2px solid; border-bottom: 2px solid; border-collapse: collapse; } table.docutils.booktabs * { border: 0px; } table.docutils.booktabs th { border-bottom: thin solid; text-align: left; } h1 tt.docutils, h2 tt.docutils, h3 tt.docutils, h4 tt.docutils, h5 tt.docutils, h6 tt.docutils { font-size: 100% } ul.auto-toc { list-style-type: none } </style> </head> <body> <div class="document" id="proposal-to-consolidate-the-pulp-3-installers"> <h1 class="title">Proposal to Consolidate the Pulp 3 Installers</h1> <div class="contents topic" id="contents"> Contents <ul class="simple"> <li><a class="reference internal" href="#what" id="id3">What?</a></li> <li><a class="reference internal" href="#why" id="id4">Why?</a><ul> <li><a class="reference internal" href="#better-installer-less-work" id="id5">Better Installer, Less Work</a></li> <li><a class="reference internal" href="#improve-pulp-s-public-image" id="id6">Improve Pulp's Public Image</a></li> <li><a class="reference internal" href="#encourage-community-involvement" id="id7">Encourage Community Involvement</a></li> <li><a class="reference internal" href="#more-installation-strategies" id="id8">More Installation Strategies</a></li> </ul> </li> <li><a class="reference internal" href="#how" id="id9">How?</a><ul> <li><a class="reference internal" href="#technology" id="id10">Technology</a></li> <li><a class="reference internal" href="#workflow-example" id="id11">Workflow Example</a></li> <li><a class="reference internal" href="#sample-playbooks" id="id12">Sample Playbooks</a></li> <li><a class="reference internal" href="#sample-role" id="id13">Sample Role</a></li> <li><a class="reference internal" href="#why-one-variable" id="id14">Why One Variable?</a></li> <li><a class="reference internal" href="#implementation-strategy" id="id15">Implementation Strategy</a><ul> <li><a class="reference internal" href="#what-code-sees-development-work" id="id16">What Code Sees Development Work?</a></li> <li><a class="reference internal" href="#what-roles-are-produced" id="id17">What Roles Are Produced?</a></li> <li><a class="reference internal" href="#what-about-vagrant" id="id18">What About Vagrant?</a></li> </ul> </li> </ul> </li> <li><a class="reference internal" href="#summary" id="id19">Summary</a></li> </ul> </div> <div class="section" id="what"> <h1><a class="toc-backref" href="#id3">What?</a></h1> Currently, one installer is used by the Pulp developers, and another installer is used by Pulp QE and end users. I propose that a single installer be used by all three audiences. This document explains why I think this change should be made, and it outlines a possible plan of action for making this change. </div> <div class="section" id="why"> <h1><a class="toc-backref" href="#id4">Why?</a></h1> Why should a single installer be used? The most succinct justification is that using a single installer improves <a class="reference external" href="https://12factor.net/dev-prod-parity">dev/prod parity</a>, where dev/prod parity means "[keeping] development, staging, and production as similar as possible." In our case, QE serves as staging, and our users serve as production. More verbosely, a single installer should be used because doing so will likely: <ol class="arabic simple"> <li>Produce a better installer.</li> <li>Reduce the workload on developers and quality engineers.</li> <li>Improve Pulp's public image.</li> <li>Encourage community involvement.</li> <li>Allow for additional installation strategies.</li> </ol> <div class="section" id="better-installer-less-work"> <h2><a class="toc-backref" href="#id5">Better Installer, Less Work</a></h2> To illustrate the first two points, consider the scenario where a quality engineer discovers an issue with the installer they're using. The engineer is obliged to check and see if the issue is also present in the second installer. If so, the engineer will either file one ticket that mentions both installers, or they will file two independent tickets, depending on their mindset. When a developer decides to work on the issues, they need to develop two similar fixes for two similar installers, they need to test both changes, and they need to file two pull requests. After the pull requests have been merged, the developer needs to either close out one or two tickets. This workflow is logically sound. So long as all parties are mindful of what they're doing, nothing will go wrong. But humans aren't like that! Consider the case of <a class="reference external" href="https://pulp.plan.io/issues/3031">Pulp #3031</a>. That ticket describes how the end user installer inserts junk text into <tt class="docutils literal">/etc/pulp/server.yaml</tt>. I filed this ticket on September 25th, and I submitted a fix for the issue on the same day. Unfortunately, I forgot to file a second ticket noting that the exact same issue is also present in the developer installer. As of this writing on November 21st, the developer installer is affected by this issue. The developer installer still works due to the particular nature of the bug, but at any time, an unrelated change to the configuration file could break the development environment. </div> <div class="section" id="improve-pulp-s-public-image"> <h2><a class="toc-backref" href="#id6">Improve Pulp's Public Image</a></h2> To illustrate the third point, consider what happened when the Pulp 3 plugin API alpha was released. The announcement was pushed back by a day or two so that QE could have a chance to play with the product before it went out the door. QE discovered several issues, one of them being the fact that the end user installer simply did not work. (The issue was fixed by <a class="reference external" href="https://github.com/pulp/devel/commit/396c93e775542484b585b8841bc9adc042b71731">pulp/devel 396c93e775542484b585b8841bc9adc042b71731</a>.) This gives the impression that the end user installer is a second class product. So long as we have two non-orthogonal installers, Pulp's public image is vulnerable to incidents like this. </div> <div class="section" id="encourage-community-involvement"> <h2><a class="toc-backref" href="#id7">Encourage Community Involvement</a></h2> To illustrate the fourth point, consider some of the differences between the end user and developer installers. Using the end user installer requires the following: <ol class="arabic"> <li>Decide where you would like to install Pulp. You could install it on your local system, on a beaker system, on a libvirt + kvm VM, on a libvirt + virtualbox VM, on a Linode VM, on a DigitalOcean VM, or just about anywhere else. </li> <li>Get shell access to the system and execute the following: <pre class="code sh literal-block"> setenforce 0 systemctl stop firewalld # or open appropriate ports </pre> </li> <li>From your local system, execute the following: <pre class="code sh literal-block"> git clone https://github.com/pulp/devel.git --branch 3.0-dev cd devel/ansible ansible-galaxy install -r requirements.yml -p roles ansible-galaxy deploy-pulp3.yml -i pulp3.example.com, </pre> </li> </ol> In contrast, using the developer installer requires the following: <ol class="arabic"> <li>Set up Vagrant. A development environment can't be deployed on any other type of host. You're SOL if you'd like to use some other type of back-end. </li> <li>From your local system, execute <a class="footnote-reference" href="#id2" id="id1">[1]</a> the following: <pre class="code sh literal-block"> git clone https://github.com/pulp/devel.git --branch 3.0-dev cd devel vagrant up </pre> </li> </ol> Those familiar with Pulp's installers might also notice that the end user installer doesn't install any plug-ins, whereas the developer installer installs the file plugin. Why should a prospective contributor need to learn two different installation work-flows, where those two work-flows require entirely different sets of infrastructure, and where the two work-flows install different Pulp 3 components? </div> <div class="section" id="more-installation-strategies"> <h2><a class="toc-backref" href="#id8">More Installation Strategies</a></h2> The developer installer installs Pulp directly from source in editable mode, along with numerous extra tools, and it enables settings like debug mode. The end user installer installs Pulp from PyPI packages, without extra tools, and without enabling settings like debug mode. Unfortunately, these two installers don't cover all of our needs. PyPI packages are — quite reasonably — updated once a week. As Pulp 3 matures, these releases may become less frequent. As a result, our Jenkins test runs may fail to detect regressions for a up to a week, and tests for fixed-but-not-released features may produce incorrect results for a up to a week. One solution to this quandary is to create a third installer that installs Pulp directly from source, without extra tools, and without enabling settings like debug mode. As an obvious example, consider <a class="reference external" href="https://pulp.plan.io/issues/3107">Pulp #3107</a>: <blockquote> Switch JWT auth token string from 'JWT' to 'Bearer'</blockquote> I had to explicitly request that new PyPI packages be uploaded after this change was made. In the future, requests like this will occur again. If a from-source installer is available, then such requests are obviated. How can we support this third installation use case with our current set-up? Should we create a third installer? That makes every other problem listed above even worse. It would be much better if we took an existing installer and added an option so that it could fulfill either use case. If we're going to do that, why not go even further and let an existing installer handle all three use cases? </div> </div> <div class="section" id="how"> <h1><a class="toc-backref" href="#id9">How?</a></h1> In concrete terms, how could the installer described above be implemented? <div class="section" id="technology"> <h2><a class="toc-backref" href="#id10">Technology</a></h2> Currently, both of our installers use Ansible. I suggest that we continue using Ansible to distribute Pulp. There are several key advantages to doing this: <ul class="simple"> <li>We can re-use some of our existing Ansible assets when developing a converged installer.</li> <li>Ansible is an official Red Hat product. Using Ansible may promote adoption of Pulp 3 within the business.</li> <li>Ansible is platform-agnostic. If someone wants to install Pulp on an unsupported platform (such as CentOS, Debian, an old version of Fedora, etc.), they can do so with little-to-no effort.</li> <li>It is hard to imagine a scenario where an RPM-based installer could produce development, from-source, or from-PyPI Pulp installations.</li> </ul> It is important to recognize that native package managers offer significant benefits, and those benefits are orthogonal to the benefits provided by Ansible. In the future, we may decide to offer native packages in addition to or instead of an Ansible-based installer. </div> <div class="section" id="workflow-example"> <h2><a class="toc-backref" href="#id11">Workflow Example</a></h2> How would such an installer work from an end user's perspective? Before going any further, let's recap some of the workflow-related problems being solved: <ul class="simple"> <li>As described in <a class="reference internal" href="#encourage-community-involvement">Encourage Community Involvement</a>, users need to learn two different installation work-flows, where those two work-flows require entirely different sets of infrastructure, and where the two work-flows install different Pulp 3 components.</li> <li>As described in <a class="reference internal" href="#more-installation-strategies">More Installation Strategies</a>, we currently support from-source developer-oriented installs and from-PyPI user-oriented installs, but we don't offer from-source user-oriented installs.</li> </ul> From an end user's perspective, a work-flow like the following would appear to solve all of these complaints: <pre class="code sh literal-block"> git clone https://github.com/pulp/devel.git --branch 3.0-dev cd devel/ansible ansible-galaxy dev-install.yml -i pulp-dev.example.com, ansible-galaxy src-install.yml -i pulp-src.example.com, ansible-galaxy pypi-install.yml -i pulp-pypi.example.com, </pre> </div> <div class="section" id="sample-playbooks"> <h2><a class="toc-backref" href="#id12">Sample Playbooks</a></h2> While this looks great from an end user's perspective, it's unclear how this would work internally. Thus, the next question is "how would this work from a developer's perspective?" To answer this question, let's take a top-down approach, starting with the playbooks. What would <tt class="docutils literal">pypi-install.yml</tt> look like? It could look like this: <pre class="code yaml literal-block"> --- # Install Pulp 3 from PyPI packages. # # The resultant installation is designed to be useful for end users. It # doesn't do things like install code in editable mode, install extra # dependencies or install Bash aliases. # # SELinux is disabled because Pulp 3 is currently incompatible with SELinux. - hosts: all become: true pre_tasks: - name: Disable selinux selinux: state: disabled roles: - 'snip!' </pre> And what would <tt class="docutils literal">dev-install.yml</tt> look like? It could look like this: <pre class="code yaml literal-block"> --- # Install Pulp 3 from locally cloned repositories. # # The resultant installation is designed to be useful for developers. It # does things like install code in editable mode, install extra dependencies # and install Bash aliases. # # SELinux is disabled because Pulp 3 is currently incompatible with SELinux. - hosts: all become: true vars: pulp_install_strategy: dev pre_tasks: - name: Disable selinux selinux: state: disabled roles: - 'snip!' </pre> The only difference between the two scripts is the addition of one variable: <pre class="code yaml literal-block"> vars: pulp_install_strategy: dev </pre> With this in mind, you can guess what <tt class="docutils literal">src-install.yml</tt> could look like: <pre class="code yaml literal-block"> --- # Install Pulp 3 from remote repositories. # # The resultant installation is designed to be useful for end users. It # doesn't do things like install code in editable mode, install extra # dependencies or install Bash aliases. # # SELinux is disabled because Pulp 3 is currently incompatible with SELinux. - hosts: all become: true vars: pulp_install_strategy: source pre_tasks: - name: Disable selinux selinux: state: disabled roles: - 'snip!' </pre> Note that SELinux is explicitly disabled in the playbooks, instead of within a role. This isn't necessary at a technical level, and in fact, doing so violates the DRY principle. It is done because disabling SELinux is a dangerous action that all users should be aware of, and placing this action in the playbooks makes it more visible. </div> <div class="section" id="sample-role"> <h2><a class="toc-backref" href="#id13">Sample Role</a></h2> Now that you know what the playbooks could look like, let's dive down another layer and look at roles. What effect does the <tt class="docutils literal">pulp_install_strategy</tt> variable have on roles? And what does code using this variable look like? The first question can be answered without looking at the documentation for a role. (It's good practice to place a <tt class="docutils literal">README.md</tt> file at the top level of a role, and the variables accepted by a role may be documented there.) Here's the documentation that could exist for a hypothetical "pulpcore" role. (I'm not advocating for a role by this name to exist. This is for learning purposes.) <pre class="code md literal-block"> pulpcore ======== This role installs the pulpcore component of Pulp 3. It also creates and customizes `/etc/pulp/server.yaml`. NOTE: `SECRET_KEY` is updated even if a value is already present! Example Usage ------------- ```yaml - hosts: all roles: - dev ``` Variables --------- This role depends on the variables exported by the `pulp-user` role, and on the `pulp_install_strategy` variable. If `pulp_install_strategy` is `dev`, then debug mode is enabled in `/etc/pulp/server.yaml`. Otherwise, debug mode is disabled. </pre> What might the actual code using <tt class="docutils literal">pulp_install_strategy</tt> look like? As an example, consider the scenario where the role discussed above wishes to enable or disable debug mode. Here's what the relevant code might look like: <pre class="code yaml literal-block"> - name: Configure DEBUG mode in server.yaml lineinfile: path: /etc/pulp/server.yaml regexp: '^DEBUG: ' line: 'DEBUG: True' state: '{{ "present" if pulp_install_strategy == "development" else "absent" }}' </pre> Or, imagine the scenario in which the role needs to find <tt class="docutils literal">server.yaml</tt> in Pulp's source code, so that it can be copied to <tt class="docutils literal">/etc/pulp/server.yaml</tt>. Depending on how Pulp was installed, the file may exist in one of several places. Here's what the relevant code might look like: <pre class="code yaml literal-block"> - name: Find server.yaml find: paths: '{{ pulp_devel_dir if pulp_install_strategy == "dev" else pulp_venv }}' recurse: true patterns: server.yaml register: result - name: Assert one server.yaml file has been found assert: that: 'result["files"] | length == 1' </pre> Or, imagine the scenario in which a large chunk of code should only be run for a single installation strategy. An elegant way to isolate it is: <pre class="code yaml literal-block"> - include: '{{ pulp_install_strategy }}.yml' </pre> </div> <div class="section" id="why-one-variable"> <h2><a class="toc-backref" href="#id14">Why One Variable?</a></h2> Why should just one variable be used to support these installation strategies? Why not use several variables, where one controls whether <tt class="docutils literal">~/.bashrc</tt> is installed, another controls whether supplemental development tools are installed, and so on? The reason to do this is to reduce combinatorial complexity. If just one argument is accepted by the installer, then the number of installation scenarios grows linearly with the number of valid values for that argument. That is, if there are three valid values for <tt class="docutils literal">pulp_install_strategy</tt>, then there are three installation scenarios to support. This makes development and testing simpler, which should improve the quality of the installer. More options can be added if necessary, of course. But starting with a simple design seems like a good idea. One might counter that this strategy greatly limits the options available to end users. There are many ways that a user might want to install Pulp, and by offering such a limiting installer, we're limiting how users can deploy Pulp. This is sort of true. But the purpose of an installer isn't to provide an infinitely flexible platform for developing new and exciting things. The purpose of an installer is to produce a specific, known-working application installation. If a Pulp user wants the ultimate in flexibility, that's what the source code is for. </div> <div class="section" id="implementation-strategy"> <h2><a class="toc-backref" href="#id15">Implementation Strategy</a></h2> Which set(s) of source code should see development work? What Ansible playbooks be produced? <div class="section" id="what-code-sees-development-work"> <h3><a class="toc-backref" href="#id16">What Code Sees Development Work?</a></h3> How can we implement such an installer while minimizing the risk to developers, QE, and most importantly, our end users? I see at least three options: <ul class="simple"> <li>Evolve the end user installer.</li> <li>Implement a brand new installer.</li> <li>Evolve the developer installer.</li> </ul> Our users are our most important assets. Without them, Pulp has no rationale for existing. Consequently, we should take steps to protect them from any mistakes that we might make. If anyone is to suffer through implementation errors and development dead-ends, it should be the development team and QE, not our end users. Thus, I think that the option of evolving the end user installer should be ruled out. Implementing an installer from scratch has one big benefit: it lets us perform development work without impacting any existing users at all. But there are big downsides to this approach as well. The biggest is that work on our existing installers will proceed while a new installer is developed. As a result, we will be stuck maintaining not two but three installers for some period of time. Many of the issues described in the <a class="reference internal" href="#why">Why?</a> section will be made even worse. Furthermore, it's quite possible that work on a new installer could stretch on for much longer than expected, perhaps due to unexpected changes in priorities. In addition, this implementation strategy means that more targets need to be tracked during development. Whoever is working on a new installer from scratch needs to keep an eye on both of the other installers, to ensure that they aren't missing out on any important implementation details or features. This "tracking" effort is troublesome and should be minimized. I think that evolving the developer installer is the best option, because it avoid both of the severe problems identified above. We don't run the risk of breaking our end users workflows (until work is done), and we don't run the risk of ending up with three installers. It is true that developers and QE may encounter breakage, but these are the audiences that are best prepared to deal with breakage. </div> <div class="section" id="what-roles-are-produced"> <h3><a class="toc-backref" href="#id17">What Roles Are Produced?</a></h3> Assuming that the existing developer installer should be evolved, what exactly would that look like? Are there any especially interesting challenges that would need to be solved? The most interesting challenge I foresee is the question of which roles should be created. Currently, the developer installer calls the following roles, in this order: <ul class="simple"> <li><tt class="docutils literal">pulp-user</tt></li> <li><tt class="docutils literal">db</tt></li> <li><tt class="docutils literal">broker</tt></li> <li><tt class="docutils literal">lazy</tt></li> <li><tt class="docutils literal">dev</tt></li> <li><tt class="docutils literal">plugin</tt></li> <li><tt class="docutils literal">systemd</tt></li> <li><tt class="docutils literal">dev_tools</tt></li> </ul> One option is to re-work the installer so that it calls the same roles as the end user installer. The end user installer calls the following roles, in this order: <ul class="simple"> <li><tt class="docutils literal">pulp3</tt></li> <li><tt class="docutils literal">pulp3_db</tt></li> <li><tt class="docutils literal">rabbitmq</tt></li> <li><tt class="docutils literal">pulp3_systemd</tt></li> </ul> I have two objections to this strategy. First, I object to using the <tt class="docutils literal">rabbitmq</tt> role, when we could use packages provided by the system package manager instead. <ul class="simple"> <li>Using an Ansible role means that we're unnecessarily pushing a particular RabbitMQ implementation and configuration on to our users, instead of respecting the differences between distributions. I imagine that the implementation of RabbitMQ differs between RHEL 7, Fedora 26 Workstation, Fedora 26 Server, Debian, Ubuntu, and so on, and the package maintainers for those distributions are in a better position to understand how RabbitMQ best fits into their particular ecosystem. This has already caused problems, in that the <tt class="docutils literal">rabbitmq</tt> role configures RabbitMQ to listen on a non-standard port by default.</li> <li>We know that packages provided by the system package manager work well. The developer installer is proof of this: it uses packages from the system package manager.</li> <li>Packages provided by the package manager are in a position to receive more quality assurance than a third-party Ansible role. This is one of the value propositions of RHEL. We should take advantage of it.</li> <li>Dropping the <tt class="docutils literal">rabbitmq</tt> role would let us also drop the <tt class="docutils literal">config_encoder_filter</tt> role. At that point, we would have no third-party role dependencies.</li> </ul> My second objection to this strategy is more general: I think that we should start with a simple, monolithic installer before getting clever and trying to draw lines between components. Consider: <ul class="simple"> <li>Drawing lines between components makes development costlier. With the current end user installer, developers must submit pull requests against as many as three repositories in order to implement a functional change in the behaviour of the installer, and four repositories when adjusting documentation too. This is a drag. It would be better if these costs are taken on only when a known problem is being solved.</li> <li>It seems far smarter to draw lines between components based on real world experience than based on conjecture.</li> <li>The Pulp developers have not yet decided on how Pulp should be deployed. Think of questions like "which web servers are supported, and how should they communicate with the back-end Pulp services?" It's far easier to respond to these changes in Pulp's implementation with a monolithic architecture.</li> </ul> Instead, what I would recommend is that we merge the various roles in the developer installer, until we're left with just one highly polished role that can perform "dev," "src," and "pypi" installations. These three installation strategies would serve the needs of developers, QE and end users, respectively. We could then move this role into the existing <a class="reference external" href="https://github.com/pulp/ansible-pulp3">pulp/ansible-pulp3</a> role, replacing the existing code in that repository. </div> <div class="section" id="what-about-vagrant"> <h3><a class="toc-backref" href="#id18">What About Vagrant?</a></h3> I see one other interesting challenge: the developer installer deploys code into a user account named "vagrant," and the end user installer deploys code into a user account named "pulp." This difference needs to be accounted for. We could solve this by parametrizing the installer, so that a configurable user may be used for deployments. However, this has two downsides: <ul class="simple"> <li>As described in <a class="reference internal" href="#why-one-variable">Why One Variable?</a>, this increases combinatorial complexity.</li> <li>This decresases <a class="reference external" href="https://12factor.net/dev-prod-parity">dev/prod parity</a>.</li> </ul> Instead, I would recommend always using a user named "pulp" for deployments, and adjusting the Vagrant file so that a user named "pulp" is used. </div> </div> </div> <div class="section" id="summary"> <h1><a class="toc-backref" href="#id19">Summary</a></h1> At a high level, this is the proposal: <ol class="arabic"> <li>Continue distributing Pulp 3 with Ansible. </li> <li>Enable the following use case: <pre class="code sh literal-block"> git clone https://github.com/pulp/devel.git --branch 3.0-dev cd devel/ansible ansible-galaxy dev-install.yml -i pulp-dev.example.com, ansible-galaxy src-install.yml -i pulp-src.example.com, ansible-galaxy pypi-install.yml -i pulp-pypi.example.com, </pre> </li> <li>Get there by evolving the developer installer. </li> </ol> <table class="docutils footnote" frame="void" id="id2" rules="none"> <colgroup><col class="label" /><col /></colgroup> <tbody valign="top"> <tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>I think this is how to use the developer installer, but I'm not sure. I've found that Vagrant frequently does not work with KVM as a back-end, and when it does work, it interferes with my already-working virtualization solution. As a result, I've never been able to create a working development environment.</td></tr> </tbody> </table> </div> </div> </body> </html>