[Pulp-dev] Announcing the plugin-template change to use containers for CI

[Mike DePaulo]

On Wed, Sep 4, 2019 at 11:14 AM Dennis Kliban <dkliban at redhat.com> wrote:

> On Wed, Sep 4, 2019 at 11:11 AM Mike DePaulo <mikedep333 at redhat.com>
> wrote:
>
>> *Why?*
>> pulp_rpm, and likely other plugins in the future, have C dependencies.
>> They are often difficult to satisfy on Ubuntu's LTS releases that Travis
>> provides. Since we are developing containers and a Kubernetes operator
>> anyway, we decided to migrate plugin-template's Travis CI scripts to using
>> them. These containers are currently based on Fedora 30, but will likely
>> later be based on CentOS 7.7 or 8.0, which contain python 3.6.
>> This is task #5004 <https://pulp.plan.io/issues/5004>.
>>
>> *Overview:*
>> 1. During install.sh, the container image named like "pulp_file" is built
>> according to pulpcore/containers/
>> 2. During install.sh, k3s, the lightweight kubernetes implementation, is
>> installed & configured on the Travis Ubuntu VM, which already has docker.
>> 3. During install.sh, pulp-operator is deployed (up.sh), which in turn
>> deploys the postgres database, redis, and the pulp_* image you built. It
>> spins the single image up 4 separate containers (services/processes):
>> pulp-api, pulp-content, pulp-resource-manager, and pulp-worker .
>> 4. During script.sh, we install some commonly needed testing tools into
>> the pulp-api container by prefixing the install command with $CMD_PREFIX .
>> So long as this container never gets restarted, they will remain installed
>> in it. These tools are not available to the other containers.
>> 5. We use $CMD_PREFIX to run the unit tests
>> 6. From the Travis Ubuntu VM, we run pulp-smash functional tests, build
>> docs, or use bindings, etc.
>> 7. Note that many other pieces of the puzzle were modified, including
>> pulp-smash, to make this integration all work.
>>
>> *Changes you may need to make:*
>> (in addition to the usual plugin-template commands):
>> 1. Ensure that in template_config.yml, "plugin_name" is set to a value
>> like "pulp_file" rather than "file"
>> 2. If you write out ~/.netrc , the pulp application password has been
>> changed from "admin" to "password" (which other pulp test / development
>> code uses.)
>>
>
> You only need to make this change if you are continuing to use
> ansible-pulp to deploy pulp on Travis. Otherwise, the containers built
> using the latest Travis config will have the right password.
>

Actually, I encountered this during pulp_file's docs tests, which did not
use ansible-pulp. Its docs/_scripts/
<https://github.com/pulp/pulp_file/tree/master/docs/_scripts> shell scripts
(which get embedded in the docs) are tested via httpie (`http`), and
therefore used the .netrc.
I suppose you could use ansible-pulp in your manually-written
.travis/{pre,post}*.sh scripts to install pulp on the Travis VM (which is
the container host), but I do not recommend it. The templated .travis/*.sh
scripts would still always deploy it in containers also. And I set things
like PYTHONPATH so you don't need to call ansible-pulp for that specific
reason.

>
>
>> 3. git rm .travis/{playbook,postgres,mariadb}.yml
>>
>
4. Append to PYTHONPATH (do not override it, your plugin's dir is in it. It
is set in .travis/script.sh).

>
>> *Planned improvements include:*
>> 1. Speeding up in the image build, largely through 3.
>> 2. Publishing images to quay.io. #5062 <https://pulp.plan.io/issues/5062>
>> 3. Only building your pulp_* image from scratch if you have a required PR
>> for pulpcore or pulpcore-plugin (or pulp-certguard.) Otherwise, your image
>> will be layered on top of the "pulpcore" image. #5062
>> <https://pulp.plan.io/issues/5062>
>> 4. re-introducing python 3.6 testing alongside 3.7 (Fedora 30 is Python
>> 3.7 based)
>> 5. re-introducing codecov
>> 6. (possible / long-term): Improving how testing packages are installed. #5404
>> <https://pulp.plan.io/issues/5404>
>>
>> *Feedback wanted on:*
>> 1. Any confusing/unclear output.
>> 2. Any reliability issues.
>>
>>
>> --
>>
>> Mike DePaulo
>>
>> He / Him / His
>>
>> Service Reliability Engineer, Pulp
>>
>> Red Hat <https://www.redhat.com/>
>>
>> IM: mikedep333
>>
>> GPG: 51745404
>> <https://www.redhat.com/>
>> _______________________________________________
>> Pulp-dev mailing list
>> Pulp-dev at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>

-- 

Mike DePaulo

He / Him / His

Service Reliability Engineer, Pulp

Red Hat <https://www.redhat.com/>

IM: mikedep333

GPG: 51745404
<https://www.redhat.com/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20190904/9735046a/attachment.htm>