[Pulp-dev] CI/CD - Upgrade Tests

[Fabricio Aguiar]

Actually, I solved the issue with the additional_plugins and I opened 2 PRs:
- https://github.com/pulp/pulpcore/pull/1327
- https://github.com/pulp/plugin_template/pull/384

I believe this thread is not needed anymore, as plugin writers can pick how
many releases they want to test.
It is safer to test only the latest release, but you can opt to add
more (the older the release, the more chances to break)

Best regards,
Fabricio Aguiar
Software Engineer, Pulp Project
Red Hat Brazil - Latam <https://www.redhat.com/>
+55 22 999000595



On Wed, May 12, 2021 at 5:34 PM Fabricio Aguiar <faguiard at redhat.com> wrote:

> Hi everyone,
>
> A while ago we met to discuss upgrade tests as a way to catch and avoid DB
> migration failures at CI time.
> I took the meeting notes [1] and started a POC [2] and I want to share my
> perspective.
> I found myself doing more workarounds than I expected, and at first, I
> thought this work was a no-go for this reason, but now I'm starting to
> think if I had to do these workarounds because we didn't have these tests
> before.
>
> I spread multiple comments on [2] to try to explain my reasoning or the
> why of the workarounds, please take a look at the PR and share your
> thoughts.
>
> Some plugins need other plugins while running tests, it adds one more
> challenge of keeping all versions compatible e.g. pulp-2to3-migration +
> pulpcore + pulp_rpm + ...
>
> Another challenge is dealing with requirements from previous versions, I
> had to do a hack for pinning django for example, and having additional
> plugins may add more complexity to it.
>
> The last challenge I noticed was about external commands or specific CI
> scripts for specific releases and breaking changes. I had to do some
> workarounds for the app label change for example and I'm not sure if I
> covered all the places (at least one of the tests is failing because of
> this). A while ago pulpcore needed an external command to be run before the
> migration (the artifact path change) I suspect this is the reason for the
> failure of another test.
>
> My personal opinion: I think it really worth having this kind of test, I
> started looking for testing only migration, but during the process, I
> became aware of dependencies, commands, and other things that may have an
> influence on the upgrade process. The far we go in the past, the more
> challenging it gets, so for starting I would only test from the last
> release to master. After doing this for a while it may get easier to test 2
> or 3 releases ago.
>
> [1] https://hackmd.io/IwhWRn6OS9ekKJNfQrGYFw
> [2] https://github.com/pulp/pulp_file/pull/503
>
> Best regards,
> Fabricio Aguiar
> Software Engineer, Pulp Project
> Red Hat Brazil - Latam <https://www.redhat.com/>
> +55 22 999000595
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20210514/16dd7e8b/attachment.htm>