[Pulp-dev] CI/CD - Upgrade Tests

[Fabricio Aguiar]

Hi everyone,

A while ago we met to discuss upgrade tests as a way to catch and avoid DB
migration failures at CI time.
I took the meeting notes [1] and started a POC [2] and I want to share my
perspective.
I found myself doing more workarounds than I expected, and at first, I
thought this work was a no-go for this reason, but now I'm starting to
think if I had to do these workarounds because we didn't have these tests
before.

I spread multiple comments on [2] to try to explain my reasoning or the why
of the workarounds, please take a look at the PR and share your thoughts.

Some plugins need other plugins while running tests, it adds one more
challenge of keeping all versions compatible e.g. pulp-2to3-migration +
pulpcore + pulp_rpm + ...

Another challenge is dealing with requirements from previous versions, I
had to do a hack for pinning django for example, and having additional
plugins may add more complexity to it.

The last challenge I noticed was about external commands or specific CI
scripts for specific releases and breaking changes. I had to do some
workarounds for the app label change for example and I'm not sure if I
covered all the places (at least one of the tests is failing because of
this). A while ago pulpcore needed an external command to be run before the
migration (the artifact path change) I suspect this is the reason for the
failure of another test.

My personal opinion: I think it really worth having this kind of test, I
started looking for testing only migration, but during the process, I
became aware of dependencies, commands, and other things that may have an
influence on the upgrade process. The far we go in the past, the more
challenging it gets, so for starting I would only test from the last
release to master. After doing this for a while it may get easier to test 2
or 3 releases ago.

[1] https://hackmd.io/IwhWRn6OS9ekKJNfQrGYFw
[2] https://github.com/pulp/pulp_file/pull/503

Best regards,
Fabricio Aguiar
Software Engineer, Pulp Project
Red Hat Brazil - Latam <https://www.redhat.com/>
+55 22 999000595
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20210512/2c24321e/attachment.htm>