[Freeipa-devel] Automated Fedora update testing

[Lukas Slebodnik]

On (28/04/17 17:07), Adam Williamson wrote:
>Hi folks! I thought this might be of interest to the FreeIPA community,
>so I thought I'd write it up here in case anyone missed it elsewhere.
>
>I work on the Fedora QA team, and we have been using the openQA
>automated test system (developed by our friends at SUSE) to run various
>functional tests on Fedora composes for the last couple of years.
>
>As FreeIPA is considered a critical part of Fedora Server, we run a few
>tests that exercise FreeIPA. The tests set up a FreeIPA server, run
>some basic checks on it, and also enrol two systems as clients of the
>domain, one using the 'realm join' command directly, one using Cockpit.
>The client tests do some basic client functionality testing (getent,
>logging in as a domain user, changing passwords, etc.) and also test
>the web UI to some extent.
>
>Until recently we ran these tests only on Fedora's nightly development
>release distribution composes. Recently, though, we deployed some
>enhancements to our openQA setup that let us run tests on Fedora
>distribution updates as well, and have the results made visible through
>the Fedora update system (Bodhi). The tests are automatically run on
>any critical path package, and as of today, they are also run on any
>update containing any of a manually-tended list of FreeIPA-related
>packages:
>
>389-ds
>389-ds-base
>bind
>bind-dyndb-ldap
>certmonger
>ding-libs
>freeipa
>krb5-server
>pki-core
>sssd
>tomcat
>cockpit
>
>This means that for any Fedora update containing one of these or any
>critical path package, Fedora's openQA FreeIPA tests should run, and
>you should see the results in the Fedora update system (Bodhi). You can
>see the results in Bodhi by clicking the Automated Updates tab for any
>update. For instance, here's a recent 389-ds-base update for Fedora 26:
>
>https://bodhi.fedoraproject.org/updates/FEDORA-2017-15e2a038b2
>
>If you look at the Automated Tests tab, you can see passes for:
>
>update.server_role_deploy_domain_controller
>update.realmd_join_cockpit
>update.realmd_join_sssd
>
>indicating that this update didn't cause any problems for FreeIPA.
>Clicking on any test result will take you to the openQA page for the
>test, where you can diagnose failures and so on (explaining how to do
>this is a bit beyond the scope of this mail, please do ask me if you're
>interested!)
>
>I hope this stuff will help us avoid shipping updates that break
>FreeIPA (and other key components). If you have any questions,
>concerns, comments, or suggestions, please do ask!
>
>To anticipate one question: you can cause *all* the tests for an update
>to be re-run by editing the update in any way (you don't have to change
>the package loadout, just changing a single character in the
>description or something will do). If you think just one test result is
>bogus and want it re-run, currently, you'll have to ask someone with
>the necessary power - either me or Jan Sedlak (garretraziel on IRC).
>I'm in North America and he's in Europe, so we should have most
>timezones covered between us. We're hoping to set up a better mechanism
>for this in future.
>
>Note, if you're interested in the results for the nightly Fedora
>distribution composes, an email summary of the results for those is
>sent each time they're run to the Fedora test@ and devel@ lists, look
>for mails with "compose check report" in the subject. Any time any of
>the FreeIPA tests fails, the failure will be listed in the mail (passed
>tests are not specifically listed, just a count of them). I usually
>keep an eye on those results and analyze failures and file bugs,
>though.

Tested with sssd and it passed as well.
https://bodhi.fedoraproject.org/updates/FEDORA-2017-8addfc0188

freeIPA has also upstream integration tests packaged in
python{2,3}-ipatests. They use pytest + python-pytest-multihost.

Will it be possible to run some of them in openQA?
e.g. test_installation.py (

LS