[Freeipa-devel] Automated Fedora update testing

[Adam Williamson]

Hi folks! I thought this might be of interest to the FreeIPA community,
so I thought I'd write it up here in case anyone missed it elsewhere.

I work on the Fedora QA team, and we have been using the openQA
automated test system (developed by our friends at SUSE) to run various
functional tests on Fedora composes for the last couple of years.

As FreeIPA is considered a critical part of Fedora Server, we run a few
tests that exercise FreeIPA. The tests set up a FreeIPA server, run
some basic checks on it, and also enrol two systems as clients of the
domain, one using the 'realm join' command directly, one using Cockpit.
The client tests do some basic client functionality testing (getent,
logging in as a domain user, changing passwords, etc.) and also test
the web UI to some extent.

Until recently we ran these tests only on Fedora's nightly development
release distribution composes. Recently, though, we deployed some
enhancements to our openQA setup that let us run tests on Fedora
distribution updates as well, and have the results made visible through
the Fedora update system (Bodhi). The tests are automatically run on
any critical path package, and as of today, they are also run on any
update containing any of a manually-tended list of FreeIPA-related
packages:

389-ds
389-ds-base
bind
bind-dyndb-ldap
certmonger
ding-libs
freeipa
krb5-server
pki-core
sssd
tomcat
cockpit

This means that for any Fedora update containing one of these or any
critical path package, Fedora's openQA FreeIPA tests should run, and
you should see the results in the Fedora update system (Bodhi). You can
see the results in Bodhi by clicking the Automated Updates tab for any
update. For instance, here's a recent 389-ds-base update for Fedora 26:

https://bodhi.fedoraproject.org/updates/FEDORA-2017-15e2a038b2

If you look at the Automated Tests tab, you can see passes for:

update.server_role_deploy_domain_controller
update.realmd_join_cockpit
update.realmd_join_sssd

indicating that this update didn't cause any problems for FreeIPA.
Clicking on any test result will take you to the openQA page for the
test, where you can diagnose failures and so on (explaining how to do
this is a bit beyond the scope of this mail, please do ask me if you're
interested!)

I hope this stuff will help us avoid shipping updates that break
FreeIPA (and other key components). If you have any questions,
concerns, comments, or suggestions, please do ask!

To anticipate one question: you can cause *all* the tests for an update
to be re-run by editing the update in any way (you don't have to change
the package loadout, just changing a single character in the
description or something will do). If you think just one test result is
bogus and want it re-run, currently, you'll have to ask someone with
the necessary power - either me or Jan Sedlak (garretraziel on IRC).
I'm in North America and he's in Europe, so we should have most
timezones covered between us. We're hoping to set up a better mechanism
for this in future.

Note, if you're interested in the results for the nightly Fedora
distribution composes, an email summary of the results for those is
sent each time they're run to the Fedora test@ and devel@ lists, look
for mails with "compose check report" in the subject. Any time any of
the FreeIPA tests fails, the failure will be listed in the mail (passed
tests are not specifically listed, just a count of them). I usually
keep an eye on those results and analyze failures and file bugs,
though.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net