[Freeipa-devel] Automated Fedora update testing

[Simo Sorce]

Top posting FTW! (sorry)

Excellent news Adam, this is awesome!

Simo.

On Fri, 2017-04-28 at 17:07 -0700, Adam Williamson wrote:
> Hi folks! I thought this might be of interest to the FreeIPA
> community,
> so I thought I'd write it up here in case anyone missed it elsewhere.
> 
> I work on the Fedora QA team, and we have been using the openQA
> automated test system (developed by our friends at SUSE) to run
> various
> functional tests on Fedora composes for the last couple of years.
> 
> As FreeIPA is considered a critical part of Fedora Server, we run a
> few
> tests that exercise FreeIPA. The tests set up a FreeIPA server, run
> some basic checks on it, and also enrol two systems as clients of the
> domain, one using the 'realm join' command directly, one using
> Cockpit.
> The client tests do some basic client functionality testing (getent,
> logging in as a domain user, changing passwords, etc.) and also test
> the web UI to some extent.
> 
> Until recently we ran these tests only on Fedora's nightly
> development
> release distribution composes. Recently, though, we deployed some
> enhancements to our openQA setup that let us run tests on Fedora
> distribution updates as well, and have the results made visible
> through
> the Fedora update system (Bodhi). The tests are automatically run on
> any critical path package, and as of today, they are also run on any
> update containing any of a manually-tended list of FreeIPA-related
> packages:
> 
> 389-ds
> 389-ds-base
> bind
> bind-dyndb-ldap
> certmonger
> ding-libs
> freeipa
> krb5-server
> pki-core
> sssd
> tomcat
> cockpit
> 
> This means that for any Fedora update containing one of these or any
> critical path package, Fedora's openQA FreeIPA tests should run, and
> you should see the results in the Fedora update system (Bodhi). You
> can
> see the results in Bodhi by clicking the Automated Updates tab for
> any
> update. For instance, here's a recent 389-ds-base update for Fedora
> 26:
> 
> https://bodhi.fedoraproject.org/updates/FEDORA-2017-15e2a038b2
> 
> If you look at the Automated Tests tab, you can see passes for:
> 
> update.server_role_deploy_domain_controller
> update.realmd_join_cockpit
> update.realmd_join_sssd
> 
> indicating that this update didn't cause any problems for FreeIPA.
> Clicking on any test result will take you to the openQA page for the
> test, where you can diagnose failures and so on (explaining how to do
> this is a bit beyond the scope of this mail, please do ask me if
> you're
> interested!)
> 
> I hope this stuff will help us avoid shipping updates that break
> FreeIPA (and other key components). If you have any questions,
> concerns, comments, or suggestions, please do ask!
> 
> To anticipate one question: you can cause *all* the tests for an
> update
> to be re-run by editing the update in any way (you don't have to
> change
> the package loadout, just changing a single character in the
> description or something will do). If you think just one test result
> is
> bogus and want it re-run, currently, you'll have to ask someone with
> the necessary power - either me or Jan Sedlak (garretraziel on IRC).
> I'm in North America and he's in Europe, so we should have most
> timezones covered between us. We're hoping to set up a better
> mechanism
> for this in future.
> 
> Note, if you're interested in the results for the nightly Fedora
> distribution composes, an email summary of the results for those is
> sent each time they're run to the Fedora test@ and devel@ lists, look
> for mails with "compose check report" in the subject. Any time any of
> the FreeIPA tests fails, the failure will be listed in the mail
> (passed
> tests are not specifically listed, just a count of them). I usually
> keep an eye on those results and analyze failures and file bugs,
> though.
> -- 
> Adam Williamson
> Fedora QA Community Monkey
> IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin .
> net
> http://www.happyassassin.net
>