[Freeipa-devel] CI: exporting test runner output

[Fraser Tweedale]

On Thu, Jan 05, 2017 at 08:53:14AM +0100, Martin Babinsky wrote:
> On 01/05/2017 08:06 AM, Fraser Tweedale wrote:
> > Hi all,
> > 
> > Although it has been discussed before and met with some skepticism,
> > here is a POC that exporting test runner output to, e.g. a pastebin,
> > does work:
> > 
> > - experimental commit: https://github.com/freeipa/freeipa/pull/370
> > - example paste: https://paste.fedoraproject.org/520085/
> >   (it is gzipped for reasons discussed in the PR)
> > 
> > I think we should proceed with getting these artifacts out of Travis
> > and stored somewhere (it doesn't have to be
> > paste.fedoraproject.org).  ``tail -n 5000`` of the log file has
> > proven to be not enough to diagnose all failures.
> > 
> Wow this is great, why have I not thought about it beforehand?
> 
> We can reduce the log size if we truncate everything before ERRORS/FAILURES
> output of pytest run (we leave the log as it is if the fail occurs before
> this stage), that should shave off considerable amount of cruft from the
> paste unless somebody sends a PR that breaks all out tests :D.
> 
> > If we stick with paste.fedoraproject.org, we can send to a
> > "project-specific" namespace e.g.
> > https://paste.fedoraproject.org/~freeipa, so that we do not clutter
> > up the main archive (I think).
> > 
> > A few questions for discussion:
> > 
> > 1. Stick with fpaste or not?  If so, use "~freeipa" namespace?
> >    (Keep in mind that the size limitation that exists for fpaste,
> >    which requires compressing the artifact, may not be a problem
> >    elsewhere).
> > 
> > 2. Export log always, or only if the build job failed?
> > 
> I would also paste the output to "freeipa" or even better "freeipa-travis"
> namespace and only send it if the job fails.
>
I might go with "freeipa-ci".

> > 3. Should pasted logs expire?  If so, what should TTL be?
> > 
> IMHO yes, but TTL is hard to determine, since the author of the PR may not
> be present to review the results immediately (because he is on PTO etc.). I
> think we should set TTL to something like 1 week and as a fallback keep
> tailing the CI results log.
>
1 week sounds reasonable.  We can change it later if we need to.

> > 4. Should we continue to `tail -n 5000` the log as we currently do,
> >    or just rely on exported log?
> > 
> > Thanks,
> > Fraser
> > 
> 
> Fraser, are you OK with waiting with this effort until we push
> https://github.com/freeipa/freeipa/pull/361 ?. I will just do some more
> adjustments there (like result log trimming) and it should be pushed ASAP.
> 
Yes, I was aware that there would be conflicts with this PR.  I
don't mind waiting.  Thanks for your input.

Cheers,
Fraser