[Crash-utility] Suggestion: Testing of crash patches before submissions

[HAGIO KAZUHITO(萩尾 一仁)]

-----Original Message-----
> On Fri, Aug 14, 2020 at 4:27 AM HAGIO KAZUHITO(萩尾　一仁)
> <k-hagio-ab at nec.com> wrote:
> >
> > -----Original Message-----
> > > Hi all,
> > >
> > > Recently I found a regression and also in the past have seen other
> > > regressions.  I am not sure what tests are currently run or expected
> > > for patches.
> > >
> > > Since this project has transitioned away from the original author, and
> > > there are many contributors, it may be more important to have some
> > > tests in the project, especially for trickier areas, uncommon options,
> > > and unique vmcores.  There are also numerous ways to validate patches
> > > in a CI system now.  I started building one but then realized it
> > > probably should be done inside the upstream project if possible.
> >
> > Hi Dave,
> >
> > Thanks for the suggestion, I'm interested :)
> >
> > As far as I know, he had tested patches with hundreds of vmcores, and
> > Lianbo and Bhupesh can use them, but seems it doesn't have a test of
> > the --minimal option.
> >
> > I also test patches with my test script, which has many commands and its
> > options fairly comprehensively, executes non-patched crash and patched
> > crash, measures elapsed time, and compare the outputs.  But it also doesn't
> > have --minimal test.  I will add it to my test.. and check out CI systems.
> >
> > Thanks,
> > Kazu
> >
> 
> If you share these tests out or at least create a framework for testing that
> is part of this project, the community can also run tests and/or add to tests
> if certain areas are changed.  It could also be part of review of certain
> patches.
> 
> One question is where to store the set of vmcores to use for testing.
> Is this mainly just maintainers that keep these vmcores privately or is
> there some safe way we can share them?  This is tricky for two reasons
> that I can think of:
> 1. Size.  A decent set of vmcores probably is in the 100's of GBs.
> 2. Security.  We cannot share sensitive data which is often inside vmcores.
> 
> Maybe there is a set of vmcores that can cover ~80% of the things needing
> to be tested and will also be small and not share sensitive data (i.e. test
> VM cores for different kernels).  The last 20% of special types of vmcores
> maybe should remain private?

As you say, we cannot share vmcores that have sensitive data or are big ones.
so personally I'm thinking about creating non-sensitive vmcores for the open
test gradually.  They will not be tricky or unique, but I think it would be
important for us to detect common failures quickly and easily so that we can
keep the base quality of the crash utility and save our development effort.

But first I still need to know what CI systems can do..

Thanks,
Kazu