[Crash-utility] Suggestion: Testing of crash patches before submissions

[David Wysochanski]

On Fri, Aug 14, 2020 at 4:27 AM HAGIO KAZUHITO(萩尾　一仁)
<k-hagio-ab at nec.com> wrote:
>
> -----Original Message-----
> > Hi all,
> >
> > Recently I found a regression and also in the past have seen other
> > regressions.  I am not sure what tests are currently run or expected
> > for patches.
> >
> > Since this project has transitioned away from the original author, and
> > there are many contributors, it may be more important to have some
> > tests in the project, especially for trickier areas, uncommon options,
> > and unique vmcores.  There are also numerous ways to validate patches
> > in a CI system now.  I started building one but then realized it
> > probably should be done inside the upstream project if possible.
>
> Hi Dave,
>
> Thanks for the suggestion, I'm interested :)
>
> As far as I know, he had tested patches with hundreds of vmcores, and
> Lianbo and Bhupesh can use them, but seems it doesn't have a test of
> the --minimal option.
>
> I also test patches with my test script, which has many commands and its
> options fairly comprehensively, executes non-patched crash and patched
> crash, measures elapsed time, and compare the outputs.  But it also doesn't
> have --minimal test.  I will add it to my test.. and check out CI systems.
>
> Thanks,
> Kazu
>

If you share these tests out or at least create a framework for testing that
is part of this project, the community can also run tests and/or add to tests
if certain areas are changed.  It could also be part of review of certain
patches.

One question is where to store the set of vmcores to use for testing.
Is this mainly just maintainers that keep these vmcores privately or is
there some safe way we can share them?  This is tricky for two reasons
that I can think of:
1. Size.  A decent set of vmcores probably is in the 100's of GBs.
2. Security.  We cannot share sensitive data which is often inside vmcores.

Maybe there is a set of vmcores that can cover ~80% of the things needing
to be tested and will also be small and not share sensitive data (i.e. test
VM cores for different kernels).  The last 20% of special types of vmcores
maybe should remain private?