[Fedora-packaging] proposed "should" requirement: clean "make check" run w/valgrind

[Jim Meyering]

Hello,

I'd like to suggest that you add a guideline (perhaps even a "should"-style
one) that encourages packagers to ensure their tests pass also when
run via valgrind.

I suspect that there are far too many packages with serious, latent bugs
that would be exposed by such testing.  Better that people find them
now (ASAP) and fix them, than waiting for reports of hard-to-reproduce
failures or news of black hat exploits.

It is not onerous to require this, because it is so easy to automate.
In addition to running tests via "make check", each package can also
run its tests via e.g.,

  env "PATH=/some/new/temp/dir:$PATH" make check 3> log

where the temporary directory, /some/new/temp/dir, contains one or
more tiny wrapper scripts, each arranging to run a tested application
via valgrind.  Here's one named "rm" that I've used in testing that
program for the GNU coreutils:

#!/bin/sh
export PATH=/cu/src:/sbin:/usr/sbin:/bin
exec /usr/bin/valgrind -v --suppressions=/tmp/cu-vg --log-fd=3 \
  --leak-check=full --track-fds=yes --num-callers=15 -- rm "$@"

Note how the PATH setting in that script lists the absolute
build directory (in my case, "/cu/src"), first in the PATH.
The following PATH dirs are necessary in case the program
under test uses $PATH -- you might need others, e.g., /usr/local/bin.
If you're in a position to generate the file, just append the original
value of $PATH.

I chose to log to file descriptor 3 (--log-fd=3), so ran the make check
with the "3> log" suffix to redirect that output.  If you prefer, use
--log-file-exactly=log or --log-file=log.

When using valgrind, you will inevitably find warnings that should
be ignored, say because they come from libraries not under test (and
hoping that they are harmless).  Such warnings are highly dependent on
the version of your C library, valgrind, and maybe on the kernel, too.
In order to avoid the distraction of such warnings (about which you can
typically do nothing), you can use a valgrind suppression file.  Above,
I used the --suppressions=/tmp/cu-vg option.

The wrapper script above also tracks file descriptor usage
(--track-fds=yes), so that if there is a file descriptor leak, valgrind
will report it.  Similarly, it uses --leak-check=full to check for
memory leaks.

Finally, I've found it useful to record more than the default number of
stack frames (--num-callers=15), but admit that does cause a bit of a
performance hit.

Jim