[libvirt PATCH 00/10] [RFC] clang-tidy CI integration

[Tim Wiederhake]

On Fri, 2021-02-12 at 14:48 +0100, Ján Tomko wrote:
> On a Friday in 2021, Tim Wiederhake wrote:
> > "clang-tidy" is a static code analysis tool for c and c++ code. See
> > https://listman.redhat.com/archives/libvir-list/2021-January/msg01152.html
> > for some issues found by clang-tidy and more background
> > information.
> > 
> > Meson has support for clang-tidy and generates target named "clang-
> > tidy" if
> > it finds a ".clang-tidy" configuration file in the project's root
> > directory.
> > 
> > There are some problems with this approach though, with regards to
> > inclusion
> > in GitLab's CI:
> > 
> > * Single-threaded runtime of a complete scan takes between 95 and
> > 110 minutes,
> > depending on the enabled checks, which is significantly longer than
> > GitLab's
> > maximum run time of 60 minutes, after which jobs are aborted.
> > 
> 
> That does seem like a deal-breaker. Did you manage to do a successful
> run by iteratively building up the cache over multiple jobs? Or by
> doing
> a multi-threaded run?
> 

I combined the soft-timeout (./scripts/run-clang-tidy.py --timeout ...)
together with caching. What this does is to stop scanning after the
specified amount of time, but will keep the cache.

Consecutive runs will pick up the cache and hopefully get further down
the list of scanned files before running out of time. This process
converges at a point where there are so few (if any) uncached results
left, that the scan completes.

The benefit of this approach is, that it actually works, takes no
longer than 30 minutes and does not require one to click "re-run"
several times until the job becomes green. And in the best case where
no code file changed aka. all cache files are recent, this reduces the
run time of the clang-tidy job to roughly three minutes; see link to
pipeline below. The first iteration did run into timeout, the second
one finished within the time window. I clicked "rerun" a second time to
get the minimum required time I quoted above.

The downside of this approach is, that for "incomplete" runs, aka when
we hit the timeout, not all files are scanned and the code may still
contain issues that would be flagged in a consecutive run.

> > * Even without this limit on runtime, this new check would double
> > to triple
> > the run time of the libVirt pipeline in GitLab.
> > 
> 
> Running it for every job sounds wasteful - it can be run on schedule
> like the coverity job - daily or even weekly.
> 
> Also, please never write libvirt like that again.
> 
> Jano
> 

I have no idea what I mixed up libvirt with, which is capitalized this
way. Good thing it's Friday.

Tim

> > * clang-tidy finds a lot of false positives (see link above for
> > explanation)
> > and has checks that contradict the libVirt code style (e.g. braces
> > around
> > blocks). This makes a quite complex configuration in ".clang-tidy"
> > neccessary.
> > 
> > * I was unable to make clang-tidy recognize the settings from the
> > configuration file for generated files, leading clang-tidy to
> > always add some
> > checks. These checks were among those that produced false
> > positives.
> > 
> > * The list of enabled / disabled checks in the yaml configuration
> > file is a
> > quite long string, making it hard to weave in some comments /
> > documentation
> > on which checks are enabled / disabled for what reason.
> > 
> > This series introduces a new script, "run-clang-tidy.py". This is a
> > replacement for the script of the same name from clang-tools-extra. 
> > It offers
> > parallel execution, caching of results and a configurable soft-
> > timeout.
> > 
> > Please see the individual commits for more details. Comments
> > welcome.
> > 
> > https://gitlab.com/twiederh/libvirt/-/pipelines/255321968 → "clang-
> > tidy".
> > 
> > Tim
> > 
> > Tim Wiederhake (10):
> >  clang-tidy: Add a simple runner
> >  clang-tidy: Run in parallel
> >  clang-tidy: Filter output
> >  clang-tidy: Add cache
> >  clang-tidy: Add timeout
> >  clang-tidy: Allow timeouts
> >  clang-tidy: Add shuffle
> >  clang-tidy: Make list of checks explicit
> >  clang-tidy: Disable irrelevant and failing checks
> >  clang-tidy: Add CI integration
> > 
> > .gitlab-ci.yml            |  88 ++++++
> > scripts/run-clang-tidy.py | 557
> > ++++++++++++++++++++++++++++++++++++++
> > 2 files changed, 645 insertions(+)
> > create mode 100755 scripts/run-clang-tidy.py
> > 
> > -- 
> > 2.26.2
> > 
> >