[libvirt] [PATCH v5 02/23] build-aux: rewrite whitespace checker in Python

[Daniel P. Berrangé]

On Fri, Nov 15, 2019 at 03:30:15PM -0500, Cole Robinson wrote:
> On 11/11/19 9:38 AM, Daniel P. Berrangé wrote:
> > As part of an goal to eliminate Perl from libvirt build tools,
> > rewrite the check-spacing.pl tool in Python.
> > 
> > This was a straight conversion, manually going line-by-line to
> > change the syntax from Perl to Python. Thus the overall structure
> > of the file and approach is the same.
> > 
> 
> I checked the speed difference mentioned in the cover letter.
> 
> Old:
> time ./build-aux/check-spacing.pl `cat ~/src/libvirt/cfiles`
> real	0m1.423s
> 
> New:
> time ./scripts/check-spacing.py `cat cfiles`
> real	0m5.607s
> 
> About around there. The script can easily be parallelized though:
> 
> diff --git a/scripts/check-spacing.py b/scripts/check-spacing.py
> index 6b9f3ec1ba..f2ce376e80 100755
> --- a/scripts/check-spacing.py
> +++ b/scripts/check-spacing.py
> @@ -222,8 +222,12 @@ def check_whitespace(filename):
> 
> 
>  ret = 0
> -for filename in sys.argv[1:]:
> -    if check_whitespace(filename):
> -        ret = 1
> +filenames = sys.argv[1:]
> +import multiprocessing
> +pool = multiprocessing.Pool()
> +results = pool.map(check_whitespace, filenames)
> +pool.close()
> +pool.join()
> +ret = int(any(results))
> 
>  sys.exit(ret)
> 
> 
> After that:
> 
> [:~/src/libvirt] (python3 *) $ time ./scripts/check-spacing.py `cat cfiles`
> 
> real	0m1.674s
> user	0m10.506s
> sys	0m0.041s
> 
> 
> Which is pretty close to the perl version, but at the cost of more
> cycles. Not sure how that will play with 'make' job control. I believe
> the Pool defaults to number of logical host CPUs, which is 8 on my
> T480s. Going above 4 doesn't seem to make much of a difference in my
> testing for this case

FWIW, I always run with "make -j 8 syntax-check" so while this will
speed up the job when run individually, it won't help a full syntax
check as then it'll be competing with the other syntx checks that
are running in parallel.

I'd already said I'd drop this check previously but mistakenly
re-posted it in this v5.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|