[Libguestfs] Scaling virt-df performance

[Richard W.M. Jones]

On Wed, Sep 10, 2014 at 01:38:16PM +0000, Dan Ryder (daryder) wrote:
> Hello,
>
> I have been looking at the "virt-df" libguestfs tool to get
> guest-level disk used/free statistics - specifically with
> Qemu-KVM/Openstack. This works great for a few Openstack instances,
> but when I begin to scale (even to ~30 instances/guests) the
> performance really takes a hit. The time it takes for the command to
> complete seems to scale linearly with the amount of guests/domains
> running on the hypervisor (note - I am using "virt-df" for all
> guests, not specifying one at a time; although I've tried that,
> too).
>
> For ~30 guests, the "virt-df" command takes around 90 seconds to
> complete. We are looking to support a scale of 3,000-30,000 guests
> disk used/free. It looks like this won't be remotely possible using
> "virt-df".

With sufficient memory, non-nested, on hardware built in the last 3
years, you should get performance of about 1 second / guest
(pipelined).  So the figure you give is about 3 times too high than
where it should be.

Just to get some basic things out of the way:

- What version of virt-df are you using and on what distro?

- Is this nested?

- What exact virt-df command(s) are you running?

- Are you using the -P option?

- How much free memory is on the system?  virt-df runs multiple
  threads in parallel, but the number to run is computed according to
  the amount of free memory[1].

Also have a look at the guestfs-performance manual[2].  I'm especially
interested in the results of the baseline measurements on that page,
but the rest of the page should answer some of your questions too.
There's also an interesting Perl script you might try playing with.

Rich.

[1] https://github.com/libguestfs/libguestfs/blob/master/df/estimate-max-threads.c
[2] http://libguestfs.org/guestfs-performance.1.html


-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW