[Libguestfs] nbdkit random seek performance

[Richard W.M. Jones]

On Thu, Aug 01, 2019 at 03:44:31PM -0700, ivo welch wrote:
> hi richard---arthur and I are working with nbdkit v1.12.3 on qemu/kvm.
> 
> we found that our linux (ubuntu 16.04 32-bit) boot time from a local .img
> file went from about 10 seconds to about 3 minutes when using the nbdkit
> file plugin instead of directly connecting qemu to the file.  on further
> inspection with bonnie++, we think the problem may be related to poor
> random seek performance:

In general I would expect some slowdown between using qemu directly
against a file and using qemu -> nbd driver -> nbdkit -> file, but not
nearly as much as you observed.

This is running bonnie++ inside the guest?

Are qemu & nbdkit running on the same machine?  Are you using a TCP
port or a Unix domain socket?  (If TCP you may want to consider
commits d088bf45, 0721aa9579 and 3842a080c).  Which version of qemu is
it?

Rich.

> DIRECT FROM QEMU TO .IMG FILE:
> ------------------------------------------------------------------------------------------------------------------------------
> Version  1.97       ------Sequential Output------ --Sequential Input-
> --Random-
> Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
> --Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec
> %CP
> TEST             4G           109468  11 58405  16           180613  33
>  3204 147
> Latency                        7125ms    5812ms               131ms
> 101ms
> ------------------------------------------------------------------------------------------------------------------------------
> ------------------------------------------------------------------------------------------------------------------------------
> 
> 
> 
> NBDKIT:
> ------------------------------------------------------------------------------------------------------------------------------
> 1.97,1.97,TEST,1,1564697809,4G,,,,109468,11,58405,16,,,180613,33,3204,147,,,,,,,,,,,,,,,,,,,7125ms,5812ms,,131ms,101ms,,,,,,
> Version  1.97       ------Sequential Output------ --Sequential Input-
> --Random-
> Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
> --Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec
> %CP
> TEST             4G           105761  11 58992  14           178472  30
>  39.3   6
> Latency                        8166ms    3683ms               187ms
>  1197ms
> ------------------------------------------------------------------------------------------------------------------------------
> ------------------------------------------------------------------------------------------------------------------------------
> 
> 
> 
> NBDKIT (-fv):
> ------------------------------------------------------------------------------------------------------------------------------
> 1.97,1.97,TEST,1,1564698256,4G,,,,105761,11,58992,14,,,178472,30,39.3,6,,,,,,,,,,,,,,,,,,,8166ms,3683ms,,187ms,1197ms,,,,,,
> Version  1.97       ------Sequential Output------ --Sequential Input-
> --Random-
> Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
> --Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec
> %CP
> TEST             4G           104941  10 56501  14           160557  28
>  39.9   6
> Latency                       15318ms    5200ms               101ms
>  1152ms
> 
> 1.97,1.97,TEST,1,1564695086,4G,,,,104941,10,56501,14,,,160557,28,39.9,6,,,,,,,,,,,,,,,,,,,15318ms,5200ms,,101ms,1152ms,,,,,,
> ------------------------------------------------------------------------------------------------------------------------------
> ------------------------------------------------------------------------------------------------------------------------------
> 
> 
> is this an error on our part?  do you have any recommendations on what we
> could investigate next?
> 
> regards,
> 
> /iaw
> 
> 
> --
> Ivo Welch (ivo.welch at ucla.edu)
> http://www.ivo-welch.info/

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org