[Libguestfs] libnbd flamegraph Odd benchmark results

[Richard W.M. Jones]

On Sat, May 29, 2021 at 09:33:58PM +0530, Abhay Raj Singh wrote:
> Rich, *nbdcopy3-original.svg* is the latest one you sent
> *nbdcopy-abhay.svg* is the one I was able to create using given
> commands.
>
> However, there was some noise when I compared my results with the
> original. Is it normal or should I try fixing it. Thanks!

Hi Abhay, I think you'll fine the mailing list has a limit on the size
of attachemnts it can accept so they probably won't appear in the
mailing list archives.

I looked at the SVGs anyhow and there is quite a bit of difference,
which is possibly unexpected.  It's useful to post the exact command
line that you used, in case we're using different flags.

Some observations:

I think you're using swap-on-zram.
(https://fedoraproject.org/wiki/Changes/SwapOnZRAM).  The kswapd
process is certainly spending a lot of time in zstd* functions.  I'm
not using zram.  There's nothing necessarily wrong with this, it's
just a difference in configuration.  However ...

Ideally the test should not be swapping at all, so it's unclear why
pageout() is being called.  How much RAM does the machine have?

The kernel function asm_exc_page_fault appears a lot, most prominently
for nbdcopy but also for nbdkit.  This indicates that the x86 page
fault trap happens a lot, which indicates that the processes are being
swapped in and out frequently
(https://elixir.bootlin.com/linux/latest/source/arch/x86/kernel/idt.c#L221)

In my trace, about 40% of the time is seen in swapper.  However I
believe this indicates CPU cores which are idle (and my test machine
has a lot of cores).  swapper is hardly seen in your trace, which just
means fewer cores are idle.  It's not really a significant difference.

My conclusion is the test machine probably doesn't have as much free
memory, and there's a lot of swapping going on.

While the test is running, try also running: "vmstat 1".  If the "si"
and "so" fields in the output are non-zero, then the machine is
swapping.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/