[Virtio-fs] tools/virtiofs: Multi threading seems to hurt performance
Dr. David Alan Gilbert
dgilbert at redhat.com
Tue Sep 22 11:09:46 UTC 2020
* Vivek Goyal (vgoyal at redhat.com) wrote:
> On Fri, Sep 18, 2020 at 05:34:36PM -0400, Vivek Goyal wrote:
> > Hi All,
> >
> > virtiofsd default thread pool size is 64. To me it feels that in most of
> > the cases thread pool size 1 performs better than thread pool size 64.
> >
> > I ran virtiofs-tests.
> >
> > https://github.com/rhvgoyal/virtiofs-tests
>
> I spent more time debugging this. First thing I noticed is that we
> are using "exclusive" glib thread pool.
>
> https://developer.gnome.org/glib/stable/glib-Thread-Pools.html#g-thread-pool-new
>
> This seems to run pre-determined number of threads dedicated to that
> thread pool. Little instrumentation of code revealed that every new
> request gets assiged to new thread (despite the fact that previous
> thread finished its job). So internally there might be some kind of
> round robin policy to choose next thread for running the job.
>
> I decided to switch to "shared" pool instead where it seemed to spin
> up new threads only if there is enough work. Also threads can be shared
> between pools.
>
> And looks like testing results are way better with "shared" pools. So
> may be we should switch to shared pool by default. (Till somebody shows
> in what cases exclusive pools are better).
>
> Second thought which came to mind was what's the impact of NUMA. What
> if qemu and virtiofsd process/threads are running on separate NUMA
> node. That should increase memory access latency and increased overhead.
> So I used "numactl --cpubind=0" to bind both qemu and virtiofsd to node
> 0. My machine seems to have two numa nodes. (Each node is having 32
> logical processors). Keeping both qemu and virtiofsd on same node
> improves throughput further.
>
> So here are the results.
>
> vtfs-none-epool --> cache=none, exclusive thread pool.
> vtfs-none-spool --> cache=none, shared thread pool.
> vtfs-none-spool-numa --> cache=none, shared thread pool, same numa node
Do you have the numbers for:
epool
epool thread-pool-size=1
spool
?
Dave
>
> NAME WORKLOAD Bandwidth IOPS
> vtfs-none-epool seqread-psync 36(MiB/s) 9392
> vtfs-none-spool seqread-psync 68(MiB/s) 17k
> vtfs-none-spool-numa seqread-psync 73(MiB/s) 18k
>
> vtfs-none-epool seqread-psync-multi 210(MiB/s) 52k
> vtfs-none-spool seqread-psync-multi 260(MiB/s) 65k
> vtfs-none-spool-numa seqread-psync-multi 309(MiB/s) 77k
>
> vtfs-none-epool seqread-libaio 286(MiB/s) 71k
> vtfs-none-spool seqread-libaio 328(MiB/s) 82k
> vtfs-none-spool-numa seqread-libaio 332(MiB/s) 83k
>
> vtfs-none-epool seqread-libaio-multi 201(MiB/s) 50k
> vtfs-none-spool seqread-libaio-multi 254(MiB/s) 63k
> vtfs-none-spool-numa seqread-libaio-multi 276(MiB/s) 69k
>
> vtfs-none-epool randread-psync 40(MiB/s) 10k
> vtfs-none-spool randread-psync 64(MiB/s) 16k
> vtfs-none-spool-numa randread-psync 72(MiB/s) 18k
>
> vtfs-none-epool randread-psync-multi 211(MiB/s) 52k
> vtfs-none-spool randread-psync-multi 252(MiB/s) 63k
> vtfs-none-spool-numa randread-psync-multi 297(MiB/s) 74k
>
> vtfs-none-epool randread-libaio 313(MiB/s) 78k
> vtfs-none-spool randread-libaio 320(MiB/s) 80k
> vtfs-none-spool-numa randread-libaio 330(MiB/s) 82k
>
> vtfs-none-epool randread-libaio-multi 257(MiB/s) 64k
> vtfs-none-spool randread-libaio-multi 274(MiB/s) 68k
> vtfs-none-spool-numa randread-libaio-multi 319(MiB/s) 79k
>
> vtfs-none-epool seqwrite-psync 34(MiB/s) 8926
> vtfs-none-spool seqwrite-psync 55(MiB/s) 13k
> vtfs-none-spool-numa seqwrite-psync 66(MiB/s) 16k
>
> vtfs-none-epool seqwrite-psync-multi 196(MiB/s) 49k
> vtfs-none-spool seqwrite-psync-multi 225(MiB/s) 56k
> vtfs-none-spool-numa seqwrite-psync-multi 270(MiB/s) 67k
>
> vtfs-none-epool seqwrite-libaio 257(MiB/s) 64k
> vtfs-none-spool seqwrite-libaio 304(MiB/s) 76k
> vtfs-none-spool-numa seqwrite-libaio 267(MiB/s) 66k
>
> vtfs-none-epool seqwrite-libaio-multi 312(MiB/s) 78k
> vtfs-none-spool seqwrite-libaio-multi 366(MiB/s) 91k
> vtfs-none-spool-numa seqwrite-libaio-multi 381(MiB/s) 95k
>
> vtfs-none-epool randwrite-psync 38(MiB/s) 9745
> vtfs-none-spool randwrite-psync 55(MiB/s) 13k
> vtfs-none-spool-numa randwrite-psync 67(MiB/s) 16k
>
> vtfs-none-epool randwrite-psync-multi 186(MiB/s) 46k
> vtfs-none-spool randwrite-psync-multi 240(MiB/s) 60k
> vtfs-none-spool-numa randwrite-psync-multi 271(MiB/s) 67k
>
> vtfs-none-epool randwrite-libaio 224(MiB/s) 56k
> vtfs-none-spool randwrite-libaio 296(MiB/s) 74k
> vtfs-none-spool-numa randwrite-libaio 290(MiB/s) 72k
>
> vtfs-none-epool randwrite-libaio-multi 300(MiB/s) 75k
> vtfs-none-spool randwrite-libaio-multi 350(MiB/s) 87k
> vtfs-none-spool-numa randwrite-libaio-multi 383(MiB/s) 95k
>
> Thanks
> Vivek
--
Dr. David Alan Gilbert / dgilbert at redhat.com / Manchester, UK
More information about the Virtio-fs
mailing list