[Virtio-fs] [QUESTION] A performance problem for buffer write compared with 9p

[wangyan]

Hi all,

I met a performance problem when I tested buffer write compared with 9p.

Guest configuration:
     Kernel: https://github.com/rhvgoyal/linux/tree/virtio-fs-dev-5.1
     2vCPU
     8GB RAM
Host configuration:
     Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz
     128GB RAM
     Linux 3.10.0
     Qemu: https://gitlab.com/virtio-fs/qemu/tree/virtio-fs-dev
     EXT4 + ramdisk for shared folder

------------------------------------------------------------------------

For virtiofs:
virtiofsd cmd:
     ./virtiofsd -o vhost_user_socket=/tmp/vhostqemu -o 
source=/mnt/share/ -o cache=always -o writeback
mount cmd:
     mount -t virtio_fs myfs /mnt/virtiofs -o 
rootmode=040000,user_id=0,group_id=0

For 9p:
mount cmd:
     mount -t 9p -o 
trans=virtio,version=9p2000.L,rw,dirsync,nodev,msize=1000000000,cache=fscache 
sharedir /mnt/virtiofs/

------------------------------------------------------------------------

Compared with 9p, the test result:
1. Latency
     Test model：
         fio -filename=/mnt/virtiofs/test -rw=write -bs=4K -size=1G 
-iodepth=1 \
             -ioengine=psync -numjobs=1 -group_reporting -name=4K 
-time_based -runtime=30

     virtiofs: avg-lat is 6.37 usec
         4K: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=psync, 
iodepth=1
         fio-2.13
         Starting 1 process
         Jobs: 1 (f=1): [W(1)] [100.0% done] [0KB/471.9MB/0KB /s] 
[0/121K/0 iops] [eta 00m:00s]
         4K: (groupid=0, jobs=1): err= 0: pid=5558: Fri Aug  9 09:21:13 2019
           write: io=13758MB, bw=469576KB/s, iops=117394, runt= 30001msec
             clat (usec): min=2, max=10316, avg= 5.75, stdev=81.80
              lat (usec): min=3, max=10317, avg= 6.37, stdev=81.80

     9p: avg-lat is 3.94 usec
         4K: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=psync, 
iodepth=1
         fio-2.13
         Starting 1 process
         Jobs: 1 (f=1): [W(1)] [100.0% done] [0KB/634.2MB/0KB /s] 
[0/162K/0 iops] [eta 00m:00s]
         4K: (groupid=0, jobs=1): err= 0: pid=5873: Fri Aug  9 09:53:46 2019
           write: io=19700MB, bw=672414KB/s, iops=168103, runt= 30001msec
             clat (usec): min=2, max=632, avg= 3.34, stdev= 3.77
              lat (usec): min=2, max=633, avg= 3.94, stdev= 3.82


2. Bandwidth
     Test model:
         fio -filename=/mnt/virtiofs/test -rw=write -bs=1M -size=1G 
-iodepth=1 \
             -ioengine=psync -numjobs=1 -group_reporting -name=1M 
-time_based -runtime=30

     virtiofs: bandwidth is 718961KB/s
         1M: (g=0): rw=write, bs=1M-1M/1M-1M/1M-1M, ioengine=psync, 
iodepth=1
         fio-2.13
         Starting 1 process
         Jobs: 1 (f=1): [W(1)] [100.0% done] [0KB/753.8MB/0KB /s] 
[0/753/0 iops] [eta 00m:00s]
         1M: (groupid=0, jobs=1): err= 0: pid=5648: Fri Aug  9 09:24:36 2019
             write: io=21064MB, bw=718961KB/s, iops=702, runt= 30001msec
              clat (usec): min=390, max=11127, avg=1361.41, stdev=1551.50
               lat (usec): min=432, max=11170, avg=1414.72, stdev=1553.28

     9p: bandwidth is 2305.5MB/s
         1M: (g=0): rw=write, bs=1M-1M/1M-1M/1M-1M, ioengine=psync, 
iodepth=1
         fio-2.13
         Starting 1 process
         Jobs: 1 (f=1): [W(1)] [100.0% done] [0KB/2406MB/0KB /s] 
[0/2406/0 iops] [eta 00m:00s]
         1M: (groupid=0, jobs=1): err= 0: pid=5907: Fri Aug  9 09:55:14 2019
           write: io=69166MB, bw=2305.5MB/s, iops=2305, runt= 30001msec
             clat (usec): min=287, max=17678, avg=352.00, stdev=503.43
              lat (usec): min=330, max=17721, avg=402.76, stdev=503.41

9p has a lower latency and higher bandwidth than virtiofs.

------------------------------------------------------------------------ 


I found that the judgement statement 'if (!TestSetPageDirty(page))' always
true in function '__set_page_dirty_nobuffers', it will waste much time
to mark inode dirty, no one page is dirty when write it the second time.
The buffer write stack:
     fuse_file_write_iter
       ->fuse_cache_write_iter
         ->generic_file_write_iter
           ->__generic_file_write_iter
             ->generic_perform_write
               ->fuse_write_end
                 ->set_page_dirty
                   ->__set_page_dirty_nobuffers

The reason for 'if (!TestSetPageDirty(page))' always true may be the pdflush
process will clean the page's dirty flags in clear_page_dirty_for_io(),
and call fuse_writepages_send() to flush all pages to the disk of the host.
So when the page is written the second time, it always not dirty.
The pdflush stack for fuse:
     pdflush
       ->...
         ->do_writepages
           ->fuse_writepages
             ->write_cache_pages         // will clear all page's dirty 
flags
               ->clear_page_dirty_for_io // clear page's dirty flags
             ->fuse_writepages_send      // write all pages to the host, 
but don't wait the result
Why not wait for getting the result of writing back pages to the host
before cleaning all page's dirty flags?

As for 9p, pdflush will call clear_page_dirty_for_io() to clean the page's
dirty flags. Then call p9_client_write() to write the page to the host,
waiting for the result, and then flush the next page. In this case, buffer
write of 9p will hit the dirty page many times before it is being write
back to the host by pdflush process.
The pdflush stack for 9p:
     pdflush
       ->...
         ->do_writepages
           ->generic_writepages
             ->write_cache_pages
               ->clear_page_dirty_for_io // clear page's dirty flags
               ->__writepage
                 ->v9fs_vfs_writepage
                   ->v9fs_vfs_writepage_locked
                     ->p9_client_write   // it will get the writing back 
page's result


According to the test result, is the handling method of 9p for page writing
back more reasonable than virtiofs?

Thanks,
Yan Wang