[Virtio-fs] [QUESTION RESEND] A performance problem for buffer write compared with 9p

Fri Aug 9 04:45:04 UTC 2019

Hi all,

I met a performance problem when I tested buffer write compared with 9p.

Guest configuration:
	Kernel: https://github.com/rhvgoyal/linux/tree/virtio-fs-dev-5.1
	2vCPU
	8GB RAM
Host configuration:
	Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz
	128GB RAM
	Linux 3.10.0
	Qemu: https://gitlab.com/virtio-fs/qemu/tree/virtio-fs-dev
	EXT4 + ramdisk for shared folder

------------------------------------------------------------------------

For virtiofs:
virtiofsd cmd:
	./virtiofsd -o vhost_user_socket=/tmp/vhostqemu -o source=/mnt/share/ 
-o cache=always -o writeback
mount cmd:
	mount -t virtio_fs myfs /mnt/virtiofs -o 
rootmode=040000,user_id=0,group_id=0

For 9p:
mount cmd:
	mount -t 9p -o 
trans=virtio,version=9p2000.L,rw,dirsync,nodev,msize=1000000000,cache=fscache 
sharedir /mnt/virtiofs/

------------------------------------------------------------------------

Compared with 9p, the test result:
1. Latency
	Test model：
		fio -filename=/mnt/virtiofs/test -rw=write -bs=4K -size=1G -iodepth=1 \
			-ioengine=psync -numjobs=1 -group_reporting -name=4K -time_based 
-runtime=30

	virtiofs: avg-lat is 6.37 usec
		4K: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=psync, iodepth=1
		fio-2.13
		Starting 1 process
		Jobs: 1 (f=1): [W(1)] [100.0% done] [0KB/471.9MB/0KB /s] [0/121K/0 
iops] [eta 00m:00s]
		4K: (groupid=0, jobs=1): err= 0: pid=5558: Fri Aug  9 09:21:13 2019
		  write: io=13758MB, bw=469576KB/s, iops=117394, runt= 30001msec
			clat (usec): min=2, max=10316, avg= 5.75, stdev=81.80
			 lat (usec): min=3, max=10317, avg= 6.37, stdev=81.80

	9p: avg-lat is 3.94 usec
		4K: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=psync, iodepth=1
		fio-2.13
		Starting 1 process
		Jobs: 1 (f=1): [W(1)] [100.0% done] [0KB/634.2MB/0KB /s] [0/162K/0 
iops] [eta 00m:00s]
		4K: (groupid=0, jobs=1): err= 0: pid=5873: Fri Aug  9 09:53:46 2019
		  write: io=19700MB, bw=672414KB/s, iops=168103, runt= 30001msec
			clat (usec): min=2, max=632, avg= 3.34, stdev= 3.77
			 lat (usec): min=2, max=633, avg= 3.94, stdev= 3.82

2. Bandwidth
	Test model:
		fio -filename=/mnt/virtiofs/test -rw=write -bs=1M -size=1G -iodepth=1 \
		    -ioengine=psync -numjobs=1 -group_reporting -name=1M -time_based 
-runtime=30

	virtiofs: bandwidth is 718961KB/s
		1M: (g=0): rw=write, bs=1M-1M/1M-1M/1M-1M, ioengine=psync, iodepth=1
		fio-2.13
		Starting 1 process
		Jobs: 1 (f=1): [W(1)] [100.0% done] [0KB/753.8MB/0KB /s] [0/753/0 
iops] [eta 00m:00s]
		1M: (groupid=0, jobs=1): err= 0: pid=5648: Fri Aug  9 09:24:36 2019
			write: io=21064MB, bw=718961KB/s, iops=702, runt= 30001msec
			 clat (usec): min=390, max=11127, avg=1361.41, stdev=1551.50
			  lat (usec): min=432, max=11170, avg=1414.72, stdev=1553.28

	9p: bandwidth is 2305.5MB/s
		1M: (g=0): rw=write, bs=1M-1M/1M-1M/1M-1M, ioengine=psync, iodepth=1
		fio-2.13
		Starting 1 process
		Jobs: 1 (f=1): [W(1)] [100.0% done] [0KB/2406MB/0KB /s] [0/2406/0 
iops] [eta 00m:00s]
		1M: (groupid=0, jobs=1): err= 0: pid=5907: Fri Aug  9 09:55:14 2019
		  write: io=69166MB, bw=2305.5MB/s, iops=2305, runt= 30001msec
			clat (usec): min=287, max=17678, avg=352.00, stdev=503.43
			 lat (usec): min=330, max=17721, avg=402.76, stdev=503.41

9p has a lower latency and higher bandwidth than virtiofs.

------------------------------------------------------------------------			

I found that the judgement statement 'if (!TestSetPageDirty(page))' always
true in function '__set_page_dirty_nobuffers', it will waste much time
to mark inode dirty, no one page is dirty when write it the second time.
The buffer write stack:
	fuse_file_write_iter
	  ->fuse_cache_write_iter
		->generic_file_write_iter
		  ->__generic_file_write_iter
			->generic_perform_write
			  ->fuse_write_end
				->set_page_dirty
				  ->__set_page_dirty_nobuffers

The reason for 'if (!TestSetPageDirty(page))' always true may be the pdflush
process will clean the page's dirty flags in clear_page_dirty_for_io(),
and call fuse_writepages_send() to flush all pages to the disk of the host.
So when the page is written the second time, it always not dirty.
The pdflush stack for fuse:
	pdflush
	  ->...
	    ->do_writepages
		  ->fuse_writepages
		    ->write_cache_pages         // will clear all page's dirty flags
			  ->clear_page_dirty_for_io // clear page's dirty flags
			->fuse_writepages_send      // write all pages to the host, but don't 
wait the result
Why not wait for getting the result of writing back pages to the host
before cleaning all page's dirty flags?

As for 9p, pdflush will call clear_page_dirty_for_io() to clean the page's
dirty flags. Then call p9_client_write() to write the page to the host,
waiting for the result, and then flush the next page. In this case, buffer
write of 9p will hit the dirty page many times before it is being write
back to the host by pdflush process.
The pdflush stack for 9p:
	pdflush
	  ->...
	    ->do_writepages
		  ->generic_writepages
		    ->write_cache_pages
			  ->clear_page_dirty_for_io // clear page's dirty flags
			  ->__writepage
				->v9fs_vfs_writepage
				  ->v9fs_vfs_writepage_locked
					->p9_client_write   // it will get the writing back page's result

According to the test result, is the handling method of 9p for page writing
back more reasonable than virtiofs?

Thanks,
Yan Wang