[Virtio-fs] [PATCH 2/2] virtiofs: add dmap flags to differentiate write mapping from read mapping

[Liu Bo]

On Wed, May 08, 2019 at 11:27:04AM -0400, Vivek Goyal wrote:
> On Tue, May 07, 2019 at 01:58:11PM +0800, Liu Bo wrote:
> > There're 2 problems in dax rw path which were found by [1][2],
> > 
> > a) setupmapping always sends a RW mapping message to virtiofs daemon side
> > no matter it's doing reads or writes, the end result is that guest reads
> > on a mapping will cause a write page fault on host, which is unnecessary.
> > 
> > b) After a successful setupmapping, it doesn't guarantee the following
> > writes can land on host, e.g. page fault on host may fail because of
> > ENOSPC.
> > 
> > This is trying to solve the problems by
> > a) marking a dmap as RO or RW to indicate it's being used for reads or
> >    writes,
> > b) setting up a RO dmap for reads
> > c) setting up a RW dmap for writes
> > d) using an existing dmap for reads if it's availalbe in the interval tree
> > e) converting an existing dmap from RO to RW for writes if it's available
> >    in the interval tree
> > 
> > The downside of the above approach is write amplification.
> 
> Another downside of using fallocate() is performance overhead. I wrote
> a problem to do small 16K file extending writes and measure total time
> with and without fallocate. Fallocate version seems to be much slower.
> 
> With fallocate()
> ===============
> # ./file-extending-writes /mnt/virtio-fs/foo.txt
> 16384 extending writes took 14 secs and 647210 us
> 16384 extending writes took 11 secs and 571941 us
> 16384 extending writes took 11 secs and 981329 us
> 
> With dax + direct write (FUSE_WRITE)
> ==============================
> # ./file-extending-writes /mnt/virtio-fs/foo.txt
> 16384 extending writes took 2 secs and 477642 us
> 16384 extending writes took 2 secs and 352413 us
> 16384 extending writes took 2 secs and 347223 us
> 
> I ran your test script as well and that does not show much difference.
> I think part of the problem is that every write launches new xfs_io
> process. So overhead of launching process, opening fd again hides the
> fallocate overhead.

It's so weird, with the below C test, I couldn't reproduce the huge
difference of w/ or w/o fallocate.

=================================
mount -t virtio_fs atest /mnt/test/ -otag=myfs-2,rootmode=040000,user_id=0,group_id=0,dax

- w/

[root at debug010000002015 virtiofs]
$ ./a.out /mnt/test/foobar
16384 extending writes took 2 secs and 407494 us
16384 extending writes took 2 secs and 368892 us
16384 extending writes took 2 secs and 347208 us
[root at debug010000002015 virtiofs]
$ ./a.out /mnt/test/foobar
16384 extending writes took 2 secs and 306272 us
16384 extending writes took 2 secs and 364296 us
16384 extending writes took 2 secs and 354999 us
[root at debug010000002015 virtiofs]
$ ./a.out /mnt/test/foobar
16384 extending writes took 2 secs and 363470 us
16384 extending writes took 2 secs and 742043 us
16384 extending writes took 2 secs and 527137 us

- w/o (Revert "fuse, dax: Do not use dax for file extnding writes" + 
Revert "virtiofsd: do fallocate for setupmapping requests with write flag")

[root at debug010000002015 virtiofs]
$ ./a.out /mnt/test/foobar
16384 extending writes took 2 secs and 331097 us
16384 extending writes took 2 secs and 298868 us
16384 extending writes took 2 secs and 337858 us
[root at debug010000002015 virtiofs]
$ ./a.out /mnt/test/foobar
16384 extending writes took 2 secs and 319989 us
16384 extending writes took 2 secs and 519414 us
16384 extending writes took 2 secs and 436929 us
[root at debug010000002015 virtiofs]
$ ./a.out /mnt/test/foobar
16384 extending writes took 2 secs and 365331 us
16384 extending writes took 2 secs and 353973 us
16384 extending writes took 2 secs and 354729 us
=================================

thanks,
-liubo

> 
> Here is my C program.
> 
> ===========================================================
> #include <stdio.h>
> #include <stdlib.h>
> #include <errno.h>
> #include <string.h>
> #include <sys/types.h>
> #include <sys/stat.h>
> #include <fcntl.h>
> #include <unistd.h>
> #include <sys/time.h>
> 
> void drop_caches(void)
> {
> 	system("sync");
> 	system("echo 3 > /proc/sys/vm/drop_caches");
> }
> 
> void run_test(int fd)
> {
> 	int nr_writes = 16 * 1024;
> 	struct timeval tv_start, tv_end, tv_elapsed;
> 	char *buf = "Hello";
> 	int buf_len = strlen(buf);
> 	ssize_t written = 0;
> 	int i;
> 
> 	drop_caches();
> 
> 	/* Extend file in a loop */		
> 	gettimeofday(&tv_start, NULL);
> 	for (i = 0; i < nr_writes; i++) {
> 		written += write(fd, buf, buf_len);
> 		if (written == -1) {
> 			fprintf(stderr, "Failed to write %d bytes:%s,"
> 				" errorno=%d\n", strlen(buf), strerror(errno),
> 				errno);
> 			exit(1);
> 		}
> 	}
> 	gettimeofday(&tv_end, NULL);
> 	timersub(&tv_end, &tv_start, &tv_elapsed);	
> 	printf("%d extending writes took %d secs and %d us\n", nr_writes, tv_elapsed.tv_sec, tv_elapsed.tv_usec);
> }
> 
> int main(int argc, char *argv[])
> {
> 	int fd, flags = 0;
> 	mode_t mode = 0;
> 	int i, nr_runs = 3;
> 
> 	if (argc != 2) {
> 		printf("Usage:%s <file-to-open>\n", argv[0]);
> 		exit(1);
> 	}
> 
> 	fd = open(argv[1], O_RDWR | O_CREAT | O_APPEND | O_TRUNC);
> 	if (fd == -1) {
> 		fprintf(stderr, "Failed to open file %s:%s, errorno=%d\n",
> 			argv[0], strerror(errno), errno);
> 		exit(1);
> 	}
> 
> 	for(i = 0; i < nr_runs; i++)
> 		run_test(fd);
> 	close(fd);
> }
> 
> ==========================================================================