[Libguestfs] [PATCH nbdkit] file: Implement cache=none and fadvise=normal|random|sequential.

[Richard W.M. Jones]

The setup is an 8G file called /var/tmp/random8 of random
data which is cached.

Then we create a new sparse file called /var/tmp/out and
copy random8 -> out.

With default settings:

$ rm -f /var/tmp/out; truncate -s 8G /var/tmp/out; free -m; time ./nbdkit file /var/tmp/out --run 'qemu-img convert -n -p -m 16 -W /var/tmp/random8 $nbd'; free -m ; cachestats /var/tmp/out
              total        used        free      shared  buff/cache   available
Mem:          32083        1181        6730           1       24171       30439
Swap:         16135          16       16119
    (100.00/100%)

real	0m6.267s
user	0m0.229s
sys	0m12.904s
              total        used        free      shared  buff/cache   available
Mem:          32083        1184         966           1       29932       30444
Swap:         16135          16       16119
pages in cache: 2038342/2097152 (97.2%)  [filesize=8388608.0K, pagesize=4K]

Notice after the run the new file is almost entirely in memory.

With fadvise=sequential cache=none:

$ rm -f /var/tmp/out; truncate -s 8G /var/tmp/out; free -m; time ./nbdkit file /var/tmp/out fadvise=sequential cache=none --run 'qemu-img convert -n -p -m 16 -W /var/tmp/random8 $nbd'; free -m ; cachestats /var/tmp/out
              total        used        free      shared  buff/cache   available
Mem:          32083        1178        6737           1       24167       30442
Swap:         16135          16       16119
    (100.00/100%)

real	0m8.791s
user	0m0.361s
sys	0m29.766s
              total        used        free      shared  buff/cache   available
Mem:          32083        1182        6716           1       24184       30438
Swap:         16135          16       16119
pages in cache: 3075/2097152 (0.1%)  [filesize=8388608.0K, pagesize=4K]

The overhead is large - about 40% - which I think is to be expected
given how we're doing this.  But it does end up with almost none of
the file in the cache.

So it works.  I think there's room for improvement here by using
double buffering or a background thread or something like that.  Hard
to implement this in the current plugin.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org