[Libguestfs] [PATCH nbdkit 0/2] Rewrite xz plugin as a filter.

[Richard W.M. Jones]

On Wed, Nov 21, 2018 at 09:59:51AM -0600, Eric Blake wrote:
> On 11/21/18 9:46 AM, Richard W.M. Jones wrote:
> >Matt asked if xz should really be a filter rather than a plugin.  The
> >answer is yes, of course it should be!  That's been something in the
> >todo file for a while.
> >
> >The commit converts the xz plugin code into a filter (leaving the
> >plugin around, but deprecating it).
> >
> >   plugin:  nbdkit xz file.xz
> >   filter:  nbdkit --filter=xz file file.xz
> >
> >   plugin:  # can't be done
> >   filter:  nbdkit --filter=xz curl url=https://example.com/disk.xz
> 
> And further:
> 
> nbdkit --filter=cache --filter=xz curl url=...
> 
> to take advantage of local caching rather than repeated curl requests.

The xz plugin includes a block cache, and now so does the filter.
(The plugin long predates our addition of filters into nbdkit).

I suppose there is a case for removing the block cache code from the
xz plugin, and relying on the cache filter instead.  I'll test that
out to see if it makes a difference.  It will certainly simplify the
xz filter code if we did that, but at the cost of making it a bit more
complex to use.

> >This is only lightly tested but it works for local files and for the
> >curl example given in the commit message.  Unfortunately because of
> >the very large block size used in the Fedora cloud image, the curl
> >example is barely usable.  We should get them to use a more reasonable
> >block size such as 16M (currently 192M).
> 
> This may be the first real random-access of a remote xz file that
> makes the argument for a smaller block size :)
> 
> Of course, when you switch to a smaller block size, the xz image
> can't compress quite as far, but hopefully the size difference is
> not that bad. Do you have actual numbers comparing the file size,
> vs. the speed changes made possible by the difference in block size?

In fact yes I do.  I measured < 1% overhead with a 16M block size (see
nbdkit-xz-plugin man page for details).  This is not particularly
surprising since 16M is still pretty large.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v