[Libguestfs] [PATCH virt-v2v] v2v: Allow temporary directory to be set on a global basis.

Tue Apr 7 11:25:02 UTC 2020

On Monday, 6 April 2020 19:45:44 CEST Daniel P. Berrangé wrote:
> On Thu, Apr 02, 2020 at 05:51:32PM +0200, Pino Toscano wrote:
> > On Thursday, 2 April 2020 17:30:39 CEST Richard W.M. Jones wrote:
> > > On Thu, Apr 02, 2020 at 03:21:14PM +0200, Pino Toscano wrote:
> > > > On Thursday, 2 April 2020 14:49:18 CEST Richard W.M. Jones wrote:
> > > > > Previously we placed large files in g#get_cachedir () (usually
> > > > > /var/tmp).  However the problem is this ties the libguestfs appliance
> > > > > and the virt-v2v overlay files to the same location.
> > > > > 
> > > > > When virt-v2v is run in a container, or any other situation where
> > > > > local storage is limited, it's helpful to be able to put the overlay
> > > > > files on an externally mounted PVC, which might be using NFS and
> > > > > shared between containers.  But putting the libguestfs appliance on
> > > > > NFS in a shared location is certainly not recommended.
> > > > > 
> > > > > This allows the two locations to be set separately:
> > > > > 
> > > > >   VIRT_V2V_TMPDIR - location of large temporary files, can use NFS
> > > > >                     and may be shared
> > > > > 
> > > > >   LIBGUESTFS_CACHEDIR - location of libguestfs appliance
> > > > > 
> > > > > Another motivation for this patch is to allow more reliable cleanup of
> > > > > temporary files by an external process, as described in the updated
> > > > > documentation.
> > > > > ---
> > > > 
> > > > I do not understand the motivation behind this, which adds yet another
> > > > location with temporary files in addition to:
> > > > - LIBGUESTFS_TMPDIR - $TMPDIR by default (which itself is /tmp by
> > > >   default)
> > > > - LIBGUESTFS_CACHEDIR - /var/tmp by default (with a .guestfs-$UID
> > > >   subdirectory for the appliance)
> > > > 
> > > > Before this patch, almost all the temporary files are stored directly
> > > > or in subdirectories of $TMPDIR, except big files such as overlays and
> > > > OVA extracted content that are in CACHEDIR. With the proposed changes,
> > > > _all_ the temporary files will be in CACHEDIR, so there are the
> > > > following problems:
> > > > - this directory will be cluttered with a lot more files than before
> > > > - if it is shared, then other places where it is mounted will see the
> > > >   same files
> > > > - if it is shared, then creating temporary files will possibly mean
> > > >   doing network I/O
> > > > - if virt-v2v exits uncleantly, there will be a lot more files to
> > > >   cleanup than now
> > > > - even without being shared, /var/tmp is persistent unlike /tmp (which
> > > >   can be tmpfs-backed on some distros/setups), meaning old temporary
> > > >   files will linger way more
> > > 
> > > How about if we confine the change to just large files (ie. ones
> > > which are currently placed in cachedir)?
> > 
> > This is already the case, isn't it?
> > 
> > > However the way that virt-v2v works at the moment means you cannot put
> > > the large files (especially v2vovl*.qcow2) in a different place from
> > > the libguestfs appliance.  This means that if you have only "some"
> > > space in /var/tmp -- enough for the appliance, but not enough for the
> > > potentially much larger space required by v2vovl*.qcow2 with multiple
> > > copies of virt-v2v running -- then you cannot separate the overlays to
> > > another directory.  This isn't just a problem for containers.
> > 
> > /var/tmp is a temporary directory, hence it ought to have enough space
> > for big temporary files. This is nothing special for libguestfs or
> > virt-v2v.
> 
> It is not valid to assume that /var/tmp is arbitrarily big enough
> for any files v2v might want to create. AFAIK, there's no real
> rule about how large it is expected to be. Linux distros typically
> don't do fine grained partitioning by default, so /var/tmp is often
> the same filesystem as /, but it is not unusual for admins to have
> / relatively small, and have a separate data partition somewhere
> which has the vast majority of storage space available to the host.
> 
> In the container world the / is indeed generally of a fairly
> limited size, as that's an unpacked container image filesystem
> usually on the local host storage.  Apps needing large storage
> capacities will usually be given an explicit volume they can
> use for this purpose.  NB, the provided volume may be persistent
> across container boots, or it may be freshly formatted on every
> boot. Whomever configures the container decides this based on
> the usage reuqirements of whatever app is to be run in the
> container.

Sure, I agree with that.

The important thing is still that that you need to have space for the
temporary files somewhere: be it /var/tmp, /mnt/scratch, whatever.
Because of this, and the fact that usually containers are created
fresh, the cache of the supermin appliance starts to make little sense,
and then a very simple solution is to point libguestfs to that extra
space:

  $ dir=$(mktemp --tmpdir -d /path/to/big/temporary/space/libguestfs.XXXXXX)
  $ export LIBGUESTGS_CACHEDIR=$dir
  $ export TMPDIR=$dir  # optionally
  $ libguestfs-test-tool
  [etc]
  $ rm -rf $dir

Easy to use, already doable, solves all the issues.

This whole problem started from a QE report on leftover files after
failed migrations: bz#1820282. What this report doesn't say, however,
is that beside the mentioned files that virt-v2v creates, there are
also leftover files that libguestfs itself creates. These files are
usually downloaded from the guest for the inspection, and generally not
that big compared to e.g. the overlays that virt-v2v creates.
Nonetheless, an abrupt exit of virt-v2v will leave those in place, and
they will still slowly fill up the space on /var/tmp (or whatever is
the location of $LIBGUESTFS_CACHEDIR).

Sure, creating a special directory for virt-v2v solves /some/ of the
issues, and most probably the ones that concern most from a disk space
POV. However, since we "uncovered" the issue and started to get our
hands on it, I don't think it would be ideal to create an ad-hoc
solution that solves just some.

-- 
Pino Toscano
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: This is a digitally signed message part.
URL: <http://listman.redhat.com/archives/libguestfs/attachments/20200407/1e5ac51d/attachment.sig>