[dm-devel] Potential enhancements to dm-thin v2

[Demi Marie Obenour]

On Mon, Apr 11, 2022 at 10:16:02AM +0200, Zdenek Kabelac wrote:
> Dne 11. 04. 22 v 0:03 Demi Marie Obenour napsal(a):
> > For quite a while, I have wanted to write a tool to manage thin volumes
> > that is not based on LVM.  The main thing holding me back is that the
> > current dm-thin interface is extremely error-prone.  The only per-thin
> > metadata stored by the kernel is a 24-bit thin ID, and userspace must
> > take great care to keep that ID in sync with its own metadata.  Failure
> > to do so results in data loss, data corruption, or even security
> > vulnerabilities.  Furthermore, having to suspend a thin volume before
> > one can take a snapshot of it creates a critical section during which
> > userspace must be very careful, as I/O or a crash can lead to deadlock.
> > I believe both of these problems can be solved without overly
> > complicating the kernel implementation.
> 
> 
> Hi
> 
> These things are coming with initial design of whole DM world - where there
> is a split of complexity between kernel & user-space. So projects like
> btrfs, ZFS, decided to go the other way and create a monolithic 'all-in-one'
> solution, where they avoid some problems related with communication between
> kernel & user-space - but at the price of having a pretty complicated and
> very hard to devel & debug  kernel code.
> 
> So let me explain one of the reasons, we have this logic with suspend is
> this basic principle:
> 
> write new lvm metadata ->  suspend (with all table preloads) ->  commit  new
> lvm2 metadata -> resume
> 
> with this we ensure the user space maintain the only valid 'view' of metadata.
> 
> Your proposal actually breaks this sequence and would move things to the
> state of  'guess at which states we are now'. (and IMHO presents much more
> risk than virtual problem with suspend from user-space - which is only a
> problem if you are using suspended device as 'swap' and 'rootfs' - so there
> are very easy ways how to orchestrate your LVs to avoid such problems).

The intent is less “guess what states we are now” and more “It looks
like dm-thin already has the data structures needed to store some
per-thin metadata, and that could make writing a simple userspace volume
manager FAR FAR easier”.  It appears to me that the only change needed
would be reserving some space (amount fixed at pool creation) after
‘struct disk_device_details’ for use by userspace, and providing a way
for userspace to enumerate the thin devices on a volume and to set and
retrieve that extra data.  Suspend isn’t actually that big of a problem,
since new Qubes OS 4.1 (and later) installs use one pool for the root
filesystem and a separate one for VMs.  As a userspace writer, the
scariest part of managing thin volumes is actually making sure I don’t
lose track of which thin ID corresponds to which volume name.  The
*only* metadata Qubes OS would need would be a per-thin name, size, thin
ID, and possibly UUID.  All of those could be put in that extra space.

> Basically you are essentially wanting to move whole management into kernel
> for some not so great speed gains (related to the rest of the running system
> (and you can certainly do that by writing your own kernel module to manage
> your ratehr unique software problem)

From a storage perspective, my problem is basically the same as Docker’s
devicemapper driver.  Unlike Docker, though, Qubes OS must work at the
block level; it can’t work at the filesystem level.  So overlayfs and
friends aren’t options.

> But IMHO creation and removal of thousands of devices in very short period
> of time rather suggest there is something sub-optimal in your original
> software design as I'm really having hard time imagining why would you need
> this ?

There very well could be (suggestions for improvement welcome).

> If you wish to operate lots of devices - keep them simply created and ready
> - and eventually blkdiscard them for next device reuse.

That would work for volatile volumes, but those are only about 1/3 of
the volumes in a Qubes OS system.  The other 2/3 are writable snapshots.
Also, Qubes OS has found blkdiscard on thins to be a performance
problem.  It used to lock up entire pools until Qubes OS moved to doing
the blkdiscard in chunks.

> I'm also unsure from where would arise any special need to instantiate  that
> many snapshots -  if there is some valid & logical purpose -   lvm2 can have
> extended user space API to create multiple snapshots at once maybe (so
> i.e.    create  10 snapshots   with      name-%d  of a single thinLV)

This would be amazing, and Qubes OS should be able to use it.  That
said, Qubes OS would prefer to be able to choose the name of each volume
separately.  Could there be a more general batching operation?  Just
supporting ‘lvm lvcreate’ and ‘lvm lvs’ would be great, but support for
‘lvm lvremove’, ‘lvm lvrename’, ‘lvm lvextend’, and ‘lvm lvchange
--activate=y’ as well would be even better.

> Not to mentioning operating that many thin volumes from a single thin-pool
> is also nothing close to high performance goal you try to reach...

Would you mind explaining?  My understanding, and the basis of
essentially all my feature requests in this area, was that virtually all
of the cost of LVM is the userspace metadata operations, udev syncing,
and device scanning.  I have been assuming that the kernel does not have
performance problems with large numbers of thin volumes.

Right now, my machine has 334 active thin volumes, split between one
pool on an NVMe drive and one on a spinning hard drive.  The pool on an
NVMe drive has 312 active thin volumes, of which I believe 64 are in use.
Are these numbers high enough to cause significant performance
penalties for dm-thin v1, and would they cause problems for dm-thin v2?
How much of a performance win can I expect from only activating the
subset of volumes I actually use?

Also, I believe a significant fraction of I/O is writes to previously
unallocated blocks.  I haven’t measured how much, though, since I am not
aware of any way to get that statistic, at least without kprobes or
similar.

The pool on a spinning hard drive has 22 thin volumes, of which I
believe only one is in use.  The HDD is mostly used for backups, so its
performance doesn’t matter that much.

-- 
Sincerely,
Demi Marie Obenour (she/her/hers)
Invisible Things Lab
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/dm-devel/attachments/20220411/c1235fdb/attachment.sig>