[libvirt] RFC backup API

[Maxim Nestratov]

21.03.2016 13:36, Daniel P. Berrange пишет:
> On Mon, Mar 21, 2016 at 01:18:19PM +0300, Maxim Nestratov wrote:
>> Hi all,
>>
>> It's been already quite a long time since qemu implemented QMP
>> "drive-backup" command to create block devices backups. Even more, since
>> qemu 2.4 there is a possibility to create incremental backups. Though it is
>> possible to backup all attached to a domain disk drives by combining them
>> into a single QMP transaction command, this way of creating them, not to
>> mention managing, remains inconvenient for an end user of libvirt. Moreover,
>> creating a single drive backup via QMP interface isn't handy either. That
>> said, it looks reasonable to introduce a *new backup API* based on QMP
>> "drive-backup" facilities.
>>
>> Though we can start from a single simple function, allowing to create a disk
>> backup by means of QMP "drive-backup" command, I'd like to discuss here the
>> level of management libvirt could provide for backup operations. To begin
>> with, here is the preliminary list of possible functions that I think make
>> sense for libvirt API.
>>
>> virDomainCreateBackup - which creates a backup full/incremental of
>> all/selected disks,
>> virListBackups - which lists all backups created for a particular
>> domain/target,
>> virRestoreBackup - which restores all/selected disks from a backup,
>> virDeleteBackup - which deletes all/selected disks from a backup.
>>
>> It looks like backup management functions,  except create one, shouldn't be
>> or might not be bound to a particular domain and we could possibly leverage
>> storage pool API with some extension. Specifically, volume definition could
>> be extended with necessary meta data related to backups.
>>
>> The *question* is: if the whole idea about this new API as described above
>> or something similar makes sense?
> Well we certainly don't want applications going directly to the QMP
> monitor command to achieve this. So given demand for this kind of
> feature, some kind of facility will be needed in libvirt. The question
> is just what any API would look like.
>
>> If yes, then let's find out requirements for it (if any) and I will try to
>> prepare a patch set with the first RFC implementation to discuss the API in
>> more details. Looking forward for your opinions on the matter.
> Have you researched what VMWare/Hyper-V/VirtualBox APIs support in this
> area ? If they have any existing models, it'd be desirable to look at
> them to align our APIs where appropriate.

As far as I know VirtualBox doesn't have any backup API, at least there 
is no reference to it at [1].
What I know about VMWare is that they use an approach that involves 
temporary snapshot creation, providing an access to quiesced disk data 
and then deleting this temporary snapshot [2]. Thus, their approach is 
close to what libvirt can have with 3d party managed backups, i.e. setup 
an NBD server on qemu side and let 3d party backup software access it 
from outside.

[1] http://download.virtualbox.org/virtualbox/SDKRef.pdf
[2] 
http://pubs.vmware.com/vsphere-60/index.jsp#com.vmware.vddk.pg.doc/vddkBkupVadp.9.2.html?path=7_5_0_6_0_0#1014219

>
> Also how does this fit in with recent discussions on QEMU mailing list
> about enabling 3rd party vendor backup software by having QEMU expose an
> NBD server to do efficient sparse backups. That suggested a very different
> kind of API where libvirt would not manage the backups, but just provide
> an API to allow an NBD target to be enabled for the 3rd party to manage.

If you mean this thread [3] then as far as I understand, there is an 
agreement that NDB protocol gets extended a bit in such a way that 
external clients are able to get dirty block bitmap and this shouldn't 
be exposed in libvirt API.

For such scenarios libvirt can have the following calls:

virDomainStartBackup,
virDomainFinishBackup

When an external backup is started, qemu starts NBD server, allocates 
CBT (changed block tracking) bitmap and returns allocated NBD port to 
the caller. After that 3d party backup software is responcible for 
reading appropriate portion of data according to requested CBT bitmap.


[3] http://lists.nongnu.org/archive/html/qemu-devel/2016-03/msg03075.html

> Regards,
> Daniel

Thus, we have two ways of creating backups. The first is external (3d 
party managed), which is backed by QMP "blockdev-backup" command, which 
doesn't need from my point of view more than two calls mentioned above. 
The second is a another one, which is backed by QMP "drive-backup" 
command. My first letter covered only the latter case and I believe that 
this backup management API will be very useful. What is unclear still 
is: if there should be disk backup facilities only or whole VM backup as 
well, should libvirt be able to restore deleted VMs, should it be able 
to list created backups and so on. As for me, using storage pools with 
some extensions of storage pool API could perfectly solve this task. 
What do you think?

Maxim