[Libvir] [PATCH] virDomainMigrate version 4 (for discussion only!)

[Daniel P. Berrange]

On Wed, Jul 18, 2007 at 08:22:37PM +0100, Richard W.M. Jones wrote:
> This is version 4 of the virDomainMigrate patch.  It includes remote 
> support, but no qemu, virsh or python yet.  And it needs lots more 
> testing which I intend to do more of tomorrow.
> 
> Interface
> ---------
> 
> The interface is now this:
> 
> virDomainPtr
>   virDomainMigrate (virDomainPtr domain, virConnectPtr dconn,
>                     unsigned long flags, const char *dname,
>                     const char *uri, unsigned long resource);
> 
> The caller may set dname, uri and resource to 0/NULL and forget about 
> them.  Or else the caller may set, in particular, uri to allow for more 
> complicated migration strategies (especially for qemu).
> 
> https://www.redhat.com/archives/libvir-list/2007-July/msg00249.html
> 
> Driver support
> --------------
> 
> As outlined in the diagram in this email:
> https://www.redhat.com/archives/libvir-list/2007-July/msg00264.html
> migration happens in two stages.
> 
> Firstly we send a "prepare" message to the destination host.  The 
> destination host may reply with a cookie.  It may also suggest a URI (in 
> the current Xen implementation it just returns gethostname).  Secondly 
> we send a "perform" message to the source host.
> 
> Correspondingly, there are two new driver API functions:
> 
>   typedef int
>       (*virDrvDomainMigratePrepare)
>                       (virConnectPtr dconn,
>                        char **cookie,
>                        int *cookielen,
>                        const char *uri_in,
>                        char **uri_out,
>                        unsigned long flags,
>                        const char *dname,
>                        unsigned long resource);
> 
>   typedef int
>       (*virDrvDomainMigratePerform)
>                       (virDomainPtr domain,
>                        const char *cookie,
>                        int cookielen,
>                        const char *uri,
>                        unsigned long flags,
>                        const char *dname,
>                        unsigned long resource);



> To make this work in the remote case I have had to export two private 
> functions from the API which only remote should call:
>   __virDomainMigratePrepare
>   __virDomainMigratePerform
> The reason for this is that libvirtd is just linked to the regular 
> libvirt.so, so can only make calls into libvirt through exported symbols 
> in the dynamic symbol table.
> 
> There are two corresponding wire messages 
> (REMOTE_PROC_DOMAIN_MIGRATE_PREPARE and 
> REMOTE_PROC_DOMAIN_MIGRATE_PERFORM) but they just do dumb argument 
> shuffling, albeit rather complicated because of the number of arguments 
> passed in and out.
> 
> The complete list of messages which go across the wire during a 
> migration is:
> 
>   client -- prepare --> destination host
>   client <-- prepare reply -- destination host
>   client -- perform --> source host
>   client <-- perform reply -- source host
>   client -- lookupbyname --> destination host
>   client <-- lookupbyname reply -- destination host

I wonder where we should allow for multiple back-forth between prepare and
perform. The 'perform' method could return a tri-state - success, failure
or continue. In the latter case it would call prepare on the destination again
before calling perform on the source a second time. eg you might end up with
an exchange like

   1. --> client calls prepare at destination
   2. <-- destination returns some metadata
   3. --> client calls perform at source with metdata
   4. <-- source returns 'continue'
   5. --> client calls prepare are destination again
   6. <-- destination returns more metdata
   7. --> client calls perform at source with more metdata
   8. <-- source returns 'success'


In fact thinking about DV's suggestion that we should have a 'finalize'
method at the destination. If we generalized just one step more we could
handle this. Have just 2 methods

   virDrvDomainMigrateSource
   virDrvDomainMigrateDestination

Both methods return a tri-state, 'complete', 'failed', 'continue'. The impl
of virDomainMigrate starts with virDrvDomainMigrateDestination, then calls
virDrvDomainMigrateSource, then calls virDrvDomainMigrateDestination again,
and then virDrvDomainMigrateSource again, etc, etc. The sequence stops when
both virDrvDomainMigrateSource & virDrvDomainMigrateDestination have returned
'complete', or one of them has returned 'failure'. 

In the case of one returning 'failure' it is possible we might still need
to talk to the other end to initiate a recovery process.

Finally the 'cookie' param is currently an 'out' parameter for prepare, and 
an 'in' parameter for perform. If we allowed for the arbitrary back-and-forth,
then I think the cookie  should probably be an 'in' and 'out' parameter for 
both calls - to allow them to pass metadata in both directions.

Maybe this is all overkill, and hopefully, no impl would ever need more than a 
single call to either method, but I'm worried that we're designing this wire
level protocol with single exchange+cookie, based on a rough idea for a potential 
future impl in Xen which has not actually been implemented yet. 

I think it could be a useful future-proofing to allow for the arbitrary back-and-
forth at the wire level, could make use of it without needing to break client 
<-> daemon compatability.

Alternatively, we could hold-off on adding a migrate API to libvirt, and get
involved in upstream Xen to sort out secure migration for XenD. Then come back
and finalize the libvirt design based on more knowledge of what we're going to
be talking to.

> I have extended capabilities with <migration_features>.  For Xen this is:
> 
> <capabilities>
>   <host>
>    <migration_features>
>      <live/>
>      <uri_transports>
>        <uri_transport>tcp</uri_transport>
>      </uri_transports>
>    </migration_features>

Makes a lot of sense

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|