[libvirt] RFC [3/3]: Lock manager usage scenarios

[Saggi Mizrahi]

On Mon, 2010-09-13 at 13:35 +0100, Daniel P. Berrange wrote:
> On Fri, Sep 10, 2010 at 02:39:41PM -0600, Eric Blake wrote:
> > On 09/10/2010 10:01 AM, Daniel P. Berrange wrote:
> > >
> > >At libvirtd startup:
> > >
> > >   driver = virLockManagerPluginLoad("sync-manager");
> > >
> > >
> > >At libvirtd shtudown:
> > >
> > >   virLockManagerPluginUnload(driver)
> > 
> > Can you load more than one lock manager at a time, or just one active 
> > lock manager?  How does a user configure which lock manager(s) to load 
> > when libvirtd is started?
> 
> The intention is that any specific libvirt driver will only use
> one lock manager, but LXC vs QEMU vs UML could each use a different
> driver if required.
> 
> > >At guest startup:
> > >
> > >   manager = virLockManagerNew(driver,
> > >                               VIR_LOCK_MANAGER_START_DOMAIN,
> > >                               0);
> > >   virLockManagerSetParameter(manager, "id", id);
> > >   virLockManagerSetParameter(manager, "uuid", uuid);
> > >   virLockManagerSetParameter(manager, "name", name);
> > >
> > >   foreach disk
> > >     virLockManagerRegisterResource(manager,
> > >                                    VIR_LOCK_MANAGER_RESOURCE_TYPE_DISK,
> > >                                    disk.path,
> > >                                    ..flags...);
> > >
> > >   char **supervisorargv;
> > >   int supervisorargc;
> > >
> > >   supervisor = virLockManagerGetSupervisorPath(manager);
> > >   virLockManagerGetSupervisorArgs(&argv,&argc);
> > >
> > >   cmd = qemuBuildCommandLine(supervisor, supervisorargv, supervisorargv);
> > >
> > >   supervisorpid = virCommandExec(cmd);
> > >
> > >   if (!virLockManagerGetChild(manager,&qemupid))
> > >     kill(supervisorpid); /* XXX or leave it running ??? */
> > 
> > Would it be better to first try virLockManagerShutdown?  And rather than 
> > a direct kill(), shouldn't this be virLockManagerFree?
> 
> Yes I guess so
> 
> > >During migration:
> > >
> > >   1. On source host
> > >
> > >        if (!virLockManagerPrepareMigrate(manager, hosturi))
> > >            ..don't start migration..
> > >
> > >   2. On dest host
> > >
> > >       manager = virLockManagerNew(driver,
> > >                                   VIR_LOCK_MANAGER_START_DOMAIN,
> > >                                   VIR_LOCK_MANAGER_NEW_MIGRATE);
> > >       virLockManagerSetParameter(manager, "id", id);
> > >       virLockManagerSetParameter(manager, "uuid", uuid);
> > >       virLockManagerSetParameter(manager, "name", name);
> > >
> > >       foreach disk
> > >         virLockManagerRegisterResource(manager,
> > >                                        VIR_LOCK_MANAGER_RESOURCE_TYPE_DISK,
> > >                                        disk.path,
> > >                                        ..flags...);
> > 
> > So if there needs to be any relaxation of locks from exclusive to 
> > shared-write for the duration of the migration, that would be the 
> > responsibility of virLockManagerPrepareMigrate, and not done directly by 
> > libvirt?
> 
> As with my other reply on this topic, I didn't want to force a particular
> design / implementation strategy for migration, so I just put in actions
> at each key stage of migration. The driver impl can decide whether todo
> a plain release+reacquire, or use some kind of shared lock
> 
> > >       char **supervisorargv;
> > >       int supervisorargc;
> > >
> > >       supervisor = virLockManagerGetSupervisorPath(manager);
> > >       virLockManagerGetSupervisorArgs(&argv,&argc);
> > >
> > >       cmd = qemuBuildCommandLine(supervisor, supervisorargv, 
> > >       supervisorargv);
> > >
> > >       supervisorpid = virCommandExec(cmd);
> > >
> > >       if (!virLockManagerGetChild(manager,&qemupid))
> > >         kill(supervisorpid); /* XXX or leave it running ??? */
> > >
> > >   3. Initiate migration in QEMU on source and wait for completion
> > >
> > >   4a. On failure
> > >
> > >       4a1 On target
> > >
> > >             virLockManagerCompleteMigrateIn(manager,
> > >                                             VIR_LOCK_MANAGER_MIGRATE_CANCEL);
> > >             virLockManagerShutdown(manager);
> > >             virLockManagerFree(manager);
> > >
> > >       4a2 On source
> > >
> > >             virLockManagerCompleteMigrateIn(manager,
> > >                                             VIR_LOCK_MANAGER_MIGRATE_CANCEL);
> > 
> > Wouldn't this be virLockManagerCompleteMigrateOut?
> 
> Opps, yes.
> 
> > 
> > >
> > >   4b. On succcess
> > >
> > >
> > >       4b1 On target
> > >
> > >             virLockManagerCompleteMigrateIn(manager, 0);
> > >
> > >       42 On source
> > >
> > >             virLockManagerCompleteMigrateIn(manager, 0);
> > 
> > Likewise?
> 
> Yes
> 
> > >             virLockManagerShutdown(manager);
> > >             virLockManagerFree(manager);
> > >
> > >
> > >Notes:
> > >
> > >   - If a lock manager impl does just VM level leases, it can
> > >     ignore all the resource paths at startup.
> > >
> > >   - If a lock manager impl does not support migrate
> > >     it can return an error from all migrate calls
> > >
> > >   - If a lock manger impl does not support hotplug
> > >     it can return an error from all resource acquire/release calls
> > >
> > 
> > Overall, this looks workable to me.  As proposed, this assumes a 1:1 
> > relation between LockManager process and managed VMs.  But I guess you 
> > can still have a central manager process that manages all the VMs, by 
> > having the lock manager plugin spawn a simple shim process that does all 
> > the communication with the central lock manager.
> 
> I could have decided it such that it didn't assume presence of a angel
> process around each VM, but I think it is easier to be able to presume
> that there is one. It can be an incredibly thin stub if desired, so I
> don't think it'll be too onerous on implementations

> We are looking into the possibility of not having a process manage a
VM but rather having the sync_manager process register with a central
daemon and exec into qemu (or anything else) so assuming there is a
process per VM is essentially false. But the verb could be used for
"unregistering" the current instance with the main manager so the verb
does have it use.

Further more, even if we decide to leave the current 'sync_manager
process per child process' system as is for now. The general direction
is a central deamon per host for managing all the leases and guarding
all processes. So be sure to keep that in mind while assembling the API.
> 
> Daniel