[libvirt] RFC: Migration with NPIV

[Osier Yang]

On 2012年11月19日 17:30, Osier Yang wrote:
> Hi,
>
> This proposal is trying to figure out a solution for migration
> of domain which uses LUN behind vHBA as disk device (QEMU
> emulated disk only at this stage). And other related NPIV
> improvements which are not related with migration. I'm not
> luck to get a environment to test if the thoughts are workable,
> but I'd like see if guys have good idea/suggestions earlier.
>
> 1) Persistent vHBA support
>
> This is the useful stuff missed for long time. Assuming
> that one created a vHBA, did masking/zoning, everything works
> as expected. However, after a system rebooting, everything is
> just lost. If the user wants to get things back, he has to
> find out the preivous WWNN & WWPN, and create the vHBA again.
>
> On the other hand, Persistent vHBA support is actually required
> for domain which uses LUN behind a vHBA. Othewise the domain
> could fail to start after a system rebooting.
>
> To support the persistent vHBA, new APIs like virNodeDeviceDefineXML,
> virNodeDeviceUndefine is required. Also it's useful to introduce
> "autostart" for vHBA, so that the vHBA could be started automatically
> after system rebooting.
>
> Proposed APIs:
>
> virNodeDevicePtr
> virNodeDeviceDefineXML(virConnectPtr conn,
> const char *xml,
> unsigned int flags);
>
> int
> virNodeDeviceUndefine(virConnectPtr conn,
> virNodeDevicePtr dev,
> unsigned int flags);
>
> int
> virNodeDeviceSetAutostart(virNodeDevicePtr dev,
> int autostart,
> unsigned int flags);
>
> int
> virNodeDeviceGetAutostart(virNodeDevicePtr dev,
> int *autostart,
> unsigned int flags);

One API missed is:

   int
   virNodeDeviceCreate(virNodeDevicePtr dev,
                       unsigned int flags);

   To create the vHBA.

>
> 2) Associate vHBA with domain XML
>
> There are two ways to attach a LUN to a domain: as an QEMU emulated
> device; or passthrough. Since passthrough a LUN is not supported in
> libvirt yet, let's focus on the emulated LUN at this stage.
>
> New attributes "wwnn" and "wwpn" are introduced to indicate the
> LUN behind the vHBA. E.g.
>
> <disk type='block' device='disk'>
> <driver name='qemu' type='raw'/>
> <source wwnn="2001001b32a9da4e" wwpn="2101001b32a90004"/>
> <target dev='vda' bus='virtio'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
> </disk>
>
> Before the domain starting, we have to check if there is LUN
> assigned to the vHBA, error out if not.
>
> Using the stable path of LUN also works, e.g.
>
> <source dev="/dev/disk/by-path/pci-0000\:00\:07.0-scsi-0\:0\:0\:0"/>
>
> But the disadvantage is the user have to figure out the stable
> path himself; And we have to do checking of every stable path to
> see if it's behind a vHBA in migration "Begin" stage. Or an new
> XML tag for element "source" to indicate that it's behind a vHBA?
> such as:
>
> <source dev="disk-by-path" model="vport"/>
>
> 3) Migration with vHBA
>
> One possible solution for migration with vHBA is to use one pair
> of WWNN & WWPN on source host, one is using for domain, one is
> reserved for migration purpose. It requires the storage admin maps
> the same LUN to the two vHBAs when doing the masking and zoning.
>
> One of the two vHBA is called "Primary vHBA", another is called
> "secondary vHBA". To maitain the relationship between these two
> vHBAs, we have to introduce new XMLs to vHBA. E.g.
>
> In XML of primary vHBA:
>
> <secondary wwpn="2101001b32a90004"/>
>
> In XML of secondary vHBA:
>
> <primary wwpn="2101001b32a90002"/>
>
> Primary vHBA is going to be guaranteed not used by any domain which
> is driven by libvirt (we do some checking eariler before the domain
> starting). And it's also guaranteed that the LUN can't be used by
> other domain with sVirt or Sanlock. So it's safe to have two vHBAs
> on source host too.
>
> To prevent one using the LUN by creating vHBA using the same WWNN &
> WWPN on another host, we must create the secondary vHBA on source
> host, even it's not being used.
>
> Both primary and secondary vHBA must be defined and marked as
> "autostart" so that the domain could be started after system
> rebooting.
>
> When do migration, we have to bake a bigger cookie with secondary
> vHBA's info (basically it's WWNN and WWPN) in migration "Begin"
> stage, and eat that in migration "Prepare" stage on target host.
>
> In "Begin" stage, the XMLs represents the secondary vHBA is
> constructed. And the secondary vHBA is destoyed on source host,
> not undefined though.
>
> In "Prepare" stage, a new vHBA is created (define and start)
> on target host with the same WWNN & WWPN as secondary vHBA on
> source host. The LUN then should be visible to target host
> automatically? and thus migration can be performed. After migration
> is finished on target host, the primary vHBA on source host is
> destroyed, not undefined.
>
> If migration fails, the new vHBA created on target host will
> be destroyed and undefined. And both primary and secondary
> vHBA on source host will be started, so that the domain could
> be resumed.
>
> Finally if migration succeeds, primary vHBA on source host
> will be transtered to target host as secondary vHBA (defined).
> And both primary and secondary vHBA on source host will be
> undefined.
>
> 4) Enrich HBA's XML
>
> It's hard to known the vHBAs created from a HBA with current
> implementation. One have to dump XML of each (v)HBAs and find
> out the clue with element "parent" of vHBAs. It's good to introduce
> new element for HBA like "vports", so that one can easily known
> what (how many) vHBAs are created from the HBA?
>
> And also it's good to have the maximum vports the HBA supports.
>
> Except these, other useful information should be exposed too,
> such as the vendor name, the HBA state, PCI address, etc.
>
> The new XMLs should be like:
>
> <vports num='2' max='64'>
> <vport name="scsi_host40" wwpn="2101001b32a90004"/>
> <vport name="scsi_host40" wwpn="2101001b32a90005"/>
> </vports>
> <online/>
> <vendor>QLogic</vendor>
> <address type="pci" domain="0" bus="0" slot="5" function="0"/>
>
> "online", "vendor", "address" make sense to vHBA too.
>
> 5) Improve the way to lookup LUN's stable path
>
> Currently, to lookup the LUN's stable path with WWNN & WWPN,
> it needs to iterate over the sysfs each time, maintaining the
> stable path in vHBA's XML doesn't make sense, as the LUN assigned
> to the vHBA could be changed as the storage admin's mood. I'm
> wondering if there is a way to notify the change asynchronously,
> if there is, then maintaining the stable path internally make
> sense.
>
> 6) Miscellaneous
>
> This is only about QEMU emulated device, passthroughed scsi_host
> with vHBA is still not covered, we have to support the vHBA passthough
> first. The good thing is the solution should be similiar.
>
> Regards,
> Osier
>
> --
> libvir-list mailing list
> libvir-list at redhat.com
> https://www.redhat.com/mailman/listinfo/libvir-list