[libvirt] [PATCH V4 4/6] libxl: support creating domain with VF assignment from a pool

[Jim Fehlig]

On 03/21/2016 02:11 AM, Chunyan Liu wrote:
> Add codes to support creating domain with network defition of assigning
> SRIOV VF from a pool.
>
> Signed-off-by: Chunyan Liu <cyliu at suse.com>
> ---
>  src/libxl/libxl_domain.c | 50 ++++++++++++++++++++++++++++++++++++++++++++++++
>  tests/Makefile.am        |  3 +++
>  2 files changed, 53 insertions(+)
>
> diff --git a/src/libxl/libxl_domain.c b/src/libxl/libxl_domain.c
> index c8d09b1..d11bf3a 100644
> --- a/src/libxl/libxl_domain.c
> +++ b/src/libxl/libxl_domain.c
> @@ -36,6 +36,7 @@
>  #include "virtime.h"
>  #include "locking/domain_lock.h"
>  #include "xen_common.h"
> +#include "network/bridge_driver.h"
>  
>  #define VIR_FROM_THIS VIR_FROM_LIBXL
>  
> @@ -764,6 +765,10 @@ libxlDomainCleanup(libxlDriverPrivatePtr driver,
>              if (net->ifname &&
>                  STRPREFIX(net->ifname, LIBXL_GENERATED_PREFIX_XEN))
>                  VIR_FREE(net->ifname);
> +
> +            /* cleanup actual device */
> +            virDomainNetRemoveHostdev(vm->def, net);
> +            networkReleaseActualDevice(vm->def, net);
>          }
>      }
>  
> @@ -960,6 +965,48 @@ libxlDomainCreateIfaceNames(virDomainDefPtr def, libxl_domain_config *d_config)
>      }
>  }
>  
> +static int
> +libxlNetworkPrepareDevices(virDomainDefPtr def)
> +{
> +    int ret = -1;
> +    size_t i;
> +
> +    for (i = 0; i < def->nnets; i++) {
> +        virDomainNetDefPtr net = def->nets[i];
> +        int actualType;
> +
> +        /* If appropriate, grab a physical device from the configured
> +         * network's pool of devices, or resolve bridge device name
> +         * to the one defined in the network definition.
> +         */
> +        if (networkAllocateActualDevice(def, net) < 0)
> +            goto cleanup;
> +
> +        actualType = virDomainNetGetActualType(net);
> +        if (actualType == VIR_DOMAIN_NET_TYPE_HOSTDEV &&
> +            net->type == VIR_DOMAIN_NET_TYPE_NETWORK) {
> +            /* Each type='hostdev' network device must also have a
> +             * corresponding entry in the hostdevs array. For netdevs
> +             * that are hardcoded as type='hostdev', this is already
> +             * done by the parser, but for those allocated from a
> +             * network / determined at runtime, we need to do it
> +             * separately.
> +             */
> +            virDomainHostdevDefPtr hostdev = virDomainNetGetActualHostdev(net);
> +            virDomainHostdevSubsysPCIPtr pcisrc = &hostdev->source.subsys.u.pci;
> +
> +            if (hostdev->mode == VIR_DOMAIN_HOSTDEV_MODE_SUBSYS &&
> +                hostdev->source.subsys.type == VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_PCI)
> +                pcisrc->backend = VIR_DOMAIN_HOSTDEV_PCI_BACKEND_XEN;
> +
> +            if (virDomainHostdevInsert(def, hostdev) < 0)
> +                goto cleanup;
> +        }
> +    }
> +    ret = 0;
> + cleanup:
> +    return ret;

Nothing to cleanup here. I think we should just return -1 on failure paths and 0
on success.

> +}
>  
>  /*
>   * Start a domain through libxenlight.
> @@ -1036,6 +1083,9 @@ libxlDomainStart(libxlDriverPrivatePtr driver, virDomainObjPtr vm,
>                                      vm, true) < 0)
>          goto cleanup;
>  
> +    if (libxlNetworkPrepareDevices(vm->def) < 0)
> +        goto cleanup;

I accessed a machine to test this series and have found a few problems.

1. When attaching an SR-IOV vf from a pool, the attach appears successful.
   I see that xen's pciback driver is bound to the device in the host. I
   didn't see the device in the guest (could be a guest problem), nor in
   dumpxml or /var/run/libvirt/libxl/<dom-name>.xml. Not surprisingly,
   this causes problems with device-detach

   # virsh detach-device dom sriov-pool-vif.xml
   error: Failed to detach device from sriov-pool-vif.xml
   error: operation failed: no device matching mac address 00:16:3e:7a:35:df found

2. After starting a domain containing an interface from sr-iov vf pool, the
   interface xml is changed from type='network' to type='hostdev'. E.g.
   before creating domain

    <interface type='network'>
      <source network='passthrough'/>
      <mac address='00:16:3e:7a:35:df'/>
    </interface>

   after creating domain

    <interface type='hostdev' managed='yes'>
      <mac address='00:16:3e:7a:35:df'/>
      <driver name='xen'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x0a' slot='0x11' function='0x1'/>
      </source>
    </interface>

   Maybe this is intended behavior, but it seems odd.

3. I started a domain containing an interface from sr-iov vf pool. Looks
   good. I then tried 'virsh detach-device ...', which never returned
   until keepalive timeout. The device was removed from the guest, but
   libvirtd got hung up in the process. Oddly, I wasn't able to interrupt
   libvirtd with gdb to see what its threads were doing.

Do you have access to an appropriate machine to test these scenarios, to see if
the issues are reproducible or just a problem with my configuration?

Regards,
Jim