[Libvirt] Attach qemu commands to an running xml domain
Moshe Levi
moshele at nvidia.com
Wed Mar 31 14:01:38 UTC 2021
> -----Original Message-----
> From: Daniel P. Berrangé <berrange at redhat.com>
> Sent: Wednesday, March 31, 2021 3:23 PM
> To: Michal Privoznik <mprivozn at redhat.com>
> Cc: Waleed Musa <waleedm at nvidia.com>; libvir-list at redhat.com; Moshe
> Levi <moshele at nvidia.com>; Adrian Chiris <adrianc at nvidia.com>
> Subject: Re: [Libvirt] Attach qemu commands to an running xml domain
>
> External email: Use caution opening links or attachments
>
>
> On Wed, Mar 31, 2021 at 02:11:40PM +0200, Michal Privoznik wrote:
> > On 3/31/21 1:36 PM, Daniel P. Berrangé wrote:
> > > On Wed, Mar 31, 2021 at 01:09:32PM +0200, Michal Privoznik wrote:
> > > > On 3/30/21 11:35 AM, Waleed Musa wrote:
> > > > > Hi all,
> > > > >
> > > > > I see in libvirt you are supporting attach/detach devices to
> > > > > existing xml domain using *attachDeviceFlags *and
> *detachDeviceFlags *APIs.
> > > > > Now we are adding some qemu command to the xml domain related
> to
> > > > > some interfaces using alias names before starting the VM, but we
> > > > > will face an issue with hot plug such devices, so I have two question
> here:
> > > > >
> > > > > 1. Is it applicable to set the alias names for interfaces because I saw
> > > > > it's ignored when I add it to xml domain before starting the VM?
> > > > > 2. Is there a way or API to attach qemu commands to running domain
> as
> > > > > you are doing in attaching the device using
> > > > > *attachDeviceFlags?*
> > > > >
> > > > > *Example of my xml*
> > > > > *<domain type='kvm' id='5'
> > > > > xmlns:qemu='https://nam11.safelinks.protection.outlook.com/?url=
> > > > >
> http%3A%2F%2Flibvirt.org%2Fschemas%2Fdomain%2Fqemu%2F1.0&da
> t
> > > > >
> a=04%7C01%7Cmoshele%40nvidia.com%7C310b722c5d9b4dbef69e08d8f43fc
> > > > >
> 7c5%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637527902060468
> > > > >
> 241%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2lu
> MzIi
> > > > >
> LCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=mGgrFOieiNdozuu
> U
> > > > > 1tYThVFou2E4caoR0ECW1jljRfk%3D&reserved=0'>
> > > > > *
> > > > > * <devices>
> > > > > *
> > > > > <interface type='vhostuser'>
> > > > > <mac address='fa:16:3e:ac:12:4c'/>
> > > > > <source type='unix'
> > > > > path='/var/lib/vhost_sockets/sockbbb6bbe9-eb5' mode='server'/>
> > > > > <target dev='tapbbb6bbe9-eb'/>
> > > > > <model type='virtio'/>
> > > > > <driver queues='4' rx_queue_size='512' tx_queue_size='512'/>
> > > > > <alias name='net0'/>
> > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
> > > > > function='0x0'/>
> > > > > </interface>
> > > > > * </devices>
> > > > > *
> > > > > * <qemu:commandline>*
> > > > > <qemu:arg value='-set'/>
> > > > > <qemu:arg value='device.net0.page-per-vq=on'/>
> > > > > <qemu:arg value='-set'/>
> > > > > <qemu:arg value='device.net0.host_mtu=8942'/>
> > > > > * </qemu:commandline>*
> > > > > *</domain>*
> > > >
> > > > Is this perhaps related to the following bug?
> > > >
> > > > "interface type='vhostuser' libvirtError: Cannot set interface MTU"
> > > >
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2F
> > > >
> bugzilla.redhat.com%2Fshow_bug.cgi%3Fid%3D1940559&data=04%7C0
> 1
> > > >
> %7Cmoshele%40nvidia.com%7C310b722c5d9b4dbef69e08d8f43fc7c5%7C430
> 83
> > > >
> d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637527902060468241%7CUnkn
> own
> > > >
> %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1ha
> Ww
> > > >
> iLCJXVCI6Mn0%3D%7C1000&sdata=g8nFl0ygr2JdDz1juugnbC0dpfpIrKX
> NG
> > > > exEL6oMnDc%3D&reserved=0
> > > >
> > > > Are you trying to work around it? I am discussing with Moshe how
> > > > libvirt can help, but honestly, I don't like the solution I proposed.
> > > >
> > > > Long story short, the interface is in different container (among
> > > > with OVS
> > > > bridge) and thus when we query ovs-vsctl it connects to the system
> > > > one and doesn't find that interface. What I proposed was to allow
> > > > specifying path to ovs db.socket but this would need to be done o per
> domain basis.
> > >
> > > If it is possible to specify a ovs db.socket path in the XML, then
> > > why can't this path simply be bind mounted to the right location in
> > > the first place.
> >
> > Because then you'd override the path for system wide OVS. I mean, by
> > default the socket is under /var/run/openvswitch/db.sock and if you
> > have another OVS running inside a container you can expose it under
> > /prefix/var/run/openvswitch/db.sock. ovs-vsctl allows this by
> > --db=$path argument.
>
> Oh, right I was mis-understanding the scenario.
>
> I've always thought if we wanted to support use of resources from other
> namespaces, then that would involve use of a namespace="$PID"
> attribute to identify the namespace.
>
> <source type='unix' namespace="523532"
> path='/var/lib/vhost_sockets/sock7f9a971a-cf3' mode='server'/>
>
> IOW, libvirt could enter the namespace, and then talk to the OVS db.sock at
> its normal location.
>
> It would be bit strange having namespace refer to a different NS for the
> OVS, but using the path=/var/lib/..... from the current namespace, but as
> long as we define the semantics clearly its not the end of the world.
Just to better explain the use case here so we have user space vdpa solution which requires the following qemu flags to work:
1. page-per-vq=on
2. host_mtu
I understand the adding page-per-vq is feature requested to I am putting that aside.
Regarding the host_mtu we want to use the libvirt xml to set mtu like in [1] so we can set mtu in hot plug.
Basically we just want that the libvirt xml mtu will just set the qemu host_mtu.
Th problem is the this using mtu in libvirt xml will query ovs (it assume that if ovs installed on the host it needs to set it mtu).
In our case we have OVS running on the host for switching and OVS in container to connect the VF netdevice with vhostuser to qemu to just forward traffic from VF to virtio using vdpa)
1. One option is to allow to disable the set of mtu in ovs so it will just add the qemu host_mtu flag (we can set the mtu on ovs by openstack)
2. Second option is to change how we query ovs to add --db flag so we can change the query to be on the ovs in the container
Both option are fine with me.
[1] -
<interface type='vhostuser'>
<mac address='fa:16:3e:92:6d:79'/>
<source type='unix' path='/var/lib/vhost_sockets/sock7f9a971a-cf3' mode='server'/>
<model type='virtio'/>
<driver rx_queue_size='512' tx_queue_size='512'/>
<mtu size='8942'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
>
> Regards,
> Daniel
> --
> |:
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fberr
> ange.com%2F&data=04%7C01%7Cmoshele%40nvidia.com%7C310b722c
> 5d9b4dbef69e08d8f43fc7c5%7C43083d15727340c1b7db39efd9ccc17a%7C0%7
> C0%7C637527902060468241%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4w
> LjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&am
> p;sdata=SRttkiVBWP9rgqQBzgDsRQSKcUL1T%2B3Z0f96i4%2B4tm8%3D&
> reserved=0 -o-
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fww
> w.flickr.com%2Fphotos%2Fdberrange&data=04%7C01%7Cmoshele%40
> nvidia.com%7C310b722c5d9b4dbef69e08d8f43fc7c5%7C43083d15727340c1b7
> db39efd9ccc17a%7C0%7C0%7C637527902060468241%7CUnknown%7CTWFpb
> GZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI
> 6Mn0%3D%7C1000&sdata=v8hkQ6F4lwKoECekJNt4WHgpReHo8GzMPz
> YbW8wv5V0%3D&reserved=0 :|
> |:
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flibvir
> t.org%2F&data=04%7C01%7Cmoshele%40nvidia.com%7C310b722c5d9b
> 4dbef69e08d8f43fc7c5%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%
> 7C637527902060468241%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAw
> MDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sda
> ta=y%2F%2BTU%2FG6qz5kdYeA8KVzZEMzhH6zwp3%2F639ylpIIT1Y%3D&am
> p;reserved=0 -o-
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Ffsto
> p138.berrange.com%2F&data=04%7C01%7Cmoshele%40nvidia.com%7C
> 310b722c5d9b4dbef69e08d8f43fc7c5%7C43083d15727340c1b7db39efd9ccc17
> a%7C0%7C0%7C637527902060468241%7CUnknown%7CTWFpbGZsb3d8eyJWI
> joiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1
> 000&sdata=WtiEpRFS83Hp1Ya9pzjvjGsVDD5SyBPdpHRHKPOop24%3D&
> amp;reserved=0 :|
> |:
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fenta
> ngle-
> photo.org%2F&data=04%7C01%7Cmoshele%40nvidia.com%7C310b722c
> 5d9b4dbef69e08d8f43fc7c5%7C43083d15727340c1b7db39efd9ccc17a%7C0%7
> C0%7C637527902060468241%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4w
> LjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&am
> p;sdata=Xok16UY9Qt2QY31nHgu01hR0ykuduqwoJeRR5K3NnUM%3D&r
> eserved=0 -o-
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fww
> w.instagram.com%2Fdberrange&data=04%7C01%7Cmoshele%40nvidia.
> com%7C310b722c5d9b4dbef69e08d8f43fc7c5%7C43083d15727340c1b7db39e
> fd9ccc17a%7C0%7C0%7C637527902060468241%7CUnknown%7CTWFpbGZsb3
> d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0
> %3D%7C1000&sdata=GtgEbUuRNH5WPXi%2F5PkE74aEbh8N7mEwJ8ntE
> te3%2FfM%3D&reserved=0 :|
More information about the libvir-list
mailing list