consuming pre-created tap - with multiqueue

[Miguel Duarte de Mora Barroso]

On Wed, Sep 23, 2020 at 6:59 PM Daniel P. Berrangé <berrange at redhat.com> wrote:
>
> On Wed, Sep 23, 2020 at 05:44:28PM +0100, Daniel P. Berrangé wrote:
> > On Tue, Sep 22, 2020 at 01:48:08PM +0200, Miguel Duarte de Mora Barroso wrote:
> > > Hello,
> > >
> > > On KubeVirt, we are trying to pre-create a tap device, then instruct
> > > libvirt to consume it (via the type=ethernet , managed='no'
> > > attributes).
> > >
> > > It works as expected, **unless** when we create a multi-queue tap device.
> > >
> > > The difference when creating the tap device is that we set the
> > > multi-queue flag; libvirt throws the following error when consuming
> > > it:
> > >
> > > ```
> > > LibvirtError(Code=38, Domain=0, Message='Unable to create tap device
> > > tap0: Invalid argument')
> > > ```
> > >
> > > After digging a bit on the libvirt code (we're using libvirt 6.0.0), I
> > > see this on the logs (immediately before the error):
> > > {"component":"virt-launcher","level":"info","msg":"Enabling
> > > IFF_VNET_HDR","pos":"virNetDevProbeVnetHdr:190","subcomponent":"libvirt","thread":"33"
> > > ,"timestamp":"2020-09-22T10:34:29.335000Z"}
> > >
> > > I do not understand how it can try to set the VNET_HDR flag, since I
> > > have not set it when I created it, which, as per [0] should only
> > > happen when requested. Here's the tap device I'm creating: (output of
> > > `ip tuntap show`)
> > >   - tap0: tap persist0x100 user 107 group 107
> >
> > IIUC the kernel code correctly, the VNET_HDR flag is not required
> > to be set when you first create the tap device - it appears to be
> > permitted any time you open a file descriptor for it.
> >
> > AFAIK, there is no problem with VNET_HDR, as it is a standard flag
> > we've set on all tap devices on Linux for 10 years.
>
> Looking at the kernel code, you need to set the MULTI_QUEUE flag
> at time you create the device and also set it when opening the
> device. In tun_set_iff():
>
>
>                 if (!!(ifr->ifr_flags & IFF_MULTI_QUEUE) !=
>                     !!(tun->flags & IFF_MULTI_QUEUE))
>                         return -EINVAL;
>
>
> so if you've configured QEMU to use multiqueue, the you need
> to use:
>
>   $ ip tuntap add dev mytap mode tap vnet_hdr multi_queue
>
> actually vnet_hdr doesn't matter as that can be set on the fly
> but multi_queue is mandatory.  Without it, I get the same EINVAL
> error as you mention.
>
Right, sorry for the noise; I found out one of our tests was abusing
the API, which caused the tap device to be created with multi-queue,
then when attempting to consume it, we were requesting a single vcpu.

My bad for not spotting this sooner. Thanks for the reply.

>
> Regards,
> Daniel
> --
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
>