[libvirt] [PATCH] util: increase libnl buffer size to 1M

[Laine Stump]

On 07/11/2017 05:06 AM, Daniel P. Berrange wrote:
> On Mon, Jul 10, 2017 at 02:51:34PM -0400, John Ferlan wrote:
>>
>> On 06/29/2017 02:05 PM, ZhiPeng Lu wrote:
>>> nl_recv() returns the error "No buffer space available"
>>> when using virsh destroy domain with 240 or more
>>> passhthrough network interfaces.
>> pass-through

(Actually, neither is correct. If you're talking about devices that are
assigned with VFIO, then they are "SRIOV VFs assigned with VFIO"
("passthrough" is a misnomer left over from Xen), and if you're talking
about macvtap, then according to the macvtap spec, the proper term is
"passthru", while the libvirt XML calls them "passthrough" (and in
either case, you should stipulate that you're talking about macvtap
network interfaces).)

>>
>>> The patch increases libnl sock receive buffer size to 1M,
>>> and nl_recv() doesn't return error when destroying domain
>>> with 512 network interfaces.
>>>
>>> Signed-off-by: ZhiPeng Lu <lu.zhipeng at zte.com.cn>
>>> ---
>>>  src/util/virnetlink.c | 4 ++--
>>>  1 file changed, 2 insertions(+), 2 deletions(-)
>>>
>> This feels like something that perhaps should be configurable - that is
>> some /etc/libvirt/libvirtd.conf variable; otherwise, we'll keep hitting
>> some conflated maximum based on the size of something.
> 1 MB matches what systemed/udevd uses, so if we hit that limit, then  the
> system as a whole is going to struggle already. So I don't think we need
> make it configurable.

What bothers me about this (now that John's research has finally
reminded me of the complete gory history that I was deeply involved in
but had already totally forgotten) is that turning on netlink message
peeking (which we've done) is supposed to eliminate the need for a large
initial buffer (at the expense of making every read a bit less
efficient) *That* was supposed to be a permanent solution to the
problem, but apparently hasn't helped.

ZhiPeng - what version of libnl and kernel are you using? The message
history John points to reminded me that there are some versions of libnl
where message peeking doesn't work properly; maybe that's the issue and
the proper solution for you is to update your libnl. Or it's possible
that we need to do something else to make message peeking work properly?

(The latter doesn't seem likely - I just looked it up, and MSG_PEEK has
actually been enabled in libnl by default since this commit:

  
https://github.com/thom311/libnl/commit/55ea6e6b6cd805f441b410971c9dd7575e783ef4

which was in libnl 3.2.29 - even if we got it wrong, it should still be
enabled).

I *was* going to suggest that pushing this patch (rather than the one
making the initial buffer size configurable) was the best course of
action. But after reading through the history and remembering
everything, I've changed my mind - I think we need to figure out why
MSG_PEEK isn't working properly on your system. The first step is to
learn the version of libnl in use on the system that's failing.