[libvirt] [PATCH v2] Add support for virtio-net.tx_queue_size

Thu Jul 27 12:05:09 UTC 2017

On Thu, Jul 20, 2017 at 21:36:28 -0400, Laine Stump wrote:
> On 07/18/2017 07:12 AM, Michal Privoznik wrote:

[...]

> >>>>> @@ -5201,6 +5201,20 @@ qemu-kvm -net nic,model=? /dev/null
> >>>>>          <b>In general you should leave this option alone, unless you
> >>>>>          are very certain you know what you are doing.</b>
> >>>>>        </dd>
> >>>>> +      <dt><code>tx_queue_size</code></dt>
> >>>>> +      <dd>
> >>>>> +        The optional <code>tx_queue_size</code> attribute controls
> >>>>> +        the size of virtio ring for each queue as described above.
> >>>>> +        The default value is hypervisor dependent and may change
> >>>>> +        across its releases. Moreover, some hypervisors may pose
> >>>>> +        some restrictions on actual value. For instance, latest
> >>>>> +        QEMU (as of 2017-07-13) requires value to be a power of two
> >> Refer to a proper version of qemu when this is supported, not a date.
> >>
> >>>>> +        from [256, 1024] range.
> >>>>> +        <span class="since">Since 3.6.0 (QEMU and KVM only)</span><br/><br/>
> >>>> This is ridiculous. Since we can't figure out how to set this, how are
> >>>> users supposed to figure this out?
> >>> Well, you've cut off the line that reads;
> >>>
> >>>   <b>In general you should leave this option alone, unless you
> >>>   are very certain you know what you are doing.</b>
> >>>
> >>> So only users that know how virtio works under the hood are expected to
> >>> also know what rx/tx queue size is and how to set it. But frankly, I
> >> This statement is ridiculous by itself.
> > Why? There are more experienced users (for whom libvirt's just a stable
> > API) and less experienced ones (for whom libvirt's and tools on the top
> > of it are great for their automatic chose of parameters, e.g. gnome boxes).
> 
> Beyond that, that same statement (or something nearly identical) is
> repeated in multiple places throughout the XML documentation. There are
> at least two classes of these attributes that I can think of:
> 
> 1) attributes that are automatically set to a sane value by libvirt when
> not specified (and they usually *aren't* specified), and saved in the
> config XML in order to assure they are set the same every time the
> domain is started (to preserve guest ABI). These are intended to record
> whatever was the default setting for the attribute at the time the
> domain was created. For example, a pcie-root-port controller might have
> <model name='ioh3420'/> set, if that was the only type of pcie-root-port
> available at the time a domain was created; this comes in handy now that
> there is a generic pcie-root-port (which also has <model
> name='pcie-root-port'/>) - existing domains don't get their ABI screwed
> up when they're migrated to a newer host.
> 
> 2) knobs that have been added in over the years at the request/demand
> from below (qemu) and above (ovirt / openstack), many of them obscure,
> difficult to explain with any amount of useful detail, and almost never
> worthy of being manually set (and if you "don't know what you're doing",
> you're just as likely to make things worse as to make them better).
> 
> tx_queue_size is one of the latter.
> 
> For either of these types of attributes, they need to be documented so
> that someone encountering them (or actively searching them out) will at
> least have a basic idea what the attribute is named / used for, but we
> also need to warn the casual user to not mess with them. I don't see
> anything ridiculous about that.

[1]

> >>> think users setting this are always gonna go with the highest value
> >>> avaliable (1024). Such detailed description is a copy of rx_virtio_queue
> >>> size description which is result of review.
> >>>
> >>>> Is it really needed? How should it be configured? Can't we or qemu pick
> >>>> a sane value?
> >>>>
> >>> No. Some users need bigger virtio rings otherwise they see a packet
> >>> drop. So this is a fine tuning that heavily depends on the use case.
> >>> Thus libvirt should not try to come up with some value.
> >> That's why I think it's wrong.  What's the drawback of setting it to
> >> maximum? Which workloads will hit it? Why is the default not sufficient?
> >>
> >> And most notably, how do the users figure out that they need it?
> 
> 
> I think it's a bit too much burden on libvirt to expect that much detail
> to be included in formatdomain.html. Heck, I don't know if anyone even
> *knows* that much detail right now.

That proves my point kind of. If there isn't knowledge how to set up
this, then why even add the option. Is there really a point?

> > I'll leave this for MST to answer.
> >
> >> In this case, there are no anchor points that users can use to figure
> >> out if they need a setting like this. In addition putting in a warning
> >> that a setting should not be touched makes it rather useless.
> 
> I disagree. The setting is documented so that people can know that it
> exists. Anyone *really* interested in performance can look up
> information about it directly from the source and/or run their own
> performance tests. My understanding is that's what's done with settings
> like this, not just in libvirt and qemu, but in the Linux kernel too -
> people like Red Hat's performance testing groups will run a bunch of
> benchmark tests with different loads, comparing the test results with
> different settings of the tuning attributes, and publish white papers
> with recommendations for settings based on what loads a customer will be
> running on their systems (over time, these performance tests may
> discover that one particular setting is the best in *all* conditions,
> and in that case either qemu or libvirt can begin setting that as the
> default). But it's impractical to expect the person adding the knob to
> perform such performance testing and document the proper setting for the
> knob before it's even been pushed upstream. Instead, we add the knobs
> and let the perf testing teams know about them, then they run their
> barrage of tests and tell everyone what (if anything) to do with those
> knobs.

I'd expect that there is a very clearly defined benefit when adding such
thing. Adding a knob not doing anything is hardly worth the time.

> > Well, it can be viewed as counter part to rx_queue_size which we already
> > have.
> >
> >> Is there any writeup that we could point users to which would explain
> >> this feature?
> >>
> > I'm afraid there's nothing else than BZ I've linked and qemu patches.
> >
> 
> I think the documentation Peter is looking for will come later, from
> someone else.

The problem is that we are using the 'prior art' argument too often
without thinking whether something even makes sense. And then we have to
support it.

When adding a feature, there should be some clear benefit from it. When
fixing a bug it also should have some clear statement why it's
happening.

In this case we have a knob, which is not really clear how to set it
which may add performance benefits (that's the non-clear feature part)
and it may fix packets being lost (that's the bug part, but in default
configuration it's not fixed, nor clear when it occurs).

This is the ridiculous part [1].

I'm not going to object to adding this any more. I just wanted some
justification for adding this and I did not get a clear one.

Please at least add proper version information as requested in my first
reply when reposting.

No-longer-complained-against-by: Peter Krempa <pkrempa at redhat.com>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Digital signature
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20170727/9e46acb2/attachment-0001.sig>