[libvirt-users] How can a bridge be optimized?

[Andrey Korolyov]

On Tue, Jun 12, 2012 at 4:07 PM, Whit Blauvelt <whit at transpect.com> wrote:
> On Tue, Jun 12, 2012 at 11:35:28AM +0400, Andrey Korolyov wrote:
>
>> Just a stupid question: did you pin guest vcpus or NIC` hw queues?
>> Unpinned vm may seriously affect I/O performance when running on same
>> core set as NIC(hwraid/fc/etc).
>
> Thanks. No question is stupid. Obviously, this shouldn't be so slow at VM
> I/O so I'm missing something. In my defense, there is no single coherent set
> of documents on this stuff, unless those are kept in a secret place. It
> would be a fine thing if a few of the people who know all the "obvious"
> stuff about libvirt-based KVM configuration would collaborate and document
> it fully somewhere.
>
> When I Google "pinned vcpu" all the top responses are about Xen. I run KVM.
> I find mention that "KVM uses the linux scheduler for distributing workload
> rather than actually assigning physical CPUs to VMs," at
> http://serverfault.com/questions/235143/can-i-provision-half-a-core-as-a-virtual-cpu.
> Is that wrong? Are you suggesting RAID will end up on one CPU core, and if
> the VM ends up on the same core there's I/O trouble? I can see that for
> software RAID. But we're running hardware RAID. Isn't that handled in the
> hardware of the RAID controller? Isn't that the point of hardware
> RAID, to keep it off the CPU?
>
> In any case the slow IO is true of all the VMs, so presumably they're
> properly spread out over CPU cores (of which there are more than VMs, and
> each VM is configured to take only one), which would rule out the general
> problem being the shared use of any one core by other system processes.
>
> Whit

You partially right, HW raid do cpu offload, but it still need an
amount of interrupts proportional to load, and same for server NICs
with hardware queues - they help to manage data outside cpu, so cpu
spent less time doing the job for peripherals, but not too much. As
for my experience, any VM(xen|qemu) should be pinned to a different
set of cores than pointed by smp_affinity for rx|tx queues and other
peripherals, if not - depending on overall system load, you may get
even freeze(my experience with crappy LSI MegaSAS from time when
2.6.18 rocks) and almost every time network benchmark lost a large
portion of throughput. Another side point when doing such tune - use
your NUMA topology to achieve best performance - e.g. do NOT pin NIC`
interrupts on neighbor socket` cores or assign ten cores of one VM to
six real ones mixed between two NUMA nodes.