[vfio-users] Disk Performance

Sun Apr 3 15:57:48 UTC 2016

Boy, I insist that I hate the extremely poor documentation. No matter how much time I spend with this, I can't get a perfect config setup, everytime I think I nailed it, someone else appears to burst my bubble.Before the other guy mail, I don't recall that even once I saw that there was something called -object iothread. I was aware that there was an experimental option for virtio-blk-pci called x-data-plane that could increase IOPS performance a lot, but not that it was already production-ready and that it transformed into two separate parameters:
http://comments.gmane.org/gmane.comp.emulators.qemu/279118
So basically, that's the latest syntax incarnation of x-data-plane=on. You need to define -object iothread,id=iothread0, then add it for your virtio-blk-pci with the parameter iothread=iothread0
-drive if=none,file=/dev/vg0/lv0,format=raw,id=drive0-object iothread,id=iothread0-device virtio-blk-pci,iothread=iothread0,drive=drive0
That's seems to be the minimalist but correct way to enable x-data-plane on latest QEMU.

Also, virtio-scsi-pci supports iothread parameter too (It was mentioned to be "in development" some years ago), so you can use it in an identical style. However, I found a weird QEMU warning that says "Guest moved index from 0 to 61440" when I do that, but its merely a warning, is not a fatal error like it happened to other people with a near identical message.
For example, this produces the warning:
#!/bin/bash
qemu-system-x86_64 \-name "Test VM" \-monitor stdio \-m 1024M \-M pc-q35-2.5,accel=kvm \-nodefaults \-device qxl-vga \-object iothread,id=iothread0 \-device virtio-scsi-pci,iothread=iothread0 \
Is nearly the lowest common denominator that causes the warning, just needs that virtio-scsi-pci has iothread parameter set. It does seems to work however, since with a more complex script including a -drive and a -device scsi-hd, both the SCSI Controller and the Hard Disk are visible with lspci and lsblk if booting from an Arch Linux ArchISO Live CD.
Here you have a mammoth script with two VirtIO SCSI Controllers that actually produces the "Guest moved index from 0 to 61440" warning twice:
#!/bin/bash
qemu-system-x86_64 \-name "Test VM" \-monitor stdio \-m 1024M \-M pc-q35-2.5,accel=kvm \-nodefaults \-drive if=none,file=archlinux-2016.03.01-dual.iso,readonly=on,format=raw,id=drive0-drive if=none,file=/dev/vg0/lv0,format=raw,id=drive1 \-drive if=none,file=filetest1.img,format=raw,id=drive2 \-drive if=none,file=filetest2.img,format=raw,id=drive3 \-device ide-cd,drive=drive0 \-device qxl-vga \-object iothread,id=iothread0 \-object iothread,id=iothread1 \-object iothread,id=iothread2 \-object iothread,id=iothread3 \-device virtio-scsi-pci,iothread=iothread0,id=scsi0 \-device virtio-scsi-pci,iothread=iothread1,id=scsi1 \-device scsi-hd,drive=drive2,bus=scsi0.0 \-device scsi-hd,drive=drive3,bus=scsi1.0 \-device virtio-blk-pci,iothread=iothread2,drive=drive1 \
Assuming no typos (And that you modify the -drive parameters to something else), it should produce two warnings on the Terminal. ArchISO sees in lsblk sda, sdb and vda, but didn't checked if sda and sdb works properly or not by formatting them and try to read/write data.You can also use info iothreads in QEMU Monitor, they appear if you use -object iothread regardless if they are used or not (There are 4 iothreads in that script, but iothread3 is not used).
Anyways, thanks to this new info, I'm one step closer to my perfect setup.

Date: Sat, 2 Apr 2016 22:26:27 -0400
Subject: Re: [vfio-users] Disk Performance
From: commendsarnex at gmail.com
To: zir_blazer at hotmail.com
CC: vfio-users at redhat.com

Hi,

Thanks for your detailed post. I believe I have already changed my script to make sense, thanks to the previous reply. I've attached it again below. For networking, I am using a wireless card where I cannot create a bridge, and my IP changes very often. Is there a simple way to manually create a tap with this? I wasn't able to get either of the 2 most popular tutorials working. 

Thanks,
sarnex

#!/bin/sh
export QEMU_AUDIO_DRV=pa
qemu-system-x86_64 -enable-kvm \
-m 5120 \
-cpu host \
-smp 6,sockets=1,cores=6,threads=1 \
-device vfio-pci,host=01:00.0,x-vga=on,multifunction=on \
-device vfio-pci,host=01:00.1 \
-vga none \
-device vfio-pci,host=00:12.0 \
-device vfio-pci,host=00:12.2 \
-device vfio-pci,host=00:16.0 \
-device vfio-pci,host=00:16.2 \
-soundhw ac97 \
-rtc base=localtime \
-netdev user,id=net0 \
-device virtio-net-pci,netdev=net0 \
-drive if=none,format=raw,cache=none,cache.direct=on,file=/media/500GB/win10.img,aio=native,id=ssd2,discard=off,detect-zeroes=off \
-object iothread,id=iothread2 \
-device virtio-blk-pci,drive=ssd2,request-merging=on,iothread=iothread2,modern-pio-notify=on,config-wce=off

On Sat, Apr 2, 2016 at 9:56 PM, Zir Blazer <zir_blazer at hotmail.com> wrote:

I spend a lot of time experimenting what works and what doesn't, through I never benchmarked anything to see performance differences.The main issue is that QEMU has a load of old syntaxis and parameters mixed with new ones that you must use to get the latest features, and documentation is extremely poor, so as an end user, I have absolutely no way to know what extra parameters I can tweak beyond the common sense ones. It takes A LOT of googling to understand how things works, and even then, it just to get them working, not even performance optimization.
The first thing you need to understand, is that just because QEMU seems to work and it starts your VM, does NOT MEANS that you get the things connected as you intend. For example, on your initial script, you are creating a VirtIO SCSI Controller, but you are NOT creating a scsi-hd to attach to it. Eventually, you will figure out that if you want to experiment, you need to use QEMU Monitor. Use -monitor stdio to have it at the Terminal you launch QEMU from. You can use info qtree to see a heavy Wall of Text that you will eventually figure out how to understand that tells you how things are connected, or basically, at what Bus or Controller each Device is attached.

At the moment, the current Storage Controllers contenders are these:
The PIIX3 IDE Controller provided by the i440FX platform, -device piix3-ide. Requires ide-hd/ide-cdThe ICH9 SATA Controller provided by the Q35 platform, which always runs in AHCI Mode (No IDE emulation) and supports NCQ, -device ahci-ich9. Requires ide-hd/ide-cdThe VirtIO Block Device, which is paravirtualized and the mainstream performance choice. -device virtio-blk-pciThe VirtIO SCSI Controller, which is also paravirtualized. Performance was nearly that of the VirtIO Block Device, but you could attach several SCSI HDs to a single controller, instead of needing one controller per drive. -device virtio-scsi-pci. Requires scsi-hd/scsi-cd
The Q35 ICH9 SATA Controller always runs in AHCI Mode, is not like the one you usually find in any Motherboard that you can set if you want IDE emulation. Basically, if you want to use Windows XP with it, you need to either slipstream the ICH9 AHCI Drivers with nLite, or pass to the VM a Floppy image with them to do the F6 installation procedure.The VirtIO Devices always need Drivers for Windows, easier way is to use the Fedora VirtIO Drivers ISO during Windows installation. VirtIO Block Device has Drivers for WXP, but VirtIO SCSI Controller does not, only Vista+.
Specifying Drives with the latest QEMU syntax is a two or three step procedure, and must be made in the following order on the command line:
1) You need to specify the host-side place that will represent the contents of the virtual Hard Disk. You do that with -drive.
-drive if=none,id=drive0,format=raw,file=/dev/vg0/w10x64
file= could be an entire Hard Disk (/dev/sdb), a physical partition (/dev/sda3), a LVM volume (/dev/vg0/lv0), a file (filebackedvm.img), etc. It can also be empty, which you can use if you want to create a CD-ROM drive that has no ISO inserted (Since ide-cd and scsi-cd requires a valid drive id parameter, but -drive doesn't requires a file=).
2) You need to create a Controller that will provide the Bus for the HDs that will contain those host-side routes to attach to:
-device virtio-scsi-pci,id=scsi0
On i440FX and Q35 you already have an IDE Bus thanks to the PIIX3 and ICH9. But we want to use the latest and greatest, so here you have the VirtIO SCSI Controller.
3) You need to create a Device that represents the virtual IDE or SCSI HD/CD with the host route from -drive, and attach it to the correct controller (IDE for either PIIX3 or ICH9, SCSI for... you guessed it, SCSI Controllers):
-device scsi-hd,drive=drive0,bus=scsi0
Important exception is the VirtIO Block Device, which is its own Controller and attaches the Drive to itself directly:
-device virtio-blk-pci,drive=drive0
There you have it working. Order is important since if you try to use -device scsi-hd before creating the SCSI Controller that provieds the SCSI Bus, QEMU will complain and refuse to create your VM.
Note that if there is only one SCSI Controller, you can avoid assigning manually bus= since there is only one SCSI Bus on the entire system, and QEMU will attach it to the only available controller, so you can merely do -device scsi-hd,drive=drive0. But if you were to use two SCSI Controllers, you may need to specify it so you get your intended topology. The more controllers you have, the more you may want to do things manually.For example, it happens when you use too many USB Devices. Even if you create two of the nec-usb-xhci Controllers, QEMU will by default try to saturate one Controller,, automatically creates an USB Hub for it, and keep attaching more USB Devices, instead of going to the second Controller. You will start to notice those very ugly things as you get friendly with QEMU Monitor info qtree.

Also important, -netdev user should be the least performing choice for Networking, and should be avoided unless you want to use it that way. The issue with QEMU Networking is that you need some understanding of how to configure a basic Bridge and TAP Device at the host side, and that may drastically change depending on your Linux distribution and the readily available tools, so I can't walkthrough those for you.For reference, you may want to check this Mail I sent to qemu-users. It has my current MACVLAN/MACVTAP (Which are supposed to be better than traditional Bridge + TAP)  templates for systemd-networkd in Arch Linux:http://lists.nongnu.org/archive/html/qemu-discuss/2016-03/msg00042.html 		 	   		  

_______________________________________________

vfio-users mailing list

vfio-users at redhat.com

https://www.redhat.com/mailman/listinfo/vfio-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20160403/2678f544/attachment.htm>