<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style></style></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1>I’m running libvirt 0.10.2 and qemu-kvm-1.2.0, both compiled from source, on CentOS 6. I’ve got a working blkio cgroup hierarchy which I’m attaching guests to using the following XML guest configs:<o:p></o:p><o:p> </o:p>VM1 (foreground):<o:p></o:p><o:p> </o:p> <cputune><o:p></o:p> <shares>2048</shares><o:p></o:p> </cputune><o:p></o:p> <blkiotune><o:p></o:p> <weight>1000</weight><o:p></o:p> </blkiotune><o:p></o:p><o:p> </o:p>VM2 (background): <o:p></o:p><o:p> </o:p> <cputune><o:p></o:p> <shares>2</shares><o:p></o:p> </cputune><o:p></o:p> <blkiotune><o:p></o:p> <weight>100</weight><o:p></o:p> </blkiotune><o:p></o:p><o:p> </o:p>I’ve tested write throughput on the host using cgexec and dd, demonstrating that libvirt has correctly set up the cgroups:<o:p></o:p><o:p> </o:p>cgexec -g blkio:libvirt/qemu/foreground time dd if=/dev/zero of=trash1.img oflag=direct bs=1M count=4096 & cgexec -g blkio:libvirt/qemu/background time dd if=/dev/zero of=trash2.img oflag=direct bs=1M count=4096 &<o:p></o:p><o:p> </o:p>Snap from iotop, showing an 8:1 ratio (should be 10:1, but 8:1 is acceptable):<o:p></o:p><o:p> </o:p>Total DISK READ: 0.00 B/s | Total DISK WRITE: 91.52 M/s<o:p></o:p> TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND<o:p></o:p> 9602 be/4 root 0.00 B/s 10.71 M/s 0.00 % 98.54 % dd if=/dev/zero of=trash2.img oflag=direct bs=1M count=4096<o:p></o:p> 9601 be/4 root 0.00 B/s 80.81 M/s 0.00 % 97.76 % dd if=/dev/zero of=trash1.img oflag=direct bs=1M count=4096<o:p></o:p><o:p> </o:p>Further, checking the task list inside each cgroup shows the guest’s main PID, plus those of the virtio kernel threads. It’s hard to tell if all the virtio kernel threads are listed, but all the ones I’ve hunted down appear to be there.<o:p></o:p><o:p> </o:p>However, when running the same dd commands inside the guests, I get roughly-equal performance – nowhere near the ~8:1 relative bandwidth enforcement I get from the host: (background ctrl-c’d right after foreground finishes, both started within 1s of each other)<o:p></o:p><o:p> </o:p>[ben@foreground ~]$ dd if=/dev/zero of=trash1.img oflag=direct bs=1M count=4096<o:p></o:p>4096+0 records in<o:p></o:p>4096+0 records out<o:p></o:p>4294967296 bytes (4.3 GB) copied, 104.645 s, 41.0 MB/s<o:p></o:p><o:p> </o:p>[ben@background ~]$ dd if=/dev/zero of=trash2.img oflag=direct bs=1M count=4096<o:p></o:p>^C4052+0 records in<o:p></o:p>4052+0 records out<o:p></o:p>4248829952 bytes (4.2 GB) copied, 106.318 s, 40.0 MB/s<o:p></o:p><o:p> </o:p>I thought based on this statement: “Currently, the Block I/O subsystem does not work for buffered write operations. It is primarily targeted at direct I/O, although it works for buffered read operations.” from this page: <a href="https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/ch-Subsystems_and_Tunable_Parameters.html">https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/ch-Subsystems_and_Tunable_Parameters.html</a> that this problem might be due to host-side buffering, but I have that explicitly disabled in my guest configs:<o:p></o:p><o:p> </o:p> <devices><o:p></o:p> <emulator>/usr/bin/qemu-kvm</emulator><o:p></o:p> <disk type="file" device="disk"><o:p></o:p> <driver name="qemu" type="raw" cache="none"/><o:p></o:p> <source file="/path/to/disk.img"/><o:p></o:p> <target dev="vda" bus="virtio"/><o:p></o:p> <alias name="virtio-disk0"/><o:p></o:p> <address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x0"/><o:p></o:p> </disk><o:p></o:p><o:p> </o:p>Here is the qemu line from ps, showing that it’s clearly being passed through from the guest XML config:<o:p></o:p><o:p> </o:p>root 5110 20.8 4.3 4491352 349312 ? Sl 11:58 0:38 /usr/bin/qemu-kvm -name background -S -M pc-1.2 -enable-kvm -m 2048 -smp 2,sockets=2,cores=1,threads=1 -uuid ea632741-c7be-36ab-bd69-da3cbe505b38 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/background.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/path/to/disk.img,if=none,id=drive-virtio-disk0,format=raw,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=20,id=hostnet0,vhost=on,vhostfd=22 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:11:22:33:44:55,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -vnc 127.0.0.1:1 -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5<o:p></o:p><o:p> </o:p>For fun I tried a few different cache options to try to force a bypass the host buffercache, including writethough and directsync, but the number of virtio kernel threads appeared to explode (especially for directsync) and the throughput dropped quite low: ~50% of “none” for writethrough and ~5% for directsync.<o:p></o:p><o:p> </o:p>With cache=none, when I generate write loads inside the VMs, I do see growth in the host’s buffer cache. Further, if I use non-direct I/O inside the VMs, and inflate the balloon (forcing the guest’s buffer cache to flush), I don’t see a corresponding drop in background throughput. Is it possible that the cache="none" directive is not being respected? <o:p></o:p><o:p> </o:p>Since cgroups is working for host-side processes I think my blkio subsystem is correctly set up (using cfq, group_isolation=1 etc). Maybe I miscompiled qemu, without some needed direct I/O support? Has anyone seen this before?<o:p></o:p><o:p> </o:p>Ben Clay<o:p></o:p>rbclay@ncsu.edu<o:p></o:p><o:p> </o:p></div></body></html>