[libvirt] blkio cgroup

[Dominik Klein]

Hi

to be as verbose and clear as possible, I will write every command I use.

This sets everything up for the test:

# setup start

importantvms="kernel1 kernel2"
notimportantvms="kernel3 kernel4 kernel5 kernel6 kernel7 kernel8"

for vm in $importantvms $notimportantvms; do
virsh domstate $vm|grep -q running || virsh start $vm;
done

echo 1 > /sys/block/sdb/queue/iosched/group_isolation

mount -t cgroup -o blkio none /mnt
cd /mnt
mkdir important
mkdir notimportant

for vm in $notimportantvms; do
cd /proc/$(pgrep -f "qemu-kvm.*$vm")/task
for task in *; do
/bin/echo $task > /mnt/notimportant/tasks
done
done

for vm in $importantvms; do
cd /proc/$(pgrep -f "qemu-kvm.*$vm")/task
for task in *; do
/bin/echo $task > /mnt/important/tasks
done
done

#ls -lH /dev/vdisks/kernel[3-8]
#brw-rw---- 1 root root 254, 25 Feb 22 13:42 kernel3
#brw-rw---- 1 root root 254, 26 Feb 22 13:42 kernel4
#brw-rw---- 1 root root 254, 27 Feb 22 13:42 kernel5
#brw-rw---- 1 root root 254, 28 Feb 22 13:42 kernel6
#brw-rw---- 1 root root 254, 29 Feb 22 13:43 kernel7
#brw-rw---- 1 root root 254, 30 Feb 22 13:43 kernel8

for i in $(seq 25 30); do
/bin/echo 254:$i 10000000 >
/mnt/notimportant/blkio.throttle.write_bps_device
done

# setup complete

> Hm..., this sounds bad. If you have put a limit of ~10Mb/s then no
> "bo" is bad. That would explain that why your box is not responding
> and you need to do power reset.
> 
> - I am assuming that you have not put any throttling limits on root group.
>   Is your system root also on /dev/sdb or on a separate disk altogether.

No throttling on root. Correct.

system root is on sda
vms are on sdb

> - This sounds like a bug in throttling logic. To narrow it down can you
>   start running "deadline" on end device. If it still happens, it is more
>   or less in throttling layer.

cat /sys/block/sdb/queue/scheduler
noop deadline [cfq]
echo deadline > /sys/block/sdb/queue/scheduler
cat /sys/block/sdb/queue/scheduler
noop [deadline] cfq

This changes things:

vmstat before test (nothing going on):
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy
id wa
 1  0      0 130231696  17968  69424    0    0     0     0 16573 32834
0  0 100  0

vmstat during test while all 8 vms are writing (2 unthrottled, 6 at 10M):
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy
id wa
 2  9      0 126257984  17968  69424    0    0     8 164462 20114 36751
 5  4 52 39

vmstat during test when only throttled vms are still dd'ing
 2 21      0 124518928  17976  69424    0    0     0 63876 17410 33670
1  1 59 39

Load is 12-ish during the test and the results are as expected:
Throttled VMs write with approx 10M, unthrottled get like 80 each, which
sums up to about the 200M max capacity of the device for direct io.

> - We can also try to remove dm layers and just create partitions on 
>   /dev/sdb and export as virtio disks to virtual machines and take
>   dm layer out of picture and see if it still happens.

Did that. Also happens when scheduler is cfq. Also does not happen when
scheduler is deadline. So it does not seem to be a dm issue.

> - In one of the mails you mentioned that with 1 virutal machine throttling
>   READs and WRITEs is working for you. So it looks like 1 virtual machine
>   does not hang but once you launch 8 virtual machines it hangs. Can we
>   try increasing the number of vitual machines gragually and confirm that
>   it happens only if some certain number of virtual machines are launched.

Back on cfq here.

1 throttled vm: works, load ~4
2 throttled vms: works, load ~6
3 throttled vms: works, load ~9
4 throttled vms: works, load ~12
6 throttled vms: works, load ~20

The number of blocked threads increases with the number of vms dd'ing.

At the beginning of each test, the blocked threads number goes really
high (4 vms 160, 6 vms 220), but then drops significantly and stays low.

So it seems that when only throttled vms are running, the problem does
not occur.

1 throttled + 1 unthrottled vm: works, load ~5
2 throttled + 1 unthrottled vm: boom

Constantly 144 blocked threads, bo=0, load increasing to 144. System
needs power reset.

So, thinking about what I did in the initial setup, I re-tested without the

for vm in $importantvms; do
cd /proc/$(pgrep -f "qemu-kvm.*$vm")/task
for task in *; do
/bin/echo $task > /mnt/important/tasks
done
done

since I don't do anything with that "important" cgroup (yet) anyway. It
did not make a difference though.

> - Can you also paste me the rules you have put on important and non-important
>   groups. Somehow I suspect that some of the rule has gone horribly bad
>   in the sense that it is very low and effectively no virtual machine
>   is making any progress.

See setup commands in the beginning of this email.

> - How long does it take to reach in this locked state where bo=0.

It goes there "instantly", right after the dd commands start.

> - you can also try to redirect blktrace output to blkparse and redirect
>   it to standard output and see capture some output by copying pasting
>   last messages.

I hope this is what you meant:

blktrace -d /dev/sdb -o - | blkparse -i -

Output is in attachment.

Regards
Dominik
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: blktrace.txt
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20110222/8afb52b7/attachment-0001.txt>