I/O Scheduling results in poor responsiveness

Chris Snook csnook at redhat.com
Wed Mar 5 19:35:41 UTC 2008


Nathan Grennan wrote:
>     Why is the command below all that is needed to bring the system to 
> it's knees? Why doesn't the io scheduler, CFQ, which is supposed to be 
> all about fairness starve other processes? Example, if I open a new file 
> in vim, and hold down "i" while this is running it will pause the 
> display of new "i"s for seconds, sometimes until the dd write is 
> completely finished. Another example is applications like firefox, 
> thunderbird, xchat, and pidgin will stop refreshing for 10+ seconds.
> 
>  dd if=/dev/zero of=test-file bs=2M count=2048
> 
>  I understand the main difference between using oflag=direct or not 
> relates to if the io scheduler is used, and if the file is cached or 
> not. I can see this clearly by watching cached rise without 
> oflag=direct, stay the same with it, and go way down when I delete the 
> file after running dd without oflag=direct.
> 
>  The system in question is running Fedora 8. It is an E6600, 4gb memory, 
> and 2x300gb Seagate sata drives. The drives are setup with md raid 1, 
> and the filesystem is ext3. But I also see this with plenty of other 
> systems with more cpu, less cpu, less memory, raid, and no raid.
> 
>  I have tried various tweaks to sys.vm settings, tried changing the 
> scheduler to as or deadline. Nothing seem to get it to behave, other 
> than oflag=direct.
> 
>  Using dd if=/dev/zero is just an easy test case.  I see this when 
> copying large files, creating large files, and using virtualization 
> software that does heavy i/o on large files.
> 
> 
> 
>  The command below seems to result in cpu idle 0 and io wait 100%. As 
> shown by "vmstat 1"
> 
> dd if=/dev/zero of=test-file bs=2M count=2048
> 
> 2048+0 records in
> 2048+0 records out
> 4294967296 bytes (4.3 GB) copied, 94.7903 s, 45.3 MB/s
> 
> 
>  The command below seems to work much better for responsiveness. The cpu 
> idle will be around 50, and the io wait will be around 50.
> 
> dd if=/dev/zero of=test-file2 bs=2M count=2048 oflag=direct
> 
> 2048+0 records in
> 2048+0 records out
> 4294967296 bytes (4.3 GB) copied, 115.733 s, 37.1 MB/s
> 

CFQ is optimized for throughput, not latency.  When you're doing dd without 
oflag=direct, you're dirtying memory faster than it can be written to disk, so 
pdflush will spawn up to 8 threads (giving it 8 threads' worth of CFQ time), 
which starves out vim's extremely frequent syncing of its session file.

The 8 threaded behavior of pdflush is a bit of a hack, and upstream is working 
on pageout improvements that should obviate it, but that work is still experimental.

vim's behavior is a performance/robustness tradeoff, and is expected to be slow 
when the system is doing a lot of I/O.

As for your virtualization, this is why most virtualization software (including 
Xen and KVM) allows you to use a block device, such as a logical volume, to 
which it can do direct I/O, which takes pdflush out of the picture.

Ultimately, if latency is a high priority for you, you should switch to the 
deadline scheduler.

-- Chris




More information about the fedora-list mailing list