[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Is irqbalance Redundant on 2.6 Kernels?

List users,

I have a server running Fedora Core 3 with the 2.6.12-1.1376_FC3smp kernel. It has four physical CPUs and I would like to balance the IRQs across them. It is a large scale VOIP server running Asterisk, so my main interest is balancing the IRQs generated by the ethernet device. Obviously, there is heavy network activity, so if the IRQs are not balanced the server will be CPU bound by the lone processor assigned to handle them.

Initially, I solved this problem by adding the following line to "/etc/rc.local" (82 is the IRQ of the ethernet device):

 echo 0f > /proc/irq/82/smp_affinity

This worked like a charm. The IRQs from the ethernet device were balanced across the four processors and each of their idle times was comparable. Unfortunately, when I duplicated this on our backup machine (adjusting the IRQ accordingly), I discovered that something was overwriting the value that I echoed to the "smp_affinity" file. The problem was that the primary machine had a service that included a fifteen second sleep in its init script. This service was started *after* the irqbalance daemon and *before* rc.local. Apparently irqbalance runs on a ten second interval and (this part is speculation based on my experience, as it is not the most well documented daemon out there) it only updates the smp affinity of ethernet devices on the first poll. All this led to irqbalance overwriting the value I echoed to the "smp_affinity" file on the backup machine, because it did not have the service with the sleep in its init script installed.

My first question is, what is the point of the irqbalance daemon if the kernel can handle balancing IRQs by itself? Googling for irqbalance brought me to the documentation for the Debian package <http://packages.debian.org/unstable/utils/irqbalance>, which states "Useful mostly just for 2.4 kernels, or 2.6 kernels with CONFIG_IRQBALANCE turned off." I grepped through the kernel source, but could not find this constant so as an experiment I turned off the irqbalance daemon and rebooted the system. Every "smp_affinity" file under "/proc/irq/*/" now contained the default value of "ffffffff" and I confirmed that all IRQs were being balanced across the CPUs by checking "/proc/interrupts".

Is it possible that irqbalance is just a holdover from the 2.4 kernel days? Can it be turned off on my kernel without any adverse affects?

I also ran the same experiment on a multi-processor SUSE box running the 2.6.5-52-smp kernel. This machine did not have irqbalance installed and the values in the "smp_affinity" files were all "ff". On this machine, all of the interrupts were being handled by CPU0. Apparently, SUSE's kernels are compiled without IRQ balancing enabled.

I've also done a little digging through "kernel-parameters.txt" and I came across a few interesting lines:

 acpi=     noirq -- do not use ACPI for IRQ routing

acpi_irq_balance [HW,ACPI] ACPI will balance active IRQs -- default in APIC mode pci= noacpi [IA-32] Do not use ACPI for IRQ routing or for PCI scanning.

noapic [SMP,APIC] Tells the kernel to not make use of any IOAPICs that may be present in the system.

 noirqbalance       [IA-32,SMP,KNL] Disable kernel irq balancing

This is all more than a little confusing to me. It appears that the kernel's IRQ balancing is contingent on ACPI and APIC, but I'm far from positive about this conclusion. If someone could explain to me how these different components interact with IRQ balancing, I would appreciate it. My fear is that in an attempt to slim down the server I will turn off a component that will disable or silently cripple IRQ balancing by the kernel.

Thank you,

Matthew Roth
InterMedia Marketing Solutions
Software Engineer and Systems Developer

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]