Memory Performance Issue with Fedora Core 2 Kernels

[Gene Heskett]

On Friday 30 July 2004 17:46, James Foris wrote:
>James Wilkinson wrote:
>> James Foris wrote:
>>>I have run into an issue with memory bandwidth using the Fedora
>>> Core 2 kernels and I need help.  I don't know what is wrong, but
>>> something killed performance of my custom driver when I ported it
>>> from RedHat 7.3 to Fedora Core 2.  I believe I have narrowed it
>>> down to the kernel.
>>>
>>>My driver requires a large amount of contiguous physical memory
>>> for DMA from a PCI device.  I use the 'mem=YYY' command line
>>> parameter to reserve the top of physical RAM for my driver.  Then
>>> I allow mapping via mmap() calls to user space.  The user space
>>> app then uses this pointer to save the data to disk.
>>>
>>><snip>
>>>
>>>I have tried this on several platforms and kernels and the results
>>> vary, but the common denominator seems to be:
>>>
>>>   Fedora kernel + 32-bit Intel = poor performance (see below)
>>
>> I suggested:
>>>Wild guess: have you tried turning off the 4G+4G support in the
>>> kernel (requires a recompile)?
>>
>> James replied
>>
>>>I have rebuilt now w/o several patches; 4Kstacks, netdump, and
>>> 4g/4g.
>>>
>>>The problem has gone away.  I suspect that 4G/4G is broken.

I'm certainly leaning in that direction also.  Since turning on the 4G 
switch, I've had literally dozens of Oppsen, most of which have been 
so fatal as to require the use of the reset button, and in one case, 
the power button, to recover.  I don't believe I have achieved a 24 
hour uptime yet.

I currently am in contact with a person who is teaching me how to 
troubleshoot this, and running a 2.6.7 kernel that usually survives 
the Oops.  A 2.6.8-rc2 kernel is totally dead for nearly the same 
Oops, which is occuring in the middle of prune_dcahe().

>> It sounds more "broken as designed", to be honest. It sounds as
>> though your setup is doing a *lot* of context switches between
>> user mode and kernel mode. The basic trade-off that 4G/4G flushes
>> the TLB each context switch, so that kernel and user both get
>> nearly 4 GB, and TLB flushes are expensive.

Ouch!  Does this help explain why my old mobo runnng a 1400XP athlon 
could do 4.5 seti nits a day, and a 2800XP with this turned on is 
only doing 6?

>> See
>> http://lwn.net/Articles/39283/
>> and
>> http://kerneltrap.org/node/view/2891
>>
>> May I recommend fedora-devel-list at redhat.com or the LKML? You
>> might not have much done about it, but they might be able to
>> recommend workarounds.
>
>Actually, we have gotten some of the kernel group at Intel into the
> game, now.
>
>It seems that the problem can be completely reproduced using the
> native "/dev/mem" interface - no custom drivers needed.  The intel
> engineer that has looked at this believes that either the 4g/4g
> patch is broken, or that there are mem driver interactions with
> 4g/4g that no one has taken into account (which to me says that
> potentially both the mem driver and 4g/4g patch need fixing).
>
>Certainly, removing the 4g/4g patch fixes this problem.
>
>We have been told that Intel will be talking/working with the Red
> Hat people directly to resolve this.  They seem to think that this
> is something significant that should be resolved.
>
>> At the very least, it would be good if real-life problems with
>> 4G/4G are reported.
>>
>> James.

-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.24% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attorneys please note, additions to this message
by Gene Heskett are:
Copyright 2004 by Maurice Eugene Heskett, all rights reserved.