Strange bdflush/swap issues

[Allen Smith]

Running on RHEL3 boxes with 2 processors and 3G of ram. Default bdflush and 
elvtune settings.

I am running java apps with big fixed heaps. Total memory used by JVMs and 
other system daemons is 2.1G.

These systems run fine during normal use, however once a night I compress old 
logs. These logs can be a gig or 2 in size uncompressed. To compress the logs 
I use:

'nice gzip -9 filename'

While this gzip is running, my jvms are stopped for periods of time which 
causes all kinds of horrible stuff to happen. It looks like the system is 
trying to swap out parts of the jvm that aren't active or it is blocking the 
few writes to disk that the JVM is doing (logging, async writes).

If I turn swap off, the system performs just fine during this time, no pauses. 

What is a good way to track down what is causing this? I think I basically 
don't want IO buffer pages to ever push app pages out of ram.

Thanks!
-Allen