[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Memory allocation jump after running for a while with a large numberof threads

I have encountered a strange problem with memory usage of a thread-heavy Java application. I'm not sure if the problem lies in JVM, the kernel, or NPTL, but I cannot rule anything out, either -- perhaps someone can shed some light on it.

The application is a chat server that serves a large number of client connections, each connection requiring two threads. The number of threads oscillates from ~100 threads in early mornings to almost 4000 threads in the evening peaks. For days, the server runs in this setting without problems; the amount of virtual memory it consumes is proportional to the number of running threads (plus a constant that accounts for the Java heap size).

The problem is that, from time to time, the memory usage suddenly jumps by a huge amount, say by a gigabyte. So the application with 3000 threads that used to consume ~800M memory suddenly consumes ~1800M! That's alarming and strange for several reasons:

* The resident size doesn't reflect the jump at all;

* The jump cannot be attributed to the growth of the Java heap, as java is run with `-server -Xms250M -Xmx250M', i.e. with a clearly specified initial and maximum heap size that allows no resizing;

* The jump cannot be attributed to an accounting mistake; `pmap' shows that the memory is actually allocated, in multiple hundreds of 1024K chunks (in fact, each megabyte is allocated in two chunks, a larger 992K one and a smaller one).

* The jump cannot be explained as a consequence of having many threads because the application has been tested to sometimes run an even larger number of threads without the additional memory. Also, the application is still running, and I've measured that creating and destroying additional threads causes memory usage to rise/drop by exactly the expected amount (number of threads * stack size). The heap jump appears to be totally unrelated.

* The addresses of the chunks that belong to the "spiked" memory are too high to be part of the heap, although I suppose they could be returned by anonymous mmap.

Has anyone seen anything like this? Does a problem like this look like it's caused by the kernel, or by the JVM, or by NPTL? Any help would be appreciated.

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]