[Linux-cachefs] Another Bug with cachefs during underlaying nfs server timeout

Romain DEGEZ romain.degez at smartjog.com
Mon May 3 13:24:01 UTC 2010


Hi David,

We faced a network outage this week and as a result, our 4 servers using 
cachefs went in a very bad shape.
Even when the nfs-server came back and while I was able to log in the server 
and browse the nfs mount, the whole cachefs part was ****ed-up.

Appart from this failure, we have not seen any problem for the past weeks.

Attached trace2.txt from one of the failed server. (running debian 
2.6.32-4-686 kernel with all your patches).

Hope this may helps

Regards,

-- 
RD
-------------- next part --------------
[1331607.427269] BUG: Bad page state in process cachefilesd  pfn:384e26                  
[1331607.427302] page:ffffea000c511850 flags:0200000000001008 count:0 mapcount:0 mapping:(null) index:2ea4
[1331607.427350] Pid: 1893, comm: cachefilesd Tainted: G    B      2.6.33-2-amd64 #1                      
[1331607.427394] Call Trace:                                                                              
[1331607.427419]  [<ffffffff810b1e09>] ? bad_page+0x116/0x129                                             
[1331607.427447]  [<ffffffff810b22e9>] ? free_pages_check+0x38/0x57                                       
[1331607.427476]  [<ffffffff810b34f6>] ? free_hot_cold_page+0x46/0x176                                    
[1331607.427512]  [<ffffffff810b368f>] ? __pagevec_free+0x69/0x80                                         
[1331607.427546]  [<ffffffff810b66fa>] ? release_pages+0x17b/0x18d                                        
[1331607.427575]  [<ffffffff81034b44>] ? __wake_up_common+0x44/0x73                                       
[1331607.427604]  [<ffffffff810387e3>] ? __wake_up+0x30/0x44                                              
[1331607.427636]  [<ffffffff810b6acd>] ? __pagevec_release+0x19/0x22                                      
[1331607.427665]  [<ffffffff810b724c>] ? truncate_inode_pages_range+0xdb/0x2b0                            
[1331607.427696]  [<ffffffff810df692>] ? virt_to_head_page+0x9/0x2a                                       
[1331607.427730]  [<ffffffffa0275ca1>] ? jbd2_journal_stop+0x222/0x235 [jbd2]                             
[1331607.427763]  [<ffffffff812e9ccb>] ? mutex_lock+0xd/0x33                                              
[1331607.427797]  [<ffffffffa0291033>] ? ext4_delete_inode+0x5d/0x246 [ext4]                              
[1331607.427833]  [<ffffffffa0290fd6>] ? ext4_delete_inode+0x0/0x246 [ext4]                               
[1331607.427864]  [<ffffffff810fa9d9>] ? generic_delete_inode+0xdc/0x168                                  
[1331607.427893]  [<ffffffff810f8faf>] ? d_delete+0x47/0xb2                                               
[1331607.427922]  [<ffffffff810f1fb5>] ? vfs_unlink+0x9f/0xa6                                             
[1331607.427951]  [<ffffffffa03d95c4>] ? cachefiles_bury_object+0x3e/0x2db [cachefiles]                   
[1331607.427997]  [<ffffffff811002e7>] ? vfs_removexattr+0x87/0x92                                        
[1331607.428030]  [<ffffffffa03d98a4>] ? cachefiles_cull+0x43/0x9c [cachefiles]                           
[1331607.428061]  [<ffffffffa03d7eb3>] ? cachefiles_daemon_cull+0xc0/0x113 [cachefiles]                   
[1331607.428115]  [<ffffffffa03d808e>] ? cachefiles_daemon_write+0x188/0x1ab [cachefiles]                 
[1331607.428212]  [<ffffffff810e95d5>] ? vfs_write+0xa9/0x102                                             
[1331607.428239]  [<ffffffff810e96ea>] ? sys_write+0x45/0x6e                                              
[1331607.428267]  [<ffffffff8103145f>] ? sysenter_dispatch+0x7/0x2e                                       
[1331607.507313] BUG: Bad page state in process cachefilesd  pfn:172caa                                   
[1331607.507352] page:ffffea000511c530 flags:0200000000001008 count:0 mapcount:0 mapping:(null) index:6264
[1331607.507406] Pid: 1893, comm: cachefilesd Tainted: G    B      2.6.33-2-amd64 #1                      
[1331607.507449] Call Trace:                                                                              
[1331607.507475]  [<ffffffff810b1e09>] ? bad_page+0x116/0x129                                             
[1331607.507503]  [<ffffffff810b22e9>] ? free_pages_check+0x38/0x57                                       
[1331607.507531]  [<ffffffff810b34f6>] ? free_hot_cold_page+0x46/0x176                                    
[1331607.507560]  [<ffffffff810b368f>] ? __pagevec_free+0x69/0x80                                         
[1331607.507589]  [<ffffffff810b66fa>] ? release_pages+0x17b/0x18d                                        
[1331607.507618]  [<ffffffff810afb59>] ? __remove_from_page_cache+0x33/0xb6                               
[1331607.507647]  [<ffffffff810b6acd>] ? __pagevec_release+0x19/0x22                                      
[1331607.507676]  [<ffffffff810b724c>] ? truncate_inode_pages_range+0xdb/0x2b0                            
[1331607.507715]  [<ffffffffa0291033>] ? ext4_delete_inode+0x5d/0x246 [ext4]                              
[1331607.507748]  [<ffffffffa0290fd6>] ? ext4_delete_inode+0x0/0x246 [ext4]                               
[1331607.507779]  [<ffffffff810fa9d9>] ? generic_delete_inode+0xdc/0x168                                  
[1331607.507808]  [<ffffffff810f8faf>] ? d_delete+0x47/0xb2                                               
[1331607.507836]  [<ffffffff810f1fb5>] ? vfs_unlink+0x9f/0xa6                                             
[1331607.507948]  [<ffffffffa03d95c4>] ? cachefiles_bury_object+0x3e/0x2db [cachefiles]                   
[1331607.507999]  [<ffffffff811002e7>] ? vfs_removexattr+0x87/0x92                                        
[1331607.508028]  [<ffffffffa03d98a4>] ? cachefiles_cull+0x43/0x9c [cachefiles]                           
[1331607.508058]  [<ffffffffa03d7eb3>] ? cachefiles_daemon_cull+0xc0/0x113 [cachefiles]                   
[1331607.508105]  [<ffffffffa03d808e>] ? cachefiles_daemon_write+0x188/0x1ab [cachefiles]                 
[1331607.508151]  [<ffffffff810e95d5>] ? vfs_write+0xa9/0x102                                             
[1331607.508179]  [<ffffffff810e96ea>] ? sys_write+0x45/0x6e                                              
[1331607.508216]  [<ffffffff8103145f>] ? sysenter_dispatch+0x7/0x2e                                       
[1331607.552946] BUG: Bad page state in process cachefilesd  pfn:1cf9ca                                   
[1331607.552978] page:ffffea000656a430 flags:0200000000001008 count:0 mapcount:0 mapping:(null) index:1f43
[1331607.553035] Pid: 1893, comm: cachefilesd Tainted: G    B      2.6.33-2-amd64 #1                      
[1331607.553079] Call Trace:                                                                              
[1331607.553102]  [<ffffffff810b1e09>] ? bad_page+0x116/0x129                                             
[1331607.553130]  [<ffffffff810b22e9>] ? free_pages_check+0x38/0x57                                       
[1331607.553159]  [<ffffffff810b34f6>] ? free_hot_cold_page+0x46/0x176                                    
[1331607.553187]  [<ffffffff810b368f>] ? __pagevec_free+0x69/0x80                                         
[1331607.553216]  [<ffffffff810b66fa>] ? release_pages+0x17b/0x18d                                        
[1331607.553269]  [<ffffffff810afb59>] ? __remove_from_page_cache+0x33/0xb6                               
[1331607.553300]  [<ffffffff810b6acd>] ? __pagevec_release+0x19/0x22                                      
[1331607.553328]  [<ffffffff810b724c>] ? truncate_inode_pages_range+0xdb/0x2b0                            
[1331607.553359]  [<ffffffff810df692>] ? virt_to_head_page+0x9/0x2a                                       
[1331607.553391]  [<ffffffffa0275ca1>] ? jbd2_journal_stop+0x222/0x235 [jbd2]                             
[1331607.553423]  [<ffffffff812e9ccb>] ? mutex_lock+0xd/0x33                                              
[1331607.553456]  [<ffffffffa0291033>] ? ext4_delete_inode+0x5d/0x246 [ext4]                              
[1331607.553490]  [<ffffffffa0290fd6>] ? ext4_delete_inode+0x0/0x246 [ext4]                               
[1331607.553524]  [<ffffffff810fa9d9>] ? generic_delete_inode+0xdc/0x168                                  
[1331607.553553]  [<ffffffff810f8faf>] ? d_delete+0x47/0xb2                                               
[1331607.553581]  [<ffffffff810f1fb5>] ? vfs_unlink+0x9f/0xa6                                             
[1331607.553610]  [<ffffffffa03d95c4>] ? cachefiles_bury_object+0x3e/0x2db [cachefiles]                   
[1331607.553665]  [<ffffffff811002e7>] ? vfs_removexattr+0x87/0x92                                        
[1331607.553695]  [<ffffffffa03d98a4>] ? cachefiles_cull+0x43/0x9c [cachefiles]                           
[1331607.553725]  [<ffffffffa03d7eb3>] ? cachefiles_daemon_cull+0xc0/0x113 [cachefiles]                   
[1331607.553776]  [<ffffffffa03d808e>] ? cachefiles_daemon_write+0x188/0x1ab [cachefiles]                 
[1331607.553826]  [<ffffffff810e95d5>] ? vfs_write+0xa9/0x102                                             
[1331607.553853]  [<ffffffff810e96ea>] ? sys_write+0x45/0x6e                                              
[1331607.553881]  [<ffffffff8103145f>] ? sysenter_dispatch+0x7/0x2e                                       
[1331881.783959] BUG: Bad page state in process lighttpd  pfn:228400                                      
[1331881.783993] page:ffffea00078ce000 flags:0200000000001000 count:0 mapcount:0 mapping:(null) index:1cfa
[1331881.784043] Pid: 29858, comm: lighttpd Tainted: G    B      2.6.33-2-amd64 #1                        
[1331881.784087] Call Trace:                                                                              
[1331881.784120]  [<ffffffff810b1e09>] ? bad_page+0x116/0x129                                             
[1331881.784154]  [<ffffffff810b3d8a>] ? get_page_from_freelist+0x4da/0x732                               
[1331881.784184]  [<ffffffff810b4593>] ? __alloc_pages_nodemask+0x2a7/0x5e0                               
[1331881.784219]  [<ffffffff810b5c99>] ? __do_page_cache_readahead+0x9b/0x1b4                             
[1331881.784263]  [<ffffffff810b5dce>] ? ra_submit+0x1c/0x20                                              
[1331881.784295]  [<ffffffff811057c2>] ? __generic_file_splice_read+0xfa/0x3ee                            
[1331881.784336]  [<ffffffff812770d8>] ? tcp_current_mss+0x3f/0x5a                                        
[1331881.784369]  [<ffffffff812ea851>] ? _raw_spin_lock_bh+0x9/0x25                                       
[1331881.784399]  [<ffffffff8123312d>] ? release_sock+0x13/0x96                                           
[1331881.784428]  [<ffffffff8126f8a4>] ? tcp_sendpage+0x44b/0x45d                                         
[1331881.784457]  [<ffffffff81103ab0>] ? pipe_to_sendpage+0x0/0x74                                        
[1331881.784486]  [<ffffffff8122ec69>] ? kernel_sendpage+0x16/0x1f                                        
[1331881.784518]  [<ffffffff8122eca7>] ? sock_sendpage+0x35/0x39                                          
[1331881.784551]  [<ffffffff81104182>] ? spd_release_page+0x0/0xe                                         
[1331881.784579]  [<ffffffff81105af0>] ? generic_file_splice_read+0x3a/0x62                               
[1331881.784609]  [<ffffffff811043d3>] ? splice_direct_to_actor+0xbe/0x188                                
[1331881.784638]  [<ffffffff81104a1c>] ? direct_splice_actor+0x0/0x1e                                     
[1331881.784669]  [<ffffffff81113274>] ? ep_scan_ready_list+0x132/0x151                                   
[1331881.784698]  [<ffffffff811044e7>] ? do_splice_direct+0x4a/0x64                                       
[1331881.791445]  [<ffffffff810e8fa8>] ? do_sendfile+0x12d/0x1a8                                          
[1331881.791474]  [<ffffffff810e906c>] ? sys_sendfile64+0x49/0x88                                         
[1331881.791503]  [<ffffffff8103145f>] ? sysenter_dispatch+0x7/0x2e                                       
[1331881.791531] BUG: Bad page state in process lighttpd  pfn:228401                                      
[1331881.791564] page:ffffea00078ce038 flags:0200000000001000 count:0 mapcount:0 mapping:(null) index:1cfb
[1331881.791616] Pid: 29858, comm: lighttpd Tainted: G    B      2.6.33-2-amd64 #1                        
[1331881.791659] Call Trace:                                                                              
[1331881.791682]  [<ffffffff810b1e09>] ? bad_page+0x116/0x129                                             
[1331881.791710]  [<ffffffff810b3d8a>] ? get_page_from_freelist+0x4da/0x732                               
[1331881.791744]  [<ffffffff810b4593>] ? __alloc_pages_nodemask+0x2a7/0x5e0                               
[1331881.791780]  [<ffffffff810b5c99>] ? __do_page_cache_readahead+0x9b/0x1b4                             
[1331881.791813]  [<ffffffff810b5dce>] ? ra_submit+0x1c/0x20                                              
[1331881.791840]  [<ffffffff811057c2>] ? __generic_file_splice_read+0xfa/0x3ee                            
[1331881.791870]  [<ffffffff812770d8>] ? tcp_current_mss+0x3f/0x5a                                        
[1331881.791899]  [<ffffffff812ea851>] ? _raw_spin_lock_bh+0x9/0x25                                       
[1331881.791927]  [<ffffffff8123312d>] ? release_sock+0x13/0x96                                           
[1331881.791954]  [<ffffffff8126f8a4>] ? tcp_sendpage+0x44b/0x45d                                         
[1331881.791983]  [<ffffffff81103ab0>] ? pipe_to_sendpage+0x0/0x74                                        
[1331881.792012]  [<ffffffff8122ec69>] ? kernel_sendpage+0x16/0x1f                                        
[1331881.792040]  [<ffffffff8122eca7>] ? sock_sendpage+0x35/0x39                                          
[1331881.792068]  [<ffffffff81104182>] ? spd_release_page+0x0/0xe                                         
[1331881.792096]  [<ffffffff81105af0>] ? generic_file_splice_read+0x3a/0x62                               
[1331881.792126]  [<ffffffff811043d3>] ? splice_direct_to_actor+0xbe/0x188                                
[1331881.792155]  [<ffffffff81104a1c>] ? direct_splice_actor+0x0/0x1e                                     
[1331881.792184]  [<ffffffff81113274>] ? ep_scan_ready_list+0x132/0x151                                   
[1331881.792213]  [<ffffffff811044e7>] ? do_splice_direct+0x4a/0x64                                       
[1331881.792260]  [<ffffffff810e8fa8>] ? do_sendfile+0x12d/0x1a8                                          
[1331881.792288]  [<ffffffff810e906c>] ? sys_sendfile64+0x49/0x88                                         
[1331881.792316]  [<ffffffff8103145f>] ? sysenter_dispatch+0x7/0x2e                                       
[1332127.874724] BUG: Bad page state in process lighttpd  pfn:18d827                                      
[1332127.874763] page:ffffea00056f4888 flags:0200000000001000 count:0 mapcount:0 mapping:(null) index:1406
[1332127.874822] Pid: 29857, comm: lighttpd Tainted: G    B      2.6.33-2-amd64 #1                        
[1332127.874866] Call Trace:                                                                              
[1332127.874895]  [<ffffffff810b1e09>] ? bad_page+0x116/0x129                                             
[1332127.874924]  [<ffffffff810b3d8a>] ? get_page_from_freelist+0x4da/0x732                               
[1332127.874954]  [<ffffffff810b43fb>] ? __alloc_pages_nodemask+0x10f/0x5e0                               
[1332127.874985]  [<ffffffff8110e03b>] ? mpage_bio_submit+0x22/0x26                                       
[1332127.875018]  [<ffffffff8110eaaf>] ? mpage_readpage+0x68/0x72                                         
[1332127.875050]  [<ffffffff810e1c63>] ? __kmalloc+0x12f/0x141                                            
[1332127.875082]  [<ffffffffa03dab65>] ? cachefiles_read_or_alloc_pages+0x3b4/0x7e5 [cachefiles]          
[1332127.875145]  [<ffffffffa03aec33>] ? nfs_readpage_from_fscache_complete+0x0/0x5c [nfs]                
[1332127.875193]  [<ffffffffa036170a>] ? __fscache_read_or_alloc_pages+0x1e4/0x260 [fscache]              
[1332127.875244]  [<ffffffffa03aeb69>] ? __nfs_readpages_from_fscache+0x77/0x141 [nfs]                    
[1332127.875294]  [<ffffffffa0396f8d>] ? nfs_readpages+0xf4/0x18d [nfs]                                   
[1332127.875324]  [<ffffffff810b5d19>] ? __do_page_cache_readahead+0x11b/0x1b4                            
[1332127.875355]  [<ffffffff810b5dce>] ? ra_submit+0x1c/0x20                                              
[1332127.875383]  [<ffffffff811057c2>] ? __generic_file_splice_read+0xfa/0x3ee                            
[1332127.875420]  [<ffffffff8126f365>] ? tcp_sendmsg+0x78a/0x87e                                          
[1332127.875454]  [<ffffffffa03115c9>] ? rpcauth_lookup_credcache+0x1c0/0x1d2 [sunrpc]                    
[1332127.875533]  [<ffffffffa038c544>] ? nfs_do_access+0x78/0x3d3 [nfs]                                   
[1332127.875566]  [<ffffffffa03115c9>] ? rpcauth_lookup_credcache+0x1c0/0x1d2 [sunrpc]                    
[1332127.875618]  [<ffffffff8105eecf>] ? bit_waitqueue+0x10/0xa0                                          
[1332127.875651]  [<ffffffff81104182>] ? spd_release_page+0x0/0xe                                         
[1332127.875679]  [<ffffffff81105af0>] ? generic_file_splice_read+0x3a/0x62                               
[1332127.875709]  [<ffffffff811043d3>] ? splice_direct_to_actor+0xbe/0x188                                
[1332127.875738]  [<ffffffff81104a1c>] ? direct_splice_actor+0x0/0x1e                                     
[1332127.875767]  [<ffffffff811044e7>] ? do_splice_direct+0x4a/0x64                                       
[1332127.875797]  [<ffffffff810e8fa8>] ? do_sendfile+0x12d/0x1a8                                          
[1332127.875825]  [<ffffffff810e906c>] ? sys_sendfile64+0x49/0x88                                         
[1332127.875854]  [<ffffffff8103145f>] ? sysenter_dispatch+0x7/0x2e                                       
[1332409.039615] BUG: Bad page state in process lighttpd  pfn:40ffe0                                      
[1332409.039649] page:ffffea000e37f900 flags:0200000000001000 count:0 mapcount:0 mapping:(null) index:1075
[1332409.039724] Pid: 29855, comm: lighttpd Tainted: G    B      2.6.33-2-amd64 #1                        
[1332409.039768] Call Trace:                                                                              
[1332409.039798]  [<ffffffff810b1e09>] ? bad_page+0x116/0x129                                             
[1332409.039837]  [<ffffffff810b3d8a>] ? get_page_from_freelist+0x4da/0x732                               
[1332409.039867]  [<ffffffff810b43fb>] ? __alloc_pages_nodemask+0x10f/0x5e0                               
[1332409.039907]  [<ffffffff810b5c99>] ? __do_page_cache_readahead+0x9b/0x1b4                             
[1332409.039938]  [<ffffffff810b5dce>] ? ra_submit+0x1c/0x20                                              
[1332409.039966]  [<ffffffff811057c2>] ? __generic_file_splice_read+0xfa/0x3ee                            
[1332409.039998]  [<ffffffff812770d8>] ? tcp_current_mss+0x3f/0x5a                                        
[1332409.040032]  [<ffffffff812ea851>] ? _raw_spin_lock_bh+0x9/0x25                                       
[1332409.040066]  [<ffffffff8123312d>] ? release_sock+0x13/0x96                                           
[1332409.040117]  [<ffffffff8126f8a4>] ? tcp_sendpage+0x44b/0x45d                                         
[1332409.040150]  [<ffffffff81103ab0>] ? pipe_to_sendpage+0x0/0x74                                        
[1332409.040182]  [<ffffffff8122ec69>] ? kernel_sendpage+0x16/0x1f                                        
[1332409.040211]  [<ffffffff8122eca7>] ? sock_sendpage+0x35/0x39                                          
[1332409.040241]  [<ffffffff81104182>] ? spd_release_page+0x0/0xe                                         
[1332409.040269]  [<ffffffff81105af0>] ? generic_file_splice_read+0x3a/0x62                               
[1332409.040300]  [<ffffffff811043d3>] ? splice_direct_to_actor+0xbe/0x188                                
[1332409.040329]  [<ffffffff81104a1c>] ? direct_splice_actor+0x0/0x1e                                     
[1332409.040360]  [<ffffffff81113274>] ? ep_scan_ready_list+0x132/0x151                                   
[1332409.040388]  [<ffffffff811044e7>] ? do_splice_direct+0x4a/0x64                                       
[1332409.040418]  [<ffffffff810e8fa8>] ? do_sendfile+0x12d/0x1a8                                          
[1332409.040446]  [<ffffffff810e906c>] ? sys_sendfile64+0x49/0x88                                         
[1332409.040475]  [<ffffffff8103145f>] ? sysenter_dispatch+0x7/0x2e                                       
[1336869.982957] BUG: Bad page state in process lighttpd  pfn:25bcd0                                      
[1336869.982990] page:ffffea0008414d80 flags:0200000000001000 count:0 mapcount:0 mapping:(null) index:1cd 
[1336869.983039] Pid: 29855, comm: lighttpd Tainted: G    B      2.6.33-2-amd64 #1                        
[1336869.983083] Call Trace:                                                                              
[1336869.983110]  [<ffffffff81218a2e>] ? loopback_xmit+0x5a/0x76                                          
[1336869.983140]  [<ffffffff810b1e09>] ? bad_page+0x116/0x129                                             
[1336869.983169]  [<ffffffff810b3d8a>] ? get_page_from_freelist+0x4da/0x732                               
[1336869.983199]  [<ffffffff810b43fb>] ? __alloc_pages_nodemask+0x10f/0x5e0                               
[1336869.983230]  [<ffffffff812382d0>] ? __alloc_skb+0x69/0x15a                                           
[1336869.983259]  [<ffffffff81297947>] ? bictcp_cong_avoid+0x12/0x247                                     
[1336869.983289]  [<ffffffff812752fc>] ? tcp_ack+0x17ac/0x197e                                            
[1336869.983317]  [<ffffffff81279e1b>] ? tcp_write_xmit+0x466/0x96c                                       
[1336869.983346]  [<ffffffff810b5c99>] ? __do_page_cache_readahead+0x9b/0x1b4                             
[1336869.983376]  [<ffffffff810b5dce>] ? ra_submit+0x1c/0x20                                              
[1336869.983405]  [<ffffffff811057c2>] ? __generic_file_splice_read+0xfa/0x3ee                            
[1336869.983435]  [<ffffffff8123a7d9>] ? sk_stream_wait_memory+0x1ae/0x1eb                                
[1336869.983467]  [<ffffffff81007689>] ? __switch_to+0x1df/0x2a6                                          
[1336869.983496]  [<ffffffff810393d8>] ? dequeue_entity+0x18/0x134                                        
[1336869.983524]  [<ffffffff81104182>] ? spd_release_page+0x0/0xe                                         
[1336869.983553]  [<ffffffff81105af0>] ? generic_file_splice_read+0x3a/0x62                               
[1336869.983582]  [<ffffffff811043d3>] ? splice_direct_to_actor+0xbe/0x188                                
[1336869.983612]  [<ffffffff81104a1c>] ? direct_splice_actor+0x0/0x1e                                     
[1336869.983642]  [<ffffffff81113274>] ? ep_scan_ready_list+0x132/0x151                                   
[1336869.983671]  [<ffffffff811044e7>] ? do_splice_direct+0x4a/0x64                                       
[1336869.983701]  [<ffffffff810e8fa8>] ? do_sendfile+0x12d/0x1a8                                          
[1336869.983729]  [<ffffffff810e906c>] ? sys_sendfile64+0x49/0x88                                         
[1336869.983758]  [<ffffffff8103145f>] ? sysenter_dispatch+0x7/0x2e                                       
[1337214.286107] BUG: Bad page state in process lighttpd  pfn:9631a                                       
[1337214.286140] page:ffffea00020dadb0 flags:0100000000001000 count:0 mapcount:0 mapping:(null) index:11d1
[1337214.286189] Pid: 29857, comm: lighttpd Tainted: G    B      2.6.33-2-amd64 #1                        
[1337214.286233] Call Trace:                                                                              
[1337214.286275]  [<ffffffff810b1e09>] ? bad_page+0x116/0x129                                             
[1337214.286304]  [<ffffffff810b3d8a>] ? get_page_from_freelist+0x4da/0x732                               
[1337214.286334]  [<ffffffff810b43fb>] ? __alloc_pages_nodemask+0x10f/0x5e0                               
[1337214.286365]  [<ffffffff8110e03b>] ? mpage_bio_submit+0x22/0x26                                       
[1337214.286394]  [<ffffffff8110eaaf>] ? mpage_readpage+0x68/0x72                                         
[1337214.286423]  [<ffffffff810e1c63>] ? __kmalloc+0x12f/0x141                                            
[1337214.286454]  [<ffffffffa03dab65>] ? cachefiles_read_or_alloc_pages+0x3b4/0x7e5 [cachefiles]          
[1337214.286512]  [<ffffffffa03aec33>] ? nfs_readpage_from_fscache_complete+0x0/0x5c [nfs]                
[1337214.286559]  [<ffffffffa036170a>] ? __fscache_read_or_alloc_pages+0x1e4/0x260 [fscache]              
[1337214.286610]  [<ffffffffa03aeb69>] ? __nfs_readpages_from_fscache+0x77/0x141 [nfs]                    
[1337214.286661]  [<ffffffffa0396f8d>] ? nfs_readpages+0xf4/0x18d [nfs]                                   
[1337214.286692]  [<ffffffff81277121>] ? tcp_init_tso_segs+0x2e/0x44                                      
[1337214.286720]  [<ffffffff81279e1b>] ? tcp_write_xmit+0x466/0x96c                                       
[1337214.286754]  [<ffffffff810b5d19>] ? __do_page_cache_readahead+0x11b/0x1b4                            
[1337214.286785]  [<ffffffff810b5dce>] ? ra_submit+0x1c/0x20                                              
[1337214.293409]  [<ffffffff811057c2>] ? __generic_file_splice_read+0xfa/0x3ee                            
[1337214.293440]  [<ffffffff812770d8>] ? tcp_current_mss+0x3f/0x5a                                        
[1337214.293471]  [<ffffffff812ea851>] ? _raw_spin_lock_bh+0x9/0x25                                       
[1337214.293513]  [<ffffffff8123312d>] ? release_sock+0x13/0x96                                           
[1337214.293541]  [<ffffffff8126f8a4>] ? tcp_sendpage+0x44b/0x45d                                         
[1337214.293570]  [<ffffffff81103ab0>] ? pipe_to_sendpage+0x0/0x74                                        
[1337214.293598]  [<ffffffff8122ec69>] ? kernel_sendpage+0x16/0x1f                                        
[1337214.293626]  [<ffffffff8122eca7>] ? sock_sendpage+0x35/0x39                                          
[1337214.293654]  [<ffffffff81104182>] ? spd_release_page+0x0/0xe                                         
[1337214.293687]  [<ffffffff81105af0>] ? generic_file_splice_read+0x3a/0x62                               
[1337214.293717]  [<ffffffff811043d3>] ? splice_direct_to_actor+0xbe/0x188                                
[1337214.293748]  [<ffffffff81104a1c>] ? direct_splice_actor+0x0/0x1e                                     
[1337214.293781]  [<ffffffff81113274>] ? ep_scan_ready_list+0x132/0x151                                   
[1337214.293818]  [<ffffffff811044e7>] ? do_splice_direct+0x4a/0x64                                       
[1337214.293852]  [<ffffffff810e8fa8>] ? do_sendfile+0x12d/0x1a8                                          
[1337214.293881]  [<ffffffff810e906c>] ? sys_sendfile64+0x49/0x88                                         
[1337214.293910]  [<ffffffff8103145f>] ? sysenter_dispatch+0x7/0x2e                                       
[1340924.082563] BUG: Bad page state in process lighttpd  pfn:23ab46                                      
[1340924.082598] page:ffffea0007cd7750 flags:0200000000001000 count:0 mapcount:0 mapping:(null) index:1276
[1340924.082646] Pid: 29854, comm: lighttpd Tainted: G    B      2.6.33-2-amd64 #1                        
[1340924.082690] Call Trace:                                                                              
[1340924.082719]  [<ffffffff810b1e09>] ? bad_page+0x116/0x129                                             
[1340924.082747]  [<ffffffff810b3d8a>] ? get_page_from_freelist+0x4da/0x732                               
[1340924.082777]  [<ffffffff810b43fb>] ? __alloc_pages_nodemask+0x10f/0x5e0                               
[1340924.082809]  [<ffffffff8104c357>] ? _local_bh_enable_ip+0x7a/0x8b                                    
[1340924.082840]  [<ffffffff8123ff43>] ? dev_queue_xmit+0x3ea/0x41c                                       
[1340924.082869]  [<ffffffff81266d93>] ? ip_queue_xmit+0x2fe/0x34b                                        
[1340924.082900]  [<ffffffff810b5c99>] ? __do_page_cache_readahead+0x9b/0x1b4                             
[1340924.082931]  [<ffffffff810b5dce>] ? ra_submit+0x1c/0x20                                              
[1340924.082960]  [<ffffffff811057c2>] ? __generic_file_splice_read+0xfa/0x3ee                            
[1340924.082991]  [<ffffffff812770d8>] ? tcp_current_mss+0x3f/0x5a                                        
[1340924.083021]  [<ffffffff812ea851>] ? _raw_spin_lock_bh+0x9/0x25                                       
[1340924.083050]  [<ffffffff8123312d>] ? release_sock+0x13/0x96                                           
[1340924.083078]  [<ffffffff8126f8a4>] ? tcp_sendpage+0x44b/0x45d                                         
[1340924.083106]  [<ffffffff81103ab0>] ? pipe_to_sendpage+0x0/0x74                                        
[1340924.083134]  [<ffffffff8122ec69>] ? kernel_sendpage+0x16/0x1f                                        
[1340924.083162]  [<ffffffff8122eca7>] ? sock_sendpage+0x35/0x39                                          
[1340924.083190]  [<ffffffff81104182>] ? spd_release_page+0x0/0xe                                         
[1340924.083218]  [<ffffffff81105af0>] ? generic_file_splice_read+0x3a/0x62                               
[1340924.083248]  [<ffffffff811043d3>] ? splice_direct_to_actor+0xbe/0x188                                
[1340924.083277]  [<ffffffff81104a1c>] ? direct_splice_actor+0x0/0x1e                                     
[1340924.083307]  [<ffffffff81113274>] ? ep_scan_ready_list+0x132/0x151                                   
[1340924.083336]  [<ffffffff811044e7>] ? do_splice_direct+0x4a/0x64                                       
[1340924.083366]  [<ffffffff810e8fa8>] ? do_sendfile+0x12d/0x1a8                                          
[1340924.083394]  [<ffffffff810e906c>] ? sys_sendfile64+0x49/0x88                                         
[1340924.083423]  [<ffffffff8103145f>] ? sysenter_dispatch+0x7/0x2e                                       
[1351956.636956] BUG: Bad page state in process cachefilesd  pfn:39ef54                                   
[1351956.636990] page:ffffea000cac5a60 flags:0200000000001008 count:0 mapcount:0 mapping:(null) index:58e2
[1351956.637038] Pid: 1893, comm: cachefilesd Tainted: G    B      2.6.33-2-amd64 #1                      
[1351956.637083] Call Trace:                                                                              
[1351956.637116]  [<ffffffff810b1e09>] ? bad_page+0x116/0x129                                             
[1351956.637144]  [<ffffffff810b22e9>] ? free_pages_check+0x38/0x57                                       
[1351956.637173]  [<ffffffff810b34f6>] ? free_hot_cold_page+0x46/0x176                                    
[1351956.637206]  [<ffffffff810b368f>] ? __pagevec_free+0x69/0x80                                         
[1351956.637240]  [<ffffffff810b66fa>] ? release_pages+0x17b/0x18d                                        
[1351956.637286]  [<ffffffff810b686c>] ? ____pagevec_lru_add+0x160/0x176                                  
[1351956.637318]  [<ffffffff810b6acd>] ? __pagevec_release+0x19/0x22                                      
[1351956.637347]  [<ffffffff810b724c>] ? truncate_inode_pages_range+0xdb/0x2b0                            
[1351956.637390]  [<ffffffff810df692>] ? virt_to_head_page+0x9/0x2a                                       
[1351956.637427]  [<ffffffffa0275ca1>] ? jbd2_journal_stop+0x222/0x235 [jbd2]                             
[1351956.637459]  [<ffffffff812e9ccb>] ? mutex_lock+0xd/0x33                                              
[1351956.637494]  [<ffffffffa0291033>] ? ext4_delete_inode+0x5d/0x246 [ext4]                              
[1351956.637528]  [<ffffffffa0290fd6>] ? ext4_delete_inode+0x0/0x246 [ext4]                               
[1351956.637565]  [<ffffffff810fa9d9>] ? generic_delete_inode+0xdc/0x168                                  
[1351956.637598]  [<ffffffff810f8faf>] ? d_delete+0x47/0xb2                                               
[1351956.637626]  [<ffffffff810f1fb5>] ? vfs_unlink+0x9f/0xa6                                             
[1351956.637655]  [<ffffffffa03d95c4>] ? cachefiles_bury_object+0x3e/0x2db [cachefiles]                   
[1351956.637701]  [<ffffffff811002e7>] ? vfs_removexattr+0x87/0x92                                        
[1351956.637730]  [<ffffffffa03d98a4>] ? cachefiles_cull+0x43/0x9c [cachefiles]                           
[1351956.637761]  [<ffffffffa03d7eb3>] ? cachefiles_daemon_cull+0xc0/0x113 [cachefiles]                   
[1351956.637807]  [<ffffffffa03d808e>] ? cachefiles_daemon_write+0x188/0x1ab [cachefiles]                 
[1351956.637853]  [<ffffffff810e95d5>] ? vfs_write+0xa9/0x102                                             
[1351956.637882]  [<ffffffff810e96ea>] ? sys_write+0x45/0x6e                                              
[1351956.637911]  [<ffffffff8103145f>] ? sysenter_dispatch+0x7/0x2e                                       
[1358901.820866] [kslowd] preemptive burial: OBJ19e0e [OBJECT_RECYCLING] ffff8804294f8840                 
[1360460.841478] [kslowd] preemptive burial: OBJ19b03 [OBJECT_RECYCLING] ffff880427801900                 
[1364084.724072] [kslowd] preemptive burial: OBJ19e3d [OBJECT_RECYCLING] ffff880427d10c00                 
[1365660.551316] BUG: Bad page state in process kswapd0  pfn:407d0c                                       
[1365660.551354] page:ffffea000e1b5aa0 flags:0200000000001008 count:0 mapcount:0 mapping:(null) index:2422
[1365660.551403] Pid: 79, comm: kswapd0 Tainted: G    B      2.6.33-2-amd64 #1                            
[1365660.551432] Call Trace:                                                                              
[1365660.551460]  [<ffffffff810b4be3>] ? clear_page_dirty_for_io+0x54/0xb8                                
[1365660.551491]  [<ffffffff810b1e09>] ? bad_page+0x116/0x129                                             
[1365660.551528]  [<ffffffffa0399b53>] ? nfs_wb_page+0x63/0xc3 [nfs]                                      
[1365660.551557]  [<ffffffff810b22e9>] ? free_pages_check+0x38/0x57                                       
[1365660.551586]  [<ffffffff810b34f6>] ? free_hot_cold_page+0x46/0x176                                    
[1365660.551615]  [<ffffffff810b368f>] ? __pagevec_free+0x69/0x80                                         
[1365660.551643]  [<ffffffff810b86eb>] ? shrink_page_list+0x51e/0x617                                     
[1365660.551672]  [<ffffffff810b7715>] ? isolate_pages_global+0x1a0/0x20f                                 
[1365660.551701]  [<ffffffff810b91bb>] ? shrink_zone+0x710/0xae9                                          
[1365660.551731]  [<ffffffff810393d8>] ? dequeue_entity+0x18/0x134                                        
[1365660.551759]  [<ffffffff810b9ff2>] ? kswapd+0x5d3/0x80c                                               
[1365660.551786]  [<ffffffff810b7575>] ? isolate_pages_global+0x0/0x20f                                   
[1365660.551817]  [<ffffffff8105efae>] ? autoremove_wake_function+0x0/0x2e                                
[1365660.551846]  [<ffffffff810b9a1f>] ? kswapd+0x0/0x80c                                                 
[1365660.551872]  [<ffffffff8105eb75>] ? kthread+0x79/0x81                                                
[1365660.551901]  [<ffffffff810098e4>] ? kernel_thread_helper+0x4/0x10                                    
[1365660.551929]  [<ffffffff8105eafc>] ? kthread+0x0/0x81                                                 
[1365660.551956]  [<ffffffff810098e0>] ? kernel_thread_helper+0x0/0x10                                    
[1365947.770464] BUG: Bad page state in process cachefilesd  pfn:3a29a9                                   
[1365947.770498] page:ffffea000cb91cf8 flags:0200000000001008 count:0 mapcount:0 mapping:(null) index:7052
[1365947.770547] Pid: 1893, comm: cachefilesd Tainted: G    B      2.6.33-2-amd64 #1                      
[1365947.770591] Call Trace:                                                                              
[1365947.770619]  [<ffffffff810b1e09>] ? bad_page+0x116/0x129                                             
[1365947.770647]  [<ffffffff810b22e9>] ? free_pages_check+0x38/0x57                                       
[1365947.770676]  [<ffffffff810b34f6>] ? free_hot_cold_page+0x46/0x176                                    
[1365947.770705]  [<ffffffff810b368f>] ? __pagevec_free+0x69/0x80                                         
[1365947.770733]  [<ffffffff810b66fa>] ? release_pages+0x17b/0x18d                                        
[1365947.770763]  [<ffffffff81034b44>] ? __wake_up_common+0x44/0x73                                       
[1365947.770793]  [<ffffffff810387e3>] ? __wake_up+0x30/0x44                                              
[1365947.770820]  [<ffffffff810b6acd>] ? __pagevec_release+0x19/0x22                                      
[1365947.770855]  [<ffffffff810b724c>] ? truncate_inode_pages_range+0xdb/0x2b0                            
[1365947.770897]  [<ffffffffa0291033>] ? ext4_delete_inode+0x5d/0x246 [ext4]                              
[1365947.770930]  [<ffffffffa0290fd6>] ? ext4_delete_inode+0x0/0x246 [ext4]                               
[1365947.770961]  [<ffffffff810fa9d9>] ? generic_delete_inode+0xdc/0x168                                  
[1365947.770990]  [<ffffffff810f8faf>] ? d_delete+0x47/0xb2                                               
[1365947.771019]  [<ffffffff810f1fb5>] ? vfs_unlink+0x9f/0xa6                                             
[1365947.771049]  [<ffffffffa03d95c4>] ? cachefiles_bury_object+0x3e/0x2db [cachefiles]                   
[1365947.771096]  [<ffffffff811002e7>] ? vfs_removexattr+0x87/0x92                                        
[1365947.771124]  [<ffffffffa03d98a4>] ? cachefiles_cull+0x43/0x9c [cachefiles]                           
[1365947.771155]  [<ffffffffa03d7eb3>] ? cachefiles_daemon_cull+0xc0/0x113 [cachefiles]                   
[1365947.771201]  [<ffffffffa03d808e>] ? cachefiles_daemon_write+0x188/0x1ab [cachefiles]                 
[1365947.771248]  [<ffffffff810e95d5>] ? vfs_write+0xa9/0x102                                             
[1365947.771276]  [<ffffffff810e96ea>] ? sys_write+0x45/0x6e                                              
[1365947.771304]  [<ffffffff8103145f>] ? sysenter_dispatch+0x7/0x2e                                       
[1365947.867728] BUG: Bad page state in process cachefilesd  pfn:2e0de6                                   
[1365947.867765] page:ffffea000a130a50 flags:0200000000001008 count:0 mapcount:0 mapping:(null) index:c611
[1365947.867819] Pid: 1893, comm: cachefilesd Tainted: G    B      2.6.33-2-amd64 #1                      
[1365947.867863] Call Trace:                                                                              
[1365947.867889]  [<ffffffff810b1e09>] ? bad_page+0x116/0x129                                             
[1365947.867917]  [<ffffffff810b22e9>] ? free_pages_check+0x38/0x57                                       
[1365947.867945]  [<ffffffff810b34f6>] ? free_hot_cold_page+0x46/0x176                                    
[1365947.867974]  [<ffffffff810b368f>] ? __pagevec_free+0x69/0x80                                         
[1365947.868003]  [<ffffffff810b66fa>] ? release_pages+0x17b/0x18d                                        
[1365947.868033]  [<ffffffff810b686c>] ? ____pagevec_lru_add+0x160/0x176                                  
[1365947.868067]  [<ffffffff810b6acd>] ? __pagevec_release+0x19/0x22                                      
[1365947.868110]  [<ffffffff810b724c>] ? truncate_inode_pages_range+0xdb/0x2b0                            
[1365947.868142]  [<ffffffff810df692>] ? virt_to_head_page+0x9/0x2a                                       
[1365947.868175]  [<ffffffffa0275ca1>] ? jbd2_journal_stop+0x222/0x235 [jbd2]                             
[1365947.868215]  [<ffffffff812e9ccb>] ? mutex_lock+0xd/0x33                                              
[1365947.868247]  [<ffffffffa0291033>] ? ext4_delete_inode+0x5d/0x246 [ext4]                              
[1365947.868281]  [<ffffffffa0290fd6>] ? ext4_delete_inode+0x0/0x246 [ext4]                               
[1365947.868316]  [<ffffffff810fa9d9>] ? generic_delete_inode+0xdc/0x168                                  
[1365947.868345]  [<ffffffff810f8faf>] ? d_delete+0x47/0xb2                                               
[1365947.868373]  [<ffffffff810f1fb5>] ? vfs_unlink+0x9f/0xa6                                             
[1365947.868402]  [<ffffffffa03d95c4>] ? cachefiles_bury_object+0x3e/0x2db [cachefiles]                   
[1365947.868448]  [<ffffffff811002e7>] ? vfs_removexattr+0x87/0x92                                        
[1365947.868477]  [<ffffffffa03d98a4>] ? cachefiles_cull+0x43/0x9c [cachefiles]                           
[1365947.868512]  [<ffffffffa03d7eb3>] ? cachefiles_daemon_cull+0xc0/0x113 [cachefiles]                   
[1365947.868562]  [<ffffffffa03d808e>] ? cachefiles_daemon_write+0x188/0x1ab [cachefiles]                 
[1365947.868635]  [<ffffffff810e95d5>] ? vfs_write+0xa9/0x102                                             
[1365947.868663]  [<ffffffff810e96ea>] ? sys_write+0x45/0x6e                                              
[1365947.868690]  [<ffffffff8103145f>] ? sysenter_dispatch+0x7/0x2e                                       
[1366021.650619] BUG: Bad page state in process lighttpd  pfn:98c85                                       
[1366021.650662] page:ffffea000216bd18 flags:0100000000001000 count:0 mapcount:0 mapping:(null) index:26cd
[1366021.650712] Pid: 2006, comm: lighttpd Tainted: G    B      2.6.33-2-amd64 #1                         
[1366021.650755] Call Trace:                                                                              
[1366021.650784]  [<ffffffff810b1e09>] ? bad_page+0x116/0x129                                             
[1366021.650823]  [<ffffffff810b3d8a>] ? get_page_from_freelist+0x4da/0x732                               
[1366021.650854]  [<ffffffff810b43fb>] ? __alloc_pages_nodemask+0x10f/0x5e0                               
[1366021.650885]  [<ffffffff8110e03b>] ? mpage_bio_submit+0x22/0x26                                       
[1366021.650913]  [<ffffffff8110eaaf>] ? mpage_readpage+0x68/0x72                                         
[1366021.650946]  [<ffffffff810e1c63>] ? __kmalloc+0x12f/0x141                                            
[1366021.650977]  [<ffffffffa03dab65>] ? cachefiles_read_or_alloc_pages+0x3b4/0x7e5 [cachefiles]          
[1366021.651046]  [<ffffffffa03aec33>] ? nfs_readpage_from_fscache_complete+0x0/0x5c [nfs]                
[1366021.651099]  [<ffffffffa036170a>] ? __fscache_read_or_alloc_pages+0x1e4/0x260 [fscache]              
[1366021.651154]  [<ffffffffa03aeb69>] ? __nfs_readpages_from_fscache+0x77/0x141 [nfs]                    
[1366021.651222]  [<ffffffffa0396f8d>] ? nfs_readpages+0xf4/0x18d [nfs]                                   
[1366021.651252]  [<ffffffff810b5d19>] ? __do_page_cache_readahead+0x11b/0x1b4                            
[1366021.651283]  [<ffffffff810b5dce>] ? ra_submit+0x1c/0x20                                              
[1366021.651326]  [<ffffffff811057c2>] ? __generic_file_splice_read+0xfa/0x3ee                            
[1366021.651357]  [<ffffffff8126f365>] ? tcp_sendmsg+0x78a/0x87e                                          
[1366021.651402]  [<ffffffffa03115c9>] ? rpcauth_lookup_credcache+0x1c0/0x1d2 [sunrpc]                    
[1366021.651460]  [<ffffffffa038c544>] ? nfs_do_access+0x78/0x3d3 [nfs]                                   
[1366021.651493]  [<ffffffffa03115c9>] ? rpcauth_lookup_credcache+0x1c0/0x1d2 [sunrpc]                    
[1366021.651544]  [<ffffffff8105eecf>] ? bit_waitqueue+0x10/0xa0                                          
[1366021.651573]  [<ffffffff81104182>] ? spd_release_page+0x0/0xe                                         
[1366021.651601]  [<ffffffff81105af0>] ? generic_file_splice_read+0x3a/0x62                               
[1366021.651631]  [<ffffffff811043d3>] ? splice_direct_to_actor+0xbe/0x188                                
[1366021.651661]  [<ffffffff81104a1c>] ? direct_splice_actor+0x0/0x1e                                     
[1366021.651690]  [<ffffffff811044e7>] ? do_splice_direct+0x4a/0x64                                       
[1366021.651723]  [<ffffffff810e8fa8>] ? do_sendfile+0x12d/0x1a8                                          
[1366021.651760]  [<ffffffff810e906c>] ? sys_sendfile64+0x49/0x88                                         
[1366021.651790]  [<ffffffff8103145f>] ? sysenter_dispatch+0x7/0x2e                                       
[1371000.117657] BUG: Bad page state in process lighttpd  pfn:40f461                                      
[1371000.117700] page:ffffea000e357538 flags:0200000000001000 count:0 mapcount:0 mapping:(null) index:15c2
[1371000.117749] Pid: 2010, comm: lighttpd Tainted: G    B      2.6.33-2-amd64 #1                         
[1371000.117797] Call Trace:                                                                              
[1371000.117826]  [<ffffffff810b1e09>] ? bad_page+0x116/0x129                                             
[1371000.117855]  [<ffffffff810b3d8a>] ? get_page_from_freelist+0x4da/0x732                               
[1371000.117889]  [<ffffffff810b43fb>] ? __alloc_pages_nodemask+0x10f/0x5e0                               
[1371000.117921]  [<ffffffff8103a380>] ? select_task_rq_fair+0x4e2/0x8bb                                  
[1371000.117955]  [<ffffffff810b5c99>] ? __do_page_cache_readahead+0x9b/0x1b4                             
[1371000.117985]  [<ffffffff810b5dce>] ? ra_submit+0x1c/0x20                                              
[1371000.118014]  [<ffffffff811057c2>] ? __generic_file_splice_read+0xfa/0x3ee                            
[1371000.118046]  [<ffffffff8126f365>] ? tcp_sendmsg+0x78a/0x87e                                          
[1371000.118082]  [<ffffffffa03115c9>] ? rpcauth_lookup_credcache+0x1c0/0x1d2 [sunrpc]                    
[1371000.118139]  [<ffffffffa038c544>] ? nfs_do_access+0x78/0x3d3 [nfs]                                   
[1371000.118172]  [<ffffffffa03115c9>] ? rpcauth_lookup_credcache+0x1c0/0x1d2 [sunrpc]                    
[1371000.118219]  [<ffffffff8105eecf>] ? bit_waitqueue+0x10/0xa0                                          
[1371000.118247]  [<ffffffff8105ef9d>] ? wake_up_bit+0x11/0x22                                            
[1371000.118279]  [<ffffffffa038f70d>] ? nfs_open+0x189/0x19a [nfs]                                       
[1371000.118307]  [<ffffffff81104182>] ? spd_release_page+0x0/0xe                                         
[1371000.118335]  [<ffffffff81105af0>] ? generic_file_splice_read+0x3a/0x62                               
[1371000.118365]  [<ffffffff811043d3>] ? splice_direct_to_actor+0xbe/0x188                                
[1371000.118394]  [<ffffffff81104a1c>] ? direct_splice_actor+0x0/0x1e                                     
[1371000.118423]  [<ffffffff811044e7>] ? do_splice_direct+0x4a/0x64                                       
[1371000.118453]  [<ffffffff810e8fa8>] ? do_sendfile+0x12d/0x1a8                                          
[1371000.118481]  [<ffffffff810e906c>] ? sys_sendfile64+0x49/0x88                                         
[1371000.118510]  [<ffffffff8103145f>] ? sysenter_dispatch+0x7/0x2e                                       
[1371000.118538] BUG: Bad page state in process lighttpd  pfn:2ec303                                      
[1371000.118567] page:ffffea000a3aa8a8 flags:0200000000001000 count:0 mapcount:0 mapping:(null) index:15c3
[1371000.118614] Pid: 2010, comm: lighttpd Tainted: G    B      2.6.33-2-amd64 #1                         
[1371000.118657] Call Trace:                                                                              
[1371000.118680]  [<ffffffff810b1e09>] ? bad_page+0x116/0x129                                             
[1371000.118707]  [<ffffffff810b3d8a>] ? get_page_from_freelist+0x4da/0x732                               
[1371000.118738]  [<ffffffff810b43fb>] ? __alloc_pages_nodemask+0x10f/0x5e0                               
[1371000.118768]  [<ffffffff8103a380>] ? select_task_rq_fair+0x4e2/0x8bb                                  
[1371000.118797]  [<ffffffff810b5c99>] ? __do_page_cache_readahead+0x9b/0x1b4                             
[1371000.118827]  [<ffffffff810b5dce>] ? ra_submit+0x1c/0x20                                              
[1371000.118855]  [<ffffffff811057c2>] ? __generic_file_splice_read+0xfa/0x3ee                            
[1371000.118885]  [<ffffffff8126f365>] ? tcp_sendmsg+0x78a/0x87e                                          
[1371000.118916]  [<ffffffffa03115c9>] ? rpcauth_lookup_credcache+0x1c0/0x1d2 [sunrpc]                    
[1371000.118966]  [<ffffffffa038c544>] ? nfs_do_access+0x78/0x3d3 [nfs]                                   
[1371000.118998]  [<ffffffffa03115c9>] ? rpcauth_lookup_credcache+0x1c0/0x1d2 [sunrpc]                    
[1371000.119051]  [<ffffffff8105eecf>] ? bit_waitqueue+0x10/0xa0                                          
[1371000.119087]  [<ffffffff8105ef9d>] ? wake_up_bit+0x11/0x22                                            
[1371000.119119]  [<ffffffffa038f70d>] ? nfs_open+0x189/0x19a [nfs]                                       
[1371000.119148]  [<ffffffff81104182>] ? spd_release_page+0x0/0xe                                         
[1371000.119176]  [<ffffffff81105af0>] ? generic_file_splice_read+0x3a/0x62                               
[1371000.119205]  [<ffffffff811043d3>] ? splice_direct_to_actor+0xbe/0x188                                
[1371000.119235]  [<ffffffff81104a1c>] ? direct_splice_actor+0x0/0x1e                                     
[1371000.119263]  [<ffffffff811044e7>] ? do_splice_direct+0x4a/0x64                                       
[1371000.119292]  [<ffffffff810e8fa8>] ? do_sendfile+0x12d/0x1a8                                          
[1371000.119320]  [<ffffffff810e906c>] ? sys_sendfile64+0x49/0x88                                         
[1371000.119352]  [<ffffffff8103145f>] ? sysenter_dispatch+0x7/0x2e                                       
[1371954.372699] [kslowd] preemptive burial: OBJ19f53 [OBJECT_RECYCLING] ffff880429654780                 
[1372603.733189] [kslowd] preemptive burial: OBJ1a2af [OBJECT_RECYCLING] ffff880427ffd480                 
[1372629.549488] BUG: Bad page state in process lighttpd  pfn:58312                                       
[1372629.549521] page:ffffea000134abf0 flags:0100000000001000 count:0 mapcount:0 mapping:(null) index:194c
[1372629.549571] Pid: 2006, comm: lighttpd Tainted: G    B      2.6.33-2-amd64 #1                         
[1372629.549614] Call Trace:                                                                              
[1372629.549642]  [<ffffffff810b1e09>] ? bad_page+0x116/0x129                                             
[1372629.549671]  [<ffffffff810b3d8a>] ? get_page_from_freelist+0x4da/0x732                               
[1372629.549701]  [<ffffffff810b43fb>] ? __alloc_pages_nodemask+0x10f/0x5e0                               
[1372629.549732]  [<ffffffff8110e03b>] ? mpage_bio_submit+0x22/0x26                                       
[1372629.549760]  [<ffffffff8110eaaf>] ? mpage_readpage+0x68/0x72                                         
[1372629.549793]  [<ffffffff810e1c63>] ? __kmalloc+0x12f/0x141                                            
[1372629.549829]  [<ffffffffa03dab65>] ? cachefiles_read_or_alloc_pages+0x3b4/0x7e5 [cachefiles]          
[1372629.549877]  [<ffffffff8105ef9d>] ? wake_up_bit+0x11/0x22                                            
[1372629.549907]  [<ffffffffa035ffe1>] ? fscache_run_op+0x20/0x38 [fscache]                               
[1372629.549946]  [<ffffffffa03aec33>] ? nfs_readpage_from_fscache_complete+0x0/0x5c [nfs]                
[1372629.549993]  [<ffffffffa036170a>] ? __fscache_read_or_alloc_pages+0x1e4/0x260 [fscache]              
[1372629.550044]  [<ffffffffa03aeb69>] ? __nfs_readpages_from_fscache+0x77/0x141 [nfs]                    
[1372629.550094]  [<ffffffffa0396f8d>] ? nfs_readpages+0xf4/0x18d [nfs]                                   
[1372629.550124]  [<ffffffff810b5d19>] ? __do_page_cache_readahead+0x11b/0x1b4                            
[1372629.550155]  [<ffffffff810b5dce>] ? ra_submit+0x1c/0x20                                              
[1372629.550182]  [<ffffffff810b60bd>] ? page_cache_async_readahead+0x75/0xad                             
[1372629.550214]  [<ffffffff811058e7>] ? __generic_file_splice_read+0x21f/0x3ee                           
[1372629.550261]  [<ffffffff812ea851>] ? _raw_spin_lock_bh+0x9/0x25                                       
[1372629.550291]  [<ffffffff8123312d>] ? release_sock+0x13/0x96                                           
[1372629.550320]  [<ffffffff8126f8a4>] ? tcp_sendpage+0x44b/0x45d                                         
[1372629.550354]  [<ffffffff81103ab0>] ? pipe_to_sendpage+0x0/0x74                                        
[1372629.550386]  [<ffffffff8122ec69>] ? kernel_sendpage+0x16/0x1f                                        
[1372629.550414]  [<ffffffff8122eca7>] ? sock_sendpage+0x35/0x39                                          
[1372629.550443]  [<ffffffff81104182>] ? spd_release_page+0x0/0xe                                         
[1372629.550471]  [<ffffffff81105af0>] ? generic_file_splice_read+0x3a/0x62                               
[1372629.550510]  [<ffffffff811043d3>] ? splice_direct_to_actor+0xbe/0x188                                
[1372629.550540]  [<ffffffff81104a1c>] ? direct_splice_actor+0x0/0x1e                                     
[1372629.550570]  [<ffffffff81113274>] ? ep_scan_ready_list+0x132/0x151                                   
[1372629.550599]  [<ffffffff811044e7>] ? do_splice_direct+0x4a/0x64                                       
[1372629.550634]  [<ffffffff810e8fa8>] ? do_sendfile+0x12d/0x1a8                                          
[1372629.550662]  [<ffffffff810e906c>] ? sys_sendfile64+0x49/0x88                                         
[1372629.550692]  [<ffffffff8103145f>] ? sysenter_dispatch+0x7/0x2e                                       
[1373850.066042] BUG: Bad page state in process cachefilesd  pfn:26761e                                   
[1373850.066077] page:ffffea000869d690 flags:0200000000001008 count:0 mapcount:0 mapping:(null) index:2d6b
[1373850.066127] Pid: 1893, comm: cachefilesd Tainted: G    B      2.6.33-2-amd64 #1                      
[1373850.066171] Call Trace:                                                                              
[1373850.066199]  [<ffffffff810b1e09>] ? bad_page+0x116/0x129                                             
[1373850.066227]  [<ffffffff810b22e9>] ? free_pages_check+0x38/0x57                                       
[1373850.066255]  [<ffffffff810b34f6>] ? free_hot_cold_page+0x46/0x176                                    
[1373850.066284]  [<ffffffff810b368f>] ? __pagevec_free+0x69/0x80                                         
[1373850.066313]  [<ffffffff810b66fa>] ? release_pages+0x17b/0x18d                                        
[1373850.066342]  [<ffffffff810b686c>] ? ____pagevec_lru_add+0x160/0x176                                  
[1373850.066372]  [<ffffffff810b6acd>] ? __pagevec_release+0x19/0x22                                      
[1373850.066400]  [<ffffffff810b724c>] ? truncate_inode_pages_range+0xdb/0x2b0                            
[1373850.066431]  [<ffffffff810df692>] ? virt_to_head_page+0x9/0x2a                                       
[1373850.066470]  [<ffffffffa0275ca1>] ? jbd2_journal_stop+0x222/0x235 [jbd2]                             
[1373850.066501]  [<ffffffff812e9ccb>] ? mutex_lock+0xd/0x33                                              
[1373850.066535]  [<ffffffffa0291033>] ? ext4_delete_inode+0x5d/0x246 [ext4]                              
[1373850.066569]  [<ffffffffa0290fd6>] ? ext4_delete_inode+0x0/0x246 [ext4]                               
[1373850.066600]  [<ffffffff810fa9d9>] ? generic_delete_inode+0xdc/0x168                                  
[1373850.066629]  [<ffffffff810f8faf>] ? d_delete+0x47/0xb2                                               
[1373850.066658]  [<ffffffff810f1fb5>] ? vfs_unlink+0x9f/0xa6                                             
[1373850.066687]  [<ffffffffa03d95c4>] ? cachefiles_bury_object+0x3e/0x2db [cachefiles]                   
[1373850.066733]  [<ffffffff811002e7>] ? vfs_removexattr+0x87/0x92                                        
[1373850.066762]  [<ffffffffa03d98a4>] ? cachefiles_cull+0x43/0x9c [cachefiles]                           
[1373850.066793]  [<ffffffffa03d7eb3>] ? cachefiles_daemon_cull+0xc0/0x113 [cachefiles]                   
[1373850.066839]  [<ffffffffa03d808e>] ? cachefiles_daemon_write+0x188/0x1ab [cachefiles]                 
[1373850.066886]  [<ffffffff810e95d5>] ? vfs_write+0xa9/0x102                                             
[1373850.066913]  [<ffffffff810e96ea>] ? sys_write+0x45/0x6e                                              
[1373850.066942]  [<ffffffff8103145f>] ? sysenter_dispatch+0x7/0x2e                                       
[1375394.416015] [kslowd] preemptive burial: OBJ1a2d0 [OBJECT_RECYCLING] ffff88013ce72900                 
[1375665.305526] BUG: Bad page state in process lighttpd  pfn:8cd61                                       
[1375665.305559] page:ffffea0001eced38 flags:0100000000001000 count:0 mapcount:0 mapping:(null) index:2e54
[1375665.305608] Pid: 1999, comm: lighttpd Tainted: G    B      2.6.33-2-amd64 #1                         
[1375665.305651] Call Trace:                                                                              
[1375665.305679]  [<ffffffff810b1e09>] ? bad_page+0x116/0x129                                             
[1375665.305708]  [<ffffffff810b3d8a>] ? get_page_from_freelist+0x4da/0x732                               
[1375665.312581]  [<ffffffff810b43fb>] ? __alloc_pages_nodemask+0x10f/0x5e0                               
[1375665.312613]  [<ffffffff8110e03b>] ? mpage_bio_submit+0x22/0x26                                       
[1375665.312642]  [<ffffffff8110eaaf>] ? mpage_readpage+0x68/0x72                                         
[1375665.312676]  [<ffffffff810e1c63>] ? __kmalloc+0x12f/0x141                                            
[1375665.312708]  [<ffffffffa03dab65>] ? cachefiles_read_or_alloc_pages+0x3b4/0x7e5 [cachefiles]          
[1375665.312787]  [<ffffffffa03aec33>] ? nfs_readpage_from_fscache_complete+0x0/0x5c [nfs]                
[1375665.312835]  [<ffffffffa036170a>] ? __fscache_read_or_alloc_pages+0x1e4/0x260 [fscache]              
[1375665.312886]  [<ffffffffa03aeb69>] ? __nfs_readpages_from_fscache+0x77/0x141 [nfs]                    
[1375665.312936]  [<ffffffffa0396f8d>] ? nfs_readpages+0xf4/0x18d [nfs]                                   
[1375665.312966]  [<ffffffff810b5d19>] ? __do_page_cache_readahead+0x11b/0x1b4                            
[1375665.312997]  [<ffffffff810b5dce>] ? ra_submit+0x1c/0x20                                              
[1375665.313026]  [<ffffffff811057c2>] ? __generic_file_splice_read+0xfa/0x3ee                            
[1375665.313058]  [<ffffffff812770d8>] ? tcp_current_mss+0x3f/0x5a                                        
[1375665.313087]  [<ffffffff812ea851>] ? _raw_spin_lock_bh+0x9/0x25                                       
[1375665.313117]  [<ffffffff8123312d>] ? release_sock+0x13/0x96                                           
[1375665.313155]  [<ffffffff8126f8a4>] ? tcp_sendpage+0x44b/0x45d                                         
[1375665.313185]  [<ffffffff81103ab0>] ? pipe_to_sendpage+0x0/0x74                                        
[1375665.313213]  [<ffffffff8122ec69>] ? kernel_sendpage+0x16/0x1f                                        
[1375665.313241]  [<ffffffff8122eca7>] ? sock_sendpage+0x35/0x39                                          
[1375665.313286]  [<ffffffff81104182>] ? spd_release_page+0x0/0xe                                         
[1375665.313323]  [<ffffffff81105af0>] ? generic_file_splice_read+0x3a/0x62                               
[1375665.313356]  [<ffffffff811043d3>] ? splice_direct_to_actor+0xbe/0x188                                
[1375665.313393]  [<ffffffff81104a1c>] ? direct_splice_actor+0x0/0x1e                                     
[1375665.313429]  [<ffffffff81113274>] ? ep_scan_ready_list+0x132/0x151                                   
[1375665.313458]  [<ffffffff811044e7>] ? do_splice_direct+0x4a/0x64                                       
[1375665.313488]  [<ffffffff810e8fa8>] ? do_sendfile+0x12d/0x1a8                                          
[1375665.313516]  [<ffffffff810e906c>] ? sys_sendfile64+0x49/0x88                                         
[1375665.313551]  [<ffffffff8103145f>] ? sysenter_dispatch+0x7/0x2e                                       
[1376138.351884] BUG: Bad page state in process lighttpd  pfn:283ab0                                      
[1376138.351918] page:ffffea0008ccd680 flags:0200000000001000 count:0 mapcount:0 mapping:(null) index:16fd
[1376138.351967] Pid: 2010, comm: lighttpd Tainted: G    B      2.6.33-2-amd64 #1                         
[1376138.352011] Call Trace:                                                                              
[1376138.352051]  [<ffffffff810b1e09>] ? bad_page+0x116/0x129                                             
[1376138.352079]  [<ffffffff810b3d8a>] ? get_page_from_freelist+0x4da/0x732                               
[1376138.352110]  [<ffffffff810b43fb>] ? __alloc_pages_nodemask+0x10f/0x5e0                               
[1376138.352140]  [<ffffffff810b5c99>] ? __do_page_cache_readahead+0x9b/0x1b4                             
[1376138.352170]  [<ffffffff810b5dce>] ? ra_submit+0x1c/0x20                                              
[1376138.352199]  [<ffffffff811057c2>] ? __generic_file_splice_read+0xfa/0x3ee                            
[1376138.352239]  [<ffffffff8126f365>] ? tcp_sendmsg+0x78a/0x87e                                          
[1376138.352291]  [<ffffffffa03115c9>] ? rpcauth_lookup_credcache+0x1c0/0x1d2 [sunrpc]                    
[1376138.352353]  [<ffffffffa038c544>] ? nfs_do_access+0x78/0x3d3 [nfs]                                   
[1376138.352386]  [<ffffffffa03115c9>] ? rpcauth_lookup_credcache+0x1c0/0x1d2 [sunrpc]                    
[1376138.352433]  [<ffffffff8105eecf>] ? bit_waitqueue+0x10/0xa0                                          
[1376138.352461]  [<ffffffff8105ef9d>] ? wake_up_bit+0x11/0x22                                            
[1376138.352493]  [<ffffffffa038f70d>] ? nfs_open+0x189/0x19a [nfs]                                       
[1376138.352522]  [<ffffffff81104182>] ? spd_release_page+0x0/0xe                                         
[1376138.352550]  [<ffffffff81105af0>] ? generic_file_splice_read+0x3a/0x62                               
[1376138.352580]  [<ffffffff811043d3>] ? splice_direct_to_actor+0xbe/0x188                                
[1376138.352609]  [<ffffffff81104a1c>] ? direct_splice_actor+0x0/0x1e                                     
[1376138.352638]  [<ffffffff811044e7>] ? do_splice_direct+0x4a/0x64                                       
[1376138.352672]  [<ffffffff810e8fa8>] ? do_sendfile+0x12d/0x1a8                                          
[1376138.352733]  [<ffffffff810e906c>] ? sys_sendfile64+0x49/0x88                                         
[1376138.352789]  [<ffffffff8103145f>] ? sysenter_dispatch+0x7/0x2e                                       
[1376401.907421] BUG: Bad page state in process kswapd0  pfn:30f01e                                       
[1376401.907473] page:ffffea000ab48690 flags:0200000000001008 count:0 mapcount:0 mapping:(null) index:c61 
[1376401.907522] Pid: 79, comm: kswapd0 Tainted: G    B      2.6.33-2-amd64 #1                            
[1376401.907557] Call Trace:                                                                              
[1376401.907590]  [<ffffffff810b4be3>] ? clear_page_dirty_for_io+0x54/0xb8                                
[1376401.907621]  [<ffffffff810b1e09>] ? bad_page+0x116/0x129                                             
[1376401.907660]  [<ffffffffa0399b53>] ? nfs_wb_page+0x63/0xc3 [nfs]                                      
[1376401.907689]  [<ffffffff810b22e9>] ? free_pages_check+0x38/0x57                                       
[1376401.907718]  [<ffffffff810b34f6>] ? free_hot_cold_page+0x46/0x176                                    
[1376401.907748]  [<ffffffff810b368f>] ? __pagevec_free+0x69/0x80                                         
[1376401.907782]  [<ffffffff810b86eb>] ? shrink_page_list+0x51e/0x617                                     
[1376401.907816]  [<ffffffff810b7715>] ? isolate_pages_global+0x1a0/0x20f                                 
[1376401.907847]  [<ffffffff810b91bb>] ? shrink_zone+0x710/0xae9                                          
[1376401.907876]  [<ffffffff810393d8>] ? dequeue_entity+0x18/0x134                                        
[1376401.907905]  [<ffffffff810b9ff2>] ? kswapd+0x5d3/0x80c                                               
[1376401.907932]  [<ffffffff810b7575>] ? isolate_pages_global+0x0/0x20f                                   
[1376401.907962]  [<ffffffff8105efae>] ? autoremove_wake_function+0x0/0x2e                                
[1376401.907992]  [<ffffffff810b9a1f>] ? kswapd+0x0/0x80c                                                 
[1376401.908018]  [<ffffffff8105eb75>] ? kthread+0x79/0x81                                                
[1376401.908046]  [<ffffffff810098e4>] ? kernel_thread_helper+0x4/0x10                                    
[1376401.908075]  [<ffffffff8105eafc>] ? kthread+0x0/0x81                                                 
[1376401.908101]  [<ffffffff810098e0>] ? kernel_thread_helper+0x0/0x10                                    
[1377387.230330] BUG: Bad page state in process lighttpd  pfn:c49ad                                       
[1377387.230365] page:ffffea0002b01dd8 flags:0100000000001000 count:0 mapcount:0 mapping:(null) index:24d3
[1377387.230418] Pid: 2010, comm: lighttpd Tainted: G    B      2.6.33-2-amd64 #1                         
[1377387.230466] Call Trace:                                                                              
[1377387.230500]  [<ffffffff810b1e09>] ? bad_page+0x116/0x129                                             
[1377387.230528]  [<ffffffff810b3d8a>] ? get_page_from_freelist+0x4da/0x732                               
[1377387.230559]  [<ffffffff810b43fb>] ? __alloc_pages_nodemask+0x10f/0x5e0                               
[1377387.230589]  [<ffffffff810b5c99>] ? __do_page_cache_readahead+0x9b/0x1b4                             
[1377387.230619]  [<ffffffff810b5dce>] ? ra_submit+0x1c/0x20                                              
[1377387.230648]  [<ffffffff811057c2>] ? __generic_file_splice_read+0xfa/0x3ee                            
[1377387.230680]  [<ffffffff8126f365>] ? tcp_sendmsg+0x78a/0x87e                                          
[1377387.230716]  [<ffffffffa03115c9>] ? rpcauth_lookup_credcache+0x1c0/0x1d2 [sunrpc]                    
[1377387.230769]  [<ffffffffa038c544>] ? nfs_do_access+0x78/0x3d3 [nfs]                                   
[1377387.230802]  [<ffffffffa03115c9>] ? rpcauth_lookup_credcache+0x1c0/0x1d2 [sunrpc]                    
[1377387.230849]  [<ffffffff8105eecf>] ? bit_waitqueue+0x10/0xa0                                          
[1377387.230877]  [<ffffffff8105ef9d>] ? wake_up_bit+0x11/0x22                                            
[1377387.230908]  [<ffffffffa038f70d>] ? nfs_open+0x189/0x19a [nfs]                                       
[1377387.230937]  [<ffffffff81104182>] ? spd_release_page+0x0/0xe                                         
[1377387.230965]  [<ffffffff81105af0>] ? generic_file_splice_read+0x3a/0x62                               
[1377387.230995]  [<ffffffff811043d3>] ? splice_direct_to_actor+0xbe/0x188                                
[1377387.231035]  [<ffffffff81104a1c>] ? direct_splice_actor+0x0/0x1e                                     
[1377387.231068]  [<ffffffff811044e7>] ? do_splice_direct+0x4a/0x64                                       
[1377387.231103]  [<ffffffff810e8fa8>] ? do_sendfile+0x12d/0x1a8                                          
[1377387.231131]  [<ffffffff810e906c>] ? sys_sendfile64+0x49/0x88                                         
[1377387.231160]  [<ffffffff8103145f>] ? sysenter_dispatch+0x7/0x2e                                       
[1381470.465990] [kslowd] preemptive burial: OBJ1a3b2 [OBJECT_RECYCLING] ffff880427d21240                 
[1381532.922939] [kslowd] preemptive burial: OBJ18998 [OBJECT_RECYCLING] ffff8803f2e71240                 
[1383344.577292] BUG: Bad page state in process lighttpd  pfn:19a0a9                                      
[1383344.577352] page:ffffea00059b24f8 flags:0200000000001000 count:0 mapcount:0 mapping:(null) index:46eb
[1383344.577405] Pid: 2009, comm: lighttpd Tainted: G    B      2.6.33-2-amd64 #1                         
[1383344.577453] Call Trace:                                                                              
[1383344.577529]  [<ffffffff810b1e09>] ? bad_page+0x116/0x129                                             
[1383344.577558]  [<ffffffff810b3d8a>] ? get_page_from_freelist+0x4da/0x732                               
[1383344.577589]  [<ffffffff810b43fb>] ? __alloc_pages_nodemask+0x10f/0x5e0                               
[1383344.577620]  [<ffffffff810b5c99>] ? __do_page_cache_readahead+0x9b/0x1b4                             
[1383344.577651]  [<ffffffff810b5dce>] ? ra_submit+0x1c/0x20                                              
[1383344.577686]  [<ffffffff811057c2>] ? __generic_file_splice_read+0xfa/0x3ee                            
[1383344.577721]  [<ffffffff8126f365>] ? tcp_sendmsg+0x78a/0x87e                                          
[1383344.577763]  [<ffffffffa03115c9>] ? rpcauth_lookup_credcache+0x1c0/0x1d2 [sunrpc]                    
[1383344.577825]  [<ffffffffa038c544>] ? nfs_do_access+0x78/0x3d3 [nfs]                                   
[1383344.577858]  [<ffffffffa03115c9>] ? rpcauth_lookup_credcache+0x1c0/0x1d2 [sunrpc]                    
[1383344.577905]  [<ffffffff8105eecf>] ? bit_waitqueue+0x10/0xa0                                          
[1383344.577933]  [<ffffffff8105ef9d>] ? wake_up_bit+0x11/0x22                                            
[1383344.577965]  [<ffffffffa038f70d>] ? nfs_open+0x189/0x19a [nfs]                                       
[1383344.577995]  [<ffffffff81104182>] ? spd_release_page+0x0/0xe                                         
[1383344.578024]  [<ffffffff81105af0>] ? generic_file_splice_read+0x3a/0x62                               
[1383344.578054]  [<ffffffff811043d3>] ? splice_direct_to_actor+0xbe/0x188                                
[1383344.578083]  [<ffffffff81104a1c>] ? direct_splice_actor+0x0/0x1e                                     
[1383344.578112]  [<ffffffff811044e7>] ? do_splice_direct+0x4a/0x64                                       
[1383344.578141]  [<ffffffff810e8fa8>] ? do_sendfile+0x12d/0x1a8                                          
[1383344.584880]  [<ffffffff810e906c>] ? sys_sendfile64+0x49/0x88                                         
[1383344.584910]  [<ffffffff8103145f>] ? sysenter_dispatch+0x7/0x2e                                       
[1385688.846934] [kslowd] preemptive burial: OBJ1a515 [OBJECT_RECYCLING] ffff880427961b40                 
[1389074.622555] BUG: Bad page state in process kswapd0  pfn:c1702                                        
[1389074.622589] page:ffffea0002a50870 flags:0100000000001008 count:0 mapcount:0 mapping:(null) index:17f8
[1389074.622638] Pid: 79, comm: kswapd0 Tainted: G    B      2.6.33-2-amd64 #1                            
[1389074.622667] Call Trace:                                                                              
[1389074.622696]  [<ffffffff810b4be3>] ? clear_page_dirty_for_io+0x54/0xb8                                
[1389074.622727]  [<ffffffff810b1e09>] ? bad_page+0x116/0x129                                             
[1389074.622766]  [<ffffffffa0399b53>] ? nfs_wb_page+0x63/0xc3 [nfs]                                      
[1389074.622795]  [<ffffffff810b22e9>] ? free_pages_check+0x38/0x57                                       
[1389074.622824]  [<ffffffff810b34f6>] ? free_hot_cold_page+0x46/0x176                                    
[1389074.622853]  [<ffffffff810b368f>] ? __pagevec_free+0x69/0x80                                         
[1389074.622886]  [<ffffffff810b86eb>] ? shrink_page_list+0x51e/0x617                                     
[1389074.622922]  [<ffffffff81187f6a>] ? __bitmap_weight+0x3a/0x7e                                        
[1389074.623004]  [<ffffffff810b7715>] ? isolate_pages_global+0x1a0/0x20f                                 
[1389074.623034]  [<ffffffff810b91bb>] ? shrink_zone+0x710/0xae9                                          
[1389074.623063]  [<ffffffff810b9ff2>] ? kswapd+0x5d3/0x80c                                               
[1389074.623090]  [<ffffffff810b7575>] ? isolate_pages_global+0x0/0x20f                                   
[1389074.623122]  [<ffffffff8105efae>] ? autoremove_wake_function+0x0/0x2e                                
[1389074.623151]  [<ffffffff810b9a1f>] ? kswapd+0x0/0x80c                                                 
[1389074.623178]  [<ffffffff8105eb75>] ? kthread+0x79/0x81                                                
[1389074.623207]  [<ffffffff810098e4>] ? kernel_thread_helper+0x4/0x10                                    
[1389074.623236]  [<ffffffff8105eafc>] ? kthread+0x0/0x81                                                 
[1389074.623264]  [<ffffffff810098e0>] ? kernel_thread_helper+0x0/0x10                                    
[1389263.668912] BUG: Bad page state in process lighttpd  pfn:357ac9                                      
[1389263.668953] page:ffffea000bb2dbf8 flags:0200000000001000 count:0 mapcount:0 mapping:(null) index:27d2
[1389263.669006] Pid: 2006, comm: lighttpd Tainted: G    B      2.6.33-2-amd64 #1                         
[1389263.669053] Call Trace:                                                                              
[1389263.669081]  [<ffffffff810b1e09>] ? bad_page+0x116/0x129                                             
[1389263.669115]  [<ffffffff810b3d8a>] ? get_page_from_freelist+0x4da/0x732                               
[1389263.669155]  [<ffffffff810b43fb>] ? __alloc_pages_nodemask+0x10f/0x5e0                               
[1389263.669191]  [<ffffffff8123ff43>] ? dev_queue_xmit+0x3ea/0x41c                                       
[1389263.669221]  [<ffffffff812752fc>] ? tcp_ack+0x17ac/0x197e                                            
[1389263.669249]  [<ffffffff810b5c99>] ? __do_page_cache_readahead+0x9b/0x1b4                             
[1389263.669279]  [<ffffffff810b5dce>] ? ra_submit+0x1c/0x20                                              
[1389263.669308]  [<ffffffff811057c2>] ? __generic_file_splice_read+0xfa/0x3ee                            
[1389263.669339]  [<ffffffff812770d8>] ? tcp_current_mss+0x3f/0x5a                                        
[1389263.669368]  [<ffffffff812ea851>] ? _raw_spin_lock_bh+0x9/0x25                                       
[1389263.669397]  [<ffffffff8123312d>] ? release_sock+0x13/0x96                                           
[1389263.669425]  [<ffffffff8126f8a4>] ? tcp_sendpage+0x44b/0x45d                                         
[1389263.669453]  [<ffffffff81103ab0>] ? pipe_to_sendpage+0x0/0x74                                        
[1389263.669481]  [<ffffffff8122ec69>] ? kernel_sendpage+0x16/0x1f                                        
[1389263.669520]  [<ffffffff8122eca7>] ? sock_sendpage+0x35/0x39                                          
[1389263.669551]  [<ffffffff81104182>] ? spd_release_page+0x0/0xe                                         
[1389263.669584]  [<ffffffff81105af0>] ? generic_file_splice_read+0x3a/0x62                               
[1389263.669614]  [<ffffffff811043d3>] ? splice_direct_to_actor+0xbe/0x188                                
[1389263.669643]  [<ffffffff81104a1c>] ? direct_splice_actor+0x0/0x1e                                     
[1389263.669675]  [<ffffffff81113274>] ? ep_scan_ready_list+0x132/0x151                                   
[1389263.669704]  [<ffffffff811044e7>] ? do_splice_direct+0x4a/0x64                                       
[1389263.669736]  [<ffffffff810e8fa8>] ? do_sendfile+0x12d/0x1a8                                          
[1389263.669765]  [<ffffffff810e906c>] ? sys_sendfile64+0x49/0x88                                         
[1389263.669795]  [<ffffffff8103145f>] ? sysenter_dispatch+0x7/0x2e                                       
[1389564.137851] [kslowd] preemptive burial: OBJ1a62e [OBJECT_RECYCLING] ffff8803784c00c0                 
[1390224.597304] BUG: Bad page state in process lighttpd  pfn:100d3d                                      
[1390224.597338] page:ffffea000382e558 flags:0200000000001000 count:0 mapcount:0 mapping:(null) index:4809
[1390224.597396] Pid: 2009, comm: lighttpd Tainted: G    B      2.6.33-2-amd64 #1                         
[1390224.597439] Call Trace:                                                                              
[1390224.597468]  [<ffffffff810b1e09>] ? bad_page+0x116/0x129                                             
[1390224.597496]  [<ffffffff810b3d8a>] ? get_page_from_freelist+0x4da/0x732                               
[1390224.597527]  [<ffffffff810b43fb>] ? __alloc_pages_nodemask+0x10f/0x5e0                               
[1390224.597646]  [<ffffffff8123ff43>] ? dev_queue_xmit+0x3ea/0x41c                                       
[1390224.597676]  [<ffffffff812752fc>] ? tcp_ack+0x17ac/0x197e                                            
[1390224.597704]  [<ffffffff810b5c99>] ? __do_page_cache_readahead+0x9b/0x1b4                             
[1390224.597735]  [<ffffffff810b5dce>] ? ra_submit+0x1c/0x20                                              
[1390224.597781]  [<ffffffff811057c2>] ? __generic_file_splice_read+0xfa/0x3ee                            
[1390224.597814]  [<ffffffff812770d8>] ? tcp_current_mss+0x3f/0x5a                                        
[1390224.597852]  [<ffffffff812ea851>] ? _raw_spin_lock_bh+0x9/0x25                                       
[1390224.597882]  [<ffffffff8123312d>] ? release_sock+0x13/0x96                                           
[1390224.597911]  [<ffffffff8126f8a4>] ? tcp_sendpage+0x44b/0x45d                                         
[1390224.597939]  [<ffffffff81103ab0>] ? pipe_to_sendpage+0x0/0x74                                        
[1390224.597968]  [<ffffffff8122ec69>] ? kernel_sendpage+0x16/0x1f                                        
[1390224.598002]  [<ffffffff8122eca7>] ? sock_sendpage+0x35/0x39                                          
[1390224.598030]  [<ffffffff81104182>] ? spd_release_page+0x0/0xe                                         
[1390224.598059]  [<ffffffff81105af0>] ? generic_file_splice_read+0x3a/0x62                               
[1390224.598093]  [<ffffffff811043d3>] ? splice_direct_to_actor+0xbe/0x188                                
[1390224.598122]  [<ffffffff81104a1c>] ? direct_splice_actor+0x0/0x1e                                     
[1390224.598153]  [<ffffffff81113274>] ? ep_scan_ready_list+0x132/0x151                                   
[1390224.598186]  [<ffffffff811044e7>] ? do_splice_direct+0x4a/0x64                                       
[1390224.598221]  [<ffffffff810e8fa8>] ? do_sendfile+0x12d/0x1a8                                          
[1390224.598249]  [<ffffffff810e906c>] ? sys_sendfile64+0x49/0x88                                         
[1390224.598278]  [<ffffffff8103145f>] ? sysenter_dispatch+0x7/0x2e                                       
[1391491.178193] BUG: Bad page state in process lighttpd  pfn:11de92                                      
[1391491.178231] page:ffffea0003e8aff0 flags:0200000000001000 count:0 mapcount:0 mapping:(null) index:1de9
[1391491.178280] Pid: 2010, comm: lighttpd Tainted: G    B      2.6.33-2-amd64 #1                         
[1391491.178323] Call Trace:                                                                              
[1391491.178351]  [<ffffffff810b1e09>] ? bad_page+0x116/0x129                                             
[1391491.178380]  [<ffffffff810b3d8a>] ? get_page_from_freelist+0x4da/0x732                               
[1391491.178410]  [<ffffffff810b43fb>] ? __alloc_pages_nodemask+0x10f/0x5e0                               
[1391491.178449]  [<ffffffffa038f2e6>] ? __nfs_revalidate_inode+0x11b/0x1a5 [nfs]                         
[1391491.178495]  [<ffffffff812eaa8e>] ? common_interrupt+0xe/0x13                                        
[1391491.178524]  [<ffffffff810da2af>] ? policy_nodemask+0x0/0x37                                         
[1391491.178553]  [<ffffffff810b5c99>] ? __do_page_cache_readahead+0x9b/0x1b4                             
[1391491.178583]  [<ffffffff810b5fc0>] ? ondemand_readahead+0x11f/0x1a7                                   
[1391491.178614]  [<ffffffff811057c2>] ? __generic_file_splice_read+0xfa/0x3ee                            
[1391491.178650]  [<ffffffff8126f365>] ? tcp_sendmsg+0x78a/0x87e                                          
[1391491.178713]  [<ffffffffa03115c9>] ? rpcauth_lookup_credcache+0x1c0/0x1d2 [sunrpc]                    
[1391491.178771]  [<ffffffffa038c544>] ? nfs_do_access+0x78/0x3d3 [nfs]                                   
[1391491.178804]  [<ffffffffa03115c9>] ? rpcauth_lookup_credcache+0x1c0/0x1d2 [sunrpc]                    
[1391491.178851]  [<ffffffff8105eecf>] ? bit_waitqueue+0x10/0xa0                                          
[1391491.178880]  [<ffffffff81104182>] ? spd_release_page+0x0/0xe                                         
[1391491.178908]  [<ffffffff81105af0>] ? generic_file_splice_read+0x3a/0x62                               
[1391491.178938]  [<ffffffff811043d3>] ? splice_direct_to_actor+0xbe/0x188                                
[1391491.178967]  [<ffffffff81104a1c>] ? direct_splice_actor+0x0/0x1e                                     
[1391491.178996]  [<ffffffff811044e7>] ? do_splice_direct+0x4a/0x64                                       
[1391491.179026]  [<ffffffff810e8fa8>] ? do_sendfile+0x12d/0x1a8                                          
[1391491.179054]  [<ffffffff810e906c>] ? sys_sendfile64+0x49/0x88                                         
[1391491.179083]  [<ffffffff8103145f>] ? sysenter_dispatch+0x7/0x2e                                       
[1391695.363181] [kslowd] preemptive burial: OBJ1a6eb [OBJECT_RECYCLING] ffff8801bc582840                 
[1392962.293688] [kslowd] preemptive burial: OBJ1a747 [OBJECT_RECYCLING] ffff8800cc4d1b40                 
[1393851.730365] BUG: Bad page state in process cachefilesd  pfn:244d07                                   
[1393851.730399] page:ffffea0007f0d988 flags:0200000000001008 count:0 mapcount:0 mapping:(null) index:3579
[1393851.730452] Pid: 1893, comm: cachefilesd Tainted: G    B      2.6.33-2-amd64 #1                      
[1393851.730509] Call Trace:                                                                              
[1393851.730537]  [<ffffffff810b1e09>] ? bad_page+0x116/0x129                                             
[1393851.730565]  [<ffffffff810b22e9>] ? free_pages_check+0x38/0x57                                       
[1393851.730594]  [<ffffffff810b34f6>] ? free_hot_cold_page+0x46/0x176                                    
[1393851.730623]  [<ffffffff810b368f>] ? __pagevec_free+0x69/0x80                                         
[1393851.730652]  [<ffffffff810b66fa>] ? release_pages+0x17b/0x18d                                        
[1393851.730682]  [<ffffffff810b686c>] ? ____pagevec_lru_add+0x160/0x176                                  
[1393851.737476]  [<ffffffff810b6acd>] ? __pagevec_release+0x19/0x22                                      
[1393851.737506]  [<ffffffff810b724c>] ? truncate_inode_pages_range+0xdb/0x2b0                            
[1393851.737537]  [<ffffffff810df692>] ? virt_to_head_page+0x9/0x2a                                       
[1393851.737578]  [<ffffffffa0275ca1>] ? jbd2_journal_stop+0x222/0x235 [jbd2]                             
[1393851.737615]  [<ffffffff812e9ccb>] ? mutex_lock+0xd/0x33                                              
[1393851.737649]  [<ffffffffa0291033>] ? ext4_delete_inode+0x5d/0x246 [ext4]                              
[1393851.737692]  [<ffffffffa0290fd6>] ? ext4_delete_inode+0x0/0x246 [ext4]                               
[1393851.737723]  [<ffffffff810fa9d9>] ? generic_delete_inode+0xdc/0x168                                  
[1393851.737763]  [<ffffffff810f8faf>] ? d_delete+0x47/0xb2                                               
[1393851.737793]  [<ffffffff810f1fb5>] ? vfs_unlink+0x9f/0xa6                                             
[1393851.737831]  [<ffffffffa03d95c4>] ? cachefiles_bury_object+0x3e/0x2db [cachefiles]                   
[1393851.737881]  [<ffffffff811002e7>] ? vfs_removexattr+0x87/0x92                                        
[1393851.737910]  [<ffffffffa03d98a4>] ? cachefiles_cull+0x43/0x9c [cachefiles]                           
[1393851.737941]  [<ffffffffa03d7eb3>] ? cachefiles_daemon_cull+0xc0/0x113 [cachefiles]                   
[1393851.737987]  [<ffffffffa03d808e>] ? cachefiles_daemon_write+0x188/0x1ab [cachefiles]                 
[1393851.738033]  [<ffffffff810e95d5>] ? vfs_write+0xa9/0x102                                             
[1393851.738061]  [<ffffffff810e96ea>] ? sys_write+0x45/0x6e                                              
[1393851.738090]  [<ffffffff8103145f>] ? sysenter_dispatch+0x7/0x2e                                       
[1394470.167054] [kslowd] preemptive burial: OBJ1a788 [OBJECT_RECYCLING] ffff88042e149d80                 
[1395168.400507] BUG: Bad page state in process lighttpd  pfn:a7642                                       
[1395168.400541] page:ffffea000249de70 flags:0100000000001000 count:0 mapcount:0 mapping:(null) index:71c 
[1395168.400599] Pid: 2008, comm: lighttpd Tainted: G    B      2.6.33-2-amd64 #1                         
[1395168.400642] Call Trace:                                                                              
[1395168.400683]  [<ffffffff810b1e09>] ? bad_page+0x116/0x129                                             
[1395168.400712]  [<ffffffff810b3d8a>] ? get_page_from_freelist+0x4da/0x732                               
[1395168.400742]  [<ffffffff810b43fb>] ? __alloc_pages_nodemask+0x10f/0x5e0                               
[1395168.400784]  [<ffffffff81053de2>] ? __mod_timer+0x141/0x153                                          
[1395168.400814]  [<ffffffff812335e3>] ? sk_reset_timer+0xf/0x1d                                          
[1395168.400843]  [<ffffffff81279e1b>] ? tcp_write_xmit+0x466/0x96c                                       
[1395168.400875]  [<ffffffff810b5c99>] ? __do_page_cache_readahead+0x9b/0x1b4                             
[1395168.400905]  [<ffffffff810b5dce>] ? ra_submit+0x1c/0x20                                              
[1395168.400934]  [<ffffffff811057c2>] ? __generic_file_splice_read+0xfa/0x3ee                            
[1395168.400964]  [<ffffffff812770d8>] ? tcp_current_mss+0x3f/0x5a                                        
[1395168.400993]  [<ffffffff812ea851>] ? _raw_spin_lock_bh+0x9/0x25                                       
[1395168.401034]  [<ffffffff8123312d>] ? release_sock+0x13/0x96                                           
[1395168.401064]  [<ffffffff8126f8a4>] ? tcp_sendpage+0x44b/0x45d                                         
[1395168.401093]  [<ffffffff81103ab0>] ? pipe_to_sendpage+0x0/0x74                                        
[1395168.401121]  [<ffffffff8122ec69>] ? kernel_sendpage+0x16/0x1f                                        
[1395168.401149]  [<ffffffff8122eca7>] ? sock_sendpage+0x35/0x39                                          
[1395168.401177]  [<ffffffff81104182>] ? spd_release_page+0x0/0xe                                         
[1395168.401205]  [<ffffffff81105af0>] ? generic_file_splice_read+0x3a/0x62                               
[1395168.401235]  [<ffffffff811043d3>] ? splice_direct_to_actor+0xbe/0x188                                
[1395168.401265]  [<ffffffff81104a1c>] ? direct_splice_actor+0x0/0x1e                                     
[1395168.401295]  [<ffffffff81113274>] ? ep_scan_ready_list+0x132/0x151                                   
[1395168.401325]  [<ffffffff811044e7>] ? do_splice_direct+0x4a/0x64                                       
[1395168.401362]  [<ffffffff810e8fa8>] ? do_sendfile+0x12d/0x1a8                                          
[1395168.401390]  [<ffffffff810e906c>] ? sys_sendfile64+0x49/0x88                                         
[1395168.401419]  [<ffffffff8103145f>] ? sysenter_dispatch+0x7/0x2e                                       
[1396895.488807] BUG: Bad page state in process lighttpd  pfn:580e6                                       
[1396895.488841] page:ffffea0001343250 flags:0100000000001000 count:0 mapcount:0 mapping:(null) index:1bd1
[1396895.488889] Pid: 2006, comm: lighttpd Tainted: G    B      2.6.33-2-amd64 #1                         
[1396895.488933] Call Trace:                                                                              
[1396895.488961]  [<ffffffff810b1e09>] ? bad_page+0x116/0x129                                             
[1396895.488990]  [<ffffffff810b3d8a>] ? get_page_from_freelist+0x4da/0x732                               
[1396895.489035]  [<ffffffff810b43fb>] ? __alloc_pages_nodemask+0x10f/0x5e0                               
[1396895.489066]  [<ffffffff8110e03b>] ? mpage_bio_submit+0x22/0x26                                       
[1396895.489094]  [<ffffffff8110eaaf>] ? mpage_readpage+0x68/0x72                                         
[1396895.489124]  [<ffffffff810e1c63>] ? __kmalloc+0x12f/0x141                                            
[1396895.489155]  [<ffffffffa03dab65>] ? cachefiles_read_or_alloc_pages+0x3b4/0x7e5 [cachefiles]          
[1396895.489216]  [<ffffffff810387e3>] ? __wake_up+0x30/0x44                                              
[1396895.489244]  [<ffffffff8105ef87>] ? __wake_up_bit+0x28/0x2d                                          
[1396895.489275]  [<ffffffffa035ffe1>] ? fscache_run_op+0x20/0x38 [fscache]                               
[1396895.489320]  [<ffffffffa03aec33>] ? nfs_readpage_from_fscache_complete+0x0/0x5c [nfs]                
[1396895.489371]  [<ffffffffa036170a>] ? __fscache_read_or_alloc_pages+0x1e4/0x260 [fscache]              
[1396895.489427]  [<ffffffffa03aeb69>] ? __nfs_readpages_from_fscache+0x77/0x141 [nfs]                    
[1396895.489482]  [<ffffffffa0396f8d>] ? nfs_readpages+0xf4/0x18d [nfs]                                   
[1396895.489517]  [<ffffffff810b5d19>] ? __do_page_cache_readahead+0x11b/0x1b4                            
[1396895.489548]  [<ffffffff810b5dce>] ? ra_submit+0x1c/0x20                                              
[1396895.489576]  [<ffffffff811057c2>] ? __generic_file_splice_read+0xfa/0x3ee                            
[1396895.489608]  [<ffffffff812770d8>] ? tcp_current_mss+0x3f/0x5a                                        
[1396895.489638]  [<ffffffff812ea851>] ? _raw_spin_lock_bh+0x9/0x25                                       
[1396895.489667]  [<ffffffff8123312d>] ? release_sock+0x13/0x96                                           
[1396895.489695]  [<ffffffff8126f8a4>] ? tcp_sendpage+0x44b/0x45d                                         
[1396895.489724]  [<ffffffff81103ab0>] ? pipe_to_sendpage+0x0/0x74                                        
[1396895.489752]  [<ffffffff8122ec69>] ? kernel_sendpage+0x16/0x1f                                        
[1396895.489780]  [<ffffffff8122eca7>] ? sock_sendpage+0x35/0x39                                          
[1396895.489808]  [<ffffffff81104182>] ? spd_release_page+0x0/0xe                                         
[1396895.489836]  [<ffffffff81105af0>] ? generic_file_splice_read+0x3a/0x62                               
[1396895.489870]  [<ffffffff811043d3>] ? splice_direct_to_actor+0xbe/0x188                                
[1396895.489924]  [<ffffffff81104a1c>] ? direct_splice_actor+0x0/0x1e                                     
[1396895.489954]  [<ffffffff81113274>] ? ep_scan_ready_list+0x132/0x151                                   
[1396895.489988]  [<ffffffff811044e7>] ? do_splice_direct+0x4a/0x64                                       
[1396895.490021]  [<ffffffff810e8fa8>] ? do_sendfile+0x12d/0x1a8                                          
[1396895.490050]  [<ffffffff8106685b>] ? getnstimeofday+0x55/0xaf                                         
[1396895.490078]  [<ffffffff810e906c>] ? sys_sendfile64+0x49/0x88                                         
[1396895.490107]  [<ffffffff8103145f>] ? sysenter_dispatch+0x7/0x2e                                       
[1399172.136795] BUG: Bad page state in process lighttpd  pfn:1607fc                                      
[1399172.136828] page:ffffea0004d1bf20 flags:0200000000001000 count:0 mapcount:0 mapping:(null) index:a17 
[1399172.136877] Pid: 2008, comm: lighttpd Tainted: G    B      2.6.33-2-amd64 #1                         
[1399172.136967] Call Trace:                                                                              
[1399172.137000]  [<ffffffff810b1e09>] ? bad_page+0x116/0x129                                             
[1399172.137038]  [<ffffffff810b3d8a>] ? get_page_from_freelist+0x4da/0x732                               
[1399172.137075]  [<ffffffff810b43fb>] ? __alloc_pages_nodemask+0x10f/0x5e0                               
[1399172.137106]  [<ffffffff8110e03b>] ? mpage_bio_submit+0x22/0x26                                       
[1399172.137134]  [<ffffffff8110eaaf>] ? mpage_readpage+0x68/0x72                                         
[1399172.137164]  [<ffffffff810e1c63>] ? __kmalloc+0x12f/0x141                                            
[1399172.137195]  [<ffffffffa03dab65>] ? cachefiles_read_or_alloc_pages+0x3b4/0x7e5 [cachefiles]          
[1399172.137263]  [<ffffffffa03aec33>] ? nfs_readpage_from_fscache_complete+0x0/0x5c [nfs]                
[1399172.137347]  [<ffffffffa036170a>] ? __fscache_read_or_alloc_pages+0x1e4/0x260 [fscache]              
[1399172.137408]  [<ffffffffa03aeb69>] ? __nfs_readpages_from_fscache+0x77/0x141 [nfs]                    
[1399172.137472]  [<ffffffffa0396f8d>] ? nfs_readpages+0xf4/0x18d [nfs]                                   
[1399172.137503]  [<ffffffff810b5d19>] ? __do_page_cache_readahead+0x11b/0x1b4                            
[1399172.137533]  [<ffffffff810b5dce>] ? ra_submit+0x1c/0x20                                              
[1399172.137562]  [<ffffffff811057c2>] ? __generic_file_splice_read+0xfa/0x3ee                            
[1399172.137594]  [<ffffffff8126f365>] ? tcp_sendmsg+0x78a/0x87e                                          
[1399172.137629]  [<ffffffffa03115c9>] ? rpcauth_lookup_credcache+0x1c0/0x1d2 [sunrpc]                    
[1399172.137682]  [<ffffffffa038c544>] ? nfs_do_access+0x78/0x3d3 [nfs]                                   
[1399172.137752]  [<ffffffffa03115c9>] ? rpcauth_lookup_credcache+0x1c0/0x1d2 [sunrpc]                    
[1399172.137815]  [<ffffffff8105eecf>] ? bit_waitqueue+0x10/0xa0                                          
[1399172.137843]  [<ffffffff81104182>] ? spd_release_page+0x0/0xe                                         
[1399172.137872]  [<ffffffff81105af0>] ? generic_file_splice_read+0x3a/0x62                               
[1399172.137922]  [<ffffffff811043d3>] ? splice_direct_to_actor+0xbe/0x188                                
[1399172.137952]  [<ffffffff81104a1c>] ? direct_splice_actor+0x0/0x1e                                     
[1399172.137980]  [<ffffffff811044e7>] ? do_splice_direct+0x4a/0x64                                       
[1399172.138009]  [<ffffffff810e8fa8>] ? do_sendfile+0x12d/0x1a8                                          
[1399172.138038]  [<ffffffff810e906c>] ? sys_sendfile64+0x49/0x88                                         
[1399172.138067]  [<ffffffff8103145f>] ? sysenter_dispatch+0x7/0x2e                                       
[1399599.766973] [kslowd] preemptive burial: OBJ1a43c [OBJECT_RECYCLING] ffff8804279720c0                 
[1400319.745134] [kslowd] preemptive burial: OBJ1a973 [OBJECT_RECYCLING] ffff88014aa66840                 
[1400380.307848] [kslowd] preemptive burial: OBJ1a97a [OBJECT_RECYCLING] ffff88042942f300                 
[1400412.475771] [kslowd] preemptive burial: OBJ1a97e [OBJECT_RECYCLING] ffff8804279549c0                 
[1401365.768195] BUG: Bad page state in process lighttpd  pfn:192d5b                                      
[1401365.768229] page:ffffea000581ebe8 flags:0200000000001000 count:0 mapcount:0 mapping:(null) index:2d375
[1401365.768291] Pid: 2010, comm: lighttpd Tainted: G    B      2.6.33-2-amd64 #1                          
[1401365.768341] Call Trace:                                                                               
[1401365.768371]  [<ffffffff810b1e09>] ? bad_page+0x116/0x129                                              
[1401365.775354]  [<ffffffff810b3d8a>] ? get_page_from_freelist+0x4da/0x732                                
[1401365.775385]  [<ffffffff810b4593>] ? __alloc_pages_nodemask+0x2a7/0x5e0                                
[1401365.775415]  [<ffffffff810b5c99>] ? __do_page_cache_readahead+0x9b/0x1b4                              
[1401365.775446]  [<ffffffff810b5dce>] ? ra_submit+0x1c/0x20                                               
[1401365.775475]  [<ffffffff811057c2>] ? __generic_file_splice_read+0xfa/0x3ee                             
[1401365.775507]  [<ffffffff8123a7d9>] ? sk_stream_wait_memory+0x1ae/0x1eb                                 
[1401365.775544]  [<ffffffff81007689>] ? __switch_to+0x1df/0x2a6                                           
[1401365.775574]  [<ffffffff810393d8>] ? dequeue_entity+0x18/0x134                                         
[1401365.775610]  [<ffffffff81104182>] ? spd_release_page+0x0/0xe                                          
[1401365.775638]  [<ffffffff81105af0>] ? generic_file_splice_read+0x3a/0x62                                
[1401365.775668]  [<ffffffff811043d3>] ? splice_direct_to_actor+0xbe/0x188                                 
[1401365.775697]  [<ffffffff81104a1c>] ? direct_splice_actor+0x0/0x1e                                      
[1401365.775736]  [<ffffffff81113274>] ? ep_scan_ready_list+0x132/0x151                                    
[1401365.775765]  [<ffffffff811044e7>] ? do_splice_direct+0x4a/0x64                                        
[1401365.775795]  [<ffffffff810e8fa8>] ? do_sendfile+0x12d/0x1a8                                           
[1401365.775828]  [<ffffffff810e906c>] ? sys_sendfile64+0x49/0x88                                          
[1401365.775862]  [<ffffffff8103145f>] ? sysenter_dispatch+0x7/0x2e                                        
[1401481.934695] [kslowd] preemptive burial: OBJ1a7e7 [OBJECT_RECYCLING] ffff8803f633d000                  
[1402026.177911] [kslowd] preemptive burial: OBJ1a9e0 [OBJECT_RECYCLING] ffff88042971da80                  
[1402189.710924] BUG: Bad page state in process lighttpd  pfn:5b461                                        
[1402189.710958] page:ffffea00013f7538 flags:0100000000001000 count:0 mapcount:0 mapping:(null) index:17fa 
[1402189.711007] Pid: 2003, comm: lighttpd Tainted: G    B      2.6.33-2-amd64 #1                          
[1402189.711050] Call Trace:                                                                               
[1402189.711079]  [<ffffffff810b1e09>] ? bad_page+0x116/0x129                                              
[1402189.711107]  [<ffffffff810b3d8a>] ? get_page_from_freelist+0x4da/0x732                                
[1402189.711138]  [<ffffffff810b43fb>] ? __alloc_pages_nodemask+0x10f/0x5e0                                
[1402189.711168]  [<ffffffff810b5c99>] ? __do_page_cache_readahead+0x9b/0x1b4                              
[1402189.711198]  [<ffffffff810b5dce>] ? ra_submit+0x1c/0x20                                               
[1402189.711232]  [<ffffffff811057c2>] ? __generic_file_splice_read+0xfa/0x3ee                             
[1402189.711268]  [<ffffffff8126f365>] ? tcp_sendmsg+0x78a/0x87e                                           
[1402189.711305]  [<ffffffffa03115c9>] ? rpcauth_lookup_credcache+0x1c0/0x1d2 [sunrpc]                     
[1402189.711358]  [<ffffffffa038c544>] ? nfs_do_access+0x78/0x3d3 [nfs]                                    
[1402189.711391]  [<ffffffffa03115c9>] ? rpcauth_lookup_credcache+0x1c0/0x1d2 [sunrpc]                     
[1402189.711443]  [<ffffffff8105eecf>] ? bit_waitqueue+0x10/0xa0                                           
[1402189.711471]  [<ffffffff81104182>] ? spd_release_page+0x0/0xe                                          
[1402189.711499]  [<ffffffff81105af0>] ? generic_file_splice_read+0x3a/0x62                                
[1402189.711529]  [<ffffffff811043d3>] ? splice_direct_to_actor+0xbe/0x188                                 
[1402189.711558]  [<ffffffff81104a1c>] ? direct_splice_actor+0x0/0x1e                                      
[1402189.711587]  [<ffffffff811044e7>] ? do_splice_direct+0x4a/0x64                                        
[1402189.711617]  [<ffffffff810e8fa8>] ? do_sendfile+0x12d/0x1a8                                           
[1402189.711645]  [<ffffffff810e906c>] ? sys_sendfile64+0x49/0x88                                          
[1402189.711693]  [<ffffffff8103145f>] ? sysenter_dispatch+0x7/0x2e                                        
[1402803.910222] [kslowd] preemptive burial: OBJ1aa28 [OBJECT_RECYCLING] ffff880427b5c840                  
[1404463.190690] [kslowd] preemptive burial: OBJ1a256 [OBJECT_RECYCLING] ffff880429546240                  
[1404492.032376] [kslowd] preemptive burial: OBJ1a67c [OBJECT_RECYCLING] ffff8804294f0600                  
[1404553.660461] [kslowd] preemptive burial: OBJ19e95 [OBJECT_RECYCLING] ffff88029366eb40                  
[1404624.564976] [kslowd] preemptive burial: OBJ1ab0d [OBJECT_RECYCLING] ffff880427ffbb40                  
[1404624.784617]                                                                                           
[1404624.784640] CacheFiles: Error: Unexpected object collision                                            
[1404624.784667] object: OBJ1ab18                                                                          
[1404624.784690] objstate=OBJECT_LOOKING_UP fl=0 swfl=a ev=0[7b]                                           
[1404624.784717] ops=0 inp=0 exc=0                                                                         
[1404624.784739] parent=ffff88042d15a280                                                                   
[1404624.784762] cookie=ffff88042ae9c780 [pr=ffff88042ae9c000 nd=ffff8803a65403f0 fl=7]                    
[1404624.784807] key=[36] '010007010100f90d000000006a76684038e942a29b3f0925b1794dc4840c0d0e84a48def'       
[1404624.784922] xobject: OBJ1a2a4                                                                         
[1404624.784949] xobjstate=OBJECT_ACTIVE fl=0 swfl=8 ev=4[7b]                                              
[1404624.784980] xops=0 inp=0 exc=0                                                                        
[1404624.785002] xparent=ffff88042d15a280                                                                  
[1404624.785026] xcookie=ffff8802b0cf28c0 [pr=ffff88042ae9c000 nd=ffff88021b1afa10 fl=8]                   
[1404624.785071] xkey=[36] '010007010100f90d000000006a76684038e942a29b3f0925b1794dc4840c0d0e84a48def'      
[1404624.785216] ------------[ cut here ]------------                                                      
[1404624.785241] kernel BUG at /usr/src/linux-2.6-2.6.33/debian/build/source_i386_none/fs/cachefiles/namei.c:200!
[1404624.785290] invalid opcode: 0000 [#1] SMP                                                                   
[1404624.785319] last sysfs file: /sys/devices/pci0000:00/0000:00:1d.7/usb4/4-2/4-2:1.0/host5/target5:0:0/5:0:0:0/block/sr0/removable                                                                                                                   
[1404624.785375] CPU 6
[1404624.785399] Pid: 13379, comm: kslowd002 Tainted: G    B      2.6.33-2-amd64 #1 0P010H/PowerEdge M600
[1404624.785446] RIP: 0010:[<ffffffffa03da055>]  [<ffffffffa03da055>] cachefiles_walk_to_object+0x461/0x7a4 [cachefiles]
[1404624.785502] RSP: 0018:ffff8800bdd15cf0  EFLAGS: 00010286
[1404624.785529] RAX: ffff88041fae6c00 RBX: 00000000ffffff97 RCX: ffffffffa03da055
[1404624.785572] RDX: 0000000000000000 RSI: ffff88000ff91f40 RDI: 0000000000000286
[1404624.785616] RBP: ffff88041fa2a000 R08: 000000000025b935 R09: ffffffff8162f4b0
[1404624.785660] R10: 0000000000000020 R11: ffff88042fc04200 R12: ffff8804234e3cc0
[1404624.785703] R13: ffff88041fa2a120 R14: ffff880427c71d80 R15: ffff880427d60780
[1404624.785748] FS:  0000000000000000(0000) GS:ffff88000ff80000(0000) knlGS:0000000000000000
[1404624.785793] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[1404624.785821] CR2: 0000000009449e8c CR3: 0000000001636000 CR4: 00000000000006e0
[1404624.785864] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[1404624.785908] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[1404624.785952] Process kslowd002 (pid: 13379, threadinfo ffff8800bdd14000, task ffff88042ba9db00)
[1404624.785998] Stack:
[1404624.786018]  0000000000000000 ffff8800bdd15d68 ffff88042ba9db00 ffff8800c7ae4800
[1404624.786053] <0> ffff88042bbdf600 0000003500000000 0000000000000000 ffff88042bbdf710
[1404624.786105] <0> ffff8804234e3d40 0000000000000000 ffff8804234e3dc8 ffff88042bbdf718
[1404624.786172] Call Trace:
[1404624.786200]  [<ffffffff81040c9b>] ? finish_task_switch+0x3a/0xa7
[1404624.786229]  [<ffffffffa03d8ad5>] ? cachefiles_lookup_object+0x75/0xcf [cachefiles]
[1404624.786277]  [<ffffffffa035f30c>] ? fscache_lookup_object+0x11b/0x150 [fscache]
[1404624.786322]  [<ffffffffa035f8dc>] ? fscache_object_slow_work_execute+0x217/0x72d [fscache]
[1404624.786371]  [<ffffffff810a70e4>] ? slow_work_execute+0x181/0x28f
[1404624.786400]  [<ffffffff810a73a3>] ? slow_work_thread+0x1b1/0x2c6
[1404624.786430]  [<ffffffff8105efae>] ? autoremove_wake_function+0x0/0x2e
[1404624.786460]  [<ffffffff810a71f2>] ? slow_work_thread+0x0/0x2c6
[1404624.786488]  [<ffffffff8105eb75>] ? kthread+0x79/0x81
[1404624.786516]  [<ffffffff810098e4>] ? kernel_thread_helper+0x4/0x10
[1404624.786544]  [<ffffffff8105eafc>] ? kthread+0x0/0x81
[1404624.786571]  [<ffffffff810098e0>] ? kernel_thread_helper+0x0/0x10
[1404624.786598] Code: 00 05 77 2b 48 c7 c7 e6 cc 3d a0 31 c0 e8 d8 e8 f0 e0 48 c7 c7 99 d0 3d a0 31 c0 e8 ca e8 f0 e0 48 89 ee 4c 89 e7 e8 c7 f3 ff ff <0f> 0b eb fe 48 8d 85 10 01 00 00 f0 ff 85 10 01 00 00 48 8b 44
[1404624.786790] RIP  [<ffffffffa03da055>] cachefiles_walk_to_object+0x461/0x7a4 [cachefiles]
[1404624.786839]  RSP <ffff8800bdd15cf0>
[1404624.787256] ---[ end trace 8193f93bdbe44ea2 ]---


More information about the Linux-cachefs mailing list