[fedora-virt] some vm die in F12
Gianluca Cecchi
gianluca.cecchi at gmail.com
Mon Nov 30 14:00:39 UTC 2009
Hello,
I had an F11 host with virt-preview repo configured and I have used it
a lot without particular problems.
Now I'm on F12 on the same host after upgrade and I'm experiencing
guests' death caused by oom killer.
Example.
- all VM (4) are qemu/kvm and x86_64
- boot of host on 26/11 about 19:20.
- two centos 5.3 guests are configured to startup automatically and
they indeed start
In messages I see this on their startup:
Nov 26 19:23:01 virtfed kernel: kvm: 3458: cpu0 unimplemented perfctr
wrmsr: 0x186 data 0x130079
Nov 26 19:23:01 virtfed kernel: kvm: 3458: cpu0 unimplemented perfctr
wrmsr: 0xc1 data 0xffcfadc8
Nov 26 19:23:01 virtfed kernel: kvm: 3458: cpu0 unimplemented perfctr
wrmsr: 0x186 data 0x530079
Nov 26 19:23:01 virtfed kernel: kvm: 3458: cpu1 unimplemented perfctr
wrmsr: 0x186 data 0x130079
Nov 26 19:23:01 virtfed kernel: kvm: 3458: cpu1 unimplemented perfctr
wrmsr: 0xc1 data 0xffcfadc8
Nov 26 19:23:01 virtfed kernel: kvm: 3458: cpu1 unimplemented perfctr
wrmsr: 0x186 data 0x530079
Nov 26 19:23:01 virtfed kernel: kvm: 3501: cpu0 unimplemented perfctr
wrmsr: 0x186 data 0x130079
Nov 26 19:23:01 virtfed kernel: kvm: 3501: cpu0 unimplemented perfctr
wrmsr: 0xc1 data 0xffcfadc8
Nov 26 19:23:01 virtfed kernel: kvm: 3501: cpu0 unimplemented perfctr
wrmsr: 0x186 data 0x530079
Nov 26 19:23:01 virtfed kernel: kvm: 3501: cpu1 unimplemented perfctr
wrmsr: 0x186 data 0x130079
Nov 26 19:23:36 virtfed kernel: kvm: emulating exchange as write
- On 27/11 at 12:03 I start other two guests in centos 5.4
Again I get rows similar to the above ones plus:
Nov 27 12:03:19 virtfed kernel: __ratelimit: 2 callbacks suppressed
The day after at 06:55
Nov 28 03:27:06 virtfed logrotate: ALERT exited abnormally with [1]
Nov 28 06:55:01 virtfed kernel: qemu-kvm invoked oom-killer:
gfp_mask=0x201da, order=0, oomkilladj=0
Nov 28 06:55:01 virtfed kernel: qemu-kvm cpuset=/ mems_allowed=0
Nov 28 06:55:01 virtfed kernel: Pid: 405, comm: qemu-kvm Tainted: G
W 2.6.31.5-127.fc12.x86_64 #1
Nov 28 06:55:01 virtfed kernel: Call Trace:
Nov 28 06:55:01 virtfed kernel: [<ffffffff8108a7d7>] ?
cpuset_print_task_mems_allowed+0x91/0x9d
Nov 28 06:55:01 virtfed kernel: [<ffffffff810c38e9>] oom_kill_process+0x98/0x256
Nov 28 06:55:01 virtfed kernel: [<ffffffff810c3d6a>] ?
select_bad_process+0xa3/0x102
Nov 28 06:55:01 virtfed kernel: [<ffffffff810c3e53>] __out_of_memory+0x8a/0x99
Nov 28 06:55:01 virtfed kernel: [<ffffffff810c3fc5>] out_of_memory+0x163/0x195
Nov 28 06:55:01 virtfed kernel: [<ffffffff810c75c1>]
__alloc_pages_nodemask+0x491/0x584
Nov 28 06:55:01 virtfed kernel: [<ffffffff810eb1c4>]
alloc_pages_current+0x95/0x9e
Nov 28 06:55:01 virtfed kernel: [<ffffffff810c15c1>]
__page_cache_alloc+0x5f/0x61
Nov 28 06:55:01 virtfed kernel: [<ffffffff810c9235>]
__do_page_cache_readahead+0x98/0x176
Nov 28 06:55:01 virtfed kernel: [<ffffffff810c9334>] ra_submit+0x21/0x25
Nov 28 06:55:01 virtfed kernel: [<ffffffff810c1abd>] filemap_fault+0x193/0x317
Nov 28 06:55:01 virtfed kernel: [<ffffffff810d6641>] __do_fault+0x54/0x3c4
Nov 28 06:55:01 virtfed kernel: [<ffffffff81017bc1>] ? read_tsc+0x9/0x1b
Nov 28 06:55:01 virtfed kernel: [<ffffffff810d881a>] handle_mm_fault+0x2f6/0x705
Nov 28 06:55:01 virtfed kernel: [<ffffffff8106a9ff>] ?
__hrtimer_start_range_ns+0x2bb/0x2cd
Nov 28 06:55:01 virtfed kernel: [<ffffffff8141cfe1>] do_page_fault+0x281/0x299
Nov 28 06:55:01 virtfed kernel: [<ffffffff8141af75>] page_fault+0x25/0x30
Nov 28 06:55:01 virtfed kernel: Mem-Info:
Nov 28 06:55:01 virtfed kernel: Node 0 DMA per-cpu:
Nov 28 06:55:01 virtfed kernel: CPU 0: hi: 0, btch: 1 usd: 0
Nov 28 06:55:01 virtfed kernel: CPU 1: hi: 0, btch: 1 usd: 0
Nov 28 06:55:01 virtfed kernel: CPU 2: hi: 0, btch: 1 usd: 0
Nov 28 06:55:01 virtfed kernel: CPU 3: hi: 0, btch: 1 usd: 0
Nov 28 06:55:01 virtfed kernel: CPU 4: hi: 0, btch: 1 usd: 0
Nov 28 06:55:01 virtfed kernel: CPU 5: hi: 0, btch: 1 usd: 0
Nov 28 06:55:01 virtfed kernel: CPU 6: hi: 0, btch: 1 usd: 0
Nov 28 06:55:01 virtfed kernel: CPU 7: hi: 0, btch: 1 usd: 0
Nov 28 06:55:01 virtfed kernel: Node 0 DMA32 per-cpu:
Nov 28 06:55:01 virtfed kernel: CPU 0: hi: 186, btch: 31 usd: 86
Nov 28 06:55:01 virtfed kernel: CPU 1: hi: 186, btch: 31 usd: 136
Nov 28 06:55:01 virtfed kernel: CPU 2: hi: 186, btch: 31 usd: 51
Nov 28 06:55:01 virtfed kernel: CPU 3: hi: 186, btch: 31 usd: 126
Nov 28 06:55:01 virtfed kernel: CPU 4: hi: 186, btch: 31 usd: 86
Nov 28 06:55:01 virtfed kernel: CPU 5: hi: 186, btch: 31 usd: 34
Nov 28 06:55:01 virtfed kernel: CPU 6: hi: 186, btch: 31 usd: 171
Nov 28 06:55:01 virtfed kernel: CPU 7: hi: 186, btch: 31 usd: 169
Nov 28 06:55:01 virtfed kernel: Node 0 Normal per-cpu:
Nov 28 06:55:01 virtfed kernel: CPU 0: hi: 186, btch: 31 usd: 173
Nov 28 06:55:01 virtfed kernel: CPU 1: hi: 186, btch: 31 usd: 42
Nov 28 06:55:01 virtfed kernel: CPU 2: hi: 186, btch: 31 usd: 160
Nov 28 06:55:01 virtfed kernel: CPU 3: hi: 186, btch: 31 usd: 109
Nov 28 06:55:01 virtfed kernel: CPU 4: hi: 186, btch: 31 usd: 112
Nov 28 06:55:01 virtfed kernel: CPU 5: hi: 186, btch: 31 usd: 112
Nov 28 06:55:01 virtfed kernel: CPU 6: hi: 186, btch: 31 usd: 174
Nov 28 06:55:01 virtfed kernel: CPU 7: hi: 186, btch: 31 usd: 118
Nov 28 06:55:01 virtfed kernel: Active_anon:2537459 active_file:3
inactive_anon:364350
Nov 28 06:55:01 virtfed kernel: inactive_file:160 unevictable:12562
dirty:0 writeback:0 unstable:0
Nov 28 06:55:01 virtfed kernel: free:16256 slab:29833 mapped:7872
pagetables:10697 bounce:0
Nov 28 06:55:01 virtfed kernel: Node 0 DMA free:15824kB min:16kB
low:20kB high:24kB active_anon:0kB inactive_anon:0kB active_file:0kB
inactive_file:0kB unevictable:0kB present:15320kB pages_scanned:0
all_unreclaimable? yes
Nov 28 06:55:01 virtfed kernel: lowmem_reserve[]: 0 3254 12092 12092
Nov 28 06:55:01 virtfed kernel: Node 0 DMA32 free:39060kB min:3784kB
low:4728kB high:5676kB active_anon:2510452kB inactive_anon:502304kB
active_file:12kB inactive_file:244kB unevictable:520kB
present:3332660kB pages_scanned:120 all_unreclaimable? no
Nov 28 06:55:01 virtfed kernel: lowmem_reserve[]: 0 0 8837 8837
Nov 28 06:55:01 virtfed kernel: Node 0 Normal free:10140kB min:10280kB
low:12848kB high:15420kB active_anon:7639384kB inactive_anon:955096kB
active_file:0kB inactive_file:396kB unevictable:49728kB
present:9049596kB pages_scanned:0 all_unreclaimable? no
Nov 28 06:55:01 virtfed kernel: lowmem_reserve[]: 0 0 0 0
Nov 28 06:55:01 virtfed kernel: Node 0 DMA: 2*4kB 1*8kB 2*16kB 1*32kB
2*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15824kB
Nov 28 06:55:01 virtfed kernel: Node 0 DMA32: 3793*4kB 2168*8kB
68*16kB 11*32kB 4*64kB 6*128kB 3*256kB 2*512kB 2*1024kB 0*2048kB
0*4096kB = 38820kB
Nov 28 06:55:01 virtfed kernel: Node 0 Normal: 1948*4kB 5*8kB 0*16kB
0*32kB 2*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB =
10264kB
Nov 28 06:55:01 virtfed kernel: 26703 total pagecache pages
Nov 28 06:55:01 virtfed kernel: 18747 pages in swap cache
Nov 28 06:55:01 virtfed kernel: Swap cache stats: add 1236041, delete
1217294, find 52896/69574
Nov 28 06:55:01 virtfed kernel: Free swap = 0kB
Nov 28 06:55:01 virtfed kernel: Total swap = 4194296kB
Nov 28 06:55:01 virtfed kernel: 3145727 pages RAM
Nov 28 06:55:01 virtfed kernel: 63020 pages reserved
Nov 28 06:55:01 virtfed kernel: 76044 pages shared
Nov 28 06:55:01 virtfed kernel: 3053641 pages non-shared
Nov 28 06:55:01 virtfed kernel: virbr0: port 3(vnet4) entering disabled state
Nov 28 06:55:01 virtfed kernel: device vnet4 left promiscuous mode
Nov 28 06:55:01 virtfed kernel: virbr0: port 3(vnet4) entering disabled state
About 1 hour and a half after, at 08:14 I get the same for the other 5.4 vm.
The same happens for other processes
Any hints on this and on how to debug? Any other one?
I don't know if it can be related, but I also tried to reboot
disabling ksm and ksmtuned:
[root at virtfed ~]# chkconfig --list ksmtuned
ksmtuned 0:off 1:off 2:off 3:off 4:off 5:off 6:off
[root at virtfed ~]# chkconfig --list ksm
ksm 0:off 1:off 2:off 3:off 4:off 5:off 6:off
getting the same problem...
strangely, I get anyway this:
[root at virtfed ~]# service ksm status
ksm is running
Why ksm is running? How can I configure a VM to be or not to be
managed with ksm?
Or is it an overall feature to enable/disable?
More information about the Fedora-virt
mailing list