<div dir="ltr"><div style="font-size:13px">Hi,</div><div style="font-size:13px"><br></div><div style="font-size:13px">One more issue from me. As I said in my previous email, we are configuring lvm with SSD caching and EBS volumes on some of our boxes in AWS. The OS for those nodes is Ubuntu 15.10 (4.2.0-16-generic).</div><div style="font-size:13px"><br></div><span style="font-size:13px"><font face="arial, helvetica, sans-serif">We already had 2 nodes down and seems to be related to the lvm caching part. On one of the nodes we found this in the logs:</font></span><br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.548785] ------------[ cut here ]------------</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552225] kernel BUG at /build/linux-AxjFAn/linux-4.2.0/drivers/md/dm-cache-policy-mq.c:1079!</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] invalid opcode: 0000 [#1] SMP </font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] Modules linked in: isofs binfmt_misc xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp bridge stp llc iptable_filter ip_tables x_tables dm_cache_mq dm_cache dm_persistent_data dm_bio_prison dm_bufio libcrc32c ppdev xen_fbfront syscopyarea sysfillrect sysimgblt fb_sys_fops serio_raw parport_pc parport autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel raid0 aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd psmouse floppy</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] CPU: 1 PID: 68058 Comm: java Not tainted 4.2.0-16-generic #19-Ubuntu</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] Hardware name: Xen HVM domU, BIOS 4.2.amazon 05/06/2015</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] task: ffff880190241b80 ti: ffff8806f3cf4000 task.ti: ffff8806f3cf4000</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] RIP: 0010:[<ffffffffc0182257>] [<ffffffffc0182257>] __mq_set_clear_dirty+0x47/0x80 [dm_cache_mq]</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] RSP: 0018:ffff8806f3cf7730 EFLAGS: 00010246</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] RAX: 0000000000000000 RBX: ffff88076a236080 RCX: ffffc90020f6aff8</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] RDX: 0000000000f7b83e RSI: ffffc9001fd39000 RDI: 0000000000000016</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] RBP: ffff8806f3cf7748 R08: 0000000000000000 R09: ffff8801adb6c7c8</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] R10: ffff88032fd31bb0 R11: ffff88076a22c858 R12: ffff88076a236000</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] R13: 0000000000000001 R14: 000000000045c6ae R15: 0000000000000000</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] FS: 00007fccc4b27700(0000) GS:ffff88076f640000(0000) knlGS:0000000000000000</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] CR2: 00007fce83a55000 CR3: 00000005b3d2b000 CR4: 00000000001406e0</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] Stack:</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] ffff88076a236080 ffff88076a236000 0000000000f7b83e ffff8806f3cf7778</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] ffffffffc0182317 0000000000000000 000000000045c6ae ffff880476c014e0</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] ffff88076744f800 ffff8806f3cf7788 ffffffffc01a9862 ffff8806f3cf7818</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] Call Trace:</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffffc0182317>] mq_set_dirty+0x37/0x50 [dm_cache_mq]</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffffc01a9862>] set_dirty+0x32/0x40 [dm_cache]</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffffc01ab3c9>] remap_cell_to_cache_dirty+0x1d9/0x240 [dm_cache]</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffffc01ab900>] cache_map+0x330/0x4d0 [dm_cache]</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffffc01a8eb0>] ? cache_resume+0x30/0x30 [dm_cache]</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff8166b2ee>] __map_bio+0x3e/0x100</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff8166d235>] __split_and_process_bio+0x285/0x3f0</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff8166d40d>] dm_make_request+0x6d/0xc0</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff813952a6>] generic_make_request+0xd6/0x110</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff810c3d61>] ? __raw_callee_save___pv_queued_spin_unlock+0x11/0x20</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff81395356>] submit_bio+0x76/0x170</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff8138f51b>] ? __bio_add_page.part.16+0x10b/0x270</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff8128c311>] ext4_io_submit+0x31/0x50</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff8128c4c8>] ext4_bio_write_page+0x168/0x410</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff81283351>] mpage_submit_page+0x61/0x80</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff812835d6>] mpage_map_and_submit_buffers+0x156/0x290</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff81288874>] ext4_writepages+0x624/0xce0</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff811903be>] do_writepages+0x1e/0x30</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff8118335c>] __filemap_fdatawrite_range+0xcc/0x100</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff8118349a>] filemap_write_and_wait_range+0x2a/0x70</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff8127f831>] ext4_sync_file+0xe1/0x2f0</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff8122fc9b>] vfs_fsync_range+0x4b/0xb0</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff8122fd5d>] do_fsync+0x3d/0x70</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff81230023>] SyS_fdatasync+0x13/0x20</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff817ef9f2>] entry_SYSCALL_64_fastpath+0x16/0x75</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] Code: 89 f2 49 8b b4 24 80 0d 00 00 e8 c5 f5 ff ff 48 85 c0 74 17 49 3b 84 24 f8 00 00 00 48 89 c3 72 0a 49 3b 84 24 00 01 00 00 72 02 <0f> 0b 48 89 c6 4c 89 e7 41 83 e5 01 e8 08 ef ff ff 0f b6 43 28 </font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] RIP [<ffffffffc0182257>] __mq_set_clear_dirty+0x47/0x80 [dm_cache_mq]</font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.552561] RSP <ffff8806f3cf7730></font></div></div><div><div><font face="monospace, monospace">Nov 17 17:03:26 localhost kernel: [1650439.740854] ---[ end trace 98483c1d54cc426e ]---</font></div></div></blockquote><div><div style="font-size:13px"><br></div></div><div style="font-size:13px">Is this something that has been seen before?</div><div style="font-size:13px">Would switching to RHEL/CentOS 7 make any difference?</div><div style="font-size:13px"><br></div><div style="font-size:13px">Thanks,</div><div style="font-size:13px">Ciprian Hacman</div><div><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr">--</div><div dir="ltr">Performance Monitoring * Log Analytics * Search Analytics<br>Solr & Elasticsearch Support * <a href="http://sematext.com/" rel="noreferrer" style="color:rgb(17,85,204)" target="_blank">http://sematext.com/</a><br></div></div></div></div></div> </div>