[dm-devel] dm thin block allocation failure

[Markus Schade]

Hi

On Linux 5.1 (confirmed up to 5.1.2) I am seeing more or less regular
lockups of the dm layer.

I first discovered this issue after upgrading to Ubuntu's 18.04 kernel
(based on 4.15).  The original bug report is at
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1777398

As can be seen, I have tested a number of their kernels and mainline
kernels since then.

I have the following storage stack:

VM (mostly single partition and ext4, but could be anything)
LVM thin volume
MD RAID 10
GPT partition
Intel P4510 NVMe

Discard passdown is enable throughout the whole stack.

VM thin volumes get created by taking a snapshot of a snapshot from a
source thin volume (containing an OS image) and then deleting the
intermediate snapshot, making the new volumes independent.

Judging from the call trace, my guess is that there is somewhere a race
condition, when a new block needs to be allocated which has still to be
discarded.

Unfortunately I have not yet been able to find an (easy) reproducer.

Best regards,
Markus


------------[ cut here ]------------
kernel BUG at drivers/md/persistent-data/dm-space-map-disk.c:178!
invalid opcode: 0000 [#1] SMP PTI
CPU: 24 PID: 4180452 Comm: kworker/u128:1 Not tainted
5.1.2-050102-generic #201905141830
Hardware name: Quanta Cloud Technology Inc. QuantaGrid D52B-1U/S5B-MB
(LBG-4), BIOS 3A13.Q402 11/16/2018
Workqueue: dm-thin do_worker [dm_thin_pool]
RIP: 0010:sm_disk_new_block+0x92/0xa0 [dm_persistent_data]
Code: 55 e4 48 83 c0 01 48 89 83 20 22 00 00 49 8b 34 24 e8 82 f7 ff ff
85 c0 75 c4 83 7d e4 01 75 0a 48 83 83 28 22 00 00 01 eb b4 <0f> 0b e8
a7 31 f8 d6 0f 1f 80 00 00 00 00 0f 1f 44 00 00 55 48 89
RSP: 0018:ffffb42036cefca8 EFLAGS: 00010297
RAX: 0000000000000000 RBX: ffff98e6e7fa0000 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff98e6ff3bfe00
RBP: ffffb42036cefcc8 R08: 0000000000000000 R09: ffffb42036cefa38
R10: 000000003dc9bc00 R11: 0000000000000000 R12: ffffb42036cefd70
R13: ffffb42036cefd70 R14: ffff98e70934e000 R15: ffff9925a7c30fb0
FS:  0000000000000000(0000) GS:ffff9927bf600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000a095d80008 CR3: 0000009cf49be004 CR4: 00000000007626e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
 dm_pool_alloc_data_block+0x3c/0x60 [dm_thin_pool]
 alloc_data_block.isra.61+0x6c/0x1a0 [dm_thin_pool]
 process_cell+0x2cd/0x4f0 [dm_thin_pool]
 ? u32_swap+0x10/0x10
 ? sort+0x13b/0x220
 ? u32_swap+0x10/0x10
 do_worker+0x27c/0x900 [dm_thin_pool]
 ? __schedule+0x2db/0x840
 process_one_work+0x20f/0x410
 worker_thread+0x34/0x400
 kthread+0x120/0x140
 ? process_one_work+0x410/0x410
 ? __kthread_parkme+0x70/0x70
 ret_from_fork+0x35/0x40
Modules linked in: xt_CHECKSUM xt_nat iptable_nat dm_snapshot veth dummy
bridge ip6table_filter ip6_tables iptable_mangle xt_tcpudp xt_conntrack
iptable_filter xt_CT iptable_raw ip_tables x_tables bpfilter vhost_net
vhost tap intel_rapl dm_thin_pool skx_edac dm_persistent_data nfit
dm_bio_prison dm_bufio x86_pkg_temp_thermal intel_powerclamp coretemp
kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul
ghash_clmulni_intel aesni_intel aes_x86_64 crypto_simd cryptd
glue_helper intel_cstate intel_rapl_perf lpc_ich ioatdma dca ipmi_ssif
acpi_power_meter acpi_pad mac_hid ipmi_si ipmi_devintf ipmi_msghandler
nf_nat_ftp nf_conntrack_ftp nf_nat_sip nf_conntrack_sip nf_nat_pptp
nf_conntrack_pptp nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4
tcp_htcp 8021q garp mrp stp llc bonding autofs4 btrfs zstd_compress
raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor
raid6_pq libcrc32c raid0 multipath linear raid1 raid10 i2c_algo_bit ttm
drm_kms_helper syscopyarea
 sysfillrect sysimgblt fb_sys_fops i40e drm nvme ahci nvme_core libahci wmi
---[ end trace 82bb0f2c4f63c2ca ]---
RIP: 0010:sm_disk_new_block+0x92/0xa0 [dm_persistent_data]
Code: 55 e4 48 83 c0 01 48 89 83 20 22 00 00 49 8b 34 24 e8 82 f7 ff ff
85 c0 75 c4 83 7d e4 01 75 0a 48 83 83 28 22 00 00 01 eb b4 <0f> 0b e8
a7 31 f8 d6 0f 1f 80 00 00 00 00 0f 1f 44 00 00 55 48 89
RSP: 0018:ffffb42036cefca8 EFLAGS: 00010297
RAX: 0000000000000000 RBX: ffff98e6e7fa0000 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff98e6ff3bfe00
RBP: ffffb42036cefcc8 R08: 0000000000000000 R09: ffffb42036cefa38
R10: 000000003dc9bc00 R11: 0000000000000000 R12: ffffb42036cefd70
R13: ffffb42036cefd70 R14: ffff98e70934e000 R15: ffff9925a7c30fb0
FS:  0000000000000000(0000) GS:ffff9927bf600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000a095d80008 CR3: 0000009cf49be004 CR4: 00000000007626e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554