[dm-devel] bcache gets stuck flushing writeback cache when used in combination with LUKS/dm-crypt and non-default bucket size
Eric Wheeler
bcache at lists.ewheeler.net
Wed May 11 01:38:38 UTC 2016
On Sun, 8 May 2016, James Johnston wrote:
> Hi,
>
> [1.] One line summary of the problem:
>
> bcache gets stuck flushing writeback cache when used in combination with
> LUKS/dm-crypt and non-default bucket size
>
> [2.] Full description of the problem/report:
>
> I've run into a problem where the bcache writeback cache can't be flushed to
> disk when the backing device is a LUKS / dm-crypt device and the cache set has
> a non-default bucket size.
You might try LUKS atop of bcache instead of under it. This might be
better for privacy too, otherwise your cached data is unencrypted.
> # Make cache set on second drive
> # IMPORTANT: Problem does not occur if I omit --bucket parameter.
> make-bcache --bucket 2M -C /dev/sdb
2MB is quite large, maybe it exceeds the 256-bvec limit. I'm not sure if
Ming Lei's patch got in to 4.6 yet, but try this:
https://lkml.org/lkml/2016/4/5/1046
and maybe Shaohua Li's patch too:
http://www.spinics.net/lists/raid/msg51830.html
--
Eric Wheeler
> # Set up LUKS/dm-crypt on second drive.
> # IMPORTANT: Problem does not occur if I omit the dm-crypt layer.
> cryptsetup luksFormat /dev/sdc
> cryptsetup open --type luks /dev/sdc backCrypt
> # Make bcache backing device & enable writeback
> make-bcache -B /dev/mapper/backCrypt
> bcache-super-show /dev/sdb | grep cset.uuid | \
> cut -f 3 > /sys/block/bcache0/bcache/attach
> echo writeback > /sys/block/bcache0/bcache/cache_mode
>
> 4. Finally, this is the kill sequence to bring the system to its knees:
>
> sudo -s
> cd /sys/block/bcache0/bcache
> echo 0 > sequential_cutoff
> # Verify that the cache is attached (i.e. does not say "no cache"). It should
> # say that it's clean since we haven't written anything yet.
> cat state
> # Copy some random data.
> dd if=/dev/urandom of=/dev/bcache0 bs=1M count=250
> # Show current state. On my system approximately 20 to 25 MB remain in
> # writeback cache.
> cat dirty_data
> cat state
> # Detach the cache set. This will start the cache set disk thrashing.
> echo 1 > detach
> # After a few moments, confirm that the cache set is not going anywhere. On
> # my system, only a few MB have been flushed as evidenced by a small decrease
> # in dirty_data. State remains dirty.
> cat dirty_data
> cat state
> # At this point, the hypervisor system reports hundreds of MB/second of reads
> # to the underlying physical SSD coming from the virtual machine; the hard drive
> # light is stuck on... hypervisor status bar shows the activity is on cache
> # set. No writes seem to be occurring on any disk.
>
> [8.] Environment
> [8.1.] Software (add the output of the ver_linux script here)
> Linux bcachetest2 4.6.0-040600rc6-generic #201605012031 SMP Mon May 2 00:33:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
>
> Util-linux 2.27.1
> Mount 2.27.1
> Module-init-tools 22
> E2fsprogs 1.42.13
> Xfsprogs 4.3.0
> Linux C Library 2.23
> Dynamic linker (ldd) 2.23
> Linux C++ Library 6.0.21
> Procps 3.3.10
> Net-tools 1.60
> Kbd 1.15.5
> Console-tools 1.15.5
> Sh-utils 8.25
> Udev 229
> Modules Loaded 8250_fintek ablk_helper aesni_intel aes_x86_64 ahci async_memcpy async_pq async_raid6_recov async_tx async_xor autofs4 btrfs configfs coretemp crc32_pclmul crct10dif_pclmul cryptd drm drm_kms_helper e1000 fb_sys_fops fjes gf128mul ghash_clmulni_intel glue_helper hid hid_generic i2c_piix4 ib_addr ib_cm ib_core ib_iser ib_mad ib_sa input_leds iscsi_tcp iw_cm joydev libahci libcrc32c libiscsi libiscsi_tcp linear lrw mac_hid mptbase mptscsih mptspi multipath nfit parport parport_pc pata_acpi ppdev psmouse raid0 raid10 raid1 raid456 raid6_pq rdma_cm scsi_transport_iscsi scsi_transport_spi serio_raw shpchp syscopyarea sysfillrect sysimgblt ttm usbhid vmw_balloon vmwgfx vmw_vmci vmw_vsock_vmci_transport vsock xor
>
> [8.2.] Processor information (from /proc/cpuinfo):
> processor : 0
> vendor_id : GenuineIntel
> cpu family : 6
> model : 42
> model name : Intel(R) Core(TM) i5-2520M CPU @ 2.50GHz
> stepping : 7
> microcode : 0x29
> cpu MHz : 2491.980
> cache size : 3072 KB
> physical id : 0
> siblings : 1
> core id : 0
> cpu cores : 1
> apicid : 0
> initial apicid : 0
> fpu : yes
> fpu_exception : yes
> cpuid level : 13
> wp : yes
> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx hypervisor lahf_lm epb tsc_adjust dtherm ida arat pln pts
> bugs :
> bogomips : 4983.96
> clflush size : 64
> cache_alignment : 64
> address sizes : 42 bits physical, 48 bits virtual
> power management:
>
> [8.3.] Module information (from /proc/modules):
> ppdev 20480 0 - Live 0x0000000000000000
> vmw_balloon 20480 0 - Live 0x0000000000000000
> vmw_vsock_vmci_transport 28672 1 - Live 0x0000000000000000
> vsock 36864 2 vmw_vsock_vmci_transport, Live 0x0000000000000000
> coretemp 16384 0 - Live 0x0000000000000000
> joydev 20480 0 - Live 0x0000000000000000
> input_leds 16384 0 - Live 0x0000000000000000
> serio_raw 16384 0 - Live 0x0000000000000000
> shpchp 36864 0 - Live 0x0000000000000000
> vmw_vmci 65536 2 vmw_balloon,vmw_vsock_vmci_transport, Live 0x0000000000000000
> i2c_piix4 24576 0 - Live 0x0000000000000000
> nfit 40960 0 - Live 0x0000000000000000
> 8250_fintek 16384 0 - Live 0x0000000000000000
> parport_pc 32768 0 - Live 0x0000000000000000
> parport 49152 2 ppdev,parport_pc, Live 0x0000000000000000
> mac_hid 16384 0 - Live 0x0000000000000000
> ib_iser 49152 0 - Live 0x0000000000000000
> rdma_cm 53248 1 ib_iser, Live 0x0000000000000000
> iw_cm 49152 1 rdma_cm, Live 0x0000000000000000
> ib_cm 45056 1 rdma_cm, Live 0x0000000000000000
> ib_sa 36864 2 rdma_cm,ib_cm, Live 0x0000000000000000
> ib_mad 49152 2 ib_cm,ib_sa, Live 0x0000000000000000
> ib_core 122880 6 ib_iser,rdma_cm,iw_cm,ib_cm,ib_sa,ib_mad, Live 0x0000000000000000
> ib_addr 20480 3 rdma_cm,ib_sa,ib_core, Live 0x0000000000000000
> configfs 40960 2 rdma_cm, Live 0x0000000000000000
> iscsi_tcp 20480 0 - Live 0x0000000000000000
> libiscsi_tcp 24576 1 iscsi_tcp, Live 0x0000000000000000
> libiscsi 53248 3 ib_iser,iscsi_tcp,libiscsi_tcp, Live 0x0000000000000000
> scsi_transport_iscsi 98304 4 ib_iser,iscsi_tcp,libiscsi, Live 0x0000000000000000
> autofs4 40960 2 - Live 0x0000000000000000
> btrfs 1024000 0 - Live 0x0000000000000000
> raid10 49152 0 - Live 0x0000000000000000
> raid456 110592 0 - Live 0x0000000000000000
> async_raid6_recov 20480 1 raid456, Live 0x0000000000000000
> async_memcpy 16384 2 raid456,async_raid6_recov, Live 0x0000000000000000
> async_pq 16384 2 raid456,async_raid6_recov, Live 0x0000000000000000
> async_xor 16384 3 raid456,async_raid6_recov,async_pq, Live 0x0000000000000000
> async_tx 16384 5 raid456,async_raid6_recov,async_memcpy,async_pq,async_xor, Live 0x0000000000000000
> xor 24576 2 btrfs,async_xor, Live 0x0000000000000000
> raid6_pq 102400 4 btrfs,raid456,async_raid6_recov,async_pq, Live 0x0000000000000000
> libcrc32c 16384 1 raid456, Live 0x0000000000000000
> raid1 36864 0 - Live 0x0000000000000000
> raid0 20480 0 - Live 0x0000000000000000
> multipath 16384 0 - Live 0x0000000000000000
> linear 16384 0 - Live 0x0000000000000000
> hid_generic 16384 0 - Live 0x0000000000000000
> usbhid 49152 0 - Live 0x0000000000000000
> hid 122880 2 hid_generic,usbhid, Live 0x0000000000000000
> crct10dif_pclmul 16384 0 - Live 0x0000000000000000
> crc32_pclmul 16384 0 - Live 0x0000000000000000
> ghash_clmulni_intel 16384 0 - Live 0x0000000000000000
> aesni_intel 167936 0 - Live 0x0000000000000000
> aes_x86_64 20480 1 aesni_intel, Live 0x0000000000000000
> lrw 16384 1 aesni_intel, Live 0x0000000000000000
> gf128mul 16384 1 lrw, Live 0x0000000000000000
> glue_helper 16384 1 aesni_intel, Live 0x0000000000000000
> ablk_helper 16384 1 aesni_intel, Live 0x0000000000000000
> cryptd 20480 3 ghash_clmulni_intel,aesni_intel,ablk_helper, Live 0x0000000000000000
> vmwgfx 237568 1 - Live 0x0000000000000000
> ttm 98304 1 vmwgfx, Live 0x0000000000000000
> drm_kms_helper 147456 1 vmwgfx, Live 0x0000000000000000
> syscopyarea 16384 1 drm_kms_helper, Live 0x0000000000000000
> psmouse 131072 0 - Live 0x0000000000000000
> sysfillrect 16384 1 drm_kms_helper, Live 0x0000000000000000
> sysimgblt 16384 1 drm_kms_helper, Live 0x0000000000000000
> fb_sys_fops 16384 1 drm_kms_helper, Live 0x0000000000000000
> drm 364544 4 vmwgfx,ttm,drm_kms_helper, Live 0x0000000000000000
> ahci 36864 2 - Live 0x0000000000000000
> libahci 32768 1 ahci, Live 0x0000000000000000
> e1000 135168 0 - Live 0x0000000000000000
> mptspi 24576 0 - Live 0x0000000000000000
> mptscsih 40960 1 mptspi, Live 0x0000000000000000
> mptbase 102400 2 mptspi,mptscsih, Live 0x0000000000000000
> scsi_transport_spi 32768 1 mptspi, Live 0x0000000000000000
> pata_acpi 16384 0 - Live 0x0000000000000000
> fjes 28672 0 - Live 0x0000000000000000
>
> [8.6.] SCSI information (from /proc/scsi/scsi)
> Attached devices:
> Host: scsi3 Channel: 00 Id: 00 Lun: 00
> Vendor: ATA Model: VMware Virtual S Rev: 0001
> Type: Direct-Access ANSI SCSI revision: 05
> Host: scsi4 Channel: 00 Id: 00 Lun: 00
> Vendor: NECVMWar Model: VMware SATA CD01 Rev: 1.00
> Type: CD-ROM ANSI SCSI revision: 05
> Host: scsi5 Channel: 00 Id: 00 Lun: 00
> Vendor: ATA Model: VMware Virtual S Rev: 0001
> Type: Direct-Access ANSI SCSI revision: 05
> Host: scsi6 Channel: 00 Id: 00 Lun: 00
> Vendor: ATA Model: VMware Virtual S Rev: 0001
> Type: Direct-Access ANSI SCSI revision: 05
>
> Best regards,
>
> James Johnston
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo at vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
More information about the dm-devel
mailing list