[linux-lvm] [dm-crypt] LuksOpen hangs ?
Ondrej Kozina
okozina at redhat.com
Mon Jun 29 14:34:17 UTC 2020
On 6/28/20 8:48 AM, B&A Consultants wrote:
> Hello,
>
> We are using LVM to hold several luks-encrypted LVs. A few days ago,
> while two of those LV were in use (at least opened for one, not sure
> there was any disk activity ; the second one was only read from at that
> time), we've had a power outage (despite UPS). Duh.
>
> On restart the next day, one of the LV fails to open. LuksOpen hangs,
> with one of the LVM disks churning (at last that's what the acces LED
> seems to tell). From then on, any LVM operation hangs, but the machine
> is properly responsive.
Could you please send debug output also for particular hanging lvm2
command? Just add -vvvv to the command. Syslog output would also help.
I can't tell you more than the cryptsetup command is stuck in I/O.
Perhaps hosting LV is suspended due to some error but I can't tell for sure.
>
> We tried a few things, the first being to check/repair the header :
>
> cryptsetup --verbose --debug repair /dev/v03/E192
> # cryptsetup 2.3.2 processing "cryptsetup --verbose --debug repair
> /dev/v03/E192"
> # Running command repair.
> # Locking memory.
> # Installing SIGINT/SIGTERM handler.
> # Unblocking interruption on signal.
> # Allocating context for crypt device /dev/v03/E192.
> # Trying to open and read device /dev/v03/E192 with direct-io.
> # Initialising device-mapper backend library.
> # Trying to load any crypt type from device /dev/v03/E192.
> # Crypto backend (OpenSSL 1.1.1g 21 Apr 2020) initialized in cryptsetup
> library version 2.3.2.
> # Detected kernel Linux 5.4.38-gentoo-LTS x86_64.
> # PBKDF pbkdf2-sha256, time_ms 2000 (iterations 0).
> # Reading LUKS header of size 1024 from device /dev/v03/E192
> # Key length 32, device size 2147483648 sectors, header size 2050 sectors.
> No known problems detected for LUKS header.
> # Releasing crypt device /dev/v03/E192 context.
> # Releasing device-mapper backend.
> # Closing read only fd for /dev/v03/E192.
> # Unlocking memory.
> Command successful.
>
> Dumping the header seems ok :
>
> cryptsetup luksDump /dev/v03/E192 --verbose --debug
> # cryptsetup 2.3.2 processing "cryptsetup luksDump /dev/v03/E192
> --verbose --debug"
> # Running command luksDump.
> # Locking memory.
> # Installing SIGINT/SIGTERM handler.
> # Unblocking interruption on signal.
> # Allocating context for crypt device /dev/v03/E192.
> # Trying to open and read device /dev/v03/E192 with direct-io.
> # Initialising device-mapper backend library.
> # Trying to load any crypt type from device /dev/v03/E192.
> # Crypto backend (OpenSSL 1.1.1g 21 Apr 2020) initialized in cryptsetup
> library version 2.3.2.
> # Detected kernel Linux 5.4.38-gentoo-LTS x86_64.
> # PBKDF pbkdf2-sha256, time_ms 2000 (iterations 0).
> # Reading LUKS header of size 1024 from device /dev/v03/E192
> # Key length 32, device size 2147483648 sectors, header size 2050 sectors.
> LUKS header information for /dev/v03/E192
>
> Version: 1
> Cipher name: aes
> Cipher mode: xts-plain64
> Hash spec: sha256
> Payload offset: 4096
> MK bits: 256
> MK digest: 52 [..] 52
> MK salt: 50 [..] 16
> a4 [..] df
> MK iterations: 365500
> UUID: 9bcee5de-aa79-42f1-bcd2-9ecfc0acc30f
>
> Key Slot 0: ENABLED
> Iterations: 2921539
> Salt: 24 [..] 79
> f1 [..] 90
> Key material offset: 8
> AF stripes: 4000
> Key Slot 1: DISABLED
> Key Slot 2: DISABLED
> Key Slot 3: DISABLED
> Key Slot 4: DISABLED
> Key Slot 5: DISABLED
> Key Slot 6: DISABLED
> Key Slot 7: DISABLED
> # Releasing crypt device /dev/v03/E192 context.
> # Releasing device-mapper backend.
> # Closing read only fd for /dev/v03/E192.
> # Unlocking memory.
> Command successful.
>
> Opening the other luks-encrypted volume is ok :
>
> # cryptsetup luksOpen /dev/v03/P50-Abc abc --verbose --debug
> # cryptsetup 2.3.2 processing "cryptsetup luksOpen /dev/v03/P50-Abc abc
> --verbose --debug"
> # Running command open.
> # Locking memory.
> # Installing SIGINT/SIGTERM handler.
> # Unblocking interruption on signal.
> # Allocating context for crypt device /dev/v03/P50-Abc.
> # Trying to open and read device /dev/v03/P50-Abc with direct-io.
> # Initialising device-mapper backend library.
>
> # Trying to load any crypt type from device /dev/v03/P50-Abc.
> # Crypto backend (OpenSSL 1.1.1g 21 Apr 2020) initialized in cryptsetup
> library version 2.3.2.
> # Detected kernel Linux 5.4.38-gentoo-LTS x86_64.
> # Loading LUKS2 header (repair disabled).
> # Acquiring read lock for device /dev/v03/P50-Abc.
> # Opening lock resource file /run/cryptsetup/L_253:4
> # Verifying lock handle for /dev/v03/P50-Abc.
> # Device /dev/v03/P50-Abc READ lock taken.
> # Trying to read primary LUKS2 header at offset 0x0.
> # Opening locked device /dev/v03/P50-Abc
> # Veryfing locked device handle (bdev)
>
> # LUKS2 header version 2 of size 16384 bytes, checksum sha256.
> #
> Checksum:19a91de16d41e5f80d15db4dbaefcb895b9f70b908b126550f3b931b3159b739 (on-disk)
> #
> Checksum:19a91de16d41e5f80d15db4dbaefcb895b9f70b908b126550f3b931b3159b739 (in-memory)
> # Trying to read secondary LUKS2 header at offset 0x4000.
> # Reusing open ro fd on device /dev/v03/P50-Abc
> # LUKS2 header version 2 of size 16384 bytes, checksum sha256.
> #
> Checksum:b0be8a6ecf4c15822d52d7d0f3a24f5ae7d4012f244f47d5e4c9d7172b3fea26 (on-disk)
> #
> Checksum:b0be8a6ecf4c15822d52d7d0f3a24f5ae7d4012f244f47d5e4c9d7172b3fea26 (in-memory)
> # Device size 644245094400, offset 16777216.
> # Device /dev/v03/P50-Abc READ lock released.
> # PBKDF argon2i, time_ms 2000 (iterations 0), max_memory_kb 1048576,
> parallel_threads 4.
> # Activating volume abc using token -1.
> # Interactive passphrase entry requested.
> Enter passphrase for /dev/v03/P50-Abc:
> # Activating volume abc [keyslot -1] using passphrase.
> # dm version [ opencount flush ] [16384] (*1)
> # dm versions [ opencount flush ] [16384] (*1)
> # Detected dm-ioctl version 4.41.0.
> # Detected dm-crypt version 1.19.0.
> # Device-mapper backend running with UDEV support enabled.
> # dm status abc [ opencount noflush ] [16384] (*1)
> # Keyslot 0 priority 1 != 2 (required), skipped.
> # Trying to open LUKS2 keyslot 0.
> # Reading keyslot area [0x8000].
> # Acquiring read lock for device /dev/v03/P50-Abc.
> # Opening lock resource file /run/cryptsetup/L_253:4
> # Verifying lock handle for /dev/v03/P50-Abc.
> # Device /dev/v03/P50-Abc READ lock taken.
> # Reusing open ro fd on device /dev/v03/P50-Abc
> # Device /dev/v03/P50-Abc READ lock released.
> # Verifying key from keyslot 0, digest 0.
> # Loading key (64 bytes, type logon) in thread keyring.
> # dm versions [ opencount flush ] [16384] (*1)
> # dm status abc [ opencount noflush ] [16384] (*1)
> # Calculated device size is 1258258432 sectors (RW), offset 32768.
> # DM-UUID is CRYPT-LUKS2-590a43f175534729a95c5c1beff86e2b-abc
> # Udev cookie 0xd4d7a75 (semid 7) created
> # Udev cookie 0xd4d7a75 (semid 7) incremented to 1
> # Udev cookie 0xd4d7a75 (semid 7) incremented to 2
> # Udev cookie 0xd4d7a75 (semid 7) assigned to CREATE task(0) with flags
> DISABLE_LIBRARY_FALLBACK (0x20)
> # dm create abc CRYPT-LUKS2-590a43f175534729a95c5c1beff86e2b-abc [
> opencount flush ] [16384] (*1)
> # dm reload abc [ opencount flush securedata ] [16384] (*1)
> # dm resume abc [ opencount flush securedata ] [16384] (*1)
> # abc: Stacking NODE_ADD (253,5) 0:0 0600 [trust_udev]
> # abc: Stacking NODE_READ_AHEAD 256 (flags=1)
> # Udev cookie 0xd4d7a75 (semid 7) decremented to 1
> # Udev cookie 0xd4d7a75 (semid 7) waiting for zero
> # Udev cookie 0xd4d7a75 (semid 7) destroyed
> # abc: Skipping NODE_ADD (253,5) 0:0 0600 [trust_udev]
> # abc: Processing NODE_READ_AHEAD 256 (flags=1)
> # abc (253:5): read ahead is 256
> # abc: retaining kernel read ahead of 256 (requested 256)
> Key slot 0 unlocked.
> # Releasing crypt device /dev/v03/P50-Abc context.
> # Releasing device-mapper backend.
> # Closing read only fd for /dev/v03/P50-Abc.
> # Unlocking memory.
> Command successful.
>
> But opening the "relevant" LV hangs when accessing the keyslot (we do
> have the proper password, thanks to password managers) :
>
> cryptsetup --verbose --debug open /dev/v03/E192 192
> # cryptsetup 2.3.2 processing "cryptsetup --verbose --debug open
> /dev/v03/E192 192"
> # Running command open.
> # Locking memory.
> # Installing SIGINT/SIGTERM handler.
> # Unblocking interruption on signal.
> # Allocating context for crypt device /dev/v03/E192.
> # Trying to open and read device /dev/v03/E192 with direct-io.
> # Initialising device-mapper backend library.
> # Trying to load any crypt type from device /dev/v03/E192.
> # Crypto backend (OpenSSL 1.1.1g 21 Apr 2020) initialized in cryptsetup
> library version 2.3.2.
> # Detected kernel Linux 5.4.38-gentoo-LTS x86_64.
> # PBKDF pbkdf2-sha256, time_ms 2000 (iterations 0).
> # Reading LUKS header of size 1024 from device /dev/v03/E192
> # Key length 32, device size 2147483648 sectors, header size 2050 sectors.
> # Activating volume 192 using token -1.
> # Interactive passphrase entry requested.
> Enter passphrase for /dev/v03/E192:
> # Activating volume 192 [keyslot -1] using passphrase.
> # dm version [ opencount flush ] [16384] (*1)
> # dm versions [ opencount flush ] [16384] (*1)
> # Detected dm-ioctl version 4.41.0.
> # Detected dm-crypt version 1.19.0.
> # Device-mapper backend running with UDEV support enabled.
> # dm status 192 [ opencount noflush ] [16384] (*1)
> # Trying to open key slot 0 [ACTIVE_LAST].
> # Reading key slot 0 area.
> # Using userspace crypto wrapper to access keyslot area.
> # Reusing open ro fd on device /dev/v03/E192 <- last message
> displayed by cryptsetup
>
> In this situation, ps says this about the cryptsetup process :
>
> 7265 pts/3 D<L+ 0:01 cryptsetup --verbose --debug open
> /dev/v03/E192 192
>
> Any idea on how to get back access to our data ?
> Or anything obviously wrong we did not see ?
>
> Tia,
>
More information about the linux-lvm
mailing list