[vdo-devel] dmsetup stuck for more than one day

[Andrew Walsh]

Hi Lukasz,

Can you please confirm a few details?  These will help us understand what
may be going on.  We may end up needing additional information, but this
will help us identify a starting point for the investigation.

**Storage Stack Configuration:**
High Level Configuration: [e.g. SSD -> MD RAID 5 -> VDO -> XFS]
Output of `blockdev --report`:
Output of `lsblk -o name,maj:min,kname,type,fstype,state,sched,uuid`:

**Hardware Information:**
 - CPU: [e.g. 2x Intel Xeon E5-1650 v2 @ 3.5GHz]
 - Memory: [e.g. 128G]
 - Storage: [e.g. Intel Optane SSD 900P]
 - Other: [e.g. iSCSI backed storage]

**Distro Information:**
 - OS: [e.g. RHEL-7.5]
 - Architecture: [e.g. x86_64]
 - Kernel: [e.g. kernel-3.10.0-862.el7]
 - VDO Version: [e.g. vdo-6.2.0.168-18.el7, or a commit hash]
 - KVDO Version: [e.g. kmod-kvdo6.2.0.153-15.el7, or a commit hash]
 - LVM Version: [e.g. 2.02.177-4.el7]
 - Output of `uname -a`: [e.g. Linux localhost.localdomain
3.10.0-862.el7.x86_64 #1 SMP Wed Mar 21 18:14:51 EDT 2018 x86_64 x86_64
x86_64 GNU/Linux]

Thanks,

Andy Walsh


On Thu, Nov 5, 2020 at 6:49 AM Łukasz Michalski <lm at zork.pl> wrote:

> Hi,
>
> I have two 20T two devices that was crashed during power outage - on two
> servers.
>
> After server restart I see in logs on the first server:
>
> [root at ixmed1 /]# dmesg |grep vdo
> [   11.223770] kvdo: modprobe: loaded version 6.1.0.168
> [   11.904949] kvdo0:dmsetup: starting device 'vdo_test' device
> instantiation 0 write policy auto
> [   11.904979] kvdo0:dmsetup: underlying device, REQ_FLUSH: not supported,
> REQ_FUA: not supported
> [   11.904985] kvdo0:dmsetup: Using mode sync automatically.
> [   11.905017] kvdo0:dmsetup: zones: 1 logical, 1 physical, 1 hash; base
> threads: 5
> [   11.966414] kvdo0:journalQ: Device was dirty, rebuilding reference
> counts
> [   12.452589] kvdo0:logQ0: Finished reading recovery journal
> [   12.458550] kvdo0:logQ0: Highest-numbered recovery journal block has
> sequence number 70548140, and the highest-numbered usable block is 70548140
> [   12.458556] kvdo0:logQ0: Replaying entries into slab journals
> [   13.538099] kvdo0:logQ0: Replayed 5568767 journal entries into slab
> journals
> [   14.174984] kvdo0:logQ0: Recreating missing journal entries
> [   14.175025] kvdo0:journalQ: Synthesized 0 missing journal entries
> [   14.177768] kvdo0:journalQ: Saving recovery progress
> [   14.636416] kvdo0:logQ0: Replaying 2528946 recovery entries into block
> map
>
> [root at ixmed1 /]# uptime
>  12:41:33 up 1 day,  4:07,  2 users,  load average: 1.06, 1.05, 1.16
>
> [root at ixmed1 /]# ps ax |grep vdo
>   1135 ?        Ss     0:00 /usr/bin/python /usr/bin/vdo start --all
> --confFile /etc/vdoconf.yml
>   1210 ?        R    21114668:39 dmsetup create vdo_Rada-ixmed --uuid
> VDO-b668b2d9-96bf-4840-a43d-6b7ab0a7f235 --table 0 72301908952 vdo
> /dev/disk/by-id/dm-name-vgStorage-LV_test 4096 disabled 0 32768 16380 on
> auto vdo_test
> ack=1,bio=4,bioRotationInterval=64,cpu=2,hash=1,logical=1,physical=1
>   1213 ?        S      1:51 [kvdo0:dedupeQ]
>   1214 ?        S      1:51 [kvdo0:journalQ]
>   1215 ?        S      1:51 [kvdo0:packerQ]
>   1216 ?        S      1:51 [kvdo0:logQ0]
>   1217 ?        S      1:51 [kvdo0:physQ0]
>   1218 ?        S      1:50 [kvdo0:hashQ0]
>   1219 ?        S      1:52 [kvdo0:bioQ0]
>   1220 ?        S      1:51 [kvdo0:bioQ1]
>   1221 ?        S      1:51 [kvdo0:bioQ2]
>   1222 ?        S      1:51 [kvdo0:bioQ3]
>   1223 ?        S      1:48 [kvdo0:ackQ]
>   1224 ?        S      1:49 [kvdo0:cpuQ0]
>   1225 ?        S      1:49 [kvdo0:cpuQ1]
>
> The only activity I see is that there are small writes shown in 'atop' to
> vdo underlying device.
>
> On the first server dmsetup takes 100% cpu (one core), on the second
> server dmsetup seems to be idle.
>
> What should I do in this situation?
>
> Regards,
> Łukasz
>
>
>
> _______________________________________________
> vdo-devel mailing list
> vdo-devel at redhat.com
> https://www.redhat.com/mailman/listinfo/vdo-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vdo-devel/attachments/20201105/f866aad0/attachment.htm>