From pbraun at nethence.com Sat Sep 19 11:21:09 2020 From: pbraun at nethence.com (Pierre-Philipp Braun) Date: Sat, 19 Sep 2020 13:21:09 +0200 Subject: [Linux-cluster] sparse-file clone breaks on GFS2 Message-ID: <1a91c0b7-68ef-1a82-5083-1dfdc2ba348f@nethence.com> Hello #linux-cluster I am attempting to host thin-provisioned virtual disks on GFS2. In that regard, I experience a weird and unexpected issue: when copying (or packing/extracting) a sparse file with a file-system on it, and which lives and gets cloned on a GFS2 mount point, it results in a corrupted file. Is this a localized bug/issue I am having only here in my farm, or can you reproduce it on yours? Here's how to reproduce. On a GFS2 file-system, creating a raw sparse file and an EXT4 file-system on it. dd if=/dev/zero of=dummy.ext4 bs=1G count=0 seek=1 mkfs.ext4 dummy.ext4 md5sum dummy.ext4 c4bd385e82afcd322a4b32a02d652d79 dummy.ext4 cp --sparse=always dummy.ext4 dummy-clone.ext4 md5sum dummy-clone.ext4 d1966a954a17894a721142a27db0f5b5 dummy-clone.ext4 fsck.ext4 dummy-clone.ext4 e2fsck 1.45.6 (20-Mar-2020) Superblock has an invalid journal (inode 8). Clear? What could be the root cause of this issue? Any ideas? Thank you -- Pierre-Philipp From g.danti at assyoma.it Sat Sep 19 12:14:47 2020 From: g.danti at assyoma.it (Gionatan Danti) Date: Sat, 19 Sep 2020 14:14:47 +0200 Subject: [Linux-cluster] sparse-file clone breaks on GFS2 In-Reply-To: <1a91c0b7-68ef-1a82-5083-1dfdc2ba348f@nethence.com> References: <1a91c0b7-68ef-1a82-5083-1dfdc2ba348f@nethence.com> Message-ID: <6fc86a8b675de6c49d2375962677021e@assyoma.it> Il 2020-09-19 13:21 Pierre-Philipp Braun ha scritto: > Hello #linux-cluster > > I am attempting to host thin-provisioned virtual disks on GFS2. In > that regard, I experience a weird and unexpected issue: when copying > (or packing/extracting) a sparse file with a file-system on it, and > which lives and gets cloned on a GFS2 mount point, it results in a > corrupted file. Hi, can you get the md5sum of *both* the original and copied files? If you use "cp --sparse=never", does it change anything? Thanks. -- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: g.danti at assyoma.it - info at assyoma.it GPG public key ID: FF5F32A8 From pbraun at nethence.com Sat Sep 19 16:57:08 2020 From: pbraun at nethence.com (Pierre-Philipp Braun) Date: Sat, 19 Sep 2020 18:57:08 +0200 Subject: [Linux-cluster] sparse-file clone breaks on GFS2 In-Reply-To: <6fc86a8b675de6c49d2375962677021e@assyoma.it> References: <1a91c0b7-68ef-1a82-5083-1dfdc2ba348f@nethence.com> <6fc86a8b675de6c49d2375962677021e@assyoma.it> Message-ID: <31072110-9e61-e41b-ad44-c97eb600d3f3@nethence.com> On 19.09.2020 14:14, Gionatan Danti wrote: > cp --sparse=never Hello neighbor (ciao from France). I am not sure what you mean, as there **was both** the original and the copied file sums in my message. Anyhow, thank you and good idea about --sparse=never. I applied the checksum multiple times -- at at each step -- just to make sure the results are consistent. cp --sparse=never dummy.ext4 dummy-never.ext4 cp --sparse=auto dummy.ext4 dummy-auto.ext4 cp --sparse=always dummy.ext4 dummy-clone.ext4 And the sums always remain the same. So I only print the output only once here below. ce2b843692051a2f4f81366f9680b675 dummy.ext4 33e84fdf0a328a64c4edd00342dd433d dummy-never.ext4 33e84fdf0a328a64c4edd00342dd433d dummy-auto.ext4 33e84fdf0a328a64c4edd00342dd433d dummy-clone.ext4 Although the GFS2 sparse-file is fine and usable as-is, it cannot be copied with `cp` nor packed with GNU tar, and it does not matter if our aim is to actually keep the sparse. I also attempted to copy a GFS2 sparse file to some EXT4 file-system. cp --sparse=always dummy.ext4 /root/dummy-clone-on-ext4.ext4 And same happens. 33e84fdf0a328a64c4edd00342dd433d /root/dummy-clone-on-ext4.ext4 While with the inverse, namely copying an EXT4 sparse-file on GFS2. cd /root/ dd if=/dev/zero of=dummy-on-ext4.ext4 bs=1G count=0 seek=1 mkfs.ext4 dummy-on-ext4.ext4 cp --sparse=always dummy-on-ext4.ext4 /data2/dummy-clone-from-ext4.ext4 delivers the sparse-file intact. No problem there. cd573cfaace07e7949bc0c46028904ff /root/dummy-on-ext4.ext4 cd573cfaace07e7949bc0c46028904ff /data2/dummy-clone-from-ext4.ext4 and... WOW! Cloning it yet again, it finally remains intact! cd573cfaace07e7949bc0c46028904ff dummy-clone-from-ext4-and-clone.ext4 cd573cfaace07e7949bc0c46028904ff dummy-clone-from-ext4.ext4 It's strange. Anyone? In case it matters, I am using vanilla Linux 4.18.20 and not the RHEL nor CentOS with patches. -- Pierre-Philipp From g.danti at assyoma.it Sat Sep 19 21:07:03 2020 From: g.danti at assyoma.it (Gionatan Danti) Date: Sat, 19 Sep 2020 23:07:03 +0200 Subject: [Linux-cluster] sparse-file clone breaks on GFS2 In-Reply-To: <31072110-9e61-e41b-ad44-c97eb600d3f3@nethence.com> References: <1a91c0b7-68ef-1a82-5083-1dfdc2ba348f@nethence.com> <6fc86a8b675de6c49d2375962677021e@assyoma.it> <31072110-9e61-e41b-ad44-c97eb600d3f3@nethence.com> Message-ID: <157fa2bd68f1f0543b2b9e22c810b727@assyoma.it> Il 2020-09-19 18:57 Pierre-Philipp Braun ha scritto: > On 19.09.2020 14:14, Gionatan Danti wrote: >> cp --sparse=never > > Hello neighbor (ciao from France). > > I am not sure what you mean, as there **was both** the original and > the copied file sums in my message. Sorry, I somewhat managed to miss them :| > Anyhow, thank you and good idea about --sparse=never. I applied the > checksum multiple times -- at at each step -- just to make sure the > results are consistent. > > cp --sparse=never dummy.ext4 dummy-never.ext4 > cp --sparse=auto dummy.ext4 dummy-auto.ext4 > cp --sparse=always dummy.ext4 dummy-clone.ext4 > > And the sums always remain the same. So I only print the output only > once here below. > > ce2b843692051a2f4f81366f9680b675 dummy.ext4 > 33e84fdf0a328a64c4edd00342dd433d dummy-never.ext4 > 33e84fdf0a328a64c4edd00342dd433d dummy-auto.ext4 > 33e84fdf0a328a64c4edd00342dd433d dummy-clone.ext4 > > Although the GFS2 sparse-file is fine and usable as-is, it cannot be > copied with `cp` nor packed with GNU tar, and it does not matter if > our aim is to actually keep the sparse. > > I also attempted to copy a GFS2 sparse file to some EXT4 file-system. > > cp --sparse=always dummy.ext4 /root/dummy-clone-on-ext4.ext4 > > And same happens. > > 33e84fdf0a328a64c4edd00342dd433d /root/dummy-clone-on-ext4.ext4 > > While with the inverse, namely copying an EXT4 sparse-file on GFS2. > > cd /root/ > dd if=/dev/zero of=dummy-on-ext4.ext4 bs=1G count=0 seek=1 > mkfs.ext4 dummy-on-ext4.ext4 > cp --sparse=always dummy-on-ext4.ext4 /data2/dummy-clone-from-ext4.ext4 > > delivers the sparse-file intact. No problem there. > > > cd573cfaace07e7949bc0c46028904ff /root/dummy-on-ext4.ext4 > cd573cfaace07e7949bc0c46028904ff /data2/dummy-clone-from-ext4.ext4 > > and... WOW! Cloning it yet again, it finally remains intact! > > cd573cfaace07e7949bc0c46028904ff dummy-clone-from-ext4-and-clone.ext4 > cd573cfaace07e7949bc0c46028904ff dummy-clone-from-ext4.ext4 > > It's strange. Anyone? Absolutely. Really strange. > In case it matters, I am using vanilla Linux 4.18.20 and not the RHEL > nor CentOS with patches. What distribution are you using (Slackware)? Any additional data about your filesystem (ie: num journals/nodes)? I would like to try to replicate the issue if I can find some spare time... Thanks. -- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: g.danti at assyoma.it - info at assyoma.it GPG public key ID: FF5F32A8 From pbraun at nethence.com Sun Sep 20 17:19:08 2020 From: pbraun at nethence.com (Pierre-Philipp Braun) Date: Sun, 20 Sep 2020 19:19:08 +0200 Subject: [Linux-cluster] sparse-file clone breaks on GFS2 In-Reply-To: <157fa2bd68f1f0543b2b9e22c810b727@assyoma.it> References: <1a91c0b7-68ef-1a82-5083-1dfdc2ba348f@nethence.com> <6fc86a8b675de6c49d2375962677021e@assyoma.it> <31072110-9e61-e41b-ad44-c97eb600d3f3@nethence.com> <157fa2bd68f1f0543b2b9e22c810b727@assyoma.it> Message-ID: <162c962b-d93b-1888-57c1-e2b7f1703361@nethence.com> >> cd573cfaace07e7949bc0c46028904ff /root/dummy-on-ext4.ext4 >> cd573cfaace07e7949bc0c46028904ff /data2/dummy-clone-from-ext4.ext4 >> >> and... WOW! Cloning it yet again, it finally remains intact! >> >> cd573cfaace07e7949bc0c46028904ff dummy-clone-from-ext4-and-clone.ext4 >> cd573cfaace07e7949bc0c46028904ff dummy-clone-from-ext4.ext4 >> >> It's strange. Anyone? > > Absolutely. Really strange. It truly happened, however when I try this again, I cannot reproduce this right now. All I see now is a corruption whenever I copy a sparse file-system from GFS2, be it created on EXT4 or not. dd if=/dev/zero of=/root/dummy.ext4 bs=1G count=0 seek=1 mkfs.ext4 /root/dummy.ext4 cp --sparse=always /root/dummy.ext4 /data2/dummy-clone-from-ext4.ext4 cp --sparse=always /data2/dummy-clone-from-ext4.ext4 /data2/dummy-clone-from-ext4-clone.ext4 cp --sparse=always /data2/dummy-clone-from-ext4.ext4 /root/dummy-clone-from-gfs2.ext4 md5sum /root/dummy.ext4 /data2/dummy-clone-from-ext4.ext4 /data2/dummy-clone-from-ext4-clone.ext4 /root/dummy-clone-from-gfs2.ext4 gives aefedffac2f2cbc0d8fe15155703e7a0 /root/dummy.ext4 aefedffac2f2cbc0d8fe15155703e7a0 /data2/dummy-clone-from-ext4.ext4 a3da5ced2af823e06a352124b9c800c7 /data2/dummy-clone-from-ext4-clone.ext4 a3da5ced2af823e06a352124b9c800c7 /root/dummy-clone-from-gfs2.ext4 and it's consistent, as shown with this second run: be123782b303fc391f93478063d21b51 /root/dummy.ext4 be123782b303fc391f93478063d21b51 /data2/dummy-clone-from-ext4.ext4 22fa85945a94001089740f024a4c3f1e /data2/dummy-clone-from-ext4-clone.ext4 22fa85945a94001089740f024a4c3f1e /root/dummy-clone-from-gfs2.ext4 same happens with XFS sparse file: dd if=/dev/zero of=/root/dummy.xfs bs=1G count=0 seek=1 mkfs.xfs /root/dummy.xfs cp --sparse=always /root/dummy.xfs /data2/dummy-clone-from-ext4.xfs cp --sparse=always /data2/dummy-clone-from-ext4.xfs /data2/dummy-clone-from-ext4-clone.xfs cp --sparse=always /data2/dummy-clone-from-ext4.xfs /root/dummy-clone-from-gfs2.xfs md5sum /root/dummy.xfs /data2/dummy-clone-from-ext4.xfs /data2/dummy-clone-from-ext4-clone.xfs /root/dummy-clone-from-gfs2.xfs 324c660d154cad3dd35f69a8531fc290 /root/dummy.xfs 324c660d154cad3dd35f69a8531fc290 /data2/dummy-clone-from-ext4.xfs 2e58d6e15a01ef3cf31b89e0ff219c7a /data2/dummy-clone-from-ext4-clone.xfs 2e58d6e15a01ef3cf31b89e0ff219c7a /root/dummy-clone-from-gfs2.xfs and thankfully this does not happen with a sparse-file without a file-system on it, namely, full of _only_ zeroes: dd if=/dev/zero of=/data2/dummy bs=1G count=0 seek=1 cp --sparse=always /data2/dummy /data2/dummy-clone md5sum /data2/dummy /data2/dummy-clone cd573cfaace07e7949bc0c46028904ff /data2/dummy cd573cfaace07e7949bc0c46028904ff /data2/dummy-clone cd573cfaace07e7949bc0c46028904ff /data2/dummy cd573cfaace07e7949bc0c46028904ff /data2/dummy-clone hexdump -C /data2/dummy 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 40000000 I will try to do other tests with self-forged data plus some sparse and see how it goes. I doubt it's related to something file-system-on-gfs2 specific. >> In case it matters, I am using vanilla Linux 4.18.20 and not the RHEL >> nor CentOS with patches. > > What distribution are you using (Slackware)? Any additional data about > your filesystem (ie: num journals/nodes)? I would like to try to > replicate the issue if I can find some spare time... Slackware current with a hopefully close-enough kernel version as RHEL8 + gfs2-utils and dlm *1 but I suppose it can be reproduced with RHEL8. Actually I would prefer if you could reproduce the issue with a rather classical setup. I would make my case easier to defend ;-) GFS2 file-system was created as such: `mkfs.gfs2 -j2 -t pro5s:data2 /dev/drbd2`, running on two DRBD8 dual-primary nodes. I still have no idea at what storage layer I am having troubles with here, so I also tested with a single mount on node2. And I also tested while having node1 as secondary. No success there and it's actually good news for me, as I wouldn't like my infrastructure to be f&^%ed up at the block-device level. # mkfs.gfs2 -V mkfs.gfs2 master (built Aug 16 2020 21:36:57) Copyright (C) Red Hat, Inc. 2004-2010 All rights reserved. oops, I should not be using the development branch so I switched to the release, but no big surprise, copying an EXT4 sparse file from GFS2, still corrupts it, here. # mkfs.gfs2 -V mkfs.gfs2 3.3.0 (built Sep 20 2020 19:28:37) Copyright (C) Red Hat, Inc. 2004-2020 All rights reserved. # dlm_controld -V dlm_controld 4.0.9 (built Aug 17 2020 07:02:31) Copyright Red Hat, Inc. 2004-2013 # tunegfs2 -l /dev/drbd2 tunegfs2 (Aug 16 2020 21:36:58) File system volume name: pro5s:data2 File system UUID: 6a95be71-e865-4706-b7c8-0c1bb0c3e232 File system magic number: 0x1161970 Block size: 4096 Block shift: 12 Root inode: 66241 Master inode: 32854 Lock protocol: lock_dlm Lock table: pro5s:data2 I am looking forward to hearing from you or anyone with a GFS2 mount in place, who's willing to reproduce the issue. Here are sample command lines on how to reproduce, meanwhile I try with something else than some mkfs. dd if=/dev/zero of=dummy.xfs bs=1G count=0 seek=1 mkfs.xfs dummy.xfs cp --sparse=always dummy.xfs dummy-clone.xfs md5sum dummy.xfs dummy-clone.xfs *1 https://pagure.io/gfs2-utils https://pagure.io/dlm BR -elge From g.danti at assyoma.it Mon Sep 21 15:56:11 2020 From: g.danti at assyoma.it (Gionatan Danti) Date: Mon, 21 Sep 2020 17:56:11 +0200 Subject: [Linux-cluster] sparse-file clone breaks on GFS2 In-Reply-To: References: <1a91c0b7-68ef-1a82-5083-1dfdc2ba348f@nethence.com> <6fc86a8b675de6c49d2375962677021e@assyoma.it> <31072110-9e61-e41b-ad44-c97eb600d3f3@nethence.com> Message-ID: Il 2020-09-21 15:45 Andreas Gruenbacher ha scritto: > On Mon, Sep 21, 2020 at 3:40 PM Pierre-Philipp Braun > wrote: >> In case it matters, I am using vanilla Linux 4.18.20 and not the RHEL >> nor CentOS with patches. > > It seem that 4.18.20 is missing the relevant parts of the following > fixes: > > a27a0c9b6a2 ("gfs2: gfs2_walk_metadata fix") > 566a2ab3c90 ("gfs2: Another gfs2_walk_metadata fix") > > Without those fixes, lseek SEEK_HOLE/SEEK_DATa will sometimes report > garbage, and so cp doesn't know what to copy. > > Andreas Well, I was just sitting down to create two CentOS 8.2 box for testing, but I really think this explains the issue reported by Pierre-Philipp. Thanks so much for these valuable information! Can you confirm that RHEL 8.x includes the above patches? Regards. -- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: g.danti at assyoma.it - info at assyoma.it GPG public key ID: FF5F32A8 From pbraun at nethence.com Thu Sep 24 09:07:04 2020 From: pbraun at nethence.com (Pierre-Philipp Braun) Date: Thu, 24 Sep 2020 11:07:04 +0200 Subject: [Linux-cluster] sparse-file clone breaks on GFS2 In-Reply-To: References: <1a91c0b7-68ef-1a82-5083-1dfdc2ba348f@nethence.com> <6fc86a8b675de6c49d2375962677021e@assyoma.it> <31072110-9e61-e41b-ad44-c97eb600d3f3@nethence.com> Message-ID: <1b692c1e-0c95-e95a-d59c-b026b57c34ff@nethence.com> >> It seem that 4.18.20 is missing the relevant parts of the following >> fixes: >> >> ? a27a0c9b6a2 ("gfs2: gfs2_walk_metadata fix") >> ? 566a2ab3c90 ("gfs2: Another gfs2_walk_metadata fix") >> >> Without those fixes, lseek SEEK_HOLE/SEEK_DATa will sometimes report >> garbage, and so cp doesn't know what to copy. >> >> Andreas Wow I was missing that message. I did not receive it nor it is available online on the archive, it states `Message not available`. https://www.redhat.com/archives/linux-cluster/2020-September/thread.html Any idea why? I also wonder how Gionatan could receive it. Yes that did the trick, thank you very much Andreas, for your patches and your (hidden) answer. > Well, I was just sitting down to create two CentOS 8.2 box for testing, > but I really think this explains the issue reported by Pierre-Philipp. > Thanks so much for these valuable information! > > Can you confirm that RHEL 8.x includes the above patches? I found those two patches referenced in the source RPM spec indeed. That's still the 4.18.0 kernel. https://git.centos.org/rpms/kernel/blob/c8s/f/SPECS/kernel.spec And then more recently over there (20 May 2020) Linux 5.4.42 https://lwn.net/Articles/820973/ So I could finally handle sparse file alright with the aforementioned version. It's a pity I was stuck with some bug-or-incompatibility with version 5.4.58 last month (https://www.redhat.com/archives/linux-cluster/2020-August/msg00000.html): I was so close to working version already. Thing is, it was probably the latest longterm kernel release available at that time. Today's longterm version is 5.4.67 and I avoided it. I tried stable Linux 5.8.x instead, and it's a success. I am finally enjoying happy virtual disks on GFS2 and from all nodes. Here again and for the record, the sparse-file check, dd if=/dev/zero of=dummy.ext4 bs=1GB count=0 seek=1 mkfs.ext4 dummy.ext4 cp --sparse=always dummy.ext4 dummy-clone.ext4 md5sum dummy.ext4 dummy-clone.ext4