From bkearney at redhat.com Fri Aug 3 14:08:05 2012 From: bkearney at redhat.com (Bryan Kearney) Date: Fri, 03 Aug 2012 10:08:05 -0400 Subject: [Thincrust-devel] TEST Message-ID: <501BDB45.5020403@redhat.com> TEST From msavy at redhat.com Fri Aug 3 14:16:15 2012 From: msavy at redhat.com (Marc Savy) Date: Fri, 03 Aug 2012 15:16:15 +0100 Subject: [Thincrust-devel] [PATCH] Install grub in chroot to ensure correct ld-linux libs used. In-Reply-To: <501BC68B.5030505@redhat.com> References: <501BC68B.5030505@redhat.com> Message-ID: <501BDD2F.2080709@redhat.com> Install Grub using chroot so that it uses the ld-linux.so downloaded for the *appliance* rather than using the *host's* ld-linux.so which may be invalid (the wrong arch type, etc). From ce06bb9c95be57b4a3b4c88442e911d2404e5cee Mon Sep 17 00:00:00 2001 From: Marc Savy Date: Tue, 31 Jul 2012 16:03:18 +0100 Subject: [PATCH] Install grub in chroot to ensure correct ld-linux libs used. To: thincrust-devel at redhat.com --- appcreate/appliance.py | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/appcreate/appliance.py b/appcreate/appliance.py index f9fc784..1a13182 100644 --- a/appcreate/appliance.py +++ b/appcreate/appliance.py @@ -287,7 +287,7 @@ class ApplianceImageCreator(ImageCreator): # Ensure all data is flushed to disk before doing grub install subprocess.call(["sync"]) - stage2 = self._instroot + "/boot/grub/stage2" + stage2 = "/boot/grub/stage2" setup = "" i = 0 @@ -301,15 +301,20 @@ class ApplianceImageCreator(ImageCreator): logging.debug("Installing grub to %s" % loopdev) - grub = subprocess.Popen([self._instroot + "/sbin/grub", "--batch", "--no-floppy"], + subprocess.call(["mount", "--bind", "/dev", self._instroot + "/dev"]) + + grub = subprocess.Popen(["chroot", self._instroot, "/sbin/grub", "--batch", "--no-floppy"], stdin=subprocess.PIPE) grub.communicate(setup) rc = grub.wait() if rc != 0: + subprocess.call(["umount", self._instroot + "/dev"]) raise MountError("Unable to install grub bootloader") + subprocess.call(["umount", self._instroot + "/dev"]) + logging.debug("Grub installed.") def _install_grub2(self): -- 1.7.11.3 -------------- next part -------------- From ce06bb9c95be57b4a3b4c88442e911d2404e5cee Mon Sep 17 00:00:00 2001 From: Marc Savy Date: Tue, 31 Jul 2012 16:03:18 +0100 Subject: [PATCH] Install grub in chroot to ensure correct ld-linux libs used. To: thincrust-devel at redhat.com --- appcreate/appliance.py | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/appcreate/appliance.py b/appcreate/appliance.py index f9fc784..1a13182 100644 --- a/appcreate/appliance.py +++ b/appcreate/appliance.py @@ -287,7 +287,7 @@ class ApplianceImageCreator(ImageCreator): # Ensure all data is flushed to disk before doing grub install subprocess.call(["sync"]) - stage2 = self._instroot + "/boot/grub/stage2" + stage2 = "/boot/grub/stage2" setup = "" i = 0 @@ -301,15 +301,20 @@ class ApplianceImageCreator(ImageCreator): logging.debug("Installing grub to %s" % loopdev) - grub = subprocess.Popen([self._instroot + "/sbin/grub", "--batch", "--no-floppy"], + subprocess.call(["mount", "--bind", "/dev", self._instroot + "/dev"]) + + grub = subprocess.Popen(["chroot", self._instroot, "/sbin/grub", "--batch", "--no-floppy"], stdin=subprocess.PIPE) grub.communicate(setup) rc = grub.wait() if rc != 0: + subprocess.call(["umount", self._instroot + "/dev"]) raise MountError("Unable to install grub bootloader") + subprocess.call(["umount", self._instroot + "/dev"]) + logging.debug("Grub installed.") def _install_grub2(self): -- 1.7.11.3 From msavy at redhat.com Fri Aug 3 14:25:05 2012 From: msavy at redhat.com (Marc Savy) Date: Fri, 03 Aug 2012 15:25:05 +0100 Subject: [Thincrust-devel] [PATCH 1/3] Remove extended partition from kpartx output to avoid partition count mismatch and erroneous device assignment Message-ID: <501BDF41.7060106@redhat.com> With msdos type layouts that have large numbers of partitions, the output from kpartx displays the extended partition, which is incorrectly interpreted as meaning an extra partition is present which aught not be. This patch pops the line from the kpartx output so that it may be safely ignored. From 2dabfe5646ca941ffa09ca3f4fbcb08bf0a2fe52 Mon Sep 17 00:00:00 2001 From: Marc Savy Date: Wed, 18 Jul 2012 14:05:20 +0100 Subject: [PATCH 1/3] Remove extended partition from kpartx output to avoid partition count mismatch and erroneous device assignment. To: thincrust-devel at redhat.com --- appcreate/partitionedfs.py | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/appcreate/partitionedfs.py b/appcreate/partitionedfs.py index 675ac71..705c2ae 100644 --- a/appcreate/partitionedfs.py +++ b/appcreate/partitionedfs.py @@ -46,6 +46,7 @@ class PartitionedMount(Mount): self.mountOrder = [] self.unmountOrder = [] self.partition_layout = partition_layout + self.has_extended = False # Has extended partition layout def add_partition(self, size, disk, mountpoint, fstype = None): self.partitions.append({'size': size, @@ -94,6 +95,7 @@ class PartitionedMount(Mount): for p in self.partitions: d = self.disks[p['disk']] if p['num'] == 5 and self.partition_layout == 'msdos': + self.has_extended = True logging.debug("Added extended part at %d of size %d" % (p['start'], d['extended'])) rc = subprocess.call(["/sbin/parted", "-s", d['disk'].device, "mkpart", "extended", "%dM" % p['start'], "%dM" % (p['start'] + d['extended'])]) @@ -125,6 +127,10 @@ class PartitionedMount(Mount): raise MountError("Failed to query partition mapping for '%s'" % d.device) + # Pop the fourth (extended) partition + if self.has_extended: + kpartxOutput.pop(3) + # Quick sanity check that the number of partitions matches # our expectation. If it doesn't, someone broke the code # further up -- 1.7.11.2 -------------- next part -------------- From 2dabfe5646ca941ffa09ca3f4fbcb08bf0a2fe52 Mon Sep 17 00:00:00 2001 From: Marc Savy Date: Wed, 18 Jul 2012 14:05:20 +0100 Subject: [PATCH 1/3] Remove extended partition from kpartx output to avoid partition count mismatch and erroneous device assignment. To: thincrust-devel at redhat.com --- appcreate/partitionedfs.py | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/appcreate/partitionedfs.py b/appcreate/partitionedfs.py index 675ac71..705c2ae 100644 --- a/appcreate/partitionedfs.py +++ b/appcreate/partitionedfs.py @@ -46,6 +46,7 @@ class PartitionedMount(Mount): self.mountOrder = [] self.unmountOrder = [] self.partition_layout = partition_layout + self.has_extended = False # Has extended partition layout def add_partition(self, size, disk, mountpoint, fstype = None): self.partitions.append({'size': size, @@ -94,6 +95,7 @@ class PartitionedMount(Mount): for p in self.partitions: d = self.disks[p['disk']] if p['num'] == 5 and self.partition_layout == 'msdos': + self.has_extended = True logging.debug("Added extended part at %d of size %d" % (p['start'], d['extended'])) rc = subprocess.call(["/sbin/parted", "-s", d['disk'].device, "mkpart", "extended", "%dM" % p['start'], "%dM" % (p['start'] + d['extended'])]) @@ -125,6 +127,10 @@ class PartitionedMount(Mount): raise MountError("Failed to query partition mapping for '%s'" % d.device) + # Pop the fourth (extended) partition + if self.has_extended: + kpartxOutput.pop(3) + # Quick sanity check that the number of partitions matches # our expectation. If it doesn't, someone broke the code # further up -- 1.7.11.2 From msavy at redhat.com Fri Aug 3 14:25:44 2012 From: msavy at redhat.com (Marc Savy) Date: Fri, 03 Aug 2012 15:25:44 +0100 Subject: [Thincrust-devel] [PATCH 2/3] Ensure that more than 10 devices can be assigned symlink, mappings that do not overlap. Message-ID: <501BDF68.7000809@redhat.com> Previously this behaviour was exhibited when over 10 devices are present: https://gist.github.com/3eda2490dde8c49e26b1 This patch ensures that the entire device number is matched. From 5efb73c15b9a2851ebb0d3b4d7a625a27bb46317 Mon Sep 17 00:00:00 2001 From: Marc Savy Date: Wed, 18 Jul 2012 14:09:44 +0100 Subject: [PATCH 2/3] Ensure that more than 10 devices can be assigned symlink mappings that do not overlap. To: thincrust-devel at redhat.com --- appcreate/partitionedfs.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/appcreate/partitionedfs.py b/appcreate/partitionedfs.py index 705c2ae..4982a69 100644 --- a/appcreate/partitionedfs.py +++ b/appcreate/partitionedfs.py @@ -24,6 +24,7 @@ import glob import shutil import subprocess import logging +import re from imgcreate.errors import * from imgcreate.fs import * @@ -141,8 +142,10 @@ class PartitionedMount(Mount): for i in range(len(kpartxOutput)): line = kpartxOutput[i] newdev = line.split()[0] + newdev_id = re.search('^loop\d+?p(\d+)$', newdev).group(1) + mapperdev = "/dev/mapper/" + newdev - loopdev = d['disk'].device + newdev[-1] + loopdev = d['disk'].device + newdev_id logging.debug("Dev %s: %s -> %s" % (newdev, loopdev, mapperdev)) pnum = d['partitions'][i] -- 1.7.11.2 -------------- next part -------------- From 5efb73c15b9a2851ebb0d3b4d7a625a27bb46317 Mon Sep 17 00:00:00 2001 From: Marc Savy Date: Wed, 18 Jul 2012 14:09:44 +0100 Subject: [PATCH 2/3] Ensure that more than 10 devices can be assigned symlink mappings that do not overlap. To: thincrust-devel at redhat.com --- appcreate/partitionedfs.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/appcreate/partitionedfs.py b/appcreate/partitionedfs.py index 705c2ae..4982a69 100644 --- a/appcreate/partitionedfs.py +++ b/appcreate/partitionedfs.py @@ -24,6 +24,7 @@ import glob import shutil import subprocess import logging +import re from imgcreate.errors import * from imgcreate.fs import * @@ -141,8 +142,10 @@ class PartitionedMount(Mount): for i in range(len(kpartxOutput)): line = kpartxOutput[i] newdev = line.split()[0] + newdev_id = re.search('^loop\d+?p(\d+)$', newdev).group(1) + mapperdev = "/dev/mapper/" + newdev - loopdev = d['disk'].device + newdev[-1] + loopdev = d['disk'].device + newdev_id logging.debug("Dev %s: %s -> %s" % (newdev, loopdev, mapperdev)) pnum = d['partitions'][i] -- 1.7.11.2 From msavy at redhat.com Fri Aug 3 14:25:57 2012 From: msavy at redhat.com (Marc Savy) Date: Fri, 03 Aug 2012 15:25:57 +0100 Subject: [Thincrust-devel] [PATCH 3/3] Ensure loop device symlinks have not been left behind, from untidy termination before assigning them. Message-ID: <501BDF75.1010305@redhat.com> It is possible in certain termination situations for the program to be unable to tidy up the symlinks it has created, causing collisions on the next run. This patch is an attempt to resolve that. From bd3f1609a36670cdc2936e9ecd7440d6ab21014f Mon Sep 17 00:00:00 2001 From: Marc Savy Date: Wed, 18 Jul 2012 14:13:09 +0100 Subject: [PATCH 3/3] Ensure loop device symlinks have not been left behind from untidy termination before assigning them. To: thincrust-devel at redhat.com --- appcreate/partitionedfs.py | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/appcreate/partitionedfs.py b/appcreate/partitionedfs.py index 4982a69..37df485 100644 --- a/appcreate/partitionedfs.py +++ b/appcreate/partitionedfs.py @@ -152,11 +152,21 @@ class PartitionedMount(Mount): self.partitions[pnum]['device'] = loopdev self.partitions[pnum]['devicemapper'] = mapperdev - # grub's install wants partitions to be named - # to match their parent device + partition num - # kpartx doesn't work like this, so we add compat - # symlinks to point to /dev/mapper - os.symlink(mapperdev, loopdev) + # Loop devices are sometimes still left hanging around from untidily + # terminated processes. + logging.debug('Creating symlink from %s to %s', loopdev, mapperdev) + try: + # grub's install wants partitions to be named + # to match their parent device + partition num + # kpartx doesn't work like this, so we add compat + # symlinks to point to /dev/mapper + os.symlink(mapperdev, loopdev) + except OSError as e: + if e.errno == errno.EEXIST: + os.unlink(loopdev) + os.symlink(mapperdev, loopdev) + else: + raise logging.debug("Adding partx mapping for %s" % d['disk'].device) rc = subprocess.call(["/sbin/kpartx", "-a", "-s", d['disk'].device]) -- 1.7.11.2 -------------- next part -------------- From bd3f1609a36670cdc2936e9ecd7440d6ab21014f Mon Sep 17 00:00:00 2001 From: Marc Savy Date: Wed, 18 Jul 2012 14:13:09 +0100 Subject: [PATCH 3/3] Ensure loop device symlinks have not been left behind from untidy termination before assigning them. To: thincrust-devel at redhat.com --- appcreate/partitionedfs.py | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/appcreate/partitionedfs.py b/appcreate/partitionedfs.py index 4982a69..37df485 100644 --- a/appcreate/partitionedfs.py +++ b/appcreate/partitionedfs.py @@ -152,11 +152,21 @@ class PartitionedMount(Mount): self.partitions[pnum]['device'] = loopdev self.partitions[pnum]['devicemapper'] = mapperdev - # grub's install wants partitions to be named - # to match their parent device + partition num - # kpartx doesn't work like this, so we add compat - # symlinks to point to /dev/mapper - os.symlink(mapperdev, loopdev) + # Loop devices are sometimes still left hanging around from untidily + # terminated processes. + logging.debug('Creating symlink from %s to %s', loopdev, mapperdev) + try: + # grub's install wants partitions to be named + # to match their parent device + partition num + # kpartx doesn't work like this, so we add compat + # symlinks to point to /dev/mapper + os.symlink(mapperdev, loopdev) + except OSError as e: + if e.errno == errno.EEXIST: + os.unlink(loopdev) + os.symlink(mapperdev, loopdev) + else: + raise logging.debug("Adding partx mapping for %s" % d['disk'].device) rc = subprocess.call(["/sbin/kpartx", "-a", "-s", d['disk'].device]) -- 1.7.11.2 From mgoldman at redhat.com Wed Aug 8 06:56:02 2012 From: mgoldman at redhat.com (Marek Goldmann) Date: Wed, 08 Aug 2012 08:56:02 +0200 Subject: [Thincrust-devel] [PATCH] Install grub in chroot to ensure correct ld-linux libs used. In-Reply-To: <501BDD2F.2080709@redhat.com> References: <501BC68B.5030505@redhat.com> <501BDD2F.2080709@redhat.com> Message-ID: <50220D82.90802@redhat.com> Hi Marc, This patch looks good for me. One question - why do you umount /dev twice if rc != 0? --Marek On 03.08.2012 16:16, Marc Savy wrote: > Install Grub using chroot so that it uses the ld-linux.so downloaded for > the *appliance* rather than using the *host's* ld-linux.so which may be > invalid (the wrong arch type, etc). > > From ce06bb9c95be57b4a3b4c88442e911d2404e5cee Mon Sep 17 00:00:00 2001 > From: Marc Savy > Date: Tue, 31 Jul 2012 16:03:18 +0100 > Subject: [PATCH] Install grub in chroot to ensure correct ld-linux libs > used. > To: thincrust-devel at redhat.com > > --- > appcreate/appliance.py | 9 +++++++-- > 1 file changed, 7 insertions(+), 2 deletions(-) > > diff --git a/appcreate/appliance.py b/appcreate/appliance.py > index f9fc784..1a13182 100644 > --- a/appcreate/appliance.py > +++ b/appcreate/appliance.py > @@ -287,7 +287,7 @@ class ApplianceImageCreator(ImageCreator): > # Ensure all data is flushed to disk before doing grub install > subprocess.call(["sync"]) > > - stage2 = self._instroot + "/boot/grub/stage2" > + stage2 = "/boot/grub/stage2" > setup = "" > > i = 0 > @@ -301,15 +301,20 @@ class ApplianceImageCreator(ImageCreator): > > logging.debug("Installing grub to %s" % loopdev) > > - grub = subprocess.Popen([self._instroot + "/sbin/grub", > "--batch", "--no-floppy"], > + subprocess.call(["mount", "--bind", "/dev", self._instroot + > "/dev"]) > + > + grub = subprocess.Popen(["chroot", self._instroot, > "/sbin/grub", "--batch", "--no-floppy"], > stdin=subprocess.PIPE) > > grub.communicate(setup) > rc = grub.wait() > > if rc != 0: > + subprocess.call(["umount", self._instroot + "/dev"]) > raise MountError("Unable to install grub bootloader") > > + subprocess.call(["umount", self._instroot + "/dev"]) > + > logging.debug("Grub installed.") > > def _install_grub2(self): > > > _______________________________________________ > Thincrust-devel mailing list > Thincrust-devel at redhat.com > https://www.redhat.com/mailman/listinfo/thincrust-devel > From mgoldman at redhat.com Wed Aug 8 07:05:01 2012 From: mgoldman at redhat.com (Marek Goldmann) Date: Wed, 08 Aug 2012 09:05:01 +0200 Subject: [Thincrust-devel] [PATCH 1/3] Remove extended partition from kpartx output to avoid partition count mismatch and erroneous device assignment In-Reply-To: <501BDF41.7060106@redhat.com> References: <501BDF41.7060106@redhat.com> Message-ID: <50220F9D.7050503@redhat.com> ACK On 03.08.2012 16:25, Marc Savy wrote: > With msdos type layouts that have large numbers of partitions, the > output from kpartx displays the extended partition, which is incorrectly > interpreted as meaning an extra partition is present which aught not be. > > This patch pops the line from the kpartx output so that it may be safely > ignored. From mgoldman at redhat.com Wed Aug 8 07:05:10 2012 From: mgoldman at redhat.com (Marek Goldmann) Date: Wed, 08 Aug 2012 09:05:10 +0200 Subject: [Thincrust-devel] [PATCH] Install grub in chroot to ensure correct ld-linux libs used. In-Reply-To: <50220D82.90802@redhat.com> References: <501BC68B.5030505@redhat.com> <501BDD2F.2080709@redhat.com> <50220D82.90802@redhat.com> Message-ID: <50220FA6.1040900@redhat.com> Please remove also whitespace after the chroot command: ..."--batch", "--no-floppy"], ^ here --Marek On 08.08.2012 08:56, Marek Goldmann wrote: > Hi Marc, > > This patch looks good for me. One question - why do you umount /dev > twice if rc != 0? > > --Marek > > On 03.08.2012 16:16, Marc Savy wrote: >> Install Grub using chroot so that it uses the ld-linux.so downloaded for >> the *appliance* rather than using the *host's* ld-linux.so which may be >> invalid (the wrong arch type, etc). >> >> From ce06bb9c95be57b4a3b4c88442e911d2404e5cee Mon Sep 17 00:00:00 2001 >> From: Marc Savy >> Date: Tue, 31 Jul 2012 16:03:18 +0100 >> Subject: [PATCH] Install grub in chroot to ensure correct ld-linux libs >> used. >> To: thincrust-devel at redhat.com >> >> --- >> appcreate/appliance.py | 9 +++++++-- >> 1 file changed, 7 insertions(+), 2 deletions(-) >> >> diff --git a/appcreate/appliance.py b/appcreate/appliance.py >> index f9fc784..1a13182 100644 >> --- a/appcreate/appliance.py >> +++ b/appcreate/appliance.py >> @@ -287,7 +287,7 @@ class ApplianceImageCreator(ImageCreator): >> # Ensure all data is flushed to disk before doing grub install >> subprocess.call(["sync"]) >> >> - stage2 = self._instroot + "/boot/grub/stage2" >> + stage2 = "/boot/grub/stage2" >> setup = "" >> >> i = 0 >> @@ -301,15 +301,20 @@ class ApplianceImageCreator(ImageCreator): >> >> logging.debug("Installing grub to %s" % loopdev) >> >> - grub = subprocess.Popen([self._instroot + "/sbin/grub", >> "--batch", "--no-floppy"], >> + subprocess.call(["mount", "--bind", "/dev", self._instroot + >> "/dev"]) >> + >> + grub = subprocess.Popen(["chroot", self._instroot, >> "/sbin/grub", "--batch", "--no-floppy"], >> stdin=subprocess.PIPE) >> >> grub.communicate(setup) >> rc = grub.wait() >> >> if rc != 0: >> + subprocess.call(["umount", self._instroot + "/dev"]) >> raise MountError("Unable to install grub bootloader") >> >> + subprocess.call(["umount", self._instroot + "/dev"]) >> + >> logging.debug("Grub installed.") >> >> def _install_grub2(self): >> >> >> _______________________________________________ >> Thincrust-devel mailing list >> Thincrust-devel at redhat.com >> https://www.redhat.com/mailman/listinfo/thincrust-devel >> > > _______________________________________________ > Thincrust-devel mailing list > Thincrust-devel at redhat.com > https://www.redhat.com/mailman/listinfo/thincrust-devel From mgoldman at redhat.com Wed Aug 8 07:05:52 2012 From: mgoldman at redhat.com (Marek Goldmann) Date: Wed, 08 Aug 2012 09:05:52 +0200 Subject: [Thincrust-devel] [PATCH 2/3] Ensure that more than 10 devices can be assigned symlink, mappings that do not overlap. In-Reply-To: <501BDF68.7000809@redhat.com> References: <501BDF68.7000809@redhat.com> Message-ID: <50220FD0.5010308@redhat.com> ACK On 03.08.2012 16:25, Marc Savy wrote: > Previously this behaviour was exhibited when over 10 devices are present: > https://gist.github.com/3eda2490dde8c49e26b1 > > This patch ensures that the entire device number is matched. From mgoldman at redhat.com Wed Aug 8 07:07:37 2012 From: mgoldman at redhat.com (Marek Goldmann) Date: Wed, 08 Aug 2012 09:07:37 +0200 Subject: [Thincrust-devel] [PATCH 3/3] Ensure loop device symlinks have not been left behind, from untidy termination before assigning them. In-Reply-To: <501BDF75.1010305@redhat.com> References: <501BDF75.1010305@redhat.com> Message-ID: <50221039.8090109@redhat.com> ACK, I'll remove the whitespace issue here myself. --Marek On 03.08.2012 16:25, Marc Savy wrote: > It is possible in certain termination situations for the program to be > unable to tidy up the symlinks it has created, causing collisions on the > next run. This patch is an attempt to resolve that.