[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

[PATCH] v3 turboLiveInst



Attached is an up to date revision of my turboLiveInst patch which incorporates the suggestions made during MarkMC's review.

Mark's executive summary of the feature:

  "Reduce installation time by not copying unused data to disk"

  "We avoid copying unused data by copying a filesystem image that has
   been reduced to the minimal possible size. This minimal image is not
   sufficient for a running LiveCD as applications need room to write
   more data, but the minimal image is efficiently created just before
   copying by applying a pre-calculated set of deltas to the original
   large filesystem image."

2nd revision post with performance numbers and decent descritpion:
(first two copy times are typo swapped)
http://www.redhat.com/archives/anaconda-devel-list/2007-July/msg00065.html

Mark's comments thread:
http://www.redhat.com/archives/fedora-livecd-list/2007-September/msg00007.html

The anaconda patch applies on top of the selinux bugfix I sent to anaconda-devel this morning, but there is no overlap.

The main things I'll mention that aren't directly related to the review thread above-

- I included a couple typo fixes to the documentation, as well as adding what seemed to be some appropriate additions.

- I included a slight cleanup of livecd-creator's resize2fsToMinimal. As discussed in the above thread, since the dumpe2fs code is going into anaconda for this patch, it seems to make sense to go ahead and include it in livecd-creator as well to remove the blocksize argument to resize2fsToMinimal, and have it calculate implicitly instead.

- I had to move the anaconda resize2fs invocation, as now the filesystem starts out full, and before resize2fs cannot even have mountpoint directories made in it. Even outside turboLiveInst this is a valid thing to do.

- renamed the main function from turboLiveInst to genMinInstDelta

- removed option for ignore-deleted, and didn't bother with option for turboliveinst. The liveinst.sh code gracefully handles the legacy configuration of no delta file.

- still not yet being a rawhide user, I tested this on an F7 livecd spin, generated with this git livecd-tools. I used the patch to patch the installed anaconda. It worked in a test under qemu. (sparse disk file with du -cms showed that only ~2.2G of data was written).

- I expect, that even just looking at it myself over the next couple days I'll probably find one or two things to change. Please give me any feedback about any possible improvements you can think of. If you have a rawhide livecd spinning enironment handy and want to test it, that would be greatly appreciated.

Enjoy...

-dmc/jdog

diff -Naur livecd.200709140049/creator/isotostick.sh livecd.turboliveinst/creator/isotostick.sh
--- livecd.200709140049/creator/isotostick.sh	2007-09-14 00:49:42.000000000 -0500
+++ livecd.turboliveinst/creator/isotostick.sh	2007-09-14 11:00:12.000000000 -0500
@@ -195,6 +195,10 @@
 elif [ -f $CDMNT/ext3fs.img ]; then
     cp $CDMNT/ext3fs.img $USBMNT/LiveOS/ext3fs.img || exitclean 
 fi
+if [ -f $CDMNT/osmin.gz ]; then
+    cp $CDMNT/osmin.gz $USBMNT/LiveOS/osmin.gz || exitclean
+fi
+
 cp $CDMNT/isolinux/* $USBMNT/$SYSLINUXPATH
 
 echo "Updating boot config file"
diff -Naur livecd.200709140049/creator/livecd-creator livecd.turboliveinst/creator/livecd-creator
--- livecd.200709140049/creator/livecd-creator	2007-09-14 00:49:42.000000000 -0500
+++ livecd.turboliveinst/creator/livecd-creator	2007-09-14 12:40:21.000000000 -0500
@@ -343,6 +343,7 @@
 
         self.image_size = 4096 # in megabytes
         self.blocksize = 4096 # in kilobytes
+        self.minimized_image_size = 0 # in kilobytes
 
     def _getRequiredPackages(self):
         return []
@@ -930,6 +931,21 @@
             shutil.move("%s/data/os.img" %(self.build_dir,),
                         "%s/out/ext3fs.img" %(self.build_dir,))
 
+    def parseField(self, output, field):
+        for line in output.split("\n"):
+            if line.startswith(field + ":"):
+                return line[len(field) + 1:].strip()
+        
+        raise KeyError("Failed to find field '%s' in output" % field)
+
+    def getBlockCountOfExt2FS(self, filesystem):
+        output = subprocess.Popen(['/sbin/dumpe2fs', '-h', filesystem],
+                                  stdout=subprocess.PIPE,
+                                  stderr=open('/dev/null', 'w')
+                                  ).communicate()[0]
+        
+        return int(self.parseField(output, "Block count"))
+
     def resize2fs(self, image, n_blocks):
         dev_null = os.open("/dev/null", os.O_WRONLY)
         try:
@@ -943,9 +959,9 @@
     # resize2fs doesn't have any kind of minimal setting, so use
     # a binary search to get it to minimal size.
     #
-    def resize2fsToMinimal(self, image, n_blocks):
+    def resize2fsToMinimal(self, image):
         bot = 0
-        top = n_blocks
+        top = self.getBlockCountOfExt2FS(image)
         while top != (bot + 1):
             t = bot + ((top - bot) / 2)
 
@@ -968,17 +984,95 @@
 
         n_blocks = os.stat(image)[stat.ST_SIZE] / self.blocksize
 
-        min_blocks = self.resize2fsToMinimal(image, n_blocks)
+        min_blocks = self.resize2fsToMinimal(image)
 
         # truncate the unused excess portion of the sparse file
         fd = os.open(image, os.O_WRONLY )
         os.ftruncate(fd, min_blocks * self.blocksize)
         os.close(fd)
 
-        print >> sys.stderr, "Installation target minimized to %dK" % (min_blocks * self.blocksize / 1024L)
+        self.minimized_image_size = min_blocks * self.blocksize / 1024L
+        print >> sys.stderr, "Installation target minimized to %dK" % (self.minimized_image_size)
 
         self.resize2fs(image, n_blocks)
 
+
+    #
+    # genMinInstDelta: generates an osmin overlay file to sit alongside
+    #                  os.img.  liveinst may then detect the existence of
+    #                  osmin, and use it to create a minimized os.img
+    #                  which can be installed more quickly, and to smaller
+    #                  destination volumes.
+    #
+    def genMinInstDelta(self):
+        # create the sparse file for the minimized overlay
+        fd = os.open("%s/out/osmin" %(self.build_dir,),
+                     os.O_WRONLY | os.O_CREAT)
+        off = long(64L * 1024L * 1024L)
+        os.lseek(fd, off, 0)
+        os.write(fd, '\x00')
+        os.close(fd)
+        
+        # associate os image with loop device
+        osloop = LoopbackMount("%s/data/os.img" %(self.build_dir,),
+                               "not_going_to_actually_get_mounted")
+        osloop.loopsetup()
+        
+        # associate overlay with loop device
+        minloop = LoopbackMount("%s/out/osmin" %(self.build_dir,),
+                                "not_going_to_actually_get_mounted")
+        minloop.loopsetup()
+        
+        # create a snapshot device
+        rc = subprocess.call(["/sbin/dmsetup",
+                              "--table",
+                              "0 %d snapshot %s %s p 8"
+                              %(self.image_size * 1024L * 2L,
+                                osloop.loopdev, minloop.loopdev),
+                              "create",
+                              "livecd-creator-%d" %(os.getpid(),) ])
+        if rc != 0:
+            raise InstallationError("Could not create genMinInstDelta snapshot device")
+        # resize snapshot device back to minimal (self.minimized_image_size)
+        rc = subprocess.call(["/sbin/resize2fs",
+                              "/dev/mapper/livecd-creator-%d" %(os.getpid(),),
+                              "%dK" %(self.minimized_image_size,)])
+        if rc != 0:
+            raise InstallationError("Could not shrink ext3fs image")
+
+        # calculate how much delta data to keep
+        dmsetupOutput = subprocess.Popen(['/sbin/dmsetup', 'status',
+                                          "livecd-creator-%d" %(os.getpid(),)],
+                                         stdout=subprocess.PIPE,
+                                         stderr=open('/dev/null', 'w')
+                                         ).communicate()[0]
+
+        try:
+            minInstDeltaDataLength = int((dmsetupOutput.split()[3]).split('/')[0])
+            print >> sys.stderr, "genMinInstDelta data length is %d 512 byte sectors" % (minInstDeltaDataLength)
+        except ValueError:
+            raise InstallationError("Could not calculate amount of data used by genMinInstDelta")            
+        
+        # tear down snapshot and loop devices
+        rc = subprocess.call(["/sbin/dmsetup", "remove",
+                              "livecd-creator-%d" %(os.getpid(),) ])
+        if rc != 0:
+            raise InstallationError("Could not remove genMinInstDelta snapshot device")
+        osloop.lounsetup()
+        minloop.lounsetup()
+
+        # truncate the unused excess portion of the sparse file
+        fd = os.open("%s/out/osmin" %(self.build_dir,), os.O_WRONLY )
+        os.ftruncate(fd, minInstDeltaDataLength * 512)
+        os.close(fd)
+
+        # the delta data is *extremely* compressible (e.g. 1.2M->7kb)
+        rc = subprocess.call(["/usr/bin/gzip", "osmin"],
+                             cwd="%s/out" %(self.build_dir,),
+                             env={"PWD": "%s/out" %(self.build_dir,)})
+        if rc != 0:
+            raise InstallationError("Could not compress genMinInstDelta data")
+
     def package(self):
         self.createSquashFS()
         self.createIso()
@@ -1300,10 +1394,6 @@
     # Don't compress the image.
     parser.add_option("-s", "--skip-compression", action="store_true", dest="skip_compression",
                       help=optparse.SUPPRESS_HELP)
-    # Don't run resize2fs to clean up wasted blocks
-    parser.add_option("", "--ignore-deleted", action="store_true", dest="ignore_deleted",
-                      help=optparse.SUPPRESS_HELP)
-
 
     (options, args) = parser.parse_args()
     if not options.kscfg or not os.path.isfile(options.kscfg):
@@ -1374,10 +1464,12 @@
 
         target.unmount()
 
-        if not options.ignore_deleted:
-            target.cleanupDeleted()
+        target.cleanupDeleted()
+
+        target.genMinInstDelta()
 
         target.package()
+
     except InstallationError, e:
         print >> sys.stderr, "Error creating Live CD : %s" % e
         target.teardown()
diff -Naur livecd.200709140049/creator/mayflower livecd.turboliveinst/creator/mayflower
--- livecd.200709140049/creator/mayflower	2007-09-14 00:49:42.000000000 -0500
+++ livecd.turboliveinst/creator/mayflower	2007-09-14 13:28:20.000000000 -0500
@@ -625,6 +625,24 @@
     mount -n -o ro,remount /sysroot
 }
 
+modprobe loop max_loop=128
+
+# we might have a genMinInstDelta delta file for anaconda to take advantage of
+if [ -e /sysroot/LiveOS/osmin.gz ]; then
+  mknod /dev/loop118 b 7 118
+  # note: osmin.gz should typically only be about 7kb. 
+  dd if=/sysroot/LiveOS/osmin.gz of=/osmin.gz bs=512 2> /dev/null
+  # pad to at least next sector boundry
+  dd if=/dev/zero of=/osmin.gz bs=512 count=1 oflag=append conv=notrunc 2> /dev/null
+  losetup /dev/loop118 /osmin.gz
+elif [ -e /sysroot/osmin.gz ] ; then
+  mknod /dev/loop118 b 7 118
+  dd if=/sysroot/osmin.gz of=/osmin.gz bs=512 2> /dev/null
+  # pad to at least next sector boundry
+  dd if=/dev/zero of=/osmin.gz bs=512 count=1 oflag=append conv=notrunc 2> /dev/null
+  losetup /dev/loop118 /osmin.gz
+fi
+
 # we might have an uncompressed embedded ext3  to use as rootfs (uncompressed live)
 #
 if [ -e /sysroot/LiveOS/ext3fs.img ]; then
@@ -638,13 +656,11 @@
         echo "setting up embedded ext3 fs "
     fi
 
-    mknod /dev/loop118 b 7 118
     mknod /dev/loop119 b 7 119
     mknod /dev/loop120 b 7 120
     mknod /dev/loop121 b 7 121
     mkdir -p /dev/mapper
     mknod /dev/mapper/control c 10 63
-    modprobe loop max_loop=128
     modprobe dm_snapshot
 
     losetup /dev/loop121 \$EXT3FS
@@ -667,13 +683,11 @@
         echo "setting up embedded squash -> ext3 fs "
     fi
 
-    mknod /dev/loop118 b 7 118
     mknod /dev/loop119 b 7 119
     mknod /dev/loop120 b 7 120
     mknod /dev/loop121 b 7 121
     mkdir -p /dev/mapper
     mknod /dev/mapper/control c 10 63
-    modprobe loop max_loop=128
     modprobe dm_snapshot
 
     if [ "\$live_ram" == "1" ] ; then
diff -Naur livecd.200709140049/README livecd.turboliveinst/README
--- livecd.200709140049/README	2007-09-14 00:49:42.000000000 -0500
+++ livecd.turboliveinst/README	2007-09-14 12:13:34.000000000 -0500
@@ -74,6 +74,13 @@
 
  o Unmounts the installation root
 
+ o Runs resize2fs to minimize and unminimize the ext3 file to remove data
+   from deleted files
+
+ o Runs resize2fs to minimize on a devicemapper snapshot, to generate a 
+   small minimized delta image file which can be used by anaconda to 
+   reduce installation time by not copying unused data to disk
+
  o Creates a squashfs file system containing only the ext3 file (compression)
 
  o Configures the boot loader
@@ -118,7 +125,7 @@
   --fslabel=Fedora-7-LiveCD-1-foo
 
 will create a live CD called "Fedora-7-LiveCD-1-foo". The name
-given by --fs-label is used. 
+given by --fslabel is used. 
 
  o as a file system label on the ext3 and iso960 file systems
    (as such it's visible on the desktop as the CD name)
@@ -155,7 +162,7 @@
 use live images.  You can take a live CD iso image and transform it so
 that it can be used on a USB stick.  To do so, use the
 livecd-iso-to-disk script.
-   /usr/bin/livecd-iso-to-stick /path/to/live.iso /dev/sdb1 
+   /usr/bin/livecd-iso-to-disk /path/to/live.iso /dev/sdb1 
 
 Replace /dev/sdb1 with the (unmounted) partition where you wish to put
 the live image.  This is not a destructive process; any data you 
diff -Naur anaconda.200709140049.bugfix_selinux/livecd.py anaconda.turboliveinst/livecd.py
--- anaconda.200709140049.bugfix_selinux/livecd.py	2007-09-14 12:41:41.000000000 -0500
+++ anaconda.turboliveinst/livecd.py	2007-09-14 12:44:17.000000000 -0500
@@ -133,25 +133,24 @@
     def getLiveBlockDevice(self):
         return self.osimg
 
-    def getLiveSizeMB(self):
-        lnk = os.readlink(self.osimg)
-        if lnk[0] != "/":
-            lnk = os.path.join(os.path.dirname(self.osimg), lnk)
-        blk = os.path.basename(lnk)
-
-        if not os.path.exists("/sys/block/%s/size" %(blk,)):
-            log.debug("Unable to determine the actual size of the live image")
-            return 0
-
-        size = open("/sys/block/%s/size" %(blk,), "r").read()
-        try:
-            size = int(size)
-        except ValueError:
-            log.debug("Unable to handle live size conversion: %s" %(size,))
-            return 0
+    def parseField(self, output, field):
+        for line in output.split("\n"):
+            if line.startswith(field + ":"):
+                return line[len(field) + 1:].strip()
+        
+        raise KeyError("Failed to find field '%s' in output" % field)
 
-        return (size * 512) / 1024 / 1024
+    def getSectorCountOfExt2FS(self, filesystem):
+        output = subprocess.Popen(['/sbin/dumpe2fs', '-h', filesystem],
+                                  stdout=subprocess.PIPE,
+                                  stderr=open('/dev/null', 'w')
+                                  ).communicate()[0]
         
+        return (int(self.parseField(output, "Block count")) *
+                int(self.parseField(output, "Block size")) / 512)
+
+    def getLiveSizeMB(self):
+        return (self.getSectorCountOfExt2FS(self.osimg) * 512) / 1024 / 1024
 
 class LiveCDCopyBackend(backend.AnacondaBackend):
     def __init__(self, method, instPath):
@@ -215,6 +214,9 @@
         wait = anaconda.intf.waitWindow(_("Doing post-installation"),
                                         _("Performing post-installation filesystem changes.  This may take several minutes..."))
 
+        # resize rootfs first, since it is 100% full due to genMinInstDelta
+        self._resizeRootfs(anaconda, wait)
+
         # remount filesystems
         anaconda.id.fsset.mountFilesystems(anaconda)
 
@@ -291,7 +293,6 @@
                 log.error("error mounting selinuxfs: %s" %(e,))
         isys.mount("/dev", "%s/dev" %(anaconda.rootPath,), bindMount = 1)
 
-        self._resizeRootfs(anaconda, wait)
         wait.pop()
 
     def _resizeRootfs(self, anaconda, win = None):
diff -Naur anaconda.200709140049.bugfix_selinux/liveinst/liveinst.sh anaconda.turboliveinst/liveinst/liveinst.sh
--- anaconda.200709140049.bugfix_selinux/liveinst/liveinst.sh	2007-04-04 13:05:42.000000000 -0500
+++ anaconda.turboliveinst/liveinst/liveinst.sh	2007-09-14 12:20:13.000000000 -0500
@@ -4,7 +4,29 @@
 #
 
 if [ -z "$LIVE_BLOCK" ]; then
-    LIVE_BLOCK="/dev/live-osimg"
+    #
+    # We avoid copying unused data by copying a filesystem image that has 
+    # been reduced to the minimal possible size. This minimal image is not 
+    # sufficient for a running LiveCD as applications need room to write 
+    # more data, but the minimal image is efficiently created just before 
+    # copying by applying a pre-calculated set of deltas to the original 
+    # large filesystem image.
+    #
+
+    # did mayflower find and expose the delta data via loop118?
+    if ( losetup /dev/loop118 > /dev/null 2>&1 ); then
+	# the delta data is exposed as gzipped data in a loop device
+	zcat /dev/loop118 > /dev/shm/osmin.img
+	# devicemapper needs a loop device with the uncompressed data
+	losetup /dev/loop117 /dev/shm/osmin.img
+	# set up the devicemapper snapshot device, which will merge
+	# the normal live fs image, and the delta, into a minimzied fs image
+	echo "0 $( blockdev --getsize /dev/loop121 ) snapshot /dev/loop121 /dev/loop117 p 8" | dmsetup create --readonly live-osimg-min
+	LIVE_BLOCK="/dev/mapper/live-osimg-min"
+    else
+	# fall back to legacy pre-genMinInstDelta behavior
+	LIVE_BLOCK="/dev/live-osimg"
+    fi
 fi
 
 if [ ! -b $LIVE_BLOCK ]; then
@@ -42,3 +64,10 @@
 if [ -n $current ]; then
     /usr/sbin/setenforce $current
 fi
+
+# cleanup genMinInstDelta stuff if needed
+if ( losetup /dev/loop118 > /dev/null 2>&1 ); then
+    dmsetup remove live-osimg-min
+    losetup -d /dev/loop117
+    rm -f /dev/shm/osmin.img
+fi

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]