Using jigdo to save on downloads

Chuck R. Anderson cra at WPI.EDU
Mon Apr 11 15:04:31 UTC 2005 

On Mon, Apr 11, 2005 at 03:13:17AM -0400, Chuck R. Anderson wrote:
> On Mon, Apr 11, 2005 at 12:45:17AM -0400, seth vidal wrote:
> > I wanna see what sort of bandwidth saving is possible. Rather than just
> > take your word and jigdo's website word for it.
> 
> Well, for the mirrors, the bandwidth savings would be pretty obvious
> since they would only need to downloaded the distro once, not thrice
> like many of them do now (tree + CDs + DVD).
> 
> > Let's get some numbers, shall we?
> 
> More interesting would be rsyncing the tree from rawhide -> test ->
> final release, then using master-server-provided jigdo files to create
> the CD and DVD ISOs. I agree that getting numbers for this would be
> worthwhile.  I'll see what I can do here.

FC4T2 size:

test/3.91/i386/
test/3.91/SRPMS/

Total 15 GiB

For anyone that maintains a rawhide mirror already (most mirrors),
going to FC4T2 you only need to download this much:

183 MiB

That includes 1.1 MiB of jigdo/template files.

You can find the jigdo/template files I generated here.  Feel free to
edit the .jigdo files themselves and change/add more lines to the
[Servers] sections.  Right now I left them pointing at the master
server.  Or you can specify a local file path during reassembly
(below).

http://download.wpi.edu/pub/jigdo/

For jigdo itself, the packages which I intend to submit to Extras are
here (built/tested on FC3):

http://angus.ind.wpi.edu/~cra/fedora/extras/jigdo/

To recreate the original CD/DVD ISO images from your file tree:

jigdo-lite http://download.wpi.edu/pub/jigdo/FC4-test2-i386-disc1.jigdo

Jigsaw Download "lite"
Copyright (C) 2001-2004  |  jigdo@
Richard Atterer          |  atterer.net

Downloading .jigdo file
--10:56:26--  http://download.wpi.edu/pub/jigdo/FC4-test2-i386-disc1.jigdo
           => `FC4-test2-i386-disc1.jigdo'
Resolving download.wpi.edu... 130.215.32.86
Connecting to download.wpi.edu[130.215.32.86]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 60,698 [text/plain]

    0K                                                      100%  165.39 MB/s

10:56:26 (165.39 MB/s) - `FC4-test2-i386-disc1.jigdo' saved [60,698/60,698]


-----------------------------------------------------------------
Images offered by `http://download.wpi.edu/pub/jigdo/FC4-test2-i386-disc1.jigdo':
  1: FC4-test2-i386-disc1.iso

-----------------------------------------------------------------
If you already have a previous version of the CD you are
downloading, jigdo can re-use files on the old CD that are also
present in the new image, and you do not need to download them
again. Mount the old CD ROM and enter the path it is mounted under
(e.g. `/mnt/cdrom').
Alternatively, just press enter if you want to start downloading
the remaining files.
Files to scan: /srv/ftp/pub/fedora/linux/core/test/3.91/i386/os

Downloading .template file
--10:56:38--  http://download.wpi.edu/pub/jigdo/FC4-test2-i386-disc1.template
           => `FC4-test2-i386-disc1.template'
Resolving download.wpi.edu... 130.215.32.86
Connecting to download.wpi.edu[130.215.32.86]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 129,022 [text/plain]

    0K .                                                    100%    5.16 MB/s

10:56:38 (5.16 MB/s) - `FC4-test2-i386-disc1.template' saved [129,022/129,022]

general:        Image file: 
general:        Jigdo:      
general:        Template:   
general:        [exit(0)]
general:        Image file: FC4-test2-i386-disc1.iso
general:        Jigdo:      FC4-test2-i386-disc1.jigdo
general:        Template:   FC4-test2-i386-disc1.template
100%       0k/0k       scanning
Successfully created `FC4-test2-i386-disc1.iso'
general:        [exit(0)]

-----------------------------------------------------------------
Finished!
The fact that you got this far is a strong indication that `FC4-test2-i386-disc1.iso'
was generated correctly. I will perform an additional, final check,
which you can interrupt safely with Ctrl-C if you do not want to wait.

general:        Image file: FC4-test2-i386-disc1.iso
general:        Jigdo:      FC4-test2-i386-disc1.jigdo
general:        Template:   FC4-test2-i386-disc1.template
  0%     256k/633206k  verifying image
general:        [exit(0)]
26.82user 9.41system 1:32.50elapsed 39%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (3major+10954minor)pagefaults 0swaps

(repeat for the other images)

Here is the procedure I used and the timings of the runs:

cd test/3.91/SRPMS/
cp -avx ../../../development/SRPMS/ .
cd ../i386/os/
cp -avx ../../../development/i386/ .
cd ../debug/
mv ../os/debug .

cd test/3.91/SRPMS/
/usr/bin/time rsync -av --delete-after \
	rsync://[your-neighborhood-friendly-mirror]/test/3.91/SRPMS/ .

receiving file list ... done
./
comps-3.91-0.20050407.src.rpm
fedora-release-3.91-1.src.rpm
deleting [a bunch of obsolete rawhide files]

sent 148 bytes  received 103420 bytes  3395.67 bytes/sec
total size is 2605906175  speedup is 25161.31
0.03user 1.56system 0:29.52elapsed 5%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+490minor)pagefaults 0swaps


cd test/3.91/i386/debug/
/usr/bin/time rsync -av --delete-after \
	rsync://[your-neighborhood-friendly--mirror]/test/3.91/i386/debug/ .

receiving file list ... done
./
headers/
repodata/
fedora-release-debuginfo-3.91-1.i386.rpm
headers/[a bunch of hdr files] 
repodata/filelists.xml.gz
repodata/other.xml.gz
repodata/primary.xml.gz
repodata/repomd.xml
deleting [a bunch of obsolete rawhide files]

sent 15153 bytes  received 9247918 bytes  173141.51 bytes/sec
total size is 1885583549  speedup is 203.56
0.11user 2.96system 0:52.33elapsed 5%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+574minor)pagefaults 0swaps


cd test/3.91/i386/os/
/usr/bin/time rsync -av --delete-after \
	rsync://[your-neighborhood-friendly-mirror]/test/3.91/i386/os/ .

receiving file list ... done
./
Fedora/
Fedora/RPMS/
Fedora/base/
.discinfo
Fedora/RPMS/comps-3.91-0.20050407.i386.rpm
Fedora/RPMS/fedora-release-3.91-1.i386.rpm
Fedora/base/comps.rpm
Fedora/base/comps.xml
Fedora/base/hdlist
Fedora/base/hdlist2
Fedora/base/hdstg2.img
Fedora/base/netstg2.img
SRPMS -> ../../SRPMS
headers/
images/
images/pxeboot/
Fedora/base/stage2.img
isolinux/
repodata/
GPL
README-Accessibility
README-en
RELEASE-NOTES-en
RPM-GPG-KEY
RPM-GPG-KEY-beta
RPM-GPG-KEY-fedora
RPM-GPG-KEY-fedora-rawhide
RPM-GPG-KEY-fedora-test
RPM-GPG-KEY-rawhide
autorun
eula.txt
headers/comps-1-3.91-0.20050407.i386.hdr
headers/comps-1-3.91-0.20050407.src.hdr
headers/fedora-release-0-3.91-1.i386.hdr
headers/fedora-release-0-3.91-1.src.hdr
headers/header.info
headers/header.src.info
images/README
images/boot.iso
images/diskboot.img
images/pxeboot/README
images/pxeboot/initrd.img
images/pxeboot/vmlinuz
images/rescuecd.iso
isolinux/boot.msg
isolinux/general.msg
isolinux/initrd.img
isolinux/isolinux.bin
isolinux/isolinux.cfg
isolinux/memtest
isolinux/options.msg
isolinux/param.msg
isolinux/rescue.msg
isolinux/snake.msg
isolinux/splash.lss
isolinux/vmlinuz
repodata/comps.xml
repodata/filelists.xml.gz
repodata/other.xml.gz
repodata/primary.xml.gz
repodata/repomd.xml
deleting yumgroups.xml
deleting i386/images/boot.iso
deleting i386/images/.listing
deleting directory i386/images
deleting directory i386
deleting [a bunch of obsolete rawhide files]

sent 264720 bytes  received 180311140 bytes  1059095.95 bytes/sec
total size is 2699311849  speedup is 14.95
3.54user 12.18system 2:50.71elapsed 9%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+2123minor)pagefaults 0swaps


/usr/bin/time jigdo-file make-template \
	--cache=/var/cache/jigdo-cache.db \
	--image=/srv/ftp/pub/fedora/linux/core/test/3.91/i386/iso/FC4-test2-i386-DVD.iso \
	--jigdo=FC4-test2-i386-DVD.jigdo \
	--template=FC4-test2-i386-DVD.template \
	--label fedora-linux-core=/srv/ftp/pub/fedora/linux/core/ \
	--uri fedora-linux-core=http://[mirror]/pub/fedora/linux/core/ \
	/srv/ftp/pub/fedora/linux/core//test/3.91/

general:        Image file: /srv/ftp/pub/fedora/linux/core/test/3.91/i386/iso/FC4-test2-i386-DVD.iso
general:        Jigdo:      FC4-test2-i386-DVD.jigdo
general:        Template:   FC4-test2-i386-DVD.template
general:        URI mapping: `fedora-linux-core' => `http://[mirror]/pub/fedora/linux/core/'
Finished - image size is 2657673216 bytes.
general:        [exit(0)]
1487.39user 18.08system 26:38.37elapsed 94%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (10major+2949minor)pagefaults 0swaps

/usr/bin/time jigdo-file make-template \
	--cache=/var/cache/jigdo-cache.db \
	--image=/srv/ftp/pub/fedora/linux/core/test/3.91/i386/iso/FC4-test2-i386-disc1.iso \
	--jigdo=FC4-test2-i386-disc1.jigdo \
	--template=FC4-test2-i386-disc1.template \
	--label fedora-linux-core=/srv/ftp/pub/fedora/linux/core/ \
	--uri fedora-linux-core=http://[mirror]/pub/fedora/linux/core/ \
	/srv/ftp/pub/fedora/linux/core//test/3.91/

general:        Image file: /srv/ftp/pub/fedora/linux/core/test/3.91/i386/iso/FC4-test2-i386-disc1.iso
general:        Jigdo:      FC4-test2-i386-disc1.jigdo
general:        Template:   FC4-test2-i386-disc1.template
general:        URI mapping: `fedora-linux-core' => `http://[mirror]/pub/fedora/linux/core/'
Finished - image size is 648402944 bytes.
general:        [exit(0)]
563.70user 1.80system 9:28.03elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (17major+3153minor)pagefaults 0swaps

/usr/bin/time jigdo-file make-template \
	--cache=/var/cache/jigdo-cache.db \
	--image=/srv/ftp/pub/fedora/linux/core/test/3.91/i386/iso/FC4-test2-i386-disc2.iso \
	--jigdo=FC4-test2-i386-disc2.jigdo \
	--template=FC4-test2-i386-disc2.template \
	--label fedora-linux-core=/srv/ftp/pub/fedora/linux/core/ \
	--uri fedora-linux-core=http://[mirror]/pub/fedora/linux/core/ \
	/srv/ftp/pub/fedora/linux/core//test/3.91/

general:        Image file: /srv/ftp/pub/fedora/linux/core/test/3.91/i386/iso/FC4-test2-i386-disc2.iso
general:        Jigdo:      FC4-test2-i386-disc2.jigdo
general:        Template:   FC4-test2-i386-disc2.template
general:        URI mapping: `fedora-linux-core' => `http://[mirror]/pub/fedora/linux/core/'
Finished - image size is 656263168 bytes.
general:        [exit(0)]
311.40user 1.58system 5:13.65elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+3034minor)pagefaults 0swaps

/usr/bin/time jigdo-file make-template \
	--cache=/var/cache/jigdo-cache.db \
	--image=/srv/ftp/pub/fedora/linux/core/test/3.91/i386/iso/FC4-test2-i386-disc3.iso \
	--jigdo=FC4-test2-i386-disc3.jigdo \
	--template=FC4-test2-i386-disc3.template \
	--label fedora-linux-core=/srv/ftp/pub/fedora/linux/core/ \
	--uri fedora-linux-core=http://[mirror]/pub/fedora/linux/core/ \
	/srv/ftp/pub/fedora/linux/core//test/3.91/

general:        Image file: /srv/ftp/pub/fedora/linux/core/test/3.91/i386/iso/FC4-test2-i386-disc3.iso
general:        Jigdo:      FC4-test2-i386-disc3.jigdo
general:        Template:   FC4-test2-i386-disc3.template
general:        URI mapping: `fedora-linux-core' => `http://[mirror]/pub/fedora/linux/core/'
Finished - image size is 669345792 bytes.
general:        [exit(0)]
260.72user 1.65system 4:23.26elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+3033minor)pagefaults 0swaps

/usr/bin/time jigdo-file make-template \
	--cache=/var/cache/jigdo-cache.db \
	--image=/srv/ftp/pub/fedora/linux/core/test/3.91/i386/iso/FC4-test2-i386-disc4.iso \
	--jigdo=FC4-test2-i386-disc4.jigdo \
	--template=FC4-test2-i386-disc4.template \
	--label fedora-linux-core=/srv/ftp/pub/fedora/linux/core/ \
	--uri fedora-linux-core=http://[mirror]/pub/fedora/linux/core/ \
	/srv/ftp/pub/fedora/linux/core//test/3.91/

general:        Image file: /srv/ftp/pub/fedora/linux/core/test/3.91/i386/iso/FC4-test2-i386-disc4.iso
general:        Jigdo:      FC4-test2-i386-disc4.jigdo
general:        Template:   FC4-test2-i386-disc4.template
general:        URI mapping: `fedora-linux-core' => `http://[mirror]/pub/fedora/linux/core/'
Finished - image size is 603430912 bytes.
general:        [exit(0)]
358.96user 1.64system 6:01.24elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+3037minor)pagefaults 0swaps

/usr/bin/time jigdo-file make-template \
	--cache=/var/cache/jigdo-cache.db \
	--image=/srv/ftp/pub/fedora/linux/core/test/3.91/i386/iso/FC4-test2-i386-SRPMS-disc1.iso \
	--jigdo=FC4-test2-i386-SRPMS-disc1.jigdo \
	--template=FC4-test2-i386-SRPMS-disc1.template \
	--label fedora-linux-core=/srv/ftp/pub/fedora/linux/core/ \
	--uri fedora-linux-core=http://[mirror]/pub/fedora/linux/core/ \
	/srv/ftp/pub/fedora/linux/core//test/3.91/

general:        Image file: /srv/ftp/pub/fedora/linux/core/test/3.91/i386/iso/FC4-test2-i386-SRPMS-disc1.iso
general:        Jigdo:      FC4-test2-i386-SRPMS-disc1.jigdo
general:        Template:   FC4-test2-i386-SRPMS-disc1.template
general:        URI mapping: `fedora-linux-core' => `http://[mirror]/pub/fedora/linux/core/'
Finished - image size is 650930176 bytes.
general:        [exit(0)]
210.17user 2.75system 3:44.35elapsed 94%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (2major+3096minor)pagefaults 0swaps

/usr/bin/time jigdo-file make-template \
	--cache=/var/cache/jigdo-cache.db \
	--image=/srv/ftp/pub/fedora/linux/core/test/3.91/i386/iso/FC4-test2-i386-SRPMS-disc2.iso \
	--jigdo=FC4-test2-i386-SRPMS-disc2.jigdo \
	--template=FC4-test2-i386-SRPMS-disc2.template \
	--label fedora-linux-core=/srv/ftp/pub/fedora/linux/core/ \
	--uri fedora-linux-core=http://[mirror]/pub/fedora/linux/core/ \
	/srv/ftp/pub/fedora/linux/core//test/3.91/

general:        Image file: /srv/ftp/pub/fedora/linux/core/test/3.91/i386/iso/FC4-test2-i386-SRPMS-disc2.iso
general:        Jigdo:      FC4-test2-i386-SRPMS-disc2.jigdo
general:        Template:   FC4-test2-i386-SRPMS-disc2.template
general:        URI mapping: `fedora-linux-core' => `http://[mirror]/pub/fedora/linux/core/'
Finished - image size is 650944512 bytes.
general:        [exit(0)]
219.23user 2.76system 3:53.31elapsed 95%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+3217minor)pagefaults 0swaps

/usr/bin/time jigdo-file make-template \
	--cache=/var/cache/jigdo-cache.db \
	--image=/srv/ftp/pub/fedora/linux/core/test/3.91/i386/iso/FC4-test2-i386-SRPMS-disc3.iso \
	--jigdo=FC4-test2-i386-SRPMS-disc3.jigdo \
	--template=FC4-test2-i386-SRPMS-disc3.template \
	--label fedora-linux-core=/srv/ftp/pub/fedora/linux/core/ \
	--uri fedora-linux-core=http://[mirror]/pub/fedora/linux/core/ \
	/srv/ftp/pub/fedora/linux/core//test/3.91/

general:        Image file: /srv/ftp/pub/fedora/linux/core/test/3.91/i386/iso/FC4-test2-i386-SRPMS-disc3.iso
general:        Jigdo:      FC4-test2-i386-SRPMS-disc3.jigdo
general:        Template:   FC4-test2-i386-SRPMS-disc3.template
general:        URI mapping: `fedora-linux-core' => `http://[mirror]/pub/fedora/linux/core/'
Finished - image size is 650930176 bytes.
general:        [exit(0)]
190.68user 2.77system 3:25.46elapsed 94%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+3180minor)pagefaults 0swaps

/usr/bin/time jigdo-file make-template \
	--cache=/var/cache/jigdo-cache.db \
	--image=/srv/ftp/pub/fedora/linux/core/test/3.91/i386/iso/FC4-test2-i386-SRPMS-disc4.iso \
	--jigdo=FC4-test2-i386-SRPMS-disc4.jigdo \
	--template=FC4-test2-i386-SRPMS-disc4.template \
	--label fedora-linux-core=/srv/ftp/pub/fedora/linux/core/ \
	--uri fedora-linux-core=http://[mirror]/pub/fedora/linux/core/ \
	/srv/ftp/pub/fedora/linux/core//test/3.91/

general:        Image file: /srv/ftp/pub/fedora/linux/core/test/3.91/i386/iso/FC4-test2-i386-SRPMS-disc4.iso
general:        Jigdo:      FC4-test2-i386-SRPMS-disc4.jigdo
general:        Template:   FC4-test2-i386-SRPMS-disc4.template
general:        URI mapping: `fedora-linux-core' => `http://[mirror]/pub/fedora/linux/core/'
Finished - image size is 650913792 bytes.
general:        [exit(0)]
217.25user 2.80system 3:52.51elapsed 94%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+3298minor)pagefaults 0swaps

More information about the fedora-test-list mailing list