Jigdo - A Professional Letter to Mike McGrath

[Jeroen van Meeuwen]

Mike McGrath wrote:
> Jonathan Steffan wrote:
>> The amount of storage and bandwidth able to be saved can be illustrated
>> by a simple comparison between the efficiency of chopping up a 3.4GB
>> iso9660 file system arbitrarily (by a static chunk size) and the same
>> file system based on contents (file by file.) For a BitTorrent,
>> Fedora's current choice for sharing Spins, the hosted data is only
>> valid for a given chunk on a single ISO. This data's footprint (equal
>> to the combined chunk sizes of the entire torrent) can be used for
>> nothing but this Spin. To be able to host 5 Spins composed from similar
>> trees via BitTorrent, we now have a footprint of 17GB, not to mention
>> "seeders" have to run BitTorrent software to be able to contribute to
>> the swarm. Alternatively, Jigdo can be used to reduce the footprint of
>> these 5 Spins to about 4GB. The amount of additional data needing to be
>> hosted for each Spin, in addition to what data is already pushed to the
>> mirrors, is about 150MB per ISO with anaconda and about 200KB for ISOs
>> without the installer bits. To help illustrate the efficiency of using
>> Jigdo vs BitTorrent, the footprint for 250 Spins is 850GB for
>> BitTorrent and about 40GB for Jigdo. Additionally, a reduction in
>> overhead can be achieved by removing the need for the BitTorrent
>> tracker and all related network traffic without requiring any
>> additional work on the part of mirror administrators.
>>   
> 
> My concern with jigdo is with how many people use it?  It seems silly to 
> host both torrent and jigdo (as much of this letter points out the 
> benefits of switching to jigdo, those benefits disappear if we simply 
> add jigdo to the mix.  Most people already have bittorrent.  Lets say we 
> were going to give Jigdo a trial run for Fedora 9

FYI, we have done so, and we are doing so officially for Fedora 9.

  and we were going to
> judge jigdo a success if a certain % (compared to bittorrent) use 
> jigdo.  What % would that be?
> 

Jigdo would in this case be particularly useful to those with a local 
mirror as they have 99% of the content already (90% if you have F9T3?). 
Because it is particularly useful to some, and completely weird and 
strange for others, the number of users that will use it if BitTorrent 
is an alternative wouldn't be a very good indicator to see if it is 
actually a viable distribution method for the whole of Fedora, neither 
is it the goal for these proposals.

However on the other hand we do have a couple of people with local 
mirrors, and last time I checked, test releases are downloaded a couple 
of times. We are hoping that these users in particular try out Jigdo and 
become happy bandwidth and time savers.

For the Fedora Project, the greatest benefit of doing a trial Jigdo 
release with Fedora 9 is to get to know the feeling, see the numbers, 
get some feedback, and not having to host 68GB of Everything spins on 
different media, instead of 450MB -giving the same results. The same 
goes for any other additional installation media composed off the 
release tree -even rebranded downstream distributions, although there 
isn't much of those as long as updates keep expiring from the mirrors. 
The original proposal was in fact that Fedora 9 CDs would be released 
and hosted by Fedora Project but it seemed to be a better path to do so 
via Release Engineering

> While google trends is sort of crazy in that I don't know what it says, 
> it does say something:
> 
> http://google.com/trends?q=bittorrent%2C+jigdo
> 

Right, I do hope our gut feeling rather then the number of hits on 
Google comes up with a decision...

Another consideration in the entire footprint discussion may be to 
expire FC{1,2,3,4,5,6} from the master mirror.

Kind regards,

Jeroen van Meeuwen
-kanarip