rsync and rawhide

[Mark Mielke]

On Fri, Oct 17, 2003 at 10:20:27AM -0400, Chris Ricker wrote:
> Now that the fedora process is opening up development a bit more, I suspect 
> more people are frequently downloading rawhide. Has there been any thought 
> about making it available usably over rsync?

I would prefer to use rsync to update my rpm set with rawhide simply because
it communicates the information more efficiently. Lately using 'lftp mirror'
I've noticed that packages sometimes download on top of themselves, perhaps
just because the timestamp of the file has changed? Even if the RPM had
changed (signed after the fact?), rsync would do a better job than ftp.

> The current drawback to using rsync is that what's posted to rawhide are
> packages, and package names change so rsync can't find the preexisting
> similar bits. Consider, just to pick one example, unixODBC. Yesterday,
> unixODBC-2.2.5-8 was pushed to rawhide. Today, unixODBC-2.2.5-9 was pushed.  
> The only difference between the two is that in -9 Fernando tweaked ~4 lines
> of text in /etc/odbcinst.ini, yet to get that I have to download the whole
> thing again b/c rsync sees them as different files and therefore doesn't
> compare them....

Note, though, that just because the change was only 4 lines of text, does not
mean that the whole RPM won't need to be downloaded. I was under the
impression that rsync communicates block checksums back and forth. How would
it handle the entire file from offset 4096 being shifted 512 bytes longer
or shorter? Would it not re-transfer the whole file?

Also, I think one could argue that rsync should have an option to deal with
this situation - file content not significantly changing, but file name
changing. At *least* for files in the same directory.

> One way to make rsync work with rawhide would be to autogenerate isos (not
> full working isos which can be booted off of and such, just the RPMs) on a
> daily basis, then to use the hardlinks script that the RH mirrors often use
> to explode the contents off of loopback-mounted isos. People could then just
> rsync ftp.redhat.com/pub/redhat/linux/rawhide/iso/daily.iso and save some 
> bandwidth on both sides....

Again, I suspect this has the same issues as I describe in my previous
paragraph, although the effect may be reduced -- which isn't to say it
would be useless, but, I wonder why rsync shouldn't be able to do a better
job with the raw data, than in a pre-packed iso form.

mark

-- 
mark at mielke.cc/markm at ncf.ca/markm at nortelnetworks.com __________________________
.  .  _  ._  . .   .__    .  . ._. .__ .   . . .__  | Neighbourhood Coder
|\/| |_| |_| |/    |_     |\/|  |  |_  |   |/  |_   | 
|  | | | | \ | \   |__ .  |  | .|. |__ |__ | \ |__  | Ottawa, Ontario, Canada

  One ring to rule them all, one ring to find them, one ring to bring them all
                       and in the darkness bind them...

                           http://mark.mielke.cc/