Repository wisdom (not lookin' fer howtos)

[Rodolfo J. Paiz]

On Thu, 2004-09-02 at 01:55, Chris Hewitt wrote:
> On Wed, 2004-09-01 at 23:58, Rodolfo J. Paiz wrote:
> > On Wed, 2004-08-18 at 13:30, Thom Paine wrote:
> > > On Wed, 2004-08-18 at 14:14, Rodolfo J. Paiz wrote:
> > > 
> > > > I used to run a mirror and I kept the whole tree on my disk at home, so
> > > > I have about 60GB of the Red Hat FTP tree local. <grin> I can share my
> > > > rsync script and my include/exclude files, if you like. No secret.
> > > > 
> I wonder if you would be kind enough to send me a copy too, please.

Here you go, files attached. Sent to the list as well since the file
sizes are so darn small.

The actual script is "mirror" and is called with one parameter: which
repository you want to mirror. So, "mirror redhat" and "mirror fedora"
are your two current options. Of course, since these take a while to run
I usually do "./mirror fedora &" so I can get on with my life.

In order to reduce load on the mirror servers, I run these once a day.
Then, in order to "give a little back" I configure all of my office
machines and some of my more clueful friends' boxes to hit this server
instead of other mirror servers, so my box at least reduces general
demand on the public servers by 25 machines or so. Makes me happy.

Note that my exclude files are carefully tailored to the stuff *I* want.
Customize heavily before using, and test the file lists you get with the
"-n" (dry-run) parameter to ensure that you will rsync what you want.
One mistake can mean an additional few *gigabytes* of downloading, which
is not what you want and is not nice on the public servers either. For
example, I do not download any languages or architectures I don't use, I
do not mirror test versions, and I only keep a select few older versions
online.

"mirror all" currently does not work as desired... don't know why,
haven't spent the time to debug it yet.

I strongly recommend that you spend some quality time with the rsync man
page. There is a *ton* of useful stuff in there but you have to dig
somewhat. And careful for one common mistake: --bw-limit used BYTES not
bits, so to limit traffic to 64Kbps you're going to want --bw-limit=8.

Place these files in their own subdirectory (anywhere... I have them in
my home dir) since each run will generate a report file. I allow for one
report daily to avoid overcrowding; running a second mirror job on the
same day will overwrite the previous file.

It should be trivial to modify this for other rsync-based mirror sites,
but yell if you want any help.

Cheers!

-- 
Rodolfo J. Paiz <rpaiz at simpaticus.com>
Simpaticus.com
-------------- next part --------------
# Top-level directories in /pub/fedora/linux/core. Basically, exclude
# everything except "1" and "2" for the moment.
debug/
development/
SRPMS/
test/
testing/
x86_64/

# Exclude the DVD and SRPM ISO images.
FC2-i386-DVD.iso
-------------- next part --------------
# Top-level directories in /pub/redhat. Basically, exclude everything
# except "linux" and "mirror-tools" for the moment.
ccm/
ccvs/
gnupro/
interchange/
rhdb/
stronghold/
support/
tux/

# Versions of Red Hat for which I don't want any files at all.
1.0/
1.1/
2.0/
2.1/
3.0.3/
4.0/
4.1/
4.2/
5.0/
5.1/
5.2/
6.0/
6.1/
7.0/
7.1/
7.2/
beta/
code/
enterprise/
preview/
rawhide/

# Updates for other products.
updates/other_prod/

# Languages I don't speak or don't want.
de/
fr/
it/
ja/

# Architectures I don't want.
alpha/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mirror
Type: application/x-shellscript
Size: 923 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/fedora-list/attachments/20040906/bf9d9749/attachment-0001.bin>