[Spacewalk-list] Need for a lightweight mirroring solution

[Mike McCune]

Michael DeHaan wrote:
> Covil, K (Kim) wrote:
>>> From: spacewalk-list-bounces at redhat.com [mailto:spacewalk-list-
>>> bounces at redhat.com] On Behalf Of Michael DeHaan
>>> Sent: 21 July 2008 13:38
>>> To: Jon Stanley
>>> Cc: spacewalk-list at redhat.com
>>> Subject: Re: [Spacewalk-list] Need for a lightweight mirroring solution
>>>
>>> Jon Stanley wrote:
>>>     
>>>> On Mon, Jul 21, 2008 at 8:05 AM, Michael DeHaan <mdehaan at redhat.com>
>>>>       
>>> wrote:
>>>     
>>>>> Many questions I get on cobbler's mailing list deal with using
>>>>> yum_rhn_plugin to mirror RHEL packages.   These are typically from
>>>>>         
>>> users who
>>>     
>>>>> have small-mid size environments that cannot use Satellite for
>>>>>         
>>> various
>>>     
>>>>> reasons.
>>>>>
>>>>>         
>>>> That;s why we have spacewalk now :)
>>>>
>>>>
>>>>       
>>> Well, no, the problem is still the same.    The small disconnected labs
>>> are looking to a solution, and they need to buy Satellite in order to
>>> do
>>> mirroring, even if they only have 10 systems.
>>>
>>>     
>> This seems like a perfect situation for the suggestion made earlier about allowing Spacewalk to refer to external repositories. Use squid to only cache the packages required by the 10 systems rather than downloading the entire RHEL/CentOS repository, and just handle package lists in Spacewalk rather than the packages themselves.
>>   
> 

I like the above ideas as well.  I've wanted to add on to Pulp's CLI and 
make a stand alone tool for content synchronization.  A while back I 
threw up some updated info in the Pulp wiki with my thoughts on what I 
thought we might need in a pulp CLI tool:

https://fedorahosted.org/pulp

""PULP ¶

Pulp is an an application for managing software content. Its goals are:

     * The replication software repositories from any location (http, 
file system, ISO, RHN) to a local onsite repository.
     * Provide and easy mechanism for systems to get access to these 
repositories and install software from them.
     * An accounting mechanism that keeps track of what systems are 
using what repositories

At its core we envision pulp being able to provide a series of input and 
output formats to synchronize content to and from various locations.

Examples would be:

Mirror F9 i386 everything repository:

   pulp mirror 
--input=http://download.fedora.redhat.com/pub/fedora/linux/releases/9/Everything/i386/os/Packages/\
     --output=file:///var/www/repo/f9-i386

Mirror Red Hat Enterprise Linux version 5 to an ISO:

pulp mirror --input=rhn://rhel-i386-server-5 
--output=iso://root/RHEL5-PACKAGES.ISO

Mirror Fedora 9 into a Spacewalk Channel:

   pulp mirror 
--input=http://download.fedora.redhat.com/pub/fedora/linux/releases/9/Everything/i386/os/Packages/\
     --output=spacewalk://somespacewalk.example.com  --username=admin 
--password=redhat

Pulp will provide an easy web, web-services, and command line interface 
for managing all of this.""

That said, my personal bandwidth for implementing such a project is 
somewhat low, although as others have said, the desire for things like 
above is definitely there.

The central problem we still have is that the only way to synchronize 
all the current, old and various arches of RHEL to a machine is by the 
use of satellite-sync to an RHN Satellite.

If we are looking to synchronize RHEL, we will need to use the 
satellite-sync API or work with the Red Hat Network team to provide new 
APIs for any new tools we develop.

If we want a tool to work with things *other* than RHEL then we are in 
better shape because the content is easier to get too.

With Spacewalk users are currently tasked with fetching their own 
content and using rhnpush to get the content into a Spacewalk channel. 
While this is workable, it isn't the most user friendly of methods.   I 
think there still is a need for a tool for fetching remote content and 
keeping it in sync with some form of local storage (disk or spacewalk 
channel) which is what Dehaan and the Cobbler users are clamoring for.

I think we should develop a new tool like described above for content 
retrieval and use Spacewalk channels to track usage.

As for the feature to make Spacewalk Channels be proxies for remote 
content, I think that is a good thing but doesn't help the users who 
want all the content available locally.

> That doesn't solve the disconnected lab case, but probably does solve 
> the problem where you need to route through a system in a DMZ for 
> updates -- probably 90% of the cases we need to solve.   I like this.

Disconnected labs will need to sync all the content to some local store 
for sure.  A CLI tool like proposed above really does help with this case.

The thing about using Spacewalk Proxy is that it still requires either a 
Spacewalk server, a RHN Satellite or a connection to rhn.redhat.com for 
it to function.  It has no notion of really what a channel is or any 
local 'management' of the content.  It is a glorified squid cache.

> If we solve it generically, this could also possibly allow using 
> Spacewalk Proxy to be a Proxy for Red Hat Network as well as just a 
> Proxy for another Spacewalk/Satellite?  

Not quite following this statement ...

Mike
-- 
Mike McCune
mmccune AT redhat.com
Engineering               | Portland, OR
RHN Satellite             | 650.567.9039x79248