[Spacewalk-list] Need for a lightweight mirroring solution

Wed Jul 23 13:52:25 UTC 2008

Thomas von Steiger wrote:
> What i think about this:
>
> There are to many very nice repo tools.
> For manage os content the best way is one way for all the different 
> software framworks.
>
> Currently for http://www.ovirt.org/ and other projects there is 
> cobbler the solution.

Part of the idea here is to make Pulp become a package management 
framework applications like Cobbler, Satellite, and ovirt can all use -- 
just as with Cobbler we are creating cobbler as install server projects 
like ovirt, Satellite, genome, and so forth can all use.
>
> For pulp there is a own repo tool with very nice ideas about content 
> handling.

Yes, but Pulp doesn't really exist yet :)

> For Satelllite there is the knowlege about the os content implemented in:
>     - Filesystem Tree with rpm name/version/arch information. There 
> you can have easy more then 100Gb of Data's with one distribution over 
> the years.
> and     - Meta Data's about all the rpm's and logical defined software 
> channels for os/cluster/virtualisation/storage/tools....what you want.
> And for Spacewalk there is currently the same concept "like the 
> satellite" implemented.
> To manage differend distros with the spacewalk it need's now a 
> filesystem tree extention.
>
> Better woul'd be one concept for all that differend tools.

Indeed.

> Because Spacewalk(Satellite also...) can not manage virtual servers 
> and storage like ovirt.

Spacewalk /CAN/ manage virtual servers.   ovirt has a 
image/appliance-centric model so possible (dare-I-say-it) "synergy" 
there still needs to be explored.  It's obviously something many of us 
are very interersted in.

> And ovirt can not manage systems, patching and os content like 
> spacewalk and satellite.

Right.
>
> With the satellite we have now some years production experience and 
> activationkeys are very practicale to attach a system to the right 
> channels.
> With spacewalk and to other important new projects we shoul'd now be 
> able to create the opensource tools/framework for next generation it.
> And it need's more than one tool. This tools should be live hand in 
> hand because there are many new synergys between available.
> Then we can share system information between all the tools or 
> import/move services/servers from one framwork to a other framwork.
>
> And for this it need's one way for repo managment for all?
> What you think about this?

Basically I want to see pulp become a common library and toolset that 
all applications that need to do package management/mirroring can use -- 
eventually including Spacewalk.    So we need to start thinking in terms 
of "horizontal" building blocks that we can use to build any sort of 
application that comes along, and Spacewalk is a "vertical" application 
that uses these building blocks for certain use cases.

If we have all applications using the same "blocks", then it becomes 
much easier to use them in the same environment -- no duplicate package 
mirroring, no multiple places to go to configure packages, etc -- same 
for the install server and so forth. 

Anyhow, that's what I am getting at here.

Whatever we do should be yum-powered on the client side and work exactly 
the same way for Fedora, RHEL, and derivative distributions -- and be 
exceedingly simple to set up and use.

--Michael
>
> Thomas
>
> On Jul 22, 2008, at 8:54 PM, Mike McCune wrote:
>
>> Michael DeHaan wrote:
>>> Covil, K (Kim) wrote:
>>>>> From: spacewalk-list-bounces at redhat.com [mailto:spacewalk-list-
>>>>> bounces at redhat.com] On Behalf Of Michael DeHaan
>>>>> Sent: 21 July 2008 13:38
>>>>> To: Jon Stanley
>>>>> Cc: spacewalk-list at redhat.com
>>>>> Subject: Re: [Spacewalk-list] Need for a lightweight mirroring 
>>>>> solution
>>>>>
>>>>> Jon Stanley wrote:
>>>>>
>>>>>> On Mon, Jul 21, 2008 at 8:05 AM, Michael DeHaan <mdehaan at redhat.com>
>>>>>>
>>>>> wrote:
>>>>>
>>>>>>> Many questions I get on cobbler's mailing list deal with using
>>>>>>> yum_rhn_plugin to mirror RHEL packages.   These are typically from
>>>>>>>
>>>>> users who
>>>>>
>>>>>>> have small-mid size environments that cannot use Satellite for
>>>>>>>
>>>>> various
>>>>>
>>>>>>> reasons.
>>>>>>>
>>>>>>>
>>>>>> That;s why we have spacewalk now :)
>>>>>>
>>>>>>
>>>>>>
>>>>> Well, no, the problem is still the same.    The small disconnected 
>>>>> labs
>>>>> are looking to a solution, and they need to buy Satellite in order to
>>>>> do
>>>>> mirroring, even if they only have 10 systems.
>>>>>
>>>>>
>>>> This seems like a perfect situation for the suggestion made earlier 
>>>> about allowing Spacewalk to refer to external repositories. Use 
>>>> squid to only cache the packages required by the 10 systems rather 
>>>> than downloading the entire RHEL/CentOS repository, and just handle 
>>>> package lists in Spacewalk rather than the packages themselves.
>>>>
>>
>> I like the above ideas as well.  I've wanted to add on to Pulp's CLI 
>> and make a stand alone tool for content synchronization.  A while 
>> back I threw up some updated info in the Pulp wiki with my thoughts 
>> on what I thought we might need in a pulp CLI tool:
>>
>> https://fedorahosted.org/pulp
>>
>> ""PULP ¶
>>
>> Pulp is an an application for managing software content. Its goals are:
>>
>>    * The replication software repositories from any location (http, 
>> file system, ISO, RHN) to a local onsite repository.
>>    * Provide and easy mechanism for systems to get access to these 
>> repositories and install software from them.
>>    * An accounting mechanism that keeps track of what systems are 
>> using what repositories
>>
>> At its core we envision pulp being able to provide a series of input 
>> and output formats to synchronize content to and from various locations.
>>
>> Examples would be:
>>
>> Mirror F9 i386 everything repository:
>>
>>  pulp mirror 
>> --input=http://download.fedora.redhat.com/pub/fedora/linux/releases/9/Everything/i386/os/Packages/\ 
>>
>>    --output=file:///var/www/repo/f9-i386
>>
>> Mirror Red Hat Enterprise Linux version 5 to an ISO:
>>
>> pulp mirror --input=rhn://rhel-i386-server-5 
>> --output=iso://root/RHEL5-PACKAGES.ISO
>>
>> Mirror Fedora 9 into a Spacewalk Channel:
>>
>>  pulp mirror 
>> --input=http://download.fedora.redhat.com/pub/fedora/linux/releases/9/Everything/i386/os/Packages/\ 
>>
>>    --output=spacewalk://somespacewalk.example.com  --username=admin 
>> --password=redhat
>>
>> Pulp will provide an easy web, web-services, and command line 
>> interface for managing all of this.""
>>
>> That said, my personal bandwidth for implementing such a project is 
>> somewhat low, although as others have said, the desire for things 
>> like above is definitely there.
>>
>> The central problem we still have is that the only way to synchronize 
>> all the current, old and various arches of RHEL to a machine is by 
>> the use of satellite-sync to an RHN Satellite.
>>
>> If we are looking to synchronize RHEL, we will need to use the 
>> satellite-sync API or work with the Red Hat Network team to provide 
>> new APIs for any new tools we develop.
>>
>> If we want a tool to work with things *other* than RHEL then we are 
>> in better shape because the content is easier to get too.
>>
>> With Spacewalk users are currently tasked with fetching their own 
>> content and using rhnpush to get the content into a Spacewalk 
>> channel. While this is workable, it isn't the most user friendly of 
>> methods.   I think there still is a need for a tool for fetching 
>> remote content and keeping it in sync with some form of local storage 
>> (disk or spacewalk channel) which is what Dehaan and the Cobbler 
>> users are clamoring for.
>>
>> I think we should develop a new tool like described above for content 
>> retrieval and use Spacewalk channels to track usage.
>>
>> As for the feature to make Spacewalk Channels be proxies for remote 
>> content, I think that is a good thing but doesn't help the users who 
>> want all the content available locally.
>>
>>> That doesn't solve the disconnected lab case, but probably does 
>>> solve the problem where you need to route through a system in a DMZ 
>>> for updates -- probably 90% of the cases we need to solve.   I like 
>>> this.
>>
>> Disconnected labs will need to sync all the content to some local 
>> store for sure.  A CLI tool like proposed above really does help with 
>> this case.
>>
>> The thing about using Spacewalk Proxy is that it still requires 
>> either a Spacewalk server, a RHN Satellite or a connection to 
>> rhn.redhat.com for it to function.  It has no notion of really what a 
>> channel is or any local 'management' of the content.  It is a 
>> glorified squid cache.
>>
>>> If we solve it generically, this could also possibly allow using 
>>> Spacewalk Proxy to be a Proxy for Red Hat Network as well as just a 
>>> Proxy for another Spacewalk/Satellite?
>>
>> Not quite following this statement ...
>>
>> Mike
>> -- 
>> Mike McCune
>> mmccune AT redhat.com
>> Engineering               | Portland, OR
>> RHN Satellite             | 650.567.9039x79248
>>
>> _______________________________________________
>> Spacewalk-list mailing list
>> Spacewalk-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/spacewalk-list
>