[Pulp-dev] file repository type

[Eric Helms]

A few suggestions from my own heavy use of the current "plugin" (and I may
repeat some previously said ideas):

 1) consider supporting syncing from a few well known types: ftp, apache
static file server
 2) Support "uploading" a file from a url
 3) Look at what wget does and recursively sync down whatever is over at
location X
 4) Use name and path as the "ID" for a file to allow re-uploads of the
same file name

I agree that syncing Git would be cool, but should be it's own entity since
Git is a known "protocol". The wget way is interesting in that I used that
to sync debian to a file repo:

 1) I synced via recursive wget a debian repository
 2) I used the script at
https://gist.github.com/ehelms/3fd956ee887db3d7bac20b29efa3dd51 to generate
a recursive (with full file path) PULP_MANIFEST
 3) I then synced that to fully mirror the repository

Steps 1-3 seem common enough that I shouldn't have to do it and/or waste
storage space just to replicate a potentially large file location.

Eric

On Tue, Jun 6, 2017 at 6:02 PM, Tom McKay <thomasmckay at redhat.com> wrote:

> As a user, the standard is "wget". In foreman I entered a url to a raw
> github file... and it failed (obscurely). While one could argue for better
> error messaging, I would argue that I was pretty explicit in indicating I
> wanted the results returned from the url. When I sync or resync the
> content, just go wget it again and stuff it into the repo. Yes, it's a repo
> of one but in foreman I make "products" which are collections of repos.
> (Agreed that my next ask would be to "sync all the files listed at a url. :)
>
>
>
>
> On Tue, Jun 6, 2017 at 5:03 PM, Michael Hrivnak <mhrivnak at redhat.com>
> wrote:
>
>> The key is in discovery of files. There's no standard we know of that
>> would be helpful, which is why we depend on a PULP_MANIFEST to tell us what
>> files are available. Maybe there are some cases when a particular service
>> has a well-defined way of expressing what files are available, in which
>> case we could instead support that as the discovery mechanism. It's a good
>> question about whether that would be an addition to the pulp_file plugin,
>> or a separate plugin. It probably needs to be evaluated case-by-case.
>>
>> As one related example, the Pulp 2 puppet importer supports both the
>> forge API and a PULP_MANIFEST for discovery of available puppet modules on
>> a remote server.
>>
>> On Tue, Jun 6, 2017 at 4:46 PM, Brian Bouterse <bbouters at redhat.com>
>> wrote:
>>
>>> What if the pulp_file plugin (the pulp3 one) would support syncing files
>>> directly from remotes without a manifest? Like syncing from a remote
>>> webserver or version control system. Maybe that would be a VCS plugin, not
>>> the file_plugin, but regardless, other software similar to Pulp supports
>>> [0] that use case.
>>>
>>> [0]: http://bit.ly/2s1pCmF
>>>
>>> On Mon, Jun 5, 2017 at 1:10 PM, Michael Hrivnak <mhrivnak at redhat.com>
>>> wrote:
>>>
>>>> You do not need to create a manifest in order to upload. You only need
>>>> one in order to sync. Here is documentation about upload:
>>>>
>>>> http://docs.pulpproject.org/plugins/pulp_rpm/user-guide/isos
>>>> .html#uploading-isos-to-a-repository
>>>>
>>>> You can also create the manifest file in 3-4 lines of bash. Maybe it
>>>> would be useful for Pulp to include a small script that can generate the
>>>> manifest.
>>>>
>>>> That said, you can always upload your files to a Pulp repo, and publish
>>>> it, which will cause Pulp to create a new manifest for you.
>>>>
>>>> On Mon, Jun 5, 2017 at 6:41 AM, Tom McKay <thomasmckay at redhat.com>
>>>> wrote:
>>>>
>>>>> I'd like to sync individual files and folders (recursively?) into pulp
>>>>> via foreman. Apparently, though, I need to create a pulp manifest in order
>>>>> to upload. Is this necessary or can pulp do it for me? Or should I build
>>>>> this into foreman? @ehelms provided a gist of the steps[1] and it seems
>>>>> pretty basic and something pulp could be doing itself.
>>>>>
>>>>> My use case is that I wanted to import a build file for openshift from
>>>>> github, then promote that file (and other artifacts) through foreman's life
>>>>> cycle environments. A lot of demos and tutorials for openshift start from
>>>>> git and other non-local components.
>>>>>
>>>>> [1] https://gist.github.com/ehelms/3fd956ee887db3d7bac20b29efa3dd51
>>>>>
>>>>> _______________________________________________
>>>>> Pulp-dev mailing list
>>>>> Pulp-dev at redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Michael Hrivnak
>>>>
>>>> Principal Software Engineer, RHCE
>>>>
>>>> Red Hat
>>>>
>>>> _______________________________________________
>>>> Pulp-dev mailing list
>>>> Pulp-dev at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>
>>>>
>>>
>>
>>
>> --
>>
>> Michael Hrivnak
>>
>> Principal Software Engineer, RHCE
>>
>> Red Hat
>>
>
>
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20170615/3104a15f/attachment.htm>