[Pulp-list] Using pulp to create monthly releases

Tue May 19 21:41:08 UTC 2015

My concern isn't pulp processing, it's the shear size and number of the
repos. I did a full sample export today and it was over 40GB for all the
different distributions (CentOS, RedHat, etc) and releases (5 and 6, etc)
that we have. The "changes" tarball will only be around 2GB.

My full-day effort time frame is not based on pulp at all, but based on our
existing procedures to do a final sync of the internet repos, use some
shell scripts that loop through the repos to perform repodiffs that
generates our "changes" tarball, then export that to media to ingest on the
downstream servers, then building the repos on the other end. I'm trying to
replace this process with pulp on both ends, hopefully it won't be a full
day process.

Moving forward, I'm going to see how long it takes to export each entire
repo and move it downstream and import there. If it's under 3-4 hours, I
think that's doable. I've also started playing around with using something
like rsync to generate a batchfile binary blob that can apply changes to an
existing downstream repo and create the next month's release. Barring that,
I could re-purpose our old legacy repodiff script to achieve the same
results.

As a side note, after some more research, it looks as though
Katello-disconnected is the supported method to do what I'm attempting.
We're not in a position to start integrating Katello right now, though.

On Mon, May 18, 2015 at 9:38 AM, Barnaby Court <bcourt at redhat.com> wrote:

> Hi Mike,
>
> When you say it's a full day effort right now, what is it that is causing
> it to be a full day effort? Is it personal involvement or the amount of
> time that Pulp is spending on processing? If it is personal involvement I
> would highly recommend repo groups so that you can configure all the
> repositories you are mirroring once and then export them in one action. If
> it Pulp processing time it would be helpful to understand how many
> repositories, how many units of each type (roughly) in each repository and
> which parameters you have enabled on all your repositories. Some parameters
> greatly increase the processing time. The biggest is generating the sqlite
> files. If that is turned on it can greatly increase the time required to
> publish. Also, is your internet connected Pulp server using local disk for
> the /var/lib/pulp directory or is it using NFS? Thanks!
>
> -Barnaby
>
> ----- Original Message -----
> From: "Mike Griffin" <mgriffin at griftech.net>
> To: "Randy Barlow" <rbarlow at redhat.com>
> Cc: pulp-list at redhat.com
> Sent: Friday, May 15, 2015 4:42:39 PM
> Subject: Re: [Pulp-list] Using pulp to create monthly releases
>
> What I'm struggling with is how I can export new/updated packages of a
> repo on my internet-connected pulp server and on the downstream server
> re-create the full release-repo, using the combination of existing packages
> on the downstream server and the updates from the upstream server. This
> task would be easy if I could just export the entire release-repo every
> month and import it downstream. It's just that due to the number of distros
> we support, that gets to be unwieldy very quickly. Even with just the
> new/updated packages every month, it's a full day effort now.
>
>
>
> On Fri, May 15, 2015 at 2:21 PM, Randy Barlow < rbarlow at redhat.com >
> wrote:
>
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>
> On 05/14/2015 07:21 PM, Mike Griffin wrote:
> > Maybe I can illustrate my requirements a little easier. For
> > example: During my first month of April, package A-1.0, B-1.0, and
> > C-1.0 are downloaded via the nightly sync. When I create my
> > release, all 3 packages appear in my export and I move them
> > downstream.
> >
> > The next month when I create my release-set, package A is updated
> > to version 2.0, B is static, C is removed upstream and D1.0 is
> > released. Since I have my nightly sync set to "remove-missing true"
> > and "retain-old-count 0", A1.0 and C1.0 should be removed from the
> > sync repo and therefore removed from the release-set. I want to
> > move A2.0 and D1.0 downstream, copy B1.0 that already exists
> > downstream and re-create the release directory there (A2.0, B1.0,
> > D1.0).
>
> Hi Mike,
>
> Perhaps you can achieve the above by removing all packages from the
> destination repo, and then copying all packages from the source. So
> long as no publish happens in-between, the final publish will
> atomically switch (i.e., no clients will ever see the empty repo).
> Does that make sense?
>
> - --
> Randy Barlow
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2
>
> iQIcBAEBCgAGBQJVVjlBAAoJEIyFaKUJtmpicv8P/3lbQ52FTs9YXzyhb80gxCwV
> 8NwqW3y0ZVKjyK4ZK1ML9J/26ig7Bz4e4rgBvjJokuzcI2AdyoPCJtuGcxucFa0l
> DZA0T5VtNlpggEET9LoVeb23C9b/mQma54NmVkuzUDBlyyqDAHL+zK11Hs+SYp3/
> LSq+p/QT6TKYI/GqjoTVZYGfKpVvpjt4GBERP6+K4YRhKD2x3m7R0zlpQQZ2++aq
> JN/+UCGuPjEBxXUZOIsLLUUuTgswWwih1Qghs3ZCxxotoSw2IfMXmf3Zl4W+0TS8
> U46lh9CKavmPx01e7fcPmeZlObZBpSpFwqFp7PFzmOeWAwEWxTLptyCT618/RAKM
> KvXY0aYT2DWGQ9MQFiXr93MoB4+Ckc7DHjv/DW2b/9rKGp7T4HF/Y9BzEwzDE1ua
> wArN8fIRXfSJWbdbOldl97i8SSroagTrqrbnpRtxa4CBdgc7i+36Ct6RnCSq59OL
> srFTEmwap78x9N9AntjaWUhFbNlai54VU8AYbkPavVOP0QLz8Vzgsaj3SJ+2dBJ7
> xTEhQRbyoVnSu3FDNY1AkQP9Rr0ZlQSUYBgVhXWhwaZU9breltMMGfX3PkC0M8iu
> jj/K6MmKPwp2fhn7TmvWDDaBnwPJqvAqGlKhUXlHN3vGkoVAngcoPvaThz/td2Na
> MWR1e2U5y8NrwJ/ml9QA
> =yMlk
> -----END PGP SIGNATURE-----
>
> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
>
>
>
> --
> Regards,
> Mike Griffin
>
> Ever grateful, ever true
>
> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
>

-- 
Regards,
Mike Griffin

Ever grateful, ever true
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20150519/331ed62d/attachment.htm>