[Pulp-list] /var/cache/pulp filling up?

[Michael Hrivnak]

This appears to be a regression in 2.7.

In 2.6 (and earlier), at the completion of each rpm download, the file
would be moved from its temporary location to the permanent location. That
means for any given sync, you would never have more files in that temporary
location than you did concurrent downloads. Using the default setting of 5
concurrent downloads, that's a maximum of 5 rpms. No problem so far.

This change turned the "move" into a "copy":

https://github.com/pulp/pulp_rpm/commit/de1895ede2f24731dcf6f52321beaee6b71c15a4

Now all of the rpm files pile up, and they don't get removed until the end
of the task. On quick glance, it appears that 2.8 suffers the same problem,
although the code has been refactored.

If you don't mind filing a bug report, it should be an easy fix: likely
just adding a delete call after the copy.

https://pulp.plan.io/projects/pulp_rpm/issues/new

Thanks,
Michael

On Thu, Mar 17, 2016 at 11:39 AM, Randy Barlow <rbarlow at redhat.com> wrote:

> On Wednesday, March 16, 2016 12:23:48 AM EDT Matthew Madey
> wrote:
> > I think it's failing to remove the files when they are done. Currently
> > there are 2G of RPM's in the cache directory, most of them are
> 0bytes. I'm
> > syncing with the RHEL7 repository, and I'm seeing every package in
> the
> > repository in the cache directory, however, once it had filled /var,
> the
> > remaining RPM files were 0 bytes. Is there a way to limit to how
> much
> > content is able to be staged in the cache directory before being
> published
> > to /var/lib/pulp?
>
> Hello again Matthew!
>
> I believe that the sync task downloads all the RPMs into /var/cache/
> pulp and then moves them into /var/lib/pulp as a later step. Several of
> us developers want to redesign our plugin API to support stream
> processing, but in the meantime you will need enough storage in /var/
> cache/pulp to support all the bytes for an entire sync. I don't know off
> the top of my head how large distro repos are, but I'd say they're
> usually on the scale of 10s of GBs.
>
> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20160317/8d9bfed5/attachment.htm>