[Pulp-list] 2to3 migration: memory usage and open file handles

[Daniel Alley]

Hi Adam,

We discovered (and fixed) some memory leaks in a library that we are using
[0] [1], which happens to be a Python extension written in C.  Right now
we're still waiting on the maintainers of that library to review the
changes before we package them and ship the RPM, but we'll definitely let
you know what that happens.

However, these issues would have affected every previous version equally,
so it's a little strange that you're only running into it now.  Nothing
else about your setup has changed I assume?

re: file descriptors, we've been testing migrating much larger systems
(300k RPMs, 600k errata) without running into problems, so I'm perplexed
about what could be causing that.  File an issue and list which
repositories you're attempting to migrate and we'll see if we can reproduce.


[0] https://github.com/rpm-software-management/createrepo_c/pull/231
[1] https://github.com/rpm-software-management/createrepo_c/pull/233

On Wed, Nov 4, 2020 at 12:40 PM Winberg Adam <Adam.Winberg at smhi.se> wrote:

> Hi,
>
>
> running a 2to3 migration with  2to3-migration-0.5.1 seems to consume a
> whole lot more memory than previous versions. My 12G RAM was quickly spent,
> i increased to 16G which wasnt enough either. Earlier migrations with 0.5.0
> didnt spend anywhere near that amount.
>
>
> Also - the migration fails with
>
> OSError: [Errno 23] Too many open files in system: ..
>
>
> The memory usage increases while running the 'Migrating rpm content to
> Pulp 3 rpm' subtask. With 16G RAM I only get to about 114000/152000 pkgs in
> that task before the memory is more or less all consumed and the OSError
> appears.
>
>
> So it seems to me that there is some type of regression here. Any pointers
> on how I can further debug or work around this?
>
>
> This is on RHEL8 with
>
> python3-pulp-rpm-3.7.0-1.el8.noarch
> python3-pulpcore-3.7.3-1.el8.noarch
> python3-pulp-2to3-migration-0.5.1-1.el8.noarch
>
> //Adam
>
> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20201104/3b4c3058/attachment.htm>