[Pulp-list] 2to3 migration: memory usage and open file handles

[Winberg Adam]

thats weird. I had a lot of memory usage while I was running on pulpcore 3.4, but after upgrading to 3.7 there was hardly any memory usage.


I will reboot and run with 2 workers instead of 4 (don't know if that is even relevant in migration) and run a new migration from scratch before filing an issue.


//Adam


________________________________
From: Daniel Alley <dalley at redhat.com>
Sent: 05 November 2020 01:58
To: Winberg Adam
Cc: pulp-list at redhat.com
Subject: Re: [Pulp-list] 2to3 migration: memory usage and open file handles

Hi Adam,

We discovered (and fixed) some memory leaks in a library that we are using [0] [1], which happens to be a Python extension written in C.  Right now we're still waiting on the maintainers of that library to review the changes before we package them and ship the RPM, but we'll definitely let you know what that happens.

However, these issues would have affected every previous version equally, so it's a little strange that you're only running into it now.  Nothing else about your setup has changed I assume?

re: file descriptors, we've been testing migrating much larger systems (300k RPMs, 600k errata) without running into problems, so I'm perplexed about what could be causing that.  File an issue and list which repositories you're attempting to migrate and we'll see if we can reproduce.


[0] https://github.com/rpm-software-management/createrepo_c/pull/231
[1] https://github.com/rpm-software-management/createrepo_c/pull/233

On Wed, Nov 4, 2020 at 12:40 PM Winberg Adam <Adam.Winberg at smhi.se<mailto:Adam.Winberg at smhi.se>> wrote:

Hi,


running a 2to3 migration with  2to3-migration-0.5.1 seems to consume a whole lot more memory than previous versions. My 12G RAM was quickly spent, i increased to 16G which wasnt enough either. Earlier migrations with 0.5.0 didnt spend anywhere near that amount.


Also - the migration fails with

OSError: [Errno 23] Too many open files in system: ..


The memory usage increases while running the 'Migrating rpm content to Pulp 3 rpm' subtask. With 16G RAM I only get to about 114000/152000 pkgs in that task before the memory is more or less all consumed and the OSError appears.


So it seems to me that there is some type of regression here. Any pointers on how I can further debug or work around this?


This is on RHEL8 with

python3-pulp-rpm-3.7.0-1.el8.noarch
python3-pulpcore-3.7.3-1.el8.noarch
python3-pulp-2to3-migration-0.5.1-1.el8.noarch

//Adam

_______________________________________________
Pulp-list mailing list
Pulp-list at redhat.com<mailto:Pulp-list at redhat.com>
https://www.redhat.com/mailman/listinfo/pulp-list
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20201105/ef7d4182/attachment.htm>