[Pulp-list] 2to3 migration: memory usage and open file handles

[Winberg Adam]

both! After fixing that script the memory usage is very manageable.


//Adam


________________________________
From: Daniel Alley <dalley at redhat.com>
Sent: 05 November 2020 15:30
To: Winberg Adam
Cc: pulp-list at redhat.com
Subject: Re: [Pulp-list] 2to3 migration: memory usage and open file handles

Do you mean that the bash script was the cause of the file descriptor issue, or the memory issue, or both?

On Thu, Nov 5, 2020 at 3:25 AM Winberg Adam <Adam.Winberg at smhi.se<mailto:Adam.Winberg at smhi.se>> wrote:

sorry, ignore my problem. It was totally unrelated to pulp, it was another process going haywire at the same time as I was running the migration (bug in bash while loop in an unrelated housekeeping script). I was so focused on the migration that I didn't even notice that the issue was elsewhere.


//Adam


________________________________
From: pulp-list-bounces at redhat.com<mailto:pulp-list-bounces at redhat.com> <pulp-list-bounces at redhat.com<mailto:pulp-list-bounces at redhat.com>> on behalf of Winberg Adam <Adam.Winberg at smhi.se<mailto:Adam.Winberg at smhi.se>>
Sent: 05 November 2020 07:42
To: Daniel Alley
Cc: pulp-list at redhat.com<mailto:pulp-list at redhat.com>
Subject: Re: [Pulp-list] 2to3 migration: memory usage and open file handles


thats weird. I had a lot of memory usage while I was running on pulpcore 3.4, but after upgrading to 3.7 there was hardly any memory usage.


I will reboot and run with 2 workers instead of 4 (don't know if that is even relevant in migration) and run a new migration from scratch before filing an issue.


//Adam


________________________________
From: Daniel Alley <dalley at redhat.com<mailto:dalley at redhat.com>>
Sent: 05 November 2020 01:58
To: Winberg Adam
Cc: pulp-list at redhat.com<mailto:pulp-list at redhat.com>
Subject: Re: [Pulp-list] 2to3 migration: memory usage and open file handles

Hi Adam,

We discovered (and fixed) some memory leaks in a library that we are using [0] [1], which happens to be a Python extension written in C.  Right now we're still waiting on the maintainers of that library to review the changes before we package them and ship the RPM, but we'll definitely let you know what that happens.

However, these issues would have affected every previous version equally, so it's a little strange that you're only running into it now.  Nothing else about your setup has changed I assume?

re: file descriptors, we've been testing migrating much larger systems (300k RPMs, 600k errata) without running into problems, so I'm perplexed about what could be causing that.  File an issue and list which repositories you're attempting to migrate and we'll see if we can reproduce.


[0] https://github.com/rpm-software-management/createrepo_c/pull/231
[1] https://github.com/rpm-software-management/createrepo_c/pull/233

On Wed, Nov 4, 2020 at 12:40 PM Winberg Adam <Adam.Winberg at smhi.se<mailto:Adam.Winberg at smhi.se>> wrote:

Hi,


running a 2to3 migration with  2to3-migration-0.5.1 seems to consume a whole lot more memory than previous versions. My 12G RAM was quickly spent, i increased to 16G which wasnt enough either. Earlier migrations with 0.5.0 didnt spend anywhere near that amount.


Also - the migration fails with

OSError: [Errno 23] Too many open files in system: ..


The memory usage increases while running the 'Migrating rpm content to Pulp 3 rpm' subtask. With 16G RAM I only get to about 114000/152000 pkgs in that task before the memory is more or less all consumed and the OSError appears.


So it seems to me that there is some type of regression here. Any pointers on how I can further debug or work around this?


This is on RHEL8 with

python3-pulp-rpm-3.7.0-1.el8.noarch
python3-pulpcore-3.7.3-1.el8.noarch
python3-pulp-2to3-migration-0.5.1-1.el8.noarch

//Adam

_______________________________________________
Pulp-list mailing list
Pulp-list at redhat.com<mailto:Pulp-list at redhat.com>
https://www.redhat.com/mailman/listinfo/pulp-list
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20201105/a1f95873/attachment.htm>