[Pulp-list] 2to3 migration: memory usage and open file handles

[Winberg Adam]

Hi,


running a 2to3 migration with  2to3-migration-0.5.1 seems to consume a whole lot more memory than previous versions. My 12G RAM was quickly spent, i increased to 16G which wasnt enough either. Earlier migrations with 0.5.0 didnt spend anywhere near that amount.


Also - the migration fails with

OSError: [Errno 23] Too many open files in system: ..


The memory usage increases while running the 'Migrating rpm content to Pulp 3 rpm' subtask. With 16G RAM I only get to about 114000/152000 pkgs in that task before the memory is more or less all consumed and the OSError appears.


So it seems to me that there is some type of regression here. Any pointers on how I can further debug or work around this?


This is on RHEL8 with

python3-pulp-rpm-3.7.0-1.el8.noarch
python3-pulpcore-3.7.3-1.el8.noarch
python3-pulp-2to3-migration-0.5.1-1.el8.noarch

//Adam
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20201104/0493f8cc/attachment.htm>