[Pulp-list] 2to3 migration: memory usage and open file handles

[Daniel Alley]

Good to know!  In any case, memory consumption should improve further
soon.  On the big system w/ 300k RPMs and a bunch of errata, etc, we were
using something like 20gb by the end of the migration, and it dropped to
<1gb after applying these fixes.

On Thu, Nov 5, 2020 at 9:34 AM Winberg Adam <Adam.Winberg at smhi.se> wrote:

> both! After fixing that script the memory usage is very manageable.
>
>
> //Adam
>
>
> ------------------------------
> *From:* Daniel Alley <dalley at redhat.com>
> *Sent:* 05 November 2020 15:30
> *To:* Winberg Adam
> *Cc:* pulp-list at redhat.com
> *Subject:* Re: [Pulp-list] 2to3 migration: memory usage and open file
> handles
>
> Do you mean that the bash script was the cause of the file descriptor
> issue, or the memory issue, or both?
>
> On Thu, Nov 5, 2020 at 3:25 AM Winberg Adam <Adam.Winberg at smhi.se> wrote:
>
>> sorry, ignore my problem. It was totally unrelated to pulp, it was
>> another process going haywire at the same time as I was running the
>> migration (bug in bash while loop in an unrelated housekeeping script). I
>> was so focused on the migration that I didn't even notice that the issue
>> was elsewhere.
>>
>>
>> //Adam
>>
>>
>> ------------------------------
>> *From:* pulp-list-bounces at redhat.com <pulp-list-bounces at redhat.com> on
>> behalf of Winberg Adam <Adam.Winberg at smhi.se>
>> *Sent:* 05 November 2020 07:42
>> *To:* Daniel Alley
>> *Cc:* pulp-list at redhat.com
>> *Subject:* Re: [Pulp-list] 2to3 migration: memory usage and open file
>> handles
>>
>>
>> thats weird. I had a lot of memory usage while I was running on pulpcore
>> 3.4, but after upgrading to 3.7 there was hardly any memory usage.
>>
>>
>> I will reboot and run with 2 workers instead of 4 (don't know if that is
>> even relevant in migration) and run a new migration from scratch before
>> filing an issue.
>>
>>
>> //Adam
>>
>>
>> ------------------------------
>> *From:* Daniel Alley <dalley at redhat.com>
>> *Sent:* 05 November 2020 01:58
>> *To:* Winberg Adam
>> *Cc:* pulp-list at redhat.com
>> *Subject:* Re: [Pulp-list] 2to3 migration: memory usage and open file
>> handles
>>
>> Hi Adam,
>>
>> We discovered (and fixed) some memory leaks in a library that we are
>> using [0] [1], which happens to be a Python extension written in C.  Right
>> now we're still waiting on the maintainers of that library to review the
>> changes before we package them and ship the RPM, but we'll definitely let
>> you know what that happens.
>>
>> However, these issues would have affected every previous version equally,
>> so it's a little strange that you're only running into it now.  Nothing
>> else about your setup has changed I assume?
>>
>> re: file descriptors, we've been testing migrating much larger systems
>> (300k RPMs, 600k errata) without running into problems, so I'm perplexed
>> about what could be causing that.  File an issue and list which
>> repositories you're attempting to migrate and we'll see if we can reproduce.
>>
>>
>> [0] https://github.com/rpm-software-management/createrepo_c/pull/231
>> [1] https://github.com/rpm-software-management/createrepo_c/pull/233
>>
>> On Wed, Nov 4, 2020 at 12:40 PM Winberg Adam <Adam.Winberg at smhi.se>
>> wrote:
>>
>>> Hi,
>>>
>>>
>>> running a 2to3 migration with  2to3-migration-0.5.1 seems to consume a
>>> whole lot more memory than previous versions. My 12G RAM was quickly spent,
>>> i increased to 16G which wasnt enough either. Earlier migrations with 0.5.0
>>> didnt spend anywhere near that amount.
>>>
>>>
>>> Also - the migration fails with
>>>
>>> OSError: [Errno 23] Too many open files in system: ..
>>>
>>>
>>> The memory usage increases while running the 'Migrating rpm content to
>>> Pulp 3 rpm' subtask. With 16G RAM I only get to about 114000/152000 pkgs in
>>> that task before the memory is more or less all consumed and the OSError
>>> appears.
>>>
>>>
>>> So it seems to me that there is some type of regression here. Any
>>> pointers on how I can further debug or work around this?
>>>
>>>
>>> This is on RHEL8 with
>>>
>>> python3-pulp-rpm-3.7.0-1.el8.noarch
>>> python3-pulpcore-3.7.3-1.el8.noarch
>>> python3-pulp-2to3-migration-0.5.1-1.el8.noarch
>>>
>>> //Adam
>>>
>>> _______________________________________________
>>> Pulp-list mailing list
>>> Pulp-list at redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-list
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20201105/c6ae8110/attachment.htm>