[Pulp-list] 2to3 migration: memory usage and open file handles

[Daniel Alley]

Do you mean that the bash script was the cause of the file descriptor
issue, or the memory issue, or both?

On Thu, Nov 5, 2020 at 3:25 AM Winberg Adam <Adam.Winberg at smhi.se> wrote:

> sorry, ignore my problem. It was totally unrelated to pulp, it was another
> process going haywire at the same time as I was running the migration (bug
> in bash while loop in an unrelated housekeeping script). I was so focused
> on the migration that I didn't even notice that the issue was elsewhere.
>
>
> //Adam
>
>
> ------------------------------
> *From:* pulp-list-bounces at redhat.com <pulp-list-bounces at redhat.com> on
> behalf of Winberg Adam <Adam.Winberg at smhi.se>
> *Sent:* 05 November 2020 07:42
> *To:* Daniel Alley
> *Cc:* pulp-list at redhat.com
> *Subject:* Re: [Pulp-list] 2to3 migration: memory usage and open file
> handles
>
>
> thats weird. I had a lot of memory usage while I was running on pulpcore
> 3.4, but after upgrading to 3.7 there was hardly any memory usage.
>
>
> I will reboot and run with 2 workers instead of 4 (don't know if that is
> even relevant in migration) and run a new migration from scratch before
> filing an issue.
>
>
> //Adam
>
>
> ------------------------------
> *From:* Daniel Alley <dalley at redhat.com>
> *Sent:* 05 November 2020 01:58
> *To:* Winberg Adam
> *Cc:* pulp-list at redhat.com
> *Subject:* Re: [Pulp-list] 2to3 migration: memory usage and open file
> handles
>
> Hi Adam,
>
> We discovered (and fixed) some memory leaks in a library that we are using
> [0] [1], which happens to be a Python extension written in C.  Right now
> we're still waiting on the maintainers of that library to review the
> changes before we package them and ship the RPM, but we'll definitely let
> you know what that happens.
>
> However, these issues would have affected every previous version equally,
> so it's a little strange that you're only running into it now.  Nothing
> else about your setup has changed I assume?
>
> re: file descriptors, we've been testing migrating much larger systems
> (300k RPMs, 600k errata) without running into problems, so I'm perplexed
> about what could be causing that.  File an issue and list which
> repositories you're attempting to migrate and we'll see if we can reproduce.
>
>
> [0] https://github.com/rpm-software-management/createrepo_c/pull/231
> [1] https://github.com/rpm-software-management/createrepo_c/pull/233
>
> On Wed, Nov 4, 2020 at 12:40 PM Winberg Adam <Adam.Winberg at smhi.se> wrote:
>
>> Hi,
>>
>>
>> running a 2to3 migration with  2to3-migration-0.5.1 seems to consume a
>> whole lot more memory than previous versions. My 12G RAM was quickly spent,
>> i increased to 16G which wasnt enough either. Earlier migrations with 0.5.0
>> didnt spend anywhere near that amount.
>>
>>
>> Also - the migration fails with
>>
>> OSError: [Errno 23] Too many open files in system: ..
>>
>>
>> The memory usage increases while running the 'Migrating rpm content to
>> Pulp 3 rpm' subtask. With 16G RAM I only get to about 114000/152000 pkgs in
>> that task before the memory is more or less all consumed and the OSError
>> appears.
>>
>>
>> So it seems to me that there is some type of regression here. Any
>> pointers on how I can further debug or work around this?
>>
>>
>> This is on RHEL8 with
>>
>> python3-pulp-rpm-3.7.0-1.el8.noarch
>> python3-pulpcore-3.7.3-1.el8.noarch
>> python3-pulp-2to3-migration-0.5.1-1.el8.noarch
>>
>> //Adam
>>
>> _______________________________________________
>> Pulp-list mailing list
>> Pulp-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-list
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20201105/4ea43be4/attachment.htm>