<div dir="ltr">Do you mean that the bash script was the cause of the file descriptor issue, or the memory issue, or both?<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Nov 5, 2020 at 3:25 AM Winberg Adam <<a href="mailto:Adam.Winberg@smhi.se">Adam.Winberg@smhi.se</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div dir="ltr"> <div id="gmail-m_-8659470818859977245divtagdefaultwrapper" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif" dir="ltr"> <p>sorry, ignore my problem. It was totally unrelated to pulp, it was another process going haywire at the same time as I was running the migration (bug in bash while loop in an unrelated housekeeping script). I was so focused on the migration that I didn't even notice that the issue was elsewhere. </p> <p><br> </p> <p>//Adam</p> <br> <br> <div style="color:rgb(0,0,0)"> <hr style="display:inline-block;width:98%"> <div id="gmail-m_-8659470818859977245divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> <a href="mailto:pulp-list-bounces@redhat.com" target="_blank">pulp-list-bounces@redhat.com</a> <<a href="mailto:pulp-list-bounces@redhat.com" target="_blank">pulp-list-bounces@redhat.com</a>> on behalf of Winberg Adam <<a href="mailto:Adam.Winberg@smhi.se" target="_blank">Adam.Winberg@smhi.se</a>><br> <b>Sent:</b> 05 November 2020 07:42<br> <b>To:</b> Daniel Alley<br> <b>Cc:</b> <a href="mailto:pulp-list@redhat.com" target="_blank">pulp-list@redhat.com</a><br> <b>Subject:</b> Re: [Pulp-list] 2to3 migration: memory usage and open file handles</font> <div> </div> </div> <div> <div id="gmail-m_-8659470818859977245divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif"> <p>thats weird. I had a lot of memory usage while I was running on pulpcore 3.4, but after upgrading to 3.7 there was hardly any memory usage. </p> <p><br> </p> <p>I will reboot and run with 2 workers instead of 4 (don't know if that is even relevant in migration) and run a new migration from scratch before filing an issue. </p> <p><br> </p> <p>//Adam</p> <br> <br> <div style="color:rgb(0,0,0)"> <hr style="display:inline-block;width:98%"> <div id="gmail-m_-8659470818859977245divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Daniel Alley <<a href="mailto:dalley@redhat.com" target="_blank">dalley@redhat.com</a>><br> <b>Sent:</b> 05 November 2020 01:58<br> <b>To:</b> Winberg Adam<br> <b>Cc:</b> <a href="mailto:pulp-list@redhat.com" target="_blank">pulp-list@redhat.com</a><br> <b>Subject:</b> Re: [Pulp-list] 2to3 migration: memory usage and open file handles</font> <div> </div> </div> <div> <div dir="ltr"> <div>Hi Adam, <br> </div> <div><br> </div> <div>We discovered (and fixed) some memory leaks in a library that we are using [0] [1], which happens to be a Python extension written in C. Right now we're still waiting on the maintainers of that library to review the changes before we package them and ship the RPM, but we'll definitely let you know what that happens.<br> </div> <div><br> </div> <div>However, these issues would have affected every previous version equally, so it's a little strange that you're only running into it now. Nothing else about your setup has changed I assume?<br> </div> <div><br> </div> <div>re: file descriptors, we've been testing migrating much larger systems (300k RPMs, 600k errata) without running into problems, so I'm perplexed about what could be causing that. File an issue and list which repositories you're attempting to migrate and we'll see if we can reproduce.</div> <div> <div><br> </div> <div><br> </div> <div>[0] <a href="https://github.com/rpm-software-management/createrepo_c/pull/231" id="gmail-m_-8659470818859977245LPlnk544080" target="_blank"> https://github.com/rpm-software-management/createrepo_c/pull/231</a></div> <div>[1] <a href="https://github.com/rpm-software-management/createrepo_c/pull/233" id="gmail-m_-8659470818859977245LPlnk845258" target="_blank"> https://github.com/rpm-software-management/createrepo_c/pull/233</a></div> </div> </div> <br> <div class="gmail_quote"> <div dir="ltr" class="gmail_attr">On Wed, Nov 4, 2020 at 12:40 PM Winberg Adam <<a href="mailto:Adam.Winberg@smhi.se" target="_blank">Adam.Winberg@smhi.se</a>> wrote:<br> </div> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div dir="ltr"> <div id="gmail-m_-8659470818859977245gmail-m_-5190763567011880832gmail-m_-2530184192362103184divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif"> <p>Hi, </p> <p><br> </p> <p>running a 2to3 migration with <span>2to3-migration-0.5.1 seems to consume a whole lot more memory than previous versions. My 12G RAM was quickly spent, i increased to 16G which wasnt enough either. Earlier migrations with <span>0.5.0 didnt spend anywhere near that amount. </span></span></p> <p><span><span><br> </span></span></p> <p><span style="font-size:12pt">Also - the migration fails with </span><br> </p> <p><span><span><span>OSError: [Errno 23] Too many open files in system: ..</span><br> </span></span></p> <p><span><span><span><br> </span></span></span></p> <p><span><span><span><span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;font-size:16px">The memory usage increases while running the '</span><span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;font-size:16px">Migrating rpm content to Pulp 3 rpm' subtask. With 16G RAM I only get to about 114000/152000 pkgs in that task before the memory is more or less all consumed and the OSError appears. </span><br> </span></span></span></p> <p><br> </p> <p>So it seems to me that there is some type of regression here. Any pointers on how I can further debug or work around this? </p> <p><br> </p> <p>This is on RHEL8 with </p> <p></p> <div>python3-pulp-rpm-3.7.0-1.el8.noarch</div> <div>python3-pulpcore-3.7.3-1.el8.noarch</div> <div>python3-pulp-2to3-migration-0.5.1-1.el8.noarch</div> <div><span style="font-size:12pt"><br> </span></div> <div><span style="font-size:12pt">//Adam</span><br> </div> <p></p> </div> </div> _______________________________________________<br> Pulp-list mailing list<br> <a href="mailto:Pulp-list@redhat.com" target="_blank">Pulp-list@redhat.com</a><br> <a href="https://www.redhat.com/mailman/listinfo/pulp-list" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-list</a></blockquote> </div> </div> </div> </div> </div> </div> </div> </div> </blockquote></div>