<div dir="ltr">Thanks for sending this nice summary. Thoughts in-line.<br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, May 9, 2017 at 6:03 PM, Tatiana Tereshchenko <span dir="ltr"><<a href="mailto:ttereshc@redhat.com" target="_blank">ttereshc@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div><div><div>Currently Pulp is able to import RPM with filelist up to ~14-15 MB which probably cover most repositories but not all of them.<br><br></div><div>Historically, for each RPM unit several potentially large data snippets are stored in db:<br> - XML snippets for RPM metadata<br></div><div> - parsed filelist<br></div><div> - parsed changelog<br><br></div><div>XML snippets are compressed and so they require much less space than a huge parsed filelist or a changelog.<br></div>Here is the issue [0] to track the effort of eliminating this limitation or at least increasing the size of filelist that Pulp can handle for each RPM.<br><br></div><div>The question is what is the best way to handle the issue, keeping in mind that any substantial change or re-design introduces more risks and efforts to Pulp 2 line and at the same time this won't be an issue in Pulp 3.<br></div></div></div></div></div></div></div></div></blockquote><div><br></div><div>And to emphasize a certain perspective, this limitation has been present for most or all of Pulp's existence.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div><div></div><div><br></div>So far the options are:<br></div> 1. Eliminate issue completely (e.g. by using GridFS)<br></div></div></div></div></div></div></blockquote><div><br></div><div>If we were sticking with mongodb long-term, this would likely be the best path. But it's a whole new area of uncertainty. I'm very hesitant to "rock the boat" with Pulp 2 at this point.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div></div> 2. Increase current limit for filelist by removing parsed version of it from db<br></div></div></div></div></div></blockquote><div><br></div><div>Probably nobody would miss it, but it would be unfortunate to change this in a Y release.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div></div> 3. Do not solve it in Pulp2, wait for Pulp3 which won't have this issue at all<br></div></div></div></div></blockquote><div><br></div><div>There doesn't seem to be any urgency from our users, so waiting for Pulp 3 should be acceptable.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div></div> 4. Any other idea/option<br></div></div></div></blockquote><div><br></div><div>We might be able to do this:</div><div>- try to save a new RPM</div><div>- if it fails with the document too large error, set the filelist and changelog to None</div><div>- try to save it again</div><div><br></div><div>This would likely be low-risk, and it would mean only newly-added content would be missing that particular data from the database. We suspect that nobody uses those fields anyway, so it would probably be fine. Although I'd hate to find out that it broke something. And schema divergence itself adds complexity; we'd need to take this into account when migrating data to Pulp3.</div><div> </div><div>On the whole, I think waiting to fix this in Pulp 3 is the best option given what we know. This is just one of many limitations we'll be able to break free of with Pulp 3, and that can't come soon enough.</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><br></div>As an additional info:<br></div> - some thoughts and options [1] which were considered several months ago<br><div><div><div><div><div><div><div><div><div><div><div> - by removing parsed filelist (and changelog?) from db we will give a room for a really large RPM metadata. Pulp will be able to import any RPM with uncompressed metadata up to ~200MB (~14-15MB currently). Just for comparison, this is ~1.5 times bigger than the filelists.xml and other.xml together of the whole EPEL7 repo.<br></div><div> - removing data from db ^ will affect at least search endpoints like this [2] where all the data for unit is returned in response.<br></div><div><br><div><div><div><div><div>[0] <a href="https://pulp.plan.io/issues/2747" target="_blank">https://pulp.plan.io/issues/<wbr>2747</a><br>[1] <a href="https://etherpad.net/p/mongodb_DocumentTooLarge" target="_blank">https://etherpad.net/p/<wbr>mongodb_DocumentTooLarge</a><br>[2] <a href="http://docs.pulpproject.org/dev-guide/integration/rest-api/repo/content.html#advanced-unit-search" target="_blank">http://docs.pulpproject.org/<wbr>dev-guide/integration/rest-<wbr>api/repo/content.html#<wbr>advanced-unit-search</a></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></blockquote></div><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><p style="color:rgb(0,0,0);font-family:overpass-mono,monospace;font-size:10px;margin:0px!important;padding:0px!important"><span style="margin:0px!important;padding:0px!important">Michael</span> <span style="margin:0px!important;padding:0px!important">Hrivnak</span></p><p style="color:rgb(0,0,0);font-family:overpass-mono,monospace;font-size:10px;margin:0px!important;padding:0px!important"></p><span style="color:rgb(0,0,0);font-family:overpass-mono,monospace;font-size:10px;margin:0px!important;padding:0px!important"><span style="margin:0px!important;padding:0px!important">Principal Software Engineer</span><span style="margin:0px!important;padding:0px!important">, <span style="margin:0px!important;padding:0px!important">RHCE</span></span> </span><span style="color:rgb(0,0,0);font-family:overpass-mono,monospace;font-size:10px"></span><br style="color:rgb(0,0,0);font-family:overpass-mono,monospace;font-size:10px;margin:0px!important;padding:0px!important"><p style="color:rgb(0,0,0);font-family:overpass-mono,monospace;font-size:10px;margin:0px!important;padding:0px!important">Red Hat</p></div></div> </div></div>