<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hi David,<br>
</p>
<div class="moz-cite-prefix">On 5/07/19 16:27, David Davis wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAHa=2Wk0wPM5psJqH=pb=v9Ts33mRF0UKco5grLRudmNXm5U9g@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">I tested and confirmed that the files are not being
deleted. Mind opening a bug for that?</div>
</blockquote>
I created an issue <a class="moz-txt-link-freetext" href="https://pulp.plan.io/issues/5092">https://pulp.plan.io/issues/5092</a><br>
<blockquote type="cite"
cite="mid:CAHa=2Wk0wPM5psJqH=pb=v9Ts33mRF0UKco5grLRudmNXm5U9g@mail.gmail.com">
<div dir="ltr">
<div><br>
</div>
<div>
<div>For the failed chunk uploads, httpie is timing out. You
can set a higher timeout (like you did) or use smaller
chunks.</div>
</div>
</div>
</blockquote>
In my case, for uploading the 300MB chunks I have no problem. Is
when creating the artifact that fails.<br>
<blockquote type="cite"
cite="mid:CAHa=2Wk0wPM5psJqH=pb=v9Ts33mRF0UKco5grLRudmNXm5U9g@mail.gmail.com">
<div dir="ltr">
<div>
<div><br>
</div>
<div>
<div>
<div dir="ltr"
class="m_-8561388604270746870gmail_signature"
data-smartmail="gmail_signature">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>I looked into why artifact creation is
failing for files < 2GB. The reason is that
it's taking too long to calculate the checksums.
There are 6 checksum types and each one takes
about 4-8 seconds from the command line in my
test environment. Calculating the digests in
Python seems to add about 1-2 seconds. The
default timeout in gunicorn is 30 seconds:</div>
<div><br>
</div>
<div>Jul 05 14:21:56 pulp3 gunicorn[13691]:
[2019-07-05 14:21:56 +0000] [13691] [CRITICAL]
WORKER TIMEOUT (pid:29843)<br>
Jul 05 14:21:57 pulp3 gunicorn[13691]:
[2019-07-05 14:21:57 +0000] [30031] [INFO]
Booting worker with pid: 30031<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
I see the same error. I have a VM with 2 CPU and 2 workers. Do I
need to add more workers?<br>
<blockquote type="cite"
cite="mid:CAHa=2Wk0wPM5psJqH=pb=v9Ts33mRF0UKco5grLRudmNXm5U9g@mail.gmail.com">
<div dir="ltr">
<div>
<div>
<div>
<div dir="ltr"
class="m_-8561388604270746870gmail_signature"
data-smartmail="gmail_signature">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div><br>
</div>
<div>You can raise this timeout or also you can
pass in the checksums when creating the
artifact[0]. I think the best solution though
might be to make artifact creation a background
task.</div>
<div><br>
</div>
<div>[0] http POST :24817/pulp/api/v3/artifacts/
upload=$UPLOAD sha256=abc...</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<p>I passed the sha256 but still have a WORKER TIMEOUT problem. I
suppose it has to calculate also md5, sha1, sha224, sha384, sha512
which takes time.</p>
<p>Where can I increase the WORKER TIMEOUT ?</p>
<p>Juan<br>
</p>
<br>
<blockquote type="cite"
cite="mid:CAHa=2Wk0wPM5psJqH=pb=v9Ts33mRF0UKco5grLRudmNXm5U9g@mail.gmail.com">
<div dir="ltr">
<div>
<div>
<div>
<div dir="ltr"
class="m_-8561388604270746870gmail_signature"
data-smartmail="gmail_signature">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div><br>
</div>
<div>David<br>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Jul 5, 2019 at 9:08 AM
Juan Cabrera <<a href="mailto:juan.cabrera@unamur.be"
target="_blank" moz-do-not-send="true">juan.cabrera@unamur.be</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi David</p>
<p>This morning I made a test sequence and open a ticket</p>
<p><a
class="gmail-m_-8561388604270746870gmail-m_-4651838809158492575moz-txt-link-freetext"
href="https://pulp.plan.io/issues/5087" target="_blank"
moz-do-not-send="true">https://pulp.plan.io/issues/5087</a><br>
</p>
<p>Yes, I see that the update file is deleted when the
artifact is created. <br>
</p>
<p>When I get the artifact creation error, the upload HREF
is not deleted, which could be normal as there was a
error.</p>
<p>But when I clean all the server uploads using API <br>
</p>
<p><tt>for u in $(http $PORT/pulp/api/v3/uploads/ | jq -r
'.results[] | ._href'); do</tt><tt><br>
</tt><tt> echo $u</tt><tt><br>
</tt><tt> http DELETE $PORT$u</tt><tt><br>
</tt><tt>done</tt><br>
</p>
<p>The files are not deleted.</p>
<p>Juan<br>
</p>
<div
class="gmail-m_-8561388604270746870gmail-m_-4651838809158492575moz-cite-prefix">On
5/07/19 12:44, David Davis wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Juan,
<div><br>
</div>
<div>There is in fact a 2GB limit currently on artifact
size. I consider this a bug and I filed this issue:</div>
<div><br>
</div>
<div><a href="https://pulp.plan.io/issues/4998"
target="_blank" moz-do-not-send="true">https://pulp.plan.io/issues/4998</a></div>
<div><br>
</div>
<div>The file in /var/lib/pulp/upload should be deleted
once it's imported as an artifact. I'm guessing it's
maybe not happening since the server is throwing an
error.<br clear="all">
<div>
<div dir="ltr"
class="gmail-m_-8561388604270746870gmail-m_-4651838809158492575gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div><br>
</div>
<div>David<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, Jul 4, 2019 at
12:31 PM Juan Cabrera <<a
href="mailto:juan.cabrera@unamur.be" target="_blank"
moz-do-not-send="true">juan.cabrera@unamur.be</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px
0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
In my previous mail I forget to say that I updated the
Pulp version to<br>
3.0.0rc3<br>
<br>
pulp_source_dir: "git+<a
href="https://github.com/pulp/pulpcore.git@3.0.0rc3"
rel="noreferrer" target="_blank"
moz-do-not-send="true">https://github.com/pulp/pulpcore.git@3.0.0rc3</a>"<br>
pulp_plugin_source_dir:<br>
"git+<a
href="https://github.com/pulp/pulpcore-plugin.git@0.1.0rc3"
rel="noreferrer" target="_blank"
moz-do-not-send="true">https://github.com/pulp/pulpcore-plugin.git@0.1.0rc3</a>"<br>
pulp_install_plugins:<br>
pulp-rpm:<br>
app_label: "rpm"<br>
source_dir: "git+<a
href="https://github.com/pulp/pulp_rpm.git@3.0.0b4"
rel="noreferrer" target="_blank"
moz-do-not-send="true">https://github.com/pulp/pulp_rpm.git@3.0.0b4</a>"<br>
<br>
Juan<br>
<br>
_______________________________________________<br>
Pulp-list mailing list<br>
<a href="mailto:Pulp-list@redhat.com" target="_blank"
moz-do-not-send="true">Pulp-list@redhat.com</a><br>
<a
href="https://www.redhat.com/mailman/listinfo/pulp-list"
rel="noreferrer" target="_blank"
moz-do-not-send="true">https://www.redhat.com/mailman/listinfo/pulp-list</a></blockquote>
</div>
</blockquote>
<div
class="gmail-m_-8561388604270746870gmail-m_-4651838809158492575moz-signature">--
<br>
<div
style="font-family:Verdana,Helvetica,Arial,sans-serif;font-size:8pt;font-weight:normal;color:rgb(97,99,101)">
<img src="cid:16bc260db83f24ddb3f1" style="display:
block; margin: 1em 0em;" moz-do-not-send="true">
<p style="margin:1em"> <span style="font-weight:bold">Juan
CABRERA</span><br>
<span style="color:rgb(105,190,40)">Correspondant
informatique</span><br>
<span>Département de Mathématiques</span> </p>
<p style="margin:1em"> <span style="font-weight:bold">T.</span>
081724919<br>
<a style="color:rgb(105,190,40)"
href="mailto:juan.cabrera@unamur.be" target="_blank"
moz-do-not-send="true">juan.cabrera@unamur.be</a><br>
<a style="color:rgb(105,190,40)"
href="http://staff.unamur.be/jbcabrer"
target="_blank" moz-do-not-send="true">http://staff.unamur.be/jbcabrer</a>
</p>
<p style="margin:1em"> <span
style="font-weight:bold;font-size:8pt">Université de
Namur ASBL</span><br>
<span>Rue de Bruxelles 61 - 5000 Namur</span><br>
<span>Belgique</span><br>
</p>
<p
style="margin:1em;font-size:6pt;color:rgb(105,190,40)">
<span>Let’s respect the environment together.</span><br>
<span style="font-weight:bold">Only print this message
if necessary!</span><br>
</p>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
<div class="moz-signature">-- <br>
<div style="font-family: Verdana, Helvetica, Arial, sans-serif;
font-size: 8pt; font-weight: normal; color: #616365;"> <img
src="cid:part13.34A6ADFC.D8242833@unamur.be" style="display:
block; margin: 1em 0em;">
<p style="margin: 1em;"> <span style="font-weight: bold;">Juan
CABRERA</span><br>
<span style="color: #69BE28;">Correspondant informatique</span><br>
<span>Département de Mathématiques</span> </p>
<p style="margin: 1em;">
<span style="font-weight: bold;">T.</span> 081724919<br>
<a style="color: #69BE28;"
href="mailto:juan.cabrera@unamur.be">juan.cabrera@unamur.be</a><br>
<a style="color: #69BE28;"
href="http://staff.unamur.be/jbcabrer">http://staff.unamur.be/jbcabrer</a>
</p>
<p style="margin: 1em;"> <span style="font-weight: bold;
font-size: 8pt;">Université de Namur ASBL</span><br>
<span>Rue de Bruxelles 61 - 5000 Namur</span><br>
<span>Belgique</span><br>
</p>
<p style="margin: 1em; font-size: 6pt; color: #69BE28;"> <span>Let’s
respect the environment together.</span><br>
<span style="font-weight: bold;">Only print this message if
necessary!</span><br>
</p>
</div>
</div>
</body>
</html>