[Pulp-list] artifact creation using chunk upload of big package

David Davis daviddavis at redhat.com
Fri Jul 5 14:27:02 UTC 2019


I tested and confirmed that the files are not being deleted. Mind opening a
bug for that?

For the failed chunk uploads, httpie is timing out. You can set a higher
timeout (like you did) or use smaller chunks.

I looked into why artifact creation is failing for files < 2GB. The reason
is that it's taking too long to calculate the checksums. There are 6
checksum types and each one takes about 4-8 seconds from the command line
in my test environment. Calculating the digests in Python seems to add
about 1-2 seconds. The default timeout in gunicorn is 30 seconds:

Jul 05 14:21:56 pulp3 gunicorn[13691]: [2019-07-05 14:21:56 +0000] [13691]
[CRITICAL] WORKER TIMEOUT (pid:29843)
Jul 05 14:21:57 pulp3 gunicorn[13691]: [2019-07-05 14:21:57 +0000] [30031]
[INFO] Booting worker with pid: 30031

You can raise this timeout or also you can pass in the checksums when
creating the artifact[0]. I think the best solution though might be to make
artifact creation a background task.

[0] http POST :24817/pulp/api/v3/artifacts/ upload=$UPLOAD sha256=abc...

David


On Fri, Jul 5, 2019 at 9:08 AM Juan Cabrera <juan.cabrera at unamur.be> wrote:

> Hi David
>
> This morning I made a test sequence and open a ticket
>
> https://pulp.plan.io/issues/5087
>
> Yes, I see that the update file is deleted when the artifact is created.
>
> When I get the artifact creation error, the upload HREF is not deleted,
> which could be normal as there was a error.
>
> But when I clean all the server uploads using API
>
> for u in $(http $PORT/pulp/api/v3/uploads/ | jq -r '.results[] | ._href');
> do
>     echo $u
>     http DELETE $PORT$u
> done
>
> The files are not deleted.
>
> Juan
> On 5/07/19 12:44, David Davis wrote:
>
> Juan,
>
> There is in fact a 2GB limit currently on artifact size. I consider this a
> bug and I filed this issue:
>
> https://pulp.plan.io/issues/4998
>
> The file in /var/lib/pulp/upload should be deleted once it's imported as
> an artifact. I'm guessing it's maybe not happening since the server is
> throwing an error.
>
> David
>
>
> On Thu, Jul 4, 2019 at 12:31 PM Juan Cabrera <juan.cabrera at unamur.be>
> wrote:
>
>> Hi,
>>
>> In my previous mail I forget to say that I updated the Pulp version to
>> 3.0.0rc3
>>
>> pulp_source_dir: "git+https://github.com/pulp/pulpcore.git@3.0.0rc3"
>> pulp_plugin_source_dir:
>> "git+https://github.com/pulp/pulpcore-plugin.git@0.1.0rc3"
>> pulp_install_plugins:
>>   pulp-rpm:
>>     app_label: "rpm"
>>     source_dir: "git+https://github.com/pulp/pulp_rpm.git@3.0.0b4"
>>
>> Juan
>>
>> _______________________________________________
>> Pulp-list mailing list
>> Pulp-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-list
>
> --
>
> Juan CABRERA
> Correspondant informatique
> Département de Mathématiques
>
> T. 081724919
> juan.cabrera at unamur.be
> http://staff.unamur.be/jbcabrer
>
> Université de Namur ASBL
> Rue de Bruxelles 61 - 5000 Namur
> Belgique
>
> Let’s respect the environment together.
> Only print this message if necessary!
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20190705/a36d2b85/attachment.htm>


More information about the Pulp-list mailing list