[Pulp-list] Issues with using S3 storage when running pulp on Amazon EC2 (pulp3)
Mike DePaulo
mikedep333 at redhat.com
Wed Nov 6 15:52:18 UTC 2019
Hi Joey,
It sounds like aws_default_acl should be documented here then:
https://docs.pulpproject.org/en/3.0/nightly/installation/storage.html?highlight=aws
Care to submit a documentation PR?
https://github.com/pulp/pulpcore/blob/master/docs/installation/storage.rst
Thanks,
-Mike
On Wed, Nov 6, 2019 at 9:07 AM Dumont, Joey <Joey.Dumont at nrc-cnrc.gc.ca>
wrote:
> Turns out the issue was on my end. I had to add aws_default_acl: None to
> the pulp_settings section of the playbook. The public-read ACL was
> incompatible with the BlockPublicAccess settings that I had set on my S3
> bucket.
>
> I'm now encountering a different issue, but I'll start another thread for
> that one.
>
> Thanks for the pointers, they were very helpful!
>
> Joey Dumont
>
> Technical Advisor, Knowledge, Information, and Technology Services
> National Research Council Canada / Governement of Canada
> joey.dumont at nrc-cnrc.gc.ca / Tel: 613-990-8152 / Cell: 438-340-7436
>
> Conseiller technique, Services du savoir, de l'information et de la
> technologie
> Conseil national de recherches Canada / Gouvernement du Canada
> joey.dumont at nrc-cnrc.gc.ca / Tél.: 613-990-8152 / Tél. cell.: 438-340-7436
> ------------------------------
> *From:* David Davis <daviddavis at redhat.com>
> *Sent:* 01 November 2019 15:51
> *To:* Dumont, Joey
> *Cc:* pulp-list at redhat.com
> *Subject:* Re: [Pulp-list] Issues with using S3 storage when running pulp
> on Amazon EC2 (pulp3)
>
> Unfortunately I don't know of a good way to debug the problem other than
> to dig into the code. If you want to debug from the Pulp code, you could
> stick a debugger in the artifact saver stage:
>
>
> https://github.com/pulp/pulpcore/blob/2203fee1407738a4ddd8e644fcbc741aab0bca63/pulpcore/plugin/stages/artifact_stages.py#L179-L200
>
> What I would probably do though is stick a debug statement here in
> django-storages to see what params it's passing to boto3:
>
>
> https://github.com/jschneier/django-storages/blob/0ab2b1e3efd2bcaf0f24540a718993acc7742d9b/storages/backends/s3boto3.py#L511
>
> You can see the location of django-storages with `pip show
> django-storages`.
>
> Sorry I don't have a better answer for you. Perhaps this is something we
> can improve in the future. Also, I'd be curious as to what the issue is as
> it sounds like everything should work in theory.
>
> David
>
>
> On Fri, Nov 1, 2019 at 2:26 PM Dumont, Joey <Joey.Dumont at nrc-cnrc.gc.ca>
> wrote:
>
>> I've installed the latest pulp3 using the Ansible installer using the
>> following playbook:
>>
>>
>> ---
>> - hosts: mirrors
>> vars:
>> prereq_pip_packages:
>> - django-storages
>> - boto3
>> pulp_use_system_wide_pkgs: True
>> pulp_default_admin_password: !vault |
>> $ANSIBLE_VAULT;1.1;AES256
>> ...
>> pulp_settings:
>> secret_key: !vault |
>> $ANSIBLE_VAULT;1.1;AES256
>> ...
>> default_file_storage: 'storages.backends.s3boto3.S3Boto3Storage'
>> aws_storage_bucket_name: 'xxx-pulp-storage'
>> aws_s3_region_name: 'ca-central-1'
>> aws_s3_addressing_style: "path"
>> media_root: '/pulp3/'
>> pulp_install_plugins:
>> pulp-file: {}
>> pulp-rpm:
>> prereq_role: "pulp.pulp_rpm_prerequisites"
>> # pulp-docker: {}
>> roles:
>> - pulp-database
>> - pulp-workers
>> - pulp-resource-manager
>> - pulp-webserver
>> - pulp-content
>> environment:
>> DJANGO_SETTINGS_MODULE: pulpcore.app.settings
>>
>> I also set up an RPM repo that uses S3 for storage. However, when I try
>> to sync, I get an AccessDenied error. I know the instance profile is
>> correct, as I can upload objects from that instance using both the AWS CLI
>> and Boto3 without specifying credentials.
>>
>> How can I debug this further? Is there a way for me know what parameters
>> are passed to the put_object boto3 call by the sync task?
>>
>> Cheers,
>>
>>
>>
>> Joey Dumont
>>
>> Technical Advisor, Knowledge, Information, and Technology Services
>> National Research Council Canada / Governement of Canada
>> joey.dumont at nrc-cnrc.gc.ca / Tel: 613-990-8152 / Cell: 438-340-7436
>>
>> Conseiller technique, Services du savoir, de l'information et de la
>> technologie
>> Conseil national de recherches Canada / Gouvernement du Canada
>> joey.dumont at nrc-cnrc.gc.ca / Tél.: 613-990-8152 / Tél. cell.:
>> 438-340-7436
>> _______________________________________________
>> Pulp-list mailing list
>> Pulp-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-list
>
> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
--
Mike DePaulo
He / Him / His
Service Reliability Engineer, Pulp
Red Hat <https://www.redhat.com/>
IM: mikedep333
GPG: 51745404
<https://www.redhat.com/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20191106/9ff48970/attachment.htm>
More information about the Pulp-list
mailing list