[Pulp-dev] S3 support for development/testing

[Fabricio Aguiar]

I'm having some trouble for making tests working on Travis,
For *minio*, I have some problems with the connections pool:
urllib3.exceptions.ProtocolError: ('Connection aborted.',
RemoteDisconnected('Remote end closed connection without response'))
travis:
https://travis-ci.org/pulp/pulp_file/jobs/654626953?utm_medium=notification&utm_source=github_status
code: https://github.com/pulp/pulp_file/pull/358/files

So I tried *localstack*, it is not clear on travis, but locally I can see
the bucket has some lifetime, and it dies before the tests finish (I
believe it happens when we are waiting for tasks to finish):
travis: https://travis-ci.com/fabricio-aguiar/pulp_file/jobs/290713046
code:
https://github.com/pulp/pulp_file/compare/master...fabricio-aguiar:6155.2

Best regards,
Fabricio Aguiar
Software Engineer, Pulp Project
Red Hat Brazil - Latam <https://www.redhat.com/>
+55 11 999652368


On Wed, Feb 19, 2020 at 1:17 PM Mike DePaulo <mikedep333 at redhat.com> wrote:

> Sounds good.
>
> See in-line replies.
>
> On Tue, Feb 18, 2020 at 2:31 PM Brian Bouterse <bmbouter at redhat.com>
> wrote:
>
>> Thanks! This story sounds great; please let me know how I can help.
>>
>> On Tue, Feb 18, 2020 at 2:28 PM David Davis <daviddavis at redhat.com>
>> wrote:
>>
>>> Yea, good catch. I meant having plugin_template test against minio and
>>> not s3.
>>>
>>> David
>>>
>>>
>>> On Tue, Feb 18, 2020 at 2:22 PM Brian Bouterse <bmbouter at redhat.com>
>>> wrote:
>>>
>>>> This testing is much needed, so I'm glad to hear it's progressing.
>>>> Thank you for organizing. I put some questions inline. +1 to prioritizing
>>>> it until it's in place.
>>>>
>>>> On Mon, Feb 17, 2020 at 3:47 PM David Davis <daviddavis at redhat.com>
>>>> wrote:
>>>>
>>>>> Recently we discovered that S3 support was broken in Pulp 3.1[0].
>>>>> Since we have at least one stakeholder using S3 in production, we've
>>>>> decided that we need to make it easier to develop and test our code against
>>>>> S3. To that end, we have filed an epic for this work and are looking for
>>>>> feedback:
>>>>>
>>>>> https://pulp.plan.io/issues/5961
>>>>>
>>>>> A few highlights/questions:
>>>>> - We plan to use minio since one of our team members has experience
>>>>> with it
>>>>>
>>>> I saw the script posted here:
> https://pulp.plan.io/issues/6155
> I am OK with us using a script indefinitely.
>
> - We'll add a plugin_template config value that will run the functional
>>>>> tests against S3. I think this should probably be a daily cron job.
>>>>>
>>>> Do you mean minio hosting S3 or actually S3?
>>>>
>>>
> The script as it is right now seems to be very reliant on conventional
> pulp installs, and will not work with the Travis host / pulp-operator
> containers. It uses "prestart", modifies /etc/cpulp/settings.py, etc.
> It *might* be faster just to use minio with the operator:
> https://docs.min.io/docs/deploy-minio-on-kubernetes.html
>
>>
>>>> - I think the pulp-devel role should always install and configure minio
>>>>> but leave the settings commented out in /etc/pulp/settings.py. That way, we
>>>>> use filesystem storage by default but developers don't have to recreate
>>>>> their dev environments if they want to switch over to S3 quickly to test
>>>>> something out.
>>>>>
>>>> This sounds great
>>>>
>>> Fine with me. We can call the script as a task, and add an option to
> call it or not.
>
>>
>>>>> Feedback is welcome.
>>>>>
>>>>
>>>>> [0] https://pulp.plan.io/issues/6154
>>>>>
>>>>> David
>>>>> _______________________________________________
>>>>> Pulp-dev mailing list
>>>>> Pulp-dev at redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>
>>>> _______________________________________________
>> Pulp-dev mailing list
>> Pulp-dev at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
> --
>
> Mike DePaulo
>
> He / Him / His
>
> Service Reliability Engineer, Pulp
>
> Red Hat <https://www.redhat.com/>
>
> IM: mikedep333
>
> GPG: 51745404
> <https://www.redhat.com/>
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20200227/84446933/attachment.htm>