[Pulp-list] advice for extracting more information from 'Forbidden' during sync jobs?

[Brian Bouterse]

Maybe those repos are requiring a cert type that the crypto library Pulp
uses isn't able to handle correctly? Did they work before? It would be
interesting to see if regenerating the certs resolves it.

On Mon, Apr 2, 2018 at 2:31 PM, Kodiak Firesmith <kfiresmith at gmail.com>
wrote:

> Hi Brian, thanks for the reply.  Sadly the logs don't show much:
> https://paste.fedoraproject.org/paste/0T5YEdTquIWW2Avhy~J6NQ
>
> I did get the idea since sending this email this morning to use curl to
> investigate the problem.  The results make me think that there is an
> upstream problem with cdn.redhat.com (or suddenly my entitlements work
> for some things but not others) - check this out:
>
> GOOD CDN:  https://paste.fedoraproject.org/paste/YJaOryGkExSfgNr9WPF9kg
>
> BAD CDN:  https://paste.fedoraproject.org/paste/AhZtnKONgqdVgJxlAVKwCw
>
> The 'X-Cache' tag contents seem intriguing but I'm certainly no CDN /
> Akamai guy and don't know what I'm talking about so....
>
> Maybe the @Redhat folks in the room know someone on the CDN side of
> things?
>
> Potentially "broken" repos are:
>   https://cdn.redhat.com/content/beta/rhel/server/7/x86_64/rhscl/1/os
>   https://cdn.redhat.com/content/beta/rhel/workstation/7/x86_64/rhscl/1/os
>   https://cdn.redhat.com/content/els/rhel/server/5/5Server/x86_64/os
>   https://cdn.redhat.com/content/els/rhel/server/5/5Server/i386/os
>
> In the mean time now that I have a proven way to interrogate via curl
> whether or not my client certs are returning content to pulp, I'm going to
> go ahead and re-generate a new entitlement certificate bundle since all 4
> repos are special repo types (beta, ELS).  I do know that at least the ELS
> repos use different PEM files for access since I get an els.pem file in the
> bundle.
>
> Thanks!
>  - Kodiak
>
> On Mon, Apr 2, 2018 at 2:03 PM, Brian Bouterse <bbouters at redhat.com>
> wrote:
>
>> Did the log show anything? I think if the task has a fatal exception it
>> logs it.
>>
>> If you don't see any errors, maybe patch the code to add some log
>> statements? Maybe someone could link to the area where a 403 error would be
>> handled?
>>
>> On Mon, Apr 2, 2018 at 11:04 AM, Kodiak Firesmith <kfiresmith at gmail.com>
>> wrote:
>>
>>> Hi Pulp People,
>>> I suspect Red Hat is having some limited trouble with their CDN lately
>>> as 2 Red Hat CDN repos started throwing forbiddens last week, and 2
>>> additional repos started throwing forbiddens over the weekend for a total
>>> of 4 out of perhaps about 80 Red Hat CDN repos currently showing forbidden
>>> during sync.
>>>
>>> The biggest problem is that it happens later into the sync and even with
>>> TCPDump running it all happens within a TLS v1.2 tunnel so I can't see what
>>> the remote origin of the 403 is.  Pulp is currently "too helpful" in that
>>> it completely obfuscates paths and response codes, even during '-vvv'
>>> runs.
>>>
>>> Here's an example:
>>>
>>>  https://cdn.redhat.com/content/beta/rhel/server/7/x86_64/rhscl/1/os
>>>
>>> $pulp-admin  rpm repo sync run --repo-id=rhel-server-rhscl-7-beta-rpms
>>> +----------------------------------------------------------------------+
>>>         Synchronizing Repository [rhel-server-rhscl-7-beta-rpms]
>>> +----------------------------------------------------------------------+
>>>
>>> This command may be exited via ctrl+c without affecting the request.
>>>
>>>
>>> Downloading metadata...
>>> [\]
>>> ... completed
>>>
>>> Downloading repository content...
>>> [-]
>>> [==================================================] 100%
>>> RPMs:       0/0 items
>>> Delta RPMs: 0/0 items
>>>
>>> ... completed
>>>
>>> Downloading distribution files...
>>> [==================================================] 100%
>>> Distributions: 0/0 items
>>>
>>> Task Failed
>>>
>>> Error retrieving metadata: Forbidden
>>>
>>>
>>> If anyone has any advice, that would be rad.  Thanks all!
>>>
>>>  - Kodiak
>>>
>>> _______________________________________________
>>> Pulp-list mailing list
>>> Pulp-list at redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-list
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20180403/f248e81f/attachment.htm>