[Pulp-list] advice for extracting more information from 'Forbidden' during sync jobs?

[Kodiak Firesmith]

Hi Brian, thanks for the reply.  Sadly the logs don't show much:
https://paste.fedoraproject.org/paste/0T5YEdTquIWW2Avhy~J6NQ

I did get the idea since sending this email this morning to use curl to
investigate the problem.  The results make me think that there is an
upstream problem with cdn.redhat.com (or suddenly my entitlements work for
some things but not others) - check this out:

GOOD CDN:  https://paste.fedoraproject.org/paste/YJaOryGkExSfgNr9WPF9kg

BAD CDN:  https://paste.fedoraproject.org/paste/AhZtnKONgqdVgJxlAVKwCw

The 'X-Cache' tag contents seem intriguing but I'm certainly no CDN /
Akamai guy and don't know what I'm talking about so....

Maybe the @Redhat folks in the room know someone on the CDN side of
things?

Potentially "broken" repos are:
  https://cdn.redhat.com/content/beta/rhel/server/7/x86_64/rhscl/1/os
  https://cdn.redhat.com/content/beta/rhel/workstation/7/x86_64/rhscl/1/os
  https://cdn.redhat.com/content/els/rhel/server/5/5Server/x86_64/os
  https://cdn.redhat.com/content/els/rhel/server/5/5Server/i386/os

In the mean time now that I have a proven way to interrogate via curl
whether or not my client certs are returning content to pulp, I'm going to
go ahead and re-generate a new entitlement certificate bundle since all 4
repos are special repo types (beta, ELS).  I do know that at least the ELS
repos use different PEM files for access since I get an els.pem file in the
bundle.

Thanks!
 - Kodiak

On Mon, Apr 2, 2018 at 2:03 PM, Brian Bouterse <bbouters at redhat.com> wrote:

> Did the log show anything? I think if the task has a fatal exception it
> logs it.
>
> If you don't see any errors, maybe patch the code to add some log
> statements? Maybe someone could link to the area where a 403 error would be
> handled?
>
> On Mon, Apr 2, 2018 at 11:04 AM, Kodiak Firesmith <kfiresmith at gmail.com>
> wrote:
>
>> Hi Pulp People,
>> I suspect Red Hat is having some limited trouble with their CDN lately as
>> 2 Red Hat CDN repos started throwing forbiddens last week, and 2 additional
>> repos started throwing forbiddens over the weekend for a total of 4 out of
>> perhaps about 80 Red Hat CDN repos currently showing forbidden during sync.
>>
>> The biggest problem is that it happens later into the sync and even with
>> TCPDump running it all happens within a TLS v1.2 tunnel so I can't see what
>> the remote origin of the 403 is.  Pulp is currently "too helpful" in that
>> it completely obfuscates paths and response codes, even during '-vvv'
>> runs.
>>
>> Here's an example:
>>
>>  https://cdn.redhat.com/content/beta/rhel/server/7/x86_64/rhscl/1/os
>>
>> $pulp-admin  rpm repo sync run --repo-id=rhel-server-rhscl-7-beta-rpms
>> +----------------------------------------------------------------------+
>>         Synchronizing Repository [rhel-server-rhscl-7-beta-rpms]
>> +----------------------------------------------------------------------+
>>
>> This command may be exited via ctrl+c without affecting the request.
>>
>>
>> Downloading metadata...
>> [\]
>> ... completed
>>
>> Downloading repository content...
>> [-]
>> [==================================================] 100%
>> RPMs:       0/0 items
>> Delta RPMs: 0/0 items
>>
>> ... completed
>>
>> Downloading distribution files...
>> [==================================================] 100%
>> Distributions: 0/0 items
>>
>> Task Failed
>>
>> Error retrieving metadata: Forbidden
>>
>>
>> If anyone has any advice, that would be rad.  Thanks all!
>>
>>  - Kodiak
>>
>> _______________________________________________
>> Pulp-list mailing list
>> Pulp-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-list
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20180402/52044faf/attachment.htm>