[Pulp-list] 500 Internal Server

Eric VS vs.eric at gmail.com
Tue Apr 13 02:46:25 UTC 2021 

Hi Gerrod,

Update.... I found how to do the upgrade... ansible-galaxy collections
install ... --force-with-deps. Now I'm running 3.12 but get the following
error.

╰─$ (⎈ k8s01.aops-dev.activeops.local:kube-system) pulp status


                                         127 ↵
Notice: Cached api is outdated. Refreshing...
{
  "versions": [
    {
      "component": "core",
      "version": "3.12.0"
    },
    {
      "component": "rpm",
      "version": "3.10.0"
    }
  ],
  "online_workers": [
    {
      "pulp_href":
"/pulp/api/v3/workers/179af4ab-6846-431c-bc51-a07db3ea9a1b/",
      "pulp_created": "2021-04-13T02:41:05.521531Z",
      "name": "22275 at ip-10-80-1-57.ap-northeast-1.compute.internal",
      "last_heartbeat": "2021-04-13T02:41:50.701103Z"
    },
    {
      "pulp_href":
"/pulp/api/v3/workers/77c08991-a27a-49d6-a2c7-87cd8c26877c/",
      "pulp_created": "2021-03-31T07:59:28.269742Z",
      "name": "resource-manager",
      "last_heartbeat": "2021-04-13T02:41:51.803953Z"
    },
    {
      "pulp_href":
"/pulp/api/v3/workers/75bc2422-2f58-41cb-93d2-17b69732d946/",
      "pulp_created": "2021-04-13T02:41:08.214845Z",
      "name": "22430 at ip-10-80-1-57.ap-northeast-1.compute.internal",
      "last_heartbeat": "2021-04-13T02:41:53.709207Z"
    }
  ],
  "online_content_apps": [
    {
      "name": "23617 at ip-10-80-1-57.ap-northeast-1.compute.internal",
      "last_heartbeat": "2021-04-13T02:41:56.489400Z"
    },
    {
      "name": "23614 at ip-10-80-1-57.ap-northeast-1.compute.internal",
      "last_heartbeat": "2021-04-13T02:41:56.646680Z"
    },
    {
      "name": "23620 at ip-10-80-1-57.ap-northeast-1.compute.internal",
      "last_heartbeat": "2021-04-13T02:41:56.918615Z"
    },
    {
      "name": "23629 at ip-10-80-1-57.ap-northeast-1.compute.internal",
      "last_heartbeat": "2021-04-13T02:41:57.224724Z"
    },
    {
      "name": "23624 at ip-10-80-1-57.ap-northeast-1.compute.internal",
      "last_heartbeat": "2021-04-13T02:41:57.319709Z"
    },
    {
      "name": "23628 at ip-10-80-1-57.ap-northeast-1.compute.internal",
      "last_heartbeat": "2021-04-13T02:41:57.328655Z"
    },
    {
      "name": "23609 at ip-10-80-1-57.ap-northeast-1.compute.internal",
      "last_heartbeat": "2021-04-13T02:41:55.609724Z"
    },
    {
      "name": "23613 at ip-10-80-1-57.ap-northeast-1.compute.internal",
      "last_heartbeat": "2021-04-13T02:41:55.824550Z"
    }
  ],
  "database_connection": {
    "connected": true
  },
  "redis_connection": {
    "connected": true
  },
  "storage": {
    "total": 161008844800,
    "used": 109266341888,
    "free": 51742502912
  }
}

(ansible-venv) ╭─ericv at ip-10-80-1-57 ~
╰─$ (⎈ k8s01.aops-dev.activeops.local:kube-system) http
https://pulp-dev.activeops.io/pulp/content/

http: error: TypeError: __str__ returned non-string (type Error)

Kind regards,

*Eric Van Steenbergen*

*E-mail: vs.eric at gmail.com <vs.eric at gmail.com>*
*Skype: ericvs2014*
*LinkedIn: Eric Van Steenbergen
<http://es.linkedin.com/pub/eric-van-steenbergen/b/8a4/51b>*


On Mon, Apr 12, 2021 at 8:35 PM Eric VS <vs.eric at gmail.com> wrote:

> Hello Gerrod,
>
> Thank you for the update. On the node that errors I have indeed 3.11
> installed, on another one I'm doing right now I have 3.12.
> Dumb question, I'm not that familiar with ansible-galaxy. How do I
> 'upgrade' pulp_installer? I tried to run the install command again which
> just says pulp.installer is already installed. Even downloading and trying
> to install from the file gives the same result.
>
> Kind regards,
>
> *Eric Van Steenbergen*
>
> *E-mail: vs.eric at gmail.com <vs.eric at gmail.com>*
> *Skype: ericvs2014*
> *LinkedIn: Eric Van Steenbergen
> <http://es.linkedin.com/pub/eric-van-steenbergen/b/8a4/51b>*
>
>
> On Mon, Apr 12, 2021 at 8:28 PM Gerrod Ubben <gubben at redhat.com> wrote:
>
>> Hi Eric,
>>
>> This bug [0] was fixed in pulpcore 3.12 which we just finished releasing
>> today.  You can use the 3.12 pulp_installer [1] to upgrade to pulpcore 3.12
>> or change the erroring line [2] in your current installation if you don't
>> want to upgrade right away. Also, all of Pulp's API endpoints require a
>> trailing forward slash or else you'll get a 301 return. So to hit the
>> status endpoint you use '/pulp/api/v3/status/', but we encourage everyone
>> to use the CLI and make feature requests [3] if you find missing
>> functionality.
>>
>> Gerrod
>>
>> [0] https://pulp.plan.io/issues/8475
>> [1] https://galaxy.ansible.com/pulp/pulp_installer
>> [2] https://git.io/JOqh0
>> [3] https://github.com/pulp/pulp-cli/issues/new/
>>
>> On Mon, Apr 12, 2021 at 8:46 PM Eric VS <vs.eric at gmail.com> wrote:
>>
>>> Hello Matthias and David,
>>>
>>> Thanks again for your swift responses, I'm in a different time zone most
>>> likely so apologies for late replies.
>>>
>>> @Matthias Dellweg <mdellweg at redhat.com> , when running the v3/status I
>>> get this:
>>>
>>> ╰─$ (⎈ k8s01.aops-dev.activeops.local:kube-system) http
>>> https://pulp-dev.activeops.io/pulp/api/v3/status
>>> HTTP/1.1 301 Moved Permanently
>>> Access-Control-Expose-Headers: Correlation-ID
>>> Connection: keep-alive
>>> Content-Length: 0
>>> Content-Type: text/html; charset=utf-8
>>> Correlation-ID: 4a98ac4213d44645ae9c963c81873780
>>> Date: Tue, 13 Apr 2021 00:40:05 GMT
>>> Location: /pulp/api/v3/status/
>>> Server: nginx/1.16.1
>>> Strict-Transport-Security: max-age=15768000
>>> (ansible-venv) ╭─ericv at ip-10-80-1-57 ~
>>>
>>> Tracing journalctl gives me the below when I run the /pulp/content with
>>> http.
>>>
>>> ╰─$ (⎈ k8s01.aops-dev.activeops.local:kube-system) sudo journalctl -f -u
>>> pulpcore-content
>>> -- Logs begin at Thu 2020-06-04 07:04:49 UTC. --
>>> Apr 13 00:42:45 ip-10-80-1-57.ap-northeast-1.compute.internal
>>> gunicorn[1216]: File
>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/web_protocol.py",
>>> line 422, in _handle_request
>>> Apr 13 00:42:45 ip-10-80-1-57.ap-northeast-1.compute.internal
>>> gunicorn[1216]: resp = await self._request_handler(request)
>>> Apr 13 00:42:45 ip-10-80-1-57.ap-northeast-1.compute.internal
>>> gunicorn[1216]: File
>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/web_app.py",
>>> line 499, in _handle
>>> Apr 13 00:42:45 ip-10-80-1-57.ap-northeast-1.compute.internal
>>> gunicorn[1216]: resp = await handler(request)
>>> Apr 13 00:42:45 ip-10-80-1-57.ap-northeast-1.compute.internal
>>> gunicorn[1216]: File
>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/content/handler.py",
>>> line 121, in list_distributions
>>> Apr 13 00:42:45 ip-10-80-1-57.ap-northeast-1.compute.internal
>>> gunicorn[1216]: directory_list = ["{}/".format(d.base_path) for d in
>>> base_paths]
>>> Apr 13 00:42:45 ip-10-80-1-57.ap-northeast-1.compute.internal
>>> gunicorn[1216]: File
>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/content/handler.py",
>>> line 121, in <listcomp>
>>> Apr 13 00:42:45 ip-10-80-1-57.ap-northeast-1.compute.internal
>>> gunicorn[1216]: directory_list = ["{}/".format(d.base_path) for d in
>>> base_paths]
>>> Apr 13 00:42:45 ip-10-80-1-57.ap-northeast-1.compute.internal
>>> gunicorn[1216]: AttributeError: 'str' object has no attribute 'base_path'
>>> Apr 13 00:42:45 ip-10-80-1-57.ap-northeast-1.compute.internal
>>> gunicorn[1216]: 127.0.0.1 [13/Apr/2021:00:42:45 +0000] "GET /pulp/content/
>>> HTTP/1.0" 500 231 "-" "HTTPie/0.9.4"
>>>
>>>
>>> Apr 13 00:43:40 ip-10-80-1-57.ap-northeast-1.compute.internal
>>> gunicorn[1216]: [2021-04-13 00:43:40 +0000] [1452] [ERROR] Error handling
>>> request
>>> Apr 13 00:43:40 ip-10-80-1-57.ap-northeast-1.compute.internal
>>> gunicorn[1216]: Traceback (most recent call last):
>>> Apr 13 00:43:40 ip-10-80-1-57.ap-northeast-1.compute.internal
>>> gunicorn[1216]: File
>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/web_protocol.py",
>>> line 422, in _handle_request
>>> Apr 13 00:43:40 ip-10-80-1-57.ap-northeast-1.compute.internal
>>> gunicorn[1216]: resp = await self._request_handler(request)
>>> Apr 13 00:43:40 ip-10-80-1-57.ap-northeast-1.compute.internal
>>> gunicorn[1216]: File
>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/web_app.py",
>>> line 499, in _handle
>>> Apr 13 00:43:40 ip-10-80-1-57.ap-northeast-1.compute.internal
>>> gunicorn[1216]: resp = await handler(request)
>>> Apr 13 00:43:40 ip-10-80-1-57.ap-northeast-1.compute.internal
>>> gunicorn[1216]: File
>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/content/handler.py",
>>> line 121, in list_distributions
>>> Apr 13 00:43:40 ip-10-80-1-57.ap-northeast-1.compute.internal
>>> gunicorn[1216]: directory_list = ["{}/".format(d.base_path) for d in
>>> base_paths]
>>> Apr 13 00:43:40 ip-10-80-1-57.ap-northeast-1.compute.internal
>>> gunicorn[1216]: File
>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/content/handler.py",
>>> line 121, in <listcomp>
>>> Apr 13 00:43:40 ip-10-80-1-57.ap-northeast-1.compute.internal
>>> gunicorn[1216]: directory_list = ["{}/".format(d.base_path) for d in
>>> base_paths]
>>> Apr 13 00:43:40 ip-10-80-1-57.ap-northeast-1.compute.internal
>>> gunicorn[1216]: AttributeError: 'str' object has no attribute 'base_path'
>>> Apr 13 00:43:40 ip-10-80-1-57.ap-northeast-1.compute.internal
>>> gunicorn[1216]: 127.0.0.1 [13/Apr/2021:00:43:40 +0000] "GET /pulp/content/
>>> HTTP/1.0" 500 231 "-" "HTTPie/0.9.4"
>>>
>>> Kind regards,
>>>
>>> *Eric Van Steenbergen*
>>>
>>> *E-mail: vs.eric at gmail.com <vs.eric at gmail.com>*
>>> *Skype: ericvs2014*
>>> *LinkedIn: Eric Van Steenbergen
>>> <http://es.linkedin.com/pub/eric-van-steenbergen/b/8a4/51b>*
>>>
>>>
>>> On Mon, Apr 12, 2021 at 9:55 AM David Davis <daviddavis at redhat.com>
>>> wrote:
>>>
>>>> It could potentially be this bug which was fixed in pulpcore 3.12:
>>>> https://pulp.plan.io/issues/8447
>>>>
>>>> The stacktrace would be helpful though to know for sure.
>>>>
>>>> David
>>>>
>>>>
>>>> On Mon, Apr 12, 2021 at 5:27 AM Matthias Dellweg <mdellweg at redhat.com>
>>>> wrote:
>>>>
>>>>> Can you monitor "journalctl -f -u pulpcore-content" while performing
>>>>> that call?
>>>>>
>>>>> On Mon, Apr 12, 2021 at 11:17 AM Matthias Dellweg <mdellweg at redhat.com>
>>>>> wrote:
>>>>>
>>>>>> Sorry, typo (e and 3 are too close on my k3yboard). Should have been
>>>>>> .../pulp/api/v3/status
>>>>>>
>>>>>> On Mon, Apr 12, 2021 at 11:06 AM Eric VS <vs.eric at gmail.com> wrote:
>>>>>>
>>>>>>> Hey Matthias,
>>>>>>>
>>>>>>> I installed Pulp using the pulp_installer Ansible playbook, no
>>>>>>> errors.
>>>>>>>
>>>>>>> This is the output from `pulp status` since the query using http
>>>>>>> didn't work either:
>>>>>>>
>>>>>>> ╰─$ (⎈ k8s01.aops-dev.activeops.local:kube-system) pulp status
>>>>>>> {
>>>>>>>   "versions": [
>>>>>>>     {
>>>>>>>       "component": "core",
>>>>>>>       "version": "3.11.0"
>>>>>>>     },
>>>>>>>     {
>>>>>>>       "component": "rpm",
>>>>>>>       "version": "3.10.0"
>>>>>>>     }
>>>>>>>   ],
>>>>>>>   "online_workers": [
>>>>>>>     {
>>>>>>>       "pulp_created": "2021-03-31T07:59:28.269742Z",
>>>>>>>       "pulp_href":
>>>>>>> "/pulp/api/v3/workers/77c08991-a27a-49d6-a2c7-87cd8c26877c/",
>>>>>>>       "name": "resource-manager",
>>>>>>>       "last_heartbeat": "2021-04-12T09:02:03.424149Z"
>>>>>>>     },
>>>>>>>     {
>>>>>>>       "pulp_created": "2021-04-05T06:46:11.689705Z",
>>>>>>>       "pulp_href":
>>>>>>> "/pulp/api/v3/workers/ed4c9dca-db6d-4d52-9f0b-331a937d2755/",
>>>>>>>       "name": "1200 at ip-10-80-1-57.ap-northeast-1.compute.internal",
>>>>>>>       "last_heartbeat": "2021-04-12T09:02:03.624644Z"
>>>>>>>     },
>>>>>>>     {
>>>>>>>       "pulp_created": "2021-04-05T06:46:11.689909Z",
>>>>>>>       "pulp_href":
>>>>>>> "/pulp/api/v3/workers/9d538294-a3aa-48d4-95ab-e0a213bae94f/",
>>>>>>>       "name": "1231 at ip-10-80-1-57.ap-northeast-1.compute.internal",
>>>>>>>       "last_heartbeat": "2021-04-12T09:02:11.745839Z"
>>>>>>>     }
>>>>>>>   ],
>>>>>>>   "online_content_apps": [
>>>>>>>     {
>>>>>>>       "name": "1452 at ip-10-80-1-57.ap-northeast-1.compute.internal",
>>>>>>>       "last_heartbeat": "2021-04-12T09:02:10.886966Z"
>>>>>>>     },
>>>>>>>     {
>>>>>>>       "name": "1453 at ip-10-80-1-57.ap-northeast-1.compute.internal",
>>>>>>>       "last_heartbeat": "2021-04-12T09:02:13.566778Z"
>>>>>>>     },
>>>>>>>     {
>>>>>>>       "name": "1448 at ip-10-80-1-57.ap-northeast-1.compute.internal",
>>>>>>>       "last_heartbeat": "2021-04-12T09:02:13.568876Z"
>>>>>>>     },
>>>>>>>     {
>>>>>>>       "name": "1447 at ip-10-80-1-57.ap-northeast-1.compute.internal",
>>>>>>>       "last_heartbeat": "2021-04-12T09:02:15.876728Z"
>>>>>>>     },
>>>>>>>     {
>>>>>>>       "name": "1450 at ip-10-80-1-57.ap-northeast-1.compute.internal",
>>>>>>>       "last_heartbeat": "2021-04-12T09:02:15.881384Z"
>>>>>>>     },
>>>>>>>     {
>>>>>>>       "name": "1454 at ip-10-80-1-57.ap-northeast-1.compute.internal",
>>>>>>>       "last_heartbeat": "2021-04-12T09:02:15.881615Z"
>>>>>>>     },
>>>>>>>     {
>>>>>>>       "name": "1451 at ip-10-80-1-57.ap-northeast-1.compute.internal",
>>>>>>>       "last_heartbeat": "2021-04-12T09:02:15.883369Z"
>>>>>>>     },
>>>>>>>     {
>>>>>>>       "name": "1449 at ip-10-80-1-57.ap-northeast-1.compute.internal",
>>>>>>>       "last_heartbeat": "2021-04-12T09:02:15.885268Z"
>>>>>>>     }
>>>>>>>   ],
>>>>>>>   "database_connection": {
>>>>>>>     "connected": true
>>>>>>>   },
>>>>>>>   "redis_connection": {
>>>>>>>     "connected": true
>>>>>>>   },
>>>>>>>   "storage": {
>>>>>>>     "total": 161008844800,
>>>>>>>     "used": 109256830976,
>>>>>>>     "free": 51752013824
>>>>>>>   }
>>>>>>> }
>>>>>>>
>>>>>>> For reference this is the query you requested using api/ve/status
>>>>>>>
>>>>>>> ╰─$ (⎈ k8s01.aops-dev.activeops.local:kube-system) http
>>>>>>> https://pulp-dev.activeops.io/pulp/api/ve/status
>>>>>>> HTTP/1.1 404 Not Found
>>>>>>> Access-Control-Expose-Headers: Correlation-ID
>>>>>>> Connection: keep-alive
>>>>>>> Content-Length: 77
>>>>>>> Content-Type: text/html
>>>>>>> Correlation-ID: 14cab8b25dc64591a243fd391424924f
>>>>>>> Date: Mon, 12 Apr 2021 09:04:06 GMT
>>>>>>> Server: nginx/1.16.1
>>>>>>> X-Frame-Options: SAMEORIGIN
>>>>>>>
>>>>>>> <h1>Not Found</h1><p>The requested resource was not found on this
>>>>>>> server.</p>
>>>>>>>
>>>>>>> Kind regards,
>>>>>>>
>>>>>>> *Eric Van Steenbergen*
>>>>>>>
>>>>>>> *E-mail: vs.eric at gmail.com <vs.eric at gmail.com>*
>>>>>>> *Skype: ericvs2014*
>>>>>>> *LinkedIn: Eric Van Steenbergen
>>>>>>> <http://es.linkedin.com/pub/eric-van-steenbergen/b/8a4/51b>*
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Apr 12, 2021 at 4:00 AM Matthias Dellweg <
>>>>>>> mdellweg at redhat.com> wrote:
>>>>>>>
>>>>>>>> Hi Eric,
>>>>>>>> can you provide more info on this?
>>>>>>>> Helpful would be:
>>>>>>>> How is the server installed? (container, pulp_installer, by hand
>>>>>>>> from pypi)
>>>>>>>> What versions of the components are installed? (http https://
>>>>>>>> <FQDN>/pulp/api/ve/status)
>>>>>>>> Thanks in advance!
>>>>>>>>
>>>>>>>> On Mon, Apr 12, 2021 at 10:57 AM Eric VS <vs.eric at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Matthias,
>>>>>>>>>
>>>>>>>>> When I run `http https://<FQDN>/pulp/content/` is when I get the
>>>>>>>>> 500 error.
>>>>>>>>> I can query the all the content with the API (pulp rpm
>>>>>>>>> repository/remote/publication/distribution) without any issue.
>>>>>>>>>
>>>>>>>>> Kind regards,
>>>>>>>>>
>>>>>>>>> *Eric Van Steenbergen*
>>>>>>>>>
>>>>>>>>> *E-mail: vs.eric at gmail.com <vs.eric at gmail.com>*
>>>>>>>>> *Skype: ericvs2014*
>>>>>>>>> *LinkedIn: Eric Van Steenbergen
>>>>>>>>> <http://es.linkedin.com/pub/eric-van-steenbergen/b/8a4/51b>*
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Apr 12, 2021 at 3:48 AM Matthias Dellweg <
>>>>>>>>> mdellweg at redhat.com> wrote:
>>>>>>>>>
>>>>>>>>>> Can you isolate the command that leads to this server response?
>>>>>>>>>>
>>>>>>>>>> On Mon, Apr 12, 2021 at 1:55 AM Eric VS <vs.eric at gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I've been setting up another Pulp server, replicating
>>>>>>>>>>> repositories from my main one for 'local' use in  an environment. When I
>>>>>>>>>>> was adding remotes and repositories at one point in time I started getting
>>>>>>>>>>> the below error:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> *HTTP/1.1 500 Internal Server ErrorConnection:
>>>>>>>>>>> keep-aliveContent-Length: 55Content-Type: text/plain; charset=utf-8Date:
>>>>>>>>>>> Sun, 11 Apr 2021 23:53:14 GMTServer: nginx/1.16.1500 Internal Server
>>>>>>>>>>> ErrorServer got itself in trouble*
>>>>>>>>>>>
>>>>>>>>>>> I have enabled debug on Nginx but can't see anything indicating
>>>>>>>>>>> on why this all of a sudden fails.
>>>>>>>>>>>
>>>>>>>>>>> Any help is greatly appreciated.
>>>>>>>>>>>
>>>>>>>>>>> Kind regards,
>>>>>>>>>>>
>>>>>>>>>>> *Eric Van Steenbergen*
>>>>>>>>>>>
>>>>>>>>>>> *E-mail: vs.eric at gmail.com <vs.eric at gmail.com>*
>>>>>>>>>>> *Skype: ericvs2014*
>>>>>>>>>>> *LinkedIn: Eric Van Steenbergen
>>>>>>>>>>> <http://es.linkedin.com/pub/eric-van-steenbergen/b/8a4/51b>*
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Pulp-list mailing list
>>>>>>>>>>> Pulp-list at redhat.com
>>>>>>>>>>> https://listman.redhat.com/mailman/listinfo/pulp-list
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>> Pulp-list mailing list
>>>>> Pulp-list at redhat.com
>>>>> https://listman.redhat.com/mailman/listinfo/pulp-list
>>>>
>>>> _______________________________________________
>>> Pulp-list mailing list
>>> Pulp-list at redhat.com
>>> https://listman.redhat.com/mailman/listinfo/pulp-list
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20210412/356f0bc4/attachment.htm>

More information about the Pulp-list mailing list