[Pulp-list] Pulp 3 internal error

Tue Jun 25 19:41:54 UTC 2019

The RPM plugin does not list directories right now. Feel free to file an
issue about this.

What happens when you GET
http://localhost:24816/pulp/content/bloomberg/systems_third_party_rhel7.2/repodata/repomd.xml
?

On Tue, Jun 25, 2019 at 3:35 PM Bin Li (BLOOMBERG/ 120 PARK) <
bli111 at bloomberg.net> wrote:

> Updated CONTENT_HOST = 'http://localhost:24816' in settings.py, restarted
> all daemons and recreated distribution. Now I am getting 404 error:
> # http get
> http://localhost:24816/pulp/content/bloomberg/systems_third_party_rhel7.2/
> HTTP/1.1 404 Not Found
> Content-Length: 14
> Content-Type: text/plain; charset=utf-8
> Date: Tue, 25 Jun 2019 19:14:28 GMT
> Server: Python/3.6 aiohttp/3.5.4
>
> 404: Not Found
>
>
> The following is in the log:
>
> Jun 25 15:19:32 ip-1-76-158-221 gunicorn[180694]: 127.0.0.1 - admin
> [25/Jun/2019:19:19:32 +0000] "GET /pulp/api/v3/publications/rpm/rpm/
> HTTP/1.1" 200 567 "-" "HTTPie/0.9.4"
> Jun 25 15:19:41 ip-1-76-158-221 rc.bloomberg[1549]: pid 149131's current
> scheduling policy: SCHED_OTHER pid 149131's current scheduling priority: 0
> Jun 25 15:19:41 ip-1-76-158-221 rc.bloomberg[1549]: chrt: failed to set
> pid 149131's policy: Operation not permitted
> Jun 25 15:19:48 ip-1-76-158-221 gunicorn[180694]: 127.0.0.1 - admin
> [25/Jun/2019:19:19:48 +0000] "GET /pulp/api/v3/repositories/ HTTP/1.1" 200
> 1353 "-" "HTTPie/0.9.4"
> Jun 25 15:19:59 ip-1-76-158-221 gunicorn[180694]: 127.0.0.1 - admin
> [25/Jun/2019:19:19:59 +0000] "GET /pulp/api/v3/repositories/ HTTP/1.1" 200
> 1353 "-" "HTTPie/0.9.4"
> Jun 25 15:20:00 ip-1-76-158-221 gunicorn[180694]: 127.0.0.1 - admin
> [25/Jun/2019:19:20:00 +0000] "GET /pulp/api/v3/remotes/rpm/rpm/ HTTP/1.1"
> 200 2383 "-" "HTTPie/0.9.4"
> Jun 25 15:20:01 ip-1-76-158-221 gunicorn[180694]: 127.0.0.1 - admin
> [25/Jun/2019:19:20:01 +0000] "GET /pulp/api/v3/content/rpm/packages/
> HTTP/1.1" 200 1873703 "-" "HTTPie/0.9.4"
>
> The distribution is:
> {
> "count": 1,
> "next": null,
> "previous": null,
> "results": [
> {
> "_created": "2019-06-25T19:13:54.154550Z",
> "_href":
> "/pulp/api/v3/distributions/rpm/rpm/f3117050-9373-479d-ae14-63ac2f37f30c/",
> "base_path": "bloomberg/systems_third_party_rhel7.2",
> "base_url": "
> http://localhost:24816/pulp/content/bloomberg/systems_third_party_rhel7.2",
>
> "content_guard": null,
> "name": "systems_third_party_rhel7.2",
> "publication":
> "/pulp/api/v3/publications/rpm/rpm/894a8109-001b-42fc-8658-4ac28bc52c0e/"
> }
> ]
> }
>
>
> Thanks
>
> From: brocha at redhat.com At: 06/25/19 15:09:04
> To: Bin Li (BLOOMBERG/ 120 PARK ) <bli111 at bloomberg.net>
> Cc: pulp-list at redhat.com
> Subject: Re: [Pulp-list] Pulp 3 internal error
>
> The port for content host recently changed to 24816
>
> CONTENT_HOST = 'http://localhost:24816'
>
> Bruno Rocha
>
> Senior Quality Engineer - Cloud Management QE - Pulp Project
> *irc*: rochacbruno
>
> *social*: http://about.me/rochacbruno
>
> “Progress is the realization of utopia.”
> <https://red.ht/sig>
>
>
> On Tue, Jun 25, 2019 at 4:01 PM Bin Li (BLOOMBERG/ 120 PARK) <
> bli111 at bloomberg.net> wrote:
>
>>
>> Adding CONTENT_HOST = 'http://localhost:8080' to setting.py fixed the
>> issue. However, got connection error when try to access a distribution.
>>
>> # http get
>> http://localhost:8080/pulp/content/bloomberg/systems_third_party_rhel7.2/
>>
>> http: error: ConnectionError: ('Connection aborted.', error(111,
>> 'Connection refused')) while doing GET request to URL:
>> http://localhost:8080/pulp/content/bloomberg/systems_third_party_rhel7.2/
>>
>> The port 8080 is not listening. What am I doing wrong?
>>
>> The distribution is as following:
>> {
>> "count": 1,
>> "next": null,
>> "previous": null,
>> "results": [
>> {
>> "_created": "2019-06-25T18:51:14.803243Z",
>> "_href":
>> "/pulp/api/v3/distributions/rpm/rpm/6cde0eda-ea68-4d22-920d-c62173c4d80e/",
>> "base_path": "bloomberg/systems_third_party_rhel7.2",
>> "base_url": "
>> http://localhost:8080/pulp/content/bloomberg/systems_third_party_rhel7.2",
>>
>> "content_guard": null,
>> "name": "systems_third_party_rhel7.2",
>> "publication":
>> "/pulp/api/v3/publications/rpm/rpm/894a8109-001b-42fc-8658-4ac28bc52c0e/"
>> }
>> ]
>> }
>>
>> Thanks
>>
>>
>> From: dkliban at redhat.com At: 06/25/19 14:07:56
>> To: Bin Li (BLOOMBERG/ 120 PARK ) <bli111 at bloomberg.net>
>> Cc: pulp-list at redhat.com
>> Subject: Re: [Pulp-list] Pulp 3 internal error
>>
>> This is the same bug as I mentioned earlier. You need to set either the
>> PULP_CONTENT_HOST environment variable or CONTENT_HOST setting in
>> /etc/pulp/settings.py
>>
>> On Tue, Jun 25, 2019 at 1:57 PM Bin Li (BLOOMBERG/ 120 PARK) <
>> bli111 at bloomberg.net> wrote:
>>
>>> Here is the traceback from the log:
>>>
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: pulp:
>>> django.request:ERROR: Internal Server Error: /pulp/api/v3/tasks/
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: Traceback (most recent
>>> call last):
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File
>>> "/opt/python/3.6.5/lib/python3.6/site-packages/django/core/handlers/exception.py",
>>> line 34, in inner
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: response =
>>> get_response(request)
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File
>>> "/opt/python/3.6.5/lib/python3.6/site-packages/django/core/handlers/base.py",
>>> line 115, in _get_response
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: response =
>>> self.process_exception_by_middleware(e, request)
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File
>>> "/opt/python/3.6.5/lib/python3.6/site-packages/django/core/handlers/base.py",
>>> line 113, in _get_response
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: response =
>>> wrapped_callback(request, *callback_args, **callback_kwargs)
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File
>>> "/opt/python/3.6.5/lib/python3.6/site-packages/django/views/decorators/csrf.py",
>>> line 54, in wrapped_view
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: return
>>> view_func(*args, **kwargs)
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File
>>> "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/viewsets.py",
>>> line 116, in view
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: return
>>> self.dispatch(request, *args, **kwargs)
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File
>>> "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/views.py",
>>> line 495, in dispatch
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: response =
>>> self.handle_exception(exc)
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File
>>> "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/views.py",
>>> line 455, in handle_exception
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]:
>>> self.raise_uncaught_exception(exc)
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File
>>> "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/views.py",
>>> line 492, in dispatch
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: response =
>>> handler(request, *args, **kwargs)
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File
>>> "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/mixins.py",
>>> line 45, in list
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: return
>>> self.get_paginated_response(serializer.data)
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File
>>> "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/serializers.py",
>>> line 768, in data
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: ret =
>>> super(ListSerializer, self).data
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File
>>> "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/serializers.py",
>>> line 262, in data
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: self._data =
>>> self.to_representation(self.instance)
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File
>>> "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/serializers.py",
>>> line 686, in to_representation
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]:
>>> self.child.to_representation(item) for item in iterable
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File
>>> "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/serializers.py",
>>> line 686, in <listcomp>
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]:
>>> self.child.to_representation(item) for item in iterable
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File
>>> "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/serializers.py",
>>> line 530, in to_representation
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: ret[field.field_name]
>>> = field.to_representation(attribute)
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File
>>> "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/relations.py",
>>> line 542, in to_representation
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: for value in iterable
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File
>>> "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/relations.py",
>>> line 542, in <listcomp>
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: for value in iterable
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File
>>> "/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore-3.0.0rc3.dev0-py3.6.egg/pulpcore/app/serializers/task.py",
>>> line 25, in to_representation
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: return
>>> serializer.data.get('_href')
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File
>>> "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/serializers.py",
>>> line 563, in data
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: ret =
>>> super(Serializer, self).data
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File
>>> "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/serializers.py",
>>> line 262, in data
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: self._data =
>>> self.to_representation(self.instance)
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File
>>> "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/serializers.py",
>>> line 530, in to_representation
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: ret[field.field_name]
>>> = field.to_representation(attribute)
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File
>>> "/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore-3.0.0rc3.dev0-py3.6.egg/pulpcore/app/serializers/fields.py",
>>> line 248, in to_representation
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: host.strip('/'),
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: AttributeError:
>>> 'NoneType' object has no attribute 'strip'
>>> Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: 127.0.0.1 - admin
>>> [25/Jun/2019:17:53:08 +0000] "GET /pulp/api/v3/tasks/ HTTP/1.1" 500 27 "-"
>>> "HTTPie/0.9.4
>>>
>>> From: dkliban at redhat.com At: 06/25/19 13:46:18
>>> To: Bin Li (BLOOMBERG/ 120 PARK ) <bli111 at bloomberg.net>
>>> Cc: pulp-list at redhat.com
>>> Subject: Re: [Pulp-list] Pulp 3 internal error
>>>
>>> The logs are available in journalctl. I usually run 'sudo journalctl -l
>>> -f' to monitor the logs. What is the traceback that is emitted when you
>>> query this task?
>>>
>>> A possibly related issue got filed today:
>>> https://pulp.plan.io/issues/4945
>>>
>>> On Tue, Jun 25, 2019 at 1:35 PM Bin Li (BLOOMBERG/ 120 PARK) <
>>> bli111 at bloomberg.net> wrote:
>>>
>>>>
>>>> I suddenly got a internal 500 error when query a task which creates a
>>>> publication. Restarted all pulp daemons didn't help. Still able to GET
>>>> repositories, publications, remotes information but query tasks and
>>>> distributions gave me internal error. Database looks fine. I am able
>>>> connect and login postgresql as pulp user. How do I troubleshot this issue?
>>>>
>>>> used the following to create distribution:
>>>> http POST
>>>> http://localhost:24817/pulp/api/v3/distributions/rpm/rpm/<<EOF
>>>> {
>>>> "base_path": "bloomberg/systems_third_party_rhel7.2",
>>>> "name": "systems_third_party_rhel7.2",
>>>> "publication":
>>>> "/pulp/api/v3/publications/rpm/rpm/894a8109-001b-42fc-8658-4ac28bc52c0e/"
>>>> }
>>>> EOF
>>>>
>>>>
>>>> Below is current status:
>>>> {
>>>> "database_connection": {
>>>> "connected": true
>>>> },
>>>> "missing_workers": [],
>>>> "online_workers": [
>>>> {
>>>> "_created": "2019-06-20T18:34:01.015187Z",
>>>> "_href": "/pulp/api/v3/workers/fd7dc86c-3bff-4631-a86d-f49f618d3a79/",
>>>> "last_heartbeat": "2019-06-25T17:14:50.374396Z",
>>>> "missing": false,
>>>> "name": "resource-manager at ip-1-76-158-221.drr1.bcpc.bloomberg.com",
>>>> "online": true
>>>> },
>>>> {
>>>> "_created": "2019-06-20T18:34:01.347733Z",
>>>> "_href": "/pulp/api/v3/workers/b4dfbb8a-69f4-4411-9311-ff84a63756c0/",
>>>> "last_heartbeat": "2019-06-25T17:14:50.575492Z",
>>>> "missing": false,
>>>> "name": "
>>>> reserved-resource-worker-1 at ip-1-76-158-221.drr1.bcpc.bloomberg.com",
>>>> "online": true
>>>> },
>>>> {
>>>> "_created": "2019-06-20T18:34:01.463024Z",
>>>> "_href": "/pulp/api/v3/workers/d1db2594-52b6-402e-8ef1-7c0a5635c3c4/",
>>>> "last_heartbeat": "2019-06-25T17:14:50.976399Z",
>>>> "missing": false,
>>>> "name": "
>>>> reserved-resource-worker-2 at ip-1-76-158-221.drr1.bcpc.bloomberg.com",
>>>> "online": true
>>>> }
>>>> ],
>>>> "redis_connection": {
>>>> "connected": true
>>>> },
>>>> "versions": [
>>>> {
>>>> "component": "pulpcore",
>>>> "version": "3.0.0rc3.dev0"
>>>> },
>>>> {
>>>> "component": "pulpcore-plugin",
>>>> "version": "0.1.0rc3.dev0"
>>>> },
>>>> {
>>>> "component": "pulp_rpm",
>>>> "version": "3.0.0b4.dev0"
>>>> }
>>>> ]
>>>> }
>>>>
>>>>
>>>> _______________________________________________
>>>> Pulp-list mailing list
>>>> Pulp-list at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/pulp-list
>>>
>>>
>>>
>> _______________________________________________
>> Pulp-list mailing list
>> Pulp-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-list
>
>
> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20190625/417ac61b/attachment.htm>