<div dir="ltr"><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jun 25, 2019 at 2:07 PM Dennis Kliban <<a href="mailto:dkliban@redhat.com">dkliban@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">This is the same bug as I mentioned earlier. You need to set either the PULP_CONTENT_HOST environment variable or CONTENT_HOST setting in /etc/pulp/settings.py<br></div></blockquote><div><br></div><div>Here[0] are the docs about this setting.</div><div><br></div><div>[0] <a href="https://docs.pulpproject.org/en/3.0/nightly/installation/configuration.html#content-host">https://docs.pulpproject.org/en/3.0/nightly/installation/configuration.html#content-host</a></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jun 25, 2019 at 1:57 PM Bin Li (BLOOMBERG/ 120 PARK) <<a href="mailto:bli111@bloomberg.net" target="_blank">bli111@bloomberg.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="gmail-m_7145593730779271698gmail-m_8854442428006997597rte-style-maintainer gmail-m_7145593730779271698gmail-m_8854442428006997597rte-pre-wrap" style="white-space:pre-wrap;font-size:small;font-family:"Courier New",Courier,"BB.FixedWidth""><div>Here is the traceback from the log:</div><div><br></div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: pulp: django.request:ERROR: Internal Server Error: /pulp/api/v3/tasks/<div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: Traceback (most recent call last):</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34, in inner</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: response = get_response(request)</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/django/core/handlers/base.py", line 115, in _get_response</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: response = self.process_exception_by_middleware(e, request)</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/django/core/handlers/base.py", line 113, in _get_response</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: response = wrapped_callback(request, *callback_args, **callback_kwargs)</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: return view_func(*args, **kwargs)</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/viewsets.py", line 116, in view</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: return self.dispatch(request, *args, **kwargs)</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/views.py", line 495, in dispatch</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: response = self.handle_exception(exc)</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/views.py", line 455, in handle_exception</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: self.raise_uncaught_exception(exc)</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/views.py", line 492, in dispatch</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: response = handler(request, *args, **kwargs)</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/mixins.py", line 45, in list</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: return self.get_paginated_response(serializer.data)</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/serializers.py", line 768, in data</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: ret = super(ListSerializer, self).data</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/serializers.py", line 262, in data</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: self._data = self.to_representation(self.instance)</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/serializers.py", line 686, in to_representation</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: self.child.to_representation(item) for item in iterable</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/serializers.py", line 686, in <listcomp></div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: self.child.to_representation(item) for item in iterable</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/serializers.py", line 530, in to_representation</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: ret[field.field_name] = field.to_representation(attribute)</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/relations.py", line 542, in to_representation</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: for value in iterable</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/relations.py", line 542, in <listcomp></div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: for value in iterable</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore-3.0.0rc3.dev0-py3.6.egg/pulpcore/app/serializers/task.py", line 25, in to_representation</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: return serializer.data.get('_href')</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/serializers.py", line 563, in data</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: ret = super(Serializer, self).data</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/serializers.py", line 262, in data</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: self._data = self.to_representation(self.instance)</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/rest_framework/serializers.py", line 530, in to_representation</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: ret[field.field_name] = field.to_representation(attribute)</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: File "/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore-3.0.0rc3.dev0-py3.6.egg/pulpcore/app/serializers/fields.py", line 248, in to_representation</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: host.strip('/'),</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: AttributeError: 'NoneType' object has no attribute 'strip'</div><div>Jun 25 13:53:08 ip-1-76-158-221 gunicorn[147692]: 127.0.0.1 - admin [25/Jun/2019:17:53:08 +0000] "GET /pulp/api/v3/tasks/ HTTP/1.1" 500 27 "-" "HTTPie/0.9.4<br><br><div class="gmail-m_7145593730779271698gmail-m_8854442428006997597rte-style-maintainer" style="font-size:small;font-family:"Courier New",Courier,"BB.FixedWidth""><div><div class="gmail-m_7145593730779271698gmail-m_8854442428006997597bbg-rte-fold-content"><div class="gmail-m_7145593730779271698gmail-m_8854442428006997597bbg-rte-fold-summary">From: <a href="mailto:dkliban@redhat.com" target="_blank">dkliban@redhat.com</a> At: 06/25/19 13:46:18</div>To: <a href="mailto:bli111@bloomberg.net" target="_blank"> Bin Li (BLOOMBERG/ 120 PARK ) </a><br>Cc: <a href="mailto:pulp-list@redhat.com" target="_blank"> pulp-list@redhat.com</a><br>Subject: Re: [Pulp-list] Pulp 3 internal error</div><br></div><div class="gmail-m_7145593730779271698gmail-m_8854442428006997597rte-internet-block-wrapper" style="background:white none repeat scroll 0% 0%;color:black;font-family:Arial,"BB.Proportional";font-size:small;white-space:normal"><div class="gmail-m_7145593730779271698gmail-m_8854442428006997597rte-internet-block"><blockquote><div dir="ltr"><div>The logs are available in journalctl. I usually run 'sudo journalctl -l -f' to monitor the logs. What is the traceback that is emitted when you query this task?</div><div><br></div><div>A possibly related issue got filed today: <a href="https://pulp.plan.io/issues/4945" target="_blank">https://pulp.plan.io/issues/4945</a></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jun 25, 2019 at 1:35 PM Bin Li (BLOOMBERG/ 120 PARK) <<a href="mailto:bli111@bloomberg.net" target="_blank">bli111@bloomberg.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="gmail-m_7145593730779271698gmail-m_8854442428006997597gmail-m_6859011548924984874rte-style-maintainer gmail-m_7145593730779271698gmail-m_8854442428006997597gmail-m_6859011548924984874rte-pre-wrap" style="white-space:pre-wrap;font-size:small;font-family:"Courier New",Courier,"BB.FixedWidth""><div><br></div><div>I suddenly got a internal 500 error when query a task which creates a publication. Restarted all pulp daemons didn't help. Still able to GET repositories, publications, remotes information but query tasks and distributions gave me internal error. Database looks fine. I am able connect and login postgresql as pulp user. How do I troubleshot this issue?</div><div><br></div><div>used the following to create distribution:</div><div>http POST <a href="http://localhost:24817/pulp/api/v3/distributions/rpm/rpm/%3C%3CEOF" target="_blank">http://localhost:24817/pulp/api/v3/distributions/rpm/rpm/<<EOF</a></div><div>{</div><div>"base_path": "bloomberg/systems_third_party_rhel7.2",</div><div>"name": "systems_third_party_rhel7.2",</div><div>"publication": "/pulp/api/v3/publications/rpm/rpm/894a8109-001b-42fc-8658-4ac28bc52c0e/"</div><div>}</div><div>EOF</div><div><br></div><div><br></div><div>Below is current status:</div><div>{</div><div> "database_connection": {</div><div> "connected": true</div><div> }, </div><div> "missing_workers": [], </div><div> "online_workers": [</div><div> {</div><div> "_created": "2019-06-20T18:34:01.015187Z", </div><div> "_href": "/pulp/api/v3/workers/fd7dc86c-3bff-4631-a86d-f49f618d3a79/", </div><div> "last_heartbeat": "2019-06-25T17:14:50.374396Z", </div><div> "missing": false, </div><div> "name": "<a href="mailto:resource-manager@ip-1-76-158-221.drr1.bcpc.bloomberg.com" target="_blank">resource-manager@ip-1-76-158-221.drr1.bcpc.bloomberg.com</a>", </div><div> "online": true</div><div> }, </div><div> {</div><div> "_created": "2019-06-20T18:34:01.347733Z", </div><div> "_href": "/pulp/api/v3/workers/b4dfbb8a-69f4-4411-9311-ff84a63756c0/", </div><div> "last_heartbeat": "2019-06-25T17:14:50.575492Z", </div><div> "missing": false, </div><div> "name": "<a href="mailto:reserved-resource-worker-1@ip-1-76-158-221.drr1.bcpc.bloomberg.com" target="_blank">reserved-resource-worker-1@ip-1-76-158-221.drr1.bcpc.bloomberg.com</a>", </div><div> "online": true</div><div> }, </div><div> {</div><div> "_created": "2019-06-20T18:34:01.463024Z", </div><div> "_href": "/pulp/api/v3/workers/d1db2594-52b6-402e-8ef1-7c0a5635c3c4/", </div><div> "last_heartbeat": "2019-06-25T17:14:50.976399Z", </div><div> "missing": false, </div><div> "name": "<a href="mailto:reserved-resource-worker-2@ip-1-76-158-221.drr1.bcpc.bloomberg.com" target="_blank">reserved-resource-worker-2@ip-1-76-158-221.drr1.bcpc.bloomberg.com</a>", </div><div> "online": true</div><div> }</div><div> ], </div><div> "redis_connection": {</div><div> "connected": true</div><div> }, </div><div> "versions": [</div><div> {</div><div> "component": "pulpcore", </div><div> "version": "3.0.0rc3.dev0"</div><div> }, </div><div> {</div><div> "component": "pulpcore-plugin", </div><div> "version": "0.1.0rc3.dev0"</div><div> }, </div><div> {</div><div> "component": "pulp_rpm", </div><div> "version": "3.0.0b4.dev0"</div><div> }</div><div> ]</div><div>}</div><div><br></div><div><br></div></div></div>_______________________________________________<br>Pulp-list mailing list<br><a href="mailto:Pulp-list@redhat.com" target="_blank">Pulp-list@redhat.com</a><br><a href="https://www.redhat.com/mailman/listinfo/pulp-list" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-list</a></blockquote></div></blockquote><br></div></div></div></div></div></div></blockquote></div> </blockquote></div></div>