<div dir="ltr"><div>Hello Gerrod, <br></div><div><br></div><div>Thank you for the update. On the node that errors I have indeed 3.11 installed, on another one I'm doing right now I have 3.12. <br></div><div>Dumb question, I'm not that familiar with ansible-galaxy. How do I 'upgrade' pulp_installer? I tried to run the install command again which just says pulp.installer is already installed. Even downloading and trying to install from the file gives the same result. <br></div><div><br></div><div>Kind regards, <br></div><div><br></div><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><b>Eric Van Steenbergen</b><div><br><div><i>E-mail: <a href="mailto:vs.eric@gmail.com" target="_blank">vs.eric@gmail.com</a></i></div><div><i>Skype: ericvs2014</i></div></div><div><i>LinkedIn: <a href="http://es.linkedin.com/pub/eric-van-steenbergen/b/8a4/51b" target="_blank">Eric Van Steenbergen</a></i></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Apr 12, 2021 at 8:28 PM Gerrod Ubben <<a href="mailto:gubben@redhat.com">gubben@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi Eric,</div><div><br></div><div>This bug [0] was fixed in pulpcore 3.12 which we just finished releasing today. You can use the 3.12 pulp_installer [1] to upgrade to pulpcore 3.12 or change the erroring line [2] in your current installation if you don't want to upgrade right away. Also, all of Pulp's API endpoints require a trailing forward slash or else you'll get a 301 return. So to hit the status endpoint you use '/pulp/api/v3/status/', but we encourage everyone to use the CLI and make feature requests [3] if you find missing functionality.</div><div><br></div><div>Gerrod</div><div><br></div><div>[0] <a href="https://pulp.plan.io/issues/8475" target="_blank">https://pulp.plan.io/issues/8475</a></div><div>[1] <a href="https://galaxy.ansible.com/pulp/pulp_installer" target="_blank">https://galaxy.ansible.com/pulp/pulp_installer</a></div><div>[2] <a href="https://git.io/JOqh0" target="_blank">https://git.io/JOqh0</a></div><div>[3] <a href="https://github.com/pulp/pulp-cli/issues/new/" target="_blank">https://github.com/pulp/pulp-cli/issues/new/</a><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Apr 12, 2021 at 8:46 PM Eric VS <<a href="mailto:vs.eric@gmail.com" target="_blank">vs.eric@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hello Matthias and David, <br></div><div><br></div><div>Thanks again for your swift responses, I'm in a different time zone most likely so apologies for late replies. <br></div><div><br></div><div><a class="gmail_plusreply" id="gmail-m_-6520694324749413712gmail-m_-1049699748274478123gmail-m_-615331699219093804gmail-m_875164633610194773plusReplyChip-0" href="mailto:mdellweg@redhat.com" target="_blank">@Matthias Dellweg</a> , when running the v3/status I get this:<span style="font-family:monospace"><br><br></span></div><div><span style="font-family:monospace">╰─$ (⎈ k8s01.aops-dev.activeops.local:kube-system) http <a href="https://pulp-dev.activeops.io/pulp/api/v3/status" target="_blank">https://pulp-dev.activeops.io/pulp/api/v3/status</a><br>HTTP/1.1 301 Moved Permanently<br>Access-Control-Expose-Headers: Correlation-ID<br>Connection: keep-alive<br>Content-Length: 0<br>Content-Type: text/html; charset=utf-8<br>Correlation-ID: 4a98ac4213d44645ae9c963c81873780<br>Date: Tue, 13 Apr 2021 00:40:05 GMT<br>Location: /pulp/api/v3/status/<br>Server: nginx/1.16.1<br>Strict-Transport-Security: max-age=15768000<br>(ansible-venv) ╭─ericv@ip-10-80-1-57 ~</span></div><div><br></div><div>Tracing journalctl gives me the below when I run the /pulp/content with http.</div><div><span style="font-family:monospace"><br></span></div><div><span style="font-family:monospace">╰─$ (⎈ k8s01.aops-dev.activeops.local:kube-system) sudo journalctl -f -u pulpcore-content<br>-- Logs begin at Thu 2020-06-04 07:04:49 UTC. --<br>Apr 13 00:42:45 ip-10-80-1-57.ap-northeast-1.compute.internal gunicorn[1216]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/web_protocol.py", line 422, in _handle_request<br>Apr 13 00:42:45 ip-10-80-1-57.ap-northeast-1.compute.internal gunicorn[1216]: resp = await self._request_handler(request)<br>Apr 13 00:42:45 ip-10-80-1-57.ap-northeast-1.compute.internal gunicorn[1216]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/web_app.py", line 499, in _handle<br>Apr 13 00:42:45 ip-10-80-1-57.ap-northeast-1.compute.internal gunicorn[1216]: resp = await handler(request)<br>Apr 13 00:42:45 ip-10-80-1-57.ap-northeast-1.compute.internal gunicorn[1216]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/content/handler.py", line 121, in list_distributions<br>Apr 13 00:42:45 ip-10-80-1-57.ap-northeast-1.compute.internal gunicorn[1216]: directory_list = ["{}/".format(d.base_path) for d in base_paths]<br>Apr 13 00:42:45 ip-10-80-1-57.ap-northeast-1.compute.internal gunicorn[1216]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/content/handler.py", line 121, in <listcomp><br>Apr 13 00:42:45 ip-10-80-1-57.ap-northeast-1.compute.internal gunicorn[1216]: directory_list = ["{}/".format(d.base_path) for d in base_paths]<br>Apr 13 00:42:45 ip-10-80-1-57.ap-northeast-1.compute.internal gunicorn[1216]: AttributeError: 'str' object has no attribute 'base_path'<br>Apr 13 00:42:45 ip-10-80-1-57.ap-northeast-1.compute.internal gunicorn[1216]: 127.0.0.1 [13/Apr/2021:00:42:45 +0000] "GET /pulp/content/ HTTP/1.0" 500 231 "-" "HTTPie/0.9.4"<br><br><br>Apr 13 00:43:40 ip-10-80-1-57.ap-northeast-1.compute.internal gunicorn[1216]: [2021-04-13 00:43:40 +0000] [1452] [ERROR] Error handling request<br>Apr 13 00:43:40 ip-10-80-1-57.ap-northeast-1.compute.internal gunicorn[1216]: Traceback (most recent call last):<br>Apr 13 00:43:40 ip-10-80-1-57.ap-northeast-1.compute.internal gunicorn[1216]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/web_protocol.py", line 422, in _handle_request<br>Apr 13 00:43:40 ip-10-80-1-57.ap-northeast-1.compute.internal gunicorn[1216]: resp = await self._request_handler(request)<br>Apr 13 00:43:40 ip-10-80-1-57.ap-northeast-1.compute.internal gunicorn[1216]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/web_app.py", line 499, in _handle<br>Apr 13 00:43:40 ip-10-80-1-57.ap-northeast-1.compute.internal gunicorn[1216]: resp = await handler(request)<br>Apr 13 00:43:40 ip-10-80-1-57.ap-northeast-1.compute.internal gunicorn[1216]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/content/handler.py", line 121, in list_distributions<br>Apr 13 00:43:40 ip-10-80-1-57.ap-northeast-1.compute.internal gunicorn[1216]: directory_list = ["{}/".format(d.base_path) for d in base_paths]<br>Apr 13 00:43:40 ip-10-80-1-57.ap-northeast-1.compute.internal gunicorn[1216]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/content/handler.py", line 121, in <listcomp><br>Apr 13 00:43:40 ip-10-80-1-57.ap-northeast-1.compute.internal gunicorn[1216]: directory_list = ["{}/".format(d.base_path) for d in base_paths]<br>Apr 13 00:43:40 ip-10-80-1-57.ap-northeast-1.compute.internal gunicorn[1216]: AttributeError: 'str' object has no attribute 'base_path'<br>Apr 13 00:43:40 ip-10-80-1-57.ap-northeast-1.compute.internal gunicorn[1216]: 127.0.0.1 [13/Apr/2021:00:43:40 +0000] "GET /pulp/content/ HTTP/1.0" 500 231 "-" "HTTPie/0.9.4"</span></div><div><br></div><div>Kind regards, <br></div><div><br></div><div><div><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><b>Eric Van Steenbergen</b><div><br><div><i>E-mail: <a href="mailto:vs.eric@gmail.com" target="_blank">vs.eric@gmail.com</a></i></div><div><i>Skype: ericvs2014</i></div></div><div><i>LinkedIn: <a href="http://es.linkedin.com/pub/eric-van-steenbergen/b/8a4/51b" target="_blank">Eric Van Steenbergen</a></i></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Apr 12, 2021 at 9:55 AM David Davis <<a href="mailto:daviddavis@redhat.com" target="_blank">daviddavis@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">It could potentially be this bug which was fixed in pulpcore 3.12: <a href="https://pulp.plan.io/issues/8447" target="_blank">https://pulp.plan.io/issues/8447</a><div><br></div><div>The stacktrace would be helpful though to know for sure.<br clear="all"><div><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><br></div><div>David</div></div></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Apr 12, 2021 at 5:27 AM Matthias Dellweg <<a href="mailto:mdellweg@redhat.com" target="_blank">mdellweg@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Can you monitor "journalctl -f -u pulpcore-content" while performing that call?<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Apr 12, 2021 at 11:17 AM Matthias Dellweg <<a href="mailto:mdellweg@redhat.com" target="_blank">mdellweg@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Sorry, typo (e and 3 are too close on my k3yboard). Should have been .../pulp/api/v3/status</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Apr 12, 2021 at 11:06 AM Eric VS <<a href="mailto:vs.eric@gmail.com" target="_blank">vs.eric@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hey Matthias, <br></div><div><br></div><div>I installed Pulp using the pulp_installer Ansible playbook, no errors. <br></div><div><br></div><div>This is the output from `pulp status` since the query using http didn't work either:<br><br><span style="font-family:monospace">╰─$ (⎈ k8s01.aops-dev.activeops.local:kube-system) pulp status<br>{<br> "versions": [<br> {<br> "component": "core",<br> "version": "3.11.0"<br> },<br> {<br> "component": "rpm",<br> "version": "3.10.0"<br> }<br> ],<br> "online_workers": [<br> {<br> "pulp_created": "2021-03-31T07:59:28.269742Z",<br> "pulp_href": "/pulp/api/v3/workers/77c08991-a27a-49d6-a2c7-87cd8c26877c/",<br> "name": "resource-manager",<br> "last_heartbeat": "2021-04-12T09:02:03.424149Z"<br> },<br> {<br> "pulp_created": "2021-04-05T06:46:11.689705Z",<br> "pulp_href": "/pulp/api/v3/workers/ed4c9dca-db6d-4d52-9f0b-331a937d2755/",<br> "name": "1200@ip-10-80-1-57.ap-northeast-1.compute.internal",<br> "last_heartbeat": "2021-04-12T09:02:03.624644Z"<br> },<br> {<br> "pulp_created": "2021-04-05T06:46:11.689909Z",<br> "pulp_href": "/pulp/api/v3/workers/9d538294-a3aa-48d4-95ab-e0a213bae94f/",<br> "name": "1231@ip-10-80-1-57.ap-northeast-1.compute.internal",<br> "last_heartbeat": "2021-04-12T09:02:11.745839Z"<br> }<br> ],<br> "online_content_apps": [<br> {<br> "name": "1452@ip-10-80-1-57.ap-northeast-1.compute.internal",<br> "last_heartbeat": "2021-04-12T09:02:10.886966Z"<br> },<br> {<br> "name": "1453@ip-10-80-1-57.ap-northeast-1.compute.internal",<br> "last_heartbeat": "2021-04-12T09:02:13.566778Z"<br> },<br> {<br> "name": "1448@ip-10-80-1-57.ap-northeast-1.compute.internal",<br> "last_heartbeat": "2021-04-12T09:02:13.568876Z"<br> },<br> {<br> "name": "1447@ip-10-80-1-57.ap-northeast-1.compute.internal",<br> "last_heartbeat": "2021-04-12T09:02:15.876728Z"<br> },<br> {<br> "name": "1450@ip-10-80-1-57.ap-northeast-1.compute.internal",<br> "last_heartbeat": "2021-04-12T09:02:15.881384Z"<br> },<br> {<br> "name": "1454@ip-10-80-1-57.ap-northeast-1.compute.internal",<br> "last_heartbeat": "2021-04-12T09:02:15.881615Z"<br> },<br> {<br> "name": "1451@ip-10-80-1-57.ap-northeast-1.compute.internal",<br> "last_heartbeat": "2021-04-12T09:02:15.883369Z"<br> },<br> {<br> "name": "1449@ip-10-80-1-57.ap-northeast-1.compute.internal",<br> "last_heartbeat": "2021-04-12T09:02:15.885268Z"<br> }<br> ],<br> "database_connection": {<br> "connected": true<br> },<br> "redis_connection": {<br> "connected": true<br> },<br> "storage": {<br> "total": 161008844800,<br> "used": 109256830976,<br> "free": 51752013824<br> }<br>}</span><br></div><div><br></div><div>For reference this is the query you requested using api/ve/status</div><div><br></div><div><span style="font-family:monospace">╰─$ (⎈ k8s01.aops-dev.activeops.local:kube-system) http <a href="https://pulp-dev.activeops.io/pulp/api/ve/status" target="_blank">https://pulp-dev.activeops.io/pulp/api/ve/status</a><br>HTTP/1.1 404 Not Found<br>Access-Control-Expose-Headers: Correlation-ID<br>Connection: keep-alive<br>Content-Length: 77<br>Content-Type: text/html<br>Correlation-ID: 14cab8b25dc64591a243fd391424924f<br>Date: Mon, 12 Apr 2021 09:04:06 GMT<br>Server: nginx/1.16.1<br>X-Frame-Options: SAMEORIGIN<br><br><h1>Not Found</h1><p>The requested resource was not found on this server.</p><br></span></div><div><br></div><div>Kind regards, <br></div><div><br></div><div><div><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><b>Eric Van Steenbergen</b><div><br><div><i>E-mail: <a href="mailto:vs.eric@gmail.com" target="_blank">vs.eric@gmail.com</a></i></div><div><i>Skype: ericvs2014</i></div></div><div><i>LinkedIn: <a href="http://es.linkedin.com/pub/eric-van-steenbergen/b/8a4/51b" target="_blank">Eric Van Steenbergen</a></i></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Apr 12, 2021 at 4:00 AM Matthias Dellweg <<a href="mailto:mdellweg@redhat.com" target="_blank">mdellweg@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi Eric,</div><div>can you provide more info on this?</div><div>Helpful would be:</div><div>How is the server installed? (container, pulp_installer, by hand from pypi)</div><div>What versions of the components are installed? (http https://<FQDN>/pulp/api/ve/status)</div><div>Thanks in advance!<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Apr 12, 2021 at 10:57 AM Eric VS <<a href="mailto:vs.eric@gmail.com" target="_blank">vs.eric@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi Matthias, <br></div><div><br></div><div>When I run `http https://<FQDN>/pulp/content/` is when I get the 500 error. <br></div><div>I can query the all the content with the API (pulp rpm repository/remote/publication/distribution) without any issue. <br></div><div><br></div><div>Kind regards, <br></div><div><br></div><div><div><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><b>Eric Van Steenbergen</b><div><br><div><i>E-mail: <a href="mailto:vs.eric@gmail.com" target="_blank">vs.eric@gmail.com</a></i></div><div><i>Skype: ericvs2014</i></div></div><div><i>LinkedIn: <a href="http://es.linkedin.com/pub/eric-van-steenbergen/b/8a4/51b" target="_blank">Eric Van Steenbergen</a></i></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Apr 12, 2021 at 3:48 AM Matthias Dellweg <<a href="mailto:mdellweg@redhat.com" target="_blank">mdellweg@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Can you isolate the command that leads to this server response?<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Apr 12, 2021 at 1:55 AM Eric VS <<a href="mailto:vs.eric@gmail.com" target="_blank">vs.eric@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi, <br></div><div><br></div><div>I've been setting up another Pulp server, replicating repositories from my main one for 'local' use in an environment. When I was adding remotes and repositories at one point in time I started getting the below error:<br></div><div><br></div><div><b><span style="font-family:monospace">HTTP/1.1 500 Internal Server Error<br>Connection: keep-alive<br>Content-Length: 55<br>Content-Type: text/plain; charset=utf-8<br>Date: Sun, 11 Apr 2021 23:53:14 GMT<br>Server: nginx/1.16.1<br><br>500 Internal Server Error<br><br>Server got itself in trouble</span></b></div><div><br></div><div>I have enabled debug on Nginx but can't see anything indicating on why this all of a sudden fails.</div><div><br></div><div>Any help is greatly appreciated. <br></div><div><br></div><div>Kind regards, <br></div><div><br></div><div><div><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><b>Eric Van Steenbergen</b><div><br><div><i>E-mail: <a href="mailto:vs.eric@gmail.com" target="_blank">vs.eric@gmail.com</a></i></div><div><i>Skype: ericvs2014</i></div></div><div><i>LinkedIn: <a href="http://es.linkedin.com/pub/eric-van-steenbergen/b/8a4/51b" target="_blank">Eric Van Steenbergen</a></i></div></div></div></div></div></div></div></div> _______________________________________________<br> Pulp-list mailing list<br> <a href="mailto:Pulp-list@redhat.com" target="_blank">Pulp-list@redhat.com</a><br> <a href="https://listman.redhat.com/mailman/listinfo/pulp-list" rel="noreferrer" target="_blank">https://listman.redhat.com/mailman/listinfo/pulp-list</a></blockquote></div> </blockquote></div> </blockquote></div> </blockquote></div> </blockquote></div> </blockquote></div> _______________________________________________<br> Pulp-list mailing list<br> <a href="mailto:Pulp-list@redhat.com" target="_blank">Pulp-list@redhat.com</a><br> <a href="https://listman.redhat.com/mailman/listinfo/pulp-list" rel="noreferrer" target="_blank">https://listman.redhat.com/mailman/listinfo/pulp-list</a></blockquote></div> </blockquote></div> _______________________________________________<br> Pulp-list mailing list<br> <a href="mailto:Pulp-list@redhat.com" target="_blank">Pulp-list@redhat.com</a><br> <a href="https://listman.redhat.com/mailman/listinfo/pulp-list" rel="noreferrer" target="_blank">https://listman.redhat.com/mailman/listinfo/pulp-list</a></blockquote></div> </blockquote></div>