<div dir="ltr">Thanks Matthias. I get 502 at <a href="http://pulp.my.domain/pulp/api/v3/status/">http://pulp.my.domain/pulp/api/v3/status/</a> as well. Below is my nginx.conf, pulled from my freshly provisioned pulp server. My skills are a little weak on the webserver side of things so I'm open to suggestions for any simplifications I can make to my config to get this working. I'm not trying to do anything fancy here. <div> </div><div>/etc/nginx/nginx.conf:</div><div> </div><div># TODO: Support IPv6. # TODO: Configure SSL certificates. # TODO: Maybe serve multiple `location`s, not just one. # Gunicorn docs suggest this value. worker_processes 1; events { worker_connections 1024; # increase if you have lots of clients accept_mutex off; # set to 'on' if nginx worker_processes > 1 } http { include mime.types; # fallback in case we can't determine a type default_type application/octet-stream; sendfile on; # If left at the default of 1024, nginx emits a warning about being unable # to build optimal hash types. types_hash_max_size 4096; upstream pulp-content { server <a href="http://127.0.0.1:24816">127.0.0.1:24816</a>; } upstream pulp-api { server <a href="http://127.0.0.1:24817">127.0.0.1:24817</a>; } server { # Gunicorn docs suggest the use of the "deferred" directive on Linux. listen 80 default_server deferred; server_name $hostname; # The default client_max_body_size is 1m. Clients uploading # files larger than this will need to chunk said files. # Gunicorn docs suggest this value. keepalive_timeout 5; location /pulp/content/ { proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header Host $http_host; # we don't want nginx trying to do something clever with # redirects, we set the Host: header above already. proxy_redirect off; proxy_pass <a href="http://pulp-content">http://pulp-content</a>; } location /pulp/api/v3/ { proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header Host $http_host; # we don't want nginx trying to do something clever with # redirects, we set the Host: header above already. proxy_redirect off; proxy_pass <a href="http://pulp-api">http://pulp-api</a>; } location /auth/login/ { proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header Host $http_host; # we don't want nginx trying to do something clever with # redirects, we set the Host: header above already. proxy_redirect off; proxy_pass <a href="http://pulp-api">http://pulp-api</a>; } include pulp/*.conf; location / { proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header Host $http_host; # we don't want nginx trying to do something clever with # redirects, we set the Host: header above already. proxy_redirect off; proxy_pass <a href="http://pulp-api">http://pulp-api</a>; # static files are served through whitenoise - <a href="http://whitenoise.evans.io/en/stable/">http://whitenoise.evans.io/en/stable/</a> } } } </div></div> <div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jul 7, 2020 at 11:56 PM Matthias Dellweg <<a href="mailto:mdellweg@redhat.com">mdellweg@redhat.com</a>> wrote: </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">The only thing that sticks out to me is `content_origin: "http://{{ ansible_fqdn }}:8080"`. This is the address seen from the outside, and since both content and api are subject to the same reverse proxy and so should be available on port 80 (and 443 soon). But that is for sure not the problem you have with the API. Can you, however, try `http <a href="http://pulp.my.domain/pulp/api/v3/status/" rel="noreferrer" target="_blank">http://pulp.my.domain/pulp/api/v3/status/`</a>? And if it still didn't produce a result, provide the content of /etc/nginx/nginx.conf ? On Tue, Jul 7, 2020 at 11:18 PM Tim Black <<a href="mailto:timblaktu@gmail.com" target="_blank">timblaktu@gmail.com</a>> wrote: > > After perusing all of the roles' READMEs more thoroughly, I have updated my playbook (pasted below) with what I believe are the correct current set of available role variables in 3.4.1, with links to the docs for each. (would be nice if the example playbook was this informative.) One thing that came up with this exercise is that the example-use playbook is not including the main pulp role, however on tag 3.4.1 the pulp role appears to be a required dependency. Does the pulp role get included by the others, implicitly? > > Anyway, after a successful run of the modified playbook, I'm now seeing all services enabled: > > pulpadmin@pulp:~$ sudo systemctl list-unit-files | grep -E "(pulp|nginx)" > nginx.service enabled > pulpcore-api.service enabled > pulpcore-content.service enabled > pulpcore-resource-manager.service enabled > pulpcore-worker@.service indirect > dev-mapper-pulp\x2d\x2dvg\x2dswap_1.swap generated > > However, I'm still getting 502 trying to connect to pulp content webserver at my specified content_origin. > > My /var/log/nginx/error.log still shows the same type errors showing nginx can't connect with an upstream application server: > > 2020/07/07 13:59:41 [error] 12936#12936: *44 connect() failed (111: Connection refused) while connecting to upstream, client: 10.212.134.131, server: pulp, request: "GET /favicon.ico HTTP/1.1", upstream: "<a href="http://127.0.0.1:24817/favicon.ico" rel="noreferrer" target="_blank">http://127.0.0.1:24817/favicon.ico</a>", host: "pulp.my.domain", referrer: "<a href="http://pulp.my.domain/" rel="noreferrer" target="_blank">http://pulp.my.domain/</a>" > > Here's my updated pulp.yml: > > --- > # Playbook to provision and manage Pulp Instances for Artifact Management > > # Requires: > # (<a href="https://pulp-installer.readthedocs.io/en/latest/#system-requirements" rel="noreferrer" target="_blank">https://pulp-installer.readthedocs.io/en/latest/#system-requirements</a>) > # 1. Debian Buster Machine Provisioned using Preseeded Installer > # a. Really just need Debian install with: > # i. sudo, openssh-server, python3 > # (after installing with only ssh-server and system utility packages selected, only need to: > # su > # vi /etc/apt/sources.list # remove CD Rom line, add buster main repo if no mirror selected during install > # apt-get install sudo) > # ii. update-alternatives --set editor `update-alternatives --list editor | grep vim` > # iii. pulpadmin user with passwordless sudoer priviledges > # (echo "pulpadmin ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers) > # iv. ansible controller user has installed its ssh key in remote host's known_hosts > # (without this you'd just need to --ask-pass and supply ssh passwd at stdin) > # TODO: capture above in a VM Snapshot in vSphere/ESXi for fast reproduction. > # 2. Ansible Roles Installed via Galaxy using `$ ansible-galaxy install -r requirements-pulp.yml` > # 3. Ansible Collection Installed via Galaxy using `$ ansible-galaxy install -r requirements-pulp.yml` > > # Run like this: > # ansible-playbook pulp.yml --user pulpadmin --ask-pass --ask-vault-pass > # > # Note ansible knows what machines to run the playbook on by the `hosts` element within the playbook, > # which should have names existing in hosts file(s) in inventory/. > > # This playbook builds upon the Engineering Services playbook template > # Check imported playbook content before adding it here. > - import_playbook: engineering-services-tmplt.yml > > - name: "Install packages we want on every Pulp instance" > hosts: engineering_services_pulp > gather_facts: false > vars: > apt_packages: > - curl > roles: > - apt > > - name: Configure admin group > become: true > hosts: engineering_services_pulp > gather_facts: false > tasks: > - name: Create admin group > group: > name: admin > > - name: Configure admin user > become: true > hosts: engineering_services_pulp > gather_facts: false > vars: > # TODO: define these as inventory variable (standard for all machines?) so it can move out of playbook task blocks > tasks: > - debug: var=ansible_fqdn > - name: Configure admin user account > user: > name: admin > groups: > - admin > > - name: Install Pulp > hosts: engineering_services_pulp > # gather_facts: false > vars: > # Main Pulp Role Variables > # <a href="https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp#role-variables" rel="noreferrer" target="_blank">https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp#role-variables</a> > pulp_settings: > secret_key: !vault | > $ANSIBLE_VAULT;1.1;AES256 > 38383631633236306565616334663761363134613835323839653962323930616639656333653865 > 3264363735643430626361383132653632316139396364370a613566396133393430663962666261 > 35356165663639613535383563366638663635326662343133353339343262646265316630616162 > 6337346131303833610a663232633339306231613738653233646466383638333934393765373034 > 63346437343834653964366666333061303634313864333031323735326134626432626535613436 > 62643731343836626436383438643862396166636263646330646332633637363765623866343733 > 616635326537346163646564653134386666 > content_origin: "http://{{ ansible_fqdn }}:8080" > pulp_install_plugins: > pulp-ansible: {} > pulp-container: {} > pulp-deb: {} > pulp-file: {} > pulp-python: {} > pulp_default_admin_password: !vault | > $ANSIBLE_VAULT;1.1;AES256 > 35636365316538376363643965323035306461643239306433353665623438633535633763613662 > 6266346236393736616532636230393136303966383339310a306563323838326431386432626465 > 30316164383265303932643865323033623938656136306665356665336262613233653866386165 > 3164396261326563640a613464353364656130396333613531383864323434316533663932303766 > 3938 > pulp_api_bind: "{{ ansible_fqdn }}" > pulp_api_workers: 4 # defaults to 1 > > # Pulp Content Role Variables > # <a href="https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp_content#pulp_content" rel="noreferrer" target="_blank">https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp_content#pulp_content</a> > # pulp_content_bind: # Defaults to <a href="http://127.0.0.1:24816" rel="noreferrer" target="_blank">127.0.0.1:24816</a> > > # Pulp Database Role Variables > # <a href="https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp_database" rel="noreferrer" target="_blank">https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp_database</a> > # None > > # Pulp Resource Manager Role Variables > # <a href="https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp_resource_manager" rel="noreferrer" target="_blank">https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp_resource_manager</a> > # pulp_resouce_manager_state: # defaults to started > # pulp_resouce_manager_enabled: # defaults to true > > # Pulp Webserver Role Variables > # <a href="https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp_webserver" rel="noreferrer" target="_blank">https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp_webserver</a> > # pulp_webserver_server: # defauls to nginx > # pulp_content_port: # defaults to 24816 > # pulp_content_host: # defaults to localhost > # pulp_api_port: # defaults to 24817 > # pulp_api_host: # defaults to localhost > # pulp_configure_firewall: # defaults to auto, which is same as firewalld. Change to none to disable. > > # Pulp Workers Role Variables > # <a href="https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp_workers" rel="noreferrer" target="_blank">https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp_workers</a> > # TODO: how is this different from pulp_api_workers in the main Pulp Role?? > # pulp_workers: 4 # defaults to 2 > > pre_tasks: > # The version string below is the highest of all those in roles' metadata: > # "min_ansible_version". It needs to be kept manually up-to-date. > - name: Verify Ansible meets min required version > assert: > that: "ansible_version.full is version_compare('2.8', '>=')" > msg: > > "You must update Ansible to at least 2.8 to use this version of Pulp 3 Installer." > roles: > # Is pulp role implicitly included by the others? > - pulp_database > - pulp_workers > - pulp_resource_manager > - pulp_webserver > - pulp_content > environment: > DJANGO_SETTINGS_MODULE: pulpcore.app.settings > > On Tue, Jul 7, 2020 at 12:24 PM Tim Black <<a href="mailto:timblaktu@gmail.com" target="_blank">timblaktu@gmail.com</a>> wrote: >> >> I just installed my first pulp instance on a fresh Debian Buster VM, using latest Ansible pulp_installer release (3.4.1), with my pulp.yml playbook (pasted below) modeled after the official example-use playbook. The playbook runs to completion, with zero failed tasks, yet I am not able to connect to the pulp content webserver using the protocol/address/port I specified in the content_origin variable. I have verified that nginx service is running, but I still get 502: Bad Gateway error. >> >> Can someone help me troubleshoot this, or direct me to troubleshooting documentation that would assist? I found this excellent explanation which seems relevant since pulp uses the same nginx/gunicorn tech cocktail. It states: >> >>> NGINX will return a 502 Bad Gateway error if it can’t successfully proxy a request to Gunicorn or if Gunicorn fails to respond. >> >> >> I learned to look in /var/log/nginx/error.log for the reason for the issue. There I found several errors similar to this: >> >> [error] 4348#4348: *28 connect() failed (111: Connection refused) while connecting to upstream, client: 10.212.134.131, server: pulp, request: "GET / HTTP/1.1", upstream: "<a href="http://127.0.1.1:24817/" rel="noreferrer" target="_blank">http://127.0.1.1:24817/</a>", host: "pulp.my.domain" >> >> I also confirmed the following pulp service statuses: >> >> pulpadmin@pulp:~$ sudo systemctl list-unit-files | grep pulp >> pulpcore-api.service disabled >> pulpcore-content.service enabled >> pulpcore-resource-manager.service enabled >> pulpcore-worker@.service indirect >> dev-mapper-pulp\x2d\x2dvg\x2dswap_1.swap generated >> >> Hmm.. Shouldn't pulpcore-api be enabled? If so, I suppose this is the "upstream" service that nginx cannot connect to? From the error log, it looks like the address is localhost:24817, and I believe this is the default I chose. Anyone see any problem with what I'm doing here? I'm simply trying to set up "hello world" with pulp_installer targeting a dedicated remote server. >> >> I applaud the pulp dev team's modularizing of the code base, but I would love to see more documentation on the architecture here, clearly illustrating all these moving parts, with links to common problems like I'm having, with troubleshooting advice. >> >> Here's my pulp.yml ansible playbook: >> >> --- >> # Playbook to provision and manage Pulp Instances for Artifact Management >> >> # Requires: >> # (<a href="https://pulp-installer.readthedocs.io/en/latest/#system-requirements" rel="noreferrer" target="_blank">https://pulp-installer.readthedocs.io/en/latest/#system-requirements</a>) >> # 1. Debian Buster Machine Provisioned using Preseeded Installer >> # a. Really just need Debian install with: >> # i. sudo, openssh-server, python3 >> # (after installing with only ssh-server and system utility packages selected, only need to: >> # su >> # vi /etc/apt/sources.list # remove CD Rom line, add buster main repo if no mirror selected during install >> # apt-get install sudo) >> # ii. update-alternatives --set editor `update-alternatives --list editor | grep vim` >> # iii. pulpadmin user with passwordless sudoer priviledges >> # (echo "pulpadmin ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers) >> # iv. ansible controller user has installed its ssh key in remote host's known_hosts >> # (without this you'd just need to --ask-pass and supply ssh passwd at stdin) >> # TODO: capture above in a VM Snapshot in vSphere/ESXi for fast reproduction. >> # 2. Ansible Roles Installed via Galaxy using `$ ansible-galaxy install -r requirements-pulp.yml` >> # 3. Ansible Collection Installed via Galaxy using `$ ansible-galaxy install -r requirements-pulp.yml` >> # >> # Run like this: >> # ansible-playbook pulp.yml --user pulpadmin -l <controlled-pulp-hostname> --ask-pass --ask-vault-pass >> >> # This playbook builds upon the Engineering Services playbook template >> # Check imported playbook content before adding it here. >> - import_playbook: engineering-services-tmplt.yml >> >> - name: "Install packages we want on every Pulp instance" >> hosts: engineering_services_pulp >> gather_facts: false >> vars: >> apt_packages: >> - curl >> roles: >> - apt >> >> - name: Configure admin group >> become: true >> hosts: engineering_services_pulp >> gather_facts: false >> tasks: >> - name: Create admin group >> group: >> name: admin >> >> - name: Configure admin user >> become: true >> hosts: engineering_services_pulp >> gather_facts: false >> vars: >> # TODO: define these as inventory variable (standard for all machines?) so it can move out of playbook task blocks >> tasks: >> - debug: var=ansible_fqdn >> - name: Configure admin user account >> user: >> name: admin >> groups: >> - admin >> >> - name: Install Pulp >> hosts: engineering_services_pulp >> # gather_facts: false >> vars: >> # required by pulp_installer: <a href="https://pulp-installer.readthedocs.io/en/latest/#system-requirements" rel="noreferrer" target="_blank">https://pulp-installer.readthedocs.io/en/latest/#system-requirements</a> >> # TODO: this is now set in ansible.cfg bc it doesn't work when set here or in inventory >> # allow_world_readable_tmpfiles: True >> pulp_settings: >> secret_key: !vault | >> $ANSIBLE_VAULT;1.1;AES256 >> 38383631633236306565616334663761363134613835323839653962323930616639656333653865 >> 3264363735643430626361383132653632316139396364370a613566396133393430663962666261 >> 35356165663639613535383563366638663635326662343133353339343262646265316630616162 >> 6337346131303833610a663232633339306231613738653233646466383638333934393765373034 >> 63346437343834653964366666333061303634313864333031323735326134626432626535613436 >> 62643731343836626436383438643862396166636263646330646332633637363765623866343733 >> 616635326537346163646564653134386666 >> content_origin: "http://{{ ansible_fqdn }}:8080" >> pulp_default_admin_password: !vault | >> $ANSIBLE_VAULT;1.1;AES256 >> 35636365316538376363643965323035306461643239306433353665623438633535633763613662 >> 6266346236393736616532636230393136303966383339310a306563323838326431386432626465 >> 30316164383265303932643865323033623938656136306665356665336262613233653866386165 >> 3164396261326563640a613464353364656130396333613531383864323434316533663932303766 >> 3938 >> pulp_content_host: "{{ ansible_fqdn }}" >> # pulp_content_port: 24816 >> pulp_content_port: 8080 >> pulp_api_host: "{{ ansible_fqdn }}" >> # pulp_content_port: 24817 >> pulp_content_bind: "{{ pulp_content_host }}:{{ pulp_content_port }}" >> pulp_install_plugins: >> # galaxy-ng: {} >> pulp-ansible: {} >> # pulp-certguard: {} >> pulp-container: {} >> # pulp-cookbook: {} >> pulp-deb: {} >> pulp-file: {} >> # pulp-gem: {} >> # pulp-maven: {} >> # pulp-npm: {} >> pulp-python: {} >> # pulp-rpm: {} >> pre_tasks: >> # The version string below is the highest of all those in roles' metadata: >> # "min_ansible_version". It needs to be kept manually up-to-date. >> - name: Verify Ansible meets min required version >> assert: >> that: "ansible_version.full is version_compare('2.8', '>=')" >> msg: > >> "You must update Ansible to at least 2.8 to use this version of Pulp 3 Installer." >> roles: >> - pulp_database >> - pulp_workers >> - pulp_resource_manager >> - pulp_webserver >> - pulp_content >> environment: >> DJANGO_SETTINGS_MODULE: pulpcore.app.settings >> >> Thanks for your help. >> >> Tim > > _______________________________________________ > Pulp-list mailing list > <a href="mailto:Pulp-list@redhat.com" target="_blank">Pulp-list@redhat.com</a> > <a href="https://www.redhat.com/mailman/listinfo/pulp-list" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-list</a> </blockquote></div>