[Pulp-list] 502 Bad Gateway error connecting to new pulp instance installed with pulp_installer
Matthias Dellweg
mdellweg at redhat.com
Wed Jul 8 06:56:29 UTC 2020
The only thing that sticks out to me is `content_origin: "http://{{
ansible_fqdn }}:8080"`. This is the address seen from the outside, and
since both content and api are subject to the same reverse proxy and
so should be available on port 80 (and 443 soon). But that is for sure
not the problem you have with the API.
Can you, however, try `http
http://pulp.my.domain/pulp/api/v3/status/`? And if it still didn't
produce a result, provide the content of /etc/nginx/nginx.conf ?
On Tue, Jul 7, 2020 at 11:18 PM Tim Black <timblaktu at gmail.com> wrote:
>
> After perusing all of the roles' READMEs more thoroughly, I have updated my playbook (pasted below) with what I believe are the correct current set of available role variables in 3.4.1, with links to the docs for each. (would be nice if the example playbook was this informative.) One thing that came up with this exercise is that the example-use playbook is not including the main pulp role, however on tag 3.4.1 the pulp role appears to be a required dependency. Does the pulp role get included by the others, implicitly?
>
> Anyway, after a successful run of the modified playbook, I'm now seeing all services enabled:
>
> pulpadmin at pulp:~$ sudo systemctl list-unit-files | grep -E "(pulp|nginx)"
> nginx.service enabled
> pulpcore-api.service enabled
> pulpcore-content.service enabled
> pulpcore-resource-manager.service enabled
> pulpcore-worker at .service indirect
> dev-mapper-pulp\x2d\x2dvg\x2dswap_1.swap generated
>
> However, I'm still getting 502 trying to connect to pulp content webserver at my specified content_origin.
>
> My /var/log/nginx/error.log still shows the same type errors showing nginx can't connect with an upstream application server:
>
> 2020/07/07 13:59:41 [error] 12936#12936: *44 connect() failed (111: Connection refused) while connecting to upstream, client: 10.212.134.131, server: pulp, request: "GET /favicon.ico HTTP/1.1", upstream: "http://127.0.0.1:24817/favicon.ico", host: "pulp.my.domain", referrer: "http://pulp.my.domain/"
>
> Here's my updated pulp.yml:
>
> ---
> # Playbook to provision and manage Pulp Instances for Artifact Management
>
> # Requires:
> # (https://pulp-installer.readthedocs.io/en/latest/#system-requirements)
> # 1. Debian Buster Machine Provisioned using Preseeded Installer
> # a. Really just need Debian install with:
> # i. sudo, openssh-server, python3
> # (after installing with only ssh-server and system utility packages selected, only need to:
> # su
> # vi /etc/apt/sources.list # remove CD Rom line, add buster main repo if no mirror selected during install
> # apt-get install sudo)
> # ii. update-alternatives --set editor `update-alternatives --list editor | grep vim`
> # iii. pulpadmin user with passwordless sudoer priviledges
> # (echo "pulpadmin ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers)
> # iv. ansible controller user has installed its ssh key in remote host's known_hosts
> # (without this you'd just need to --ask-pass and supply ssh passwd at stdin)
> # TODO: capture above in a VM Snapshot in vSphere/ESXi for fast reproduction.
> # 2. Ansible Roles Installed via Galaxy using `$ ansible-galaxy install -r requirements-pulp.yml`
> # 3. Ansible Collection Installed via Galaxy using `$ ansible-galaxy install -r requirements-pulp.yml`
>
> # Run like this:
> # ansible-playbook pulp.yml --user pulpadmin --ask-pass --ask-vault-pass
> #
> # Note ansible knows what machines to run the playbook on by the `hosts` element within the playbook,
> # which should have names existing in hosts file(s) in inventory/.
>
> # This playbook builds upon the Engineering Services playbook template
> # Check imported playbook content before adding it here.
> - import_playbook: engineering-services-tmplt.yml
>
> - name: "Install packages we want on every Pulp instance"
> hosts: engineering_services_pulp
> gather_facts: false
> vars:
> apt_packages:
> - curl
> roles:
> - apt
>
> - name: Configure admin group
> become: true
> hosts: engineering_services_pulp
> gather_facts: false
> tasks:
> - name: Create admin group
> group:
> name: admin
>
> - name: Configure admin user
> become: true
> hosts: engineering_services_pulp
> gather_facts: false
> vars:
> # TODO: define these as inventory variable (standard for all machines?) so it can move out of playbook task blocks
> tasks:
> - debug: var=ansible_fqdn
> - name: Configure admin user account
> user:
> name: admin
> groups:
> - admin
>
> - name: Install Pulp
> hosts: engineering_services_pulp
> # gather_facts: false
> vars:
> # Main Pulp Role Variables
> # https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp#role-variables
> pulp_settings:
> secret_key: !vault |
> $ANSIBLE_VAULT;1.1;AES256
> 38383631633236306565616334663761363134613835323839653962323930616639656333653865
> 3264363735643430626361383132653632316139396364370a613566396133393430663962666261
> 35356165663639613535383563366638663635326662343133353339343262646265316630616162
> 6337346131303833610a663232633339306231613738653233646466383638333934393765373034
> 63346437343834653964366666333061303634313864333031323735326134626432626535613436
> 62643731343836626436383438643862396166636263646330646332633637363765623866343733
> 616635326537346163646564653134386666
> content_origin: "http://{{ ansible_fqdn }}:8080"
> pulp_install_plugins:
> pulp-ansible: {}
> pulp-container: {}
> pulp-deb: {}
> pulp-file: {}
> pulp-python: {}
> pulp_default_admin_password: !vault |
> $ANSIBLE_VAULT;1.1;AES256
> 35636365316538376363643965323035306461643239306433353665623438633535633763613662
> 6266346236393736616532636230393136303966383339310a306563323838326431386432626465
> 30316164383265303932643865323033623938656136306665356665336262613233653866386165
> 3164396261326563640a613464353364656130396333613531383864323434316533663932303766
> 3938
> pulp_api_bind: "{{ ansible_fqdn }}"
> pulp_api_workers: 4 # defaults to 1
>
> # Pulp Content Role Variables
> # https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp_content#pulp_content
> # pulp_content_bind: # Defaults to 127.0.0.1:24816
>
> # Pulp Database Role Variables
> # https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp_database
> # None
>
> # Pulp Resource Manager Role Variables
> # https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp_resource_manager
> # pulp_resouce_manager_state: # defaults to started
> # pulp_resouce_manager_enabled: # defaults to true
>
> # Pulp Webserver Role Variables
> # https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp_webserver
> # pulp_webserver_server: # defauls to nginx
> # pulp_content_port: # defaults to 24816
> # pulp_content_host: # defaults to localhost
> # pulp_api_port: # defaults to 24817
> # pulp_api_host: # defaults to localhost
> # pulp_configure_firewall: # defaults to auto, which is same as firewalld. Change to none to disable.
>
> # Pulp Workers Role Variables
> # https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp_workers
> # TODO: how is this different from pulp_api_workers in the main Pulp Role??
> # pulp_workers: 4 # defaults to 2
>
> pre_tasks:
> # The version string below is the highest of all those in roles' metadata:
> # "min_ansible_version". It needs to be kept manually up-to-date.
> - name: Verify Ansible meets min required version
> assert:
> that: "ansible_version.full is version_compare('2.8', '>=')"
> msg: >
> "You must update Ansible to at least 2.8 to use this version of Pulp 3 Installer."
> roles:
> # Is pulp role implicitly included by the others?
> - pulp_database
> - pulp_workers
> - pulp_resource_manager
> - pulp_webserver
> - pulp_content
> environment:
> DJANGO_SETTINGS_MODULE: pulpcore.app.settings
>
> On Tue, Jul 7, 2020 at 12:24 PM Tim Black <timblaktu at gmail.com> wrote:
>>
>> I just installed my first pulp instance on a fresh Debian Buster VM, using latest Ansible pulp_installer release (3.4.1), with my pulp.yml playbook (pasted below) modeled after the official example-use playbook. The playbook runs to completion, with zero failed tasks, yet I am not able to connect to the pulp content webserver using the protocol/address/port I specified in the content_origin variable. I have verified that nginx service is running, but I still get 502: Bad Gateway error.
>>
>> Can someone help me troubleshoot this, or direct me to troubleshooting documentation that would assist? I found this excellent explanation which seems relevant since pulp uses the same nginx/gunicorn tech cocktail. It states:
>>
>>> NGINX will return a 502 Bad Gateway error if it can’t successfully proxy a request to Gunicorn or if Gunicorn fails to respond.
>>
>>
>> I learned to look in /var/log/nginx/error.log for the reason for the issue. There I found several errors similar to this:
>>
>> [error] 4348#4348: *28 connect() failed (111: Connection refused) while connecting to upstream, client: 10.212.134.131, server: pulp, request: "GET / HTTP/1.1", upstream: "http://127.0.1.1:24817/", host: "pulp.my.domain"
>>
>> I also confirmed the following pulp service statuses:
>>
>> pulpadmin at pulp:~$ sudo systemctl list-unit-files | grep pulp
>> pulpcore-api.service disabled
>> pulpcore-content.service enabled
>> pulpcore-resource-manager.service enabled
>> pulpcore-worker at .service indirect
>> dev-mapper-pulp\x2d\x2dvg\x2dswap_1.swap generated
>>
>> Hmm.. Shouldn't pulpcore-api be enabled? If so, I suppose this is the "upstream" service that nginx cannot connect to? From the error log, it looks like the address is localhost:24817, and I believe this is the default I chose. Anyone see any problem with what I'm doing here? I'm simply trying to set up "hello world" with pulp_installer targeting a dedicated remote server.
>>
>> I applaud the pulp dev team's modularizing of the code base, but I would love to see more documentation on the architecture here, clearly illustrating all these moving parts, with links to common problems like I'm having, with troubleshooting advice.
>>
>> Here's my pulp.yml ansible playbook:
>>
>> ---
>> # Playbook to provision and manage Pulp Instances for Artifact Management
>>
>> # Requires:
>> # (https://pulp-installer.readthedocs.io/en/latest/#system-requirements)
>> # 1. Debian Buster Machine Provisioned using Preseeded Installer
>> # a. Really just need Debian install with:
>> # i. sudo, openssh-server, python3
>> # (after installing with only ssh-server and system utility packages selected, only need to:
>> # su
>> # vi /etc/apt/sources.list # remove CD Rom line, add buster main repo if no mirror selected during install
>> # apt-get install sudo)
>> # ii. update-alternatives --set editor `update-alternatives --list editor | grep vim`
>> # iii. pulpadmin user with passwordless sudoer priviledges
>> # (echo "pulpadmin ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers)
>> # iv. ansible controller user has installed its ssh key in remote host's known_hosts
>> # (without this you'd just need to --ask-pass and supply ssh passwd at stdin)
>> # TODO: capture above in a VM Snapshot in vSphere/ESXi for fast reproduction.
>> # 2. Ansible Roles Installed via Galaxy using `$ ansible-galaxy install -r requirements-pulp.yml`
>> # 3. Ansible Collection Installed via Galaxy using `$ ansible-galaxy install -r requirements-pulp.yml`
>> #
>> # Run like this:
>> # ansible-playbook pulp.yml --user pulpadmin -l <controlled-pulp-hostname> --ask-pass --ask-vault-pass
>>
>> # This playbook builds upon the Engineering Services playbook template
>> # Check imported playbook content before adding it here.
>> - import_playbook: engineering-services-tmplt.yml
>>
>> - name: "Install packages we want on every Pulp instance"
>> hosts: engineering_services_pulp
>> gather_facts: false
>> vars:
>> apt_packages:
>> - curl
>> roles:
>> - apt
>>
>> - name: Configure admin group
>> become: true
>> hosts: engineering_services_pulp
>> gather_facts: false
>> tasks:
>> - name: Create admin group
>> group:
>> name: admin
>>
>> - name: Configure admin user
>> become: true
>> hosts: engineering_services_pulp
>> gather_facts: false
>> vars:
>> # TODO: define these as inventory variable (standard for all machines?) so it can move out of playbook task blocks
>> tasks:
>> - debug: var=ansible_fqdn
>> - name: Configure admin user account
>> user:
>> name: admin
>> groups:
>> - admin
>>
>> - name: Install Pulp
>> hosts: engineering_services_pulp
>> # gather_facts: false
>> vars:
>> # required by pulp_installer: https://pulp-installer.readthedocs.io/en/latest/#system-requirements
>> # TODO: this is now set in ansible.cfg bc it doesn't work when set here or in inventory
>> # allow_world_readable_tmpfiles: True
>> pulp_settings:
>> secret_key: !vault |
>> $ANSIBLE_VAULT;1.1;AES256
>> 38383631633236306565616334663761363134613835323839653962323930616639656333653865
>> 3264363735643430626361383132653632316139396364370a613566396133393430663962666261
>> 35356165663639613535383563366638663635326662343133353339343262646265316630616162
>> 6337346131303833610a663232633339306231613738653233646466383638333934393765373034
>> 63346437343834653964366666333061303634313864333031323735326134626432626535613436
>> 62643731343836626436383438643862396166636263646330646332633637363765623866343733
>> 616635326537346163646564653134386666
>> content_origin: "http://{{ ansible_fqdn }}:8080"
>> pulp_default_admin_password: !vault |
>> $ANSIBLE_VAULT;1.1;AES256
>> 35636365316538376363643965323035306461643239306433353665623438633535633763613662
>> 6266346236393736616532636230393136303966383339310a306563323838326431386432626465
>> 30316164383265303932643865323033623938656136306665356665336262613233653866386165
>> 3164396261326563640a613464353364656130396333613531383864323434316533663932303766
>> 3938
>> pulp_content_host: "{{ ansible_fqdn }}"
>> # pulp_content_port: 24816
>> pulp_content_port: 8080
>> pulp_api_host: "{{ ansible_fqdn }}"
>> # pulp_content_port: 24817
>> pulp_content_bind: "{{ pulp_content_host }}:{{ pulp_content_port }}"
>> pulp_install_plugins:
>> # galaxy-ng: {}
>> pulp-ansible: {}
>> # pulp-certguard: {}
>> pulp-container: {}
>> # pulp-cookbook: {}
>> pulp-deb: {}
>> pulp-file: {}
>> # pulp-gem: {}
>> # pulp-maven: {}
>> # pulp-npm: {}
>> pulp-python: {}
>> # pulp-rpm: {}
>> pre_tasks:
>> # The version string below is the highest of all those in roles' metadata:
>> # "min_ansible_version". It needs to be kept manually up-to-date.
>> - name: Verify Ansible meets min required version
>> assert:
>> that: "ansible_version.full is version_compare('2.8', '>=')"
>> msg: >
>> "You must update Ansible to at least 2.8 to use this version of Pulp 3 Installer."
>> roles:
>> - pulp_database
>> - pulp_workers
>> - pulp_resource_manager
>> - pulp_webserver
>> - pulp_content
>> environment:
>> DJANGO_SETTINGS_MODULE: pulpcore.app.settings
>>
>> Thanks for your help.
>>
>> Tim
>
> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
More information about the Pulp-list
mailing list