[Pulp-list] 502 Bad Gateway error connecting to new pulp instance installed with pulp_installer
Tim Black
timblaktu at gmail.com
Tue Jul 7 21:18:09 UTC 2020
After perusing all of the roles' READMEs more thoroughly, I have updated my
playbook (pasted below) with what I believe are the correct current set of
available role variables in 3.4.1, with links to the docs for each. (would
be nice if the example playbook was this informative.) One thing that came
up with this exercise is that the example-use playbook is not including the
main pulp role, however on tag 3.4.1 the pulp role appears to be a required
dependency. Does the pulp role get included by the others, implicitly?
Anyway, after a successful run of the modified playbook, I'm now seeing all
services enabled:
pulpadmin at pulp:~$ sudo systemctl list-unit-files | grep -E "(pulp|nginx)"
nginx.service enabled
pulpcore-api.service enabled
pulpcore-content.service enabled
pulpcore-resource-manager.service enabled
pulpcore-worker at .service indirect
dev-mapper-pulp\x2d\x2dvg\x2dswap_1.swap generated
However, I'm still getting 502 trying to connect to pulp content webserver
at my specified content_origin.
My /var/log/nginx/error.log still shows the same type errors showing nginx
can't connect with an upstream application server:
2020/07/07 13:59:41 [error] 12936#12936: *44 connect() failed (111:
Connection refused) while connecting to upstream, client: 10.212.134.131,
server: pulp, request: "GET /favicon.ico HTTP/1.1", upstream: "
http://127.0.0.1:24817/favicon.ico", host: "pulp.my.domain", referrer: "
http://pulp.my.domain/"
Here's my updated pulp.yml:
---
# Playbook to provision and manage Pulp Instances for Artifact Management
# Requires:
# (https://pulp-installer.readthedocs.io/en/latest/#system-requirements)
# 1. Debian Buster Machine Provisioned using Preseeded Installer
# a. Really just need Debian install with:
# i. sudo, openssh-server, python3
# (after installing with only ssh-server and system utility
packages selected, only need to:
# su
# vi /etc/apt/sources.list # remove CD Rom line, add buster main
repo if no mirror selected during install
# apt-get install sudo)
# ii. update-alternatives --set editor `update-alternatives --list
editor | grep vim`
# iii. pulpadmin user with passwordless sudoer priviledges
# (echo "pulpadmin ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers)
# iv. ansible controller user has installed its ssh key in remote
host's known_hosts
# (without this you'd just need to --ask-pass and supply ssh
passwd at stdin)
# TODO: capture above in a VM Snapshot in vSphere/ESXi for fast
reproduction.
# 2. Ansible Roles Installed via Galaxy using `$ ansible-galaxy install -r
requirements-pulp.yml`
# 3. Ansible Collection Installed via Galaxy using `$ ansible-galaxy
install -r requirements-pulp.yml`
# Run like this:
# ansible-playbook pulp.yml --user pulpadmin --ask-pass --ask-vault-pass
#
# Note ansible knows what machines to run the playbook on by the `hosts`
element within the playbook,
# which should have names existing in hosts file(s) in inventory/.
# This playbook builds upon the Engineering Services playbook template
# Check imported playbook content before adding it here.
- import_playbook: engineering-services-tmplt.yml
- name: "Install packages we want on every Pulp instance"
hosts: engineering_services_pulp
gather_facts: false
vars:
apt_packages:
- curl
roles:
- apt
- name: Configure admin group
become: true
hosts: engineering_services_pulp
gather_facts: false
tasks:
- name: Create admin group
group:
name: admin
- name: Configure admin user
become: true
hosts: engineering_services_pulp
gather_facts: false
vars:
# TODO: define these as inventory variable (standard for all machines?)
so it can move out of playbook task blocks
tasks:
- debug: var=ansible_fqdn
- name: Configure admin user account
user:
name: admin
groups:
- admin
- name: Install Pulp
hosts: engineering_services_pulp
# gather_facts: false
vars:
# Main Pulp Role Variables
#
https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp#role-variables
pulp_settings:
secret_key: !vault |
$ANSIBLE_VAULT;1.1;AES256
38383631633236306565616334663761363134613835323839653962323930616639656333653865
3264363735643430626361383132653632316139396364370a613566396133393430663962666261
35356165663639613535383563366638663635326662343133353339343262646265316630616162
6337346131303833610a663232633339306231613738653233646466383638333934393765373034
63346437343834653964366666333061303634313864333031323735326134626432626535613436
62643731343836626436383438643862396166636263646330646332633637363765623866343733
616635326537346163646564653134386666
content_origin: "http://{{ ansible_fqdn }}:8080"
pulp_install_plugins:
pulp-ansible: {}
pulp-container: {}
pulp-deb: {}
pulp-file: {}
pulp-python: {}
pulp_default_admin_password: !vault |
$ANSIBLE_VAULT;1.1;AES256
35636365316538376363643965323035306461643239306433353665623438633535633763613662
6266346236393736616532636230393136303966383339310a306563323838326431386432626465
30316164383265303932643865323033623938656136306665356665336262613233653866386165
3164396261326563640a613464353364656130396333613531383864323434316533663932303766
3938
pulp_api_bind: "{{ ansible_fqdn }}"
pulp_api_workers: 4 # defaults to 1
# Pulp Content Role Variables
#
https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp_content#pulp_content
# pulp_content_bind: # Defaults to 127.0.0.1:24816
# Pulp Database Role Variables
#
https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp_database
# None
# Pulp Resource Manager Role Variables
#
https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp_resource_manager
# pulp_resouce_manager_state: # defaults to started
# pulp_resouce_manager_enabled: # defaults to true
# Pulp Webserver Role Variables
#
https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp_webserver
# pulp_webserver_server: # defauls to nginx
# pulp_content_port: # defaults to 24816
# pulp_content_host: # defaults to localhost
# pulp_api_port: # defaults to 24817
# pulp_api_host: # defaults to localhost
# pulp_configure_firewall: # defaults to auto, which is same as
firewalld. Change to none to disable.
# Pulp Workers Role Variables
# https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp_workers
# TODO: how is this different from pulp_api_workers in the main Pulp
Role??
# pulp_workers: 4 # defaults to 2
pre_tasks:
# The version string below is the highest of all those in roles'
metadata:
# "min_ansible_version". It needs to be kept manually up-to-date.
- name: Verify Ansible meets min required version
assert:
that: "ansible_version.full is version_compare('2.8', '>=')"
msg: >
"You must update Ansible to at least 2.8 to use this version of
Pulp 3 Installer."
roles:
# Is pulp role implicitly included by the others?
- pulp_database
- pulp_workers
- pulp_resource_manager
- pulp_webserver
- pulp_content
environment:
DJANGO_SETTINGS_MODULE: pulpcore.app.settings
On Tue, Jul 7, 2020 at 12:24 PM Tim Black <timblaktu at gmail.com> wrote:
> I just installed my first pulp instance on a fresh Debian Buster VM, using
> latest Ansible pulp_installer release (3.4.1), with my pulp.yml playbook
> (pasted below) modeled after the official example-use playbook
> <https://github.com/pulp/pulp_installer/tree/3.4.1/playbooks/example-use>.
> The playbook runs to completion, with zero failed tasks, yet I am not able
> to connect to the pulp content webserver using the protocol/address/port I
> specified in the content_origin variable. I have verified that nginx
> service is running, but I still get 502: Bad Gateway error.
>
> Can someone help me troubleshoot this, or direct me to troubleshooting
> documentation that would assist? I found this excellent explanation
> <https://www.datadoghq.com/blog/nginx-502-bad-gateway-errors-gunicorn/>which
> seems relevant since pulp uses the same nginx/gunicorn tech cocktail. It
> states:
>
> NGINX will return a 502 Bad Gateway error if it can’t successfully proxy a
>> request to Gunicorn or if Gunicorn fails to respond.
>
>
> I learned to look in /var/log/nginx/error.log for the reason for the
> issue. There I found several errors similar to this:
>
> [error] 4348#4348: *28 connect() failed (111: Connection refused) while
> connecting to upstream, client: 10.212.134.131, server: pulp, request: "GET
> / HTTP/1.1", upstream: "http://127.0.1.1:24817/", host: "pulp.my.domain"
>
> I also confirmed the following pulp service statuses:
>
> pulpadmin at pulp:~$ sudo systemctl list-unit-files | grep pulp
>
> *pulpcore-api.service disabled*pulpcore-content.service
> enabled
> pulpcore-resource-manager.service enabled
> pulpcore-worker at .service indirect
> dev-mapper-pulp\x2d\x2dvg\x2dswap_1.swap generated
>
> Hmm.. *Shouldn't pulpcore-api be enabled*? If so, I suppose this is the
> "upstream" service that nginx cannot connect to? From the error log, it
> looks like the address is localhost:24817, and I believe this is the
> default I chose. Anyone see any problem with what I'm doing here? I'm
> simply trying to set up "hello world" with pulp_installer targeting a
> dedicated remote server.
>
> I applaud the pulp dev team's modularizing of the code base, but I would
> love to see more documentation on the architecture here, clearly
> illustrating all these moving parts, with links to common problems like I'm
> having, with troubleshooting advice.
>
> Here's my pulp.yml ansible playbook:
>
> ---
> # Playbook to provision and manage Pulp Instances for Artifact Management
>
> # Requires:
> # (https://pulp-installer.readthedocs.io/en/latest/#system-requirements)
> # 1. Debian Buster Machine Provisioned using Preseeded Installer
> # a. Really just need Debian install with:
> # i. sudo, openssh-server, python3
> # (after installing with only ssh-server and system utility
> packages selected, only need to:
> # su
> # vi /etc/apt/sources.list # remove CD Rom line, add buster
> main repo if no mirror selected during install
> # apt-get install sudo)
> # ii. update-alternatives --set editor `update-alternatives --list
> editor | grep vim`
> # iii. pulpadmin user with passwordless sudoer priviledges
> # (echo "pulpadmin ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers)
> # iv. ansible controller user has installed its ssh key in remote
> host's known_hosts
> # (without this you'd just need to --ask-pass and supply ssh
> passwd at stdin)
> # TODO: capture above in a VM Snapshot in vSphere/ESXi for fast
> reproduction.
> # 2. Ansible Roles Installed via Galaxy using `$ ansible-galaxy install -r
> requirements-pulp.yml`
> # 3. Ansible Collection Installed via Galaxy using `$ ansible-galaxy
> install -r requirements-pulp.yml`
> #
> # Run like this:
> # ansible-playbook pulp.yml --user pulpadmin -l
> <controlled-pulp-hostname> --ask-pass --ask-vault-pass
>
> # This playbook builds upon the Engineering Services playbook template
> # Check imported playbook content before adding it here.
> - import_playbook: engineering-services-tmplt.yml
>
> - name: "Install packages we want on every Pulp instance"
> hosts: engineering_services_pulp
> gather_facts: false
> vars:
> apt_packages:
> - curl
> roles:
> - apt
>
> - name: Configure admin group
> become: true
> hosts: engineering_services_pulp
> gather_facts: false
> tasks:
> - name: Create admin group
> group:
> name: admin
>
> - name: Configure admin user
> become: true
> hosts: engineering_services_pulp
> gather_facts: false
> vars:
> # TODO: define these as inventory variable (standard for all
> machines?) so it can move out of playbook task blocks
> tasks:
> - debug: var=ansible_fqdn
> - name: Configure admin user account
> user:
> name: admin
> groups:
> - admin
>
> - name: Install Pulp
> hosts: engineering_services_pulp
> # gather_facts: false
> vars:
> # required by pulp_installer:
> https://pulp-installer.readthedocs.io/en/latest/#system-requirements
> # TODO: this is now set in ansible.cfg bc it doesn't work when set
> here or in inventory
> # allow_world_readable_tmpfiles: True
> pulp_settings:
> secret_key: !vault |
> $ANSIBLE_VAULT;1.1;AES256
>
> 38383631633236306565616334663761363134613835323839653962323930616639656333653865
>
> 3264363735643430626361383132653632316139396364370a613566396133393430663962666261
>
> 35356165663639613535383563366638663635326662343133353339343262646265316630616162
>
> 6337346131303833610a663232633339306231613738653233646466383638333934393765373034
>
> 63346437343834653964366666333061303634313864333031323735326134626432626535613436
>
> 62643731343836626436383438643862396166636263646330646332633637363765623866343733
> 616635326537346163646564653134386666
> content_origin: "http://{{ ansible_fqdn }}:8080"
> pulp_default_admin_password: !vault |
> $ANSIBLE_VAULT;1.1;AES256
>
> 35636365316538376363643965323035306461643239306433353665623438633535633763613662
>
> 6266346236393736616532636230393136303966383339310a306563323838326431386432626465
>
> 30316164383265303932643865323033623938656136306665356665336262613233653866386165
>
> 3164396261326563640a613464353364656130396333613531383864323434316533663932303766
> 3938
> pulp_content_host: "{{ ansible_fqdn }}"
> # pulp_content_port: 24816
> pulp_content_port: 8080
> pulp_api_host: "{{ ansible_fqdn }}"
> # pulp_content_port: 24817
> pulp_content_bind: "{{ pulp_content_host }}:{{ pulp_content_port }}"
> pulp_install_plugins:
> # galaxy-ng: {}
> pulp-ansible: {}
> # pulp-certguard: {}
> pulp-container: {}
> # pulp-cookbook: {}
> pulp-deb: {}
> pulp-file: {}
> # pulp-gem: {}
> # pulp-maven: {}
> # pulp-npm: {}
> pulp-python: {}
> # pulp-rpm: {}
> pre_tasks:
> # The version string below is the highest of all those in roles'
> metadata:
> # "min_ansible_version". It needs to be kept manually up-to-date.
> - name: Verify Ansible meets min required version
> assert:
> that: "ansible_version.full is version_compare('2.8', '>=')"
> msg: >
> "You must update Ansible to at least 2.8 to use this version of
> Pulp 3 Installer."
> roles:
> - pulp_database
> - pulp_workers
> - pulp_resource_manager
> - pulp_webserver
> - pulp_content
> environment:
> DJANGO_SETTINGS_MODULE: pulpcore.app.settings
>
> Thanks for your help.
>
> Tim
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20200707/fcd6096c/attachment.htm>
More information about the Pulp-list
mailing list