[Pulp-list] 502 Bad Gateway error connecting to new pulp instance installed with pulp_installer

Tue Jul 7 19:24:40 UTC 2020

I just installed my first pulp instance on a fresh Debian Buster VM, using
latest Ansible pulp_installer release (3.4.1), with my pulp.yml playbook
(pasted below) modeled after the official example-use playbook
<https://github.com/pulp/pulp_installer/tree/3.4.1/playbooks/example-use>.
The playbook runs to completion, with zero failed tasks, yet I am not able
to connect to the pulp content webserver using the protocol/address/port I
specified in the content_origin variable. I have verified that nginx
service is running, but I still get 502: Bad Gateway error.

Can someone help me troubleshoot this, or direct me to troubleshooting
documentation that would assist? I found this excellent explanation
<https://www.datadoghq.com/blog/nginx-502-bad-gateway-errors-gunicorn/>which
seems relevant since pulp uses the same nginx/gunicorn tech cocktail. It
states:

NGINX will return a 502 Bad Gateway error if it can’t successfully proxy a
> request to Gunicorn or if Gunicorn fails to respond.

I learned to look in /var/log/nginx/error.log for the reason for the issue.
There I found several errors similar to this:

[error] 4348#4348: *28 connect() failed (111: Connection refused) while
connecting to upstream, client: 10.212.134.131, server: pulp, request: "GET
/ HTTP/1.1", upstream: "http://127.0.1.1:24817/", host: "pulp.my.domain"

I also confirmed the following pulp service statuses:

pulpadmin at pulp:~$ sudo systemctl list-unit-files | grep pulp

*pulpcore-api.service                      disabled*pulpcore-content.service
                 enabled
pulpcore-resource-manager.service         enabled
pulpcore-worker at .service                  indirect
dev-mapper-pulp\x2d\x2dvg\x2dswap_1.swap  generated

Hmm.. *Shouldn't pulpcore-api be enabled*? If so, I suppose this is the
"upstream" service that nginx cannot connect to? From the error log, it
looks like the address is localhost:24817, and I believe this is the
default I chose. Anyone see any problem with what I'm doing here? I'm
simply trying to set up "hello world" with pulp_installer targeting a
dedicated remote server.

I applaud the pulp dev team's modularizing of the code base, but I would
love to see more documentation on the architecture here, clearly
illustrating all these moving parts, with links to common problems like I'm
having, with troubleshooting advice.

Here's my pulp.yml ansible playbook:

---
# Playbook to provision and manage Pulp Instances for Artifact Management

# Requires:
#   (https://pulp-installer.readthedocs.io/en/latest/#system-requirements)
# 1. Debian Buster Machine Provisioned using Preseeded Installer
#    a. Really just need Debian install with:
#       i. sudo, openssh-server, python3
#          (after installing with only ssh-server and system utility
packages selected, only need to:
#           su
#           vi /etc/apt/sources.list  # remove CD Rom line, add buster main
repo if no mirror selected during install
#           apt-get install sudo)
#       ii. update-alternatives --set editor `update-alternatives --list
editor | grep vim`
#       iii. pulpadmin user with passwordless sudoer priviledges
#           (echo "pulpadmin ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers)
#       iv. ansible controller user has installed its ssh key in remote
host's known_hosts
#           (without this you'd just need to --ask-pass and supply ssh
passwd at stdin)
#       TODO: capture above in a VM Snapshot in vSphere/ESXi for fast
reproduction.
# 2. Ansible Roles Installed via Galaxy using `$ ansible-galaxy install -r
requirements-pulp.yml`
# 3. Ansible Collection Installed via Galaxy using `$ ansible-galaxy
install -r requirements-pulp.yml`
#
# Run like this:
#    ansible-playbook pulp.yml --user pulpadmin -l
<controlled-pulp-hostname> --ask-pass --ask-vault-pass

# This playbook builds upon the Engineering Services playbook template
# Check imported playbook content before adding it here.
- import_playbook: engineering-services-tmplt.yml

- name: "Install packages we want on every Pulp instance"
  hosts: engineering_services_pulp
  gather_facts: false
  vars:
    apt_packages:
      - curl
  roles:
    - apt

- name: Configure admin group
  become: true
  hosts: engineering_services_pulp
  gather_facts: false
  tasks:
    - name: Create admin group
      group:
        name: admin

- name: Configure admin user
  become: true
  hosts: engineering_services_pulp
  gather_facts: false
  vars:
    # TODO: define these as inventory variable (standard for all machines?)
so it can move out of playbook task blocks
  tasks:
    - debug: var=ansible_fqdn
    - name: Configure admin user account
      user:
        name: admin
        groups:
          - admin

- name: Install Pulp
  hosts: engineering_services_pulp
  # gather_facts: false
  vars:
    # required by pulp_installer:
https://pulp-installer.readthedocs.io/en/latest/#system-requirements
    # TODO: this is now set in ansible.cfg bc it doesn't work when set here
or in inventory
    # allow_world_readable_tmpfiles: True
    pulp_settings:
      secret_key: !vault |
          $ANSIBLE_VAULT;1.1;AES256

38383631633236306565616334663761363134613835323839653962323930616639656333653865

3264363735643430626361383132653632316139396364370a613566396133393430663962666261

35356165663639613535383563366638663635326662343133353339343262646265316630616162

6337346131303833610a663232633339306231613738653233646466383638333934393765373034

63346437343834653964366666333061303634313864333031323735326134626432626535613436

62643731343836626436383438643862396166636263646330646332633637363765623866343733
          616635326537346163646564653134386666
      content_origin: "http://{{ ansible_fqdn }}:8080"
    pulp_default_admin_password: !vault |
          $ANSIBLE_VAULT;1.1;AES256

35636365316538376363643965323035306461643239306433353665623438633535633763613662

6266346236393736616532636230393136303966383339310a306563323838326431386432626465

30316164383265303932643865323033623938656136306665356665336262613233653866386165

3164396261326563640a613464353364656130396333613531383864323434316533663932303766
          3938
    pulp_content_host: "{{ ansible_fqdn }}"
    # pulp_content_port: 24816
    pulp_content_port: 8080
    pulp_api_host: "{{ ansible_fqdn }}"
    # pulp_content_port: 24817
    pulp_content_bind: "{{ pulp_content_host }}:{{ pulp_content_port }}"
    pulp_install_plugins:
      # galaxy-ng: {}
      pulp-ansible: {}
      # pulp-certguard: {}
      pulp-container: {}
      # pulp-cookbook: {}
      pulp-deb: {}
      pulp-file: {}
      # pulp-gem: {}
      # pulp-maven: {}
      # pulp-npm: {}
      pulp-python: {}
      # pulp-rpm: {}
  pre_tasks:
    # The version string below is the highest of all those in roles'
metadata:
    # "min_ansible_version". It needs to be kept manually up-to-date.
    - name: Verify Ansible meets min required version
      assert:
        that: "ansible_version.full is version_compare('2.8', '>=')"
        msg: >
          "You must update Ansible to at least 2.8 to use this version of
Pulp 3 Installer."
  roles:
    - pulp_database
    - pulp_workers
    - pulp_resource_manager
    - pulp_webserver
    - pulp_content
  environment:
    DJANGO_SETTINGS_MODULE: pulpcore.app.settings

Thanks for your help.

Tim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20200707/be588ef4/attachment.htm>