[Pulp-list] 502 Bad Gateway error connecting to new pulp instance installed with pulp_installer
Dennis Kliban
dkliban at redhat.com
Sat Jul 11 20:48:43 UTC 2020
I would recommend re-running the installer on a fresh VM that is running
CentOS 7.7+. I've experienced this problem before when the installer had to
be run multiple times due to various failures. In my case, the database
migrations had not been run and the output of "systemctl status pulpcore*"
showed that Pulp services were failing to start due to database issues. I
suspected it was due to permissions problems with /etc/pulp/settings.py,
however, I never confirmed this by actually fixing the install. I've always
just reprovisioned on a new VM.
If you can reproduce this issue again on a new VM, I would recommend filing
an issue at https://pulp.plan.io/issues/new/. The installer is definitely
doing something wrong, but I am not sure how to reproduce the issue
consistently.
On Fri, Jul 10, 2020 at 11:12 PM Tim Black <timblaktu at gmail.com> wrote:
> Thanks Matthias. I get 502 at http://pulp.my.domain/pulp/api/v3/status/
> as well. Below is my nginx.conf, pulled from my freshly provisioned pulp
> server. My skills are a little weak on the webserver side of things so I'm
> open to suggestions for any simplifications I can make to my config to get
> this working. I'm not trying to do anything fancy here.
>
> /etc/nginx/nginx.conf:
>
> # TODO: Support IPv6.
> # TODO: Configure SSL certificates.
> # TODO: Maybe serve multiple `location`s, not just one.
>
> # Gunicorn docs suggest this value.
> worker_processes 1;
> events {
> worker_connections 1024; # increase if you have lots of clients
> accept_mutex off; # set to 'on' if nginx worker_processes > 1
> }
>
> http {
> include mime.types;
> # fallback in case we can't determine a type
> default_type application/octet-stream;
> sendfile on;
>
> # If left at the default of 1024, nginx emits a warning about being
> unable
> # to build optimal hash types.
> types_hash_max_size 4096;
>
> upstream pulp-content {
> server 127.0.0.1:24816;
> }
>
> upstream pulp-api {
> server 127.0.0.1:24817;
> }
>
> server {
> # Gunicorn docs suggest the use of the "deferred" directive on
> Linux.
> listen 80 default_server deferred;
> server_name $hostname;
>
> # The default client_max_body_size is 1m. Clients uploading
> # files larger than this will need to chunk said files.
>
> # Gunicorn docs suggest this value.
> keepalive_timeout 5;
>
> location /pulp/content/ {
> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
> proxy_set_header X-Forwarded-Proto $scheme;
> proxy_set_header Host $http_host;
> # we don't want nginx trying to do something clever with
> # redirects, we set the Host: header above already.
> proxy_redirect off;
> proxy_pass http://pulp-content;
> }
>
> location /pulp/api/v3/ {
> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
> proxy_set_header X-Forwarded-Proto $scheme;
> proxy_set_header Host $http_host;
> # we don't want nginx trying to do something clever with
> # redirects, we set the Host: header above already.
> proxy_redirect off;
> proxy_pass http://pulp-api;
> }
>
> location /auth/login/ {
> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
> proxy_set_header X-Forwarded-Proto $scheme;
> proxy_set_header Host $http_host;
> # we don't want nginx trying to do something clever with
> # redirects, we set the Host: header above already.
> proxy_redirect off;
> proxy_pass http://pulp-api;
> }
>
> include pulp/*.conf;
>
> location / {
> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
> proxy_set_header X-Forwarded-Proto $scheme;
> proxy_set_header Host $http_host;
> # we don't want nginx trying to do something clever with
> # redirects, we set the Host: header above already.
> proxy_redirect off;
> proxy_pass http://pulp-api;
> # static files are served through whitenoise -
> http://whitenoise.evans.io/en/stable/
> }
> }
> }
>
> On Tue, Jul 7, 2020 at 11:56 PM Matthias Dellweg <mdellweg at redhat.com>
> wrote:
>
>> The only thing that sticks out to me is `content_origin: "http://{{
>> ansible_fqdn }}:8080"`. This is the address seen from the outside, and
>> since both content and api are subject to the same reverse proxy and
>> so should be available on port 80 (and 443 soon). But that is for sure
>> not the problem you have with the API.
>> Can you, however, try `http
>> http://pulp.my.domain/pulp/api/v3/status/`
>> <http://pulp.my.domain/pulp/api/v3/status/>? And if it still didn't
>> produce a result, provide the content of /etc/nginx/nginx.conf ?
>>
>> On Tue, Jul 7, 2020 at 11:18 PM Tim Black <timblaktu at gmail.com> wrote:
>> >
>> > After perusing all of the roles' READMEs more thoroughly, I have
>> updated my playbook (pasted below) with what I believe are the correct
>> current set of available role variables in 3.4.1, with links to the docs
>> for each. (would be nice if the example playbook was this informative.) One
>> thing that came up with this exercise is that the example-use playbook is
>> not including the main pulp role, however on tag 3.4.1 the pulp role
>> appears to be a required dependency. Does the pulp role get included by the
>> others, implicitly?
>> >
>> > Anyway, after a successful run of the modified playbook, I'm now seeing
>> all services enabled:
>> >
>> > pulpadmin at pulp:~$ sudo systemctl list-unit-files | grep -E
>> "(pulp|nginx)"
>> > nginx.service enabled
>> > pulpcore-api.service enabled
>> > pulpcore-content.service enabled
>> > pulpcore-resource-manager.service enabled
>> > pulpcore-worker at .service indirect
>> > dev-mapper-pulp\x2d\x2dvg\x2dswap_1.swap generated
>> >
>> > However, I'm still getting 502 trying to connect to pulp content
>> webserver at my specified content_origin.
>> >
>> > My /var/log/nginx/error.log still shows the same type errors showing
>> nginx can't connect with an upstream application server:
>> >
>> > 2020/07/07 13:59:41 [error] 12936#12936: *44 connect() failed (111:
>> Connection refused) while connecting to upstream, client: 10.212.134.131,
>> server: pulp, request: "GET /favicon.ico HTTP/1.1", upstream: "
>> http://127.0.0.1:24817/favicon.ico", host: "pulp.my.domain", referrer: "
>> http://pulp.my.domain/"
>> >
>> > Here's my updated pulp.yml:
>> >
>> > ---
>> > # Playbook to provision and manage Pulp Instances for Artifact
>> Management
>> >
>> > # Requires:
>> > # (
>> https://pulp-installer.readthedocs.io/en/latest/#system-requirements)
>> > # 1. Debian Buster Machine Provisioned using Preseeded Installer
>> > # a. Really just need Debian install with:
>> > # i. sudo, openssh-server, python3
>> > # (after installing with only ssh-server and system utility
>> packages selected, only need to:
>> > # su
>> > # vi /etc/apt/sources.list # remove CD Rom line, add buster
>> main repo if no mirror selected during install
>> > # apt-get install sudo)
>> > # ii. update-alternatives --set editor `update-alternatives
>> --list editor | grep vim`
>> > # iii. pulpadmin user with passwordless sudoer priviledges
>> > # (echo "pulpadmin ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers)
>> > # iv. ansible controller user has installed its ssh key in remote
>> host's known_hosts
>> > # (without this you'd just need to --ask-pass and supply ssh
>> passwd at stdin)
>> > # TODO: capture above in a VM Snapshot in vSphere/ESXi for fast
>> reproduction.
>> > # 2. Ansible Roles Installed via Galaxy using `$ ansible-galaxy install
>> -r requirements-pulp.yml`
>> > # 3. Ansible Collection Installed via Galaxy using `$ ansible-galaxy
>> install -r requirements-pulp.yml`
>> >
>> > # Run like this:
>> > # ansible-playbook pulp.yml --user pulpadmin --ask-pass
>> --ask-vault-pass
>> > #
>> > # Note ansible knows what machines to run the playbook on by the
>> `hosts` element within the playbook,
>> > # which should have names existing in hosts file(s) in inventory/.
>> >
>> > # This playbook builds upon the Engineering Services playbook template
>> > # Check imported playbook content before adding it here.
>> > - import_playbook: engineering-services-tmplt.yml
>> >
>> > - name: "Install packages we want on every Pulp instance"
>> > hosts: engineering_services_pulp
>> > gather_facts: false
>> > vars:
>> > apt_packages:
>> > - curl
>> > roles:
>> > - apt
>> >
>> > - name: Configure admin group
>> > become: true
>> > hosts: engineering_services_pulp
>> > gather_facts: false
>> > tasks:
>> > - name: Create admin group
>> > group:
>> > name: admin
>> >
>> > - name: Configure admin user
>> > become: true
>> > hosts: engineering_services_pulp
>> > gather_facts: false
>> > vars:
>> > # TODO: define these as inventory variable (standard for all
>> machines?) so it can move out of playbook task blocks
>> > tasks:
>> > - debug: var=ansible_fqdn
>> > - name: Configure admin user account
>> > user:
>> > name: admin
>> > groups:
>> > - admin
>> >
>> > - name: Install Pulp
>> > hosts: engineering_services_pulp
>> > # gather_facts: false
>> > vars:
>> > # Main Pulp Role Variables
>> > #
>> https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp#role-variables
>> > pulp_settings:
>> > secret_key: !vault |
>> > $ANSIBLE_VAULT;1.1;AES256
>> >
>> 38383631633236306565616334663761363134613835323839653962323930616639656333653865
>> >
>> 3264363735643430626361383132653632316139396364370a613566396133393430663962666261
>> >
>> 35356165663639613535383563366638663635326662343133353339343262646265316630616162
>> >
>> 6337346131303833610a663232633339306231613738653233646466383638333934393765373034
>> >
>> 63346437343834653964366666333061303634313864333031323735326134626432626535613436
>> >
>> 62643731343836626436383438643862396166636263646330646332633637363765623866343733
>> > 616635326537346163646564653134386666
>> > content_origin: "http://{{ ansible_fqdn }}:8080"
>> > pulp_install_plugins:
>> > pulp-ansible: {}
>> > pulp-container: {}
>> > pulp-deb: {}
>> > pulp-file: {}
>> > pulp-python: {}
>> > pulp_default_admin_password: !vault |
>> > $ANSIBLE_VAULT;1.1;AES256
>> >
>> 35636365316538376363643965323035306461643239306433353665623438633535633763613662
>> >
>> 6266346236393736616532636230393136303966383339310a306563323838326431386432626465
>> >
>> 30316164383265303932643865323033623938656136306665356665336262613233653866386165
>> >
>> 3164396261326563640a613464353364656130396333613531383864323434316533663932303766
>> > 3938
>> > pulp_api_bind: "{{ ansible_fqdn }}"
>> > pulp_api_workers: 4 # defaults to 1
>> >
>> > # Pulp Content Role Variables
>> > #
>> https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp_content#pulp_content
>> > # pulp_content_bind: # Defaults to 127.0.0.1:24816
>> >
>> > # Pulp Database Role Variables
>> > #
>> https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp_database
>> > # None
>> >
>> > # Pulp Resource Manager Role Variables
>> > #
>> https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp_resource_manager
>> > # pulp_resouce_manager_state: # defaults to started
>> > # pulp_resouce_manager_enabled: # defaults to true
>> >
>> > # Pulp Webserver Role Variables
>> > #
>> https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp_webserver
>> > # pulp_webserver_server: # defauls to nginx
>> > # pulp_content_port: # defaults to 24816
>> > # pulp_content_host: # defaults to localhost
>> > # pulp_api_port: # defaults to 24817
>> > # pulp_api_host: # defaults to localhost
>> > # pulp_configure_firewall: # defaults to auto, which is same as
>> firewalld. Change to none to disable.
>> >
>> > # Pulp Workers Role Variables
>> > #
>> https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp_workers
>> > # TODO: how is this different from pulp_api_workers in the main
>> Pulp Role??
>> > # pulp_workers: 4 # defaults to 2
>> >
>> > pre_tasks:
>> > # The version string below is the highest of all those in roles'
>> metadata:
>> > # "min_ansible_version". It needs to be kept manually up-to-date.
>> > - name: Verify Ansible meets min required version
>> > assert:
>> > that: "ansible_version.full is version_compare('2.8', '>=')"
>> > msg: >
>> > "You must update Ansible to at least 2.8 to use this version
>> of Pulp 3 Installer."
>> > roles:
>> > # Is pulp role implicitly included by the others?
>> > - pulp_database
>> > - pulp_workers
>> > - pulp_resource_manager
>> > - pulp_webserver
>> > - pulp_content
>> > environment:
>> > DJANGO_SETTINGS_MODULE: pulpcore.app.settings
>> >
>> > On Tue, Jul 7, 2020 at 12:24 PM Tim Black <timblaktu at gmail.com> wrote:
>> >>
>> >> I just installed my first pulp instance on a fresh Debian Buster VM,
>> using latest Ansible pulp_installer release (3.4.1), with my pulp.yml
>> playbook (pasted below) modeled after the official example-use playbook.
>> The playbook runs to completion, with zero failed tasks, yet I am not able
>> to connect to the pulp content webserver using the protocol/address/port I
>> specified in the content_origin variable. I have verified that nginx
>> service is running, but I still get 502: Bad Gateway error.
>> >>
>> >> Can someone help me troubleshoot this, or direct me to troubleshooting
>> documentation that would assist? I found this excellent explanation which
>> seems relevant since pulp uses the same nginx/gunicorn tech cocktail. It
>> states:
>> >>
>> >>> NGINX will return a 502 Bad Gateway error if it can’t successfully
>> proxy a request to Gunicorn or if Gunicorn fails to respond.
>> >>
>> >>
>> >> I learned to look in /var/log/nginx/error.log for the reason for the
>> issue. There I found several errors similar to this:
>> >>
>> >> [error] 4348#4348: *28 connect() failed (111: Connection refused)
>> while connecting to upstream, client: 10.212.134.131, server: pulp,
>> request: "GET / HTTP/1.1", upstream: "http://127.0.1.1:24817/", host:
>> "pulp.my.domain"
>> >>
>> >> I also confirmed the following pulp service statuses:
>> >>
>> >> pulpadmin at pulp:~$ sudo systemctl list-unit-files | grep pulp
>> >> pulpcore-api.service disabled
>> >> pulpcore-content.service enabled
>> >> pulpcore-resource-manager.service enabled
>> >> pulpcore-worker at .service indirect
>> >> dev-mapper-pulp\x2d\x2dvg\x2dswap_1.swap generated
>> >>
>> >> Hmm.. Shouldn't pulpcore-api be enabled? If so, I suppose this is the
>> "upstream" service that nginx cannot connect to? From the error log, it
>> looks like the address is localhost:24817, and I believe this is the
>> default I chose. Anyone see any problem with what I'm doing here? I'm
>> simply trying to set up "hello world" with pulp_installer targeting a
>> dedicated remote server.
>> >>
>> >> I applaud the pulp dev team's modularizing of the code base, but I
>> would love to see more documentation on the architecture here, clearly
>> illustrating all these moving parts, with links to common problems like I'm
>> having, with troubleshooting advice.
>> >>
>> >> Here's my pulp.yml ansible playbook:
>> >>
>> >> ---
>> >> # Playbook to provision and manage Pulp Instances for Artifact
>> Management
>> >>
>> >> # Requires:
>> >> # (
>> https://pulp-installer.readthedocs.io/en/latest/#system-requirements)
>> >> # 1. Debian Buster Machine Provisioned using Preseeded Installer
>> >> # a. Really just need Debian install with:
>> >> # i. sudo, openssh-server, python3
>> >> # (after installing with only ssh-server and system utility
>> packages selected, only need to:
>> >> # su
>> >> # vi /etc/apt/sources.list # remove CD Rom line, add buster
>> main repo if no mirror selected during install
>> >> # apt-get install sudo)
>> >> # ii. update-alternatives --set editor `update-alternatives
>> --list editor | grep vim`
>> >> # iii. pulpadmin user with passwordless sudoer priviledges
>> >> # (echo "pulpadmin ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers)
>> >> # iv. ansible controller user has installed its ssh key in
>> remote host's known_hosts
>> >> # (without this you'd just need to --ask-pass and supply ssh
>> passwd at stdin)
>> >> # TODO: capture above in a VM Snapshot in vSphere/ESXi for fast
>> reproduction.
>> >> # 2. Ansible Roles Installed via Galaxy using `$ ansible-galaxy
>> install -r requirements-pulp.yml`
>> >> # 3. Ansible Collection Installed via Galaxy using `$ ansible-galaxy
>> install -r requirements-pulp.yml`
>> >> #
>> >> # Run like this:
>> >> # ansible-playbook pulp.yml --user pulpadmin -l
>> <controlled-pulp-hostname> --ask-pass --ask-vault-pass
>> >>
>> >> # This playbook builds upon the Engineering Services playbook template
>> >> # Check imported playbook content before adding it here.
>> >> - import_playbook: engineering-services-tmplt.yml
>> >>
>> >> - name: "Install packages we want on every Pulp instance"
>> >> hosts: engineering_services_pulp
>> >> gather_facts: false
>> >> vars:
>> >> apt_packages:
>> >> - curl
>> >> roles:
>> >> - apt
>> >>
>> >> - name: Configure admin group
>> >> become: true
>> >> hosts: engineering_services_pulp
>> >> gather_facts: false
>> >> tasks:
>> >> - name: Create admin group
>> >> group:
>> >> name: admin
>> >>
>> >> - name: Configure admin user
>> >> become: true
>> >> hosts: engineering_services_pulp
>> >> gather_facts: false
>> >> vars:
>> >> # TODO: define these as inventory variable (standard for all
>> machines?) so it can move out of playbook task blocks
>> >> tasks:
>> >> - debug: var=ansible_fqdn
>> >> - name: Configure admin user account
>> >> user:
>> >> name: admin
>> >> groups:
>> >> - admin
>> >>
>> >> - name: Install Pulp
>> >> hosts: engineering_services_pulp
>> >> # gather_facts: false
>> >> vars:
>> >> # required by pulp_installer:
>> https://pulp-installer.readthedocs.io/en/latest/#system-requirements
>> >> # TODO: this is now set in ansible.cfg bc it doesn't work when set
>> here or in inventory
>> >> # allow_world_readable_tmpfiles: True
>> >> pulp_settings:
>> >> secret_key: !vault |
>> >> $ANSIBLE_VAULT;1.1;AES256
>> >>
>> 38383631633236306565616334663761363134613835323839653962323930616639656333653865
>> >>
>> 3264363735643430626361383132653632316139396364370a613566396133393430663962666261
>> >>
>> 35356165663639613535383563366638663635326662343133353339343262646265316630616162
>> >>
>> 6337346131303833610a663232633339306231613738653233646466383638333934393765373034
>> >>
>> 63346437343834653964366666333061303634313864333031323735326134626432626535613436
>> >>
>> 62643731343836626436383438643862396166636263646330646332633637363765623866343733
>> >> 616635326537346163646564653134386666
>> >> content_origin: "http://{{ ansible_fqdn }}:8080"
>> >> pulp_default_admin_password: !vault |
>> >> $ANSIBLE_VAULT;1.1;AES256
>> >>
>> 35636365316538376363643965323035306461643239306433353665623438633535633763613662
>> >>
>> 6266346236393736616532636230393136303966383339310a306563323838326431386432626465
>> >>
>> 30316164383265303932643865323033623938656136306665356665336262613233653866386165
>> >>
>> 3164396261326563640a613464353364656130396333613531383864323434316533663932303766
>> >> 3938
>> >> pulp_content_host: "{{ ansible_fqdn }}"
>> >> # pulp_content_port: 24816
>> >> pulp_content_port: 8080
>> >> pulp_api_host: "{{ ansible_fqdn }}"
>> >> # pulp_content_port: 24817
>> >> pulp_content_bind: "{{ pulp_content_host }}:{{ pulp_content_port
>> }}"
>> >> pulp_install_plugins:
>> >> # galaxy-ng: {}
>> >> pulp-ansible: {}
>> >> # pulp-certguard: {}
>> >> pulp-container: {}
>> >> # pulp-cookbook: {}
>> >> pulp-deb: {}
>> >> pulp-file: {}
>> >> # pulp-gem: {}
>> >> # pulp-maven: {}
>> >> # pulp-npm: {}
>> >> pulp-python: {}
>> >> # pulp-rpm: {}
>> >> pre_tasks:
>> >> # The version string below is the highest of all those in roles'
>> metadata:
>> >> # "min_ansible_version". It needs to be kept manually up-to-date.
>> >> - name: Verify Ansible meets min required version
>> >> assert:
>> >> that: "ansible_version.full is version_compare('2.8', '>=')"
>> >> msg: >
>> >> "You must update Ansible to at least 2.8 to use this version
>> of Pulp 3 Installer."
>> >> roles:
>> >> - pulp_database
>> >> - pulp_workers
>> >> - pulp_resource_manager
>> >> - pulp_webserver
>> >> - pulp_content
>> >> environment:
>> >> DJANGO_SETTINGS_MODULE: pulpcore.app.settings
>> >>
>> >> Thanks for your help.
>> >>
>> >> Tim
>> >
>> > _______________________________________________
>> > Pulp-list mailing list
>> > Pulp-list at redhat.com
>> > https://www.redhat.com/mailman/listinfo/pulp-list
>>
>> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20200711/f49499ce/attachment.htm>
More information about the Pulp-list
mailing list