[Pulp-list] 502 Bad Gateway error connecting to new pulp instance installed with pulp_installer
Tim Black
timblaktu at gmail.com
Sat Jul 11 03:12:00 UTC 2020
Thanks Matthias. I get 502 at http://pulp.my.domain/pulp/api/v3/status/ as
well. Below is my nginx.conf, pulled from my freshly provisioned pulp
server. My skills are a little weak on the webserver side of things so I'm
open to suggestions for any simplifications I can make to my config to get
this working. I'm not trying to do anything fancy here.
/etc/nginx/nginx.conf:
# TODO: Support IPv6.
# TODO: Configure SSL certificates.
# TODO: Maybe serve multiple `location`s, not just one.
# Gunicorn docs suggest this value.
worker_processes 1;
events {
worker_connections 1024; # increase if you have lots of clients
accept_mutex off; # set to 'on' if nginx worker_processes > 1
}
http {
include mime.types;
# fallback in case we can't determine a type
default_type application/octet-stream;
sendfile on;
# If left at the default of 1024, nginx emits a warning about being
unable
# to build optimal hash types.
types_hash_max_size 4096;
upstream pulp-content {
server 127.0.0.1:24816;
}
upstream pulp-api {
server 127.0.0.1:24817;
}
server {
# Gunicorn docs suggest the use of the "deferred" directive on
Linux.
listen 80 default_server deferred;
server_name $hostname;
# The default client_max_body_size is 1m. Clients uploading
# files larger than this will need to chunk said files.
# Gunicorn docs suggest this value.
keepalive_timeout 5;
location /pulp/content/ {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $http_host;
# we don't want nginx trying to do something clever with
# redirects, we set the Host: header above already.
proxy_redirect off;
proxy_pass http://pulp-content;
}
location /pulp/api/v3/ {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $http_host;
# we don't want nginx trying to do something clever with
# redirects, we set the Host: header above already.
proxy_redirect off;
proxy_pass http://pulp-api;
}
location /auth/login/ {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $http_host;
# we don't want nginx trying to do something clever with
# redirects, we set the Host: header above already.
proxy_redirect off;
proxy_pass http://pulp-api;
}
include pulp/*.conf;
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $http_host;
# we don't want nginx trying to do something clever with
# redirects, we set the Host: header above already.
proxy_redirect off;
proxy_pass http://pulp-api;
# static files are served through whitenoise -
http://whitenoise.evans.io/en/stable/
}
}
}
On Tue, Jul 7, 2020 at 11:56 PM Matthias Dellweg <mdellweg at redhat.com>
wrote:
> The only thing that sticks out to me is `content_origin: "http://{{
> ansible_fqdn }}:8080"`. This is the address seen from the outside, and
> since both content and api are subject to the same reverse proxy and
> so should be available on port 80 (and 443 soon). But that is for sure
> not the problem you have with the API.
> Can you, however, try `http
> http://pulp.my.domain/pulp/api/v3/status/`
> <http://pulp.my.domain/pulp/api/v3/status/>? And if it still didn't
> produce a result, provide the content of /etc/nginx/nginx.conf ?
>
> On Tue, Jul 7, 2020 at 11:18 PM Tim Black <timblaktu at gmail.com> wrote:
> >
> > After perusing all of the roles' READMEs more thoroughly, I have updated
> my playbook (pasted below) with what I believe are the correct current set
> of available role variables in 3.4.1, with links to the docs for each.
> (would be nice if the example playbook was this informative.) One thing
> that came up with this exercise is that the example-use playbook is not
> including the main pulp role, however on tag 3.4.1 the pulp role appears to
> be a required dependency. Does the pulp role get included by the others,
> implicitly?
> >
> > Anyway, after a successful run of the modified playbook, I'm now seeing
> all services enabled:
> >
> > pulpadmin at pulp:~$ sudo systemctl list-unit-files | grep -E
> "(pulp|nginx)"
> > nginx.service enabled
> > pulpcore-api.service enabled
> > pulpcore-content.service enabled
> > pulpcore-resource-manager.service enabled
> > pulpcore-worker at .service indirect
> > dev-mapper-pulp\x2d\x2dvg\x2dswap_1.swap generated
> >
> > However, I'm still getting 502 trying to connect to pulp content
> webserver at my specified content_origin.
> >
> > My /var/log/nginx/error.log still shows the same type errors showing
> nginx can't connect with an upstream application server:
> >
> > 2020/07/07 13:59:41 [error] 12936#12936: *44 connect() failed (111:
> Connection refused) while connecting to upstream, client: 10.212.134.131,
> server: pulp, request: "GET /favicon.ico HTTP/1.1", upstream: "
> http://127.0.0.1:24817/favicon.ico", host: "pulp.my.domain", referrer: "
> http://pulp.my.domain/"
> >
> > Here's my updated pulp.yml:
> >
> > ---
> > # Playbook to provision and manage Pulp Instances for Artifact Management
> >
> > # Requires:
> > # (
> https://pulp-installer.readthedocs.io/en/latest/#system-requirements)
> > # 1. Debian Buster Machine Provisioned using Preseeded Installer
> > # a. Really just need Debian install with:
> > # i. sudo, openssh-server, python3
> > # (after installing with only ssh-server and system utility
> packages selected, only need to:
> > # su
> > # vi /etc/apt/sources.list # remove CD Rom line, add buster
> main repo if no mirror selected during install
> > # apt-get install sudo)
> > # ii. update-alternatives --set editor `update-alternatives --list
> editor | grep vim`
> > # iii. pulpadmin user with passwordless sudoer priviledges
> > # (echo "pulpadmin ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers)
> > # iv. ansible controller user has installed its ssh key in remote
> host's known_hosts
> > # (without this you'd just need to --ask-pass and supply ssh
> passwd at stdin)
> > # TODO: capture above in a VM Snapshot in vSphere/ESXi for fast
> reproduction.
> > # 2. Ansible Roles Installed via Galaxy using `$ ansible-galaxy install
> -r requirements-pulp.yml`
> > # 3. Ansible Collection Installed via Galaxy using `$ ansible-galaxy
> install -r requirements-pulp.yml`
> >
> > # Run like this:
> > # ansible-playbook pulp.yml --user pulpadmin --ask-pass
> --ask-vault-pass
> > #
> > # Note ansible knows what machines to run the playbook on by the `hosts`
> element within the playbook,
> > # which should have names existing in hosts file(s) in inventory/.
> >
> > # This playbook builds upon the Engineering Services playbook template
> > # Check imported playbook content before adding it here.
> > - import_playbook: engineering-services-tmplt.yml
> >
> > - name: "Install packages we want on every Pulp instance"
> > hosts: engineering_services_pulp
> > gather_facts: false
> > vars:
> > apt_packages:
> > - curl
> > roles:
> > - apt
> >
> > - name: Configure admin group
> > become: true
> > hosts: engineering_services_pulp
> > gather_facts: false
> > tasks:
> > - name: Create admin group
> > group:
> > name: admin
> >
> > - name: Configure admin user
> > become: true
> > hosts: engineering_services_pulp
> > gather_facts: false
> > vars:
> > # TODO: define these as inventory variable (standard for all
> machines?) so it can move out of playbook task blocks
> > tasks:
> > - debug: var=ansible_fqdn
> > - name: Configure admin user account
> > user:
> > name: admin
> > groups:
> > - admin
> >
> > - name: Install Pulp
> > hosts: engineering_services_pulp
> > # gather_facts: false
> > vars:
> > # Main Pulp Role Variables
> > #
> https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp#role-variables
> > pulp_settings:
> > secret_key: !vault |
> > $ANSIBLE_VAULT;1.1;AES256
> >
> 38383631633236306565616334663761363134613835323839653962323930616639656333653865
> >
> 3264363735643430626361383132653632316139396364370a613566396133393430663962666261
> >
> 35356165663639613535383563366638663635326662343133353339343262646265316630616162
> >
> 6337346131303833610a663232633339306231613738653233646466383638333934393765373034
> >
> 63346437343834653964366666333061303634313864333031323735326134626432626535613436
> >
> 62643731343836626436383438643862396166636263646330646332633637363765623866343733
> > 616635326537346163646564653134386666
> > content_origin: "http://{{ ansible_fqdn }}:8080"
> > pulp_install_plugins:
> > pulp-ansible: {}
> > pulp-container: {}
> > pulp-deb: {}
> > pulp-file: {}
> > pulp-python: {}
> > pulp_default_admin_password: !vault |
> > $ANSIBLE_VAULT;1.1;AES256
> >
> 35636365316538376363643965323035306461643239306433353665623438633535633763613662
> >
> 6266346236393736616532636230393136303966383339310a306563323838326431386432626465
> >
> 30316164383265303932643865323033623938656136306665356665336262613233653866386165
> >
> 3164396261326563640a613464353364656130396333613531383864323434316533663932303766
> > 3938
> > pulp_api_bind: "{{ ansible_fqdn }}"
> > pulp_api_workers: 4 # defaults to 1
> >
> > # Pulp Content Role Variables
> > #
> https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp_content#pulp_content
> > # pulp_content_bind: # Defaults to 127.0.0.1:24816
> >
> > # Pulp Database Role Variables
> > #
> https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp_database
> > # None
> >
> > # Pulp Resource Manager Role Variables
> > #
> https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp_resource_manager
> > # pulp_resouce_manager_state: # defaults to started
> > # pulp_resouce_manager_enabled: # defaults to true
> >
> > # Pulp Webserver Role Variables
> > #
> https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp_webserver
> > # pulp_webserver_server: # defauls to nginx
> > # pulp_content_port: # defaults to 24816
> > # pulp_content_host: # defaults to localhost
> > # pulp_api_port: # defaults to 24817
> > # pulp_api_host: # defaults to localhost
> > # pulp_configure_firewall: # defaults to auto, which is same as
> firewalld. Change to none to disable.
> >
> > # Pulp Workers Role Variables
> > #
> https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp_workers
> > # TODO: how is this different from pulp_api_workers in the main Pulp
> Role??
> > # pulp_workers: 4 # defaults to 2
> >
> > pre_tasks:
> > # The version string below is the highest of all those in roles'
> metadata:
> > # "min_ansible_version". It needs to be kept manually up-to-date.
> > - name: Verify Ansible meets min required version
> > assert:
> > that: "ansible_version.full is version_compare('2.8', '>=')"
> > msg: >
> > "You must update Ansible to at least 2.8 to use this version
> of Pulp 3 Installer."
> > roles:
> > # Is pulp role implicitly included by the others?
> > - pulp_database
> > - pulp_workers
> > - pulp_resource_manager
> > - pulp_webserver
> > - pulp_content
> > environment:
> > DJANGO_SETTINGS_MODULE: pulpcore.app.settings
> >
> > On Tue, Jul 7, 2020 at 12:24 PM Tim Black <timblaktu at gmail.com> wrote:
> >>
> >> I just installed my first pulp instance on a fresh Debian Buster VM,
> using latest Ansible pulp_installer release (3.4.1), with my pulp.yml
> playbook (pasted below) modeled after the official example-use playbook.
> The playbook runs to completion, with zero failed tasks, yet I am not able
> to connect to the pulp content webserver using the protocol/address/port I
> specified in the content_origin variable. I have verified that nginx
> service is running, but I still get 502: Bad Gateway error.
> >>
> >> Can someone help me troubleshoot this, or direct me to troubleshooting
> documentation that would assist? I found this excellent explanation which
> seems relevant since pulp uses the same nginx/gunicorn tech cocktail. It
> states:
> >>
> >>> NGINX will return a 502 Bad Gateway error if it can’t successfully
> proxy a request to Gunicorn or if Gunicorn fails to respond.
> >>
> >>
> >> I learned to look in /var/log/nginx/error.log for the reason for the
> issue. There I found several errors similar to this:
> >>
> >> [error] 4348#4348: *28 connect() failed (111: Connection refused) while
> connecting to upstream, client: 10.212.134.131, server: pulp, request: "GET
> / HTTP/1.1", upstream: "http://127.0.1.1:24817/", host: "pulp.my.domain"
> >>
> >> I also confirmed the following pulp service statuses:
> >>
> >> pulpadmin at pulp:~$ sudo systemctl list-unit-files | grep pulp
> >> pulpcore-api.service disabled
> >> pulpcore-content.service enabled
> >> pulpcore-resource-manager.service enabled
> >> pulpcore-worker at .service indirect
> >> dev-mapper-pulp\x2d\x2dvg\x2dswap_1.swap generated
> >>
> >> Hmm.. Shouldn't pulpcore-api be enabled? If so, I suppose this is the
> "upstream" service that nginx cannot connect to? From the error log, it
> looks like the address is localhost:24817, and I believe this is the
> default I chose. Anyone see any problem with what I'm doing here? I'm
> simply trying to set up "hello world" with pulp_installer targeting a
> dedicated remote server.
> >>
> >> I applaud the pulp dev team's modularizing of the code base, but I
> would love to see more documentation on the architecture here, clearly
> illustrating all these moving parts, with links to common problems like I'm
> having, with troubleshooting advice.
> >>
> >> Here's my pulp.yml ansible playbook:
> >>
> >> ---
> >> # Playbook to provision and manage Pulp Instances for Artifact
> Management
> >>
> >> # Requires:
> >> # (
> https://pulp-installer.readthedocs.io/en/latest/#system-requirements)
> >> # 1. Debian Buster Machine Provisioned using Preseeded Installer
> >> # a. Really just need Debian install with:
> >> # i. sudo, openssh-server, python3
> >> # (after installing with only ssh-server and system utility
> packages selected, only need to:
> >> # su
> >> # vi /etc/apt/sources.list # remove CD Rom line, add buster
> main repo if no mirror selected during install
> >> # apt-get install sudo)
> >> # ii. update-alternatives --set editor `update-alternatives
> --list editor | grep vim`
> >> # iii. pulpadmin user with passwordless sudoer priviledges
> >> # (echo "pulpadmin ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers)
> >> # iv. ansible controller user has installed its ssh key in remote
> host's known_hosts
> >> # (without this you'd just need to --ask-pass and supply ssh
> passwd at stdin)
> >> # TODO: capture above in a VM Snapshot in vSphere/ESXi for fast
> reproduction.
> >> # 2. Ansible Roles Installed via Galaxy using `$ ansible-galaxy install
> -r requirements-pulp.yml`
> >> # 3. Ansible Collection Installed via Galaxy using `$ ansible-galaxy
> install -r requirements-pulp.yml`
> >> #
> >> # Run like this:
> >> # ansible-playbook pulp.yml --user pulpadmin -l
> <controlled-pulp-hostname> --ask-pass --ask-vault-pass
> >>
> >> # This playbook builds upon the Engineering Services playbook template
> >> # Check imported playbook content before adding it here.
> >> - import_playbook: engineering-services-tmplt.yml
> >>
> >> - name: "Install packages we want on every Pulp instance"
> >> hosts: engineering_services_pulp
> >> gather_facts: false
> >> vars:
> >> apt_packages:
> >> - curl
> >> roles:
> >> - apt
> >>
> >> - name: Configure admin group
> >> become: true
> >> hosts: engineering_services_pulp
> >> gather_facts: false
> >> tasks:
> >> - name: Create admin group
> >> group:
> >> name: admin
> >>
> >> - name: Configure admin user
> >> become: true
> >> hosts: engineering_services_pulp
> >> gather_facts: false
> >> vars:
> >> # TODO: define these as inventory variable (standard for all
> machines?) so it can move out of playbook task blocks
> >> tasks:
> >> - debug: var=ansible_fqdn
> >> - name: Configure admin user account
> >> user:
> >> name: admin
> >> groups:
> >> - admin
> >>
> >> - name: Install Pulp
> >> hosts: engineering_services_pulp
> >> # gather_facts: false
> >> vars:
> >> # required by pulp_installer:
> https://pulp-installer.readthedocs.io/en/latest/#system-requirements
> >> # TODO: this is now set in ansible.cfg bc it doesn't work when set
> here or in inventory
> >> # allow_world_readable_tmpfiles: True
> >> pulp_settings:
> >> secret_key: !vault |
> >> $ANSIBLE_VAULT;1.1;AES256
> >>
> 38383631633236306565616334663761363134613835323839653962323930616639656333653865
> >>
> 3264363735643430626361383132653632316139396364370a613566396133393430663962666261
> >>
> 35356165663639613535383563366638663635326662343133353339343262646265316630616162
> >>
> 6337346131303833610a663232633339306231613738653233646466383638333934393765373034
> >>
> 63346437343834653964366666333061303634313864333031323735326134626432626535613436
> >>
> 62643731343836626436383438643862396166636263646330646332633637363765623866343733
> >> 616635326537346163646564653134386666
> >> content_origin: "http://{{ ansible_fqdn }}:8080"
> >> pulp_default_admin_password: !vault |
> >> $ANSIBLE_VAULT;1.1;AES256
> >>
> 35636365316538376363643965323035306461643239306433353665623438633535633763613662
> >>
> 6266346236393736616532636230393136303966383339310a306563323838326431386432626465
> >>
> 30316164383265303932643865323033623938656136306665356665336262613233653866386165
> >>
> 3164396261326563640a613464353364656130396333613531383864323434316533663932303766
> >> 3938
> >> pulp_content_host: "{{ ansible_fqdn }}"
> >> # pulp_content_port: 24816
> >> pulp_content_port: 8080
> >> pulp_api_host: "{{ ansible_fqdn }}"
> >> # pulp_content_port: 24817
> >> pulp_content_bind: "{{ pulp_content_host }}:{{ pulp_content_port }}"
> >> pulp_install_plugins:
> >> # galaxy-ng: {}
> >> pulp-ansible: {}
> >> # pulp-certguard: {}
> >> pulp-container: {}
> >> # pulp-cookbook: {}
> >> pulp-deb: {}
> >> pulp-file: {}
> >> # pulp-gem: {}
> >> # pulp-maven: {}
> >> # pulp-npm: {}
> >> pulp-python: {}
> >> # pulp-rpm: {}
> >> pre_tasks:
> >> # The version string below is the highest of all those in roles'
> metadata:
> >> # "min_ansible_version". It needs to be kept manually up-to-date.
> >> - name: Verify Ansible meets min required version
> >> assert:
> >> that: "ansible_version.full is version_compare('2.8', '>=')"
> >> msg: >
> >> "You must update Ansible to at least 2.8 to use this version
> of Pulp 3 Installer."
> >> roles:
> >> - pulp_database
> >> - pulp_workers
> >> - pulp_resource_manager
> >> - pulp_webserver
> >> - pulp_content
> >> environment:
> >> DJANGO_SETTINGS_MODULE: pulpcore.app.settings
> >>
> >> Thanks for your help.
> >>
> >> Tim
> >
> > _______________________________________________
> > Pulp-list mailing list
> > Pulp-list at redhat.com
> > https://www.redhat.com/mailman/listinfo/pulp-list
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20200710/a7c894cf/attachment.htm>
More information about the Pulp-list
mailing list