[Pulp-list] 502 Bad Gateway error connecting to new pulp instance installed with pulp_installer
Tim Black
timblaktu at gmail.com
Mon Jul 27 22:04:09 UTC 2020
Correction: using pulp_installer 3.5.0, I am still getting the same error
pulpcore/plugin compatibility error message I was getting with 3.4.1. (I
got past the secret_key error by specifying it in plain text in my playbook
instead of using vault (for now).) I am at a bit of a standstill, and am
going to shift gears and wait for some guidance or suggestions for how to
move forward with using pulp. Thanks again.
On Mon, Jul 27, 2020 at 2:53 PM Tim Black <timblaktu at gmail.com> wrote:
> Also.. I notice that on the 3.5.0 tag of pulp_installer, the example-use
> playbook
> <https://github.com/pulp/pulp_installer/blob/3.5.0/playbooks/example-use/playbook.yml>
> now has gone back to using the "pulp_all_services" role instead of listing
> each role separately, like it was doing before. Since I'm now using 3.5.0
> pulp_installer, should I be following this new pattern?
>
> I would like to also reiterate my request for a vagrant-virtualbox-based
> solution that "just works" that can be shared with me and other newbies.
> Seems like enabling this level of turnkey automation is the whole goal of
> using ansible to begin with. Does this exist somewhere? Thanks.
>
> On Mon, Jul 27, 2020 at 2:47 PM Tim Black <timblaktu at gmail.com> wrote:
>
>> Thanks Dennis. I finally got some time to work on this, and have started
>> over again, this time using the latest centos iso: 8.2.2004. I do not have
>> support for centos in my ansible bootstrapping playbooks, which typically
>> operate on a debian-based machine/snapshot with a fixed hostname and user.
>> So, for now I've done the following manual steps post centOS install,
>> before running my *slightly simplified pulp.yml ansible playbook:
>>
>> (* all my pulp.yml is doing now is configuring an admin/admin user/group,
>> then running the pulp_installer, with same options as I posted before.)
>>
>> 1. ssh-copy-id -i ~/.ssh/id_rsa.pub ansible at pulpcentos and confirm
>> that I can:
>> 1. ssh as ansible user without password
>> 2. sudo as ansible user with password
>> 2. sudo yum install python3
>>
>> Unfortunately, now I get an error in the compatibility check between
>> pulpcore and plugins:
>>
>> TASK [Run pip-compile to check pulpcore/plugin compatibility]
>> *****************************************************************************************************[20/7382]
>> Monday 27 July 2020 14:23:18 -0700 (0:00:00.287) 0:00:46.377
>> ***********
>> [WARNING]: conditional statements should not
>> include jinja2 templating delimiters such as {{ }} or {% %}. Found: {{
>> failed_condition | default("compatibility.rc != 0") }} fatal:
>> [pulpcentos]: FAILED! => changed=false
>>
>> cmd:
>>
>> - /usr/local/lib/pulp/bin/pip-compile
>>
>> delta:
>> '0:00:03.171889'
>>
>> end: '2020-07-27 14:23:21.863378'
>>
>> failed_when_result: true
>>
>> msg: non-zero return
>> code
>>
>> rc: 2
>>
>> start: '2020-07-27 14:23:18.691489'
>>
>> stderr: |-
>>
>> Could not
>> find a version that matches pulpcore<3.5,<3.6,==3.4.1,>=3.0,>=3.4,>=3.5
>> from
>> https://files.pythonhosted.org/packages/5c/40/8dab8ccfe73982ef3a5e48489af2d83974b0e7677ca52ec232fcb4b49dfa/pulpcore-3.4.1-py3-none-any.whl#sha256=e33ca32f867201e1a18b888d72ef07e85c2cd11273a8e422e33d6a2910a64fac
>> (from -r requirements.in (line 1)) Tried: 3.0.0, 3.0.0,
>> 3.0.1, 3.0.1, 3.1.0, 3.1.0, 3.1.1, 3.1.1, 3.2.0, 3.2.0, 3.2.1, 3.2.1,
>> 3.3.0, 3.3.0, 3.3.1, 3.3.1, 3.4.0, 3.4.0, 3.4.1, 3.4.1, 3.5.0, 3.5.0
>> Skipped pre-versions: 3.0.0b1, 3.0.0b1, 3.0.0b2, 3.0.0b2, 3.0.0b3,
>> 3.0.0b3, 3.0.0b4, 3.0.0b4, 3.0.0b5, 3.0.0b5, 3.0.0b6, 3.0.0b6, 3.0.0b7,
>> 3.0.0b7, 3.0.0b8, 3.0.0b8, 3.0.0b9, 3.0.0b9, 3.0.0b10, 3.0.0b10, 3.0.0b11,
>> 3.0.0b11, 3.0.0b12, 3.0.0b12, 3.0.0b13, 3.0.0b13, 3.0.0b14, 3.0.0b14,
>> 3.0.0b15, 3.0.0b15, 3.0.0b16, 3.0.0b16, 3.0.0b17, 3.0.0b18, 3.0.0b18,
>> 3.0.0b19, 3.0.0b19, 3.0.0b20, 3.0.0b20, 3.0.0b21, 3.0.0b21, 3.0.0b22,
>> 3.0.0b22, 3.0.0b23, 3.0.0b23, 3.0.0rc1, 3.0.0rc1, 3.0.0rc2, 3.0.0rc2,
>> 3.0.0rc3, 3.0.0rc3, 3.0.0rc4, 3.0.0rc4, 3.0.0rc5, 3.0.0rc5, 3.0.0rc6,
>> 3.0.0rc6, 3.0.0rc7, 3.0.0rc7, 3.0.0rc8, 3.0.0rc8, 3.0.0rc9, 3.0.0rc9
>> There are incompatible
>> versions in the resolved dependencies:
>>
>> pulpcore==3.4.1 from
>> https://files.pythonhosted.org/packages/5c/40/8dab8ccfe73982ef3a5e48489af2d83974b0e7677ca52ec232fcb4b49dfa/pulpcore-3.4.1-py3-none-any.whl#sha256=e33ca32f867201e1a18b888d72ef07e85c2cd11273a8e422e33d6a2910a64fac
>> (from -r requirements.in (line 1))
>> pulpcore<3.6,>=3.4 (from pulp-file==1.1.0->-r requirements.in
>> (line 5))
>> pulpcore<3.6,>=3.4 (from pulp-container==1.4.2->-r requirements.in
>> (line 3))
>> pulpcore<3.5,>=3.4 (from pulp-python==3.0.0b9->-r requirements.in
>> (line 6))
>> pulpcore<3.6,>=3.5 (from pulp-deb==2.5.0b1->-r requirements.in
>> (line 4))
>> pulpcore<3.6,>=3.0 (from pulp-ansible==0.2.0b15->-r requirements.in
>> (line 2))
>> stderr_lines: <omitted>
>> stdout: ''
>> stdout_lines: <omitted>
>>
>> PLAY RECAP
>> *****************************************************************************************************************************************************************pulpcentos
>> : ok=33 changed=14 unreachable=0 failed=1
>> skipped=16 rescued=0 ignored=0
>>
>> I believe this means that the version of pulp_installer role(s) I
>> have/had installed have become broken bc of compatibility changes made to
>> one or more versions they were referencing. This seems bad, nevertheless, I
>> went ahead and updated my pulp_installer to a newer tag (from 3.4.1 to
>> 3.5.0), and reran the pulp.yml playbook, with the following results:
>>
>> With 3.5.0 pulp_installer, running against fresh new centos 8 machine, it
>> got past the pulpcore/plugin version check, but failed here, in
>> pulp_common's check for required variables. This worked fine before (on my
>> debian-based machine) as you can see in my playbook I'm using an
>> ansible-vault encrypted string as the secret_key.
>>
>> TASK [pulp_common : Check if required variables are set]
>> *******************************************************************************************************************Monday
>> 27 July 2020 14:34:27 -0700 (0:00:00.024) 0:00:19.821 ***********
>>
>> ok: [pulpcentos] => (item=pulp_settings.content_origin) =>
>> changed=false
>> ansible_loop_var: item
>>
>> item:
>> pulp_settings.content_origin
>>
>> msg: All assertions passed
>>
>> fatal: [pulpcentos]: FAILED! =>
>> msg: 'The conditional check ''pulp_settings.secret_key | default('''',
>> true) | length > 0'' failed. The error was: Unexpected templating type
>> error occurred on ({% if pulp_settings.secret_key | default('''', true) |
>> length > 0 %} True {% else %} False {% endif %}): object of type
>> ''AnsibleVaultEncryptedUnicode'' has no len()'
>>
>> Not sure what's up, but at the very least so far it's not working any
>> better with CentOS. I'm all ears for suggestions.
>>
>> Does anyone have a turnkey, fully-automated solution they can share, like
>> a vagrant box that brings up a pulp instance from scratch? Seems like I'm
>> doing a lot more work here than should be required to bring this thing up.
>> Thanks.
>>
>> On Sat, Jul 11, 2020 at 1:49 PM Dennis Kliban <dkliban at redhat.com> wrote:
>>
>>> I would recommend re-running the installer on a fresh VM that is running
>>> CentOS 7.7+. I've experienced this problem before when the installer had to
>>> be run multiple times due to various failures. In my case, the database
>>> migrations had not been run and the output of "systemctl status pulpcore*"
>>> showed that Pulp services were failing to start due to database issues. I
>>> suspected it was due to permissions problems with /etc/pulp/settings.py,
>>> however, I never confirmed this by actually fixing the install. I've always
>>> just reprovisioned on a new VM.
>>>
>>> If you can reproduce this issue again on a new VM, I would recommend
>>> filing an issue at https://pulp.plan.io/issues/new/. The installer is
>>> definitely doing something wrong, but I am not sure how to reproduce the
>>> issue consistently.
>>>
>>>
>>> On Fri, Jul 10, 2020 at 11:12 PM Tim Black <timblaktu at gmail.com> wrote:
>>>
>>>> Thanks Matthias. I get 502 at http://pulp.my.domain/pulp/api/v3/status/
>>>> as well. Below is my nginx.conf, pulled from my freshly provisioned pulp
>>>> server. My skills are a little weak on the webserver side of things so I'm
>>>> open to suggestions for any simplifications I can make to my config to get
>>>> this working. I'm not trying to do anything fancy here.
>>>>
>>>> /etc/nginx/nginx.conf:
>>>>
>>>> # TODO: Support IPv6.
>>>> # TODO: Configure SSL certificates.
>>>> # TODO: Maybe serve multiple `location`s, not just one.
>>>>
>>>> # Gunicorn docs suggest this value.
>>>> worker_processes 1;
>>>> events {
>>>> worker_connections 1024; # increase if you have lots of clients
>>>> accept_mutex off; # set to 'on' if nginx worker_processes > 1
>>>> }
>>>>
>>>> http {
>>>> include mime.types;
>>>> # fallback in case we can't determine a type
>>>> default_type application/octet-stream;
>>>> sendfile on;
>>>>
>>>> # If left at the default of 1024, nginx emits a warning about being
>>>> unable
>>>> # to build optimal hash types.
>>>> types_hash_max_size 4096;
>>>>
>>>> upstream pulp-content {
>>>> server 127.0.0.1:24816;
>>>> }
>>>>
>>>> upstream pulp-api {
>>>> server 127.0.0.1:24817;
>>>> }
>>>>
>>>> server {
>>>> # Gunicorn docs suggest the use of the "deferred" directive on
>>>> Linux.
>>>> listen 80 default_server deferred;
>>>> server_name $hostname;
>>>>
>>>> # The default client_max_body_size is 1m. Clients uploading
>>>> # files larger than this will need to chunk said files.
>>>>
>>>> # Gunicorn docs suggest this value.
>>>> keepalive_timeout 5;
>>>>
>>>> location /pulp/content/ {
>>>> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
>>>> proxy_set_header X-Forwarded-Proto $scheme;
>>>> proxy_set_header Host $http_host;
>>>> # we don't want nginx trying to do something clever with
>>>> # redirects, we set the Host: header above already.
>>>> proxy_redirect off;
>>>> proxy_pass http://pulp-content;
>>>> }
>>>>
>>>> location /pulp/api/v3/ {
>>>> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
>>>> proxy_set_header X-Forwarded-Proto $scheme;
>>>> proxy_set_header Host $http_host;
>>>> # we don't want nginx trying to do something clever with
>>>> # redirects, we set the Host: header above already.
>>>> proxy_redirect off;
>>>> proxy_pass http://pulp-api;
>>>> }
>>>>
>>>> location /auth/login/ {
>>>> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
>>>> proxy_set_header X-Forwarded-Proto $scheme;
>>>> proxy_set_header Host $http_host;
>>>> # we don't want nginx trying to do something clever with
>>>> # redirects, we set the Host: header above already.
>>>> proxy_redirect off;
>>>> proxy_pass http://pulp-api;
>>>> }
>>>>
>>>> include pulp/*.conf;
>>>>
>>>> location / {
>>>> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
>>>> proxy_set_header X-Forwarded-Proto $scheme;
>>>> proxy_set_header Host $http_host;
>>>> # we don't want nginx trying to do something clever with
>>>> # redirects, we set the Host: header above already.
>>>> proxy_redirect off;
>>>> proxy_pass http://pulp-api;
>>>> # static files are served through whitenoise -
>>>> http://whitenoise.evans.io/en/stable/
>>>> }
>>>> }
>>>> }
>>>>
>>>> On Tue, Jul 7, 2020 at 11:56 PM Matthias Dellweg <mdellweg at redhat.com>
>>>> wrote:
>>>>
>>>>> The only thing that sticks out to me is `content_origin: "http://{{
>>>>> ansible_fqdn }}:8080"`. This is the address seen from the outside, and
>>>>> since both content and api are subject to the same reverse proxy and
>>>>> so should be available on port 80 (and 443 soon). But that is for sure
>>>>> not the problem you have with the API.
>>>>> Can you, however, try `http
>>>>> http://pulp.my.domain/pulp/api/v3/status/`
>>>>> <http://pulp.my.domain/pulp/api/v3/status/>? And if it still didn't
>>>>> produce a result, provide the content of /etc/nginx/nginx.conf ?
>>>>>
>>>>> On Tue, Jul 7, 2020 at 11:18 PM Tim Black <timblaktu at gmail.com> wrote:
>>>>> >
>>>>> > After perusing all of the roles' READMEs more thoroughly, I have
>>>>> updated my playbook (pasted below) with what I believe are the correct
>>>>> current set of available role variables in 3.4.1, with links to the docs
>>>>> for each. (would be nice if the example playbook was this informative.) One
>>>>> thing that came up with this exercise is that the example-use playbook is
>>>>> not including the main pulp role, however on tag 3.4.1 the pulp role
>>>>> appears to be a required dependency. Does the pulp role get included by the
>>>>> others, implicitly?
>>>>> >
>>>>> > Anyway, after a successful run of the modified playbook, I'm now
>>>>> seeing all services enabled:
>>>>> >
>>>>> > pulpadmin at pulp:~$ sudo systemctl list-unit-files | grep -E
>>>>> "(pulp|nginx)"
>>>>> > nginx.service enabled
>>>>> > pulpcore-api.service enabled
>>>>> > pulpcore-content.service enabled
>>>>> > pulpcore-resource-manager.service enabled
>>>>> > pulpcore-worker at .service indirect
>>>>> > dev-mapper-pulp\x2d\x2dvg\x2dswap_1.swap generated
>>>>> >
>>>>> > However, I'm still getting 502 trying to connect to pulp content
>>>>> webserver at my specified content_origin.
>>>>> >
>>>>> > My /var/log/nginx/error.log still shows the same type errors showing
>>>>> nginx can't connect with an upstream application server:
>>>>> >
>>>>> > 2020/07/07 13:59:41 [error] 12936#12936: *44 connect() failed (111:
>>>>> Connection refused) while connecting to upstream, client: 10.212.134.131,
>>>>> server: pulp, request: "GET /favicon.ico HTTP/1.1", upstream: "
>>>>> http://127.0.0.1:24817/favicon.ico", host: "pulp.my.domain",
>>>>> referrer: "http://pulp.my.domain/"
>>>>> >
>>>>> > Here's my updated pulp.yml:
>>>>> >
>>>>> > ---
>>>>> > # Playbook to provision and manage Pulp Instances for Artifact
>>>>> Management
>>>>> >
>>>>> > # Requires:
>>>>> > # (
>>>>> https://pulp-installer.readthedocs.io/en/latest/#system-requirements)
>>>>> > # 1. Debian Buster Machine Provisioned using Preseeded Installer
>>>>> > # a. Really just need Debian install with:
>>>>> > # i. sudo, openssh-server, python3
>>>>> > # (after installing with only ssh-server and system utility
>>>>> packages selected, only need to:
>>>>> > # su
>>>>> > # vi /etc/apt/sources.list # remove CD Rom line, add
>>>>> buster main repo if no mirror selected during install
>>>>> > # apt-get install sudo)
>>>>> > # ii. update-alternatives --set editor `update-alternatives
>>>>> --list editor | grep vim`
>>>>> > # iii. pulpadmin user with passwordless sudoer priviledges
>>>>> > # (echo "pulpadmin ALL=(ALL) NOPASSWD: ALL" >>
>>>>> /etc/sudoers)
>>>>> > # iv. ansible controller user has installed its ssh key in
>>>>> remote host's known_hosts
>>>>> > # (without this you'd just need to --ask-pass and supply
>>>>> ssh passwd at stdin)
>>>>> > # TODO: capture above in a VM Snapshot in vSphere/ESXi for
>>>>> fast reproduction.
>>>>> > # 2. Ansible Roles Installed via Galaxy using `$ ansible-galaxy
>>>>> install -r requirements-pulp.yml`
>>>>> > # 3. Ansible Collection Installed via Galaxy using `$ ansible-galaxy
>>>>> install -r requirements-pulp.yml`
>>>>> >
>>>>> > # Run like this:
>>>>> > # ansible-playbook pulp.yml --user pulpadmin --ask-pass
>>>>> --ask-vault-pass
>>>>> > #
>>>>> > # Note ansible knows what machines to run the playbook on by the
>>>>> `hosts` element within the playbook,
>>>>> > # which should have names existing in hosts file(s) in inventory/.
>>>>> >
>>>>> > # This playbook builds upon the Engineering Services playbook
>>>>> template
>>>>> > # Check imported playbook content before adding it here.
>>>>> > - import_playbook: engineering-services-tmplt.yml
>>>>> >
>>>>> > - name: "Install packages we want on every Pulp instance"
>>>>> > hosts: engineering_services_pulp
>>>>> > gather_facts: false
>>>>> > vars:
>>>>> > apt_packages:
>>>>> > - curl
>>>>> > roles:
>>>>> > - apt
>>>>> >
>>>>> > - name: Configure admin group
>>>>> > become: true
>>>>> > hosts: engineering_services_pulp
>>>>> > gather_facts: false
>>>>> > tasks:
>>>>> > - name: Create admin group
>>>>> > group:
>>>>> > name: admin
>>>>> >
>>>>> > - name: Configure admin user
>>>>> > become: true
>>>>> > hosts: engineering_services_pulp
>>>>> > gather_facts: false
>>>>> > vars:
>>>>> > # TODO: define these as inventory variable (standard for all
>>>>> machines?) so it can move out of playbook task blocks
>>>>> > tasks:
>>>>> > - debug: var=ansible_fqdn
>>>>> > - name: Configure admin user account
>>>>> > user:
>>>>> > name: admin
>>>>> > groups:
>>>>> > - admin
>>>>> >
>>>>> > - name: Install Pulp
>>>>> > hosts: engineering_services_pulp
>>>>> > # gather_facts: false
>>>>> > vars:
>>>>> > # Main Pulp Role Variables
>>>>> > #
>>>>> https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp#role-variables
>>>>> > pulp_settings:
>>>>> > secret_key: !vault |
>>>>> > $ANSIBLE_VAULT;1.1;AES256
>>>>> >
>>>>> 38383631633236306565616334663761363134613835323839653962323930616639656333653865
>>>>> >
>>>>> 3264363735643430626361383132653632316139396364370a613566396133393430663962666261
>>>>> >
>>>>> 35356165663639613535383563366638663635326662343133353339343262646265316630616162
>>>>> >
>>>>> 6337346131303833610a663232633339306231613738653233646466383638333934393765373034
>>>>> >
>>>>> 63346437343834653964366666333061303634313864333031323735326134626432626535613436
>>>>> >
>>>>> 62643731343836626436383438643862396166636263646330646332633637363765623866343733
>>>>> > 616635326537346163646564653134386666
>>>>> > content_origin: "http://{{ ansible_fqdn }}:8080"
>>>>> > pulp_install_plugins:
>>>>> > pulp-ansible: {}
>>>>> > pulp-container: {}
>>>>> > pulp-deb: {}
>>>>> > pulp-file: {}
>>>>> > pulp-python: {}
>>>>> > pulp_default_admin_password: !vault |
>>>>> > $ANSIBLE_VAULT;1.1;AES256
>>>>> >
>>>>> 35636365316538376363643965323035306461643239306433353665623438633535633763613662
>>>>> >
>>>>> 6266346236393736616532636230393136303966383339310a306563323838326431386432626465
>>>>> >
>>>>> 30316164383265303932643865323033623938656136306665356665336262613233653866386165
>>>>> >
>>>>> 3164396261326563640a613464353364656130396333613531383864323434316533663932303766
>>>>> > 3938
>>>>> > pulp_api_bind: "{{ ansible_fqdn }}"
>>>>> > pulp_api_workers: 4 # defaults to 1
>>>>> >
>>>>> > # Pulp Content Role Variables
>>>>> > #
>>>>> https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp_content#pulp_content
>>>>> > # pulp_content_bind: # Defaults to 127.0.0.1:24816
>>>>> >
>>>>> > # Pulp Database Role Variables
>>>>> > #
>>>>> https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp_database
>>>>> > # None
>>>>> >
>>>>> > # Pulp Resource Manager Role Variables
>>>>> > #
>>>>> https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp_resource_manager
>>>>> > # pulp_resouce_manager_state: # defaults to started
>>>>> > # pulp_resouce_manager_enabled: # defaults to true
>>>>> >
>>>>> > # Pulp Webserver Role Variables
>>>>> > #
>>>>> https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp_webserver
>>>>> > # pulp_webserver_server: # defauls to nginx
>>>>> > # pulp_content_port: # defaults to 24816
>>>>> > # pulp_content_host: # defaults to localhost
>>>>> > # pulp_api_port: # defaults to 24817
>>>>> > # pulp_api_host: # defaults to localhost
>>>>> > # pulp_configure_firewall: # defaults to auto, which is same as
>>>>> firewalld. Change to none to disable.
>>>>> >
>>>>> > # Pulp Workers Role Variables
>>>>> > #
>>>>> https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp_workers
>>>>> > # TODO: how is this different from pulp_api_workers in the main
>>>>> Pulp Role??
>>>>> > # pulp_workers: 4 # defaults to 2
>>>>> >
>>>>> > pre_tasks:
>>>>> > # The version string below is the highest of all those in roles'
>>>>> metadata:
>>>>> > # "min_ansible_version". It needs to be kept manually up-to-date.
>>>>> > - name: Verify Ansible meets min required version
>>>>> > assert:
>>>>> > that: "ansible_version.full is version_compare('2.8', '>=')"
>>>>> > msg: >
>>>>> > "You must update Ansible to at least 2.8 to use this
>>>>> version of Pulp 3 Installer."
>>>>> > roles:
>>>>> > # Is pulp role implicitly included by the others?
>>>>> > - pulp_database
>>>>> > - pulp_workers
>>>>> > - pulp_resource_manager
>>>>> > - pulp_webserver
>>>>> > - pulp_content
>>>>> > environment:
>>>>> > DJANGO_SETTINGS_MODULE: pulpcore.app.settings
>>>>> >
>>>>> > On Tue, Jul 7, 2020 at 12:24 PM Tim Black <timblaktu at gmail.com>
>>>>> wrote:
>>>>> >>
>>>>> >> I just installed my first pulp instance on a fresh Debian Buster
>>>>> VM, using latest Ansible pulp_installer release (3.4.1), with my pulp.yml
>>>>> playbook (pasted below) modeled after the official example-use playbook.
>>>>> The playbook runs to completion, with zero failed tasks, yet I am not able
>>>>> to connect to the pulp content webserver using the protocol/address/port I
>>>>> specified in the content_origin variable. I have verified that nginx
>>>>> service is running, but I still get 502: Bad Gateway error.
>>>>> >>
>>>>> >> Can someone help me troubleshoot this, or direct me to
>>>>> troubleshooting documentation that would assist? I found this excellent
>>>>> explanation which seems relevant since pulp uses the same nginx/gunicorn
>>>>> tech cocktail. It states:
>>>>> >>
>>>>> >>> NGINX will return a 502 Bad Gateway error if it can’t successfully
>>>>> proxy a request to Gunicorn or if Gunicorn fails to respond.
>>>>> >>
>>>>> >>
>>>>> >> I learned to look in /var/log/nginx/error.log for the reason for
>>>>> the issue. There I found several errors similar to this:
>>>>> >>
>>>>> >> [error] 4348#4348: *28 connect() failed (111: Connection refused)
>>>>> while connecting to upstream, client: 10.212.134.131, server: pulp,
>>>>> request: "GET / HTTP/1.1", upstream: "http://127.0.1.1:24817/", host:
>>>>> "pulp.my.domain"
>>>>> >>
>>>>> >> I also confirmed the following pulp service statuses:
>>>>> >>
>>>>> >> pulpadmin at pulp:~$ sudo systemctl list-unit-files | grep pulp
>>>>> >> pulpcore-api.service disabled
>>>>> >> pulpcore-content.service enabled
>>>>> >> pulpcore-resource-manager.service enabled
>>>>> >> pulpcore-worker at .service indirect
>>>>> >> dev-mapper-pulp\x2d\x2dvg\x2dswap_1.swap generated
>>>>> >>
>>>>> >> Hmm.. Shouldn't pulpcore-api be enabled? If so, I suppose this is
>>>>> the "upstream" service that nginx cannot connect to? From the error log, it
>>>>> looks like the address is localhost:24817, and I believe this is the
>>>>> default I chose. Anyone see any problem with what I'm doing here? I'm
>>>>> simply trying to set up "hello world" with pulp_installer targeting a
>>>>> dedicated remote server.
>>>>> >>
>>>>> >> I applaud the pulp dev team's modularizing of the code base, but I
>>>>> would love to see more documentation on the architecture here, clearly
>>>>> illustrating all these moving parts, with links to common problems like I'm
>>>>> having, with troubleshooting advice.
>>>>> >>
>>>>> >> Here's my pulp.yml ansible playbook:
>>>>> >>
>>>>> >> ---
>>>>> >> # Playbook to provision and manage Pulp Instances for Artifact
>>>>> Management
>>>>> >>
>>>>> >> # Requires:
>>>>> >> # (
>>>>> https://pulp-installer.readthedocs.io/en/latest/#system-requirements)
>>>>> >> # 1. Debian Buster Machine Provisioned using Preseeded Installer
>>>>> >> # a. Really just need Debian install with:
>>>>> >> # i. sudo, openssh-server, python3
>>>>> >> # (after installing with only ssh-server and system
>>>>> utility packages selected, only need to:
>>>>> >> # su
>>>>> >> # vi /etc/apt/sources.list # remove CD Rom line, add
>>>>> buster main repo if no mirror selected during install
>>>>> >> # apt-get install sudo)
>>>>> >> # ii. update-alternatives --set editor `update-alternatives
>>>>> --list editor | grep vim`
>>>>> >> # iii. pulpadmin user with passwordless sudoer priviledges
>>>>> >> # (echo "pulpadmin ALL=(ALL) NOPASSWD: ALL" >>
>>>>> /etc/sudoers)
>>>>> >> # iv. ansible controller user has installed its ssh key in
>>>>> remote host's known_hosts
>>>>> >> # (without this you'd just need to --ask-pass and supply
>>>>> ssh passwd at stdin)
>>>>> >> # TODO: capture above in a VM Snapshot in vSphere/ESXi for
>>>>> fast reproduction.
>>>>> >> # 2. Ansible Roles Installed via Galaxy using `$ ansible-galaxy
>>>>> install -r requirements-pulp.yml`
>>>>> >> # 3. Ansible Collection Installed via Galaxy using `$
>>>>> ansible-galaxy install -r requirements-pulp.yml`
>>>>> >> #
>>>>> >> # Run like this:
>>>>> >> # ansible-playbook pulp.yml --user pulpadmin -l
>>>>> <controlled-pulp-hostname> --ask-pass --ask-vault-pass
>>>>> >>
>>>>> >> # This playbook builds upon the Engineering Services playbook
>>>>> template
>>>>> >> # Check imported playbook content before adding it here.
>>>>> >> - import_playbook: engineering-services-tmplt.yml
>>>>> >>
>>>>> >> - name: "Install packages we want on every Pulp instance"
>>>>> >> hosts: engineering_services_pulp
>>>>> >> gather_facts: false
>>>>> >> vars:
>>>>> >> apt_packages:
>>>>> >> - curl
>>>>> >> roles:
>>>>> >> - apt
>>>>> >>
>>>>> >> - name: Configure admin group
>>>>> >> become: true
>>>>> >> hosts: engineering_services_pulp
>>>>> >> gather_facts: false
>>>>> >> tasks:
>>>>> >> - name: Create admin group
>>>>> >> group:
>>>>> >> name: admin
>>>>> >>
>>>>> >> - name: Configure admin user
>>>>> >> become: true
>>>>> >> hosts: engineering_services_pulp
>>>>> >> gather_facts: false
>>>>> >> vars:
>>>>> >> # TODO: define these as inventory variable (standard for all
>>>>> machines?) so it can move out of playbook task blocks
>>>>> >> tasks:
>>>>> >> - debug: var=ansible_fqdn
>>>>> >> - name: Configure admin user account
>>>>> >> user:
>>>>> >> name: admin
>>>>> >> groups:
>>>>> >> - admin
>>>>> >>
>>>>> >> - name: Install Pulp
>>>>> >> hosts: engineering_services_pulp
>>>>> >> # gather_facts: false
>>>>> >> vars:
>>>>> >> # required by pulp_installer:
>>>>> https://pulp-installer.readthedocs.io/en/latest/#system-requirements
>>>>> >> # TODO: this is now set in ansible.cfg bc it doesn't work when
>>>>> set here or in inventory
>>>>> >> # allow_world_readable_tmpfiles: True
>>>>> >> pulp_settings:
>>>>> >> secret_key: !vault |
>>>>> >> $ANSIBLE_VAULT;1.1;AES256
>>>>> >>
>>>>> 38383631633236306565616334663761363134613835323839653962323930616639656333653865
>>>>> >>
>>>>> 3264363735643430626361383132653632316139396364370a613566396133393430663962666261
>>>>> >>
>>>>> 35356165663639613535383563366638663635326662343133353339343262646265316630616162
>>>>> >>
>>>>> 6337346131303833610a663232633339306231613738653233646466383638333934393765373034
>>>>> >>
>>>>> 63346437343834653964366666333061303634313864333031323735326134626432626535613436
>>>>> >>
>>>>> 62643731343836626436383438643862396166636263646330646332633637363765623866343733
>>>>> >> 616635326537346163646564653134386666
>>>>> >> content_origin: "http://{{ ansible_fqdn }}:8080"
>>>>> >> pulp_default_admin_password: !vault |
>>>>> >> $ANSIBLE_VAULT;1.1;AES256
>>>>> >>
>>>>> 35636365316538376363643965323035306461643239306433353665623438633535633763613662
>>>>> >>
>>>>> 6266346236393736616532636230393136303966383339310a306563323838326431386432626465
>>>>> >>
>>>>> 30316164383265303932643865323033623938656136306665356665336262613233653866386165
>>>>> >>
>>>>> 3164396261326563640a613464353364656130396333613531383864323434316533663932303766
>>>>> >> 3938
>>>>> >> pulp_content_host: "{{ ansible_fqdn }}"
>>>>> >> # pulp_content_port: 24816
>>>>> >> pulp_content_port: 8080
>>>>> >> pulp_api_host: "{{ ansible_fqdn }}"
>>>>> >> # pulp_content_port: 24817
>>>>> >> pulp_content_bind: "{{ pulp_content_host }}:{{
>>>>> pulp_content_port }}"
>>>>> >> pulp_install_plugins:
>>>>> >> # galaxy-ng: {}
>>>>> >> pulp-ansible: {}
>>>>> >> # pulp-certguard: {}
>>>>> >> pulp-container: {}
>>>>> >> # pulp-cookbook: {}
>>>>> >> pulp-deb: {}
>>>>> >> pulp-file: {}
>>>>> >> # pulp-gem: {}
>>>>> >> # pulp-maven: {}
>>>>> >> # pulp-npm: {}
>>>>> >> pulp-python: {}
>>>>> >> # pulp-rpm: {}
>>>>> >> pre_tasks:
>>>>> >> # The version string below is the highest of all those in
>>>>> roles' metadata:
>>>>> >> # "min_ansible_version". It needs to be kept manually
>>>>> up-to-date.
>>>>> >> - name: Verify Ansible meets min required version
>>>>> >> assert:
>>>>> >> that: "ansible_version.full is version_compare('2.8', '>=')"
>>>>> >> msg: >
>>>>> >> "You must update Ansible to at least 2.8 to use this
>>>>> version of Pulp 3 Installer."
>>>>> >> roles:
>>>>> >> - pulp_database
>>>>> >> - pulp_workers
>>>>> >> - pulp_resource_manager
>>>>> >> - pulp_webserver
>>>>> >> - pulp_content
>>>>> >> environment:
>>>>> >> DJANGO_SETTINGS_MODULE: pulpcore.app.settings
>>>>> >>
>>>>> >> Thanks for your help.
>>>>> >>
>>>>> >> Tim
>>>>> >
>>>>> > _______________________________________________
>>>>> > Pulp-list mailing list
>>>>> > Pulp-list at redhat.com
>>>>> > https://www.redhat.com/mailman/listinfo/pulp-list
>>>>>
>>>>> _______________________________________________
>>>> Pulp-list mailing list
>>>> Pulp-list at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/pulp-list
>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20200727/d87e9c17/attachment.htm>
More information about the Pulp-list
mailing list