[Pulp-list] 502 Bad Gateway error connecting to new pulp instance installed with pulp_installer
Dennis Kliban
dkliban at redhat.com
Wed Aug 5 20:13:28 UTC 2020
Looks like you filed a bug about this[0]. The problem is that you were
trying to install pulp_python and it didn't have a 3.5.0 compatible
release. We are releasing it now.
In the future when a similar situation occurs, you can use an older release
of the installer and specify the 'pulp_install_plugins' variable with
specific versions of plugins and their versions[1].
[0] https://pulp.plan.io/issues/7223
[1]
https://pulp-installer.readthedocs.io/en/latest/roles/pulp_common/#role-variables
On Mon, Jul 27, 2020 at 7:53 PM Tim Black <timblaktu at gmail.com> wrote:
> Regarding my question about a turnkey vagrant solution, searching around
> pulpproject i found pulplift <https://github.com/pulp/pulplift>, which
> appears to contain vagrant boxes for bringing up and developing pulp on
> various OSes. When I get some more time, I'll try to have a deeper look and
> see if any of them work out of the box with my same pulp_installer version
> and os versions..
>
> On Mon, Jul 27, 2020 at 3:34 PM Tim Black <timblaktu at gmail.com> wrote:
>
>> Created this: https://pulp.plan.io/issues/7223
>>
>> But would still love to get advice on how to get ANY pulp instance
>> brought up as nothing I've tried so far has worked. Can anyone share a
>> working vm settings/ansible playbook that "just works"? Even one that just
>> brings it up on localhost would be fine for now.
>>
>> On Mon, Jul 27, 2020 at 3:15 PM Tim Black <timblaktu at gmail.com> wrote:
>>
>>> Using pulp_installer 3.5.0 and this:
>>>
>>> roles:
>>> - pulp_all_services
>>>
>>> also produces the version compatibility error (posted above) like I was
>>> getting using 3.4.1 which uses a different role pattern:
>>>
>>> roles:
>>> - pulp_database
>>> - pulp_workers
>>> - pulp_resource_manager
>>> - pulp_webserver
>>> - pulp_content
>>>
>>> I will file a bug.
>>>
>>> On Mon, Jul 27, 2020 at 3:04 PM Tim Black <timblaktu at gmail.com> wrote:
>>>
>>>> Correction: using pulp_installer 3.5.0, I am still getting the same
>>>> error pulpcore/plugin compatibility error message I was getting with 3.4.1.
>>>> (I got past the secret_key error by specifying it in plain text in my
>>>> playbook instead of using vault (for now).) I am at a bit of a standstill,
>>>> and am going to shift gears and wait for some guidance or suggestions for
>>>> how to move forward with using pulp. Thanks again.
>>>>
>>>> On Mon, Jul 27, 2020 at 2:53 PM Tim Black <timblaktu at gmail.com> wrote:
>>>>
>>>>> Also.. I notice that on the 3.5.0 tag of pulp_installer, the
>>>>> example-use playbook
>>>>> <https://github.com/pulp/pulp_installer/blob/3.5.0/playbooks/example-use/playbook.yml>
>>>>> now has gone back to using the "pulp_all_services" role instead of listing
>>>>> each role separately, like it was doing before. Since I'm now using 3.5.0
>>>>> pulp_installer, should I be following this new pattern?
>>>>>
>>>>> I would like to also reiterate my request for a
>>>>> vagrant-virtualbox-based solution that "just works" that can be shared with
>>>>> me and other newbies. Seems like enabling this level of turnkey automation
>>>>> is the whole goal of using ansible to begin with. Does this exist
>>>>> somewhere? Thanks.
>>>>>
>>>>> On Mon, Jul 27, 2020 at 2:47 PM Tim Black <timblaktu at gmail.com> wrote:
>>>>>
>>>>>> Thanks Dennis. I finally got some time to work on this, and have
>>>>>> started over again, this time using the latest centos iso: 8.2.2004. I do
>>>>>> not have support for centos in my ansible bootstrapping playbooks, which
>>>>>> typically operate on a debian-based machine/snapshot with a fixed hostname
>>>>>> and user. So, for now I've done the following manual steps post centOS
>>>>>> install, before running my *slightly simplified pulp.yml ansible
>>>>>> playbook:
>>>>>>
>>>>>> (* all my pulp.yml is doing now is configuring an admin/admin
>>>>>> user/group, then running the pulp_installer, with same options as I posted
>>>>>> before.)
>>>>>>
>>>>>> 1. ssh-copy-id -i ~/.ssh/id_rsa.pub ansible at pulpcentos and confirm
>>>>>> that I can:
>>>>>> 1. ssh as ansible user without password
>>>>>> 2. sudo as ansible user with password
>>>>>> 2. sudo yum install python3
>>>>>>
>>>>>> Unfortunately, now I get an error in the compatibility check between
>>>>>> pulpcore and plugins:
>>>>>>
>>>>>> TASK [Run pip-compile to check pulpcore/plugin compatibility]
>>>>>> *****************************************************************************************************[20/7382]
>>>>>> Monday 27 July 2020 14:23:18 -0700 (0:00:00.287) 0:00:46.377
>>>>>> ***********
>>>>>> [WARNING]: conditional statements should not
>>>>>> include jinja2 templating delimiters such as {{ }} or {% %}. Found: {{
>>>>>> failed_condition | default("compatibility.rc != 0") }} fatal:
>>>>>> [pulpcentos]: FAILED! => changed=false
>>>>>>
>>>>>> cmd:
>>>>>>
>>>>>> - /usr/local/lib/pulp/bin/pip-compile
>>>>>>
>>>>>> delta:
>>>>>> '0:00:03.171889'
>>>>>>
>>>>>> end: '2020-07-27 14:23:21.863378'
>>>>>>
>>>>>> failed_when_result: true
>>>>>>
>>>>>> msg: non-zero return
>>>>>> code
>>>>>>
>>>>>> rc: 2
>>>>>>
>>>>>> start: '2020-07-27 14:23:18.691489'
>>>>>>
>>>>>> stderr: |-
>>>>>>
>>>>>> Could not
>>>>>> find a version that matches pulpcore<3.5,<3.6,==3.4.1,>=3.0,>=3.4,>=3.5
>>>>>> from
>>>>>> https://files.pythonhosted.org/packages/5c/40/8dab8ccfe73982ef3a5e48489af2d83974b0e7677ca52ec232fcb4b49dfa/pulpcore-3.4.1-py3-none-any.whl#sha256=e33ca32f867201e1a18b888d72ef07e85c2cd11273a8e422e33d6a2910a64fac
>>>>>> (from -r requirements.in (line 1)) Tried: 3.0.0, 3.0.0,
>>>>>> 3.0.1, 3.0.1, 3.1.0, 3.1.0, 3.1.1, 3.1.1, 3.2.0, 3.2.0, 3.2.1, 3.2.1,
>>>>>> 3.3.0, 3.3.0, 3.3.1, 3.3.1, 3.4.0, 3.4.0, 3.4.1, 3.4.1, 3.5.0, 3.5.0
>>>>>> Skipped pre-versions: 3.0.0b1, 3.0.0b1, 3.0.0b2, 3.0.0b2, 3.0.0b3,
>>>>>> 3.0.0b3, 3.0.0b4, 3.0.0b4, 3.0.0b5, 3.0.0b5, 3.0.0b6, 3.0.0b6, 3.0.0b7,
>>>>>> 3.0.0b7, 3.0.0b8, 3.0.0b8, 3.0.0b9, 3.0.0b9, 3.0.0b10, 3.0.0b10, 3.0.0b11,
>>>>>> 3.0.0b11, 3.0.0b12, 3.0.0b12, 3.0.0b13, 3.0.0b13, 3.0.0b14, 3.0.0b14,
>>>>>> 3.0.0b15, 3.0.0b15, 3.0.0b16, 3.0.0b16, 3.0.0b17, 3.0.0b18, 3.0.0b18,
>>>>>> 3.0.0b19, 3.0.0b19, 3.0.0b20, 3.0.0b20, 3.0.0b21, 3.0.0b21, 3.0.0b22,
>>>>>> 3.0.0b22, 3.0.0b23, 3.0.0b23, 3.0.0rc1, 3.0.0rc1, 3.0.0rc2, 3.0.0rc2,
>>>>>> 3.0.0rc3, 3.0.0rc3, 3.0.0rc4, 3.0.0rc4, 3.0.0rc5, 3.0.0rc5, 3.0.0rc6,
>>>>>> 3.0.0rc6, 3.0.0rc7, 3.0.0rc7, 3.0.0rc8, 3.0.0rc8, 3.0.0rc9, 3.0.0rc9
>>>>>> There are incompatible
>>>>>> versions in the resolved dependencies:
>>>>>>
>>>>>> pulpcore==3.4.1 from
>>>>>> https://files.pythonhosted.org/packages/5c/40/8dab8ccfe73982ef3a5e48489af2d83974b0e7677ca52ec232fcb4b49dfa/pulpcore-3.4.1-py3-none-any.whl#sha256=e33ca32f867201e1a18b888d72ef07e85c2cd11273a8e422e33d6a2910a64fac
>>>>>> (from -r requirements.in (line 1))
>>>>>> pulpcore<3.6,>=3.4 (from pulp-file==1.1.0->-r requirements.in
>>>>>> (line 5))
>>>>>> pulpcore<3.6,>=3.4 (from pulp-container==1.4.2->-r
>>>>>> requirements.in (line 3))
>>>>>> pulpcore<3.5,>=3.4 (from pulp-python==3.0.0b9->-r
>>>>>> requirements.in (line 6))
>>>>>> pulpcore<3.6,>=3.5 (from pulp-deb==2.5.0b1->-r requirements.in
>>>>>> (line 4))
>>>>>> pulpcore<3.6,>=3.0 (from pulp-ansible==0.2.0b15->-r
>>>>>> requirements.in (line 2))
>>>>>> stderr_lines: <omitted>
>>>>>> stdout: ''
>>>>>> stdout_lines: <omitted>
>>>>>>
>>>>>> PLAY RECAP
>>>>>> *****************************************************************************************************************************************************************pulpcentos
>>>>>> : ok=33 changed=14 unreachable=0 failed=1
>>>>>> skipped=16 rescued=0 ignored=0
>>>>>>
>>>>>> I believe this means that the version of pulp_installer role(s) I
>>>>>> have/had installed have become broken bc of compatibility changes made to
>>>>>> one or more versions they were referencing. This seems bad, nevertheless, I
>>>>>> went ahead and updated my pulp_installer to a newer tag (from 3.4.1 to
>>>>>> 3.5.0), and reran the pulp.yml playbook, with the following results:
>>>>>>
>>>>>> With 3.5.0 pulp_installer, running against fresh new centos 8
>>>>>> machine, it got past the pulpcore/plugin version check, but failed here, in
>>>>>> pulp_common's check for required variables. This worked fine before (on my
>>>>>> debian-based machine) as you can see in my playbook I'm using an
>>>>>> ansible-vault encrypted string as the secret_key.
>>>>>>
>>>>>> TASK [pulp_common : Check if required variables are set]
>>>>>> *******************************************************************************************************************Monday
>>>>>> 27 July 2020 14:34:27 -0700 (0:00:00.024) 0:00:19.821 ***********
>>>>>>
>>>>>> ok: [pulpcentos] => (item=pulp_settings.content_origin) =>
>>>>>> changed=false
>>>>>> ansible_loop_var: item
>>>>>>
>>>>>> item:
>>>>>> pulp_settings.content_origin
>>>>>>
>>>>>> msg: All assertions passed
>>>>>>
>>>>>> fatal: [pulpcentos]: FAILED! =>
>>>>>> msg: 'The conditional check ''pulp_settings.secret_key |
>>>>>> default('''', true) | length > 0'' failed. The error was: Unexpected
>>>>>> templating type error occurred on ({% if pulp_settings.secret_key |
>>>>>> default('''', true) | length > 0 %} True {% else %} False {% endif %}):
>>>>>> object of type ''AnsibleVaultEncryptedUnicode'' has no len()'
>>>>>>
>>>>>> Not sure what's up, but at the very least so far it's not working any
>>>>>> better with CentOS. I'm all ears for suggestions.
>>>>>>
>>>>>> Does anyone have a turnkey, fully-automated solution they can share,
>>>>>> like a vagrant box that brings up a pulp instance from scratch? Seems like
>>>>>> I'm doing a lot more work here than should be required to bring this thing
>>>>>> up. Thanks.
>>>>>>
>>>>>> On Sat, Jul 11, 2020 at 1:49 PM Dennis Kliban <dkliban at redhat.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I would recommend re-running the installer on a fresh VM that is
>>>>>>> running CentOS 7.7+. I've experienced this problem before when the
>>>>>>> installer had to be run multiple times due to various failures. In my case,
>>>>>>> the database migrations had not been run and the output of "systemctl
>>>>>>> status pulpcore*" showed that Pulp services were failing to start due to
>>>>>>> database issues. I suspected it was due to permissions problems with
>>>>>>> /etc/pulp/settings.py, however, I never confirmed this by actually fixing
>>>>>>> the install. I've always just reprovisioned on a new VM.
>>>>>>>
>>>>>>> If you can reproduce this issue again on a new VM, I would recommend
>>>>>>> filing an issue at https://pulp.plan.io/issues/new/. The installer
>>>>>>> is definitely doing something wrong, but I am not sure how to reproduce the
>>>>>>> issue consistently.
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Jul 10, 2020 at 11:12 PM Tim Black <timblaktu at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thanks Matthias. I get 502 at
>>>>>>>> http://pulp.my.domain/pulp/api/v3/status/ as well. Below is my
>>>>>>>> nginx.conf, pulled from my freshly provisioned pulp server. My skills are a
>>>>>>>> little weak on the webserver side of things so I'm open to suggestions for
>>>>>>>> any simplifications I can make to my config to get this working. I'm not
>>>>>>>> trying to do anything fancy here.
>>>>>>>>
>>>>>>>> /etc/nginx/nginx.conf:
>>>>>>>>
>>>>>>>> # TODO: Support IPv6.
>>>>>>>> # TODO: Configure SSL certificates.
>>>>>>>> # TODO: Maybe serve multiple `location`s, not just one.
>>>>>>>>
>>>>>>>> # Gunicorn docs suggest this value.
>>>>>>>> worker_processes 1;
>>>>>>>> events {
>>>>>>>> worker_connections 1024; # increase if you have lots of clients
>>>>>>>> accept_mutex off; # set to 'on' if nginx worker_processes > 1
>>>>>>>> }
>>>>>>>>
>>>>>>>> http {
>>>>>>>> include mime.types;
>>>>>>>> # fallback in case we can't determine a type
>>>>>>>> default_type application/octet-stream;
>>>>>>>> sendfile on;
>>>>>>>>
>>>>>>>> # If left at the default of 1024, nginx emits a warning about
>>>>>>>> being unable
>>>>>>>> # to build optimal hash types.
>>>>>>>> types_hash_max_size 4096;
>>>>>>>>
>>>>>>>> upstream pulp-content {
>>>>>>>> server 127.0.0.1:24816;
>>>>>>>> }
>>>>>>>>
>>>>>>>> upstream pulp-api {
>>>>>>>> server 127.0.0.1:24817;
>>>>>>>> }
>>>>>>>>
>>>>>>>> server {
>>>>>>>> # Gunicorn docs suggest the use of the "deferred" directive
>>>>>>>> on Linux.
>>>>>>>> listen 80 default_server deferred;
>>>>>>>> server_name $hostname;
>>>>>>>>
>>>>>>>> # The default client_max_body_size is 1m. Clients uploading
>>>>>>>> # files larger than this will need to chunk said files.
>>>>>>>>
>>>>>>>> # Gunicorn docs suggest this value.
>>>>>>>> keepalive_timeout 5;
>>>>>>>>
>>>>>>>> location /pulp/content/ {
>>>>>>>> proxy_set_header X-Forwarded-For
>>>>>>>> $proxy_add_x_forwarded_for;
>>>>>>>> proxy_set_header X-Forwarded-Proto $scheme;
>>>>>>>> proxy_set_header Host $http_host;
>>>>>>>> # we don't want nginx trying to do something clever with
>>>>>>>> # redirects, we set the Host: header above already.
>>>>>>>> proxy_redirect off;
>>>>>>>> proxy_pass http://pulp-content;
>>>>>>>> }
>>>>>>>>
>>>>>>>> location /pulp/api/v3/ {
>>>>>>>> proxy_set_header X-Forwarded-For
>>>>>>>> $proxy_add_x_forwarded_for;
>>>>>>>> proxy_set_header X-Forwarded-Proto $scheme;
>>>>>>>> proxy_set_header Host $http_host;
>>>>>>>> # we don't want nginx trying to do something clever with
>>>>>>>> # redirects, we set the Host: header above already.
>>>>>>>> proxy_redirect off;
>>>>>>>> proxy_pass http://pulp-api;
>>>>>>>> }
>>>>>>>>
>>>>>>>> location /auth/login/ {
>>>>>>>> proxy_set_header X-Forwarded-For
>>>>>>>> $proxy_add_x_forwarded_for;
>>>>>>>> proxy_set_header X-Forwarded-Proto $scheme;
>>>>>>>> proxy_set_header Host $http_host;
>>>>>>>> # we don't want nginx trying to do something clever with
>>>>>>>> # redirects, we set the Host: header above already.
>>>>>>>> proxy_redirect off;
>>>>>>>> proxy_pass http://pulp-api;
>>>>>>>> }
>>>>>>>>
>>>>>>>> include pulp/*.conf;
>>>>>>>>
>>>>>>>> location / {
>>>>>>>> proxy_set_header X-Forwarded-For
>>>>>>>> $proxy_add_x_forwarded_for;
>>>>>>>> proxy_set_header X-Forwarded-Proto $scheme;
>>>>>>>> proxy_set_header Host $http_host;
>>>>>>>> # we don't want nginx trying to do something clever with
>>>>>>>> # redirects, we set the Host: header above already.
>>>>>>>> proxy_redirect off;
>>>>>>>> proxy_pass http://pulp-api;
>>>>>>>> # static files are served through whitenoise -
>>>>>>>> http://whitenoise.evans.io/en/stable/
>>>>>>>> }
>>>>>>>> }
>>>>>>>> }
>>>>>>>>
>>>>>>>> On Tue, Jul 7, 2020 at 11:56 PM Matthias Dellweg <
>>>>>>>> mdellweg at redhat.com> wrote:
>>>>>>>>
>>>>>>>>> The only thing that sticks out to me is `content_origin: "http://
>>>>>>>>> {{
>>>>>>>>> ansible_fqdn }}:8080"`. This is the address seen from the outside,
>>>>>>>>> and
>>>>>>>>> since both content and api are subject to the same reverse proxy
>>>>>>>>> and
>>>>>>>>> so should be available on port 80 (and 443 soon). But that is for
>>>>>>>>> sure
>>>>>>>>> not the problem you have with the API.
>>>>>>>>> Can you, however, try `http
>>>>>>>>> http://pulp.my.domain/pulp/api/v3/status/`
>>>>>>>>> <http://pulp.my.domain/pulp/api/v3/status/>? And if it still
>>>>>>>>> didn't
>>>>>>>>> produce a result, provide the content of /etc/nginx/nginx.conf ?
>>>>>>>>>
>>>>>>>>> On Tue, Jul 7, 2020 at 11:18 PM Tim Black <timblaktu at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>> >
>>>>>>>>> > After perusing all of the roles' READMEs more thoroughly, I have
>>>>>>>>> updated my playbook (pasted below) with what I believe are the correct
>>>>>>>>> current set of available role variables in 3.4.1, with links to the docs
>>>>>>>>> for each. (would be nice if the example playbook was this informative.) One
>>>>>>>>> thing that came up with this exercise is that the example-use playbook is
>>>>>>>>> not including the main pulp role, however on tag 3.4.1 the pulp role
>>>>>>>>> appears to be a required dependency. Does the pulp role get included by the
>>>>>>>>> others, implicitly?
>>>>>>>>> >
>>>>>>>>> > Anyway, after a successful run of the modified playbook, I'm now
>>>>>>>>> seeing all services enabled:
>>>>>>>>> >
>>>>>>>>> > pulpadmin at pulp:~$ sudo systemctl list-unit-files | grep -E
>>>>>>>>> "(pulp|nginx)"
>>>>>>>>> > nginx.service enabled
>>>>>>>>> > pulpcore-api.service enabled
>>>>>>>>> > pulpcore-content.service enabled
>>>>>>>>> > pulpcore-resource-manager.service enabled
>>>>>>>>> > pulpcore-worker at .service indirect
>>>>>>>>> > dev-mapper-pulp\x2d\x2dvg\x2dswap_1.swap generated
>>>>>>>>> >
>>>>>>>>> > However, I'm still getting 502 trying to connect to pulp content
>>>>>>>>> webserver at my specified content_origin.
>>>>>>>>> >
>>>>>>>>> > My /var/log/nginx/error.log still shows the same type errors
>>>>>>>>> showing nginx can't connect with an upstream application server:
>>>>>>>>> >
>>>>>>>>> > 2020/07/07 13:59:41 [error] 12936#12936: *44 connect() failed
>>>>>>>>> (111: Connection refused) while connecting to upstream, client:
>>>>>>>>> 10.212.134.131, server: pulp, request: "GET /favicon.ico HTTP/1.1",
>>>>>>>>> upstream: "http://127.0.0.1:24817/favicon.ico", host:
>>>>>>>>> "pulp.my.domain", referrer: "http://pulp.my.domain/"
>>>>>>>>> >
>>>>>>>>> > Here's my updated pulp.yml:
>>>>>>>>> >
>>>>>>>>> > ---
>>>>>>>>> > # Playbook to provision and manage Pulp Instances for Artifact
>>>>>>>>> Management
>>>>>>>>> >
>>>>>>>>> > # Requires:
>>>>>>>>> > # (
>>>>>>>>> https://pulp-installer.readthedocs.io/en/latest/#system-requirements
>>>>>>>>> )
>>>>>>>>> > # 1. Debian Buster Machine Provisioned using Preseeded Installer
>>>>>>>>> > # a. Really just need Debian install with:
>>>>>>>>> > # i. sudo, openssh-server, python3
>>>>>>>>> > # (after installing with only ssh-server and system
>>>>>>>>> utility packages selected, only need to:
>>>>>>>>> > # su
>>>>>>>>> > # vi /etc/apt/sources.list # remove CD Rom line, add
>>>>>>>>> buster main repo if no mirror selected during install
>>>>>>>>> > # apt-get install sudo)
>>>>>>>>> > # ii. update-alternatives --set editor
>>>>>>>>> `update-alternatives --list editor | grep vim`
>>>>>>>>> > # iii. pulpadmin user with passwordless sudoer priviledges
>>>>>>>>> > # (echo "pulpadmin ALL=(ALL) NOPASSWD: ALL" >>
>>>>>>>>> /etc/sudoers)
>>>>>>>>> > # iv. ansible controller user has installed its ssh key in
>>>>>>>>> remote host's known_hosts
>>>>>>>>> > # (without this you'd just need to --ask-pass and
>>>>>>>>> supply ssh passwd at stdin)
>>>>>>>>> > # TODO: capture above in a VM Snapshot in vSphere/ESXi for
>>>>>>>>> fast reproduction.
>>>>>>>>> > # 2. Ansible Roles Installed via Galaxy using `$ ansible-galaxy
>>>>>>>>> install -r requirements-pulp.yml`
>>>>>>>>> > # 3. Ansible Collection Installed via Galaxy using `$
>>>>>>>>> ansible-galaxy install -r requirements-pulp.yml`
>>>>>>>>> >
>>>>>>>>> > # Run like this:
>>>>>>>>> > # ansible-playbook pulp.yml --user pulpadmin --ask-pass
>>>>>>>>> --ask-vault-pass
>>>>>>>>> > #
>>>>>>>>> > # Note ansible knows what machines to run the playbook on by the
>>>>>>>>> `hosts` element within the playbook,
>>>>>>>>> > # which should have names existing in hosts file(s) in
>>>>>>>>> inventory/.
>>>>>>>>> >
>>>>>>>>> > # This playbook builds upon the Engineering Services playbook
>>>>>>>>> template
>>>>>>>>> > # Check imported playbook content before adding it here.
>>>>>>>>> > - import_playbook: engineering-services-tmplt.yml
>>>>>>>>> >
>>>>>>>>> > - name: "Install packages we want on every Pulp instance"
>>>>>>>>> > hosts: engineering_services_pulp
>>>>>>>>> > gather_facts: false
>>>>>>>>> > vars:
>>>>>>>>> > apt_packages:
>>>>>>>>> > - curl
>>>>>>>>> > roles:
>>>>>>>>> > - apt
>>>>>>>>> >
>>>>>>>>> > - name: Configure admin group
>>>>>>>>> > become: true
>>>>>>>>> > hosts: engineering_services_pulp
>>>>>>>>> > gather_facts: false
>>>>>>>>> > tasks:
>>>>>>>>> > - name: Create admin group
>>>>>>>>> > group:
>>>>>>>>> > name: admin
>>>>>>>>> >
>>>>>>>>> > - name: Configure admin user
>>>>>>>>> > become: true
>>>>>>>>> > hosts: engineering_services_pulp
>>>>>>>>> > gather_facts: false
>>>>>>>>> > vars:
>>>>>>>>> > # TODO: define these as inventory variable (standard for all
>>>>>>>>> machines?) so it can move out of playbook task blocks
>>>>>>>>> > tasks:
>>>>>>>>> > - debug: var=ansible_fqdn
>>>>>>>>> > - name: Configure admin user account
>>>>>>>>> > user:
>>>>>>>>> > name: admin
>>>>>>>>> > groups:
>>>>>>>>> > - admin
>>>>>>>>> >
>>>>>>>>> > - name: Install Pulp
>>>>>>>>> > hosts: engineering_services_pulp
>>>>>>>>> > # gather_facts: false
>>>>>>>>> > vars:
>>>>>>>>> > # Main Pulp Role Variables
>>>>>>>>> > #
>>>>>>>>> https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp#role-variables
>>>>>>>>> > pulp_settings:
>>>>>>>>> > secret_key: !vault |
>>>>>>>>> > $ANSIBLE_VAULT;1.1;AES256
>>>>>>>>> >
>>>>>>>>> 38383631633236306565616334663761363134613835323839653962323930616639656333653865
>>>>>>>>> >
>>>>>>>>> 3264363735643430626361383132653632316139396364370a613566396133393430663962666261
>>>>>>>>> >
>>>>>>>>> 35356165663639613535383563366638663635326662343133353339343262646265316630616162
>>>>>>>>> >
>>>>>>>>> 6337346131303833610a663232633339306231613738653233646466383638333934393765373034
>>>>>>>>> >
>>>>>>>>> 63346437343834653964366666333061303634313864333031323735326134626432626535613436
>>>>>>>>> >
>>>>>>>>> 62643731343836626436383438643862396166636263646330646332633637363765623866343733
>>>>>>>>> > 616635326537346163646564653134386666
>>>>>>>>> > content_origin: "http://{{ ansible_fqdn }}:8080"
>>>>>>>>> > pulp_install_plugins:
>>>>>>>>> > pulp-ansible: {}
>>>>>>>>> > pulp-container: {}
>>>>>>>>> > pulp-deb: {}
>>>>>>>>> > pulp-file: {}
>>>>>>>>> > pulp-python: {}
>>>>>>>>> > pulp_default_admin_password: !vault |
>>>>>>>>> > $ANSIBLE_VAULT;1.1;AES256
>>>>>>>>> >
>>>>>>>>> 35636365316538376363643965323035306461643239306433353665623438633535633763613662
>>>>>>>>> >
>>>>>>>>> 6266346236393736616532636230393136303966383339310a306563323838326431386432626465
>>>>>>>>> >
>>>>>>>>> 30316164383265303932643865323033623938656136306665356665336262613233653866386165
>>>>>>>>> >
>>>>>>>>> 3164396261326563640a613464353364656130396333613531383864323434316533663932303766
>>>>>>>>> > 3938
>>>>>>>>> > pulp_api_bind: "{{ ansible_fqdn }}"
>>>>>>>>> > pulp_api_workers: 4 # defaults to 1
>>>>>>>>> >
>>>>>>>>> > # Pulp Content Role Variables
>>>>>>>>> > #
>>>>>>>>> https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp_content#pulp_content
>>>>>>>>> > # pulp_content_bind: # Defaults to 127.0.0.1:24816
>>>>>>>>> >
>>>>>>>>> > # Pulp Database Role Variables
>>>>>>>>> > #
>>>>>>>>> https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp_database
>>>>>>>>> > # None
>>>>>>>>> >
>>>>>>>>> > # Pulp Resource Manager Role Variables
>>>>>>>>> > #
>>>>>>>>> https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp_resource_manager
>>>>>>>>> > # pulp_resouce_manager_state: # defaults to started
>>>>>>>>> > # pulp_resouce_manager_enabled: # defaults to true
>>>>>>>>> >
>>>>>>>>> > # Pulp Webserver Role Variables
>>>>>>>>> > #
>>>>>>>>> https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp_webserver
>>>>>>>>> > # pulp_webserver_server: # defauls to nginx
>>>>>>>>> > # pulp_content_port: # defaults to 24816
>>>>>>>>> > # pulp_content_host: # defaults to localhost
>>>>>>>>> > # pulp_api_port: # defaults to 24817
>>>>>>>>> > # pulp_api_host: # defaults to localhost
>>>>>>>>> > # pulp_configure_firewall: # defaults to auto, which is
>>>>>>>>> same as firewalld. Change to none to disable.
>>>>>>>>> >
>>>>>>>>> > # Pulp Workers Role Variables
>>>>>>>>> > #
>>>>>>>>> https://github.com/pulp/pulp_installer/tree/3.4.1/roles/pulp_workers
>>>>>>>>> > # TODO: how is this different from pulp_api_workers in the
>>>>>>>>> main Pulp Role??
>>>>>>>>> > # pulp_workers: 4 # defaults to 2
>>>>>>>>> >
>>>>>>>>> > pre_tasks:
>>>>>>>>> > # The version string below is the highest of all those in
>>>>>>>>> roles' metadata:
>>>>>>>>> > # "min_ansible_version". It needs to be kept manually
>>>>>>>>> up-to-date.
>>>>>>>>> > - name: Verify Ansible meets min required version
>>>>>>>>> > assert:
>>>>>>>>> > that: "ansible_version.full is version_compare('2.8',
>>>>>>>>> '>=')"
>>>>>>>>> > msg: >
>>>>>>>>> > "You must update Ansible to at least 2.8 to use this
>>>>>>>>> version of Pulp 3 Installer."
>>>>>>>>> > roles:
>>>>>>>>> > # Is pulp role implicitly included by the others?
>>>>>>>>> > - pulp_database
>>>>>>>>> > - pulp_workers
>>>>>>>>> > - pulp_resource_manager
>>>>>>>>> > - pulp_webserver
>>>>>>>>> > - pulp_content
>>>>>>>>> > environment:
>>>>>>>>> > DJANGO_SETTINGS_MODULE: pulpcore.app.settings
>>>>>>>>> >
>>>>>>>>> > On Tue, Jul 7, 2020 at 12:24 PM Tim Black <timblaktu at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>> >>
>>>>>>>>> >> I just installed my first pulp instance on a fresh Debian
>>>>>>>>> Buster VM, using latest Ansible pulp_installer release (3.4.1), with my
>>>>>>>>> pulp.yml playbook (pasted below) modeled after the official example-use
>>>>>>>>> playbook. The playbook runs to completion, with zero failed tasks, yet I am
>>>>>>>>> not able to connect to the pulp content webserver using the
>>>>>>>>> protocol/address/port I specified in the content_origin variable. I have
>>>>>>>>> verified that nginx service is running, but I still get 502: Bad Gateway
>>>>>>>>> error.
>>>>>>>>> >>
>>>>>>>>> >> Can someone help me troubleshoot this, or direct me to
>>>>>>>>> troubleshooting documentation that would assist? I found this excellent
>>>>>>>>> explanation which seems relevant since pulp uses the same nginx/gunicorn
>>>>>>>>> tech cocktail. It states:
>>>>>>>>> >>
>>>>>>>>> >>> NGINX will return a 502 Bad Gateway error if it can’t
>>>>>>>>> successfully proxy a request to Gunicorn or if Gunicorn fails to respond.
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >> I learned to look in /var/log/nginx/error.log for the reason
>>>>>>>>> for the issue. There I found several errors similar to this:
>>>>>>>>> >>
>>>>>>>>> >> [error] 4348#4348: *28 connect() failed (111: Connection
>>>>>>>>> refused) while connecting to upstream, client: 10.212.134.131, server:
>>>>>>>>> pulp, request: "GET / HTTP/1.1", upstream: "
>>>>>>>>> http://127.0.1.1:24817/", host: "pulp.my.domain"
>>>>>>>>> >>
>>>>>>>>> >> I also confirmed the following pulp service statuses:
>>>>>>>>> >>
>>>>>>>>> >> pulpadmin at pulp:~$ sudo systemctl list-unit-files | grep pulp
>>>>>>>>> >> pulpcore-api.service disabled
>>>>>>>>> >> pulpcore-content.service enabled
>>>>>>>>> >> pulpcore-resource-manager.service enabled
>>>>>>>>> >> pulpcore-worker at .service indirect
>>>>>>>>> >> dev-mapper-pulp\x2d\x2dvg\x2dswap_1.swap generated
>>>>>>>>> >>
>>>>>>>>> >> Hmm.. Shouldn't pulpcore-api be enabled? If so, I suppose this
>>>>>>>>> is the "upstream" service that nginx cannot connect to? From the error log,
>>>>>>>>> it looks like the address is localhost:24817, and I believe this is the
>>>>>>>>> default I chose. Anyone see any problem with what I'm doing here? I'm
>>>>>>>>> simply trying to set up "hello world" with pulp_installer targeting a
>>>>>>>>> dedicated remote server.
>>>>>>>>> >>
>>>>>>>>> >> I applaud the pulp dev team's modularizing of the code base,
>>>>>>>>> but I would love to see more documentation on the architecture here,
>>>>>>>>> clearly illustrating all these moving parts, with links to common problems
>>>>>>>>> like I'm having, with troubleshooting advice.
>>>>>>>>> >>
>>>>>>>>> >> Here's my pulp.yml ansible playbook:
>>>>>>>>> >>
>>>>>>>>> >> ---
>>>>>>>>> >> # Playbook to provision and manage Pulp Instances for Artifact
>>>>>>>>> Management
>>>>>>>>> >>
>>>>>>>>> >> # Requires:
>>>>>>>>> >> # (
>>>>>>>>> https://pulp-installer.readthedocs.io/en/latest/#system-requirements
>>>>>>>>> )
>>>>>>>>> >> # 1. Debian Buster Machine Provisioned using Preseeded Installer
>>>>>>>>> >> # a. Really just need Debian install with:
>>>>>>>>> >> # i. sudo, openssh-server, python3
>>>>>>>>> >> # (after installing with only ssh-server and system
>>>>>>>>> utility packages selected, only need to:
>>>>>>>>> >> # su
>>>>>>>>> >> # vi /etc/apt/sources.list # remove CD Rom line, add
>>>>>>>>> buster main repo if no mirror selected during install
>>>>>>>>> >> # apt-get install sudo)
>>>>>>>>> >> # ii. update-alternatives --set editor
>>>>>>>>> `update-alternatives --list editor | grep vim`
>>>>>>>>> >> # iii. pulpadmin user with passwordless sudoer priviledges
>>>>>>>>> >> # (echo "pulpadmin ALL=(ALL) NOPASSWD: ALL" >>
>>>>>>>>> /etc/sudoers)
>>>>>>>>> >> # iv. ansible controller user has installed its ssh key
>>>>>>>>> in remote host's known_hosts
>>>>>>>>> >> # (without this you'd just need to --ask-pass and
>>>>>>>>> supply ssh passwd at stdin)
>>>>>>>>> >> # TODO: capture above in a VM Snapshot in vSphere/ESXi
>>>>>>>>> for fast reproduction.
>>>>>>>>> >> # 2. Ansible Roles Installed via Galaxy using `$ ansible-galaxy
>>>>>>>>> install -r requirements-pulp.yml`
>>>>>>>>> >> # 3. Ansible Collection Installed via Galaxy using `$
>>>>>>>>> ansible-galaxy install -r requirements-pulp.yml`
>>>>>>>>> >> #
>>>>>>>>> >> # Run like this:
>>>>>>>>> >> # ansible-playbook pulp.yml --user pulpadmin -l
>>>>>>>>> <controlled-pulp-hostname> --ask-pass --ask-vault-pass
>>>>>>>>> >>
>>>>>>>>> >> # This playbook builds upon the Engineering Services playbook
>>>>>>>>> template
>>>>>>>>> >> # Check imported playbook content before adding it here.
>>>>>>>>> >> - import_playbook: engineering-services-tmplt.yml
>>>>>>>>> >>
>>>>>>>>> >> - name: "Install packages we want on every Pulp instance"
>>>>>>>>> >> hosts: engineering_services_pulp
>>>>>>>>> >> gather_facts: false
>>>>>>>>> >> vars:
>>>>>>>>> >> apt_packages:
>>>>>>>>> >> - curl
>>>>>>>>> >> roles:
>>>>>>>>> >> - apt
>>>>>>>>> >>
>>>>>>>>> >> - name: Configure admin group
>>>>>>>>> >> become: true
>>>>>>>>> >> hosts: engineering_services_pulp
>>>>>>>>> >> gather_facts: false
>>>>>>>>> >> tasks:
>>>>>>>>> >> - name: Create admin group
>>>>>>>>> >> group:
>>>>>>>>> >> name: admin
>>>>>>>>> >>
>>>>>>>>> >> - name: Configure admin user
>>>>>>>>> >> become: true
>>>>>>>>> >> hosts: engineering_services_pulp
>>>>>>>>> >> gather_facts: false
>>>>>>>>> >> vars:
>>>>>>>>> >> # TODO: define these as inventory variable (standard for
>>>>>>>>> all machines?) so it can move out of playbook task blocks
>>>>>>>>> >> tasks:
>>>>>>>>> >> - debug: var=ansible_fqdn
>>>>>>>>> >> - name: Configure admin user account
>>>>>>>>> >> user:
>>>>>>>>> >> name: admin
>>>>>>>>> >> groups:
>>>>>>>>> >> - admin
>>>>>>>>> >>
>>>>>>>>> >> - name: Install Pulp
>>>>>>>>> >> hosts: engineering_services_pulp
>>>>>>>>> >> # gather_facts: false
>>>>>>>>> >> vars:
>>>>>>>>> >> # required by pulp_installer:
>>>>>>>>> https://pulp-installer.readthedocs.io/en/latest/#system-requirements
>>>>>>>>> >> # TODO: this is now set in ansible.cfg bc it doesn't work
>>>>>>>>> when set here or in inventory
>>>>>>>>> >> # allow_world_readable_tmpfiles: True
>>>>>>>>> >> pulp_settings:
>>>>>>>>> >> secret_key: !vault |
>>>>>>>>> >> $ANSIBLE_VAULT;1.1;AES256
>>>>>>>>> >>
>>>>>>>>> 38383631633236306565616334663761363134613835323839653962323930616639656333653865
>>>>>>>>> >>
>>>>>>>>> 3264363735643430626361383132653632316139396364370a613566396133393430663962666261
>>>>>>>>> >>
>>>>>>>>> 35356165663639613535383563366638663635326662343133353339343262646265316630616162
>>>>>>>>> >>
>>>>>>>>> 6337346131303833610a663232633339306231613738653233646466383638333934393765373034
>>>>>>>>> >>
>>>>>>>>> 63346437343834653964366666333061303634313864333031323735326134626432626535613436
>>>>>>>>> >>
>>>>>>>>> 62643731343836626436383438643862396166636263646330646332633637363765623866343733
>>>>>>>>> >> 616635326537346163646564653134386666
>>>>>>>>> >> content_origin: "http://{{ ansible_fqdn }}:8080"
>>>>>>>>> >> pulp_default_admin_password: !vault |
>>>>>>>>> >> $ANSIBLE_VAULT;1.1;AES256
>>>>>>>>> >>
>>>>>>>>> 35636365316538376363643965323035306461643239306433353665623438633535633763613662
>>>>>>>>> >>
>>>>>>>>> 6266346236393736616532636230393136303966383339310a306563323838326431386432626465
>>>>>>>>> >>
>>>>>>>>> 30316164383265303932643865323033623938656136306665356665336262613233653866386165
>>>>>>>>> >>
>>>>>>>>> 3164396261326563640a613464353364656130396333613531383864323434316533663932303766
>>>>>>>>> >> 3938
>>>>>>>>> >> pulp_content_host: "{{ ansible_fqdn }}"
>>>>>>>>> >> # pulp_content_port: 24816
>>>>>>>>> >> pulp_content_port: 8080
>>>>>>>>> >> pulp_api_host: "{{ ansible_fqdn }}"
>>>>>>>>> >> # pulp_content_port: 24817
>>>>>>>>> >> pulp_content_bind: "{{ pulp_content_host }}:{{
>>>>>>>>> pulp_content_port }}"
>>>>>>>>> >> pulp_install_plugins:
>>>>>>>>> >> # galaxy-ng: {}
>>>>>>>>> >> pulp-ansible: {}
>>>>>>>>> >> # pulp-certguard: {}
>>>>>>>>> >> pulp-container: {}
>>>>>>>>> >> # pulp-cookbook: {}
>>>>>>>>> >> pulp-deb: {}
>>>>>>>>> >> pulp-file: {}
>>>>>>>>> >> # pulp-gem: {}
>>>>>>>>> >> # pulp-maven: {}
>>>>>>>>> >> # pulp-npm: {}
>>>>>>>>> >> pulp-python: {}
>>>>>>>>> >> # pulp-rpm: {}
>>>>>>>>> >> pre_tasks:
>>>>>>>>> >> # The version string below is the highest of all those in
>>>>>>>>> roles' metadata:
>>>>>>>>> >> # "min_ansible_version". It needs to be kept manually
>>>>>>>>> up-to-date.
>>>>>>>>> >> - name: Verify Ansible meets min required version
>>>>>>>>> >> assert:
>>>>>>>>> >> that: "ansible_version.full is version_compare('2.8',
>>>>>>>>> '>=')"
>>>>>>>>> >> msg: >
>>>>>>>>> >> "You must update Ansible to at least 2.8 to use this
>>>>>>>>> version of Pulp 3 Installer."
>>>>>>>>> >> roles:
>>>>>>>>> >> - pulp_database
>>>>>>>>> >> - pulp_workers
>>>>>>>>> >> - pulp_resource_manager
>>>>>>>>> >> - pulp_webserver
>>>>>>>>> >> - pulp_content
>>>>>>>>> >> environment:
>>>>>>>>> >> DJANGO_SETTINGS_MODULE: pulpcore.app.settings
>>>>>>>>> >>
>>>>>>>>> >> Thanks for your help.
>>>>>>>>> >>
>>>>>>>>> >> Tim
>>>>>>>>> >
>>>>>>>>> > _______________________________________________
>>>>>>>>> > Pulp-list mailing list
>>>>>>>>> > Pulp-list at redhat.com
>>>>>>>>> > https://www.redhat.com/mailman/listinfo/pulp-list
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>> Pulp-list mailing list
>>>>>>>> Pulp-list at redhat.com
>>>>>>>> https://www.redhat.com/mailman/listinfo/pulp-list
>>>>>>>
>>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20200805/c2aeb13e/attachment.htm>
More information about the Pulp-list
mailing list