<html><head><title></title></head><body><div class="rte-style-maintainer rte-pre-wrap" data-color="global-default" bbg-color="default" data-bb-font-size="medium"bbg-font-size="medium" style="white-space: pre-wrap; font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth";"style="white-space: pre-wrap; font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth"; color: rgb(0, 0, 0);">Brian,<div><br></div><div>I filed a bug to track this issue "<a bbg-destination="rte:bind" data-destination="rte:bind" href="https://pulp.plan.io/issues/6449">https://pulp.plan.io/issues/6449</a>". In the meantime, is it possible to recover from this issue or we need to erase the database and reinstall?</div><div><br></div><div>Thanks</div><div><br><br><div class="rte-style-maintainer" data-color="global-default"bbg-color="default" data-bb-font-size="medium" bbg-font-size="medium" style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth";"style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth"; color: rgb(0, 0, 0);"><div><div class="bbg-rte-fold-content" data-header="From: bmbouter@redhat.com At: 04/03/20 16:45:00" data-digest="From: bmbouter@redhat.com At: 04/03/20 16:45:00" style=""><div class="bbg-rte-fold-summary">From: bmbouter@redhat.com At: 04/03/20 16:45:00</div>To: <a spellcheck="false" bbg-destination="mailto:bli111@bloomberg.net" href="mailto:bli111@bloomberg.net"> Bin Li (BLOOMBERG/ 120 PARK ) </a><br>Cc: <a spellcheck="false"bbg-destination="mailto:pulp-list@redhat.com" href="mailto:pulp-list@redhat.com"> pulp-list@redhat.com</a><br>Subject: Re: [Pulp-list] Pulp 3 waiting tasks</div><br></div><div class="rte-internet-block-wrapper" style="background: white; color: black; font-family: Arial, "BB.Proportional"; font-size: small; white-space: normal;"><div class="rte-internet-block"><blockquote><div dir="ltr"><div>So the problematic thing I see in this output is the "resource-manager | 0". This tells me that Pulp's record of the task is in postgresql (and was never run), but RQ has lost the task from the "resource-manager" queue in Redis. So the next question is how did that happen?</div><div><br></div><div>Would you be willing to file a bug and link it here so that I could try to reproduce it on our end?</div><div><br></div><div>Thanks!</div><div>Brian</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Apr 3, 2020 at 4:35 PM Bin Li (BLOOMBERG/ 120 PARK) <<a spellcheck="false"bbg-destination="mailto:rte:bind" class="" href="mailto:bli111@bloomberg.net" data-destination="mailto:rte:bind">bli111@bloomberg.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth"; white-space: pre-wrap;"><div>Brian,</div><div><br></div><div>Here is rq info output. Thanks for look into this.</div><div><br></div><div># rq --version</div><div>rq, version 1.2.2</div><div><br></div><div># rq info<div>134692@pulpmaster |â–ˆâ–ˆ 7</div><div>182536@pulpmaster | 0</div><div>134343@pulpmaster |â–ˆâ–ˆ 7</div><div>191144@pulpmaster | 0</div><div>130945@pulpmaster |â–ˆâ–ˆ 7</div><div>135922@pulpmaster | 0</div><div>182528@pulpmaster | 0</div><div>182532@pulpmaster | 0</div><div>191145@pulpmaster | 0</div><div>135796@pulpmaster | 0</div><div>191148@pulpmaster | 0</div><div>191152@pulpmaster | 0</div><div>191151@pulpmaster | 0</div><div>135306@pulpmaster | 0</div><div>135679@pulpmaster | 0</div><div>182539@pulpmaster | 0</div><div>182547@pulpmaster | 0</div><div>182530@pulpmaster | 0</div><div>134332@pulpmaster |â–ˆâ–ˆ 7</div><div>191147@pulpmaster | 0</div><div>131701@pulpmaster |â–ˆâ–ˆ 5</div><div>134330@pulpmaster |â–ˆâ–ˆ 7</div><div>134688@pulpmaster |â–ˆâ–ˆ 5</div><div>182548@pulpmaster | 0</div><div>134929@pulpmaster | 0</div><div>135180@pulpmaster | 0</div><div>135503@pulpmaster | 0</div><div>182546@pulpmaster | 0</div><div>131485@pulpmaster |â–ˆâ–ˆ 7</div><div>131269@pulpmaster |â–ˆâ–ˆ 7</div><div>32603@pulpmaster | 0</div><div>191146@pulpmaster | 0</div><div>131053@pulpmaster |â–ˆâ–ˆ 7</div><div>134339@pulpmaster |â–ˆâ–ˆ 7</div><div>134336@pulpmaster |â–ˆâ–ˆ 7</div><div>191150@pulpmaster | 0</div><div>182542@pulpmaster | 0</div><div>182540@pulpmaster | 0</div><div>32609@pulpmaster | 0</div><div>191153@pulpmaster | 0</div><div>131593@pulpmaster |â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆ 21</div><div>135051@pulpmaster | 0</div><div>134696@pulpmaster |â–ˆâ–ˆ 7</div><div>191149@pulpmaster | 0</div><div>131377@pulpmaster |â–ˆâ–ˆ 5</div><div>134694@pulpmaster |â–ˆâ–ˆ 7</div><div>134690@pulpmaster |â–ˆâ–ˆ 7</div><div>131161@pulpmaster |â–ˆâ–ˆ 5</div><div>136342@pulpmaster | 0</div><div>32626@pulpmaster | 0</div><div>131810@pulpmaster |â–ˆâ–ˆ 7</div><div>136462@pulpmaster | 0</div><div>130836@pulpmaster |â–ˆâ–ˆ 5</div><div>resource-manager | 0</div><div>54 queues, 144 jobs total</div><div><br></div><div>191146@pulpmaster (b'pulpp-ob-581' 191146): idle 191146@pulpmaster</div><div>191147@pulpmaster (b'pulpp-ob-581' 191147): idle 191147@pulpmaster</div><div>191153@pulpmaster (b'pulpp-ob-581' 191153): idle 191153@pulpmaster</div><div>resource-manager (b'pulpp-ob-581' 187238): idle resource-manager</div><div>191144@pulpmaster (b'pulpp-ob-581' 191144): idle 191144@pulpmaster</div><div>191151@pulpmaster (b'pulpp-ob-581' 191151): idle 191151@pulpmaster</div><div>191149@pulpmaster (b'pulpp-ob-581' 191149): idle 191149@pulpmaster</div><div>191145@pulpmaster (b'pulpp-ob-581' 191145): idle 191145@pulpmaster</div><div>191148@pulpmaster (b'pulpp-ob-581' 191148): idle 191148@pulpmaster</div><div>191150@pulpmaster (b'pulpp-ob-581' 191150): idle 191150@pulpmaster</div><div>191152@pulpmaster (b'pulpp-ob-581' 191152): idle 191152@pulpmaster</div><div>11 workers, 54 queues</div><div><br></div><div>Updated: 2020-04-03 16:30:23.373244</div><div><br><div style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth";"><div><div><div>From: <a spellcheck="false" bbg-destination="mailto:rte:bind"class="" href="mailto:bmbouter@redhat.com" data-destination="mailto:rte:bind">bmbouter@redhat.com</a> At: 04/03/20 16:23:22</div>To: <a spellcheck="false" bbg-destination="mailto:bli111@bloomberg.net" class="" href="mailto:bli111@bloomberg.net" data-destination="mailto:bli111@bloomberg.net"> Bin Li (BLOOMBERG/ 120 PARK ) </a><br>Cc: <a spellcheck="false"bbg-destination="mailto:pulp-list@redhat.com" class="" href="mailto:pulp-list@redhat.com" data-destination="mailto:pulp-list@redhat.com"> pulp-list@redhat.com</a><br>Subject: Re: [Pulp-list] Pulp 3 waiting tasks</div><br></div><div style="background: none 0% 0% repeat scroll white; color: black; font-family: Arial, "BB.Proportional"; font-size: small; white-space: normal;"><div><blockquote><div dir="ltr"><div>Since the task that is stalled has a "worker" unassigned it tells me it has not traveled through the resource-manager yet. All tests in Pulp3 (currently) go through the resource-manager. I can see from your ps output there is 1 resource-manager running (which is good), and the status API agrees with that (also good).</div><div><br></div><div>So what does RQ thing the situation is? Can you paste the output of `rq info` please?</div><div><br></div><div>Also what version of RQ are do you have installed?</div><div><br></div><div>Thanks,</div><div>Brian</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Apr 3, 2020 at 9:39 AM Bin Li (BLOOMBERG/ 120 PARK) <<a spellcheck="false"bbg-destination="mailto:rte:bind" class="" href="mailto:bli111@bloomberg.net" data-destination="mailto:rte:bind">bli111@bloomberg.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth"; white-space: pre-wrap;">Here is the more info. Log is very big. I will send you shortly.<div><br><div> # ./sget status<div>{</div><div> "database_connection": {</div><div> "connected": true</div><div> },</div><div> "online_content_apps": [</div><div> {</div><div> "last_heartbeat": "2020-04-03T13:10:30.135954Z",</div><div> "name": "187254@pulpp-ob-581"</div><div> },</div><div> {</div><div> "last_heartbeat": "2020-04-03T13:10:30.132849Z",</div><div> "name": "187257@pulpp-ob-581"</div><div> }</div><div> ],</div><div> "online_workers": [</div><div> {</div><div> "last_heartbeat": "2020-04-03T13:10:29.898377Z",</div><div> "name": "<a spellcheck="false" bbg-destination="mailto:rte:bind" class="" href="mailto:191147@pulpp-ob-581.bloomberg.com" data-destination="mailto:rte:bind">191147@pulpp-ob-581.bloomberg.com</a>",</div><div> "pulp_created": "2020-04-02T13:36:11.796937Z",</div><div> "pulp_href": "/pulp/api/v3/workers/268261b9-f46d-4d37-ab47-0b50ca382637/"</div><div> },</div><div> {</div><div> "last_heartbeat": "2020-04-03T13:10:19.087502Z",</div><div> "name": "<a spellcheck="false" bbg-destination="mailto:rte:bind" class="" href="mailto:191150@pulpp-ob-581.bloomberg.com" data-destination="mailto:rte:bind">191150@pulpp-ob-581.bloomberg.com</a>",</div><div> "pulp_created": "2020-04-02T13:36:11.807418Z",</div><div> "pulp_href": "/pulp/api/v3/workers/4fb4d87c-2c3c-4f64-b6f3-e05d9aaf6fc0/"</div><div> },</div><div> {</div><div> "last_heartbeat": "2020-04-03T13:10:29.498852Z",</div><div> "name": "<a spellcheck="false" bbg-destination="mailto:rte:bind" class="" href="mailto:191146@pulpp-ob-581.bloomberg.com" data-destination="mailto:rte:bind">191146@pulpp-ob-581.bloomberg.com</a>",</div><div> "pulp_created": "2020-04-02T13:36:11.810402Z",</div><div> "pulp_href": "/pulp/api/v3/workers/7b15b6bd-1437-47b8-9832-0b44b326e0fa/"</div><div> },</div><div> {</div><div> "last_heartbeat": "2020-04-03T13:10:29.798941Z",</div><div> "name": "<a spellcheck="false" bbg-destination="mailto:rte:bind" class="" href="mailto:191149@pulpp-ob-581.bloomberg.com" data-destination="mailto:rte:bind">191149@pulpp-ob-581.bloomberg.com</a>",</div><div> "pulp_created": "2020-04-02T13:36:11.817391Z",</div><div> "pulp_href": "/pulp/api/v3/workers/62523740-e109-4828-bcbb-e8459c0944c5/"</div><div> },</div><div> {</div><div> "last_heartbeat": "2020-04-03T13:10:29.598962Z",</div><div> "name": "<a spellcheck="false" bbg-destination="mailto:rte:bind" class="" href="mailto:191144@pulpp-ob-581.bloomberg.com" data-destination="mailto:rte:bind">191144@pulpp-ob-581.bloomberg.com</a>",</div><div> "pulp_created": "2020-04-02T13:36:11.818322Z",</div><div> "pulp_href": "/pulp/api/v3/workers/02e33d62-797d-4797-8fdc-b999efc8cd12/"</div><div> },</div><div> {</div><div> "last_heartbeat": "2020-04-03T13:10:16.685771Z",</div><div> "name": "<a spellcheck="false" bbg-destination="mailto:rte:bind" class="" href="mailto:191153@pulpp-ob-581.bloomberg.com" data-destination="mailto:rte:bind">191153@pulpp-ob-581.bloomberg.com</a>",</div><div> "pulp_created": "2020-04-02T13:36:11.831154Z",</div><div> "pulp_href": "/pulp/api/v3/workers/23e2a484-a877-4083-bcd4-38a0e89fcb49/"</div><div> },</div><div> {</div><div> "last_heartbeat": "2020-04-03T13:10:18.487964Z",</div><div> "name": "<a spellcheck="false" bbg-destination="mailto:rte:bind" class="" href="mailto:191145@pulpp-ob-581.bloomberg.com" data-destination="mailto:rte:bind">191145@pulpp-ob-581.bloomberg.com</a>",</div><div> "pulp_created": "2020-04-02T13:36:11.869871Z",</div><div> "pulp_href": "/pulp/api/v3/workers/9e63708f-bbc0-473d-8de1-8788a1c91f51/"</div><div> },</div><div> {</div><div> "last_heartbeat": "2020-04-03T13:10:29.898354Z",</div><div> "name": "<a spellcheck="false" bbg-destination="mailto:rte:bind" class="" href="mailto:191151@pulpp-ob-581.bloomberg.com" data-destination="mailto:rte:bind">191151@pulpp-ob-581.bloomberg.com</a>",</div><div> "pulp_created": "2020-04-02T13:36:11.880995Z",</div><div> "pulp_href": "/pulp/api/v3/workers/ddd49126-5531-471a-bea1-3aab07bcf8b4/"</div><div> },</div><div> {</div><div> "last_heartbeat": "2020-04-03T13:10:18.887949Z",</div><div> "name": "<a spellcheck="false" bbg-destination="mailto:rte:bind" class="" href="mailto:191148@pulpp-ob-581.bloomberg.com" data-destination="mailto:rte:bind">191148@pulpp-ob-581.bloomberg.com</a>",</div><div> "pulp_created": "2020-04-02T13:36:11.893280Z",</div><div> "pulp_href": "/pulp/api/v3/workers/2ef1e562-845f-4ae7-8007-9b7db8cf73a0/"</div><div> },</div><div> {</div><div> "last_heartbeat": "2020-04-03T13:10:29.798877Z",</div><div> "name": "<a spellcheck="false" bbg-destination="mailto:rte:bind" class="" href="mailto:191152@pulpp-ob-581.bloomberg.com" data-destination="mailto:rte:bind">191152@pulpp-ob-581.bloomberg.com</a>",</div><div> "pulp_created": "2020-04-02T13:36:11.917095Z",</div><div> "pulp_href": "/pulp/api/v3/workers/6e2cf918-af8e-4c5d-bc8f-bef3d3a83dca/"</div><div> },</div><div> {</div><div> "last_heartbeat": "2020-04-03T13:10:15.684710Z",</div><div> "name": "resource-manager",</div><div> "pulp_created": "2020-01-23T18:24:49.246717Z",</div><div> "pulp_href": "/pulp/api/v3/workers/d46e4da0-9735-445b-a502-2aff7ce13ef7/"</div><div> }</div><div> ],</div><div> "redis_connection": {</div><div> "connected": true</div><div> },</div><div> "storage": {</div><div> "free": 32543019880448,</div><div> "total": 33521607376896,</div><div> "used": 978587496448</div><div> },</div><div> "versions": [</div><div> {</div><div> "component": "pulpcore",</div><div> "version": "3.2.1"</div><div> },</div><div> {</div><div> "component": "pulp_rpm",</div><div> "version": "3.2.0"</div><div> },</div><div> {</div><div> "component": "pulp_file",</div><div> "version": "0.2.0"</div><div> }</div><div> ]</div><div><br></div><div><br></div><div># ps -awfux |grep pulp</div><div>root 180078 0.0 0.0 107992 616 pts/1 S+ Apr02 0:00 | \_ tail -f /var/log/pulp/pulp-config.log</div><div>root 184836 0.0 0.0 124448 2044 pts/2 S+ Apr02 0:00 | \_ vi bbpulp3.py</div><div>root 43270 0.0 0.0 112708 984 pts/3 S+ 09:11 0:00 \_ grep --color=auto pulp</div><div>pulp 187224 0.0 0.0 228600 19188 ? Ss Apr02 0:04 /opt/utils/venv/pulp/3.7.3/bin/python3 /opt/utils/venv/pulp/3.7.3/bin/gunicorn pulpcore.app.wsgi:application --bind <a spellcheck="false"class="">127.0.0.1:24817</a> --access-logfile -</div><div>pulp 187251 1.4 0.0 528708 109752 ? S Apr02 20:48 \_ /opt/utils/venv/pulp/3.7.3/bin/python3 /opt/utils/venv/pulp/3.7.3/bin/gunicorn pulpcore.app.wsgi:application --bind <a spellcheck="false"class="">127.0.0.1:24817</a> --access-logfile -</div><div>pulp 187231 0.0 0.0 269476 27976 ? Ss Apr02 0:05 /opt/utils/venv/pulp/3.7.3/bin/python3 /opt/utils/venv/pulp/3.7.3/bin/gunicorn pulpcore.content:server --bind <a spellcheck="false"class="">127.0.0.1:24816</a> --worker-class aiohttp.GunicornWebWorker -w 2 --access-logfile -</div><div>pulp 187254 0.0 0.0 485860 68592 ? S Apr02 0:18 \_ /opt/utils/venv/pulp/3.7.3/bin/python3 /opt/utils/venv/pulp/3.7.3/bin/gunicorn pulpcore.content:server --bind <a spellcheck="false"class="">127.0.0.1:24816</a> --worker-class aiohttp.GunicornWebWorker -w 2 --access-logfile -</div><div>pulp 187257 0.0 0.0 486132 68604 ? S Apr02 0:19 \_ /opt/utils/venv/pulp/3.7.3/bin/python3 /opt/utils/venv/pulp/3.7.3/bin/gunicorn pulpcore.content:server --bind <a spellcheck="false"class="">127.0.0.1:24816</a> --worker-class aiohttp.GunicornWebWorker -w 2 --access-logfile -</div><div>pulp 187238 0.0 0.0 486428 71128 ? Ss Apr02 1:20 /opt/utils/venv/pulp/3.7.3/bin/python3 /opt/utils/venv/pulp/3.7.3/bin/rq worker -w pulpcore.tasking.worker.PulpWorker -n resource-manager --pid=/var/run/pulpcore-resource-manager/resource-manager.pid -c pulpcore.rqconfig --disable-job-desc-logging</div><div>pulp 191144 0.0 0.0 486392 71064 ? Ss Apr02 0:51 /opt/utils/venv/pulp/3.7.3/bin/python3 /opt/utils/venv/pulp/3.7.3/bin/rq worker -w pulpcore.tasking.worker.PulpWorker --pid=/var/run/pulpcore-worker-1/reserved-resource-worker-1.pid -c pulpcore.rqconfig --disable-job-desc-logging</div><div>pulp 191145 0.0 0.0 486404 71064 ? Ss Apr02 0:50 /opt/utils/venv/pulp/3.7.3/bin/python3 /opt/utils/venv/pulp/3.7.3/bin/rq worker -w pulpcore.tasking.worker.PulpWorker --pid=/var/run/pulpcore-worker-2/reserved-resource-worker-2.pid -c pulpcore.rqconfig --disable-job-desc-logging</div><div>pulp 191146 0.0 0.0 486404 71044 ? Ss Apr02 0:50 /opt/utils/venv/pulp/3.7.3/bin/python3 /opt/utils/venv/pulp/3.7.3/bin/rq worker -w pulpcore.tasking.worker.PulpWorker --pid=/var/run/pulpcore-worker-3/reserved-resource-worker-3.pid -c pulpcore.rqconfig --disable-job-desc-logging</div><div>pulp 191147 0.0 0.0 486404 71036 ? Ss Apr02 0:52 /opt/utils/venv/pulp/3.7.3/bin/python3 /opt/utils/venv/pulp/3.7.3/bin/rq worker -w pulpcore.tasking.worker.PulpWorker --pid=/var/run/pulpcore-worker-4/reserved-resource-worker-4.pid -c pulpcore.rqconfig --disable-job-desc-logging</div><div>pulp 191148 0.0 0.0 486164 71056 ? Ss Apr02 0:51 /opt/utils/venv/pulp/3.7.3/bin/python3 /opt/utils/venv/pulp/3.7.3/bin/rq worker -w pulpcore.tasking.worker.PulpWorker --pid=/var/run/pulpcore-worker-5/reserved-resource-worker-5.pid -c pulpcore.rqconfig --disable-job-desc-logging</div><div>pulp 191149 0.0 0.0 486168 71060 ? Ss Apr02 0:52 /opt/utils/venv/pulp/3.7.3/bin/python3 /opt/utils/venv/pulp/3.7.3/bin/rq worker -w pulpcore.tasking.worker.PulpWorker --pid=/var/run/pulpcore-worker-6/reserved-resource-worker-6.pid -c pulpcore.rqconfig --disable-job-desc-logging</div><div>pulp 191150 0.0 0.0 486148 71040 ? Ss Apr02 0:50 /opt/utils/venv/pulp/3.7.3/bin/python3 /opt/utils/venv/pulp/3.7.3/bin/rq worker -w pulpcore.tasking.worker.PulpWorker --pid=/var/run/pulpcore-worker-7/reserved-resource-worker-7.pid -c pulpcore.rqconfig --disable-job-desc-logging</div><div>pulp 191151 0.0 0.0 486400 71060 ? Ss Apr02 0:51 /opt/utils/venv/pulp/3.7.3/bin/python3 /opt/utils/venv/pulp/3.7.3/bin/rq worker -w pulpcore.tasking.worker.PulpWorker --pid=/var/run/pulpcore-worker-8/reserved-resource-worker-8.pid -c pulpcore.rqconfig --disable-job-desc-logging</div><div>pulp 191152 0.0 0.0 486164 71044 ? Ss Apr02 0:52 /opt/utils/venv/pulp/3.7.3/bin/python3 /opt/utils/venv/pulp/3.7.3/bin/rq worker -w pulpcore.tasking.worker.PulpWorker --pid=/var/run/pulpcore-worker-9/reserved-resource-worker-9.pid -c pulpcore.rqconfig --disable-job-desc-logging</div><div>pulp 191153 0.0 0.0 486392 71068 ? Ss Apr02 0:52 /opt/utils/venv/pulp/3.7.3/bin/python3 /opt/utils/venv/pulp/3.7.3/bin/rq worker -w pulpcore.tasking.worker.PulpWorker --pid=/var/run/pulpcore-worker-10/reserved-resource-worker-10.pid -c pulpcore.rqconfig --disable-job-desc-logging</div><div><br><div style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth";"><div><div><div>From: <a spellcheck="false" bbg-destination="mailto:rte:bind"class="" href="mailto:bmbouter@redhat.com" data-destination="mailto:rte:bind">bmbouter@redhat.com</a> At: 04/03/20 09:05:47</div>To: <a spellcheck="false" bbg-destination="mailto:bli111@bloomberg.net" class="" href="mailto:bli111@bloomberg.net" data-destination="mailto:bli111@bloomberg.net"> Bin Li (BLOOMBERG/ 120 PARK ) </a><br>Cc: <a spellcheck="false"bbg-destination="mailto:pulp-list@redhat.com" class="" href="mailto:pulp-list@redhat.com" data-destination="mailto:pulp-list@redhat.com"> pulp-list@redhat.com</a><br>Subject: Re: [Pulp-list] Pulp 3 waiting tasks</div><br></div><div style="background: none 0% 0% repeat scroll white; color: black; font-family: Arial, "BB.Proportional"; font-size: small; white-space: normal;"><div><blockquote><div dir="ltr"><div>While you are experiencing the issue, can you capture the status API output?</div><div><br></div><div>Also can you paste an output of the workers on that system with `ps -awfux | grep pulp`.</div><div><br></div><div>Also do you see any errors in the log? Could you share a copy of the log?<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Apr 3, 2020 at 9:01 AM Bin Li (BLOOMBERG/ 120 PARK) <<a spellcheck="false"bbg-destination="mailto:rte:bind" class="" href="mailto:bli111@bloomberg.net" data-destination="mailto:rte:bind">bli111@bloomberg.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth"; white-space: pre-wrap;">We have been seeing many waiting tasks. They seem to be stuck forever. <div>e.g.<div>pulpp-ob-581 /home/bli4/pulp3-script # ./get /pulp/api/v3/tasks/14b76b27-9f34-4297-88ed-5ec13cbe5e50/</div><div>HTTP/1.1 200 OK</div><div>Allow: GET, PATCH, DELETE, HEAD, OPTIONS</div><div>Connection: keep-alive</div><div>Content-Length: 323</div><div>Content-Type: application/json</div><div>Date: Fri, 03 Apr 2020 12:56:02 GMT</div><div>Server: nginx/1.16.1</div><div>Vary: Accept, Cookie</div><div>X-Frame-Options: SAMEORIGIN</div><div><br></div><div>{</div><div> "created_resources": [], </div><div> "error": null, </div><div> "finished_at": null, </div><div> "name": "pulpcore.app.tasks.base.general_update", </div><div> "progress_reports": [], </div><div> "pulp_created": "2020-04-02T13:00:14.881212Z", </div><div> "pulp_href": "/pulp/api/v3/tasks/14b76b27-9f34-4297-88ed-5ec13cbe5e50/", </div><div> "reserved_resources_record": [], </div><div> "started_at": null, </div><div> "state": "waiting", </div><div> "worker": null</div><div>}</div><div><br></div><div>What could be the reason for these stuck waiting tasks? How should we troubleshot the issue? </div></div></div>_______________________________________________<br>Pulp-list mailing list<br><a spellcheck="false"bbg-destination="mailto:rte:bind" class="" href="mailto:Pulp-list@redhat.com" data-destination="mailto:rte:bind">Pulp-list@redhat.com</a><br><a spellcheck="false"bbg-destination="rte:bind" class="" href="https://www.redhat.com/mailman/listinfo/pulp-list"data-destination="rte:bind">https://www.redhat.com/mailman/listinfo/pulp-list</a></blockquote></div></blockquote><br></div></div></div></div></div></div></div></blockquote></div></blockquote><br></div></div></div></div></div></div></blockquote></div></blockquote><br></div></div></div></div></div></body></html>