<div dir="ltr"><a class="gmail_plusreply" id="plusReplyChip-1" href="mailto:dellweg@atix.de" tabindex="-1">@Matthias Dellweg</a> Would you be willing to file an issue?<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jan 21, 2020 at 4:51 PM Daniel Alley <<a href="mailto:dalley@redhat.com">dalley@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>I'm not sure why I can't reproduce the issue on either of my machines but I think it would be proper to re-pin redis until we get it figured out. PR here: <a href="https://github.com/pulp/pulpcore/pull/513" target="_blank">https://github.com/pulp/pulpcore/pull/513</a></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jan 21, 2020 at 4:37 PM Simon Baatz <<a href="mailto:gmbnomis@gmail.com" target="_blank">gmbnomis@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I can reproduce the problem described by Matthias with pulp_cookbook's<br>
tests almost instantly (I haven't seen a single complete run of the<br>
functional tests).<br>
<br>
Box is pulp3-source-fedora31, redis-py 3.3.1, redis 5.0.7.<br>
I don't see these hangs anymore after downgrading redis-py to 3.1.0.<br>
<br>
Vagrant's VM provider is libvirt with an additional<br>
`domain.volume_cache = 'unsafe'` setting. Write I/O is probably<br>
extremely fast in this setup (VM can't force flush to disk)<br>
<br>
<br>
On Tue, Jan 21, 2020 at 12:22:33PM -0500, Daniel Alley wrote:<br>
> 4 runs on Fedora 31 has not triggered it either. This is using the<br>
> latest version of redis-py, 3.3.11, and redis 5.0.7.<br>
> <br>
> On Tue, Jan 21, 2020 at 11:16 AM Daniel Alley <[1]<a href="mailto:dalley@redhat.com" target="_blank">dalley@redhat.com</a>><br>
> wrote:<br>
> <br>
> So far I've run it 4 times back to back on a CentOS 7 box and not had<br>
> any lockups. I'll try Fedora<br>
> <br>
> On Tue, Jan 21, 2020 at 9:36 AM Matthias Dellweg <[2]<a href="mailto:dellweg@atix.de" target="_blank">dellweg@atix.de</a>><br>
> wrote:<br>
> <br>
> @Daniel<br>
> It is happening with fresh installations (pulplift with libvirt) on<br>
> both boxes i have tried (pulp3-source-debian10 and<br>
> pulp3-source-fedora30). To test, i run `prestart; django-admin test<br>
> pulpcore` multiple times until it hangs (though i have seen it hang<br>
> on<br>
> the first try). There is no modification that i know of. Also, up<br>
> until<br>
> now, i seem to be able to reproduce reliably, while downgrading<br>
> (pinning) redis-py completely solves the issue.<br>
> On Tue, 21 Jan 2020 09:16:43 -0500<br>
> Daniel Alley <[3]<a href="mailto:dalley@redhat.com" target="_blank">dalley@redhat.com</a>> wrote:<br>
> > @Matthias Dellweg <[4]<a href="mailto:dellweg@atix.de" target="_blank">dellweg@atix.de</a>><br>
> > did you restart Pulp after upgrading redis-py? Are you seeing<br>
> this on<br>
> > fresh boxes, or does it require some modification to reproduce? I<br>
> > don't think any of us have experienced this thus far.<br>
> ><br>
> > The only thing I can think of is that maybe when the worker<br>
> process<br>
> > forks it ends up using a different version of redis-py than the<br>
> > parent worker.<br>
> ><br>
> > On Tue, Jan 21, 2020 at 4:49 AM Matthias Dellweg<br>
> <[5]<a href="mailto:dellweg@atix.de" target="_blank">dellweg@atix.de</a>><br>
> > wrote:<br>
> ><br>
> > > [@ Brian:<br>
> > ><br>
> > ><br>
> [6]<a href="https://github.com/pulp/pulpcore/commit/e36e7b5f0eccc176a6e6298df" rel="noreferrer" target="_blank">https://github.com/pulp/pulpcore/commit/e36e7b5f0eccc176a6e6298df</a><br>
> 29293b014f4710c<br>
> > > ]<br>
> > ><br>
> > > Hi Daniel,<br>
> > > thank you for looking into this.<br>
> > > What i am seeing is:<br>
> > ><br>
> > > Jan 21 09:23:10 [7]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a><br>
> > > gunicorn[23274]: 127.0.0.1 - admin [21/Jan/2020:09:23:10 +0000]<br>
> > > "PATCH<br>
> > ><br>
> /pulp/api/v3/repositories/file/file/3a31ed13-585d-4a36-8398-7df40560<br>
> ffa4/<br>
> > > HTTP/1.1" 202 67 "-" "python-requests/2.22.0" Jan 21 09:23:10<br>
> > > [8]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a> rq[23269]: pulp:<br>
> > > rq.worker:INFO:<br>
> [9]<a href="mailto:23269@pulp3-source-fedora30.anubis.example.com" target="_blank">23269@pulp3-source-fedora30.anubis.example.com</a>:<br>
> > > 3a776d4d-ff0c-4b44-afae-08dc7c6cc415 Jan 21 09:23:10<br>
> > > [10]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a> rq[23270]: pulp:<br>
> > > rq.worker:INFO: resource-manager: Job OK<br>
> > > (aa22aed0-0363-45ae-8254-4ab14893a4a1) Jan 21 09:23:10<br>
> > > [11]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a> rq[23269]: pulp:<br>
> > > rq.worker:ERROR: Worker<br>
> > > [12]<a href="mailto:rq%3Aworker%3A23269@pulp3-source-fedora30.anubis.example.com" target="_blank">rq:worker:23269@pulp3-source-fedora30.anubis.example.com</a>:<br>
> found an<br>
> > > unhandled exception, quitting... Jan 21 09:23:10<br>
> > > [13]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a> rq[23269]:<br>
> Traceback (most<br>
> > > recent call last): Jan 21 09:23:10<br>
> > > [14]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a> rq[23269]: File<br>
> > ><br>
> "/usr/local/lib/pulp/lib64/python3.7/site-packages/rq/worker.py",<br>
> > > line 782, in prepare_job_execution Jan 21 09:23:10<br>
> > > [15]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a> rq[23269]:<br>
> > > pipeline.execute() Jan 21 09:23:10<br>
> > > [16]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a> rq[23269]: File<br>
> > ><br>
> "/usr/local/lib/pulp/lib64/python3.7/site-packages/redis/client.py",<br>
> > > line 3707, in execute Jan 21 09:23:10<br>
> > > [17]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a> rq[23269]:<br>
> > > self.reset() Jan 21 09:23:10<br>
> > > [18]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a> rq[23269]: File<br>
> > ><br>
> "/usr/local/lib/pulp/lib64/python3.7/site-packages/redis/client.py",<br>
> > > line 3476, in reset Jan 21 09:23:10<br>
> > > [19]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a> rq[23269]:<br>
> > > self.connection_pool.release(self.connection) Jan 21 09:23:10<br>
> > > [20]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a> rq[23269]: File<br>
> > ><br>
> "/usr/local/lib/pulp/lib64/python3.7/site-packages/redis/connection.<br>
> py",<br>
> > > line 1114, in release Jan 21 09:23:10<br>
> > > [21]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a> rq[23269]:<br>
> > > self._in_use_connections.remove(connection) Jan 21 09:23:10<br>
> > > [22]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a> rq[23269]:<br>
> KeyError:<br>
> > > Connection<host=localhost,port=6379,db=0> Jan 21 09:23:10<br>
> > > [23]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a> rq[23269]: During<br>
> handling<br>
> > > of the above exception, another exception occurred: Jan 21<br>
> 09:23:10<br>
> > > [24]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a> rq[23269]:<br>
> Traceback (most<br>
> > > recent call last): Jan 21 09:23:10<br>
> > > [25]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a> rq[23269]: File<br>
> > ><br>
> "/usr/local/lib/pulp/lib64/python3.7/site-packages/rq/worker.py",<br>
> > > line 515, in work Jan 21 09:23:10<br>
> > > [26]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a> rq[23269]:<br>
> > > self.execute_job(job, queue) Jan 21 09:23:10<br>
> > > [27]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a> rq[23269]: File<br>
> > > "/home/vagrant/devel/pulpcore/pulpcore/tasking/worker.py", line<br>
> 72,<br>
> > > in execute_job Jan 21 09:23:10<br>
> > > [28]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a> rq[23269]:<br>
> > > super().execute_job(*args, **kwargs) Jan 21 09:23:10<br>
> > > [29]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a> rq[23269]: File<br>
> > ><br>
> "/usr/local/lib/pulp/lib64/python3.7/site-packages/rq/worker.py",<br>
> > > line 727, in execute_job Jan 21 09:23:10<br>
> > > [30]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a> rq[23269]:<br>
> > > self.fork_work_horse(job, queue) Jan 21 09:23:10<br>
> > > [31]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a> rq[23269]: File<br>
> > ><br>
> "/usr/local/lib/pulp/lib64/python3.7/site-packages/rq/worker.py",<br>
> > > line 667, in fork_work_horse Jan 21 09:23:10<br>
> > > [32]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a> rq[23269]:<br>
> > > self.main_work_horse(job, queue) Jan 21 09:23:10<br>
> > > [33]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a> rq[23269]: File<br>
> > ><br>
> "/usr/local/lib/pulp/lib64/python3.7/site-packages/rq/worker.py",<br>
> > > line 744, in main_work_horse Jan 21 09:23:10<br>
> > > [34]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a> rq[23269]:<br>
> raise e Jan<br>
> > > 21 09:23:10 [35]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a><br>
> rq[23269]:<br>
> > > File<br>
> > ><br>
> "/usr/local/lib/pulp/lib64/python3.7/site-packages/rq/worker.py",<br>
> > > line 741, in main_work_horse Jan 21 09:23:10<br>
> > > [36]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a> rq[23269]:<br>
> > > self.perform_job(job, queue) Jan 21 09:23:10<br>
> > > [37]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a> rq[23269]: File<br>
> > > "/home/vagrant/devel/pulpcore/pulpcore/tasking/worker.py", line<br>
> > > 103, in perform_job Jan 21 09:23:10<br>
> > > [38]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a> rq[23269]:<br>
> return<br>
> > > super().perform_job(job, queue) Jan 21 09:23:10<br>
> > > [39]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a> rq[23269]: File<br>
> > ><br>
> "/usr/local/lib/pulp/lib64/python3.7/site-packages/rq/worker.py",<br>
> > > line 866, in perform_job Jan 21 09:23:10<br>
> > > [40]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a> rq[23269]:<br>
> > > self.prepare_job_execution(job, heartbeat_ttl) Jan 21 09:23:10<br>
> > > [41]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a> rq[23269]: File<br>
> > ><br>
> "/usr/local/lib/pulp/lib64/python3.7/site-packages/rq/worker.py",<br>
> > > line 782, in prepare_job_execution Jan 21 09:23:10<br>
> > > [42]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a> rq[23269]:<br>
> > > pipeline.execute() Jan 21 09:23:10<br>
> > > [43]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a> rq[23269]: File<br>
> > ><br>
> "/usr/local/lib/pulp/lib64/python3.7/site-packages/redis/client.py",<br>
> > > line 3445, in __exit__ Jan 21 09:23:10<br>
> > > [44]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a> rq[23269]:<br>
> > > self.reset() Jan 21 09:23:10<br>
> > > [45]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a> rq[23269]: File<br>
> > ><br>
> "/usr/local/lib/pulp/lib64/python3.7/site-packages/redis/client.py",<br>
> > > line 3476, in reset Jan 21 09:23:10<br>
> > > [46]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a> rq[23269]:<br>
> > > self.connection_pool.release(self.connection) Jan 21 09:23:10<br>
> > > [47]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a> rq[23269]: File<br>
> > ><br>
> "/usr/local/lib/pulp/lib64/python3.7/site-packages/redis/connection.<br>
> py",<br>
> > > line 1114, in release Jan 21 09:23:10<br>
> > > [48]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a> rq[23269]:<br>
> > > self._in_use_connections.remove(connection) Jan 21 09:23:10<br>
> > > [49]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a> rq[23269]:<br>
> KeyError:<br>
> > > Connection<host=localhost,port=6379,db=0> Jan 21 09:23:10<br>
> > > [50]<a href="http://pulp3-source-fedora30.anubis.example.com" rel="noreferrer" target="_blank">pulp3-source-fedora30.anubis.example.com</a> gunicorn[23274]:<br>
> 127.0.0.1<br>
> > > - admin [21/Jan/2020:09:23:10 +0000] "GET<br>
> > > /pulp/api/v3/tasks/3a776d4d-ff0c-4b44-afae-08dc7c6cc415/<br>
> HTTP/1.1"<br>
> > > 200 477 "-" "python-requests/2.22.0"<br>
> > ><br>
> > > and that last line repeats forever.<br>
> > ><br>
> > > On Fri, 17 Jan 2020 09:09:01 -0500<br>
> > > Daniel Alley <[51]<a href="mailto:dalley@redhat.com" target="_blank">dalley@redhat.com</a>> wrote:<br>
> > ><br>
> > > > Different issue perhaps? Are you seeing anything in the logs<br>
> that<br>
> > > > looks like this? [52]<a href="https://github.com/rq/rq/issues/1044" rel="noreferrer" target="_blank">https://github.com/rq/rq/issues/1044</a><br>
> > > ><br>
> > > > On Fri, Jan 17, 2020 at 9:06 AM Daniel Alley<br>
> <[53]<a href="mailto:dalley@redhat.com" target="_blank">dalley@redhat.com</a>><br>
> > > > wrote:<br>
> > > ><br>
> > > > > Strange, I'm pretty sure an issue like this is why we pinned<br>
> > > > > originally, but upstream said that that particular issue was<br>
> > > > > (supposedly) fixed.<br>
> > > > ><br>
> > > > ><br>
> > ><br>
> [54]<a href="https://github.com/andymccurdy/redis-py/issues/1136#issuecomment" rel="noreferrer" target="_blank">https://github.com/andymccurdy/redis-py/issues/1136#issuecomment</a><br>
> -571168161<br>
> > ><br>
> > > > ><br>
> > > > > On Fri, Jan 17, 2020 at 4:15 AM Matthias Dellweg<br>
> > > > > <[55]<a href="mailto:dellweg@atix.de" target="_blank">dellweg@atix.de</a>> wrote:<br>
> > > > >> Hello all,<br>
> > > > >> I believe i have found a new incarnation of hanging tasks<br>
> > > > >> (tm). This time it is pulp3 and as hard to nail down as<br>
> ever.<br>
> > > > >> I think, it is introduced by<br>
> > > > >> e36e7b5f0eccc176a6e6298df29293b014f4710c. Where the<br>
> dependency<br>
> > > > >> on redis was dropped with the result that 3.3.smth instead<br>
> of<br>
> > > > >> 3.1.smth was installed.<br>
> > > > >><br>
> > > > >> Before filing an issue, is there anyone out there to share<br>
> that<br>
> > > > >> experience?<br>
> > > > >><br>
> > > > >> Also as a thought protocol of how to reproduce:<br>
> > > > >> I have seen tasks hanging in both "waiting" and "running"<br>
> state<br>
> > > > >> when using the command `prestart; django-admin test<br>
> pulp_deb`<br>
> > > > >> or `<...> test pulpcore`. All the tasks I have seen were<br>
> > > > >> `sync`, `general_create` or `general_delete` and looked<br>
> like<br>
> > > > >> they never started to do anything for real. To have<br>
> consistent<br>
> > > > >> results, i had the impression that i needed to rebuild the<br>
> > > > >> vagrant boxes for every bisecting step. Also updating the<br>
> > > > >> python-redis package on a box that worked, produced a<br>
> hanging<br>
> > > > >> task in the next run.<br>
> > > > >><br>
> > > > >> Have a good day,<br>
> > > > >> Matthias<br>
> <br>
> References<br>
> <br>
> 1. mailto:<a href="mailto:dalley@redhat.com" target="_blank">dalley@redhat.com</a><br>
> 2. mailto:<a href="mailto:dellweg@atix.de" target="_blank">dellweg@atix.de</a><br>
> 3. mailto:<a href="mailto:dalley@redhat.com" target="_blank">dalley@redhat.com</a><br>
> 4. mailto:<a href="mailto:dellweg@atix.de" target="_blank">dellweg@atix.de</a><br>
> 5. mailto:<a href="mailto:dellweg@atix.de" target="_blank">dellweg@atix.de</a><br>
> 6. <a href="https://github.com/pulp/pulpcore/commit/e36e7b5f0eccc176a6e6298df29293b014f4710c" rel="noreferrer" target="_blank">https://github.com/pulp/pulpcore/commit/e36e7b5f0eccc176a6e6298df29293b014f4710c</a><br>
> 7. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 8. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 9. mailto:<a href="mailto:23269@pulp3-source-fedora30.anubis.example.com" target="_blank">23269@pulp3-source-fedora30.anubis.example.com</a><br>
> 10. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 11. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 12. mailto:<a href="mailto:rq" target="_blank">rq</a>:<a href="mailto:worker%3A23269@pulp3-source-fedora30.anubis.example.com" target="_blank">worker:23269@pulp3-source-fedora30.anubis.example.com</a><br>
> 13. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 14. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 15. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 16. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 17. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 18. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 19. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 20. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 21. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 22. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 23. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 24. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 25. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 26. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 27. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 28. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 29. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 30. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 31. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 32. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 33. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 34. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 35. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 36. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 37. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 38. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 39. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 40. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 41. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 42. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 43. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 44. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 45. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 46. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 47. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 48. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 49. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 50. <a href="http://pulp3-source-fedora30.anubis.example.com/" rel="noreferrer" target="_blank">http://pulp3-source-fedora30.anubis.example.com/</a><br>
> 51. mailto:<a href="mailto:dalley@redhat.com" target="_blank">dalley@redhat.com</a><br>
> 52. <a href="https://github.com/rq/rq/issues/1044" rel="noreferrer" target="_blank">https://github.com/rq/rq/issues/1044</a><br>
> 53. mailto:<a href="mailto:dalley@redhat.com" target="_blank">dalley@redhat.com</a><br>
> 54. <a href="https://github.com/andymccurdy/redis-py/issues/1136#issuecomment-571168161" rel="noreferrer" target="_blank">https://github.com/andymccurdy/redis-py/issues/1136#issuecomment-571168161</a><br>
> 55. mailto:<a href="mailto:dellweg@atix.de" target="_blank">dellweg@atix.de</a><br>
<br>
> _______________________________________________<br>
> Pulp-dev mailing list<br>
> <a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br>
> <a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-dev</a><br>
<br>
<br>
-- <br>
Simon Baatz <<a href="mailto:gmbnomis@gmail.com" target="_blank">gmbnomis@gmail.com</a>><br>
<br>
</blockquote></div>
</blockquote></div>