plague: Job waited too long for repo to unlock. Killing it...

[Michael Schwendt]

If in a failed job.log you see the message

    Job waited too long for repo to unlock. Killing it...

please notify me.

It's a problem in the plague server code that results in a denial of
service for subsequent build jobs. I have a traceback from Dec 28th, but
in the context of the source code it doesn't make sense yet (because a few
lines earlier the code ensures that the files to be copied exist and are
readable). Buildsys runs a slightly modified version that adds a bit more
debug output in this area.

[...]

37683 (grads/ppc): Build result files - [ 'state.log', 'grads-1.9b4-21.el5.src.rpm', 'grads-debuginfo-1.9b4-21.el5.ppc.rpm', 'grads-1.9b4-21.el5.ppc.rpm', 'root.log', 'build.log', 'job.log' ]
37683 (grads/x86_64): Build result files - [ 'root.log', 'build.log', 'grads-debuginfo-1.9b4-21.el5.x86_64.rpm', 'grads-1.9b4-21.el5.x86_64.rpm', 'grads-1.9b4-21.el5.src.rpm', 'state.log', 'job.log' ]
37683 (grads/i386): Build result files - [ 'build.log', 'state.log', 'grads-1.9b4-21.el5.i386.rpm', 'job.log', 'root.log', 'grads-debuginfo-1.9b4-21.el5.i386.rpm', 'grads-1.9b4-21.el5.src.rpm' ]
Repo 'fedora-5-epel': updating repository metadata...
Exception in thread Repo: fedora-5-epel:
Traceback (most recent call last):
  File "/usr/lib64/python2.3/threading.py", line 436, in __bootstrap
    self.run()
  File "/usr/share/plague/server/Repo.py", line 233, in run
    self._update_repo()
  File "/usr/share/plague/server/Repo.py", line 137, in _update_repo
    shutil.copy(src, file_in_dst)
  File "/usr/lib64/python2.3/shutil.py", line 72, in copy
    copymode(src, dst)
  File "/usr/lib64/python2.3/shutil.py", line 49, in copymode
    st = os.stat(src)
OSError: [Errno 2] No such file or directory: '/srv/rpmbuild/server_work/fedora-5-epel/37683-grads-1.9b4-21.el5/x86_64/grads-1.9b4-21.el5.x86_64.rpm'
37683 (grads): Job finished.