Extras build failures...

[Michael Schwendt]

On Fri, 09 Jun 2006 07:37:21 -0400, seth vidal wrote:

> On Fri, 2006-06-09 at 12:31 +0100, David Woodhouse wrote:
> > On Thu, 2006-06-08 at 08:11 -0400, Dan Williams wrote:
> > > I think we should bump up the timeout here, actually.  It's not a
> > > problem on the builders, but an issue with the server and Extras
> > > release scripts. 
> > 
> > Can't we just fix the locking? Surely we don't need to lock the repo for
> > the _whole_ period of the upload. You can upload the new packages, then
> > lock, then create metadata, then unlock.

Danger. It would be possible for the push script to fetch partial uploads
from the needsign repo. Unless you upload to a tmp dir, then lock, move,
createrepo, unlock.

> > You don't need it locked for
> > more than the few seconds it takes to update the metadata for the new
> > packages, surely?
> 
> it takes longer than a few seconds to update the metadata.

No repo is locked when repodata and repoview are updated by the push
script. The push script locks the needsign queue only for a short period.

See my recent message to buildsys-list, which explains it.

There is only one lock file per dist repo, which blocks plague from
updating a needsign repository while the push script is accessing it and
vice versa. This file is locked only for the short time it takes to enter
the passphrase, sign packages and let them install into the local master
repository.

The time-consuming operations (repomanage, createrepo, repoview, sync to
public master repo) are performed on the local master repo, where no
locking is implemented.  Btw, it would not be possible to protect any
yum/mock clients from accessing the public (!) master repository while a
sync is in progress.  You could only play tricks like doing multiple
syncs, first a no-delete sync of all new packages, excluding repodata,
then another sync with the new repodata, trying to make the files
available as quickly as possible, then another sync to delete gone
files. And it still would not be bullet-proof -- adding files plus
updating metadata would need to be an atomic operation.

One major problem with the push script was that _it modified rpms_ in the
needsign repo. It signed them, thereby invalidating the metadata. It
_moved_ them away into the local master repo, leaving build servers with a
temporarily broken needsign repo until the updates arrived at
download.fedora.redhat.com. (The older push script even mailed the build
report long before the actual sync.)

However, the situation has changed last night, and I've modified the new
push script even further to work on copies of rpms. Packages in the
needsign repo now are not modified or moved by the push script. Pushed and
unneeded builds expire after some time (48 hours currently).