Build fails in koji, but not anywhere else ... how to debug?

[Tom Lane]

I've been pursuing, with increasing frustration, the seemingly simple
goal of getting mysql to rebuild in rawhide since the mass rebuild.
It failed in the mass rebuild (on the same source code which had worked
fine a few weeks before), and has failed multiple attempts since then,
for example
http://koji.fedoraproject.org/koji/getfile?taskID=1596536&name=build.log
http://koji.fedoraproject.org/koji/getfile?taskID=1606630&name=build.log
http://koji.fedoraproject.org/koji/getfile?taskID=1606834&name=build.log
The symptoms are not consistent, although there is some repetition;
for example the first and third runs above encountered the same failure.
It's usually the x86_64 build that dies, but I think that may just be
a reflection of the x86_64 builder being faster than the others.

What is frustrating me is that I can't reproduce the problem outside
koji where I might have a shot at debugging it.  It works fine in F-10,
F-11, rawhide, rawhide-via-mock, and everything else I've tried on my
own machines.  I've tried to try it on RHTS machines, but since neither
F-11 nor rawhide install successfully on those machines, that attempt
didn't get far either.  (And shouldn't somebody be paying closer
attention to that?)

I'm starting to wonder about corrupted ccache on the koji machines,
although unless they all share a common cache that theory doesn't seem
to hold much water.  I wonder if anyone else has a theory, or at least a
suggestion how to debug this problem?

BTW, the last time mysql started unexplainably failing its regression
tests in koji, it turned out to be because the build machines had been
switched to new RHEL-5 kernels that had a different page size on PPC
machines.  mysql is unduly sensitive to things like that :-(  So I'd
also be interested in information about any low-level changes that
might have been applied to the build machines a month or so ago.

			regards, tom lane