Buildsys strangeness
Dan Williams
dcbw at redhat.com
Wed Jun 7 14:40:49 UTC 2006
On Wed, 2006-06-07 at 13:20 +0200, Ralf Corsepius wrote:
> On Wed, 2006-06-07 at 12:06 +0200, Hans de Goede wrote:
> > Hi all,
> >
> > I had a job which was building since yesterday evening, so it was
> > clearly stuck hence i killed the job and requeued it now its stuck again
> > in a very strange way:
> >
> > 10586: dia (dia-0_95-4_fc6) j.w.r.degoede at hhs.nl building/in-progress
> > hammer3.fedora.redhat.com(i386):
> > 3524044740cc821119c18f0b86d24a4a99e51884 done/building
> > hammer3.fedora.redhat.com(i386):
> > f70c9a2aea3719453f700a1ac04a1e632fd3b01f running/building
> > hammer3.fedora.redhat.com(x86_64):
> > da9431d21b048c528bcaff326544a1efd4d8a3b0 done/done
> > ppc2.fedora.redhat.com(ppc):
> > 653e112d8edb708fd7f730776e5cf5b6dba318ca done/done
> >
> > Notice how its building for i386 twice (and on the same machine).
>
> I am facing a similar issue:
>
> # plague-client detail 10603
>
> Detail for Job ID 10603 (OpenSceneGraph):
> --------------------------------------------------------------------------------
> Source: OpenSceneGraph-0_9_9-5_fc3
> Target: fedora-3-extras
> Submitter: rc040203 at freenet.de
> Status: building/
> Archjobs:
> i386: hammer1.fedora.redhat.com done/cleanup
> x86_64: hammer2.fedora.redhat.com running/prepping
> i386: hammer1.fedora.redhat.com running/prepping
Any ideas on how to further debug the following?
I looked at tmraz's vpnc build (10608) and the builder's mock buildroot
setup processes were waiting on in fcntl() to lock a file somewhere deep
in RPM:
strace:
Process 18882 attached - interrupt to quit
fcntl(15, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=0, len=0}
<unfinished ...>
backtrace:
#0 0x0000003fdab0b216 in fcntl () from /lib64/tls/libpthread.so.0
#1 0x00002aaaaacc91d4 in rpmcliVerify () from /usr/lib64/librpm-4.3.so
#2 0x00002aaaaacc927c in rpmtsAcquireLock () from /usr/lib64/librpm-4.3.so
#3 0x00002aaaaacc57cd in rpmtsRun () from /usr/lib64/librpm-4.3.so
#4 0x00002aaaaab7987e in rpmte_Wrap () from /usr/lib64/python2.3/site-packages/rpmmodule.so
#5 0x0000003fd988973f in _PyEval_SliceIndex () from /usr/lib64/libpython2.3.so.1.0
#6 0x0000003fd9889c01 in _PyEval_SliceIndex () from /usr/lib64/libpython2.3.so.1.0
#7 0x0000003fd988b2b0 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.3.so.1.0
#8 0x0000003fd9889a8a in _PyEval_SliceIndex () from /usr/lib64/libpython2.3.so.1.0
#9 0x0000003fd988b2b0 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.3.so.1.0
#10 0x0000003fd988b512 in PyEval_EvalCode () from /usr/lib64/libpython2.3.so.1.0
#11 0x0000003fd98a4129 in PyErr_Display () from /usr/lib64/libpython2.3.so.1.0
#12 0x0000003fd98a510d in PyRun_SimpleFileExFlags () from /usr/lib64/libpython2.3.so.1.0
#13 0x0000003fd98aa808 in Py_Main () from /usr/lib64/libpython2.3.so.1.0
#14 0x0000003fd911c4bb in __libc_start_main () from /lib64/tls/libc.so.6
And 'lsof' output (doesn't look abnormal to me, but I have no idea what to look for):
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
mock-yum 18882 root cwd DIR 9,2 4096 2 /
mock-yum 18882 root rtd DIR 9,2 4096 2 /
mock-yum 18882 root txt REG 9,2 8344 53479 /usr/bin/python
mock-yum 18882 root mem REG 0,0 0 [heap] (stat: No such file or directory)
mock-yum 18882 root mem REG 9,2 115168 336007 /lib64/ld-2.3.6.so
mock-yum 18882 root mem REG 9,2 1613968 336009 /lib64/tls/libc-2.3.6.so
mock-yum 18882 root mem REG 9,2 19880 336029 /lib64/libdl-2.3.6.so
mock-yum 18882 root mem REG 9,2 643408 336011 /lib64/tls/libm-2.3.6.so
mock-yum 18882 root mem REG 9,2 1138632 49241 /usr/lib64/libpython2.3.so.1.0
mock-yum 18882 root mem REG 9,2 61384 336039 /lib64/tls/librt-2.3.6.so
mock-yum 18882 root mem REG 9,2 69080 53421 /usr/lib64/libelf-0.96.so
mock-yum 18882 root mem REG 9,2 64192 49170 /usr/lib64/libbz2.so.1.0.2
mock-yum 18882 root mem REG 9,2 34112 49172 /usr/lib64/libpopt.so.0.0.0
mock-yum 18882 root mem REG 9,2 1031496 49174 /usr/lib64/librpmdb-4.3.so
mock-yum 18882 root mem REG 9,2 386472 49171 /usr/lib64/librpmio-4.3.so
mock-yum 18882 root mem REG 9,2 133424 49169 /usr/lib64/libbeecrypt.so.6.2.0
mock-yum 18882 root mem REG 9,2 130024 336033 /lib64/tls/libpthread-2.3.6.so
mock-yum 18882 root mem REG 9,2 19160 336070 /lib64/libutil-2.3.6.so
mock-yum 18882 root mem REG 9,2 118816 98887 /usr/lib64/python2.3/site-packages/rpmmodule.so
mock-yum 18882 root mem REG 9,2 350760 49175 /usr/lib64/librpm-4.3.so
mock-yum 18882 root mem REG 9,2 79240 49159 /usr/lib64/libz.so.1.2.1.2
mock-yum 18882 root mem REG 9,2 62504 336031 /lib64/libselinux.so.1
mock-yum 18882 root mem REG 9,2 24896 98150 /usr/lib64/python2.3/lib-dynload/strop.so
mock-yum 18882 root mem REG 9,2 20480 98148 /usr/lib64/python2.3/lib-dynload/timemodule.so
mock-yum 18882 root mem REG 9,2 24232 98155 /usr/lib64/python2.3/lib-dynload/structmodule.so
mock-yum 18882 root mem REG 9,2 21952 98153 /usr/lib64/python2.3/lib-dynload/zlibmodule.so
mock-yum 18882 root mem REG 9,2 21672 98030 /usr/lib64/python2.3/lib-dynload/_localemodule.so
mock-yum 18882 root mem REG 9,2 357184 98885 /usr/lib64/python2.3/site-packages/libxml2mod.so
mock-yum 18882 root mem REG 9,2 1073016 49167 /usr/lib64/libxml2.so.2.6.16
mock-yum 18882 root mem REG 9,2 51120 98032 /usr/lib64/python2.3/lib-dynload/_socketmodule.so
mock-yum 18882 root mem REG 9,2 17824 98033 /usr/lib64/python2.3/lib-dynload/_ssl.so
mock-yum 18882 root mem REG 9,2 244320 336040 /lib64/libssl.so.0.9.7a
mock-yum 18882 root mem REG 9,2 1221368 336030 /lib64/libcrypto.so.0.9.7a
mock-yum 18882 root mem REG 9,2 93832 53341 /usr/lib64/libgssapi_krb5.so.2.2
mock-yum 18882 root mem REG 9,2 464040 53340 /usr/lib64/libkrb5.so.3.2
mock-yum 18882 root mem REG 9,2 10384 336023 /lib64/libcom_err.so.2.1
mock-yum 18882 root mem REG 9,2 145424 49152 /usr/lib64/libk5crypto.so.3.0
mock-yum 18882 root mem REG 9,2 93624 336019 /lib64/libresolv-2.3.6.so
mock-yum 18882 root mem REG 9,2 21224 98042 /usr/lib64/python2.3/lib-dynload/binascii.so
mock-yum 18882 root mem REG 9,2 18352 98060 /usr/lib64/python2.3/lib-dynload/mathmodule.so
mock-yum 18882 root mem REG 9,2 11808 98031 /usr/lib64/python2.3/lib-dynload/_random.so
mock-yum 18882 root mem REG 9,2 15528 98053 /usr/lib64/python2.3/lib-dynload/fcntlmodule.so
mock-yum 18882 root mem REG 9,2 19096 98045 /usr/lib64/python2.3/lib-dynload/cStringIO.so
mock-yum 18882 root mem REG 9,2 13456 98127 /usr/lib64/python2.3/lib-dynload/md5module.so
mock-yum 18882 root mem REG 9,2 13008 98142 /usr/lib64/python2.3/lib-dynload/shamodule.so
mock-yum 18882 root mem REG 9,2 75544 98044 /usr/lib64/python2.3/lib-dynload/cPickle.so
mock-yum 18882 root mem REG 9,2 19936 98136 /usr/lib64/python2.3/lib-dynload/readline.so
mock-yum 18882 root mem REG 9,2 229824 49201 /usr/lib64/libreadline.so.4.3
mock-yum 18882 root mem REG 9,2 15744 336225 /lib64/libtermcap.so.2.0.8
mock-yum 18882 root mem REG 9,2 407640 98684 /usr/lib64/python2.3/lib-dynload/unicodedata.so
mock-yum 18882 root mem REG 9,2 27800 98058 /usr/lib64/python2.3/lib-dynload/itertools.so
mock-yum 18882 root mem REG 9,2 38592 98040 /usr/lib64/python2.3/lib-dynload/arraymodule.so
mock-yum 18882 root mem REG 9,2 58840 336105 /lib64/libnss_files-2.3.6.so
mock-yum 18882 root mem REG 9,2 24712 336102 /lib64/libnss_dns-2.3.6.so
mock-yum 18882 root mem REG 9,3 16384 16369286 /var/lib/mock/fedora-development-x86_64-core-839bf2f17e49475381b3254433701b5c7d5234c5/root/var/lib/rpm/__db.001
mock-yum 18882 root mem REG 9,3 1318912 16369287 /var/lib/mock/fedora-development-x86_64-core-839bf2f17e49475381b3254433701b5c7d5234c5/root/var/lib/rpm/__db.002
mock-yum 18882 root mem REG 9,3 663552 16369288 /var/lib/mock/fedora-development-x86_64-core-839bf2f17e49475381b3254433701b5c7d5234c5/root/var/lib/rpm/__db.003
mock-yum 18882 root 0u CHR 1,3 2442 /dev/null
mock-yum 18882 root 1w FIFO 0,6 2493184 pipe
mock-yum 18882 root 2w FIFO 0,6 2493184 pipe
mock-yum 18882 root 3w REG 9,2 7580121 160431 /var/log/plague-builder.log
mock-yum 18882 root 4u REG 9,3 3939 16369251 /mnt/build/builder_work/839bf2f17e49475381b3254433701b5c7d5234c5/result/root.log
mock-yum 18882 root 5u IPv4 7990 TCP *:8889 (LISTEN)
mock-yum 18882 root 6u IPv4 8396 TCP *:8888 (LISTEN)
mock-yum 18882 root 7u REG 9,3 1003 16369244 /mnt/build/builder_work/839bf2f17e49475381b3254433701b5c7d5234c5/result/job.log
mock-yum 18882 root 8u REG 9,3 1017 7356641 /mnt/build/builder_work/9de1339d8bff65b824cf4258bc107452768e5d48/result/job.log
mock-yum 18882 root 9u REG 9,3 0 16369252 /mnt/build/builder_work/839bf2f17e49475381b3254433701b5c7d5234c5/result/build.log
mock-yum 18882 root 10w REG 9,3 0 16369282 /var/lib/mock/fedora-development-x86_64-core-839bf2f17e49475381b3254433701b5c7d5234c5/root/var/log/yum.log
mock-yum 18882 root 11r REG 9,3 12288 16369289 /var/lib/mock/fedora-development-x86_64-core-839bf2f17e49475381b3254433701b5c7d5234c5/root/var/lib/rpm/Packages
mock-yum 18882 root 12r REG 9,3 12288 16369289 /var/lib/mock/fedora-development-x86_64-core-839bf2f17e49475381b3254433701b5c7d5234c5/root/var/lib/rpm/Packages
mock-yum 18882 root 13r REG 9,3 12288 16369290 /var/lib/mock/fedora-development-x86_64-core-839bf2f17e49475381b3254433701b5c7d5234c5/root/var/lib/rpm/Providename
mock-yum 18882 root 14r REG 9,3 12288 16369376 /var/lib/mock/fedora-development-x86_64-core-839bf2f17e49475381b3254433701b5c7d5234c5/root/var/lib/rpm/Conflictname
mock-yum 18882 root 15u REG 9,2 0 160839 /var/lock/rpm/transaction
More information about the fedora-extras-list
mailing list