From dcbw at redhat.com Wed Feb 1 03:35:03 2006 From: dcbw at redhat.com (Dan Williams) Date: Tue, 31 Jan 2006 22:35:03 -0500 Subject: pthread_sigmask module In-Reply-To: References: Message-ID: <1138764903.4059.0.camel@localhost.localdomain> On Tue, 2006-01-31 at 19:46 -0400, Jeff Sheltren wrote: > Hi, I am having some problems with the build server hanging (ie. > there are no jobs running, but when I submit a job it sits with > status 'waiting')... > > Since this is running on EL4 (python 2.3.4), I'm getting this message > when I start up plague-builder: > > WARNING: You might want to install the pthread_sigmask module. > Python versions earlier than 2.4 have problems with signals and > spawned processes. > > I'm guessing this is related to my problem :) So the question is, > where can I find this module? I haven't quite gotten around to putting it in extras yet, but you can grab the srpm here: http://people.redhat.com/dcbw/py_pthread_sigmask-0.1-1.src.rpm Dan From sheltren at cs.ucsb.edu Wed Feb 1 10:25:44 2006 From: sheltren at cs.ucsb.edu (Jeff Sheltren) Date: Wed, 1 Feb 2006 06:25:44 -0400 Subject: pthread_sigmask module In-Reply-To: <1138764903.4059.0.camel@localhost.localdomain> References: <1138764903.4059.0.camel@localhost.localdomain> Message-ID: On Jan 31, 2006, at 11:35 PM, Dan Williams wrote: > On Tue, 2006-01-31 at 19:46 -0400, Jeff Sheltren wrote: >> Hi, I am having some problems with the build server hanging (ie. >> there are no jobs running, but when I submit a job it sits with >> status 'waiting')... >> >> Since this is running on EL4 (python 2.3.4), I'm getting this message >> when I start up plague-builder: >> >> WARNING: You might want to install the pthread_sigmask module. >> Python versions earlier than 2.4 have problems with signals and >> spawned processes. >> >> I'm guessing this is related to my problem :) So the question is, >> where can I find this module? > > I haven't quite gotten around to putting it in extras yet, but you can > grab the srpm here: > > http://people.redhat.com/dcbw/py_pthread_sigmask-0.1-1.src.rpm > > Dan Great, thank you. I'll give it a try. -Jeff From andreas.bierfert at lowlatency.de Thu Feb 9 08:40:45 2006 From: andreas.bierfert at lowlatency.de (Andreas Bierfert) Date: Thu, 09 Feb 2006 09:40:45 +0100 Subject: x86_64-devel koffice builds fail randomly (on random builders) Message-ID: <43EB000D.8010502@lowlatency.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, when I pushed koffice 1.4.90-{1,2} everything worked fine. Not there was a bug with some desktop file and I fixed it by adding an %exclude to the file section (so no changes in the sources or the build part or the alike) but now the build keeps failing randomly. See the logs from try 2-5 (lost the first one). Any help would be very much welcome... - - Andreas This is from the 2nd try: creating libkiviocommon_la.all_cpp.cpp ... /bin/sh ../../libtool --silent --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H -I. - -I. -I../.. -I./ui -Iui -I./config -Iconfig -I./kiviosdk -I./tools - -I../../lib/kofficeui -I../../lib/kofficeui -I../../lib/kofficecore - -I../../lib/kofficecore -I../../lib/store -I../../lib/store -I../../lib/kwmf - -I../../lib/kwmf -I../../lib/kopalette -I../../lib/kopalette - -I../../lib/kopalette -I../../lib/kopalette -I../../lib/kotext - -I../../lib/kotext -I../../lib/kformula -I/usr/include/kde - -I/usr/lib64/qt-3.3/include -I. -DQT_THREAD_SUPPORT -D_REENTRANT - -Wno-long-long -Wundef -ansi -D_XOPEN_SOURCE=500 -D_BSD_SOURCE -Wcast-align - -Wconversion -Wchar-subscripts -Wall -W -Wpointer-arith -O2 -O2 -g -pipe -Wall - -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 - -m64 -mtune=generic -Wformat-security -Wmissing-format-attribute - -Wno-non-virtual-dtor -fno-exceptions -fno-check-new -fno-common - -DQT_CLEAN_NAMESPACE -DQT_NO_ASCII_CAST -DQT_NO_STL -DQT_NO_COMPAT -DQT_NO_TRANS LATION -DHAVE_KNEWSTUFF -c -o libkiviocommon_la.all_cpp.lo libkiviocommon_la.all_cpp.cpp /bin/sh ../../libtool --silent --tag=CXX --mode=link g++ -Wno-long-long -Wundef - -ansi -D_XOPEN_SOURCE=500 -D_BSD_SOURCE -Wcast-align -Wconversion - -Wchar-subscripts -Wall -W -Wpointer-arith -O2 -O2 -g -pipe -Wall - -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 - -m64 -mtune=generic -Wformat-security -Wmissing-format-attribute - -Wno-non-virtual-dtor -fno-exceptions -fno-check-new -fno-common - -DQT_CLEAN_NAMESPACE -DQT_NO_ASCII_CAST -DQT_NO_STL -DQT_NO_COMPAT - -DQT_NO_TRANSLATION -DHAVE_KNEWSTUFF -o libkiviocommon.la -rpath /usr/lib64 - -L/usr/lib64/qt-3.3/lib -L/usr/lib64 -L/usr/lib64 -no-undefined - -Wl,--no-undefined -Wl,--allow-shlib-undefined libkiviocommon_la.all_cc.lo libkiviocommon_la.all_cpp.lo ../../kivio/kiviopart/tools/libtools.la ../../kivio/kiviopart/kiviosdk/libkiviosdk.la ../../kivio/kiviopart/config/libkivioconfig.la ../../kivio/kiviopart/ui/libui.la ../../lib/kofficeui/libkofficeui.la ../../lib/kopainter/libkopainter.la ../../lib /kopalette/libkopalette.la ../../lib/kotext/libkotext.la -lpython2.4 -ldl -ldl -L/usr/lib64 : : No such file or directory : : No such file or directory libtool: link: `/usr/lib64/libkutils.la' is not a valid libtool archive make[4]: *** [libkiviocommon.la] Error 1 make[4]: Leaving directory `/builddir/build/BUILD/koffice-1.4.90/kivio/kiviopart' make[3]: *** [all-recursive] Error 1 make[3]: Leaving directory `/builddir/build/BUILD/koffice-1.4.90/kivio/kiviopart' make[2]: *** [all-recursive] Error 1 make[2]: Leaving directory `/builddir/build/BUILD/koffice-1.4.90/kivio' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/builddir/build/BUILD/koffice-1.4.90' make: *** [all] Error 2 error: Bad exit status from /var/tmp/rpm-tmp.25214 (%build) RPM build errors: Bad exit status from /var/tmp/rpm-tmp.25214 (%build) 3rd try: /bin/sh ../libtool --silent --tag=CXX --mode=link g++ -Wno-long-long -Wundef - -ansi -D_XOPEN_SOURCE=500 -D_BSD_SOURCE -Wcast-align -Wconversion - -Wchar-subscripts -Wall -W -Wpointer-arith -O2 -O2 -g -pipe -Wall - -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 - -m64 -mtune=generic -Wformat-security -Wmissing-format-attribute - -Wno-non-virtual-dtor -fno-exceptions -fno-check-new -fno-common - -DQT_CLEAN_NAMESPACE -DQT_NO_ASCII_CAST -DQT_NO_STL -DQT_NO_COMPAT - -DQT_NO_TRANSLATION -DHAVE_KNEWSTUFF -o kpresenter -no-undefined - -L/usr/lib64/qt-3.3/lib -L/usr/lib64 -L/usr/lib64 kpresenter.la.o libkdeinit_kpresenter.la libtool: link: `/usr/lib64/libkdecore.la' is not a valid libtool archive make[3]: *** [kpresenter] Error 1 make[3]: Leaving directory `/builddir/build/BUILD/koffice-1.4.90/kpresenter' make[2]: *** [all-recursive] Error 1 make[2]: Leaving directory `/builddir/build/BUILD/koffice-1.4.90/kpresenter' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/builddir/build/BUILD/koffice-1.4.90' make: *** [all] Error 2 error: Bad exit status from /var/tmp/rpm-tmp.96654 (%build) RPM build errors: Bad exit status from /var/tmp/rpm-tmp.96654 (%build) 4th try: /bin/sh ../../../libtool --silent --tag=CXX --mode=link g++ -Wno-long-long - -Wundef -ansi -D_XOPEN_SOURCE=500 -D_BSD_SOURCE -Wcast-align -Wconversion - -Wchar-subscripts -Wall -W -Wpointer-arith -O2 -O2 -g -pipe -Wall - -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 - -m64 -mtune=generic -Wformat-security -Wmissing-format-attribute - -Wno-non-virtual-dtor -fno-exceptions -fno-check-new -fno-common - -DQT_CLEAN_NAMESPACE -DQT_NO_ASCII_CAST -DQT_NO_STL -DQT_NO_COMPAT - -DQT_NO_TRANSLATION -DHAVE_KNEWSTUFF -fexceptions -o krossruby.la -rpath /usr/lib64/kde3 -module -lruby -L/usr/lib64/qt-3.3/lib -L/usr/lib64 - -L/usr/lib64 -avoid-version -module -no-undefined -Wl,--no-undefined - -Wl,--allow-shlib-undefined -version-info 1:0:0 krossruby_la.all_cpp.lo - -lqt-mt -lz -lpng -lz -lm -lXext -lX11 -lSM -lICE -lpthread -lkdecore ../api/libkrossapi.la ../main/libkrossmain.la libtool: link: `/usr/lib64/libkio.la' is not a valid libtool archive make[4]: *** [krossruby.la] Error 1 make[4]: Leaving directory `/builddir/build/BUILD/koffice-1.4.90/lib/kross/ruby' make[3]: *** [all-recursive] Error 1 make[3]: Leaving directory `/builddir/build/BUILD/koffice-1.4.90/lib/kross' make[2]: *** [all-recursive] Error 1 make[2]: Leaving directory `/builddir/build/BUILD/koffice-1.4.90/lib' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/builddir/build/BUILD/koffice-1.4.90' make: *** [all] Error 2 error: Bad exit status from /var/tmp/rpm-tmp.10097 (%build) RPM build errors: Bad exit status from /var/tmp/rpm-tmp.10097 (%build) 5th try: bin/sh ../libtool --silent --tag=CXX --mode=link g++ -Wno-long-long -Wundef - -ansi -D_XOPEN_SOURCE=500 -D_BSD_SOURCE -Wcast-align -Wconversion - -Wchar-subscripts -Wall -W -Wpointer-arith -O2 -O2 -g -pipe -Wall - -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 - -m64 -mtune=generic -Wformat-security -Wmissing-format-attribute - -Wno-non-virtual-dtor -fno-exceptions -fno-check-new -fno-common - -DQT_CLEAN_NAMESPACE -DQT_NO_ASCII_CAST -DQT_NO_STL -DQT_NO_COMPAT - -DQT_NO_TRANSLATION -DHAVE_KNEWSTUFF -o libkdgantt.la - -L/usr/lib64/qt-3.3/lib -L/usr/lib64 -L/usr/lib64 -no-undefined - -Wl,--no-undefined -Wl,--allow-shlib-undefined libkdgantt_la.all_cpp.lo -lkdeui libtool: link: `/usr/lib64/libkdefx.la' is not a valid libtool archive make[2]: *** [libkdgantt.la] Error 1 make[2]: Leaving directory `/builddir/build/BUILD/koffice-1.4.90/kdgantt' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/builddir/build/BUILD/koffice-1.4.90' make: *** [all] Error 2 error: Bad exit status from /var/tmp/rpm-tmp.40707 (%build) RPM build errors: Bad exit status from /var/tmp/rpm-tmp.40707 (%build) - -- Andreas Bierfert | http://awbsworld.de | GPG: C58CF1CB andreas.bierfert at lowlatency.de | http://lowlatency.de | signed/encrypted phone: +49 2402 102373 | cell: +49 172 9789968 | mail preferred -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFD6wANQEQyPsWM8csRAsPMAJwIzdk8K2zR9AfJRkWnpkhwtN206wCfXBwJ sc6WBZYbexp7WyxRLduvyJc= =GFC8 -----END PGP SIGNATURE----- From fedora at leemhuis.info Mon Feb 13 18:40:51 2006 From: fedora at leemhuis.info (Thorsten Leemhuis) Date: Mon, 13 Feb 2006 19:40:51 +0100 Subject: Kernel-Modules in the buildsys Message-ID: <1139856052.6436.4.camel@localhost.localdomain> Hi all! The kernel-module discussion on fedora-extras list was quite successful and we have a kernel-module scheme now where nobody complained loudly (at least until now). Two examples that are based on the latest results from the discussion are found in extras-cvs in the packages lirc-kmod and thinkpad-kmod. What we don't have yet: Support for the that kernel-module scheme in the buildsystem. What's needed? Roughly this: - we need to get the kernel-verrel to the spec-file/rpmbuild somehow - we need to get the kernel-variants to the spec-file/rpmbuild somehow Example: To rebuild lirc-kmod for the kernel 2.6.15-1.1831_FC4 and 2.6.15-1.1831_FC4smp on i686 one would run this command with the current scheme: $ rpmbuild --rebuild lirc-kmod-0.8.0-2.2.6.15_1.1831_FC4.src.rpm --define 'kver 2.6.15-1.1831_FC4' --define 'kvariants "" smp ' --target i686 So guys, how to we teach that the buildsystem? Do we handle that in mock? Or in plague? As a yum-plugin? I don't know plague and mock to much, so any help/suggestions appreciated! /me looks at the command and wonders why we use "kver" for kversion -- we either should use "kver" and "kvars" or "kversion" and "kvariants" -- Thorsten Leemhuis From cweyl at alumni.drew.edu Mon Feb 13 19:00:50 2006 From: cweyl at alumni.drew.edu (Chris Weyl) Date: Mon, 13 Feb 2006 14:00:50 -0500 Subject: Kernel-Modules in the buildsys In-Reply-To: <1139856052.6436.4.camel@localhost.localdomain> References: <1139856052.6436.4.camel@localhost.localdomain> Message-ID: <7dd7ab490602131100y5b12b659h995fa4967d41a027@mail.gmail.com> On 2/13/06, Thorsten Leemhuis wrote: > - we need to get the kernel-verrel to the spec-file/rpmbuild somehow > - we need to get the kernel-variants to the spec-file/rpmbuild somehow > > Example: To rebuild lirc-kmod for the kernel 2.6.15-1.1831_FC4 and > 2.6.15-1.1831_FC4smp on i686 one would run this command with the current > scheme: > > $ rpmbuild --rebuild lirc-kmod-0.8.0-2.2.6.15_1.1831_FC4.src.rpm > --define 'kver 2.6.15-1.1831_FC4' --define 'kvariants "" smp ' --target > i686 > > So guys, how to we teach that the buildsystem? Do we handle that in > mock? Or in plague? As a yum-plugin? I don't know plague and mock to > much, so any help/suggestions appreciated! One thing that leaps to mind -- and this may be a bit simplistic, or covered over in fedora-extras, I'll readily admit -- but how about embedding kver and kvariants in the spec file itself? This would require no tweaks to the buildsys (it would seem to me), and if %kver is part of the release tag, it wouldn't involve tweaking that as well. + no tweaks to plague needed + everything to do this is in place already - every time a new kernel is released, the spec would have to be tweaked and a rebuild job submitted. - every time a new version of the kernel module is released, see above -Chris From dcbw at redhat.com Mon Feb 13 20:19:01 2006 From: dcbw at redhat.com (Dan Williams) Date: Mon, 13 Feb 2006 15:19:01 -0500 Subject: Kernel-Modules in the buildsys In-Reply-To: <7dd7ab490602131100y5b12b659h995fa4967d41a027@mail.gmail.com> References: <1139856052.6436.4.camel@localhost.localdomain> <7dd7ab490602131100y5b12b659h995fa4967d41a027@mail.gmail.com> Message-ID: <1139861942.12431.1.camel@dhcp83-74.boston.redhat.com> On Mon, 2006-02-13 at 14:00 -0500, Chris Weyl wrote: > - every time a new kernel is released, the spec would have to be > tweaked and a rebuild job submitted. > - every time a new version of the kernel module is released, see above We'd like to have the buildsys trigger builds of modules automatically when a new kernel gets built though, so these two minuses are pretty large... If this is the scheme, we'll need some changes in: 1) Mock: allow passing of values to the rpmbuild command 2) Plague: figure out what variables to pass to mock based on a specific kernel, and also auto-rebuild kernel modules when the kernel changes So we can't do this without changes to plague & mock. Dan From katzj at redhat.com Mon Feb 13 20:40:55 2006 From: katzj at redhat.com (Jeremy Katz) Date: Mon, 13 Feb 2006 15:40:55 -0500 Subject: Kernel-Modules in the buildsys In-Reply-To: <1139861942.12431.1.camel@dhcp83-74.boston.redhat.com> References: <1139856052.6436.4.camel@localhost.localdomain> <7dd7ab490602131100y5b12b659h995fa4967d41a027@mail.gmail.com> <1139861942.12431.1.camel@dhcp83-74.boston.redhat.com> Message-ID: <1139863255.11251.21.camel@bree.local.net> On Mon, 2006-02-13 at 15:19 -0500, Dan Williams wrote: > On Mon, 2006-02-13 at 14:00 -0500, Chris Weyl wrote: > > - every time a new kernel is released, the spec would have to be > > tweaked and a rebuild job submitted. > > - every time a new version of the kernel module is released, see above > > We'd like to have the buildsys trigger builds of modules automatically > when a new kernel gets built though, so these two minuses are pretty > large... At the same time, not doing this means that we break the assumption that you can do cvs co -r $(simple_transform_of_nvr) module Jeremy From wtogami at redhat.com Thu Feb 16 22:04:34 2006 From: wtogami at redhat.com (Warren Togami) Date: Thu, 16 Feb 2006 17:04:34 -0500 Subject: kmod arch, ExlusiveArch, and buildsys? Message-ID: <43F4F6F2.7050606@redhat.com> Hi Dan, I am supposed to ask you about your opinion about how to ask buildsys to build for specific archs for a kernel module. I believe that we should use ExclusiveArch with explicit listed archs, and buildsys should loop through each listed arch and build each. This would also work for multi-arch performance intensive userspace packages similar to what we do in glibc or openssl in Core. [1] Contributors may also list any other arch like s390x or alpha in ExclusiveArch, and buildsys simply ignores it if it isn't supported in Extras. I believe this is the best approach to take for Extras, because Core has been similar to this for many years. Do you have any opinion on this, or know of a better way to handle this? Here is some related discussion: the problem with exclusivearch is that let's say you list i586 i686 x86_64 ppc and someone wants to rebuild it for alpha -> won't build, even if the package/software would work scop, you can include other archs in the list, buildsys can simply ignore Extras non-supported archs. warren, it's abuse of the tag nevertheless scop, I'd personally disagree, but either way this is very simple and it already behaves similar to this in Core literally forever. more importantly the simplicity of this allows us to move forward sooner (assuming dcbw doesn't have a better idea) that I agree with Warren Togami wtogami at redhat.com [1] However we should disallow this, unless someone proves through benchmarks that there is a huge benefit of publishing both i386 and i686 packages and our tools support selection of the right arch directly. Currently I believe it doesn't. From enrico.scholz at informatik.tu-chemnitz.de Fri Feb 17 13:38:46 2006 From: enrico.scholz at informatik.tu-chemnitz.de (Enrico Scholz) Date: Fri, 17 Feb 2006 14:38:46 +0100 Subject: Sugg: Improve for mock (caching) In-Reply-To: <1131051274.20403.34.camel@cutter> (seth vidal's message of "Thu, 03 Nov 2005 15:54:33 -0500") References: <20051103195543.GA29411@myhome> <1131047950.20403.15.camel@cutter> <87oe51wjdl.fsf@kosh.bigo.ensc.de> <1131051274.20403.34.camel@cutter> Message-ID: <87vevef6l5.fsf@kosh.bigo.ensc.de> skvidal at phy.duke.edu (seth vidal) writes: >> The speedup gained by caching is not (primarily) related to the >> network connection but to the rpm installation and disk-IO. On the >> fedora.us server we used caching also and the initial buildroot was >> created in <1min (afair). >> >> Local tests at that time showed, that the initial buildroot can be >> created in 20 seconds when a tmpfs will be used instead of a real disk >> with ext3. > > So the numbers I'm curious about that you may have: > - remote repos - but lan-like connection times > - local-disk repos - chroot creation - all on ext3 > - tmpfs targets for the chroot creation and repos > > do you have anything like that? ok, a little bit late, but I implemented caching now so I can provide some real numbers: Traditional 'mock init' on a tmpfs takes | Init buildroot... | | real 1m37.445s | user 1m15.329s | sys 0m8.009s tar'ing this filesystem, and extracting it takes | Extracting cached buildroot... | | real 0m1.266s | user 0m0.188s | sys 0m0.944s Both operations are resulting into the same root-fs; the executed code was: | mock=( env http_proxy= mock --resultdir="$results" -r "$MOCK_ROOT" ) | mock_yum=( env http_proxy= /usr/sbin/mock-helper yum --installroot "$BUILD_DIR" ) | | function initBuildroot() { | "${mock[@]}" clean >/dev/null | | if test -e "$CACHE_NAME"; then | echo "Extracting cached buildroot..." | time /usr/sbin/mock-helper uncache "$MOCK_ROOT" | | echo "Updating metadata..." | "${mock_yum[@]}" check-update || : | else | lock | echo "Init buildroot..." | time "${mock[@]}" init >/dev/null | unlock | | echo "Cleaning up root..." | "${mock_yum[@]}" clean packages headers | echo "Creating cache..." | /usr/sbin/mock-helper cache "$MOCK_ROOT" | fi | } The 'cache' and 'uncache' operations of 'mock-helper' were added by me and are simple 'tar' wrappers. The resulting tarball is uncompressed and 280MB sized, both the rootfs and the tarball are located in the tmpfs. Environment: mock-machine: Intel Celeron 2.80GHz, 1500MB RAM repository-server: PII 266MHz, 196MB RAM, http-access to repo (Apache2) 100Mb/s LAN; 'wget' download rate between repo and mock machine appr. 7-8MB/s Enrico From skvidal at linux.duke.edu Fri Feb 17 20:25:02 2006 From: skvidal at linux.duke.edu (seth vidal) Date: Fri, 17 Feb 2006 15:25:02 -0500 Subject: Sugg: Improve for mock (caching) In-Reply-To: <87vevef6l5.fsf@kosh.bigo.ensc.de> References: <20051103195543.GA29411@myhome> <1131047950.20403.15.camel@cutter> <87oe51wjdl.fsf@kosh.bigo.ensc.de> <1131051274.20403.34.camel@cutter> <87vevef6l5.fsf@kosh.bigo.ensc.de> Message-ID: <1140207903.9671.13.camel@cutter> > ok, a little bit late, but I implemented caching now so I can provide > some real numbers: > > Traditional 'mock init' on a tmpfs takes > > | Init buildroot... > | > | real 1m37.445s > | user 1m15.329s > | sys 0m8.009s > > tar'ing this filesystem, and extracting it takes > > | Extracting cached buildroot... > | > | real 0m1.266s > | user 0m0.188s > | sys 0m0.944s > > > Both operations are resulting into the same root-fs; the executed code > was: > > | mock=( env http_proxy= mock --resultdir="$results" -r "$MOCK_ROOT" ) > | mock_yum=( env http_proxy= /usr/sbin/mock-helper yum --installroot "$BUILD_DIR" ) > | > | function initBuildroot() { > | "${mock[@]}" clean >/dev/null > | > | if test -e "$CACHE_NAME"; then > | echo "Extracting cached buildroot..." > | time /usr/sbin/mock-helper uncache "$MOCK_ROOT" > | > | echo "Updating metadata..." > | "${mock_yum[@]}" check-update || : > | else > | lock > | echo "Init buildroot..." > | time "${mock[@]}" init >/dev/null > | unlock > | > | echo "Cleaning up root..." > | "${mock_yum[@]}" clean packages headers > | echo "Creating cache..." > | /usr/sbin/mock-helper cache "$MOCK_ROOT" > | fi > | } > > The 'cache' and 'uncache' operations of 'mock-helper' were added by me > and are simple 'tar' wrappers. The resulting tarball is uncompressed and > 280MB sized, both the rootfs and the tarball are located in the tmpfs. > > Environment: > mock-machine: Intel Celeron 2.80GHz, 1500MB RAM > repository-server: PII 266MHz, 196MB RAM, http-access to repo (Apache2) > 100Mb/s LAN; 'wget' download rate between repo and mock machine appr. 7-8MB/s > > okay - that looks great. could I talk you into applying those patches to the mock package in fedora cvs and testing it out? If it works and people are happy with it then I don't see any reason not to use it. fair enough? -sv From mharris at mharris.ca Fri Feb 17 20:41:57 2006 From: mharris at mharris.ca (Mike A. Harris) Date: Fri, 17 Feb 2006 15:41:57 -0500 Subject: kmod arch, ExlusiveArch, and buildsys? In-Reply-To: <43F4F6F2.7050606@redhat.com> References: <43F4F6F2.7050606@redhat.com> Message-ID: <43F63515.4040701@mharris.ca> Warren Togami wrote: > Hi Dan, > > I am supposed to ask you about your opinion about how to ask buildsys to > build for specific archs for a kernel module. I believe that we should > use ExclusiveArch with explicit listed archs, and buildsys should loop > through each listed arch and build each. This would also work for > multi-arch performance intensive userspace packages similar to what we > do in glibc or openssl in Core. [1] Contributors may also list any > other arch like s390x or alpha in ExclusiveArch, and buildsys simply > ignores it if it isn't supported in Extras. > > I believe this is the best approach to take for Extras, because Core has > been similar to this for many years. Do you have any opinion on this, > or know of a better way to handle this? > > Here is some related discussion: > the problem with exclusivearch is that let's say you list i586 > i686 x86_64 ppc and someone wants to rebuild it for alpha -> won't > build, even if the package/software would work > scop, you can include other archs in the list, buildsys can > simply ignore Extras non-supported archs. > warren, it's abuse of the tag nevertheless > scop, I'd personally disagree, but either way this is very > simple and it already behaves similar to this in Core literally forever. > more importantly the simplicity of this allows us to move > forward sooner > (assuming dcbw doesn't have a better idea) > that I agree with The alternative tag is ExcludeArch. It's sometimes better to use that, and simply exclude the arches we do _not_ want to build for, but which there are machines that would otherwise go ahead and build the packages. The approach I've taken for the Xorg driver rpms is to use the ExclusiveArch tag, and pre-fill them with the upstream default drivers that got built on each arch previously, modulo any unintentional ommisions/etc. I covered sparc/sparc64/alpha as well. Using this mechanism to solve this type of problem automatically has a hard coded assumption that you want to either include all by default and then blacklist individual ones, or you want to blacklist all by default and then include individual ones. On a side note, except under very specific circumstances, do not use "i386" or "i686" as an architecture to list in ExclusiveArch or Excludearch. Use the rpm macro "%{ix86}" instead, as that expands to all existing architecture aliases past, present and future. Only use i386 or i686 et al. if you really do want to intentionally restrict the package from being built on other x86 compatible architectures, such as i486 for example. Keeping this data _in_ the rpms means that the package itself is in sync with what arches it should be building on. No external synchronization is necessary. Trying to store such information somewhere else, and let the layer above take care of it (the buildsystem, or the tree maker software, leaves things open to be desynchronized, etc. It also leaves more room for errors to happen, such as a database being changed accidentally or restored from backup and losing changes. -- Mike A. Harris * Open Source Advocate * http://mharris.ca Proud Canadian. From dcbw at redhat.com Fri Feb 17 21:10:56 2006 From: dcbw at redhat.com (Dan Williams) Date: Fri, 17 Feb 2006 16:10:56 -0500 Subject: kmod arch, ExlusiveArch, and buildsys? In-Reply-To: <43F63515.4040701@mharris.ca> References: <43F4F6F2.7050606@redhat.com> <43F63515.4040701@mharris.ca> Message-ID: <1140210657.6735.17.camel@localhost.localdomain> On Fri, 2006-02-17 at 15:41 -0500, Mike A. Harris wrote: > Warren Togami wrote: > > Hi Dan, > > > > I am supposed to ask you about your opinion about how to ask buildsys to > > build for specific archs for a kernel module. I believe that we should > > use ExclusiveArch with explicit listed archs, and buildsys should loop > > through each listed arch and build each. This would also work for > > multi-arch performance intensive userspace packages similar to what we > > do in glibc or openssl in Core. [1] Contributors may also list any > > other arch like s390x or alpha in ExclusiveArch, and buildsys simply > > ignores it if it isn't supported in Extras. > > > > I believe this is the best approach to take for Extras, because Core has > > been similar to this for many years. Do you have any opinion on this, > > or know of a better way to handle this? > > > > Here is some related discussion: > > the problem with exclusivearch is that let's say you list i586 > > i686 x86_64 ppc and someone wants to rebuild it for alpha -> won't > > build, even if the package/software would work > > scop, you can include other archs in the list, buildsys can > > simply ignore Extras non-supported archs. > > warren, it's abuse of the tag nevertheless > > scop, I'd personally disagree, but either way this is very > > simple and it already behaves similar to this in Core literally forever. > > more importantly the simplicity of this allows us to move > > forward sooner > > (assuming dcbw doesn't have a better idea) > > that I agree with > On a side note, except under very specific circumstances, do not > use "i386" or "i686" as an architecture to list in ExclusiveArch > or Excludearch. Use the rpm macro "%{ix86}" instead, as that > expands to all existing architecture aliases past, present and > future. Only use i386 or i686 et al. if you really do want to > intentionally restrict the package from being built on other x86 > compatible architectures, such as i486 for example. Right, here's my take. In the end, it might just be best to leave this up to the buildsystem, since there's a "canonical" list of arches that a particular Distro maintains. There's two factors here: 1) The list of arches a _distro_ supports 2) The list of arches a particular package supports The two are not identical. And even though, for example, all of FC3 was compiled for ppc internally, the original FC3 was not released for ppc at all, and so therefore kernel modules should be released for FC3/ppc either. Control over the package set that the distro supports is up to the distro maintainers and the build system. So what I think we should end up doing here is to: a) Let packages do whatever the hell they want with their Exclusive, Exclude, BuildArch tags, including using %{ix86} as Mike suggests b) Have the buildsystem recognize kmod packages somehow (which we have to do anyway), then filter kmod packages through a "supported" list of sub-arches, including i586, i686, x86_64, ppc, athlon. There's some support for this already in the buildsystem. This puts a minimum burden on maintainers since lots of this arch stuff is black magic anyway, and should probably be kept in the same place. The nobody gets confused about arches, and there doesn't need to be a 50ft long wiki page about it. Of course, all this is completely separate from the whole xen/hugemem/smp/up fiasco, and much simpler IMHO than those. Let me know what you think. Dan From williams at redhat.com Fri Feb 24 23:02:37 2006 From: williams at redhat.com (Clark Williams) Date: Fri, 24 Feb 2006 17:02:37 -0600 Subject: bugzilla #164441 (mock-helper and basedir) Message-ID: <1140822157.6364.57.camel@localhost.localdomain> Hi all, I've been looking at how to change mock so that the build chroot's can be placed at arbitrary locations (as opposed to the current hardcoded /var/lib/mock). The main reason I want to do this is that I build target root filesystems for embedded targets in our compile farm and we use a big NFS filesystem as our main storage. I can access the build directory and our cross-toolchains from any system in our farm and nothing that's specific to a build is on any farm system (I can kickstart it at will and not lose anything). Yes, we've successfully redirected /var/lib/mock to the NFS storage with a mount --bind but I just find that kinda scary. I'm just not very confident in remounting an NFS filesystem to a local filesystem. The bottom line is that I'd like to remove dependencies on the build system's local filesystem and move it all to the NFS storage. So, the problem seems to be how to modify mock-helper so that it allows the move, yet doesn't become a handy-dandy-attack-point for some cracker wannabe. Can we say that the following is true? "The mock-helper program always begins execution *outside* of a chroot and is the only mechanism mock uses to *enter* a chroot." If so, then we can check to insure that the intended chroot directory: 1. Is not '/' 2. Is not a symlink to '/' 3. Is not in a list of special directories (/bin, /sbin, /lib, etc.) Are we "safe" if we insure that whatever command is executed by mock-helper is executed in a chroot directory that does not contain a system critical component? Clark -- Clark Williams -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 191 bytes Desc: This is a digitally signed message part URL: From dcbw at redhat.com Fri Feb 24 23:10:24 2006 From: dcbw at redhat.com (Dan Williams) Date: Fri, 24 Feb 2006 18:10:24 -0500 Subject: bugzilla #164441 (mock-helper and basedir) In-Reply-To: <1140822157.6364.57.camel@localhost.localdomain> References: <1140822157.6364.57.camel@localhost.localdomain> Message-ID: <1140822625.3579.8.camel@dhcp83-74.boston.redhat.com> On Fri, 2006-02-24 at 17:02 -0600, Clark Williams wrote: > Hi all, > > I've been looking at how to change mock so that the build chroot's can > be placed at arbitrary locations (as opposed to the current > hardcoded /var/lib/mock). The main reason I want to do this is that I > build target root filesystems for embedded targets in our compile farm > and we use a big NFS filesystem as our main storage. I can access the > build directory and our cross-toolchains from any system in our farm and > nothing that's specific to a build is on any farm system (I can > kickstart it at will and not lose anything). Yes, we've successfully > redirected /var/lib/mock to the NFS storage with a mount --bind but I > just find that kinda scary. I'm just not very confident in remounting an > NFS filesystem to a local filesystem. The bottom line is that I'd like > to remove dependencies on the build system's local filesystem and move > it all to the NFS storage. I can't speak for Seth, but I'm fairly sure the answer here is "use bind mounts". Seth is very big on the FHS, and in the FHS, that's where the buildroots and other stuff for mock are supposed to live. I've asked about this long ago, when starting to use mock for the build system, and that was the answer. Seriously, bind mounts are neither complicated, nor problematic, though I haven't done a ton of testing with NFS. The other question is why you want the buildroots on NFS in the first place. Unless you have blazingly fast storage, it's likely that local SCSI disk will be quite a bit faster than NFS. Since creating buildroot and doing the actual builds involves _quite_ a lot of IO, the same issue applies; local storage is likely better unless you've got really a fast path to the shared storage. Dan From williams at redhat.com Sat Feb 25 04:38:29 2006 From: williams at redhat.com (Clark Williams) Date: Fri, 24 Feb 2006 22:38:29 -0600 Subject: bugzilla #164441 (mock-helper and basedir) In-Reply-To: <1140822625.3579.8.camel@dhcp83-74.boston.redhat.com> References: <1140822157.6364.57.camel@localhost.localdomain> <1140822625.3579.8.camel@dhcp83-74.boston.redhat.com> Message-ID: <1140842309.17460.50.camel@localhost.localdomain> On Fri, 2006-02-24 at 18:10 -0500, Dan Williams wrote: > > I can't speak for Seth, but I'm fairly sure the answer here is "use bind > mounts". Seth is very big on the FHS, and in the FHS, that's where the > buildroots and other stuff for mock are supposed to live. I've asked > about this long ago, when starting to use mock for the build system, and > that was the answer. Seriously, bind mounts are neither complicated, > nor problematic, though I haven't done a ton of testing with NFS. > I didn't mean to imply that I'm not confident of my ability to *use* bind mounts, just that I'm suspicious of remounting a network filesystem. Fifteen or so years of wrestling with NFS have made me leery of layering anything on top of it. > The other question is why you want the buildroots on NFS in the first > place. Unless you have blazingly fast storage, it's likely that local > SCSI disk will be quite a bit faster than NFS. Since creating buildroot > and doing the actual builds involves _quite_ a lot of IO, the same issue > applies; local storage is likely better unless you've got really a fast > path to the shared storage. The main reason I'd like to have an arbitrary location for buildroots is that in our farm system, multiple builds may be occurring on a compile host at the same time (e.g. one person building for a MIPS and another building for ARM). Having them both collide in /var/lib/mock wouldn't be fun. Since each architecture has it's own space on the NFS server (which is connected via gigabit ethernet to the compile systems), if you stay in your own sandbox, you don't collide with anyone else. I'm not sure I want to use mock in the same way it's used by Plague (i.e. clean the chroot, populate it, build one package). Since we currently use a custom set of rpmmacros, rpmrc and a few command line definitions to cross-build our RPMs, I'd rather use mock to setup one chroot for an architecture and then build multiple packages out of it. I'm not certain of that now, but that's the way I'm leaning. Clark -- Clark Williams -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 191 bytes Desc: This is a digitally signed message part URL: From dcbw at redhat.com Sun Feb 26 02:03:11 2006 From: dcbw at redhat.com (Dan Williams) Date: Sat, 25 Feb 2006 21:03:11 -0500 Subject: bugzilla #164441 (mock-helper and basedir) In-Reply-To: <1140842309.17460.50.camel@localhost.localdomain> References: <1140822157.6364.57.camel@localhost.localdomain> <1140822625.3579.8.camel@dhcp83-74.boston.redhat.com> <1140842309.17460.50.camel@localhost.localdomain> Message-ID: <1140919392.2570.5.camel@localhost.localdomain> On Fri, 2006-02-24 at 22:38 -0600, Clark Williams wrote: > On Fri, 2006-02-24 at 18:10 -0500, Dan Williams wrote: > > > > I can't speak for Seth, but I'm fairly sure the answer here is "use bind > > mounts". Seth is very big on the FHS, and in the FHS, that's where the > > buildroots and other stuff for mock are supposed to live. I've asked > > about this long ago, when starting to use mock for the build system, and > > that was the answer. Seriously, bind mounts are neither complicated, > > nor problematic, though I haven't done a ton of testing with NFS. > > > > I didn't mean to imply that I'm not confident of my ability to *use* > bind mounts, just that I'm suspicious of remounting a network > filesystem. Fifteen or so years of wrestling with NFS have made me leery > of layering anything on top of it. > > > The other question is why you want the buildroots on NFS in the first > > place. Unless you have blazingly fast storage, it's likely that local > > SCSI disk will be quite a bit faster than NFS. Since creating buildroot > > and doing the actual builds involves _quite_ a lot of IO, the same issue > > applies; local storage is likely better unless you've got really a fast > > path to the shared storage. > > The main reason I'd like to have an arbitrary location for buildroots is > that in our farm system, multiple builds may be occurring on a compile > host at the same time (e.g. one person building for a MIPS and another > building for ARM). Having them both collide in /var/lib/mock wouldn't be > fun. Since each architecture has it's own space on the NFS server (which > is connected via gigabit ethernet to the compile systems), if you stay > in your own sandbox, you don't collide with anyone else. Ok; if you want different directories for different architectures, there are two ways to do it: 1) in the mock config file for the target, change the 'root' option to include your architecture. Then pass '-r ' on the mock command line 2) use the --uniqueext= option on the mock command line to futher unique-ify your buildroot Dan From skvidal at linux.duke.edu Sun Feb 26 16:01:45 2006 From: skvidal at linux.duke.edu (seth vidal) Date: Sun, 26 Feb 2006 11:01:45 -0500 Subject: bugzilla #164441 (mock-helper and basedir) In-Reply-To: <1140842309.17460.50.camel@localhost.localdomain> References: <1140822157.6364.57.camel@localhost.localdomain> <1140822625.3579.8.camel@dhcp83-74.boston.redhat.com> <1140842309.17460.50.camel@localhost.localdomain> Message-ID: <1140969705.1723.5.camel@cutter> > The main reason I'd like to have an arbitrary location for buildroots is > that in our farm system, multiple builds may be occurring on a compile > host at the same time (e.g. one person building for a MIPS and another > building for ARM). Having them both collide in /var/lib/mock wouldn't be > fun. Since each architecture has it's own space on the NFS server (which > is connected via gigabit ethernet to the compile systems), if you stay > in your own sandbox, you don't collide with anyone else. will the rpmdb's get created properly on nfs? I thought they went a bit nutty? > I'm not sure I want to use mock in the same way it's used by Plague > (i.e. clean the chroot, populate it, build one package). Since we > currently use a custom set of rpmmacros, rpmrc and a few command line > definitions to cross-build our RPMs, I'd rather use mock to setup one > chroot for an architecture and then build multiple packages out of it. > I'm not certain of that now, but that's the way I'm leaning. you should look at the mock config file - you can put your rpmmacros and rpmrc specifications in to the config file and it will be pushed into the chroot automatically. -sv From skvidal at linux.duke.edu Sun Feb 26 16:03:03 2006 From: skvidal at linux.duke.edu (seth vidal) Date: Sun, 26 Feb 2006 11:03:03 -0500 Subject: bugzilla #164441 (mock-helper and basedir) In-Reply-To: <1140822625.3579.8.camel@dhcp83-74.boston.redhat.com> References: <1140822157.6364.57.camel@localhost.localdomain> <1140822625.3579.8.camel@dhcp83-74.boston.redhat.com> Message-ID: <1140969783.1723.8.camel@cutter> > I can't speak for Seth, but I'm fairly sure the answer here is "use bind > mounts". Seth is very big on the FHS, and in the FHS, that's where the > buildroots and other stuff for mock are supposed to live. I've asked > about this long ago, when starting to use mock for the build system, and > that was the answer. Seriously, bind mounts are neither complicated, > nor problematic, though I haven't done a ton of testing with NFS. The fhs is our friend. :) However I don't think we're suggesting changing the default location for mock just making it possible for someone else to do something like that. And if we can do that securely then, sure. -sv From wtogami at redhat.com Sun Feb 26 20:38:16 2006 From: wtogami at redhat.com (Warren Togami) Date: Sun, 26 Feb 2006 15:38:16 -0500 Subject: bugzilla #164441 (mock-helper and basedir) In-Reply-To: <1140822157.6364.57.camel@localhost.localdomain> References: <1140822157.6364.57.camel@localhost.localdomain> Message-ID: <440211B8.3090708@redhat.com> My main thought here is, what does putting your buildroots on NFS gain you? Do you realize that one of the main reasons beehive had failures in past years was due to NFS problems? Warren Togami wtogami at redhat.com From dsmith at redhat.com Mon Feb 27 14:45:30 2006 From: dsmith at redhat.com (David Smith) Date: Mon, 27 Feb 2006 08:45:30 -0600 Subject: bugzilla #164441 (mock-helper and basedir) In-Reply-To: <1140969705.1723.5.camel@cutter> References: <1140822157.6364.57.camel@localhost.localdomain> <1140822625.3579.8.camel@dhcp83-74.boston.redhat.com> <1140842309.17460.50.camel@localhost.localdomain> <1140969705.1723.5.camel@cutter> Message-ID: <1141051530.20489.27.camel@dhcp-2.hsv.redhat.com> On Sun, 2006-02-26 at 11:01 -0500, seth vidal wrote: > > The main reason I'd like to have an arbitrary location for buildroots is > > that in our farm system, multiple builds may be occurring on a compile > > host at the same time (e.g. one person building for a MIPS and another > > building for ARM). Having them both collide in /var/lib/mock wouldn't be > > fun. Since each architecture has it's own space on the NFS server (which > > is connected via gigabit ethernet to the compile systems), if you stay > > in your own sandbox, you don't collide with anyone else. > > will the rpmdb's get created properly on nfs? I thought they went a bit > nutty? I work with Clark so I'll throw in a bit here. Storing a rpmdb on NFS does OK for awhile, then goes wonky (hope that's not too technical for you ;-). Since our target rpmdb's were fairly transitory, it wasn't a huge deal, just annoying. It finally got *too* annoying one day and I found a workaround (with the help of every programmer's friend - google). I added the following to our rpmmacros file: # We're adding the private token. This should help when storing the db # on a NFS file system. %__dbi_other %{?_tmppath:tmpdir=%{_tmppath}} %{?__dbi_cdb} \ private Adding 'private' to %__dbi_other causes rpm to use the DB_PRIVATE flag. Here's a description of DB_PRIVATE from : ================================= If the DB_PRIVATE flag is specified to the DB_ENV->open method, regions are created in per-process heap memory; that is, memory returned by malloc(3). This flag should not be specified if more than a single process is accessing the environment because it is likely to cause database corruption and unpredictable behavior. For example, if both a server application and Berkeley DB utilities (for example, db_archive, db_checkpoint or db_stat) are expected to access the environment, the DB_PRIVATE flag should not be specified. ... If no memory-related flags are specified to DB_ENV->open, memory backed by the filesystem is used to store the regions. On UNIX systems, the Berkeley DB library will use the POSIX mmap interface. If mmap is not available, the UNIX shmget interfaces may be used instead, if they are available. ================================= After adding "private", our use of rpmdbs on NFS has been rock-solid. (However, your mileage may vary.) > > I'm not sure I want to use mock in the same way it's used by Plague > > (i.e. clean the chroot, populate it, build one package). Since we > > currently use a custom set of rpmmacros, rpmrc and a few command line > > definitions to cross-build our RPMs, I'd rather use mock to setup one > > chroot for an architecture and then build multiple packages out of it. > > I'm not certain of that now, but that's the way I'm leaning. > > you should look at the mock config file - you can put your rpmmacros and > rpmrc specifications in to the config file and it will be pushed into > the chroot automatically. > > -sv Hmm, didn't know you could specify different rpmmacros and rpmrc files in the config file. I'll look into that. The only hold up for us might be that we do both native and cross compiles during our full builds and we normally use different rpmmacros/rpmrc files based on what we're compiling for. -- David Smith dsmith at redhat.com Red Hat, Inc. http://www.redhat.com 256.217.0141 (direct) 256.837.0057 (fax) From williams at redhat.com Mon Feb 27 15:18:40 2006 From: williams at redhat.com (Clark Williams) Date: Mon, 27 Feb 2006 09:18:40 -0600 Subject: bugzilla #164441 (mock-helper and basedir) In-Reply-To: <1140969783.1723.8.camel@cutter> References: <1140822157.6364.57.camel@localhost.localdomain> <1140822625.3579.8.camel@dhcp83-74.boston.redhat.com> <1140969783.1723.8.camel@cutter> Message-ID: <1141053520.21956.46.camel@localhost.localdomain> On Sun, 2006-02-26 at 11:03 -0500, seth vidal wrote: > > The fhs is our friend. :) However I don't think we're suggesting > changing the default location for mock just making it possible for > someone else to do something like that. And if we can do that securely > then, sure. > You've hit the nail on the head. I realize that some folks think badly of NFS and don't want to use it. That's fine (yeah, I'm not in love with NFS, but I still want to use it), but what if someone's using FibreChannel or InfiniBand attached storage? My argument that we should be able to move the root is still valid. So, back to my original question: if we *exclude* certain directories as candidates for chroot'ing, can we securely move the root? I'm thinking of something like the attached patch (minus the #ifdefs). Clark -- Clark Williams -------------- next part -------------- A non-text attachment was scrubbed... Name: arbitrary.diff Type: text/x-patch Size: 1321 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 191 bytes Desc: This is a digitally signed message part URL: From williams at redhat.com Mon Feb 27 15:58:18 2006 From: williams at redhat.com (Clark Williams) Date: Mon, 27 Feb 2006 09:58:18 -0600 Subject: bugzilla #164441 (mock-helper and basedir) In-Reply-To: <1141053520.21956.46.camel@localhost.localdomain> References: <1140822157.6364.57.camel@localhost.localdomain> <1140822625.3579.8.camel@dhcp83-74.boston.redhat.com> <1140969783.1723.8.camel@cutter> <1141053520.21956.46.camel@localhost.localdomain> Message-ID: <1141055898.21956.53.camel@localhost.localdomain> On Mon, 2006-02-27 at 09:18 -0600, Clark Williams wrote: > So, back to my original question: if we *exclude* certain directories as > candidates for chroot'ing, can we securely move the root? I'm thinking > of something like the attached patch (minus the #ifdefs). Grrr. That's what I get for doing something in a hurry. I sent the wrong patch and I didn't inline it. Sigh. Here's the patch I *meant* to send: Index: mock-helper.c =================================================================== RCS file: /cvs/fedora/mock/src/mock-helper.c,v retrieving revision 1.7 diff -u -r1.7 mock-helper.c --- mock-helper.c 14 Jul 2005 18:00:26 +++ mock-helper.c 27 Feb 2006 15:54:09 @@ -55,6 +55,12 @@ exit (1); } +#ifdef ARBITRARY_CHROOT +const char *disallowed[] = {"/bin", "/sbin/", "/usr", "/lib", + "/boot", "/dev", "/etc", "/var" +}; +#endif + /* * perform checks on the given dir * - is the given dir under the allowed hierarchy ? @@ -68,9 +74,21 @@ char last; int retval; +#ifdef ARBITRARY_CHROOT + int i; + + if (strncmp(given, "/", 1) != 0) + error("can't chroot to '/'"); + + for (i=0; i < sizeof(disallowed) / sizeof(char *); i++) { + if (strncmp(given, disallowed[i], strlen(given)) != 0) + error("%s: chroot not allowed\n", disallowed[i]); + } +#else /* does given start with allowed ? */ if (strncmp (given, allowed, strlen (allowed)) != 0) error ("%s: not under allowed directory (%s)", given, allowed); +#endif /* does it try to fool us by using .. ? */ if (strstr (given, "..") != 0) -- Clark Williams -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 191 bytes Desc: This is a digitally signed message part URL: From dsmith at redhat.com Mon Feb 27 17:52:23 2006 From: dsmith at redhat.com (David Smith) Date: Mon, 27 Feb 2006 11:52:23 -0600 Subject: bugzilla #164441 (mock-helper and basedir) In-Reply-To: <440211B8.3090708@redhat.com> References: <1140822157.6364.57.camel@localhost.localdomain> <440211B8.3090708@redhat.com> Message-ID: <1141062743.20489.84.camel@dhcp-2.hsv.redhat.com> On Sun, 2006-02-26 at 15:38 -0500, Warren Togami wrote: > My main thought here is, what does putting your buildroots on NFS gain > you? Do you realize that one of the main reasons beehive had failures > in past years was due to NFS problems? > > Warren Togami > wtogami at redhat.com Warren, Perhaps I should better explain what Clark and I are really doing, and what we hope to gain by using mock so you can see where we are coming from (and where we want to go). And no, I didn't realize beehive had NFS problems since I've never used beehive... Our group does embedded Linux ports, cross-compiling the specific architecture on an x86 system. We don't do native compiles because we typically don't create native compilers and the embedded systems are typically quite slow. For sometime we used a home-grown system of Makefiles and source downloaded off the net. This system was quite fragile in several ways, both at build time and at run time. We got frustrated with that system and looked around for something else. I really wanted to use Fedora SRPMS as a base to start with. I looked around and found an old system that H.J. Lu had written to cross-compile part of RHL7.3 for mips. Clark and I took that system and modified and extended it. Here's how it currently works. We start with unmodified FCx SRPMs. Each SRPM has a directory where we store our modified spec file (and patches if necessary) to enable cross-compiling. We start with an empty target arch buildroot and cross compile and install packages. As you can imagine build order is quite important at first (for instance glibc is one of the first packages we build). We build and install the packages other packages need, then build the remaining packages. We currently have about 120 or so packages in our distribution (typically taking 4-6 hours to do a complete build). Warren, you asked what does putting our buildroots on NFS gain us. I guess the main answer is machine independence. We have several build farm systems all mounting the same NFS shares. If a build system becomes slow or goes down, I can just switch machines and restart the build at the SRPM that died. Note that we don't support building multiple RPMs simultaneously on the same buildroot (either on the same machine or on multiple machines). So, what do we hope to gain by using mock? We hope to make the cross- compile porting process easier and we hope to gain even more machine independence. Since our customers build on RHEL, part of getting the packages to cross-compile entails getting the FCx packages to build on RHELy. For instance, we build FC3 packages on RHEL3, FC4 packages on RHEL4. Unfortunately, when we're done, we've typically mangled the packages to only build on that version of RHEL (because of autoconf/automake/etc. changes). If instead we built the FCx SRPM in a mock FCx chroot, that part of the port effort wouldn't be needed. In addition, if we're building in a mock FCx chroot, the host OS doesn't really matter. -- David Smith dsmith at redhat.com Red Hat, Inc. http://www.redhat.com 256.217.0141 (direct) 256.837.0057 (fax) From sopwith at redhat.com Mon Feb 27 17:55:57 2006 From: sopwith at redhat.com (Elliot Lee) Date: Mon, 27 Feb 2006 12:55:57 -0500 (EST) Subject: bugzilla #164441 (mock-helper and basedir) In-Reply-To: <1141062743.20489.84.camel@dhcp-2.hsv.redhat.com> References: <1140822157.6364.57.camel@localhost.localdomain> <440211B8.3090708@redhat.com> <1141062743.20489.84.camel@dhcp-2.hsv.redhat.com> Message-ID: > On Sun, 2006-02-26 at 15:38 -0500, Warren Togami wrote: > > My main thought here is, what does putting your buildroots on NFS gain > > you? Do you realize that one of the main reasons beehive had failures > > in past years was due to NFS problems? Which turned out to be completely unrelated to NFS, hence the running joke :) Best, -- Elliot From dsmith at redhat.com Mon Feb 27 22:14:25 2006 From: dsmith at redhat.com (David Smith) Date: Mon, 27 Feb 2006 16:14:25 -0600 Subject: adding packages to mock chroot Message-ID: <1141078465.20489.97.camel@dhcp-2.hsv.redhat.com> When you create a mock chroot, all the packages you want to install in it need to be available from a yum repostory. I've got the situation where I'm trying to bootstrap a bit, so, I'd like to be able to: - build a chroot - build a small set of packages - add those packages to the chroot to be used by future builds Off the top of my head, possibilities include: (1) creating a base chroot, building the rpms, add the new packages to a yum repo, creating a second chroot that contains the new packages (2) creating the base chroot, building the rpms, editing the buildroots.xml file and adding in the new packages, updating the yum repo, then run "mock init" again (hoping that the new packages would get added to the chroot) Are there better ways of doing this? Thanks. -- David Smith dsmith at redhat.com Red Hat, Inc. http://www.redhat.com 256.217.0141 (direct) 256.837.0057 (fax) From sheltren at cs.ucsb.edu Mon Feb 27 23:38:38 2006 From: sheltren at cs.ucsb.edu (Jeff Sheltren) Date: Mon, 27 Feb 2006 19:38:38 -0400 Subject: adding packages to mock chroot In-Reply-To: <1141078465.20489.97.camel@dhcp-2.hsv.redhat.com> References: <1141078465.20489.97.camel@dhcp-2.hsv.redhat.com> Message-ID: <76A4CA97-3F19-4DF0-9373-198B17C742CE@cs.ucsb.edu> On Feb 27, 2006, at 6:14 PM, David Smith wrote: > When you create a mock chroot, all the packages you want to install in > it need to be available from a yum repostory. I've got the situation > where I'm trying to bootstrap a bit, so, I'd like to be able to: > > - build a chroot > - build a small set of packages > - add those packages to the chroot to be used by future builds > > Off the top of my head, possibilities include: > > (1) creating a base chroot, building the rpms, add the new packages > to a > yum repo, creating a second chroot that contains the new packages > > (2) creating the base chroot, building the rpms, editing the > buildroots.xml file and adding in the new packages, updating the yum > repo, then run "mock init" again (hoping that the new packages > would get > added to the chroot) > > Are there better ways of doing this? > > Thanks. > > David, I may be missing something, but I think the easiest way would be to create a local directory on your build machine which contains the packages you want included. Run 'createrepo' there, and then reference that directory (now a yum repo) using file:///path/to/rpms in your /etc/mock/whatever.conf file(s). This way you have a local yum repo you can add your "special" packages to. There's no need to edit the buildroots.xml file unless you want these to *always* be installed - otherwise they'll just get pulled in as needed for dependencies. -Jeff From jkeating at j2solutions.net Tue Feb 28 00:34:26 2006 From: jkeating at j2solutions.net (Jesse Keating) Date: Mon, 27 Feb 2006 16:34:26 -0800 Subject: adding packages to mock chroot In-Reply-To: <1141078465.20489.97.camel@dhcp-2.hsv.redhat.com> References: <1141078465.20489.97.camel@dhcp-2.hsv.redhat.com> Message-ID: <1141086866.21520.78.camel@ender> On Mon, 2006-02-27 at 16:14 -0600, David Smith wrote: > > Are there better ways of doing this? > Since mock uses a fresh chroot each time, you should make the packages you build available in a localized yum repo using createrepo. Then when you build the next package, your build root creation can pull from this local yum repo to satisfy the BuildRequires of your given package. -- Jesse Keating RHCE (geek.j2solutions.net) Fedora Legacy Team (www.fedoralegacy.org) GPG Public Key (geek.j2solutions.net/jkeating.j2solutions.pub) -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 191 bytes Desc: This is a digitally signed message part URL: