From skvidal at linux.duke.edu Fri May 5 02:31:01 2006 From: skvidal at linux.duke.edu (seth vidal) Date: Thu, 04 May 2006 22:31:01 -0400 Subject: [Fedora-sysadmin-list] rpm on build machines In-Reply-To: <1146792575.7031.1.camel@ender> References: <1146792575.7031.1.camel@ender> Message-ID: <1146796261.16918.0.camel@cutter> On Thu, 2006-05-04 at 21:29 -0400, Jesse Keating wrote: > Due to the bug we found in rpm, tickled by the glibc package in rawhide, > we'll need to get an updated rpm onto the build machines. > > We have one package for RHEL4/CentOS4 that skips over the bug, but Paul > Nasrat is working on a real fix. Until we get something on the build > machines no builds on development can happen. > > I can provide the rpms that skip over the bug for now, your call. What bug is this? Does it have a bug #? -sv From Matt_Domsch at dell.com Mon May 8 16:45:13 2006 From: Matt_Domsch at dell.com (Matt Domsch) Date: Mon, 8 May 2006 11:45:13 -0500 Subject: SLES9 and SLES10 enablement patches committed Message-ID: <20060508164512.GA20026@lists.us.dell.com> I've committed the two patches below to mock in CVS, which enables mock to be used to create build environments for Novell/SuSE SLES 9 and SLES10. In their environments, some files already exist so mock shouldn't create them; useradd needs the -m flag to create the users's home dir; we moved several iterations of chmod/chroot invocations out of the loop, and instead do a 'chown -R' once (saves calling chroot as often); and finally make sure files in /etc/ that should be writeable are. Patches are by Michael E. Brown and myself. Thanks, Matt -- Matt Domsch Software Architect Dell Linux Solutions linux.dell.com & www.dell.com/linux Linux on Dell mailing lists @ http://lists.us.dell.com --- /usr/bin/mock 2006-04-19 14:31:47.000000000 -0500 +++ ./mock 2006-04-19 14:32:16.000000000 -0500 @@ -566,8 +566,9 @@ for item in [os.path.join(self.rootdir, 'etc', 'mtab'), os.path.join(self.rootdir, 'etc', 'fstab'), os.path.join(self.rootdir, 'var', 'log', 'yum.log')]: - fo = open(item, 'w') - fo.close() + if not os.path.exists(item): + fo = open(item, 'w') + fo.close() # write in yum.conf into chroot yumconf = os.path.join(self.rootdir, 'etc', 'yum.conf') @@ -589,7 +590,7 @@ if not os.path.exists(self.rootdir + self.homedir): if not os.path.exists(os.path.join(self.rootdir, 'usr/sbin/useradd')): raise RootError, "Could not find useradd in chroot, maybe the install failed?" - cmd = '/usr/sbin/useradd -u %s -d %s %s' % (self.config['chrootuid'], + cmd = '/usr/sbin/useradd -m -u %s -d %s %s' % (self.config['chrootuid'], self.homedir, self.config['chrootuser']) self.do_chroot(cmd, fatal = True) @@ -604,9 +605,11 @@ for subdir in ('RPMS', 'SRPMS', 'SOURCES', 'SPECS', 'BUILD', 'originals'): cmd = "mkdir -p %s/%s" % (self.builddir, subdir) self.do_chroot(cmd, fatal = True) - cmd = "chown %s.%s %s/%s" % (self.config['chrootuser'], - self.config['chrootgroup'], self.builddir, subdir) - self.do_chroot(cmd, fatal = True) + + # change ownership so we can write to build home dir + cmd = "chown -R %s.%s %s" % (self.config['chrootuser'], + self.config['chrootgroup'], self.homedir) + self.do_chroot(cmd, fatal = True) # rpmmacros default macrofile_out = '%s%s/.rpmmacros' % (self.rootdir, self.homedir) --- mock 2006-04-24 10:04:14.000000000 -0500 +++ mock.new 2006-05-05 16:16:09.000000000 -0500 @@ -579,6 +579,13 @@ # files in /etc that need doing filedict = self.config['files'] for key in filedict: + # ensure permisssions + if os.path.exists( "%s%s" % (self.rootdir, key) ): + cmd = "chown %s.%s %s" % (self.config['chrootuser'], + self.config['chrootgroup'], "%s" % key) + self.do_chroot(cmd, fatal = True) + + # write file fn = '%s%s' % (self.rootdir, key) fo = open(fn, 'w') fo.write(filedict[key]) From cweyl at alumni.drew.edu Mon May 8 16:57:48 2006 From: cweyl at alumni.drew.edu (Chris Weyl) Date: Mon, 8 May 2006 09:57:48 -0700 Subject: SLES9 and SLES10 enablement patches committed In-Reply-To: <20060508164512.GA20026@lists.us.dell.com> References: <20060508164512.GA20026@lists.us.dell.com> Message-ID: <7dd7ab490605080957x1ef89b82ub037407058004c3@mail.gmail.com> On 5/8/06, Matt Domsch wrote: > I've committed the two patches below to mock in CVS, which enables > mock to be used to create build environments for Novell/SuSE SLES 9 > and SLES10. In their environments, some files already exist so mock > shouldn't create them; useradd needs the -m flag to create the users's > home dir; we moved several iterations of chmod/chroot invocations out of the loop, and > instead do a 'chown -R' once (saves calling chroot as often); and > finally make sure files in /etc/ that should be writeable are. > > Patches are by Michael E. Brown and myself. Thanks guys! Do you happen to have a working set of config files to do these builds with mock? (Or advice along those lines) -Chris From Matt_Domsch at dell.com Mon May 8 17:05:19 2006 From: Matt_Domsch at dell.com (Matt Domsch) Date: Mon, 8 May 2006 12:05:19 -0500 Subject: SLES9 and SLES10 enablement patches committed In-Reply-To: <7dd7ab490605080957x1ef89b82ub037407058004c3@mail.gmail.com> References: <20060508164512.GA20026@lists.us.dell.com> <7dd7ab490605080957x1ef89b82ub037407058004c3@mail.gmail.com> Message-ID: <20060508170519.GC20026@lists.us.dell.com> On Mon, May 08, 2006 at 09:57:48AM -0700, Chris Weyl wrote: > On 5/8/06, Matt Domsch wrote: > >I've committed the two patches below to mock in CVS, which enables > >mock to be used to create build environments for Novell/SuSE SLES 9 > >and SLES10. In their environments, some files already exist so mock > >shouldn't create them; useradd needs the -m flag to create the users's > >home dir; we moved several iterations of chmod/chroot invocations out of > >the loop, and > >instead do a 'chown -R' once (saves calling chroot as often); and > >finally make sure files in /etc/ that should be writeable are. > > > >Patches are by Michael E. Brown and myself. > > Thanks guys! > > Do you happen to have a working set of config files to do these builds > with mock? (Or advice along those lines) http://linux.dell.com/files/mock/ has config files and buildgroups files. We haven't yet made the switch to the new buildsys-build.spec model for pulling in dependencies. -- Matt Domsch Software Architect Dell Linux Solutions linux.dell.com & www.dell.com/linux Linux on Dell mailing lists @ http://lists.us.dell.com From skvidal at linux.duke.edu Wed May 10 19:21:12 2006 From: skvidal at linux.duke.edu (seth vidal) Date: Wed, 10 May 2006 15:21:12 -0400 Subject: buildsys-build and buildgroups location Message-ID: <1147288873.15425.28.camel@cutter> Hi folks, I was looking at how things are arranged with the buildgroups repository for the groups installed by mock for the buildsys. I was going to move the files used from fedoraproject.org to buildsys.fedoraproject.org then put a redirect on fedoraproject.org to point to buildsys.fedoraproject.org for that path. any objections to that? Also, I've put it off for the last week or so. I'm building the buildsys-build pkgs which will be used by mock 0.5 for creating the build environments we need. cool? -sv From Michael_E_Brown at Dell.com Wed May 10 19:35:51 2006 From: Michael_E_Brown at Dell.com (Michael_E_Brown at Dell.com) Date: Wed, 10 May 2006 14:35:51 -0500 Subject: [RFC, PATCH] Build multiple srpms Message-ID: <35C9A9D68AB3FA4AB63692802656D9EC010F4372@ausx3mps303.aus.amer.dell.com> Here is a patch for review. I have made the "obvious" changes in order to be able to do the following: mock -r CFG srpm1 srpm2 srpm3 srpm4 Limitations: - There are no checks for dependencies between the srpms, so they should be independent. (no re-ordering of SRPMS) - Building one SRPM can leave the buildroot in an inconsistent state for the next SRPM. (no clean of chroot between builds) - Failure to build one SRPM stops the whole process. - Resulting RPMs are not installed into the build environment for use by subsequent SRPMS. Reasoning: One problem I have been having with mock is speed. We are using mock to build for 16 distinct configurations. Doing prep on each configuration was taking a minute and a half to two minutes. As an optimization, our mock wrapper script was doing a single prep per chroot and then using --no-clean, but this complicates our wrapper script. I am building a set of related RPMS, so I don have much concern about cross-pollution. This simple patch simplifies my wrapper script considerably. Future: One future direction I'd like to take this is a parallel mock that can prep/build multiple configurations at the same time to try to amortize the cost of prep stage by running prep of one environment in parallel with build of a different configuration. Currently, it is taking me an hour to build 16 sets of RPMS (5 SRPMS per set), and I am hoping to get this down. We have already implemented squid and some other measures to try to speed things up. -- Michael <> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: mock-multi.patch Type: application/octet-stream Size: 2223 bytes Desc: mock-multi.patch URL: From cweyl at alumni.drew.edu Wed May 10 20:18:24 2006 From: cweyl at alumni.drew.edu (Chris Weyl) Date: Wed, 10 May 2006 13:18:24 -0700 Subject: buildsys-build and buildgroups location In-Reply-To: <1147288873.15425.28.camel@cutter> References: <1147288873.15425.28.camel@cutter> Message-ID: <7dd7ab490605101318s3330f268wa4e0167e786bac2a@mail.gmail.com> On 5/10/06, seth vidal wrote: > cool? Works by me :) If you get a chance, drop us a note so I can start updating my mock configs when the groups repos is moved to buildsys.f.o? (Which I know the redirect will handle, but it's always nice to be prepared...) -Chris -- Chris Weyl Ex astris, scientia From andreas at bawue.net Wed May 10 20:26:42 2006 From: andreas at bawue.net (Andreas Thienemann) Date: Wed, 10 May 2006 22:26:42 +0200 (CEST) Subject: Patches for shared configurations items and specifying additional dependencies Message-ID: Hello List, as it seems to be patch submission time, I'm pushing two patches to the list, I had lying around for some time now. The first patch allows mock to parse a generic side-wide config file, where general configuration-items can be stored and later overridden by specific chroot-definitions. The second patch, is a bit different: While pushing the Fedora Core SRPMS through the mock buildsystem we noticed many SRPMS which do not correctly define their buildrequirements. Until the fedora core developer responsible for the package has updated the spec in cvs and pushed an updated srpm into rawhide, there is no way to make the package in question build in mock without having to change the .spec. Thus, in order to allow even these broken packages being built, we needed a way to work around these broken dependencies to see if anything else was missing. This is done by the second patch, which adds a more_buildreqs config option. If set the given dependencies are installed _in addition_ to the ones from the .spec file. While the second patch should not be used during normal use and is definetly not for the fedora-extras buildservers, it still is of interest to other parties, using mock. regards, andreas -------------- next part -------------- Provide generic config defaults. * Support a generic /etc/mock/defaults.cfg config file. The values defined in this file are overwritten by any definitions in the specific /etc/mock/CHROOT.cfg files. This part just introduces mock-0.4/etc/defaults.cfg and adds the last chunk in the patch to mock-0.4/mock.py. This patch was written in 2006 for bawue.net by Hans Ulrich Niedermann in 2006 after an idea by Andreas Thienemann. Testing and final clean-up was one by Andreas Thienemann. The source code this patch is based on has been downloaded from http://fedoraproject.org/projects/mock/releases/mock-0.4.tar.gz diff -Nurp mock-0.4.orig/etc/defaults.cfg mock-0.4.bawue/etc/defaults.cfg --- mock-0.4.orig/etc/defaults.cfg 1970-01-01 01:00:00.000000000 +0100 +++ mock-0.4.bawue/etc/defaults.cfg 2006-04-05 21:59:22.000000000 +0200 @@ -0,0 +1,8 @@ +# mock defaults +# +# Define default values here. +# These values are overwritten in the /etc/mock/CHROOT.cfg files. +# +# Example: +# +# config_opts['foo'] = bar diff -Nurp mock-0.4.orig/mock.py mock-0.4.bawue/mock.py --- mock-0.4.orig/mock.py 2005-07-26 22:33:52.000000000 +0200 +++ mock-0.4.bawue/mock.py 2006-04-06 06:59:51.000000000 +0200 @@ -683,6 +701,14 @@ def main(): if len(args) < 1: error("No srpm or command specified - nothing to do") sys.exit(50) + + # Read in the default values which can be overwritten + # with the more specific config being loaded below. + cfg = os.path.join(config_path, 'defaults.cfg') + if os.path.exists(cfg): + execfile(cfg) + else: + pass # not finding the defaults.cfg file is no error # read in the config file by chroot name if options.chroot.endswith('.cfg'): -------------- next part -------------- Provide extra build requirements for packages. * Introduce a new config variable 'more_buildreqs' which allows defining additional build dependencies for specific packages. This is useful for buggy SRPMs or SRPMs which compile some parts conditionally, depending on the software installed. This mechanism should not be used in normal operation. The extra build requirements can be configured like in the following example, in which we use the example of a fictitional SRPM file called 'ser-0.9.6-3'. # matches only the 'ser-0.9.6-3' package config_opts['more_buildreqs']['ser-0.9.6-3'] = [ 'smtp-devel >= 1.2', 'mysql-devel >= 5.0', 'png-devel < 0.1', 'tcsh' ] # matches all 'ser-0.9.6-*' packages config_opts['more_buildreqs']['ser-0.9.6'] = [ 'smtp-devel >= 1.2', 'mysql-devel >= 5.0', 'png-devel < 0.1', 'tcsh' ] # matches all 'ser-*-*' packages config_opts['more_buildreqs']['ser'] = [ 'smtp-devel >= 1.2', 'mysql-devel >= 5.0', 'png-devel < 0.1', 'tcsh' ] The most specific match is used, i.e. if requirements for 'ser-0.9.6-3' are defined, the definitions for 'ser-0.9.6' and 'ser' are ignored. Each time, /etc/mock/CHROOT.cfg is checked first, and only if there is no dependency definition here, the definition from /etc/mock/defaults.cfg is checked. This patch was written in 2006 for bawue.net by Hans Ulrich Niedermann in 2006 after an idea by Andreas Thienemann. Testing and final clean-up was done by Andreas Thienemann. The source code this patch is based on has been downloaded from http://fedoraproject.org/projects/mock/releases/mock-0.4.tar.gz diff -Nurp mock-0.4.orig/mock.py mock-0.4.bawue/mock.py --- mock-0.4.orig/mock.py 2005-07-26 22:33:52.000000000 +0200 +++ mock-0.4.bawue/mock.py 2006-04-06 06:59:51.000000000 +0200 @@ -278,7 +278,7 @@ class Root: hdr = rpmUtils.miscutils.hdrFromPackage(ts, srpm) # get text buildreqs - buildreqs = self._text_requires_from_hdr(hdr) + buildreqs = self._text_requires_from_hdr(hdr, srpm) arg_string = "" for item in buildreqs: @@ -469,7 +469,7 @@ class Root: return (ret, output) - def _text_requires_from_hdr(self, hdr): + def _text_requires_from_hdr(self, hdr, srpm): """take a header and hand back a unique'd list of the requires as strings""" @@ -487,6 +487,23 @@ class Root: req = rpmUtils.miscutils.formatRequire(n, v, f) reqlist.append(req) + # Extract SRPM name components - still not nice, shouldn't this + # be somewhere in the "hdr" parameter? + fname = os.path.split(str(srpm))[1] + name, ver, rel, epoch, arch = rpmUtils.miscutils.splitFilename(fname) + + # Add the 'more_buildreqs' for this SRPM (if defined) + for this_srpm in ['-'.join([name,ver,rel]), + '-'.join([name,ver]), + '-'.join([name]),]: + if self.config['more_buildreqs'].has_key(this_srpm): + more_reqs = self.config['more_buildreqs'][this_srpm] + if type(more_reqs) in (type(u''), type(''),): + more_reqs = [more_reqs] # be nice if we get a string + for req in more_reqs: + reqlist.append(req) + break + return rpmUtils.miscutils.unique(reqlist) def _prep_install(self): @@ -674,6 +691,7 @@ def main(): """ % config_opts['chroothome'] + config_opts['more_buildreqs'] = {} config_opts['files']['/etc/resolv.conf'] = "nameserver 192.168.1.1\n" config_opts['files']['/etc/hosts'] = "127.0.0.1 localhost localhost.localdomain\n" From williams at redhat.com Wed May 10 20:59:08 2006 From: williams at redhat.com (Clark Williams) Date: Wed, 10 May 2006 15:59:08 -0500 Subject: [RFC, PATCH] Build multiple srpms In-Reply-To: <35C9A9D68AB3FA4AB63692802656D9EC010F4372@ausx3mps303.aus.amer.dell.com> References: <35C9A9D68AB3FA4AB63692802656D9EC010F4372@ausx3mps303.aus.amer.dell.com> Message-ID: <4462541C.1040301@redhat.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Michael_E_Brown at Dell.com wrote: > [RFC, PATCH] Build multiple srpms > > Here is a patch for review. I have made the "obvious" changes in > order to be able to do the following: > mock -r CFG srpm1 srpm2 srpm3 srpm4 > I like the patch. It doesn't change the currently expected behavior with one srpm, just adds the ability to build multiple srpms from one chroot. Have you tried it with plague? If not, I'll put your version on my local plague server and see if it causes any grief (don't expect it to). > > Limitations: > - There are no checks for dependencies between the srpms, so they > should be independent. (no re-ordering of SRPMS) > - Building one SRPM can leave the buildroot in an inconsistent state > for the next SRPM. (no clean of chroot between builds) > I don't have a problem with this. I'm not sure I buy the argument that we need to do a clean of the chroot every time. Partially that's because I do a lot of cross-tools stuff which requires that I keep a chroot around for multiple builds. But even discounting that, I don't see what building an srpm in a chroot can do that will corrupt the chroot so that a subsequent build will fail or be incorrect. Mostly you're in there because you want a particular set of binaries (programs and libraries). Once those are installed, who cares if the rpm database gets trashed or the passwd file has some crufty entries in it? > - Failure to build one SRPM stops the whole process. > I'm not sure that I would consider the "failure stops everything" a limitation, since it saves you having to dig through tons of log file entries to find where the failure occurred (I never liked that make option anyway :)). You could probably get away with removing the sys.exit() in the for loop, but then you'd have to remember the exit status, etc. > - Resulting RPMs are not installed into the build environment for > use by subsequent SRPMS. > > Reasoning: > One problem I have been having with mock is speed. We are > using mock to build for 16 distinct configurations. Doing prep on > each configuration was taking a minute and a half to two minutes. As > an optimization, our mock wrapper script was doing a single prep per > chroot and then using --no-clean, but this complicates our wrapper > script. I am building a set of related RPMS, so I don have much > concern about cross-pollution. This simple patch simplifies my > wrapper script considerably. > What sort of speed improvement are you seeing now? > Future: > One future direction I'd like to take this is a parallel > mock that can prep/build multiple configurations at the same time to > try to amortize the cost of prep stage by running prep of one > environment in parallel with build of a different configuration. > Currently, it is taking me an hour to build 16 sets of RPMS (5 SRPMS > per set), and I am hoping to get this down. We have already > implemented squid and some other measures to try to speed things up. > You could probably achieve the same result by using plague and invoking multiple plague-client builds from your wrapper script. You wouldn't be able to "pipeline" the builds (i.e. wait for build1 to reach build state, then start build2), but this will be simpler. Certainly simpler than threading mock. Clark -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFEYlQcHyuj/+TTEp0RAlZFAJwKz7oV2d0lSjpvsjHn98196PX+vQCg29VZ TdCTiQSwsDG45VM+ILRZkro= =+h5q -----END PGP SIGNATURE----- From Michael_E_Brown at Dell.com Wed May 10 21:21:08 2006 From: Michael_E_Brown at Dell.com (Michael_E_Brown at Dell.com) Date: Wed, 10 May 2006 16:21:08 -0500 Subject: [RFC PATCH] Mock multi-config build patch -- prepare Message-ID: <35C9A9D68AB3FA4AB63692802656D9EC010F4376@ausx3mps303.aus.amer.dell.com> All, Here is stage one of a patch series to implement multi-config builds with mock. I believe this patch stands on its own, so I am sending it separately. It is applied on top of the earlier patch today to do multi-srpm builds. The patch simply re-arranges main() by taking and putting several sections into their own function. For example, the clean and prep sections were very similar and have been factored into their own function. The setting of defaults for config_opts is moved into its own function. The result is no functional change to the operation of mock, but it makes the next patch in my series much easier, which is the addition of multiple config builds, like this: $ mock -r CFG1 -r CFG2 -r CFG3 -r CFG4 ... srpm1 srpm2 srpm3 ... The next patch in the series will be a straightforward patch to implement the above on top of this patch. The last patch I would like to send would be to be able to run the prep() sections for the different configs in parallel (simple, via fork()) to reduce setup time. Comments welcome. -- Michael <> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: mock-multicfg.patch Type: application/octet-stream Size: 7600 bytes Desc: mock-multicfg.patch URL: From Michael_E_Brown at Dell.com Wed May 10 21:32:58 2006 From: Michael_E_Brown at Dell.com (Michael_E_Brown at Dell.com) Date: Wed, 10 May 2006 16:32:58 -0500 Subject: [RFC, PATCH] Build multiple srpms Message-ID: <35C9A9D68AB3FA4AB63692802656D9EC010F4377@ausx3mps303.aus.amer.dell.com> Thanks for the comments. Responses below. > -----Original Message----- > From: fedora-buildsys-list-bounces at redhat.com > [mailto:fedora-buildsys-list-bounces at redhat.com] On Behalf Of > Clark Williams > Sent: Wednesday, May 10, 2006 3:59 PM > To: Discussion of Fedora build system > Subject: Re: [RFC, PATCH] Build multiple srpms > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Michael_E_Brown at Dell.com wrote: > > [RFC, PATCH] Build multiple srpms > > > > Here is a patch for review. I have made the "obvious" > changes in order > > to be able to do the following: > > mock -r CFG srpm1 srpm2 srpm3 srpm4 > > > > I like the patch. It doesn't change the currently expected > behavior with one srpm, just adds the ability to build > multiple srpms from one chroot. Exactly my intent. > > Have you tried it with plague? If not, I'll put your version > on my local plague server and see if it causes any grief > (don't expect it to). I did not see an *easy* way to set up a plague server, so we do not have one, yet. > > > > Limitations: > > - There are no checks for dependencies between the srpms, so they > > should be independent. (no re-ordering of SRPMS) > > - Building one SRPM can leave the buildroot in an > inconsistent state > > for the next SRPM. (no clean of chroot between builds) > > > I don't have a problem with this. I'm not sure I buy the > argument that we need to do a clean of the chroot every time. > Partially that's because I do a lot of cross-tools stuff > which requires that I keep a chroot around for multiple > builds. But even discounting that, I don't see what building > an srpm in a chroot can do that will corrupt the chroot so > that a subsequent build will fail or be incorrect. Mostly > you're in there because you want a particular set of binaries > (programs and libraries). Once those are installed, who cares > if the rpm database gets trashed or the passwd file has some > crufty entries in it? My thoughts as well. :) > > > - Failure to build one SRPM stops the whole process. > > > > I'm not sure that I would consider the "failure stops > everything" a limitation, since it saves you having to dig > through tons of log file entries to find where the failure > occurred (I never liked that make option anyway :)). You > could probably get away with removing the > sys.exit() in the for loop, but then you'd have to remember > the exit status, etc. Same here. > > > - Resulting RPMs are not installed into the build > environment for use > > by subsequent SRPMS. > > > > Reasoning: > > One problem I have been having with mock is speed. We are > > using mock to build for 16 distinct configurations. Doing > prep on each > > configuration was taking a minute and a half to two minutes. As an > > optimization, our mock wrapper script was doing a single prep per > > chroot and then using --no-clean, but this complicates our wrapper > > script. I am building a set of related RPMS, so I don have much > > concern about cross-pollution. This simple patch simplifies > my wrapper > > script considerably. > > > > What sort of speed improvement are you seeing now? None, yet. Right now, this patch really just functions to make the rest of my scripting simpler. I believe that the speed boost will probably happen in stage 3. This patch is stage 1, and the patch I just sent next is stage 2. In stage 3, I would like to overlap prep() of one build root with building in another. > > > Future: > > One future direction I'd like to take this is a > parallel mock > > that can prep/build multiple configurations at the same > time to try to > > amortize the cost of prep stage by running prep of one > environment in > > parallel with build of a different configuration. > > Currently, it is taking me an hour to build 16 sets of RPMS > (5 SRPMS > > per set), and I am hoping to get this down. We have already > > implemented squid and some other measures to try to speed things up. > > > You could probably achieve the same result by using plague > and invoking multiple plague-client builds from your wrapper > script. You wouldn't be able to "pipeline" the builds (i.e. > wait for build1 to reach build state, then start build2), but > this will be simpler. > Certainly simpler than threading mock. Plague looked a bit complicated to set up when I last looked at it. I will probably look at it again soon. -- Michael From Michael_E_Brown at Dell.com Wed May 10 22:02:19 2006 From: Michael_E_Brown at Dell.com (Michael_E_Brown at Dell.com) Date: Wed, 10 May 2006 17:02:19 -0500 Subject: Patches for shared configurations items and specifying additional dependencies Message-ID: <35C9A9D68AB3FA4AB63692802656D9EC010F4378@ausx3mps303.aus.amer.dell.com> I cannot really speak for the mock guys, but to me, these patches look very good. -- self-contained -- limited scope -- elegant They add required functionality without making changes across the codebase. In particular, your solution for the additional dependencies looks elegant and cool. -- Michael > -----Original Message----- > From: fedora-buildsys-list-bounces at redhat.com > [mailto:fedora-buildsys-list-bounces at redhat.com] On Behalf Of > Andreas Thienemann > Sent: Wednesday, May 10, 2006 3:27 PM > To: Discussion of Fedora build system > Subject: Patches for shared configurations items and > specifying additional dependencies > > Hello List, > > as it seems to be patch submission time, I'm pushing two > patches to the list, I had lying around for some time now. > > The first patch allows mock to parse a generic side-wide > config file, where general configuration-items can be stored > and later overridden by specific chroot-definitions. > > > The second patch, is a bit different: > While pushing the Fedora Core SRPMS through the mock > buildsystem we noticed many SRPMS which do not correctly > define their buildrequirements. > Until the fedora core developer responsible for the package > has updated the spec in cvs and pushed an updated srpm into > rawhide, there is no way to make the package in question > build in mock without having to change the .spec. > Thus, in order to allow even these broken packages being > built, we needed a way to work around these broken > dependencies to see if anything else was missing. > This is done by the second patch, which adds a more_buildreqs > config option. If set the given dependencies are installed > _in addition_ to the ones from the .spec file. > > While the second patch should not be used during normal use > and is definetly not for the fedora-extras buildservers, it > still is of interest to other parties, using mock. > > > > regards, > andreas > From Michael_E_Brown at Dell.com Wed May 10 22:04:28 2006 From: Michael_E_Brown at Dell.com (Michael_E_Brown at Dell.com) Date: Wed, 10 May 2006 17:04:28 -0500 Subject: [RFC PATCH] Mock multi-config build patch -- implement multi-config Message-ID: <35C9A9D68AB3FA4AB63692802656D9EC010F4379@ausx3mps303.aus.amer.dell.com> And here is the multi-srpm, multi-config patch, built on top of the previous two. The changes are relatively minor, just re-indenting the last part of main(), no significant code changes except changing options.chroot to an array and adding a loop. Please let me know what you think. -- Michael ________________________________ From: fedora-buildsys-list-bounces at redhat.com [mailto:fedora-buildsys-list-bounces at redhat.com] On Behalf Of Michael_E_Brown at Dell.com Sent: Wednesday, May 10, 2006 4:21 PM To: fedora-buildsys-list at redhat.com Subject: [RFC PATCH] Mock multi-config build patch -- prepare All, Here is stage one of a patch series to implement multi-config builds with mock. I believe this patch stands on its own, so I am sending it separately. It is applied on top of the earlier patch today to do multi-srpm builds. The patch simply re-arranges main() by taking and putting several sections into their own function. For example, the clean and prep sections were very similar and have been factored into their own function. The setting of defaults for config_opts is moved into its own function. The result is no functional change to the operation of mock, but it makes the next patch in my series much easier, which is the addition of multiple config builds, like this: $ mock -r CFG1 -r CFG2 -r CFG3 -r CFG4 ... srpm1 srpm2 srpm3 ... The next patch in the series will be a straightforward patch to implement the above on top of this patch. The last patch I would like to send would be to be able to run the prep() sections for the different configs in parallel (simple, via fork()) to reduce setup time. Comments welcome. -- Michael <> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: mock-multicfg2.patch Type: application/octet-stream Size: 3905 bytes Desc: mock-multicfg2.patch URL: From jkeating at j2solutions.net Thu May 11 03:27:15 2006 From: jkeating at j2solutions.net (Jesse Keating) Date: Wed, 10 May 2006 23:27:15 -0400 Subject: Patches for shared configurations items and specifying additional dependencies In-Reply-To: References: Message-ID: <1147318035.19623.11.camel@ender> On Wed, 2006-05-10 at 22:26 +0200, Andreas Thienemann wrote: > The second patch, is a bit different: > While pushing the Fedora Core SRPMS through the mock buildsystem we > noticed many SRPMS which do not correctly define their > buildrequirements. > Until the fedora core developer responsible for the package has > updated > the spec in cvs and pushed an updated srpm into rawhide, there is no > way > to make the package in question build in mock without having to change > the > .spec. > Thus, in order to allow even these broken packages being built, we > needed > a way to work around these broken dependencies to see if anything else > was > missing. > This is done by the second patch, which adds a more_buildreqs config > option. If set the given dependencies are installed _in addition_ to > the > ones from the .spec file. > > While the second patch should not be used during normal use and is > definetly not for the fedora-extras buildservers, it still is of > interest > to other parties, using mock. > How is this different than just expanding what is in the list for 'init'ing a buildroot? -- Jesse Keating RHCE (geek.j2solutions.net) Fedora Legacy Team (www.fedoralegacy.org) GPG Public Key (geek.j2solutions.net/jkeating.j2solutions.pub) -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From Michael_E_Brown at dell.com Thu May 11 03:48:57 2006 From: Michael_E_Brown at dell.com (Michael E Brown) Date: Wed, 10 May 2006 22:48:57 -0500 Subject: Patches for shared configurations items and specifying additional dependencies In-Reply-To: <1147318035.19623.11.camel@ender> References: <1147318035.19623.11.camel@ender> Message-ID: <1147319337.13955.11.camel@soltek.michaels-house.net> On Wed, 2006-05-10 at 23:27 -0400, Jesse Keating wrote: > How is this different than just expanding what is in the list for > 'init'ing a buildroot? > I'm not the original author, but it looks good to me because you don't have to burden _every_ srpm you build with extra packages in the buildroot. It takes time to download and install these extra packages, after all. -- Michael From jkeating at j2solutions.net Thu May 11 03:54:48 2006 From: jkeating at j2solutions.net (Jesse Keating) Date: Wed, 10 May 2006 23:54:48 -0400 Subject: Patches for shared configurations items and specifying additional dependencies In-Reply-To: <1147319337.13955.11.camel@soltek.michaels-house.net> References: <1147318035.19623.11.camel@ender> <1147319337.13955.11.camel@soltek.michaels-house.net> Message-ID: <1147319688.19623.15.camel@ender> On Wed, 2006-05-10 at 22:48 -0500, Michael E Brown wrote: > I'm not the original author, but it looks good to me because you don't > have to burden _every_ srpm you build with extra packages in the > buildroot. It takes time to download and install these extra packages, > after all. True. We're trying to solve a similar problem with RHEL packages, and we want the rhel init to be much larger for all packages we push through. Trying to figure out which ones need which buildreqs and constantly adding to the list would not seem very fun, so we're probably just going to enable those build configs to have a much "fatter" init. -- Jesse Keating RHCE (geek.j2solutions.net) Fedora Legacy Team (www.fedoralegacy.org) GPG Public Key (geek.j2solutions.net/jkeating.j2solutions.pub) -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From skvidal at linux.duke.edu Thu May 11 04:00:11 2006 From: skvidal at linux.duke.edu (seth vidal) Date: Thu, 11 May 2006 00:00:11 -0400 Subject: Patches for shared configurations items and specifying additional dependencies In-Reply-To: <1147319688.19623.15.camel@ender> References: <1147318035.19623.11.camel@ender> <1147319337.13955.11.camel@soltek.michaels-house.net> <1147319688.19623.15.camel@ender> Message-ID: <1147320011.19154.3.camel@cutter> On Wed, 2006-05-10 at 23:54 -0400, Jesse Keating wrote: > On Wed, 2006-05-10 at 22:48 -0500, Michael E Brown wrote: > > I'm not the original author, but it looks good to me because you don't > > have to burden _every_ srpm you build with extra packages in the > > buildroot. It takes time to download and install these extra packages, > > after all. > > True. We're trying to solve a similar problem with RHEL packages, and > we want the rhel init to be much larger for all packages we push > through. Trying to figure out which ones need which buildreqs and > constantly adding to the list would not seem very fun, so we're probably > just going to enable those build configs to have a much "fatter" init. > why would we want to do that? If a package has insufficient buildreqs then the package is broken. We should not write code to work around broken packages that can be fixed. If there is a problem with the default set of things installed in a minimal build chroot that's a different discussion but adding code in to compensate for broken packages just seems like the cart leading the horse. -sv From andreas at bawue.net Thu May 11 09:17:14 2006 From: andreas at bawue.net (Andreas Thienemann) Date: Thu, 11 May 2006 11:17:14 +0200 (CEST) Subject: Patches for shared configurations items and specifying additional dependencies In-Reply-To: <1147318035.19623.11.camel@ender> References: <1147318035.19623.11.camel@ender> Message-ID: On Wed, 10 May 2006, Jesse Keating wrote: > How is this different than just expanding what is in the list for > 'init'ing a buildroot? Adding it to the buildroot.xml file is global. moredeps works on a per package level. bye, andreas From andreas at bawue.net Thu May 11 09:30:48 2006 From: andreas at bawue.net (Andreas Thienemann) Date: Thu, 11 May 2006 11:30:48 +0200 (CEST) Subject: Patches for shared configurations items and specifying additional dependencies In-Reply-To: <1147320011.19154.3.camel@cutter> References: <1147318035.19623.11.camel@ender> <1147319337.13955.11.camel@soltek.michaels-house.net> <1147319688.19623.15.camel@ender> <1147320011.19154.3.camel@cutter> Message-ID: On Thu, 11 May 2006, seth vidal wrote: >> [...] so we're probably just going to enable those build configs to >> have a much "fatter" init. > why would we want to do that? I do not know what Jesse is trying to build there, but I suspect he's talking about rebuilding RHEL SRPMs in mock. If he doesn't want to fix the SRPMs for his rebuild, as the policy is to deliver exactly the same SRPMs as upstream he has to work around these broken SRPMs which won't build in mock until upstream has fixed it. In this case, fattening up the buildroots definitions might help, but it is rather global and IMHO definitely not elegant at all. > If a package has insufficient buildreqs then the package is broken. Agreed. For FE use, fattening up the buildroots or even using my moredeps patch is a very bad idea (tm) as it would allow broken packages to slip through. But it seems to me that plague and even more so mock is increasingly being used by other people and projects for building their rpms. And if they cannot fix their packages (e.g. wanting to keep their SRPMs packages in synx with upstream) they can at least build the packages until upstream has fixed them. > We should not write code to work around broken packages that can be > fixed. For FE, no, we should not. For other purposes, I'd welcome a solution to at least allow me to work around problems. bye, andreas From skvidal at linux.duke.edu Thu May 11 11:16:17 2006 From: skvidal at linux.duke.edu (seth vidal) Date: Thu, 11 May 2006 07:16:17 -0400 Subject: Patches for shared configurations items and specifying additional dependencies In-Reply-To: References: <1147318035.19623.11.camel@ender> <1147319337.13955.11.camel@soltek.michaels-house.net> <1147319688.19623.15.camel@ender> <1147320011.19154.3.camel@cutter> Message-ID: <1147346177.19154.7.camel@cutter> On Thu, 2006-05-11 at 11:30 +0200, Andreas Thienemann wrote: > On Thu, 11 May 2006, seth vidal wrote: > > >> [...] so we're probably just going to enable those build configs to > >> have a much "fatter" init. > > why would we want to do that? > > I do not know what Jesse is trying to build there, but I suspect he's > talking about rebuilding RHEL SRPMs in mock. > > If he doesn't want to fix the SRPMs for his rebuild, as the policy is to > deliver exactly the same SRPMs as upstream he has to work around these > broken SRPMs which won't build in mock until upstream has fixed it. > > In this case, fattening up the buildroots definitions might help, but it > is rather global and IMHO definitely not elegant at all. > > > If a package has insufficient buildreqs then the package is broken. > > Agreed. For FE use, fattening up the buildroots or even using my moredeps > patch is a very bad idea (tm) as it would allow broken packages to slip > through. > > But it seems to me that plague and even more so mock is increasingly being > used by other people and projects for building their rpms. > > And if they cannot fix their packages (e.g. wanting to keep their > SRPMs packages in synx with upstream) they can at least build the packages > until upstream has fixed them. > > > We should not write code to work around broken packages that can be > > fixed. > For FE, no, we should not. For other purposes, I'd welcome a solution to > at least allow me to work around problems. okay, I can accept that. But the first time I hear someone who wants us to do special things to mock to allow us to build fedora packages I'm going to smack them around. :) -sv From skvidal at linux.duke.edu Thu May 11 13:49:27 2006 From: skvidal at linux.duke.edu (seth vidal) Date: Thu, 11 May 2006 09:49:27 -0400 Subject: [RFC, PATCH] Build multiple srpms In-Reply-To: <4462541C.1040301@redhat.com> References: <35C9A9D68AB3FA4AB63692802656D9EC010F4372@ausx3mps303.aus.amer.dell.com> <4462541C.1040301@redhat.com> Message-ID: <1147355368.1918.7.camel@cutter> On Wed, 2006-05-10 at 15:59 -0500, Clark Williams wrote: > I don't have a problem with this. I'm not sure I buy the argument that > we need to do a clean of the chroot every time. Partially that's > because I do a lot of cross-tools stuff which requires that I keep a > chroot around for multiple builds. But even discounting that, I don't > see what building an srpm in a chroot can do that will corrupt the > chroot so that a subsequent build will fail or be incorrect. Mostly > you're in there because you want a particular set of binaries > (programs and libraries). Once those are installed, who cares if the > rpm database gets trashed or the passwd file has some crufty entries > in it? > The clean is non-negotiable. Pollution of chroot is a big deal, especially in wanting to make sure we've created consistent and repeatable builds, not to mention security. As I mentioned to matt, Jochen, from several months ago, wrote a patch to do manual creation of a cached chroot so we could simply copy that image into place if it exists and run a 'yum update' on it to make sure it is current. The clean is important for consistent builds and we must always have a clean chroot for our builds of fedora. Moving away from that requirement in mock (w/o special options) is just setting up users for confusion and failure. > I'm not sure that I would consider the "failure stops everything" a > limitation, since it saves you having to dig through tons of log file > entries to find where the failure occurred (I never liked that make > option anyway :)). You could probably get away with removing the > sys.exit() in the for loop, but then you'd have to remember the exit > status, etc. failure should stop everything, especially for related but not _Required_ builds happening before. -sv From skvidal at linux.duke.edu Thu May 11 13:54:03 2006 From: skvidal at linux.duke.edu (seth vidal) Date: Thu, 11 May 2006 09:54:03 -0400 Subject: [RFC PATCH] Mock multi-config build patch -- implement multi-config In-Reply-To: <35C9A9D68AB3FA4AB63692802656D9EC010F4379@ausx3mps303.aus.amer.dell.com> References: <35C9A9D68AB3FA4AB63692802656D9EC010F4379@ausx3mps303.aus.amer.dell.com> Message-ID: <1147355643.1918.12.camel@cutter> On Wed, 2006-05-10 at 17:04 -0500, Michael_E_Brown at Dell.com wrote: > And here is the multi-srpm, multi-config patch, built on top of the > previous two. The changes are relatively minor, just re-indenting the > last part of main(), no significant code changes except changing > options.chroot to an array and adding a loop. > > Please let me know what you think. > -- > Michael > > > ______________________________________________________________ > From: fedora-buildsys-list-bounces at redhat.com > [mailto:fedora-buildsys-list-bounces at redhat.com] On Behalf Of > Michael_E_Brown at Dell.com > Sent: Wednesday, May 10, 2006 4:21 PM > To: fedora-buildsys-list at redhat.com > Subject: [RFC PATCH] Mock multi-config build patch -- prepare > > > > All, > Here is stage one of a patch series to implement > multi-config builds with mock. I believe this patch stands on > its own, so I am sending it separately. It is applied on top > of the earlier patch today to do multi-srpm builds. > > The patch simply re-arranges main() by taking and > putting several sections into their own function. For example, > the clean and prep sections were very similar and have been > factored into their own function. The setting of defaults for > config_opts is moved into its own function. > > The result is no functional change to the operation of > mock, but it makes the next patch in my series much easier, > which is the addition of multiple config builds, like this: > > $ mock -r CFG1 -r CFG2 -r CFG3 -r CFG4 ? srpm1 srpm2 > srpm3 ? > > The next patch in the series will be a straightforward > patch to implement the above on top of this patch. The last > patch I would like to send would be to be able to run the > prep() sections for the different configs in parallel (simple, > via fork()) to reduce setup time. why would this be valuable? why not simply spawn off the jobs in multiple terminal windows or screen sessions and not complicate the code with the forking and also confusing the user as to what -r config -r config -r config does? This would be adding A LOT of complexity to mock where it isn't necessary and making the user case more difficult to know what you would get. As an example does the above mean: srpm1 will be built with cfg1 srpm2 will be built with cfg2 srpm3 will be build with cfg3 or does it mean srpm1 will be built with configs 1, 2 and 3 srpm2 will be built with configs 1, 2 and 3 srpm3 will be built with configs 1, 2 and 3 let's not clutter up interfaces for some special case we're targeting. -sv From skvidal at linux.duke.edu Thu May 11 13:58:33 2006 From: skvidal at linux.duke.edu (seth vidal) Date: Thu, 11 May 2006 09:58:33 -0400 Subject: [RFC, PATCH] Build multiple srpms In-Reply-To: <35C9A9D68AB3FA4AB63692802656D9EC010F4372@ausx3mps303.aus.amer.dell.com> References: <35C9A9D68AB3FA4AB63692802656D9EC010F4372@ausx3mps303.aus.amer.dell.com> Message-ID: <1147355913.1918.16.camel@cutter> On Wed, 2006-05-10 at 14:35 -0500, Michael_E_Brown at Dell.com wrote: > Future: > One future direction I'd like to take this is a parallel mock > that can prep/build multiple configurations at the same time to try to > amortize the cost of prep stage by running prep of one environment in > parallel with build of a different configuration. Currently, it is > taking me an hour to build 16 sets of RPMS (5 SRPMS per set), and I am > hoping to get this down. We have already implemented squid and some > other measures to try to speed things up. > parallelizing mock will NOT happen in the fedora mock branch. Period. Distributing out the items like that are suited to plague, not mock. Adding the features into mock is putting a lot of complexity where it is not needed. with regard to not cleaning the chroot b/t rebuilds: If you want to do it for the srpms then pass --no-clean and pass in a series of srpms. But breaking the expectation to the user that the chroots will be completely pristine is not safe and, to me, violates the purpose of mock. -sv From skvidal at linux.duke.edu Thu May 11 14:14:37 2006 From: skvidal at linux.duke.edu (seth vidal) Date: Thu, 11 May 2006 10:14:37 -0400 Subject: mock goals and non-goals Message-ID: <1147356877.1918.34.camel@cutter> Hi folks, Given the patchfest from yesterday I felt it would be best to have some discussion of goals/non-goals for mock in/around fedora's build environment. This is my opinion on the subject. It is not final nor terminal, it just is where I am: goals: - make clean and consistent build environments to be used by fedora's buildsystem and packagers - simplify building and testing process for packagers - be able to canonically identify the 'build environment' for fedora systems to discourage confusion on what should be where - simple interface with good defaults. non-goals: - this is not a tool to manage many, many builds with queuing, etc - that is plague. So, with regard to the patches we've seen so far here's what makes sense to me: - global/shared config to reduce the replication in the mock configs - specifying multiple srpms on the command line for a SINGLE config - provided that the buildroot is refreshed/cleaned each time - unless, of course, the --no-clean option is passed in - caching mechanism to create a cached pristine chroot that will allow for quickly recreating the chroot into that state simply by putting that cached copy back. Patches that don't make sense to me: - multiconfig - this is what shells and/or plague is for, use one of them. - parallel builds - plague is your friend. Patches which I am ambivalent about: - the fake buildreqs/sneaky buildreqs patch for rebuilding items with BROKEN buildrequires: - I hate the idea of enabling broken shit - I realize that there are lots of broken things that people need to work around and I'm sympathetic to that, to some extent. Doesn't mean I can't hate it. so, there, my flames are all away, feel free to bombard me with your own. -sv From dcbw at redhat.com Thu May 11 14:17:28 2006 From: dcbw at redhat.com (Dan Williams) Date: Thu, 11 May 2006 10:17:28 -0400 Subject: [RFC, PATCH] Build multiple srpms In-Reply-To: <35C9A9D68AB3FA4AB63692802656D9EC010F4377@ausx3mps303.aus.amer.dell.com> References: <35C9A9D68AB3FA4AB63692802656D9EC010F4377@ausx3mps303.aus.amer.dell.com> Message-ID: <1147357048.4933.12.camel@dhcp83-75.boston.redhat.com> On Wed, 2006-05-10 at 16:32 -0500, Michael_E_Brown at Dell.com wrote: > > Have you tried it with plague? If not, I'll put your version > > on my local plague server and see if it causes any grief > > (don't expect it to). > > I did not see an *easy* way to set up a plague server, so we do not have > one, yet. Sorry about that, I hope to fix this soon with nice documentation. There are also gotchas that can be cleaned up in the code, like ensuring that all necessary directories are created and accessible before the server is open for business. The config files also need explanation. I'd gladly help out on configuration at any point though. > Plague looked a bit complicated to set up when I last looked at it. I > will probably look at it again soon. The setup is the complicated bit, and it's (slightly) more complicated when you're using SSL, because then you have to do the SSL certificate generation and _nobody_ thinks openssl is simple :) Dan From williams at redhat.com Thu May 11 14:24:08 2006 From: williams at redhat.com (Clark Williams) Date: Thu, 11 May 2006 09:24:08 -0500 Subject: [RFC, PATCH] Build multiple srpms In-Reply-To: <1147357048.4933.12.camel@dhcp83-75.boston.redhat.com> References: <35C9A9D68AB3FA4AB63692802656D9EC010F4377@ausx3mps303.aus.amer.dell.com> <1147357048.4933.12.camel@dhcp83-75.boston.redhat.com> Message-ID: <44634908.4010500@redhat.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Dan Williams wrote: > On Wed, 2006-05-10 at 16:32 -0500, Michael_E_Brown at Dell.com wrote: >>> Have you tried it with plague? If not, I'll put your version >>> on my local plague server and see if it causes any grief >>> (don't expect it to). >> I did not see an *easy* way to set up a plague server, so we do not have >> one, yet. > > Sorry about that, I hope to fix this soon with nice documentation. > There are also gotchas that can be cleaned up in the code, like ensuring > that all necessary directories are created and accessible before the > server is open for business. The config files also need explanation. > > I'd gladly help out on configuration at any point though. I've set up one plague server that was entirely local (i.e. server and builder on the same box) and didn't use SSL. It was not really difficult, my problems were mainly getting my head wrapped around how it all works. > >> Plague looked a bit complicated to set up when I last looked at it. I >> will probably look at it again soon. > > The setup is the complicated bit, and it's (slightly) more complicated > when you're using SSL, because then you have to do the SSL certificate > generation and _nobody_ thinks openssl is simple :) > I'm going to set up another plague system on my home server. I'll document those steps and send them to you; maybe we can either make a Plague for the Clueless document (I'm *eminently* qualified) or we can automate it so that you could install an RPM and voila! a plague server. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFEY0kIHyuj/+TTEp0RAiN+AJ9Y7KA1L9n1nXryuzL/piDLd4od9wCgl2vi gtU5V5UgGdCyhlgYtpHZGNI= =V5AH -----END PGP SIGNATURE----- From williams at redhat.com Thu May 11 14:36:08 2006 From: williams at redhat.com (Clark Williams) Date: Thu, 11 May 2006 09:36:08 -0500 Subject: [RFC, PATCH] Build multiple srpms In-Reply-To: <1147355368.1918.7.camel@cutter> References: <35C9A9D68AB3FA4AB63692802656D9EC010F4372@ausx3mps303.aus.amer.dell.com> <4462541C.1040301@redhat.com> <1147355368.1918.7.camel@cutter> Message-ID: <44634BD8.6070807@redhat.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 seth vidal wrote: > On Wed, 2006-05-10 at 15:59 -0500, Clark Williams wrote: >> I don't have a problem with this. I'm not sure I buy the argument that >> we need to do a clean of the chroot every time. Partially that's >> because I do a lot of cross-tools stuff which requires that I keep a >> chroot around for multiple builds. But even discounting that, I don't >> see what building an srpm in a chroot can do that will corrupt the >> chroot so that a subsequent build will fail or be incorrect. Mostly >> you're in there because you want a particular set of binaries >> (programs and libraries). Once those are installed, who cares if the >> rpm database gets trashed or the passwd file has some crufty entries >> in it? >> > > > The clean is non-negotiable. Pollution of chroot is a big deal, > especially in wanting to make sure we've created consistent and > repeatable builds, not to mention security. > > As I mentioned to matt, Jochen, from several months ago, wrote a patch > to do manual creation of a cached chroot so we could simply copy that > image into place if it exists and run a 'yum update' on it to make sure > it is current. I remember seeing it in the archives. Do you think we should look at merging that patch into the current yum? > > The clean is important for consistent builds and we must always have a > clean chroot for our builds of fedora. Moving away from that requirement > in mock (w/o special options) is just setting up users for confusion and > failure. > I'm not opposed to clean being the default; I understand the paranoia :). My situation is a bit strange, since I use mock to maintain a "host" environment that contains a "sysroot". The sysroot is a set of target libraries and headers files that are used to generate packages for a non-x86 Linux system. So, I have an environment that gets built up over time as packages state their requirements. Since a clean of the chroot would blow away my sysroot, I don't want that to happen and go to great pains to insure that all the config files default 'clean' to false and all command lines have --no-clean. I suppose I could copy the sysroot out after the build and back in before a new build, but that just seems hokey... > >> I'm not sure that I would consider the "failure stops everything" a >> limitation, since it saves you having to dig through tons of log file >> entries to find where the failure occurred (I never liked that make >> option anyway :)). You could probably get away with removing the >> sys.exit() in the for loop, but then you'd have to remember the exit >> status, etc. > > failure should stop everything, especially for related but not > _Required_ builds happening before. > Hey, I build with -Werror! Clark -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFEY0vXHyuj/+TTEp0RAuBzAJ9arsw8mRN120BCuWEuSEONIoPygACgwHah lOzChbkhhduhqfyM68FM/Ac= =fWw1 -----END PGP SIGNATURE----- From dcbw at redhat.com Thu May 11 15:08:05 2006 From: dcbw at redhat.com (Dan Williams) Date: Thu, 11 May 2006 11:08:05 -0400 Subject: Plague 0.5 update: builder-to-server communication Message-ID: <1147360085.6367.6.camel@dhcp83-75.boston.redhat.com> Hi, I checked in the last bits for builder-initiated communication earlier this week to cvs HEAD. This means that builder instances no longer have to run _any_ server (XML-RPC, HTTP, or otherwise), and that builders can be behind a firewall without problems. I call these "Active" builders, as they initiate all communication to to the build server. It appears to work with light use, but there are still bugs and edge-cases to work out. Next steps include beating down bugs introduced by this refactor, finishing up the server-to-builder ("Passive") communication code, writing documentation, and getting more testing. I'd also like to make sure the plague directory structure is FHS-clean Somebody is sending me an SVN patch too. Dan From Michael_E_Brown at Dell.com Thu May 11 15:55:26 2006 From: Michael_E_Brown at Dell.com (Michael_E_Brown at Dell.com) Date: Thu, 11 May 2006 10:55:26 -0500 Subject: [RFC, PATCH] Build multiple srpms Message-ID: <35C9A9D68AB3FA4AB63692802656D9EC010F4383@ausx3mps303.aus.amer.dell.com> > -----Original Message----- > From: fedora-buildsys-list-bounces at redhat.com > [mailto:fedora-buildsys-list-bounces at redhat.com] On Behalf Of > Clark Williams > > I'm going to set up another plague system on my home server. > I'll document those steps and send them to you; maybe we can > either make a Plague for the Clueless document (I'm > *eminently* qualified) or we can automate it so that you > could install an RPM and voila! a plague server. That would be much appreciated. An RPM that installs a functioning plague server would be goodness. One nice thing about mock is that you install the RPM and have a working (but slow) configuration. Make a couple of obvious config changes to use a local mirror and everything is good. An equivalent situation for plague would be goodness. -- Michael From Michael_E_Brown at Dell.com Thu May 11 16:05:51 2006 From: Michael_E_Brown at Dell.com (Michael_E_Brown at Dell.com) Date: Thu, 11 May 2006 11:05:51 -0500 Subject: mock goals and non-goals Message-ID: <35C9A9D68AB3FA4AB63692802656D9EC010F4384@ausx3mps303.aus.amer.dell.com> > -----Original Message----- > From: fedora-buildsys-list-bounces at redhat.com > [mailto:fedora-buildsys-list-bounces at redhat.com] On Behalf Of > seth vidal > Sent: Thursday, May 11, 2006 9:15 AM > To: Discussion of Fedora build system > Subject: mock goals and non-goals Thanks for the valuable and constructive feedback. I was just testing the limits, as I have a mildly complex build environment that I am trying to set up and manage. Looks like I will be investigating plague a bit more. > So, with regard to the patches we've seen so far here's what > makes sense to me: > - global/shared config to reduce the replication in the mock configs > - specifying multiple srpms on the command line for a SINGLE > config - provided that the buildroot is refreshed/cleaned > each time - unless, of course, the --no-clean option is passed in I will refresh my patch from yesterday to do this and resend. Doing it this way is a one line change. > - caching mechanism to create a cached pristine chroot that > will allow for quickly recreating the chroot into that state > simply by putting that cached copy back. I will dig out the patch from the archives and test/resend (though I am not the original author). I have 5 RPMS I build across 16 configs. With clean, this takes about 5 hours. With single clean/init to set up the environment and --no-clean for each individual RPM, that reduces the time to 45-mins to an hour. > > Patches that don't make sense to me: > - multiconfig - this is what shells and/or plague is for, use > one of them. Secret message received and understood. :) > - parallel builds - plague is your friend. Ditto. Thanks, Michael From Michael_E_Brown at Dell.com Thu May 11 16:41:01 2006 From: Michael_E_Brown at Dell.com (Michael_E_Brown at Dell.com) Date: Thu, 11 May 2006 11:41:01 -0500 Subject: [PATCH] resubmit: multisrpm build Message-ID: <35C9A9D68AB3FA4AB63692802656D9EC010F4385@ausx3mps303.aus.amer.dell.com> This is a resubmit of the patch from yesterday to build multiple srpms in one command line: $ mock -r CFG srpm1 srpm2 srpm3 It exits immediately on the first failure. It cleans and preps the build environment for each srpm unless --no-clean is passed. -- Michael <> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: multisrpm.patch Type: application/octet-stream Size: 2250 bytes Desc: multisrpm.patch URL: From williams at redhat.com Thu May 11 17:12:10 2006 From: williams at redhat.com (Clark Williams) Date: Thu, 11 May 2006 12:12:10 -0500 Subject: interactive shell in chroot patch Message-ID: <4463706A.2050400@redhat.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 While we're in patch-a-thon mode, thought I'd submit this for comment. While attempting to track down a bizarre bug with elinks, I hacked mock so that I could get an interactive shell in the chroot environment. Not useful for everyone, but handy if you want to look at the environment in which something is failing. Oh and I converted some prints in my chroot command to use the debug method. Index: mock.py =================================================================== RCS file: /cvs/fedora/mock/mock.py,v retrieving revision 1.43 diff -u -r1.43 mock.py - --- mock.py 8 May 2006 16:39:53 -0000 1.43 +++ mock.py 11 May 2006 17:04:34 -0000 @@ -800,13 +800,28 @@ print 'Finished initializing root' elif args[0] == 'chroot': + # catch-all for executing arbitrary commands in the chroot config_opts['clean'] = config_opts['quiet'] = False - - my= Root(config_opts) + my = Root(config_opts) + cmd = ' '.join(args[1:]) + my.debug("executing: %s" % cmd) my._mount() - - my.do_chroot(' '.join(args[1:]), True) + my.do_chroot(cmd, True) my.close() - - print 'Finished chroot command' + my.debug('finished chroot command') + elif args[0] == 'shell': + # debugging tool for interactive poking around in the chroot + config_opts['clean'] = config_opts['quiet'] = False + my = Root(config_opts) + cmd = "PS1='mock-chroot> ' %s %s /bin/bash" % (config_opts['chroot'], + my.rootdir) + my.debug("executing: %s" % cmd) + my._mount() + ret = os.system(cmd) + my.close() + my.debug("finished shell with retval %d" % ret) + else: if args[0] == 'rebuild': if len(args) > 1: Index: src/mock-helper.c =================================================================== RCS file: /cvs/fedora/mock/src/mock-helper.c,v retrieving revision 1.7 diff -u -r1.7 mock-helper.c - --- src/mock-helper.c 14 Jul 2005 18:00:26 -0000 1.7 +++ src/mock-helper.c 11 May 2006 17:04:34 -0000 @@ -25,7 +25,7 @@ static char const * const ALLOWED_ENV[] = { "dist", - - "ftp_proxy", "http_proxy", "https_proxy", "no_proxy" + "ftp_proxy", "http_proxy", "https_proxy", "no_proxy", "PS1", }; #define ALLOWED_ENV_SIZE (sizeof (ALLOWED_ENV) / sizeof (ALLOWED_ENV[0])) -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFEY3BpHyuj/+TTEp0RAvnBAJ9tqOtxB102szDxsv/QZGbHkF5SFgCg1mCn QzgvjxfcsjBrfcKXOAg3Efs= =iMVS -----END PGP SIGNATURE----- From skvidal at linux.duke.edu Thu May 11 18:36:58 2006 From: skvidal at linux.duke.edu (seth vidal) Date: Thu, 11 May 2006 14:36:58 -0400 Subject: [RFC, PATCH] Build multiple srpms In-Reply-To: <44634BD8.6070807@redhat.com> References: <35C9A9D68AB3FA4AB63692802656D9EC010F4372@ausx3mps303.aus.amer.dell.com> <4462541C.1040301@redhat.com> <1147355368.1918.7.camel@cutter> <44634BD8.6070807@redhat.com> Message-ID: <1147372619.1918.58.camel@cutter> On Thu, 2006-05-11 at 09:36 -0500, Clark Williams wrote: > > As I mentioned to matt, Jochen, from several months ago, wrote a patch > > to do manual creation of a cached chroot so we could simply copy that > > image into place if it exists and run a 'yum update' on it to make sure > > it is current. > > I remember seeing it in the archives. Do you think we should look at > merging that patch into the current yum? into yum? it doesn't need to go into yum. The feature need only be added to mock. > My situation is a bit strange, since I use mock to maintain a "host" > environment that contains a "sysroot". The sysroot is a set of target > libraries and headers files that are used to generate packages for a > non-x86 Linux system. So, I have an environment that gets built up > over time as packages state their requirements. Since a clean of the > chroot would blow away my sysroot, I don't want that to happen and go > to great pains to insure that all the config files default 'clean' to > false and all command lines have --no-clean. I suppose I could copy > the sysroot out after the build and back in before a new build, but > that just seems hokey... so is there something WRONG with --no-clean? Does that not meet your requirements? -sv From skvidal at linux.duke.edu Thu May 11 18:38:11 2006 From: skvidal at linux.duke.edu (seth vidal) Date: Thu, 11 May 2006 14:38:11 -0400 Subject: buildsys-build and buildgroups location In-Reply-To: <7dd7ab490605101318s3330f268wa4e0167e786bac2a@mail.gmail.com> References: <1147288873.15425.28.camel@cutter> <7dd7ab490605101318s3330f268wa4e0167e786bac2a@mail.gmail.com> Message-ID: <1147372691.1918.60.camel@cutter> On Wed, 2006-05-10 at 13:18 -0700, Chris Weyl wrote: > On 5/10/06, seth vidal wrote: > > cool? > > Works by me :) If you get a chance, drop us a note so I can start > updating my mock configs when the groups repos is moved to > buildsys.f.o? (Which I know the redirect will handle, but it's always > nice to be prepared...) > I'll announce it here. -sv From williams at redhat.com Thu May 11 18:48:10 2006 From: williams at redhat.com (Clark Williams) Date: Thu, 11 May 2006 13:48:10 -0500 Subject: [RFC, PATCH] Build multiple srpms In-Reply-To: <1147372619.1918.58.camel@cutter> References: <35C9A9D68AB3FA4AB63692802656D9EC010F4372@ausx3mps303.aus.amer.dell.com> <4462541C.1040301@redhat.com> <1147355368.1918.7.camel@cutter> <44634BD8.6070807@redhat.com> <1147372619.1918.58.camel@cutter> Message-ID: <446386EA.1050300@redhat.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 seth vidal wrote: > On Thu, 2006-05-11 at 09:36 -0500, Clark Williams wrote: >>> As I mentioned to matt, Jochen, from several months ago, wrote a patch >>> to do manual creation of a cached chroot so we could simply copy that >>> image into place if it exists and run a 'yum update' on it to make sure >>> it is current. >> I remember seeing it in the archives. Do you think we should look at >> merging that patch into the current yum? > > into yum? it doesn't need to go into yum. The feature need only be added > to mock. > Gah. I meant mock, sorry... >> My situation is a bit strange, since I use mock to maintain a "host" >> environment that contains a "sysroot". The sysroot is a set of target >> libraries and headers files that are used to generate packages for a >> non-x86 Linux system. So, I have an environment that gets built up >> over time as packages state their requirements. Since a clean of the >> chroot would blow away my sysroot, I don't want that to happen and go >> to great pains to insure that all the config files default 'clean' to >> false and all command lines have --no-clean. I suppose I could copy >> the sysroot out after the build and back in before a new build, but >> that just seems hokey... > > so is there something WRONG with --no-clean? Does that not meet your > requirements? No, I see nothing wrong with --no-clean (although I prefer setting the 'clean' option to false in the config file). I was just trying to explain my point-of-view (which admittedly is not the norm) so when I submit a patch, those on this list can see why it might be useful. Clark -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFEY4bpHyuj/+TTEp0RAq0RAKCkRM2TvVUsbdpJ+4go37YclmKr1gCgzseW v4iKitNSq0y2yEQiyNg8tFo= =ZJkn -----END PGP SIGNATURE----- From andreas at bawue.net Thu May 11 19:10:18 2006 From: andreas at bawue.net (Andreas Thienemann) Date: Thu, 11 May 2006 21:10:18 +0200 (CEST) Subject: mock goals and non-goals In-Reply-To: <1147356877.1918.34.camel@cutter> References: <1147356877.1918.34.camel@cutter> Message-ID: Hello, On Thu, 11 May 2006, seth vidal wrote: > goals: > - make clean and consistent build environments to be used by fedora's > buildsystem and packagers > - simplify building and testing process for packagers > - be able to canonically identify the 'build environment' for fedora > systems to discourage confusion on what should be where > - simple interface with good defaults. Why not a bit more generic and see mock as a tool to build rpms in a clean and consistent buildroot. After all, we heard that SuSE RPMs can be built with it too. And that's a feat s the last time I looked SuSE RPMs were pretty bad dependency-wise. > non-goals: > - this is not a tool to manage many, many builds with queuing, etc - > that is plague. ack IMHO. Even though plague needs still a bit of work it seems. livna's plague installation only became stable as soon as I sent them a few patches, which dcbw handed me on irc after bugging him about a few problems. ;-D > - specifying multiple srpms on the command line for a SINGLE config - > provided that the buildroot is refreshed/cleaned each time - unless, of This time, I do not really see the difference between calling mock with several srpms and calling mock several times on the shell. > course, the --no-clean option is passed in > - caching mechanism to create a cached pristine chroot that will allow > for quickly recreating the chroot into that state simply by putting that > cached copy back. This would be simply great. Right now, the cycle of cearting the chroot, installing the base-packages, doing the full depsolve-cycle, building the rpm and then removing the buildroot again takes ages. Something faster there might be great. > Patches which I am ambivalent about: > - the fake buildreqs/sneaky buildreqs patch for rebuilding items with > BROKEN buildrequires: > - I hate the idea of enabling broken shit > - I realize that there are lots of broken things that people need to > work around and I'm sympathetic to that, to some extent. Doesn't mean I > can't hate it. Well, nobody expects you to like it, it just has to be in mock. ;-D bye, andreas From Michael_E_Brown at Dell.com Thu May 11 19:55:25 2006 From: Michael_E_Brown at Dell.com (Michael_E_Brown at Dell.com) Date: Thu, 11 May 2006 14:55:25 -0500 Subject: mock goals and non-goals Message-ID: <35C9A9D68AB3FA4AB63692802656D9EC010F438D@ausx3mps303.aus.amer.dell.com> > -----Original Message----- > From: fedora-buildsys-list-bounces at redhat.com > [mailto:fedora-buildsys-list-bounces at redhat.com] On Behalf Of > Andreas Thienemann > Sent: Thursday, May 11, 2006 2:10 PM > To: Discussion of Fedora build system > Subject: Re: mock goals and non-goals > > Hello, > > On Thu, 11 May 2006, seth vidal wrote: > > > goals: > > - make clean and consistent build environments to be used > by fedora's > > buildsystem and packagers > > - simplify building and testing process for packagers > > - be able to canonically identify the 'build environment' > for fedora > > systems to discourage confusion on what should be where > > - simple interface with good defaults. > Why not a bit more generic and see mock as a tool to build > rpms in a clean and consistent buildroot. After all, we heard > that SuSE RPMs can be built with it too. And that's a feat s > the last time I looked SuSE RPMs were pretty bad dependency-wise. SuSE has taken a little bit of work to get working with mock. My position, though, is that I cannot afford, time-wise, the effort it would take to set up hundreds of different build environments for every distro out there. It is much more efficient for me to just add the extra bits to support SUSE in mock. I really would like to see one build environment to rule them all(TM). Mock seems simple, flexible, and extensible enough to handle lots of different distros with a modicum of cooperation from the distros. Yes, there are a few problems, and no, you cannot do an entire distro build of SUSE out of the box. But it works for my stuff, which is all I need. >From my perspective, every major rpm-based distro coming out has native yum support, so mock (built on yum) looks like the winner. > > - specifying multiple srpms on the command line for a > SINGLE config - > > provided that the buildroot is refreshed/cleaned each time > - unless, > > of > This time, I do not really see the difference between calling > mock with several srpms and calling mock several times on the shell. It is just a matter of complexity. The added complexity in my script to handle this is a pain, but inside mock itself, the changes are minimal. Or, stated another way, the total system complexity is smaller doing this inside mock. > > > course, the --no-clean option is passed in > > - caching mechanism to create a cached pristine chroot that > will allow > > for quickly recreating the chroot into that state simply by putting > > that cached copy back. > This would be simply great. Right now, the cycle of cearting > the chroot, installing the base-packages, doing the full > depsolve-cycle, building the rpm and then removing the > buildroot again takes ages. > Something faster there might be great. Seconded. -- Michael From Michael_E_Brown at Dell.com Thu May 11 20:06:55 2006 From: Michael_E_Brown at Dell.com (Michael_E_Brown at Dell.com) Date: Thu, 11 May 2006 15:06:55 -0500 Subject: Patches for shared configurations items and specifying additional dependencies Message-ID: <35C9A9D68AB3FA4AB63692802656D9EC010F438E@ausx3mps303.aus.amer.dell.com> Don't know if anybody else brought this up, but could the shared-configuration be implemented by putting an "execfile('/etc/mock/default.cfg')" at the beginning of the config file? No patches necessary. -- Michael > -----Original Message----- > From: fedora-buildsys-list-bounces at redhat.com > [mailto:fedora-buildsys-list-bounces at redhat.com] On Behalf Of > Andreas Thienemann > Sent: Wednesday, May 10, 2006 3:27 PM > To: Discussion of Fedora build system > Subject: Patches for shared configurations items and > specifying additional dependencies > > Hello List, > > as it seems to be patch submission time, I'm pushing two > patches to the list, I had lying around for some time now. > > The first patch allows mock to parse a generic side-wide > config file, where general configuration-items can be stored > and later overridden by specific chroot-definitions. > > > The second patch, is a bit different: > While pushing the Fedora Core SRPMS through the mock > buildsystem we noticed many SRPMS which do not correctly > define their buildrequirements. > Until the fedora core developer responsible for the package > has updated the spec in cvs and pushed an updated srpm into > rawhide, there is no way to make the package in question > build in mock without having to change the .spec. > Thus, in order to allow even these broken packages being > built, we needed a way to work around these broken > dependencies to see if anything else was missing. > This is done by the second patch, which adds a more_buildreqs > config option. If set the given dependencies are installed > _in addition_ to the ones from the .spec file. > > While the second patch should not be used during normal use > and is definetly not for the fedora-extras buildservers, it > still is of interest to other parties, using mock. > > > > regards, > andreas > From skvidal at linux.duke.edu Fri May 12 20:53:32 2006 From: skvidal at linux.duke.edu (seth vidal) Date: Fri, 12 May 2006 16:53:32 -0400 Subject: interactive shell in chroot patch In-Reply-To: <4463706A.2050400@redhat.com> References: <4463706A.2050400@redhat.com> Message-ID: <1147467213.3404.46.camel@cutter> On Thu, 2006-05-11 at 12:12 -0500, Clark Williams wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > While we're in patch-a-thon mode, thought I'd submit this for comment. > > While attempting to track down a bizarre bug with elinks, I hacked > mock so that I could get an interactive shell in the chroot > environment. Not useful for everyone, but handy if you want to look at > the environment in which something is failing. Oh and I converted some > prints in my chroot command to use the debug method. > check it in. -sv From skvidal at linux.duke.edu Fri May 12 20:56:29 2006 From: skvidal at linux.duke.edu (seth vidal) Date: Fri, 12 May 2006 16:56:29 -0400 Subject: mock goals and non-goals In-Reply-To: References: <1147356877.1918.34.camel@cutter> Message-ID: <1147467389.3404.50.camel@cutter> On Thu, 2006-05-11 at 21:10 +0200, Andreas Thienemann wrote: > Hello, > > On Thu, 11 May 2006, seth vidal wrote: > > > goals: > > - make clean and consistent build environments to be used by fedora's > > buildsystem and packagers > > - simplify building and testing process for packagers > > - be able to canonically identify the 'build environment' for fedora > > systems to discourage confusion on what should be where > > - simple interface with good defaults. > Why not a bit more generic and see mock as a tool to build rpms in a clean > and consistent buildroot. After all, we heard that SuSE RPMs can be built > with it too. And that's a feat s the last time I looked SuSE RPMs were > pretty bad dependency-wise. then the suse config would have a different buildsys-build package/set of pkgs. > non-goals: > > - this is not a tool to manage many, many builds with queuing, etc - > > that is plague. > ack IMHO. Even though plague needs still a bit of work it seems. livna's > plague installation only became stable as soon as I sent them a few > patches, which dcbw handed me on irc after bugging him about a few > problems. ;-D which is why plague gets worked on. good. > > course, the --no-clean option is passed in > > - caching mechanism to create a cached pristine chroot that will allow > > for quickly recreating the chroot into that state simply by putting that > > cached copy back. > This would be simply great. Right now, the cycle of cearting the chroot, > installing the base-packages, doing the full depsolve-cycle, building the > rpm and then removing the buildroot again takes ages. > Something faster there might be great. look at the caching patch jochen submitted some time back. Clean it up and write up what it is it does and we'll put it in. > > Patches which I am ambivalent about: > > - the fake buildreqs/sneaky buildreqs patch for rebuilding items with > > BROKEN buildrequires: > > - I hate the idea of enabling broken shit > > - I realize that there are lots of broken things that people need to > > work around and I'm sympathetic to that, to some extent. Doesn't mean I > > can't hate it. > Well, nobody expects you to like it, it just has to be in mock. ;-D which is why i put it under ambivalence. -sv From skvidal at linux.duke.edu Fri May 12 20:59:53 2006 From: skvidal at linux.duke.edu (seth vidal) Date: Fri, 12 May 2006 16:59:53 -0400 Subject: [PATCH] resubmit: multisrpm build In-Reply-To: <35C9A9D68AB3FA4AB63692802656D9EC010F4385@ausx3mps303.aus.amer.dell.com> References: <35C9A9D68AB3FA4AB63692802656D9EC010F4385@ausx3mps303.aus.amer.dell.com> Message-ID: <1147467593.3404.53.camel@cutter> On Thu, 2006-05-11 at 11:41 -0500, Michael_E_Brown at Dell.com wrote: > This is a resubmit of the patch from yesterday to build > multiple srpms in one command line: > > $ mock -r CFG srpm1 srpm2 srpm3 > > It exits immediately on the first failure. > It cleans and preps the build environment for each srpm unless > --no-clean is passed. committed -sv From skvidal at linux.duke.edu Fri May 12 21:01:11 2006 From: skvidal at linux.duke.edu (seth vidal) Date: Fri, 12 May 2006 17:01:11 -0400 Subject: Patches for shared configurations items and specifying additional dependencies In-Reply-To: References: Message-ID: <1147467671.3404.55.camel@cutter> On Wed, 2006-05-10 at 22:26 +0200, Andreas Thienemann wrote: > Hello List, > > as it seems to be patch submission time, I'm pushing two patches to the > list, I had lying around for some time now. > > The first patch allows mock to parse a generic side-wide config file, > where general configuration-items can be stored and later overridden by > specific chroot-definitions. > Andreas, Can you recheck both of these patches against a cvs check out of mock - make sure they apply? thanks, -sv From williams at redhat.com Fri May 12 20:58:44 2006 From: williams at redhat.com (Clark Williams) Date: Fri, 12 May 2006 15:58:44 -0500 Subject: interactive shell in chroot patch In-Reply-To: <1147467213.3404.46.camel@cutter> References: <4463706A.2050400@redhat.com> <1147467213.3404.46.camel@cutter> Message-ID: <4464F704.3020200@redhat.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 seth vidal wrote: > check it in. > > -sv Done Clark -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFEZPcEHyuj/+TTEp0RAkb5AJ4x+c78RaEOjOUss9NiQ4bIQLZyZQCeNiNd n15cs5OgapTA/F+pH/LpnDQ= =f5QF -----END PGP SIGNATURE----- From hun at n-dimensional.de Fri May 12 23:46:59 2006 From: hun at n-dimensional.de (Hans Ulrich Niedermann) Date: Sat, 13 May 2006 01:46:59 +0200 Subject: mock goals and non-goals In-Reply-To: <1147356877.1918.34.camel@cutter> (seth vidal's message of "Thu, 11 May 2006 10:14:37 -0400") References: <1147356877.1918.34.camel@cutter> Message-ID: <86wtcqu7po.fsf@n-dimensional.de> seth vidal writes: > Patches which I am ambivalent about: > - the fake buildreqs/sneaky buildreqs patch for rebuilding items with > BROKEN buildrequires: > - I hate the idea of enabling broken shit > - I realize that there are lots of broken things that people need to > work around and I'm sympathetic to that, to some extent. Doesn't mean I > can't hate it. Would it help if I tried to sell you that feature as a "build dependency problem debugging tool"? These are the arguments: 1. Allowing an easy way add fake buildreqs allows one to easily add, remove again and thus test build dependencies in a reproducible way. This helps greatly to determine the exact set of missing builddeps. 2. The additional builddeps determined in step 1. are in a simple and convenient data format for conversion into a bug reports to the respective package maintainers. Documenting the feature using this line of reasoning might also help people get the right idea and do the right thing. Uli From sheltren at cs.ucsb.edu Sat May 13 11:34:53 2006 From: sheltren at cs.ucsb.edu (Jeff Sheltren) Date: Sat, 13 May 2006 07:34:53 -0400 Subject: Avoiding install of x86_64 packages for i386 builds Message-ID: <3D00B383-9729-45BD-911F-D19B4AB54292@cs.ucsb.edu> I'm having a problem when building packages which have a dependency sitting in the plague 'repo_dir'. Since plague puts all arches of a package in the same sub directory under 'repo_dir', if a package has both i386 and x86_64 binaries, later builds which pull in those packages as a dependency will install the x86_64 binary rather than the i386 one. Yes, I have brought this problem on myself - I want to have stuff in repo_dir be usable for builds even before those packages are pushed to the main repository. In order to do this, I have to point mock to the same repodir for both i386 and x86_64. So, I have a couple of questions: is there a way to have plague sort packages in the repo_dir differently than it does by default? How does Extras avoid this problem? Here's an example of the directory structure used: ---- Snip from /etc/mock/fedora-3-i386-core.cfg --- [updates-plague] name=updates-plague baseurl=file:///build/plague/repodir/fedora-3-core/ ---- End Snip ---- $ pwd /build/plague/repodir/fedora-3-core $ ls mozilla repodata $ ls mozilla/1.7.13-1.3.1.legacy/ i386 SRPM x86_64 Thanks, Jeff From dennis at ausil.us Sat May 13 13:43:58 2006 From: dennis at ausil.us (Dennis Gilmore) Date: Sat, 13 May 2006 08:43:58 -0500 Subject: Avoiding install of x86_64 packages for i386 builds In-Reply-To: <3D00B383-9729-45BD-911F-D19B4AB54292@cs.ucsb.edu> References: <3D00B383-9729-45BD-911F-D19B4AB54292@cs.ucsb.edu> Message-ID: <200605130843.58973.dennis@ausil.us> On Saturday 13 May 2006 6:34 am, Jeff Sheltren wrote: > I'm having a problem when building packages which have a dependency > sitting in the plague 'repo_dir'. Since plague puts all arches of a > package in the same sub directory under 'repo_dir', if a package has > both i386 and x86_64 binaries, later builds which pull in those > packages as a dependency will install the x86_64 binary rather than > the i386 one. Yes, I have brought this problem on myself - I want to > have stuff in repo_dir be usable for builds even before those > packages are pushed to the main repository. In order to do this, I > have to point mock to the same repodir for both i386 and x86_64. > > So, I have a couple of questions: is there a way to have plague sort > packages in the repo_dir differently than it does by default? How > does Extras avoid this problem? have a look in the x86_64 mock config files there is a line exclude=*.i386 *.i686 *.athlon *.i586 *.486 to your i386 mock configs add exclude=*.x86_64 -- Dennis Gilmore, RHCE Proud Australian From sheltren at cs.ucsb.edu Sat May 13 14:23:12 2006 From: sheltren at cs.ucsb.edu (Jeff Sheltren) Date: Sat, 13 May 2006 10:23:12 -0400 Subject: Avoiding install of x86_64 packages for i386 builds In-Reply-To: <200605130843.58973.dennis@ausil.us> References: <3D00B383-9729-45BD-911F-D19B4AB54292@cs.ucsb.edu> <200605130843.58973.dennis@ausil.us> Message-ID: <64FA3367-11FA-4C99-B165-E8BF2D521DEC@cs.ucsb.edu> On May 13, 2006, at 9:43 AM, Dennis Gilmore wrote: > On Saturday 13 May 2006 6:34 am, Jeff Sheltren wrote: >> I'm having a problem when building packages which have a dependency >> sitting in the plague 'repo_dir'. Since plague puts all arches of a >> package in the same sub directory under 'repo_dir', if a package has >> both i386 and x86_64 binaries, later builds which pull in those >> packages as a dependency will install the x86_64 binary rather than >> the i386 one. Yes, I have brought this problem on myself - I want to >> have stuff in repo_dir be usable for builds even before those >> packages are pushed to the main repository. In order to do this, I >> have to point mock to the same repodir for both i386 and x86_64. >> >> So, I have a couple of questions: is there a way to have plague sort >> packages in the repo_dir differently than it does by default? How >> does Extras avoid this problem? > have a look in the x86_64 mock config files there is a line > exclude=*.i386 *.i686 *.athlon *.i586 *.486 > > to your i386 mock configs add > exclude=*.x86_64 That was easy - thanks for bringing it to my attention! I don't know why I didn't think of that... -Jeff From andreas at bawue.net Sat May 13 21:53:44 2006 From: andreas at bawue.net (Andreas Thienemann) Date: Sat, 13 May 2006 23:53:44 +0200 (CEST) Subject: Patches for shared configurations items and specifying additional dependencies In-Reply-To: <1147467671.3404.55.camel@cutter> References: <1147467671.3404.55.camel@cutter> Message-ID: Hi Seth, On Fri, 12 May 2006, seth vidal wrote: > Can you recheck both of these patches against a cvs check out of mock - > make sure they apply? Done. The shared-defaults patch had to be refactored a bit, but they do apply now. Both are attached. regards, andreas -- Bawue.Net http://www.bawue.net Demetriusweg 15 Tel: +49 7033-4007956 70563 Stuttgart Fax: +49 7033-4007955 -------------- next part -------------- Provide generic config defaults. * Support a generic /etc/mock/defaults.cfg config file. The values defined in this file are overwritten by any definitions in the specific /etc/mock/CHROOT.cfg files. This part just introduces mock-0.4/etc/defaults.cfg and adds the last chunk in the patch to mock-0.4/mock.py. This patch was written in 2006 for bawue.net by Hans Ulrich Niedermann in 2006 after an idea by Andreas Thienemann. Testing and final clean-up was one by Andreas Thienemann. The source code this patch is based on has been downloaded from http://fedoraproject.org/projects/mock/releases/mock-0.4.tar.gz etc/defaults.cfg | 8 ++++++++ mock.py | 8 ++++++++ 2 files changed, 16 insertions(+) diff -uNr mock-20060513.orig/etc/defaults.cfg mock-20060513/etc/defaults.cfg --- mock-20060513.orig/etc/defaults.cfg 1970-01-01 01:00:00.000000000 +0100 +++ mock-20060513/etc/defaults.cfg 2006-05-13 16:46:19.480038297 +0200 @@ -0,0 +1,8 @@ +# mock defaults +# +# Define default values here. +# These values are overwritten in the /etc/mock/CHROOT.cfg files. +# +# Example: +# +# config_opts['foo'] = bar diff -uNr mock-20060513.orig/mock.py mock-20060513/mock.py --- mock-20060513.orig/mock.py 2006-05-13 16:38:46.487358000 +0200 +++ mock-20060513/mock.py 2006-05-13 16:47:52.874952632 +0200 @@ -735,6 +735,14 @@ if options.configdir: config_path = options.configdir + # Read in the default values which can be overwritten + # with the more specific config being loaded below. + cfg = os.path.join(config_path, 'defaults.cfg') + if os.path.exists(cfg): + execfile(cfg) + else: + pass # not finding the defaults.cfg file is no error + # read in the config file by chroot name if options.chroot.endswith('.cfg'): cfg = '%s/%s' % (config_path, options.chroot) -------------- next part -------------- Provide extra build requirements for packages. * Introduce a new config variable 'more_buildreqs' which allows defining additional build dependencies for specific packages. This is useful for buggy SRPMs or SRPMs which compile some parts conditionally, depending on the software installed. This mechanism should not be used in normal operation. The extra build requirements can be configured like in the following example, in which we use the example of a fictitional SRPM file called 'ser-0.9.6-3'. # matches only the 'ser-0.9.6-3' package config_opts['more_buildreqs']['ser-0.9.6-3'] = [ 'smtp-devel >= 1.2', 'mysql-devel >= 5.0', 'png-devel < 0.1', 'tcsh' ] # matches all 'ser-0.9.6-*' packages config_opts['more_buildreqs']['ser-0.9.6'] = [ 'smtp-devel >= 1.2', 'mysql-devel >= 5.0', 'png-devel < 0.1', 'tcsh' ] # matches all 'ser-*-*' packages config_opts['more_buildreqs']['ser'] = [ 'smtp-devel >= 1.2', 'mysql-devel >= 5.0', 'png-devel < 0.1', 'tcsh' ] The most specific match is used, i.e. if requirements for 'ser-0.9.6-3' are defined, the definitions for 'ser-0.9.6' and 'ser' are ignored. Each time, /etc/mock/CHROOT.cfg is checked first, and only if there is no dependency definition here, the definition from /etc/mock/defaults.cfg is checked. This patch was written in 2006 for bawue.net by Hans Ulrich Niedermann in 2006 after an idea by Andreas Thienemann. Testing and final clean-up was done by Andreas Thienemann. The source code this patch is based on has been downloaded from http://fedoraproject.org/projects/mock/releases/mock-0.4.tar.gz mock.py | 22 ++++++++++++++++++++-- 1 files changed, 20 insertions(+), 2 deletions(-) diff -uNr mock-20060513.orig/mock.py mock-20060513/mock.py --- mock-20060513.orig/mock.py 2006-05-13 16:38:46.487358000 +0200 +++ mock-20060513/mock.py 2006-05-13 23:46:51.789001516 +0200 @@ -289,7 +289,7 @@ hdr = rpmUtils.miscutils.hdrFromPackage(ts, srpm) # get text buildreqs - buildreqs = self._text_requires_from_hdr(hdr) + buildreqs = self._text_requires_from_hdr(hdr, srpm) arg_string = "" for item in buildreqs: @@ -495,7 +495,7 @@ return (ret, output) - def _text_requires_from_hdr(self, hdr): + def _text_requires_from_hdr(self, hdr, srpm): """take a header and hand back a unique'd list of the requires as strings""" @@ -513,6 +513,23 @@ req = rpmUtils.miscutils.formatRequire(n, v, f) reqlist.append(req) + # Extract SRPM name components - still not nice, shouldn't this + # be somewhere in the "hdr" parameter? + fname = os.path.split(str(srpm))[1] + name, ver, rel, epoch, arch = rpmUtils.miscutils.splitFilename(fname) + + # Add the 'more_buildreqs' for this SRPM (if defined) + for this_srpm in ['-'.join([name,ver,rel]), + '-'.join([name,ver]), + '-'.join([name]),]: + if self.config['more_buildreqs'].has_key(this_srpm): + more_reqs = self.config['more_buildreqs'][this_srpm] + if type(more_reqs) in (type(u''), type(''),): + more_reqs = [more_reqs] # be nice if we get a string + for req in more_reqs: + reqlist.append(req) + break + return rpmUtils.miscutils.unique(reqlist) def _prep_install(self): @@ -722,6 +739,7 @@ """ % config_opts['chroothome'] + config_opts['more_buildreqs'] = {} config_opts['files']['/etc/resolv.conf'] = "nameserver 192.168.1.1\n" config_opts['files']['/etc/hosts'] = "127.0.0.1 localhost localhost.localdomain\n" From skvidal at linux.duke.edu Sun May 14 04:58:34 2006 From: skvidal at linux.duke.edu (seth vidal) Date: Sun, 14 May 2006 00:58:34 -0400 Subject: Patches for shared configurations items and specifying additional dependencies In-Reply-To: References: <1147467671.3404.55.camel@cutter> Message-ID: <1147582715.8403.33.camel@cutter> On Sat, 2006-05-13 at 23:53 +0200, Andreas Thienemann wrote: > Hi Seth, > > On Fri, 12 May 2006, seth vidal wrote: > > > Can you recheck both of these patches against a cvs check out of mock - > > make sure they apply? > Done. > The shared-defaults patch had to be refactored a bit, but they do apply > now. > > Both are attached. these have been committed, thanks. -sv From Michael_E_Brown at dell.com Tue May 16 03:55:31 2006 From: Michael_E_Brown at dell.com (Michael E Brown) Date: Mon, 15 May 2006 22:55:31 -0500 Subject: [TRIVIAL PATCH] - Tab cleanup Message-ID: <1147751732.4046.8.camel@soltek.michaels-house.net> Attached is a patch to clean up a tab-error in my patch from last week. I accidentally let a tab creep into one of the comments. Please apply. -------------- next part -------------- A non-text attachment was scrubbed... Name: mock-smallfixes.patch Type: text/x-patch Size: 575 bytes Desc: not available URL: From Michael_E_Brown at dell.com Tue May 16 03:59:44 2006 From: Michael_E_Brown at dell.com (Michael E Brown) Date: Mon, 15 May 2006 22:59:44 -0500 Subject: [PATCH] slim main() Message-ID: <1147751984.4046.14.camel@soltek.michaels-house.net> The attached patch contains no functional changes. It simply introduces a few helper functions and slims down main(). def setup_default_config_opts(config_opts): move the setup of default config_opts to a helper function def set_config_opts_per_cmdline(config_opts, options): All of the logic that sets config_opts stuff based on cmdline options is here. def do_clean(config_opts, init=0): Consolidate clean/init handling into one function def do_run_cmd(config_opts, cmd, raw_chroot=0): Consolidate chroot/shell handling into one function def ensure_filetype_srpm(srpms): Check all cmdline srpm args to ensure they exist and are srpms def do_rebuild(config_opts, srpms): rebuild logic. Please apply. -------------- next part -------------- A non-text attachment was scrubbed... Name: mock-slimmain.patch Type: text/x-patch Size: 9248 bytes Desc: not available URL: From skvidal at linux.duke.edu Tue May 16 04:16:19 2006 From: skvidal at linux.duke.edu (seth vidal) Date: Tue, 16 May 2006 00:16:19 -0400 Subject: [TRIVIAL PATCH] - Tab cleanup In-Reply-To: <1147751732.4046.8.camel@soltek.michaels-house.net> References: <1147751732.4046.8.camel@soltek.michaels-house.net> Message-ID: <1147752979.3034.2.camel@cutter> On Mon, 2006-05-15 at 22:55 -0500, Michael E Brown wrote: > Attached is a patch to clean up a tab-error in my patch from last week. > I accidentally let a tab creep into one of the comments. > > Please apply. applied, thanks. -sv From skvidal at linux.duke.edu Tue May 16 04:19:14 2006 From: skvidal at linux.duke.edu (seth vidal) Date: Tue, 16 May 2006 00:19:14 -0400 Subject: [PATCH] slim main() In-Reply-To: <1147751984.4046.14.camel@soltek.michaels-house.net> References: <1147751984.4046.14.camel@soltek.michaels-house.net> Message-ID: <1147753154.3034.4.camel@cutter> On Mon, 2006-05-15 at 22:59 -0500, Michael E Brown wrote: > The attached patch contains no functional changes. It simply introduces > a few helper functions and slims down main(). > also applied, thanks. -sv From andreas at bawue.net Tue May 16 09:25:43 2006 From: andreas at bawue.net (Andreas Thienemann) Date: Tue, 16 May 2006 11:25:43 +0200 (CEST) Subject: Multiarch support for mock Message-ID: Hello, I noticed some problems with mock, when building tools such as valgrind, syslinux etc. on x86_64. The default mock config says to exclude i386 and similar packages. However, these programs depend on stubs-32.h from the i386 glibc-devel package. The x86_64 glibc-devel only brings stubs-64.h, which makes the compilation of these tools stop, because stubs-32.h could not be found. What would be the correct way of fixing this? regards, andreas From laroche at redhat.com Tue May 16 09:33:54 2006 From: laroche at redhat.com (Florian La Roche) Date: Tue, 16 May 2006 11:33:54 +0200 Subject: Multiarch support for mock In-Reply-To: References: Message-ID: <20060516093354.GA3956@dudweiler.stuttgart.redhat.com> On Tue, May 16, 2006 at 11:25:43AM +0200, Andreas Thienemann wrote: > Hello, > > I noticed some problems with mock, when building tools such as valgrind, > syslinux etc. on x86_64. > > The default mock config says to exclude i386 and similar packages. > > However, these programs depend on stubs-32.h from the i386 glibc-devel > package. > The x86_64 glibc-devel only brings stubs-64.h, which makes the compilation > of these tools stop, because stubs-32.h could not be found. > > What would be the correct way of fixing this? You should still keep the 32bit glibc rpm packages. I've had this running at some point with the following yum repo: [devel-x86_64] name=Fedora Core Development for x86_64 baseurl=file:/mnt/raid/fedora/development/x86_64 exclude=*-debuginfo-* [!g]*.i686 [!g]*.i386 g[!l]*.i386 glib.i386 glib2.i386 Not sure the above still works or if there are nicer ways now todo the same within mock via other mechanisms. It should be something like "exclude all *.i686/*.i386 packages, but still keep glibc*.i686 installed". regards, Florian La Roche From skvidal at linux.duke.edu Tue May 16 14:48:39 2006 From: skvidal at linux.duke.edu (seth vidal) Date: Tue, 16 May 2006 10:48:39 -0400 Subject: Multiarch support for mock In-Reply-To: <20060516093354.GA3956@dudweiler.stuttgart.redhat.com> References: <20060516093354.GA3956@dudweiler.stuttgart.redhat.com> Message-ID: <1147790919.4737.8.camel@cutter> On Tue, 2006-05-16 at 11:33 +0200, Florian La Roche wrote: > On Tue, May 16, 2006 at 11:25:43AM +0200, Andreas Thienemann wrote: > > Hello, > > > > I noticed some problems with mock, when building tools such as valgrind, > > syslinux etc. on x86_64. > > > > The default mock config says to exclude i386 and similar packages. > > > > However, these programs depend on stubs-32.h from the i386 glibc-devel > > package. > > The x86_64 glibc-devel only brings stubs-64.h, which makes the compilation > > of these tools stop, because stubs-32.h could not be found. > > > > What would be the correct way of fixing this? > > You should still keep the 32bit glibc rpm packages. > > I've had this running at some point with the following yum repo: > [devel-x86_64] > name=Fedora Core Development for x86_64 > baseurl=file:/mnt/raid/fedora/development/x86_64 > exclude=*-debuginfo-* [!g]*.i686 [!g]*.i386 g[!l]*.i386 glib.i386 glib2.i386 > > > Not sure the above still works or if there are nicer ways now todo the > same within mock via other mechanisms. It should be something like > "exclude all *.i686/*.i386 packages, but still keep glibc*.i686 installed". > > So you're saying that in order to build 64bit packages we REQUIRE glibc*.i686? Are you serious? So there's no way to build x86_64 w/o i386 first being built? -sv From Michael_E_Brown at Dell.com Tue May 16 16:08:51 2006 From: Michael_E_Brown at Dell.com (Michael_E_Brown at Dell.com) Date: Tue, 16 May 2006 11:08:51 -0500 Subject: Multiarch support for mock Message-ID: <35C9A9D68AB3FA4AB63692802656D9EC010F43A3@ausx3mps303.aus.amer.dell.com> > -----Original Message----- > From: fedora-buildsys-list-bounces at redhat.com > [mailto:fedora-buildsys-list-bounces at redhat.com] On Behalf Of > seth vidal > On Tue, 2006-05-16 at 11:33 +0200, Florian La Roche wrote: > > On Tue, May 16, 2006 at 11:25:43AM +0200, Andreas Thienemann wrote: > > > Hello, > > > > > > I noticed some problems with mock, when building tools such as > > > valgrind, syslinux etc. on x86_64. > > > > > > The default mock config says to exclude i386 and similar packages. > > > > > > However, these programs depend on stubs-32.h from the i386 > > > glibc-devel package. > > > The x86_64 glibc-devel only brings stubs-64.h, which makes the > > > compilation of these tools stop, because stubs-32.h could > not be found. > > > > > > What would be the correct way of fixing this? > > > > You should still keep the 32bit glibc rpm packages. > > > > I've had this running at some point with the following yum repo: > > [devel-x86_64] > > name=Fedora Core Development for x86_64 > > baseurl=file:/mnt/raid/fedora/development/x86_64 > > exclude=*-debuginfo-* [!g]*.i686 [!g]*.i386 g[!l]*.i386 glib.i386 > > glib2.i386 > > > > > > Not sure the above still works or if there are nicer ways > now todo the > > same within mock via other mechanisms. It should be something like > > "exclude all *.i686/*.i386 packages, but still keep > glibc*.i686 installed". > > > > > > So you're saying that in order to build 64bit packages we > REQUIRE glibc*.i686? > > Are you serious? So there's no way to build x86_64 w/o i386 > first being built? Wouldn't a "BuildRequires: glibc.i686" work ok? Or am I missing something? -- Michael From skvidal at linux.duke.edu Tue May 16 18:26:54 2006 From: skvidal at linux.duke.edu (seth vidal) Date: Tue, 16 May 2006 14:26:54 -0400 Subject: Multiarch support for mock In-Reply-To: <35C9A9D68AB3FA4AB63692802656D9EC010F43A3@ausx3mps303.aus.amer.dell.com> References: <35C9A9D68AB3FA4AB63692802656D9EC010F43A3@ausx3mps303.aus.amer.dell.com> Message-ID: <1147804014.4737.17.camel@cutter> On Tue, 2006-05-16 at 11:08 -0500, Michael_E_Brown at Dell.com wrote: > > -----Original Message----- > > From: fedora-buildsys-list-bounces at redhat.com > > [mailto:fedora-buildsys-list-bounces at redhat.com] On Behalf Of > > seth vidal > > On Tue, 2006-05-16 at 11:33 +0200, Florian La Roche wrote: > > > On Tue, May 16, 2006 at 11:25:43AM +0200, Andreas Thienemann wrote: > > > > Hello, > > > > > > > > I noticed some problems with mock, when building tools such as > > > > valgrind, syslinux etc. on x86_64. > > > > > > > > The default mock config says to exclude i386 and similar packages. > > > > > > > > However, these programs depend on stubs-32.h from the i386 > > > > glibc-devel package. > > > > The x86_64 glibc-devel only brings stubs-64.h, which makes the > > > > compilation of these tools stop, because stubs-32.h could > > not be found. > > > > > > > > What would be the correct way of fixing this? > > > > > > You should still keep the 32bit glibc rpm packages. > > > > > > I've had this running at some point with the following yum repo: > > > [devel-x86_64] > > > name=Fedora Core Development for x86_64 > > > baseurl=file:/mnt/raid/fedora/development/x86_64 > > > exclude=*-debuginfo-* [!g]*.i686 [!g]*.i386 g[!l]*.i386 glib.i386 > > > glib2.i386 > > > > > > > > > Not sure the above still works or if there are nicer ways > > now todo the > > > same within mock via other mechanisms. It should be something like > > > "exclude all *.i686/*.i386 packages, but still keep > > glibc*.i686 installed". > > > > > > > > > > So you're saying that in order to build 64bit packages we > > REQUIRE glibc*.i686? > > > > Are you serious? So there's no way to build x86_64 w/o i386 > > first being built? > > Wouldn't a "BuildRequires: glibc.i686" work ok? Or am I missing > something? it would work to pull in the package in mock - b/c it would be passed directly to yum which understands that string. However, it won't work otherwise b/c rpm doesn't know about archs in Requires or BuildRequires. -sv From Michael_E_Brown at Dell.com Tue May 16 19:01:24 2006 From: Michael_E_Brown at Dell.com (Michael_E_Brown at Dell.com) Date: Tue, 16 May 2006 14:01:24 -0500 Subject: [PATCH] Fix PS1 prompt for shell Message-ID: <35C9A9D68AB3FA4AB63692802656D9EC010F43A8@ausx3mps303.aus.amer.dell.com> Not quite a brown-paper-bag bug, but I broke the PS1 prompt for the shell command. Not all that earth shattering, but the following trivial patch fixes it. It was due to using an old mock-helper which didn't have the PS1 in ALLOWED_ENVS and the fact that I didn't notice that os.system() doesn't pass along changes to os.environ. -- Michael <> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: mock-fixps1.patch Type: application/octet-stream Size: 1136 bytes Desc: mock-fixps1.patch URL: From Michael_E_Brown at Dell.com Tue May 16 19:07:28 2006 From: Michael_E_Brown at Dell.com (Michael_E_Brown at Dell.com) Date: Tue, 16 May 2006 14:07:28 -0500 Subject: [PATCH - RFC] unify config parsing Message-ID: <35C9A9D68AB3FA4AB63692802656D9EC010F43A9@ausx3mps303.aus.amer.dell.com> The recent changes to add default.cfg processing bumped the number of lines in main(). The following patch unifies all config file parsing into one small loop, removing about a dozen redundant lines of code. It also removes an unnecessary variable, config_path and uses options.configdir directly. This is an RFC because it changes mock behaviour in the following way, which requres discussion: The "-r" option is no longer required. You will not get an error message for a missing "-r" option. I think this might be useful for the case where somebody sets up their 'default.cfg' to be a full config. Comments? -- Michael <> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: mock-cfgunify.patch Type: application/octet-stream Size: 2045 bytes Desc: mock-cfgunify.patch URL: From Michael_E_Brown at Dell.com Tue May 16 19:11:25 2006 From: Michael_E_Brown at Dell.com (Michael_E_Brown at Dell.com) Date: Tue, 16 May 2006 14:11:25 -0500 Subject: Multiarch support for mock Message-ID: <35C9A9D68AB3FA4AB63692802656D9EC010F43AA@ausx3mps303.aus.amer.dell.com> > -----Original Message----- > From: fedora-buildsys-list-bounces at redhat.com > [mailto:fedora-buildsys-list-bounces at redhat.com] On Behalf Of > seth vidal > Sent: Tuesday, May 16, 2006 1:27 PM > To: Discussion of Fedora build system > Subject: RE: Multiarch support for mock > > On Tue, 2006-05-16 at 11:08 -0500, Michael_E_Brown at Dell.com wrote: > > Wouldn't a "BuildRequires: glibc.i686" work ok? Or am I missing > > something? > > it would work to pull in the package in mock - b/c it would > be passed directly to yum which understands that string. > However, it won't work otherwise b/c rpm doesn't know about > archs in Requires or BuildRequires. SUSE build.rpm has a format that it uses for buildrequires, which I thought was somewhat 'special' (short-bus type): it uses specially-formatted comments in the spec file to embed build requires lines. Does it make sense to add this as a workaround for this special-case? Ie. Add a special "# mock-build-requires: glibc.i386" for this RPM so that we do not break RPM, but still work with mock? This probably isn't optimal, but a better solution eludes me atm. -- Michael From Michael_E_Brown at Dell.com Tue May 16 19:52:46 2006 From: Michael_E_Brown at Dell.com (Michael_E_Brown at Dell.com) Date: Tue, 16 May 2006 14:52:46 -0500 Subject: [RFC] in progress: autocache patch Message-ID: <35C9A9D68AB3FA4AB63692802656D9EC010F43AB@ausx3mps303.aus.amer.dell.com> Here is my current autocache patch for comments. I have two features left to implement: 1) permissions on tarballs, and 2) ability to select compressed/uncompressed tarballs. Usage: There are two added command line options. First is "--autocache". You should be able to just add this command line option to all mock invocations to automatically and transparently speed up operations. It will automatically build cache the first time it is invoked. It will automatically use the cache if it exists. It will automatically expire the cache and rebuild it on a periodic basis. The second option is "--rebuildcache". This option is for the rare occasions when you would like to force a rebuild of the cache. Theory: The theory behind this patch is that it will automatically untar the buildroot from a tarball, if the tarball exists. It will automatically create the tarball if it does not exist (only when --autocache is specified). The tarball has an expiration date, default is 15 days. After the tarball is older than this, it will automatically be rebuilt. Test results: Initial testing with FC5 yum repo mirror on local network and squid proxy running on local machine resulted in a "init" of the build root at 2:20. Using an already-cached build root takes this time down to 0:13. -- Michael <> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: mock-autocache.patch Type: application/octet-stream Size: 4466 bytes Desc: mock-autocache.patch URL: From Michael_E_Brown at Dell.com Tue May 16 20:13:46 2006 From: Michael_E_Brown at Dell.com (Michael_E_Brown at Dell.com) Date: Tue, 16 May 2006 15:13:46 -0500 Subject: [RFC] in progress: autocache patch Message-ID: <35C9A9D68AB3FA4AB63692802656D9EC010F43AC@ausx3mps303.aus.amer.dell.com> A couple of notes: Credit should be given to Jochen Schmitt for his original patch: http://www.redhat.com/archives/fedora-extras-list/2005-November/msg00169 .html My mock-helper changes are straight from Jochen Schmitt's patch. All mock.py changes are mine, with inspiration from Jochen's patch. New Config options: There are several new config options that can be placed in the mock config file to control caching. See the additions to setup_default_config_opts() for details, they should be relatively self-explanatory. Note that "cache_topdir" is concatenated with "basedir" with os.path.join(). This means that if cache_topdir begins with a /, it will completely override basedir. -- Michael ________________________________ From: fedora-buildsys-list-bounces at redhat.com [mailto:fedora-buildsys-list-bounces at redhat.com] On Behalf Of Michael_E_Brown at Dell.com Sent: Tuesday, May 16, 2006 2:53 PM To: fedora-buildsys-list at redhat.com Subject: [RFC] in progress: autocache patch Here is my current autocache patch for comments. I have two features left to implement: 1) permissions on tarballs, and 2) ability to select compressed/uncompressed tarballs. Usage: There are two added command line options. First is "--autocache". You should be able to just add this command line option to all mock invocations to automatically and transparently speed up operations. It will automatically build cache the first time it is invoked. It will automatically use the cache if it exists. It will automatically expire the cache and rebuild it on a periodic basis. The second option is "--rebuildcache". This option is for the rare occasions when you would like to force a rebuild of the cache. Theory: The theory behind this patch is that it will automatically untar the buildroot from a tarball, if the tarball exists. It will automatically create the tarball if it does not exist (only when --autocache is specified). The tarball has an expiration date, default is 15 days. After the tarball is older than this, it will automatically be rebuilt. Test results: Initial testing with FC5 yum repo mirror on local network and squid proxy running on local machine resulted in a "init" of the build root at 2:20. Using an already-cached build root takes this time down to 0:13. -- Michael <> -------------- next part -------------- An HTML attachment was scrubbed... URL: From Michael_E_Brown at Dell.com Tue May 16 22:03:23 2006 From: Michael_E_Brown at Dell.com (Michael_E_Brown at Dell.com) Date: Tue, 16 May 2006 17:03:23 -0500 Subject: [PATCH] autocache patch for mock Message-ID: <35C9A9D68AB3FA4AB63692802656D9EC010F43AF@ausx3mps303.aus.amer.dell.com> Here is what I would consider to be a patch ready to check-in to implement automatic caching of the mock buildroot. This feature is disabled by default but can be turned on by the "--autocache" cli option or the "use_cache" config file option. The only feature not implmented is selectable compression. This can be implemented by changing mock-helper.c to check the file extension and only compress if end is .bz2 or .gz. I have not implemented this, after looking at the speed difference between compressed and uncompressed. It turns out that uncompressed is slower, probably due to disk IO bandwidth. (compressed tarball for fc5 is ~200MB. Uncompressed tarball is ~500MB) Feedback appreciated. -- Michael <> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: mock-autocache.patch Type: application/octet-stream Size: 6426 bytes Desc: mock-autocache.patch URL: From laroche at redhat.com Tue May 16 22:03:43 2006 From: laroche at redhat.com (Florian La Roche) Date: Wed, 17 May 2006 00:03:43 +0200 Subject: Multiarch support for mock In-Reply-To: <1147790919.4737.8.camel@cutter> References: <20060516093354.GA3956@dudweiler.stuttgart.redhat.com> <1147790919.4737.8.camel@cutter> Message-ID: <20060516220343.GA5707@dudweiler.stuttgart.redhat.com> On Tue, May 16, 2006 at 10:48:39AM -0400, seth vidal wrote: > On Tue, 2006-05-16 at 11:33 +0200, Florian La Roche wrote: > > On Tue, May 16, 2006 at 11:25:43AM +0200, Andreas Thienemann wrote: > > > Hello, > > > > > > I noticed some problems with mock, when building tools such as valgrind, > > > syslinux etc. on x86_64. > > > > > > The default mock config says to exclude i386 and similar packages. > > > > > > However, these programs depend on stubs-32.h from the i386 glibc-devel > > > package. > > > The x86_64 glibc-devel only brings stubs-64.h, which makes the compilation > > > of these tools stop, because stubs-32.h could not be found. > > > > > > What would be the correct way of fixing this? > > > > You should still keep the 32bit glibc rpm packages. > > > > I've had this running at some point with the following yum repo: > > [devel-x86_64] > > name=Fedora Core Development for x86_64 > > baseurl=file:/mnt/raid/fedora/development/x86_64 > > exclude=*-debuginfo-* [!g]*.i686 [!g]*.i386 g[!l]*.i386 glib.i386 glib2.i386 > > > > > > Not sure the above still works or if there are nicer ways now todo the > > same within mock via other mechanisms. It should be something like > > "exclude all *.i686/*.i386 packages, but still keep glibc*.i686 installed". > > > > > > So you're saying that in order to build 64bit packages we REQUIRE > glibc*.i686? > > Are you serious? So there's no way to build x86_64 w/o i386 first being > built? Hello Seth, No, just that glibc is an exception to the otherwise "only install 64bit rpm packages" rule for a buildroot. regards, Florian La Roche From tibbs at math.uh.edu Wed May 17 03:06:33 2006 From: tibbs at math.uh.edu (Jason L Tibbitts III) Date: Tue, 16 May 2006 22:06:33 -0500 Subject: [PATCH] autocache patch for mock In-Reply-To: <35C9A9D68AB3FA4AB63692802656D9EC010F43AF@ausx3mps303.aus.amer.dell.com> (Michael E. Brown's message of "Tue, 16 May 2006 17:03:23 -0500") References: <35C9A9D68AB3FA4AB63692802656D9EC010F43AF@ausx3mps303.aus.amer.dell.com> Message-ID: OK, since I do a bunch of builds I just had to try this out. It took me a while to get building again after moving from mock 0.4 to 0.5; I'm still not sure where the buildsys-build package is supposed to come from, so I just built the spec I found in mock CVS and stuck it in my local repo. Basic summary: this saves me 30 seconds per build; the time goes from 2:40 to 2:10 when building a basic package (perl-Expect) which has only one dependency outside of base. These timings are stable and run cache-warm. I'm building on a dual core quad socket Opteron 880 (2.4GHz) machine with 16GB of RAM; disk is a plain throwaway 250GB Western Digital connected via SATA to an onboard SiI 3114 controller. The machine is otherwise idle. It has sufficient memory to cache pretty much everything involved in the build process. Things are mostly IO bound on this machine, except for unpacking the cached buildroot which is completely CPU bound. I think that switching to gzip might help. Of course, building the cache takes some time; bzip is just incredibly slow. Time for a build after deleting the cache file is 5:43. Again, gzip would probably help; bzip2 is just so slow. I noticed one issue: tar is traversing proc. I thought I saw a patch where it was called with -l (--one-file-system) which would prevent this: tar: root/proc/acpi/event: Cannot open: Device or resource busy tar: root/proc/irq/11: file changed as we read it tar: root/proc/irq: file changed as we read it tar: root/proc/sys/dev/scsi: file changed as we read it tar: root/proc/sys/dev: file changed as we read it tar: root/proc/sys/net/ipv6/neigh/default: file changed as we read it tar: root/proc/sys/net/ipv6/neigh: file changed as we read it - J< From Michael_E_Brown at dell.com Wed May 17 03:18:47 2006 From: Michael_E_Brown at dell.com (Michael E Brown) Date: Tue, 16 May 2006 22:18:47 -0500 Subject: [PATCH] autocache patch for mock In-Reply-To: References: <35C9A9D68AB3FA4AB63692802656D9EC010F43AF@ausx3mps303.aus.amer.dell.com> Message-ID: <1147835927.4046.42.camel@soltek.michaels-house.net> Thanks for the feedback! On Tue, 2006-05-16 at 22:06 -0500, Jason L Tibbitts III wrote: > OK, since I do a bunch of builds I just had to try this out. It took > me a while to get building again after moving from mock 0.4 to 0.5; > I'm still not sure where the buildsys-build package is supposed to > come from, so I just built the spec I found in mock CVS and stuck it > in my local repo. To get my stuff working, I changed the config: config_opts['chroot_setup_cmd'] = 'groupinstall build build-minimal build-base' This emulates the old behaviour. It is a bit easier for now than building the spec, especially since it won't work for my suse builds yet. > > Basic summary: this saves me 30 seconds per build; the time goes from > 2:40 to 2:10 when building a basic package (perl-Expect) which has > only one dependency outside of base. These timings are stable and run > cache-warm. I tested just 'init' of the buildroot. The timings went from 2:30 down to :13 with warm cache on my machine. Of course first build of cache takes longer. From my experience, bzip2 gets much better compression than gzip, and since you are IO bound, doing less IO (smaller cache) should improve the timings. (see next point) > > I'm building on a dual core quad socket Opteron 880 (2.4GHz) machine > with 16GB of RAM; disk is a plain throwaway 250GB Western Digital > connected via SATA to an onboard SiI 3114 controller. The machine is > otherwise idle. It has sufficient memory to cache pretty much > everything involved in the build process. > > Things are mostly IO bound on this machine, except for unpacking the > cached buildroot which is completely CPU bound. I think that > switching to gzip might help. I'll work on that next. I'll make tar look at the file extension and switch between bzip2/gzip/no-compression. > > Of course, building the cache takes some time; bzip is just incredibly > slow. Time for a build after deleting the cache file is 5:43. Again, > gzip would probably help; bzip2 is just so slow. > > I noticed one issue: tar is traversing proc. I thought I saw a patch > where it was called with -l (--one-file-system) which would prevent this: Thanks! That should help a bit, I'll make this change to mock-helper.c. > > tar: root/proc/acpi/event: Cannot open: Device or resource busy > tar: root/proc/irq/11: file changed as we read it > tar: root/proc/irq: file changed as we read it > tar: root/proc/sys/dev/scsi: file changed as we read it > tar: root/proc/sys/dev: file changed as we read it > tar: root/proc/sys/net/ipv6/neigh/default: file changed as we read it > tar: root/proc/sys/net/ipv6/neigh: file changed as we read it I'll send a new patch shortly. -- Michael From Michael_E_Brown at dell.com Wed May 17 03:51:12 2006 From: Michael_E_Brown at dell.com (Michael E Brown) Date: Tue, 16 May 2006 22:51:12 -0500 Subject: [PATCH] autocache patch for mock -- try 2 In-Reply-To: <1147835927.4046.42.camel@soltek.michaels-house.net> References: <35C9A9D68AB3FA4AB63692802656D9EC010F43AF@ausx3mps303.aus.amer.dell.com> <1147835927.4046.42.camel@soltek.michaels-house.net> Message-ID: <1147837873.4046.50.camel@soltek.michaels-house.net> On Tue, 2006-05-16 at 22:18 -0500, Michael E Brown wrote: > On Tue, 2006-05-16 at 22:06 -0500, Jason L Tibbitts III wrote: > > > > I'm building on a dual core quad socket Opteron 880 (2.4GHz) machine > > with 16GB of RAM; disk is a plain throwaway 250GB Western Digital > > connected via SATA to an onboard SiI 3114 controller. The machine is > > otherwise idle. It has sufficient memory to cache pretty much > > everything involved in the build process. > > > > Things are mostly IO bound on this machine, except for unpacking the > > cached buildroot which is completely CPU bound. I think that > > switching to gzip might help. > > I'll work on that next. I'll make tar look at the file extension and > switch between bzip2/gzip/no-compression. Implemented. Change the cache_ext config option. Endings with .bz2 and .gz are handled appropriately. Others are uncached. Attached is a lightly-tested patch. I wanted to get it out there, I'll finish a few more test runs on it tomorrow. > > > > > Of course, building the cache takes some time; bzip is just incredibly > > slow. Time for a build after deleting the cache file is 5:43. Again, > > gzip would probably help; bzip2 is just so slow. > > > > I noticed one issue: tar is traversing proc. I thought I saw a patch > > where it was called with -l (--one-file-system) which would prevent this: > > Thanks! That should help a bit, I'll make this change to mock-helper.c. Fixed. > I'll send a new patch shortly. Attached. Also fixed: one small bug where if you did separate "init" and then "rebuild --noclean" steps, it re-unpacked the cache unnecessarily. Also fixed: command line option is "--no-clean", opt parser variable is "dirty", and the config_opts variable is "clean". I've changed the last two to "clean" to make them consistent because I kept getting things backwards. -- Michael -------------- next part -------------- A non-text attachment was scrubbed... Name: mock-autocache-2.patch Type: text/x-patch Size: 7605 bytes Desc: not available URL: From Michael_E_Brown at dell.com Wed May 17 04:28:35 2006 From: Michael_E_Brown at dell.com (Michael E Brown) Date: Tue, 16 May 2006 23:28:35 -0500 Subject: [PATCH] autocache patch for mock -- try 2 In-Reply-To: <1147837873.4046.50.camel@soltek.michaels-house.net> References: <35C9A9D68AB3FA4AB63692802656D9EC010F43AF@ausx3mps303.aus.amer.dell.com> <1147835927.4046.42.camel@soltek.michaels-house.net> <1147837873.4046.50.camel@soltek.michaels-house.net> Message-ID: <1147840116.4046.60.camel@soltek.michaels-house.net> Attached are timings from various runs with bzip2, gzip, and no compression. Summary: gzip kicks butt. Mr Tibbitts was correct. Attached is a diff to change the default to gzip. :) Here is a summary (warm cache): No caching: real 1m36.247s build bz2 cache: real 4m28.174s build gzip cache: real 2m25.889s build nocomp cache: real 1m47.684s use bz2 cache: real 1m13.208s use gzip cache: real 0m20.854s use nocomp cache: real 0m24.060s The numbers above were from building a very small RPM which has no compilation. -- Michael -------------- next part -------------- A non-text attachment was scrubbed... Name: mock-autocache-gzip.patch Type: text/x-patch Size: 553 bytes Desc: not available URL: -------------- next part -------------- Full build, no cache, don't build cache: [michael_e_brown at mdomsch-pe1425-a ~]$ time mock -r fc5-i386 dell-repository-11-0.fc5.src.rpm init clean prep This may take a while setup build ending done Results and/or logs in: /var/lib/mock/fedora-5-i386-core/result First Run: real 1m51.105s user 0m46.835s sys 0m18.117s Second Run: real 1m36.247s user 0m46.895s sys 0m16.797s Full build, autocache, rebuild cache (bzip2): [michael_e_brown at mdomsch-pe1425-a ~]$ time mock -r fc5-i386 dell-repository-11-0.fc5.src.rpm --autocache init clean prep This may take a while create cache setup build ending done Results and/or logs in: /var/lib/mock/fedora-5-i386-core/result real 4m28.174s user 3m36.874s sys 0m20.877s Full build, autocache, using cache (bzip2): [michael_e_brown at mdomsch-pe1425-a ~]$ time mock -r fc5-i386 dell-repository-11-0.fc5.src.rpm --autocache init clean prep This may take a while unpack cache setup build ending done Results and/or logs in: /var/lib/mock/fedora-5-i386-core/result real 1m13.208s user 1m0.336s sys 0m11.269s Full build, autocache, rebuild cache (gzip): [michael_e_brown at mdomsch-pe1425-a ~]$ time mock -r fc5-i386 dell-repository-11-0.fc5.src.rpm --autocache init clean prep This may take a while create cache setup build ending done Results and/or logs in: /var/lib/mock/fedora-5-i386-core/result real 2m25.889s user 1m35.166s sys 0m20.537s Full build, autocache, using cache (gzip): [michael_e_brown at mdomsch-pe1425-a ~]$ time mock -r fc5-i386 dell-repository-11-0.fc5.src.rpm --autocache init clean prep This may take a while unpack cache setup build ending done Results and/or logs in: /var/lib/mock/fedora-5-i386-core/result real 0m20.854s user 0m8.289s sys 0m10.001s Full build, autocache, rebuild cache (no compression): [michael_e_brown at mdomsch-pe1425-a ~]$ time mock -r fc5-i386 dell-repository-11-0.fc5.src.rpm --autocache init clean prep This may take a while create cache setup build ending done Results and/or logs in: /var/lib/mock/fedora-5-i386-core/result real 1m47.684s user 0m46.639s sys 0m21.197s Full build, autocache, using cache (no compression): [michael_e_brown at mdomsch-pe1425-a ~]$ time mock -r fc5-i386 dell-repository-11-0.fc5.src.rpm --autocache init clean prep This may take a while unpack cache setup build ending done Results and/or logs in: /var/lib/mock/fedora-5-i386-core/result real 0m24.060s user 0m1.484s sys 0m9.169s From bugs.michael at gmx.net Wed May 17 09:59:59 2006 From: bugs.michael at gmx.net (Michael Schwendt) Date: Wed, 17 May 2006 11:59:59 +0200 Subject: [PATCH] plague server Repo.py Message-ID: <20060517115959.897ec37a.bugs.michael@gmx.net> The file glob for debuginfo packages should be "*-debuginfo-*" since the debuginfo rpms are called %{name}-debuginfo-%{version}-%{release}.%{arch}.rpm and not: *.debuginfo.rpm -------------- next part -------------- A non-text attachment was scrubbed... Name: Repo.py.patch Type: application/octet-stream Size: 625 bytes Desc: not available URL: From bugs.michael at gmx.net Wed May 17 10:00:56 2006 From: bugs.michael at gmx.net (Michael Schwendt) Date: Wed, 17 May 2006 12:00:56 +0200 Subject: Adding a lock file to plague server Repo.py _update_repo() Message-ID: <20060517120056.fa573f73.bugs.michael@gmx.net> Currently, when we sign and push built packages, we move them out of a "repodir" while the build server may be copying new packages to that directory. We have not cleaned up repodir for a long time either, because it could not be done automatically without risk. The corresponding section in the push script contains a "todo" comment only. [Recently I've used a separate script, however, which cleans up the directory after a push, provided that the build system is idle and no package is about to be copied into the directory at the same time.] I've looked at Repo.py _update_repo() where I see nothing which prevents concurrent access to this repodir. Is there anything I'm missing when thinking that it is possible to add a simple lock file at the root of repodir in order to control access to this directory? The build master would lock the file when copying new packages to it. And the push script would lock the file when moving packages and when cleaning up. Both would sleep when they cannot acquire the lock. From dcbw at redhat.com Wed May 17 14:00:42 2006 From: dcbw at redhat.com (Dan Williams) Date: Wed, 17 May 2006 10:00:42 -0400 Subject: [PATCH] plague server Repo.py In-Reply-To: <20060517115959.897ec37a.bugs.michael@gmx.net> References: <20060517115959.897ec37a.bugs.michael@gmx.net> Message-ID: <1147874442.2179.2.camel@localhost.localdomain> On Wed, 2006-05-17 at 11:59 +0200, Michael Schwendt wrote: > The file glob for debuginfo packages should be "*-debuginfo-*" since > the debuginfo rpms are called > > %{name}-debuginfo-%{version}-%{release}.%{arch}.rpm > > and not: > > *.debuginfo.rpm Thanks, committed to stable and HEAD. Dan From Michael_E_Brown at dell.com Wed May 17 14:39:52 2006 From: Michael_E_Brown at dell.com (Michael E Brown) Date: Wed, 17 May 2006 09:39:52 -0500 Subject: rfc -- multibuild script Message-ID: <1147876792.4046.74.camel@soltek.michaels-house.net> Hi all. skvidal asked me to send a copy of my multibuild script to the list. Here it is, in all its customized for my environment glory. If there is interest, I could clean it up a bit more and add more cmdline options instead of hardcoding some of the options. The main advantage of this script is: A) speed: runs a configurable number of builds in parallel. B) auto repository creation: creates yum repo as output. C) multi config build: builds a set of rpms for multiple configs Note that this script assumes mock config files named like this: fcX-ARCH.cfg rhelX-ARCH.cfg slesX-ARCH.cfg That is because it outputs a yum repository structured like this: /repo/fcX/{RPMS,SRPMS,repodata} /repo/rhelX/{RPMS,SRPMS,repodata} /repo/slesX/{RPMS,SRPMS,repodata} Usage: multibuild.sh file1.src.rpm file2.src.rpm ... Options: p: http_proxy c: config list (override builtin) e: mock uniqueext d debug o: output directory (override automatically-deleted tempdir) For now, I have a few variables hardcoded in the script. These could probably be made cli options. See the CHANGEME entries. For testing, I would recommend using the "-o /output/dir" option so you can examine the output directory structure. -- Michael -------------- next part -------------- A non-text attachment was scrubbed... Name: multibuild.sh Type: application/x-shellscript Size: 4008 bytes Desc: not available URL: From williams at redhat.com Wed May 17 15:13:53 2006 From: williams at redhat.com (Clark Williams) Date: Wed, 17 May 2006 10:13:53 -0500 Subject: [PATCH] Fix PS1 prompt for shell In-Reply-To: <35C9A9D68AB3FA4AB63692802656D9EC010F43A8@ausx3mps303.aus.amer.dell.com> References: <35C9A9D68AB3FA4AB63692802656D9EC010F43A8@ausx3mps303.aus.amer.dell.com> Message-ID: <446B3DB1.3050001@redhat.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Michael_E_Brown at Dell.com wrote: > [PATCH] Fix PS1 prompt for shell > > Not quite a brown-paper-bag bug, but I broke the PS1 prompt for the > shell command. Not all that earth shattering, but the following > trivial patch fixes it. > > It was due to using an old mock-helper which didn't have the PS1 in > ALLOWED_ENVS and the fact that I didn't notice that os.system() > doesn't pass along changes to os.environ. > > - > Yeah, I was surprised that os.system didn't pass os.environ too (which is why I put PS1 on the command line :). committed. Clark -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFEaz2xHyuj/+TTEp0RAhEoAJ0ahgl6H/HCO/kKpPTQO2hq9m81owCfTigF zAFGnbgMjSj3+jPRsH5i3SE= =wI+z -----END PGP SIGNATURE----- From tibbs at math.uh.edu Wed May 17 16:05:54 2006 From: tibbs at math.uh.edu (Jason L Tibbitts III) Date: Wed, 17 May 2006 11:05:54 -0500 Subject: [PATCH] autocache patch for mock -- try 2 In-Reply-To: <1147837873.4046.50.camel@soltek.michaels-house.net> (Michael E. Brown's message of "Tue, 16 May 2006 22:51:12 -0500") References: <35C9A9D68AB3FA4AB63692802656D9EC010F43AF@ausx3mps303.aus.amer.dell.com> <1147835927.4046.42.camel@soltek.michaels-house.net> <1147837873.4046.50.camel@soltek.michaels-house.net> Message-ID: With this and the "gzip by default" patch my build times are down by another 40 seconds (to 1:30, repeatably). Time to build the cache is down by 2:32 to 3:20. I'll certainly be running this from now on. - J< From skvidal at linux.duke.edu Thu May 18 04:57:46 2006 From: skvidal at linux.duke.edu (seth vidal) Date: Thu, 18 May 2006 00:57:46 -0400 Subject: Multiarch support for mock In-Reply-To: <20060516220343.GA5707@dudweiler.stuttgart.redhat.com> References: <20060516093354.GA3956@dudweiler.stuttgart.redhat.com> <1147790919.4737.8.camel@cutter> <20060516220343.GA5707@dudweiler.stuttgart.redhat.com> Message-ID: <1147928267.14273.14.camel@cutter> On Wed, 2006-05-17 at 00:03 +0200, Florian La Roche wrote: > > So you're saying that in order to build 64bit packages we REQUIRE > > glibc*.i686? > > > > Are you serious? So there's no way to build x86_64 w/o i386 first being > > built? > > Hello Seth, > > No, just that glibc is an exception to the otherwise "only install 64bit > rpm packages" rule for a buildroot. How is that not what I just asked? If you cannot completely build an x86_64 pkg w/o the presence of an i386 pkg then the x86_64 pkg DEPENDS on their being i386 pkg already existing and installed. -sv From Michael_E_Brown at Dell.com Fri May 19 19:45:14 2006 From: Michael_E_Brown at Dell.com (Michael_E_Brown at Dell.com) Date: Fri, 19 May 2006 14:45:14 -0500 Subject: [PATCH] Multiuser environment permissions Message-ID: <35C9A9D68AB3FA4AB63692802656D9EC010F43CC@ausx3mps303.aus.amer.dell.com> I have found one difficulty with the autocache patch, not a showstopper, but annoying. The problem occurs when you have multiple users on the same machine. If one user creates the cache, several files inside will be owned by that user. The other user then cannot access /builddir/ due to permissions. The attached patch fixes this situation. Basically, it is very localized to the _make_our_user() function. It checks to see if the user already exists with a different UID from the current UID. If it detects this, it deletes and recreates the user and chown's the appropriate files. Feedback appreciated. If this is ok, please apply with the autocache patches. Other than this issue, we have been using the autocache patch here for a few days with no other issues. -- Michael <> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: mock-permfix.patch Type: application/octet-stream Size: 2265 bytes Desc: mock-permfix.patch URL: From Michael_E_Brown at Dell.com Fri May 19 19:52:15 2006 From: Michael_E_Brown at Dell.com (Michael_E_Brown at Dell.com) Date: Fri, 19 May 2006 14:52:15 -0500 Subject: [PATCH] Multiuser environment permissions Message-ID: <35C9A9D68AB3FA4AB63692802656D9EC010F43CD@ausx3mps303.aus.amer.dell.com> Ok, further testing indicates that I need to check group as well. I'll send a corrected patch in a few mins. -- Michael ________________________________ From: fedora-buildsys-list-bounces at redhat.com [mailto:fedora-buildsys-list-bounces at redhat.com] On Behalf Of Michael_E_Brown at Dell.com Sent: Friday, May 19, 2006 2:45 PM To: fedora-buildsys-list at redhat.com Subject: [PATCH] Multiuser environment permissions I have found one difficulty with the autocache patch, not a showstopper, but annoying. The problem occurs when you have multiple users on the same machine. If one user creates the cache, several files inside will be owned by that user. The other user then cannot access /builddir/ due to permissions. The attached patch fixes this situation. Basically, it is very localized to the _make_our_user() function. It checks to see if the user already exists with a different UID from the current UID. If it detects this, it deletes and recreates the user and chown's the appropriate files. Feedback appreciated. If this is ok, please apply with the autocache patches. Other than this issue, we have been using the autocache patch here for a few days with no other issues. -- Michael <> -------------- next part -------------- An HTML attachment was scrubbed... URL: From Michael_E_Brown at Dell.com Fri May 19 20:19:38 2006 From: Michael_E_Brown at Dell.com (Michael_E_Brown at Dell.com) Date: Fri, 19 May 2006 15:19:38 -0500 Subject: [PATCH - updated] Multiuser environment permissions Message-ID: <35C9A9D68AB3FA4AB63692802656D9EC010F43CE@ausx3mps303.aus.amer.dell.com> Attached is the updated patch. There is a single line addition from the previous patch, in which I delete the build group as well. -- Michael ________________________________ From: fedora-buildsys-list-bounces at redhat.com [mailto:fedora-buildsys-list-bounces at redhat.com] On Behalf Of Michael_E_Brown at Dell.com Sent: Friday, May 19, 2006 2:52 PM To: fedora-buildsys-list at redhat.com Subject: RE: [PATCH] Multiuser environment permissions Ok, further testing indicates that I need to check group as well. I'll send a corrected patch in a few mins. -- Michael ________________________________ From: fedora-buildsys-list-bounces at redhat.com [mailto:fedora-buildsys-list-bounces at redhat.com] On Behalf Of Michael_E_Brown at Dell.com Sent: Friday, May 19, 2006 2:45 PM To: fedora-buildsys-list at redhat.com Subject: [PATCH] Multiuser environment permissions I have found one difficulty with the autocache patch, not a showstopper, but annoying. The problem occurs when you have multiple users on the same machine. If one user creates the cache, several files inside will be owned by that user. The other user then cannot access /builddir/ due to permissions. The attached patch fixes this situation. Basically, it is very localized to the _make_our_user() function. It checks to see if the user already exists with a different UID from the current UID. If it detects this, it deletes and recreates the user and chown's the appropriate files. Feedback appreciated. If this is ok, please apply with the autocache patches. Other than this issue, we have been using the autocache patch here for a few days with no other issues. -- Michael <> -------------- next part -------------- A non-text attachment was scrubbed... Name: mock-permfix2.patch Type: application/octet-stream Size: 2433 bytes Desc: mock-permfix2.patch URL: From Michael_E_Brown at Dell.com Fri May 19 20:52:18 2006 From: Michael_E_Brown at Dell.com (Michael_E_Brown at Dell.com) Date: Fri, 19 May 2006 15:52:18 -0500 Subject: [PATCH] - cli usability: rebuildcache implies autocache Message-ID: <35C9A9D68AB3FA4AB63692802656D9EC010F43D0@ausx3mps303.aus.amer.dell.com> Here is a two line patch to increase usability/user-friendliness of cmdline. Currently the --rebuildcache option doesn't do anything by itself, you have to specify both --autocache and --rebuildcache. With this patch, --rebuildcache will automatically turn on autocache. -- Michael <> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: mock-cli-userfriendly.patch Type: application/octet-stream Size: 543 bytes Desc: mock-cli-userfriendly.patch URL: From bugs.michael at gmx.net Sat May 20 14:50:19 2006 From: bugs.michael at gmx.net (Michael Schwendt) Date: Sat, 20 May 2006 16:50:19 +0200 Subject: [PATCH] Re: Adding a lock file to plague server Repo.py _update_repo() In-Reply-To: <20060517120056.fa573f73.bugs.michael@gmx.net> References: <20060517120056.fa573f73.bugs.michael@gmx.net> Message-ID: <20060520165019.33dee276.bugs.michael@gmx.net> On Wed, 17 May 2006 12:00:56 +0200, Michael Schwendt wrote: > Currently, when we sign and push built packages, we move them out of a > "repodir" while the build server may be copying new packages to that > directory. > > We have not cleaned up repodir for a long time either, because it could > not be done automatically without risk. The corresponding section in the > push script contains a "todo" comment only. [Recently I've used a separate > script, however, which cleans up the directory after a push, provided that > the build system is idle and no package is about to be copied into the > directory at the same time.] > > I've looked at Repo.py _update_repo() where I see nothing which prevents > concurrent access to this repodir. Is there anything I'm missing when > thinking that it is possible to add a simple lock file at the root of > repodir in order to control access to this directory? The build master > would lock the file when copying new packages to it. And the push script > would lock the file when moving packages and when cleaning up. Both would > sleep when they cannot acquire the lock. Attached patch explains in code what I have in mind. -------------- next part -------------- A non-text attachment was scrubbed... Name: Repo.py.patch Type: application/octet-stream Size: 1204 bytes Desc: not available URL: From Michael_E_Brown at dell.com Tue May 23 05:06:29 2006 From: Michael_E_Brown at dell.com (Michael E Brown) Date: Tue, 23 May 2006 00:06:29 -0500 Subject: [PATCH] autocache patche -- resend, updated Message-ID: <1148360789.23485.26.camel@soltek.michaels-house.net> Here is a resend/consolidation of the autocache patches from last week. I have combined the patches sent last week. It now defaults to gzip format. I have also made the following fixes: -- integrate the mock permfix patch. This checks if cache was created with UID != current user UID and adjusts permissions appropriately. -- Integrate a couple of small fixes regarding permissions. _prep_install() happens before the above fix, so I have changed two chown calls from config['chrootuser'] to use config['chrootuid'] (as well as corresponding group/gid). This ensures that cached yum.conf is always up-to-date. Previously, a cached yum.conf would never be updated until the cache expired and was rebuilt. This change means cached builds will never use wrong configuration. -- Use the -l option to tar when creating cache. This prevents tar from saving /proc or /dev/pts/ -- cmdline usability: --rebuildcache implies --autocache Feedback and comments welcome... Usage: Add --autocache to cmdline, or config_opts['use_cache'] = 1 to config file to enable caching. Cache is automatically built and saved to /var/lib/mock/root-cache/ on a per-config basis. Cache will be rebuilt by default every 15 days, or when --rebuildcache is specified on cmdline. This option is meant to be self-administering and low overhead. -- Michael -------------- next part -------------- A non-text attachment was scrubbed... Name: mock-autocache-v6.patch Type: text/x-patch Size: 10872 bytes Desc: not available URL: From Michael_E_Brown at dell.com Tue May 23 05:14:10 2006 From: Michael_E_Brown at dell.com (Michael E Brown) Date: Tue, 23 May 2006 00:14:10 -0500 Subject: [PATCH - RFC] config unify patch Message-ID: <1148361250.23485.32.camel@soltek.michaels-house.net> Here is a patch for discussion. It is similar to the patch sent last week, with the exception that I have added two condition checks that ensure the behaviour is the same with this patch as without. Net reduction in code size is 4 lines, but it also moves all the config file parsing to one loop. My belief is that either one of the two condition checks are not necessary. IMO, the condition check prior to the config parsing loop where it checks for file existence can be removed. We will then rely on the check after config parsing which tests for config_opts['root'] to ensure a proper config is given. Patch is for discussion. I can send an updated version after decisions are made. -- Michael -------------- next part -------------- A non-text attachment was scrubbed... Name: mock-cfgunify.patch Type: text/x-patch Size: 2638 bytes Desc: not available URL: From skvidal at linux.duke.edu Wed May 24 15:21:03 2006 From: skvidal at linux.duke.edu (seth vidal) Date: Wed, 24 May 2006 11:21:03 -0400 Subject: [PATCH] autocache patche -- resend, updated In-Reply-To: <1148360789.23485.26.camel@soltek.michaels-house.net> References: <1148360789.23485.26.camel@soltek.michaels-house.net> Message-ID: <1148484063.2940.6.camel@cutter> On Tue, 2006-05-23 at 00:06 -0500, Michael E Brown wrote: > Here is a resend/consolidation of the autocache patches from last week. > I have combined the patches sent last week. It now defaults to gzip > format. I have also made the following fixes: > -- integrate the mock permfix patch. This checks if cache was created > with UID != current user UID and adjusts permissions appropriately. > -- Integrate a couple of small fixes regarding permissions. > _prep_install() happens before the above fix, so I have changed two > chown calls from config['chrootuser'] to use config['chrootuid'] (as > well as corresponding group/gid). This ensures that cached yum.conf is > always up-to-date. Previously, a cached yum.conf would never be updated > until the cache expired and was rebuilt. This change means cached builds > will never use wrong configuration. > -- Use the -l option to tar when creating cache. This prevents tar from > saving /proc or /dev/pts/ > -- cmdline usability: --rebuildcache implies --autocache > > Feedback and comments welcome... > > Usage: > Add --autocache to cmdline, or config_opts['use_cache'] = 1 to config > file to enable caching. Cache is automatically built and saved > to /var/lib/mock/root-cache/ on a per-config basis. Cache will be > rebuilt by default every 15 days, or when --rebuildcache is specified on > cmdline. This option is meant to be self-administering and low > overhead. > merged. Thank you. -sv From mike at redhat.com Fri May 26 01:43:40 2006 From: mike at redhat.com (Mike McLean) Date: Thu, 25 May 2006 21:43:40 -0400 Subject: more mounts in mock Message-ID: <44765D4C.6050404@redhat.com> Attached are a couple of patches that expand the mounts created in the chroot by mock. These are mounts that we've used for builds within Red Hat for years and some packages need them to compile properly. more_mounts.patch is the larger patch, it refactors _mount() so that the mounts to be created are specified in a list and looped over. I've also changed to the unmounting code to make it more paranoid. In order to allow these mounts, I had to make some changes to mock-helper. bind_dev.patch builds on the the previous patch and provides an option to have /dev bind mounted in the chroot (instead of the skeletal /dev that mock sets up). The patch makes this an option with the original behavior as the default. -------------- next part -------------- A non-text attachment was scrubbed... Name: more_mounts.patch Type: text/x-patch Size: 11538 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: bind_dev.patch Type: text/x-patch Size: 4073 bytes Desc: not available URL: From mike at redhat.com Fri May 26 02:10:23 2006 From: mike at redhat.com (Mike McLean) Date: Thu, 25 May 2006 22:10:23 -0400 Subject: fast repo updates Message-ID: <4476638F.3010208@redhat.com> I posted this patch to rpm-metadata-list, but I figure some folks here might be interested as well. http://people.redhat.com/mikem/software/createrepo-update2.patch This patch allows createrepo to recycle existing metadata so that it only has to scan the rpms that have been changed or added. For situations where you might want to make small incremental changes to a yum repo (e.g. mock), this can provide significant time savings. From Michael_E_Brown at dell.com Fri May 26 05:03:16 2006 From: Michael_E_Brown at dell.com (Michael E Brown) Date: Fri, 26 May 2006 00:03:16 -0500 Subject: more mounts in mock In-Reply-To: <44765D4C.6050404@redhat.com> References: <44765D4C.6050404@redhat.com> Message-ID: <1148619796.1085.13.camel@soltek.michaels-house.net> Just wanted to run this idea past, while we are discussing mount points. I have noticed that mock mounts directories when they are not strictly necessary. It also mounts the directories before some operations where it is necessary, but does not unmount them afterwards. The result of this is that it is pretty easy to interrupt mock and leave behind mounted dirs. I believe that the only times that mount points are strictly needed are in the do_chroot() code path, as well as (possibly) during yum commands. Currently, there are many calls to _mount() scattered throughout the code. What I would propose is to place _mount() and _umount() calls in do_chroot(), place _mount() and _umount() calls in yum(), and then remove these calls from the various other places where they are currently. The effect of this would be that it would be relatively easy to encapsulate this behaviour in a try/except block that would ensure that mountpoints are always unmounted on exit. -- Michael On Thu, 2006-05-25 at 21:43 -0400, Mike McLean wrote: > Attached are a couple of patches that expand the mounts created in the > chroot by mock. These are mounts that we've used for builds within Red > Hat for years and some packages need them to compile properly. > > more_mounts.patch is the larger patch, it refactors _mount() so that the > mounts to be created are specified in a list and looped over. I've also > changed to the unmounting code to make it more paranoid. In order to > allow these mounts, I had to make some changes to mock-helper. > > bind_dev.patch builds on the the previous patch and provides an option > to have /dev bind mounted in the chroot (instead of the skeletal /dev > that mock sets up). The patch makes this an option with the original > behavior as the default. > > -- > Fedora-buildsys-list mailing list > Fedora-buildsys-list at redhat.com > https://www.redhat.com/mailman/listinfo/fedora-buildsys-list From Michael_E_Brown at dell.com Fri May 26 05:42:42 2006 From: Michael_E_Brown at dell.com (Michael E Brown) Date: Fri, 26 May 2006 00:42:42 -0500 Subject: more mounts in mock In-Reply-To: <44765D4C.6050404@redhat.com> References: <44765D4C.6050404@redhat.com> Message-ID: <1148622163.1085.34.camel@soltek.michaels-house.net> Comments below... On Thu, 2006-05-25 at 21:43 -0400, Mike McLean wrote: > Attached are a couple of patches that expand the mounts created in > the > chroot by mock. These are mounts that we've used for builds within > Red > Hat for years and some packages need them to compile properly. > > more_mounts.patch is the larger patch, it refactors _mount() so that > the > mounts to be created are specified in a list and looped over. I've > also > changed to the unmounting code to make it more paranoid. In order to > allow these mounts, I had to make some changes to mock-helper. > > bind_dev.patch builds on the the previous patch and provides an > option > to have /dev bind mounted in the chroot (instead of the skeletal /dev > that mock sets up). The patch makes this an option with the original > behavior as the default. > > > > > > > > > differences > between files > attachment > (more_mounts.patch) > > ? more_mounts.patch > Index: mock.py > =================================================================== > RCS file: /cvs/fedora/mock/mock.py,v > retrieving revision 1.51 > diff -u -r1.51 mock.py > --- mock.py 24 May 2006 15:15:19 -0000 1.51 > +++ mock.py 26 May 2006 01:21:35 -0000 > @@ -177,11 +177,9 @@ > self.state("clean") > > self.root_log('Cleaning Root') > - if os.path.exists('%s/%s' % (self.rootdir, 'proc')): > - self._umount('proc') > - if os.path.exists('%s/%s' % (self.rootdir, 'dev/pts')): > - self._umount('dev/pts') > - > + self._umount_by_proc() > + self._umount_by_file() > + > if os.path.exists(self.basedir): > cmd = '%s -rfv %s' % (self.config['rm'], self.basedir) > (retval, output) = self.do(cmd) > @@ -381,6 +379,7 @@ > """unmount things and clean up a bit""" > self.root_log("Cleaning up...") > self.state("ending") > + self._umount_by_proc() > self._umount_by_file() > self._build_log.close() > self.state("done") > @@ -403,44 +402,49 @@ > > def _mount(self): > """mount proc and devpts into chroot""" > + > + #dev,path(relative),type,opts > + mounts = [('proc','proc','proc',None)] > + if os.path.exists('/selinux'): > + mounts.append(('/selinux','selinux','none','bind')) How will this affect builds of eg. fedora development pkgs on build systems that dont have selinux, or do not have it enabled? > + #these have to come after /dev > + mounts.extend([ > + ('pts','dev/pts','devpts', 'gid=5,mode=620'), > + ('shm','dev/shm','tmpfs',None),]) tmpfs filesystems can have args specifying max mem usage. Would that be useful? > + > + #read existing mounts > + current = self._find_mounts() > mf = os.path.join(self.statedir, 'mounted-locations') > track = open(mf, 'w+') > > - # make the procdir if we don't have it > - # mount up proc > - procdir = os.path.join(self.rootdir, 'proc') > - self._ensure_dir(procdir) > - > - self.debug("mounting proc in %s" % procdir) > - command = '%s -t proc proc %s/proc' % (self.config['mount'], > - self.rootdir) > - track.write('proc\n') > - (retval, output) = self.do(command) > - track.flush() > - > - if retval != 0: > - if output.find('already mounted') == -1: # probably won't > work in other LOCALES > - error("could not mount proc error was: %s" % output) > - > - # devpts > - # > - devptsdir = os.path.join(self.rootdir, 'dev/pts') > - self._ensure_dir(devptsdir) > - self.debug("mounting devpts in %s" % devptsdir) > - command = '%s -t devpts devpts %s' % (self.config['mount'], > devptsdir) > - track.write('dev/pts\n') > - (retval, output) = self.do(command) > - track.flush() > + # mount all of the items listed in mounts > + for dev,path,type,opts in mounts: > + if path in current: > + self.debug("Already mounted: %s" % path) > + continue > + mount_dir = os.path.normpath('/'.join([self.rootdir, > path])) > + self._ensure_dir(mount_dir) > + self.debug("mounting %s in %s" % (path, mount_dir)) > + command = [self.config['mount'], '-t', type] > + if opts is not None: > + command.extend(['-o', opts]) > + command.extend([dev, mount_dir]) > + > + track.write('%s\n' % path) > + (retval, output) = self.do(' '.join(command)) > + track.flush() > + if retval != 0: > + if output.find('already mounted') == -1: # probably > won't work in other LOCALES > + raise RootError, "could not mount %s error was: % > s" % (mount_dir, output) > track.close() > > - if retval != 0: > - if output.find('already mounted') == -1: # probably won't > work in other LOCALES > - raise RootError, "could not mount /dev/pts error was: > %s" % output > - > - > def _umount(self, path): > > item = '%s/%s' % (self.rootdir, path) > + if not os.path.exists(item): > + #skip > + self.debug("skipping umount: no such path: %s" % item) > + return superfluous comment. > command = '%s %s' % (self.config['umount'], item) > (retval, output) = self.do(command) > > @@ -448,7 +452,36 @@ > if output.find('not mounted') == -1: # this probably > won't work in other LOCALES > raise RootError, "could not umount %s error was: %s" > % (path, output) > > - > + def _find_mounts(self): > + #read existing mounts > + ret = [] > + rootdir = os.path.normpath(self.rootdir) > + fo = file('/proc/mounts','r') > + for line in fo.readlines(): > + path = line.split()[1] > + if not path.startswith(rootdir): > + continue > + path = path[len(rootdir):] > + if path[:1] == '/': > + path = path[1:] > + if path: > + ret.append(path) > + fo.close() > + #reverse sort so deeper mounts come first > + ret.sort() > + ret.reverse() > + self.debug("Current mounts: %r" % ret) > + return ret > + > + def _umount_by_proc(self): > + for path in self._find_mounts(): > + self.debug("ummounting /proc/mounts entry: %r" % path) > + self._umount(path) > + #check > + remain = self._find_mounts() > + if remain: > + raise RootError, "mounts remain: %r" % remain > + > def _umount_by_file(self): > > mf = os.path.join(self.statedir, 'mounted-locations') > @@ -459,11 +492,14 @@ > lines = track.readlines() > track.close() > > + lines.sort() > + lines.reverse() > for item in lines: > item = item.replace('\n','') > if len(item.strip()) < 1: > continue > > + self.debug("ummounting file entry: %r" % item) > self._umount(item) > > # poof, no more file > Index: src/config.h > =================================================================== > RCS file: /cvs/fedora/mock/src/config.h,v > retrieving revision 1.1.1.1 > diff -u -r1.1.1.1 config.h > --- src/config.h 16 May 2005 02:44:00 -0000 1.1.1.1 > +++ src/config.h 26 May 2006 01:21:35 -0000 > @@ -44,7 +44,7 @@ > #define PACKAGE "mock" > > /* Location of mock roots */ > -#define ROOTSDIR "/var/lib/mock" > +#define ROOTSDIR "/var/lib/mock/" > > /* Define to 1 if you have the ANSI C header files. */ > #define STDC_HEADERS 1 > Index: src/mock-helper.c > =================================================================== > RCS file: /cvs/fedora/mock/src/mock-helper.c,v > retrieving revision 1.9 > diff -u -r1.9 mock-helper.c > --- src/mock-helper.c 24 May 2006 15:15:20 -0000 1.9 > +++ src/mock-helper.c 26 May 2006 01:21:35 -0000 > @@ -32,6 +32,28 @@ > > #define ALLOWED_ENV_SIZE (sizeof (ALLOWED_ENV) / sizeof > (ALLOWED_ENV[0])) > > +/* allowed values to mount type */ > +static char const * const ALLOWED_MTYPE[] = > +{ > + "proc", > + "devpts", > + "tmpfs", > + "none", > +}; > + > +#define ALLOWED_MTYPE_SIZE (sizeof (ALLOWED_MTYPE) / sizeof > (ALLOWED_MTYPE[0])) > + > +/* allowed mount options */ > +static char const * const ALLOWED_MOPT[] = > +{ > + "bind", > + "gid=5", > + "mode=620", > +}; Why is gid hard-coded here? > + > +#define ALLOWED_MOPT_SIZE (sizeof (ALLOWED_MOPT) / sizeof > (ALLOWED_MOPT[0])) > +#define MOPT_BUFLEN 128 > + > /* > * helper functions > */ > @@ -58,6 +80,54 @@ > } > > /* > + * check mount type against allowed list > + */ > +void check_mount_type(const char *type) > +{ > + size_t i; > + for (i = 0; i < ALLOWED_MTYPE_SIZE; ++i) > + { > + char const *ptr = ALLOWED_MTYPE[i]; > + if (strcmp (type, ptr) == 0) { > + return; > + } > + } > + error ("mount type not allowed: %s", type); > +} > + > +/* > + * check mount options against allowed list > + */ > +void check_mount_opts(const char *optstring) > +{ > + size_t i,j; > + char *buf = strndup(optstring, MOPT_BUFLEN); > + char *opt; > + if (ALLOWED_MOPT_SIZE == 0) > + /* allowed option list empty */ > + error ("no mount options allowed"); > + /* iterate over comma-separated list */ > + opt = strtok(buf,","); > + while (opt) > + { > + int allowed = 0; > + /* iterate over allowed opts */ > + for (j = 0; j < ALLOWED_MOPT_SIZE; ++j) > + { > + char const *ptr = ALLOWED_MOPT[j]; > + if (strcmp (opt, ptr) == 0) { > + allowed = 1; > + break; > + } > + } > + if (!allowed) > + error ("mount option not allowed: %s", opt); > + opt = strtok(NULL,","); > + } > + free(buf); > +} > + > +/* > * perform checks on the given dir > * - is the given dir under the allowed hierarchy ? > * - is it an actual dir ? > @@ -204,40 +274,92 @@ > } > > /* > - * allow proc mounts: > - * mount -t proc proc (root)/proc > - * allow devpts mounts: > - * mount -t devpts devpts (root)/dev/pts > + * check against: ALLOWED_MTYPE, ALLOWED_MOPT > + * check that mountpoint lies under rootsdir > + * verify that needed params are present > */ > void > do_mount (int argc, char *argv[]) probably just me, but this function seems a bit long. > { > - /* see if we have enough arguments for it to be what we want, ie. 5 > */ > - if (argc < 5) > - error ("not enough arguments"); > - > - /* see if it's -t proc or -t devpts */ > - if ((strncmp ("-t", argv[2], 2) == 0) && > - (strncmp ("proc", argv[3], 4) == 0)) > + char *mopts = NULL; > + char *mtype = NULL; > + char *dev = NULL; > + char *dir = NULL; > + char *cmd[10]; > + int i; > + > + if (argc > 8) > + error("too many arguments"); > + > + /* scan args */ > + char opt = '\0'; > + int count = 0; > + /* note we skip the first two args, which should be ('mock-helper', > 'mount') */ > + for (i=2; i { > - /* see if we're mounting proc to somewhere in rootsdir */ > - if (strncmp (rootsdir, argv[5], strlen (rootsdir)) != 0) > - error ("proc: mount not allowed on %s", argv[5]); > + if (strcmp(argv[i], "-t") == 0) > + { > + if (++i >= argc) > + error("incomplete type specification"); > + mtype = argv[i]; why not check_mount_type() here? Saves conditional later. > + } > + else if (strcmp(argv[i], "-o") == 0) > + { > + if (++i >= argc) > + error("incomplete option specification"); > + mopts = argv[i]; why not check_mount_opts() here? saves conditional later. > + } > + else if (*argv[i] == '-') > + error ("option not allowed: %s", strndup(argv[i],10)); > + else > + { > + /* normal parameter */ > + if (count == 0) > + { > + dev = argv[i]; > + count++; > + } > + else if (count == 1) > + { > + dir = argv[i]; > + count++; > + } > + else > + error("Too many parameters"); > + } > } > - else if ((strncmp ("-t", argv[2], 2) == 0) && > - (strncmp ("devpts", argv[3], 6) == 0)) > + > + if (!dev) > + error("No device provided"); > + if (!dir) > + error("No dir given"); > + if (!mtype) > + error("No type given"); > + /* verify that type is allowed */ > + check_mount_type(mtype); > + if (mopts) > + /* verify mount options are allowed */ > + check_mount_opts(mopts); > + > + /* see if we're mounting to somewhere in rootsdir */ > + if (strncmp (rootsdir, dir, strlen(rootsdir)) != 0) > + error ("mount not allowed on %s", strndup(dir,80)); looks like a security problem here. You should use check_dir_allowed() here to prevent /../ and the like. > + > + i = 0; > + cmd[i++] = "/bin/mount"; > + cmd[i++] = "-t"; > + cmd[i++] = mtype; > + if (mopts) > { > - if (argc < 5) > - error ("devpts: not enough mount arguments"); > - /* see if we're mounting devpts to somewhere in rootsdir */ > - else if (strncmp (rootsdir, argv[5], strlen (rootsdir)) != 0) > - error ("devpts: mount not allowed on %s", argv[5]); > + cmd[i++] = "-o"; > + cmd[i++] = mopts; > } > - else > - error ("unallowed mount type"); > + cmd[i++] = dev; > + cmd[i++] = dir; > + cmd[i++] = NULL; > > /* all checks passed, execute */ > - do_command ("/bin/mount", &(argv[1]), 0); > + do_command ("/bin/mount", cmd, 0); > } > > /* clean out a chroot dir */ > > > > > > > > differences > between files > attachment > (bind_dev.patch) > > diff -u mock.py mock.py > --- mock.py 26 May 2006 01:12:29 -0000 > +++ mock.py 26 May 2006 01:09:15 -0000 > @@ -407,6 +409,8 @@ > mounts = [('proc','proc','proc',None)] > if os.path.exists('/selinux'): > mounts.append(('/selinux','selinux','none','bind')) > + if self.config['bind_dev']: > + mounts.append(('/dev','dev','none','bind')) > #these have to come after /dev > mounts.extend([ > ('pts','dev/pts','devpts', 'gid=5,mode=620'), > @@ -601,24 +605,9 @@ > break > > return rpmUtils.miscutils.unique(reqlist) > - > - def _prep_install(self): > - """prep chroot for installation""" > - # make chroot dir > - # make /dev, mount /proc > - # > - for item in [self.basedir, self.rootdir, self.statedir, > self.resultdir, > - os.path.join(self.rootdir, 'var/lib/rpm'), > - os.path.join(self.rootdir, 'var/log'), > - os.path.join(self.rootdir, 'dev'), > - os.path.join(self.rootdir, 'etc/rpm'), > - os.path.join(self.rootdir, 'tmp'), > - os.path.join(self.rootdir, 'var/tmp'), > - os.path.join(self.rootdir, 'etc/yum.repos.d')]: > - self._ensure_dir(item) > - > - self._mount() > > + def _prep_dev(self): > + """prep basic /dev contents in the chroot""" > # we need stuff > devices = [('null', 'c', '1', '3', '666'), > ('urandom', 'c', '1', '9', '644'), > @@ -641,7 +630,7 @@ > devpath = os.path.join(self.rootdir, 'dev/fd') > if not os.path.exists(devpath): > os.symlink('../proc/self/fd', devpath) > - > + > fd = 0 > for item in ('stdin', 'stdout', 'stderr'): > devpath = os.path.join(self.rootdir, 'dev', item) > @@ -650,6 +639,27 @@ > os.symlink(fdpath, devpath) > fd += 1 > > + def _prep_install(self): > + """prep chroot for installation""" > + # make chroot dir > + # make /dev, mount /proc > + # > + for item in [self.basedir, self.rootdir, self.statedir, > self.resultdir, > + os.path.join(self.rootdir, 'var/lib/rpm'), > + os.path.join(self.rootdir, 'var/log'), > + os.path.join(self.rootdir, 'dev'), > + os.path.join(self.rootdir, 'etc/rpm'), > + os.path.join(self.rootdir, 'tmp'), > + os.path.join(self.rootdir, 'var/tmp'), > + os.path.join(self.rootdir, 'etc/yum.repos.d')]: > + self._ensure_dir(item) > + > + self._mount() per previous email, I believe that do_chroot is a better place for _mount(), but I suppose that could be a separate patch. > + > + if not self.config['bind_dev']: > + #if we don't bind mount dev, at least provide the basics > + self._prep_dev() > + > for item in [os.path.join(self.rootdir, 'etc', 'mtab'), > os.path.join(self.rootdir, 'etc', 'fstab'), > os.path.join(self.rootdir, 'var', 'log', > 'yum.log')]: > @@ -771,6 +781,8 @@ > parser.add_option("-r", action="store", type="string", > dest="chroot", > help="chroot name/config file name default: %default", > default='default') > + parser.add_option("--bind-dev", action ="store_true", > + help="bind mount /dev in the chroot") Why not have 'default=False' here? > parser.add_option("--no-clean", action ="store_false", > dest="clean", > help="do not clean chroot before building", default=True) > parser.add_option("--arch", action ="store", dest="arch", > @@ -813,6 +825,7 @@ > config_opts['debug'] = False > config_opts['quiet'] = False > config_opts['target_arch'] = 'i386' > + config_opts['bind_dev'] = False > config_opts['files'] = {} > config_opts['yum.conf'] = '' > config_opts['macros'] = """ > @@ -856,6 +869,9 @@ > if options.uniqueext: > config_opts['unique-ext'] = options.uniqueext > > + if options.bind_dev is not None: > + config_opts['bind_dev'] = options.bind_dev And then you can drop the conditional here. -- Michael From enrico.scholz at informatik.tu-chemnitz.de Fri May 26 10:24:32 2006 From: enrico.scholz at informatik.tu-chemnitz.de (Enrico Scholz) Date: Fri, 26 May 2006 12:24:32 +0200 Subject: more mounts in mock In-Reply-To: <44765D4C.6050404@redhat.com> (Mike McLean's message of "Thu, 25 May 2006 21:43:40 -0400") References: <44765D4C.6050404@redhat.com> Message-ID: <87slmxozhb.fsf@kosh.bigo.ensc.de> mike at redhat.com (Mike McLean) writes: > Attached are a couple of patches that expand the mounts created in the > chroot by mock. These are mounts that we've used for builds within Red > Hat for years and some packages need them to compile properly. 1. 'mock' should be run in an own namespace; then you would not need to track the mounted filesystems 2. most of the mounts should be done directly with the mount(2) syscall; NFS filesystem are the only exception I am aware of 3. a secure way to mount the filesystems is | chroot(ROOTDIR); | mount(...); Current path-checks (e.g. for '/../') are completely useless because they will not protect against symlink attacks. > more_mounts.patch is the larger patch, it refactors _mount() so that > the mounts to be created are specified in a list and looped over. > I've also changed to the unmounting code to make it more paranoid. With namespaces, unmounting would not be needed... > In order to allow these mounts, I had to make some changes to > mock-helper. > > bind_dev.patch builds on the the previous patch and provides an option > to have /dev bind mounted in the chroot (instead of the skeletal /dev > that mock sets up). When packages require special devices to build these packages are broken... Making a full /dev available lowers security significantly in environments which remove CAP_MKNOD for the buildsys. Enrico -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 480 bytes Desc: not available URL: From enrico.scholz at informatik.tu-chemnitz.de Fri May 26 11:04:35 2006 From: enrico.scholz at informatik.tu-chemnitz.de (Enrico Scholz) Date: Fri, 26 May 2006 13:04:35 +0200 Subject: [PATCH] autocache patche -- resend, updated In-Reply-To: <1148360789.23485.26.camel@soltek.michaels-house.net> (Michael E. Brown's message of "Tue, 23 May 2006 00:06:29 -0500") References: <1148360789.23485.26.camel@soltek.michaels-house.net> Message-ID: <87odxloxmk.fsf@kosh.bigo.ensc.de> Michael_E_Brown at dell.com (Michael E Brown) writes: > Here is a resend/consolidation of the autocache patches from last week. > I have combined the patches sent last week. It now defaults to gzip > format. I have also made the following fixes: > + check_dir_allowed (rootsdir, argv[2]); When you do security checks, then please make them correctly. Everything else will give a wrong feeling about security: * a | check_dir(); | operate_in_dir(); // tar ... opens always a window for symlink attacks. Better do | chdirSafe( From rdieter at math.unl.edu Fri May 26 12:47:53 2006 From: rdieter at math.unl.edu (Rex Dieter) Date: Fri, 26 May 2006 07:47:53 -0500 Subject: fast repo updates References: <4476638F.3010208@redhat.com> Message-ID: Mike McLean wrote: > This patch allows createrepo to recycle existing metadata so that it > only has to scan the rpms that have been changed or added. How is this different than the existing --cachedir option? -- Rex From skvidal at linux.duke.edu Fri May 26 14:22:50 2006 From: skvidal at linux.duke.edu (seth vidal) Date: Fri, 26 May 2006 10:22:50 -0400 Subject: fast repo updates In-Reply-To: References: <4476638F.3010208@redhat.com> Message-ID: <1148653370.2742.0.camel@cutter> On Fri, 2006-05-26 at 07:47 -0500, Rex Dieter wrote: > Mike McLean wrote: > > > > This patch allows createrepo to recycle existing metadata so that it > > only has to scan the rpms that have been changed or added. > > How is this different than the existing --cachedir option? It looks like it reads in the old metadata, using its own parser to help chop them up. The only part I'm confused by is why it has its own parser to do this. -sv From Michael_E_Brown at dell.com Mon May 29 20:46:14 2006 From: Michael_E_Brown at dell.com (Michael E Brown) Date: Mon, 29 May 2006 15:46:14 -0500 Subject: [PATCH] autocache patche -- resend, updated In-Reply-To: <87odxloxmk.fsf@kosh.bigo.ensc.de> References: <1148360789.23485.26.camel@soltek.michaels-house.net> <87odxloxmk.fsf@kosh.bigo.ensc.de> Message-ID: <1148935575.1085.74.camel@soltek.michaels-house.net> On Fri, 2006-05-26 at 13:04 +0200, Enrico Scholz wrote: > Michael_E_Brown at dell.com (Michael E Brown) writes: > > > Here is a resend/consolidation of the autocache patches from last week. > > I have combined the patches sent last week. It now defaults to gzip > > format. I have also made the following fixes: > > > + check_dir_allowed (rootsdir, argv[2]); > > When you do security checks, then please make them correctly. Everything > else will give a wrong feeling about security: Enrico, I welcome the feedback, thanks much. > > * a > > | check_dir(); > | operate_in_dir(); // tar ... Dir always has to be under /var/lib/mock/. It is not possible for unprivileged users to create symlinks here. I suppose the symlink attack could be done by somebody in the mock group. 1) mock -r CFG some.src.rpm #to force creation of builddir with ownership by me. 2) cd /var/lib/mock/CFG/root/builddir/ 3) ln -s / blastroot 4) mock-helper unpack /var/lib/mock/CFG/root/builddir/blastroot my-bad.tar.gz I believe that the fix for this would be to add a check to ensure that the tarball is always sourced from /var/lib/mock/root-cache/ and that the perms are correct. Feedback? > > opens always a window for symlink attacks. Better do > > | chdirSafe( | operate_in_dir("."); // tar ... > > The security of 'tar' operations is another question; extraction can > be made secure by extracting into a private dir and doing an atomic > rename(2) then. ATM, I do not see a way how to implement tarball > creation securely. Make sure we are always tarring /var/lib/mock/CFG/root and always place the tar under /var/lib/mock/root-cache. Users in group 'mock' would not have sufficient perms to do anything harmful under this dir. > > * you should check permissions of the tar-file; else, user can provide > e.g. a public writable /dev/kmem device in the tarball > > Ok, the question remains whether such checks (inclusive check_dir_allowed()) > are needed overall. 'mock' gives practically root rights to everybody in the > mock. Personally, I think they should be there. It makes things much more difficult, though. Suggested resolution: For a patch, I could change both pack and unpack to only take the file name and config name. It would then always look for or create the tar under /var/lib/mock/root-cache/, and it would only ever tar or untar to /var/lib/mock/CFG/root. -- Michael From enrico.scholz at informatik.tu-chemnitz.de Mon May 29 22:40:05 2006 From: enrico.scholz at informatik.tu-chemnitz.de (Enrico Scholz) Date: Tue, 30 May 2006 00:40:05 +0200 Subject: [PATCH] autocache patche -- resend, updated In-Reply-To: <1148935575.1085.74.camel@soltek.michaels-house.net> (Michael E. Brown's message of "Mon, 29 May 2006 15:46:14 -0500") References: <1148360789.23485.26.camel@soltek.michaels-house.net> <87odxloxmk.fsf@kosh.bigo.ensc.de> <1148935575.1085.74.camel@soltek.michaels-house.net> Message-ID: <87k684sbei.fsf@kosh.bigo.ensc.de> Michael_E_Brown at dell.com (Michael E Brown) writes: >> * a >> >> | check_dir(); >> | operate_in_dir(); // tar ... > > Dir always has to be under /var/lib/mock/. It is not possible for > unprivileged users to create symlinks here. /var/lib/mock is writable by everybody in the 'mock' group. > I suppose the symlink attack could be done by somebody in the mock > group. > 1) mock -r CFG some.src.rpm #to force creation of builddir with > ownership by me. > 2) cd /var/lib/mock/CFG/root/builddir/ > 3) ln -s / blastroot > 4) mock-helper unpack /var/lib/mock/CFG/root/builddir/blastroot > my-bad.tar.gz > > I believe that the fix for this would be to add a check to ensure that > the tarball is always sourced from /var/lib/mock/root-cache/ and that > the perms are correct. Feedback? right thing would be, to open the tarball in the helper, check it and pipe it into tar's stdin. See http://ensc.de/mock/mock-0.4-cache.diff It protects against all symlink attacks until the 'tar' run. >> opens always a window for symlink attacks. Better do >> >> | chdirSafe(> | operate_in_dir("."); // tar ... >> >> The security of 'tar' operations is another question; extraction can >> be made secure by extracting into a private dir and doing an atomic >> rename(2) then. ATM, I do not see a way how to implement tarball >> creation securely. > > Make sure we are always tarring /var/lib/mock/CFG/root and always place > the tar under /var/lib/mock/root-cache. I mean the following attack | tar attacker | | check whether 'etc' is a dir | rm -rf etc | ln -s /etc etc | chdir('etc') | pack content I simple do not know, whether the 'chdir' in tar is done in a secure way (e.g. lstat(dir, &exp_st) && chdir(dir) && stat(".", &cur_st) && compare(exp_st, cur_st)) Enrico -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 480 bytes Desc: not available URL: