From Michael_E_Brown at dell.com Sat Sep 1 02:27:11 2007 From: Michael_E_Brown at dell.com (Michael E Brown) Date: Fri, 31 Aug 2007 21:27:11 -0500 Subject: mock fix for packages with glib2-devel dependency Message-ID: <20070901022711.GA2102@humbolt.us.dell.com> All, https://bugzilla.redhat.com/show_bug.cgi?id=273481 Builds in mock on x86_64 fail for packages with buildrequires on glib2-devel. Reported on EPEL 5. The suggested fix is to add glib2-devel/glib-devel to the exclude list. Question: is this something we want to apply for FC6/F7/et al? -- Michael PS: thanks to Jonathan Steffan for the report and suggested fix. From jkeating at redhat.com Sat Sep 1 11:00:44 2007 From: jkeating at redhat.com (Jesse Keating) Date: Sat, 1 Sep 2007 07:00:44 -0400 Subject: mock fix for packages with glib2-devel dependency In-Reply-To: <20070901022711.GA2102@humbolt.us.dell.com> References: <20070901022711.GA2102@humbolt.us.dell.com> Message-ID: <20070901070044.63a1574d@ender> On Fri, 31 Aug 2007 21:27:11 -0500 Michael E Brown wrote: > Builds in mock on x86_64 fail for packages with buildrequires on > glib2-devel. Reported on EPEL 5. > > The suggested fix is to add glib2-devel/glib-devel to the exclude > list. > > Question: is this something we want to apply for FC6/F7/et al? I didn't commit that change or put that change up for patch as it's incomplete. There are plenty of other packages in the multilib set that start with gl, and this will be an ever chasing rathole. Instead I asked Seth to help us come up with a plugin or syntax that would allow us to include rather than exclude. There was a long conversation on IRC about this weeks ago, and the result was a yum plugin that would do what we want. Unfortunately we don't have a good facility to use plugins with mock, and we really need it for other things as well (like repo priorities to lower bandwidth usage of koji.fp.o if static-repos are enabled). I think we need to work on yum plugin support in mock so that we can fix this properly without continually futzing with regexes to just get glibc and glibc-devel. -- Jesse Keating Fedora -- All my bits are free, are yours? -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From markmc at redhat.com Sat Sep 1 18:54:16 2007 From: markmc at redhat.com (Mark McLoughlin) Date: Sat, 01 Sep 2007 19:54:16 +0100 Subject: [patch 1/4] Remove Python 2.5 usage In-Reply-To: <20070831192341.1ce7a32d@ender> References: <20070831155221.672428000@redhat.com> <20070831155430.074450000@redhat.com> <20070831192341.1ce7a32d@ender> Message-ID: <1188672856.2706.7.camel@blaa> Hi Jesse, On Fri, 2007-08-31 at 19:23 -0400, Jesse Keating wrote: > Curious what you're attempting to do with pungi on a < python2.5 > platform... I'm certainly not thinking about backwards compat when > developing future versions of pungi to compose future versions of > Fedora... Basically, I'm just using gather to compose a number of yum repos together ... it seems to work fine on RHEL5 with this patch for older pykickstart: http://www.gnome.org/~markmc/code/pungi-old-pkickstart.patch I'm not suggesting this is something you should worry about ... I just sent the patch since it's not exactly an important feature for pungi to rely on. Cheers, Mark. From skvidal at fedoraproject.org Sun Sep 2 14:13:01 2007 From: skvidal at fedoraproject.org (seth vidal) Date: Sun, 02 Sep 2007 10:13:01 -0400 Subject: mock fix for packages with glib2-devel dependency In-Reply-To: <20070901070044.63a1574d@ender> References: <20070901022711.GA2102@humbolt.us.dell.com> <20070901070044.63a1574d@ender> Message-ID: <1188742381.29226.55.camel@cutter> On Sat, 2007-09-01 at 07:00 -0400, Jesse Keating wrote: > On Fri, 31 Aug 2007 21:27:11 -0500 > Michael E Brown wrote: > > > Builds in mock on x86_64 fail for packages with buildrequires on > > glib2-devel. Reported on EPEL 5. > > > > The suggested fix is to add glib2-devel/glib-devel to the exclude > > list. > > > > Question: is this something we want to apply for FC6/F7/et al? > > I didn't commit that change or put that change up for patch as it's > incomplete. There are plenty of other packages in the multilib set > that start with gl, and this will be an ever chasing rathole. > > Instead I asked Seth to help us come up with a plugin or syntax that > would allow us to include rather than exclude. There was a long > conversation on IRC about this weeks ago, and the result was a yum > plugin that would do what we want. Unfortunately we don't have a good > facility to use plugins with mock, and we really need it for other > things as well (like repo priorities to lower bandwidth usage of > koji.fp.o if static-repos are enabled). > > I think we need to work on yum plugin support in mock so that we can > fix this properly without continually futzing with regexes to just get > glibc and glibc-devel. Okay, so mock should be able to do this, I think, using the mock config file format. Since it generates a yum.conf and you can generate arbitrary files within it - we should be able to hack up something in the short term and do something better in the longer term. -sv From jkeating at redhat.com Sun Sep 2 14:26:19 2007 From: jkeating at redhat.com (Jesse Keating) Date: Sun, 2 Sep 2007 10:26:19 -0400 Subject: [patch 1/4] Remove Python 2.5 usage In-Reply-To: <1188672856.2706.7.camel@blaa> References: <20070831155221.672428000@redhat.com> <20070831155430.074450000@redhat.com> <20070831192341.1ce7a32d@ender> <1188672856.2706.7.camel@blaa> Message-ID: <20070902102619.3d61799f@ender> On Sat, 01 Sep 2007 19:54:16 +0100 Mark McLoughlin wrote: > Basically, I'm just using gather to compose a number of yum > repos together ... it seems to work fine on RHEL5 with this patch for > older pykickstart: > > http://www.gnome.org/~markmc/code/pungi-old-pkickstart.patch > > I'm not suggesting this is something you should worry > about ... I just sent the patch since it's not exactly an important > feature for pungi to rely on. Fair enough. I don't see how the provided patches would hurt my continued development so I don't have any qualms with applying them. I just wanted to be sure I wasn't providing an unreasonable expectation (: -- Jesse Keating Fedora -- All my bits are free, are yours? -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From markmc at redhat.com Mon Sep 3 09:54:32 2007 From: markmc at redhat.com (Mark McLoughlin) Date: Mon, 03 Sep 2007 10:54:32 +0100 Subject: [patch 4/4] Add a Config class In-Reply-To: <20070831155430.236623000@redhat.com>> References: <20070831155221.672428000@redhat.com> > <20070831155430.236623000@redhat.com>> Message-ID: <1188813272.16980.5.camel@blaa> Hi, Here's a version of this patch which applies against the latest upstream repo. Cheers, Mark. -------------- next part -------------- A non-text attachment was scrubbed... Name: pungi-add-config-object.patch Type: text/x-patch Size: 7967 bytes Desc: not available URL: From markmc at redhat.com Mon Sep 3 14:34:22 2007 From: markmc at redhat.com (Mark McLoughlin) Date: Mon, 03 Sep 2007 15:34:22 +0100 Subject: [patch 0/3] Support repo priorities Message-ID: <20070903142650.048726000@redhat.com>> Hi, If you want to compose a tree from multiple repositories and, for example, you want a repository's contents to take precendence over other packages, then you need some way to prioritize repos. e.g. if you "forked" Fedora, changed the kernel and wanted to build a tree consisting of Fedora and your kernel, even if the n-v-r of your kernel package is older than that of the latest Fedora kernel. The yum-priorities plugin supports this, and recently pykickstart has grown support for specifying the priority of a repo: repo --name=myrepo --baseurl=file:///my/repo --priority=0 repo --name=fedora --baseurl=file:///my/copy/of/fedora --priority=100 The following patches add support for priorities to pungi. Cheers, Mark. -- From markmc at redhat.com Mon Sep 3 14:42:23 2007 From: markmc at redhat.com (Mark McLoughlin) Date: Mon, 03 Sep 2007 15:42:23 +0100 Subject: [patch 1/3] Enable yum plugins References: <20070903144113.332700000@redhat.com>> Message-ID: <20070903144119.392483000@redhat.com>> An embedded and charset-unspecified text was scrubbed... Name: pungi-enable-plugins.patch URL: From markmc at redhat.com Mon Sep 3 14:42:43 2007 From: markmc at redhat.com (Mark McLoughlin) Date: Mon, 03 Sep 2007 15:42:43 +0100 Subject: [patch 2/3] Support repository priorities References: <20070903144113.332700000@redhat.com>> Message-ID: <20070903144119.429367000@redhat.com>> An embedded and charset-unspecified text was scrubbed... Name: pungi-support-kickstart-repo-priorities.patch URL: From markmc at redhat.com Mon Sep 3 14:43:03 2007 From: markmc at redhat.com (Mark McLoughlin) Date: Mon, 03 Sep 2007 15:43:03 +0100 Subject: [patch 3/3] Add a --prioritize option References: <20070903144113.332700000@redhat.com>> Message-ID: <20070903144119.489939000@redhat.com>> An embedded and charset-unspecified text was scrubbed... Name: pungi-prioritize-repos.patch URL: From sergio at sergiomb.no-ip.org Thu Sep 6 14:26:03 2007 From: sergio at sergiomb.no-ip.org (Sergio Monteiro Basto) Date: Thu, 06 Sep 2007 15:26:03 +0100 Subject: pungi -P doesn't work Re: No such file or directory: u'/var/tmp/yum-root-O3By9L/anaconda/headers/ifd-egate-0.05-17.i386.hdr' In-Reply-To: <20070825235502.A21647@xos037.xos.nl> References: <20070713210151.28d55e51@ender> <20070714221320.20dd85dc@ender> <1184600098.9130.4.camel@localhost.localdomain> <20070716114315.38444de5@localhost.localdomain> <20070716114847.1b67cddf@localhost.localdomain> <469BD849.3030309@kanarip.com> <20070825220959.A20769@xos037.xos.nl> <46D09BF0.1010306@kanarip.com> <20070825235502.A21647@xos037.xos.nl> Message-ID: <1189088763.2722.4.camel@localhost.localdomain> On Sat, 2007-08-25 at 23:55 +0200, Jos Vos wrote: > Hi Jeroen, > > On Sat, Aug 25, 2007 at 11:15:28PM +0200, Jeroen van Meeuwen wrote: > > > The key item here may be yum. We've seen similar issues before. The > > system's yum version doesn't seem to matter, but the one pulled into the > > tree does. 3.2.0 works, 3.2.1 and 3.2.2 don't. > > Correct, but I saw in this thread Jesse saying that the problem was > identified and you mentioned that the fix should be in anaconda's > development version. > > Anyway, I seem to have solved it at least for my RHEL environment > given Jesse's hint for creating a headers directory: in my pkgorder > I now added at the end of setup(): > > os.mkdir(os.path.join(cachedir, 'anaconda', 'headers'), 0755) > > which seems to solve the problem. > > I don't see anything fixing the problem in anaconda-11.3.0.19 (devel), > but maybe that's because in the meantime yum is fixed (or better: > behaving differently, as I don't know the yum API well enough to say > that yum was wrong). I want to stick to yum 3.2.1, as long as this > the proposed version for RHEL 5.1 Just confirming that this patch on anaconda package solve the build problem. but trying installing this isos will be or not a problem anaconda don't have this patch ? Thanks in advance -- S?rgio M. B. -------------- next part -------------- A non-text attachment was scrubbed... Name: pkorder.diff Type: text/x-patch Size: 447 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 2192 bytes Desc: not available URL: From pochinet.org at gmail.com Fri Sep 7 07:35:17 2007 From: pochinet.org at gmail.com (pochi_ken) Date: Fri, 7 Sep 2007 16:35:17 +0900 Subject: pungi -P doesn't work Re: No such file or directory: u'/var/tmp/yum-root-O3By9L/anaconda/headers/ifd-egate-0.05-17.i386.hdr' In-Reply-To: <46D0A78F.8010408@kanarip.com> References: <20070713210151.28d55e51@ender> <1184600098.9130.4.camel@localhost.localdomain> <20070716114315.38444de5@localhost.localdomain> <20070716114847.1b67cddf@localhost.localdomain> <469BD849.3030309@kanarip.com> <20070825220959.A20769@xos037.xos.nl> <46D09BF0.1010306@kanarip.com> <20070825235502.A21647@xos037.xos.nl> <46D0A78F.8010408@kanarip.com> Message-ID: <7f6dd19f0709070035h1467a15en8dc93b68e0dc48f2@mail.gmail.com> 2007/8/26, Jeroen van Meeuwen : > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Jos Vos wrote: > > Hi Jeroen, > > > > On Sat, Aug 25, 2007 at 11:15:28PM +0200, Jeroen van Meeuwen wrote: > > > >> The key item here may be yum. We've seen similar issues before. The > >> system's yum version doesn't seem to matter, but the one pulled into the > >> tree does. 3.2.0 works, 3.2.1 and 3.2.2 don't. > > > > Correct, but I saw in this thread Jesse saying that the problem was > > identified and you mentioned that the fix should be in anaconda's > > development version. > > > > Anyway, I seem to have solved it at least for my RHEL environment > > given Jesse's hint for creating a headers directory: in my pkgorder > > I now added at the end of setup(): > > > > os.mkdir(os.path.join(cachedir, 'anaconda', 'headers'), 0755) > > > > which seems to solve the problem. > > > > I don't see anything fixing the problem in anaconda-11.3.0.19 (devel), > > but maybe that's because in the meantime yum is fixed (or better: > > behaving differently, as I don't know the yum API well enough to say > > that yum was wrong). I want to stick to yum 3.2.1, as long as this > > the proposed version for RHEL 5.1 > > > > Cheers, > > > > The problem was with yum not creating the headers/ directory anymore in > yum-3.2.1, and although Seth tried to fix that in yum-3.2.2, it didn't > do the trick. But yeah, if you create the directory at some point > manually, it should fix stuff. > > - -- > Kind regards, > > Jeroen van Meeuwen > - -kanarip > I made yum-3.2.4-2 based fix file. yum-3.2.1/3.2.2/3.2.4 have same problem. New ISOs made that include New RPMS files at pungi or revisor, but during installation process exception error occured. Apply patch to yum/yumRepo.py --- yum/yumRepo.py 2007-08-29 02:38:39.000000000 +0900 +++ yum/yumRepo.py~ 2007-09-07 12:54:14.000000000 +0900 @@ -455,7 +455,7 @@ cookie = self.cachedir + '/' + self.metadata_cookie_fn self.setAttribute('metadata_cookie', cookie) - for dir in [self.cachedir, self.pkgdir]: + for dir in [self.cachedir, self.hdrdir, self.pkgdir]: if self.cache == 0: if os.path.exists(dir) and os.path.isdir(dir): continue please apply and re-make yum packages (ex: yum-3.2.4-2 -> yum-3.2.4-3) And new ISOs made at pungi or revisor. You will successfully install at new ISOs. best regard. From kanarip at kanarip.com Fri Sep 7 08:29:28 2007 From: kanarip at kanarip.com (Jeroen van Meeuwen) Date: Fri, 07 Sep 2007 10:29:28 +0200 Subject: pungi -P doesn't work Re: No such file or directory: u'/var/tmp/yum-root-O3By9L/anaconda/headers/ifd-egate-0.05-17.i386.hdr' In-Reply-To: <7f6dd19f0709070035h1467a15en8dc93b68e0dc48f2@mail.gmail.com> References: <20070713210151.28d55e51@ender> <1184600098.9130.4.camel@localhost.localdomain> <20070716114315.38444de5@localhost.localdomain> <20070716114847.1b67cddf@localhost.localdomain> <469BD849.3030309@kanarip.com> <20070825220959.A20769@xos037.xos.nl> <46D09BF0.1010306@kanarip.com> <20070825235502.A21647@xos037.xos.nl> <46D0A78F.8010408@kanarip.com> <7f6dd19f0709070035h1467a15en8dc93b68e0dc48f2@mail.gmail.com> Message-ID: <46E10BE8.1080502@kanarip.com> pochi_ken wrote: > please apply and re-make yum packages (ex: yum-3.2.4-2 -> yum-3.2.4-3) > And new ISOs made at pungi or revisor. > You will successfully install at new ISOs. > > best regard. > You cannot just go an spin ISOs even if this yum bug gets solved in F7 properly. Another "fix" in RPM though causes anaconda to go flat on it's face so you cannot compose images without also excluding the rpm* updated packages. RPM 4.4.2.1-1.fc7 does not allow packages to be added to the transaction more then once; it throws an exception and although that exception may be caught and handled appropriately, anaconda-runtime's pkgorder doesn't do so (but does add packages to the transaction multiple times). This (for F7) cannot be fixed, as anaconda and anaconda-runtime will not get updated in between releases. Revisor has a way to work around this, but I'm not amused at all we had to find such work around in the first place. Apparently, the packages which anaconda depends upon can just break that compatibility (one way or the other) and get away with it, posing anyone of us for a number of challenges. My complete list of excluded updates now is: exclude=yum yum-updatesd rpm-build rpm-libs rpm rpm-devel rpm-python -- Kind regards, Jeroen van Meeuwen -kanarip -- http://www.kanarip.com/ RHCE, LPIC-2, MCP, CCNA C6B0 7FB4 43E6 CDDA D258 F70B 28DE 9FDA 9342 BF08 From aberoham at gmail.com Mon Sep 10 17:03:18 2007 From: aberoham at gmail.com (aberoham at gmail.com) Date: Mon, 10 Sep 2007 10:03:18 -0700 Subject: koji initialization, docs Message-ID: <3bdb07840709101003i2c9fb465vd9604dc5033d47a5@mail.gmail.com> I have latest trunk version of koji, kojid, kojira, etc setup and running. But now what? Does any administration documentation exist? To figure out how to put gas in this engine do I really have to read the source? Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rob.myers at gtri.gatech.edu Tue Sep 11 13:31:33 2007 From: rob.myers at gtri.gatech.edu (rob myers) Date: Tue, 11 Sep 2007 09:31:33 -0400 Subject: koji initialization, docs In-Reply-To: <3bdb07840709101003i2c9fb465vd9604dc5033d47a5@mail.gmail.com> References: <3bdb07840709101003i2c9fb465vd9604dc5033d47a5@mail.gmail.com> Message-ID: <1189517493.22264.7.camel@rxm-581b.stl.gtri.gatech.edu> On Mon, 2007-09-10 at 10:03 -0700, aberoham at gmail.com wrote: > > I have latest trunk version of koji, kojid, kojira, etc setup and > running. But now what? Does any administration documentation exist? To > figure out how to put gas in this engine do I really have to read the > source? from what i remember, the first step is to import rpms for the distribution you are building against. koji assigns buildids in the order that the rpms are imported or built, so take care to import the oldest rpms first. assign the imported rpms a tag, and then create a build tag. if you take notes of what is required, it would probably help others to add them here: http://fedoraproject.org/wiki/Koji/ServerHowTo good luck! rob. From enrico.scholz at informatik.tu-chemnitz.de Tue Sep 11 14:36:31 2007 From: enrico.scholz at informatik.tu-chemnitz.de (Enrico Scholz) Date: Tue, 11 Sep 2007 16:36:31 +0200 Subject: koji initialization, docs In-Reply-To: <1189517493.22264.7.camel@rxm-581b.stl.gtri.gatech.edu> (rob myers's message of "Tue, 11 Sep 2007 09:31:33 -0400") References: <3bdb07840709101003i2c9fb465vd9604dc5033d47a5@mail.gmail.com> <1189517493.22264.7.camel@rxm-581b.stl.gtri.gatech.edu> Message-ID: <87wsuxmfps.fsf@fc5.bigo.ensc.de> rob myers writes: > from what i remember, the first step is to import rpms for the > distribution you are building against. Can be avoided by applying https://www.redhat.com/archives/fedora-buildsys-list/2007-May/msg00023.html and using an external repository. Enrico From dennis at ausil.us Tue Sep 11 14:38:38 2007 From: dennis at ausil.us (Dennis Gilmore) Date: Tue, 11 Sep 2007 09:38:38 -0500 Subject: koji initialization, docs In-Reply-To: <87wsuxmfps.fsf@fc5.bigo.ensc.de> References: <3bdb07840709101003i2c9fb465vd9604dc5033d47a5@mail.gmail.com> <1189517493.22264.7.camel@rxm-581b.stl.gtri.gatech.edu> <87wsuxmfps.fsf@fc5.bigo.ensc.de> Message-ID: <200709110938.39108.dennis@ausil.us> On Tuesday 11 September 2007 9:36:31 am Enrico Scholz wrote: > rob myers writes: > > from what i remember, the first step is to import rpms for the > > distribution you are building against. > > Can be avoided by applying > > > https://www.redhat.com/archives/fedora-buildsys-list/2007-May/msg00023.html > > and using an external repository. > > > Enrico Which breaks many things in koji. such as reproducability. koji keeps track of and holds all rpms that are used in buildroots. Dennis From enrico.scholz at informatik.tu-chemnitz.de Tue Sep 11 15:16:18 2007 From: enrico.scholz at informatik.tu-chemnitz.de (Enrico Scholz) Date: Tue, 11 Sep 2007 17:16:18 +0200 Subject: koji initialization, docs In-Reply-To: <200709110938.39108.dennis@ausil.us> (Dennis Gilmore's message of "Tue, 11 Sep 2007 09:38:38 -0500") References: <3bdb07840709101003i2c9fb465vd9604dc5033d47a5@mail.gmail.com> <1189517493.22264.7.camel@rxm-581b.stl.gtri.gatech.edu> <87wsuxmfps.fsf@fc5.bigo.ensc.de> <200709110938.39108.dennis@ausil.us> Message-ID: <87sl5lmdvh.fsf@fc5.bigo.ensc.de> Dennis Gilmore writes: >> https://www.redhat.com/archives/fedora-buildsys-list/2007-May/msg00023.html > > Which breaks many things in koji. many? I am only aware of reproducability and did 735 builds/10968 tasks with this patch. > such as reproducability. koji keeps track of and holds all rpms that > are used in buildroots. yes; this requires an enormous amount of space, backup capacity, bandwidth and effort to keep repository up-to-date. This is far too much overhead for my use case... Enrico From ob.system at gmail.com Thu Sep 13 22:49:04 2007 From: ob.system at gmail.com (Oscar Victorio Calixto Bacho) Date: Thu, 13 Sep 2007 17:49:04 -0500 Subject: Koji Message-ID: <6a28481b0709131549m73e2b87bhe7b4bc0dd6c88394@mail.gmail.com> Hi list koji is a great software, but which are case use for koji? in the college, we are trying to create a distribution based on Fedora. Att Oscar Bacho Instituto Tecnologico de Acapulco -------------- next part -------------- An HTML attachment was scrubbed... URL: From jkeating at redhat.com Fri Sep 14 00:36:38 2007 From: jkeating at redhat.com (Jesse Keating) Date: Thu, 13 Sep 2007 20:36:38 -0400 Subject: Koji In-Reply-To: <6a28481b0709131549m73e2b87bhe7b4bc0dd6c88394@mail.gmail.com> References: <6a28481b0709131549m73e2b87bhe7b4bc0dd6c88394@mail.gmail.com> Message-ID: <20070913203638.149b3e5b@lumos.localdomain> On Thu, 13 Sep 2007 17:49:04 -0500 "Oscar Victorio Calixto Bacho" wrote: > which are case use for koji? Koji is a system to manage rpm builds via mock on a farm of builders. It does a lot to manage this, but at the end of the day that is what it is for. > > in the college, we are trying to create a distribution based on > Fedora. Koji may be a bit overkill if you're only adding a few packages. Perhaps you're looking for the tools needed to actually take those packages and turn them into an installable set? -- Jesse Keating Fedora -- All my bits are free, are yours? -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From ob.system at gmail.com Fri Sep 14 17:32:16 2007 From: ob.system at gmail.com (Oscar Victorio Calixto Bacho) Date: Fri, 14 Sep 2007 12:32:16 -0500 Subject: Koji In-Reply-To: <20070913203638.149b3e5b@lumos.localdomain> References: <6a28481b0709131549m73e2b87bhe7b4bc0dd6c88394@mail.gmail.com> <20070913203638.149b3e5b@lumos.localdomain> Message-ID: <6a28481b0709141032v73bf102fs1833073e60771541@mail.gmail.com> > which are case use for koji? > > Koji is a system to manage rpm builds via mock on a farm of builders. > It does a lot to manage this, but at the end of the day that is what it > is for. case use from development, list actors, etc., administrator manual. > > > in the college, we are trying to create a distribution based on > > Fedora. > > Koji may be a bit overkill if you're only adding a few packages. > Perhaps you're looking for the tools needed to actually take those > packages and turn them into an installable set? No, we want use koji, for our own distribucion, we have infrastructure so what we need is only information about implementation. thanks And contribute with fedora in translation, maintenance with packages. No being spin of fedora. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikem at redhat.com Fri Sep 14 19:42:52 2007 From: mikem at redhat.com (Mike McLean) Date: Fri, 14 Sep 2007 15:42:52 -0400 Subject: Koji In-Reply-To: <6a28481b0709141032v73bf102fs1833073e60771541@mail.gmail.com> References: <6a28481b0709131549m73e2b87bhe7b4bc0dd6c88394@mail.gmail.com> <20070913203638.149b3e5b@lumos.localdomain> <6a28481b0709141032v73bf102fs1833073e60771541@mail.gmail.com> Message-ID: <46EAE43C.2020909@redhat.com> Oscar Victorio Calixto Bacho wrote: > > which are case use for koji? > > Koji is a system to manage rpm builds via mock on a farm of builders. > It does a lot to manage this, but at the end of the day that is what it > is for. > > case use from development, list actors, etc., administrator manual. I feel like we're losing something in translation. Are you looking for documentation? We're a little light on administrative docs right now, but I plan to have something better in the next month or so. In the meantime, there is this page: http://fedoraproject.org/wiki/Koji/ServerHowTo From sergio at sergiomb.no-ip.org Sat Sep 15 17:50:08 2007 From: sergio at sergiomb.no-ip.org (Sergio Monteiro Basto) Date: Sat, 15 Sep 2007 18:50:08 +0100 Subject: pungi -P doesn't work Re: No such file or directory: u'/var/tmp/yum-root-O3By9L/anaconda/headers/ifd-egate-0.05-17.i386.hdr' In-Reply-To: <7f6dd19f0709070035h1467a15en8dc93b68e0dc48f2@mail.gmail.com> References: <20070713210151.28d55e51@ender> <1184600098.9130.4.camel@localhost.localdomain> <20070716114315.38444de5@localhost.localdomain> <20070716114847.1b67cddf@localhost.localdomain> <469BD849.3030309@kanarip.com> <20070825220959.A20769@xos037.xos.nl> <46D09BF0.1010306@kanarip.com> <20070825235502.A21647@xos037.xos.nl> <46D0A78F.8010408@kanarip.com> <7f6dd19f0709070035h1467a15en8dc93b68e0dc48f2@mail.gmail.com> Message-ID: <1189878608.18104.3.camel@localhost.localdomain> On Fri, 2007-09-07 at 16:35 +0900, pochi_ken wrote: > > I made yum-3.2.4-2 based fix file. > yum-3.2.1/3.2.2/3.2.4 have same problem. > New ISOs made that include New RPMS files at pungi or revisor, but > during installation process exception error occured. > > Apply patch to yum/yumRepo.py > > --- yum/yumRepo.py 2007-08-29 02:38:39.000000000 +0900 > +++ yum/yumRepo.py~ 2007-09-07 12:54:14.000000000 +0900 > @@ -455,7 +455,7 @@ > cookie = self.cachedir + '/' + self.metadata_cookie_fn > self.setAttribute('metadata_cookie', cookie) > > - for dir in [self.cachedir, self.pkgdir]: > + for dir in [self.cachedir, self.hdrdir, self.pkgdir]: > if self.cache == 0: > if os.path.exists(dir) and os.path.isdir(dir): > continue > > please apply and re-make yum packages (ex: yum-3.2.4-2 -> yum-3.2.4-3) > And new ISOs made at pungi or revisor. > You will successfully install at new ISOs. > > best regard. Hi, Nice trick :) thanks, may be this patch should be upstreamed into yum. Well I don't try the new ISOs yet... but looks the right way to do it. Thanks, -- S?rgio M. B. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 2192 bytes Desc: not available URL: From enrico.scholz at informatik.tu-chemnitz.de Sat Sep 15 19:31:58 2007 From: enrico.scholz at informatik.tu-chemnitz.de (Enrico Scholz) Date: Sat, 15 Sep 2007 21:31:58 +0200 Subject: [PATCH koji] Added support for building from git repositories Message-ID: <20070915193158.11662.35393.stgit@kosh.bigo.ensc.de> This patch adds support for building from git:// respositories. It is used like koji call build 'git://?#' The repository specified above must have the following layout: / |- common/ | `- .git/ |- ... |- | `- .git/ |- ... This means that every package is hold in an own git repository. There must be a 'common' repository which contains files required for building the srpm. There is no explicit support for branches; the patch just checks out the given tag and builds from it. Perhaps it can be enhanced to check whether branch-point is some child of the tag... TODO: the git and cvs handlers share some common, non trivial code. This should be generalized. TODO: use a separate user account for the 'git' and 'make srpm' operations; currently they are executed under the uid 'kojid' is running (which is 'root). Hence, a '%(/sbin/killall5)' in the spec file can bring down the box. Signed-off-by: Enrico Scholz --- builder/kojid | 130 +++++++++++++++++++++++++++++++++++++++++++- cli/koji | 9 ++- koji.spec | 2 - koji/__init__.py | 11 ++++ www/kojiweb/taskinfo.chtml | 7 ++ www/kojiweb/tasks.chtml | 1 6 files changed, 153 insertions(+), 7 deletions(-) diff --git a/builder/kojid b/builder/kojid index 5940954..bbb9774 100755 --- a/builder/kojid +++ b/builder/kojid @@ -1396,7 +1396,7 @@ class ChainBuildTask(BaseTaskHandler): subtasks = [] build_tasks = [] for src in build_level: - if src.startswith('cvs://'): + if src.startswith('cvs://') or src.startswith('git://'): task_id = session.host.subtask(method='build', arglist=[src, target, opts], parent=self.id) @@ -1437,8 +1437,10 @@ class BuildTask(BaseTaskHandler): raise koji.BuildError, "arch_override is only allowed for scratch builds" task_info = session.getTaskInfo(self.id) # only allow admins to perform non-scratch builds from srpm - if not src.startswith('cvs://') and not opts.get('scratch') \ - and not 'admin' in session.getUserPerms(task_info['owner']): + if not src.startswith('cvs://') and \ + not src.startswith('git://') and \ + not opts.get('scratch') and \ + not 'admin' in session.getUserPerms(task_info['owner']): raise koji.BuildError, "only admins may peform non-scratch builds from srpm" target_info = session.getBuildTarget(target) if not target_info: @@ -1496,6 +1498,8 @@ class BuildTask(BaseTaskHandler): if isinstance(src,str): if src.startswith('cvs://'): return self.getSRPMFromCVS(src) + if src.startswith('git://'): + return self.getSRPMFromGit(src) else: #assume this is a path under uploads return src @@ -1514,6 +1518,17 @@ class BuildTask(BaseTaskHandler): srpm = result['srpm'] return srpm + def getSRPMFromGit(self, url): + #TODO - allow different ways to get the srpm + task_id = session.host.subtask(method='buildSRPMFromGit', + arglist=[url], + label='srpm', + parent=self.id) + # wait for subtask to finish + result = self.wait(task_id)[task_id] + srpm = result['srpm'] + return srpm + def readSRPMHeader(self, srpm): #srpm arg should be a path relative to /work global options @@ -1767,6 +1782,115 @@ class TagBuildTask(BaseTaskHandler): exctype, value = sys.exc_info()[:2] session.host.tagNotification(False, tag_id, fromtag, build_id, user_id, ignore_success, "%s: %s" % (exctype, value)) raise e + + +class BuildSRPMFromGitTask(BaseTaskHandler): + + Methods = ['buildSRPMFromGit'] + _taskWeight = 0.75 + + def spec_sanity_checks(self, filename): + spec = open(filename).read() + for tag in ("Packager", "Distribution", "Vendor"): + if re.match("%s:" % tag, spec, re.M): + raise koji.BuildError, "%s is not allowed to be set in spec file" % tag + for tag in ("packager", "distribution", "vendor"): + if re.match("%%define\s+%s\s+" % tag, spec, re.M): + raise koji.BuildError, "%s is not allowed to be defined in spec file" % tag + + def handler(self,url): + if not url.startswith('git://'): + raise koji.BuildError("invalid git URL: %s" % url) + + # Hack it because it refuses to parse it properly otherwise + scheme, netloc, path, params, query, fragment = urlparse.urlparse('http'+url[3:]) + if not (netloc and path and fragment and query): + raise koji.BuildError("invalid git URL: %s" % url) + + # Steps: + # 1. GIT clone into tempdir + # 3. Run 'make srpm' + + gitdir = self.workdir + '/git' + koji.ensuredir(gitdir) + logfile = self.workdir + "/srpm.log" + uploadpath = self.getUploadDir() + sourcedir = '%s/%s' % (gitdir, query) + + cmd = ['git', 'clone', 'git://%s%s/%s' % (netloc, path, query), query] + if log_output(cmd[0], cmd, logfile, uploadpath, cwd=gitdir, logerror=1, append=1): + output = "(none)" + try: + output = open(logfile).read() + except IOError: + pass + raise koji.BuildError, "Error with clone 'git://%s%s/%s: %s" % (netloc, path, query, output) + + cmd = ['git', 'clone', 'git://%s%s/common' % (netloc, path)] + self.logger.debug("executing: %s" % cmd) + if log_output(cmd[0], cmd, logfile, uploadpath, cwd=gitdir, logerror=1, append=1): + output = "(none)" + try: + output = open(logfile).read() + except IOError: + pass + raise koji.BuildError, "Error with clone 'git://%s%s/common: %s" % (netloc, path, output) + + cmd = ['git', 'ls-files', '-t', '-s'] + log_output(cmd[0], cmd, logfile, uploadpath, cwd='%s/common' % gitdir, logerror=1, append=1) + + cmd = ['git', 'checkout', '-b', 'kojibuild', fragment] + self.logger.debug("executing: %s" % cmd) + if log_output(cmd[0], cmd, logfile, uploadpath, cwd=sourcedir, logerror=1, append=1): + output = "(none)" + try: + output = open(logfile).read() + except IOError: + pass + raise koji.BuildError, "Error with checking out tag '%s' from git://%s%s/%s: %s" % (fragment, netloc, path, query, output) + + cmd = ['git', 'ls-files', '-t', '-s'] + log_output(cmd[0], cmd, logfile, uploadpath, cwd=sourcedir, logerror=1, append=1) + + spec_files = glob.glob("%s/*.spec" % sourcedir) + if len(spec_files) == 0: + raise koji.BuildError("No spec file found") + elif len(spec_files) > 1: + raise koji.BuildError("Multiple spec files found: %s" % spec_files) + spec_file = spec_files[0] + + # Run spec file sanity checks. Any failures will throw a BuildError + self.spec_sanity_checks(spec_file) + + #build srpm + cmd = ['make', '-C', sourcedir, 'srpm', '_KOJI=1.2.2', '_KOJI_TAG=%s' % fragment] + self.logger.debug("executing: %s" % cmd) + if log_output(cmd[0], cmd, logfile, uploadpath, cwd=gitdir, logerror=1, append=1): + raise koji.BuildError, "Error building SRPM" + + srpms = glob.glob('%s/*.src.rpm' % sourcedir) + if len(srpms) == 0: + raise koji.BuildError, "No srpms found in %s" % sourcedir + elif len(srpms) > 1: + raise koji.BuildError, "Multiple srpms found in %s: %s" % (sourcedir, ", ".join(srpms)) + else: + srpm = srpms[0] + + # check srpm name + h = koji.get_rpm_header(srpm) + name = h[rpm.RPMTAG_NAME] + version = h[rpm.RPMTAG_VERSION] + release = h[rpm.RPMTAG_RELEASE] + srpm_name = "%(name)s-%(version)s-%(release)s.src.rpm" % locals() + if srpm_name != os.path.basename(srpm): + raise koji.BuildError, 'srpm name mismatch: %s != %s' % (srpm_name, os.path.basename(srpm)) + + #upload srpm and return + self.uploadFile(srpm) + return { + 'srpm' : "%s/%s" % (uploadpath, srpm_name), + 'log' : "%s/srpm.log" % uploadpath, + } class BuildSRPMFromCVSTask(BaseTaskHandler): diff --git a/cli/koji b/cli/koji index 718b6e5..4f05500 100755 --- a/cli/koji +++ b/cli/koji @@ -669,7 +669,7 @@ def handle_build(options, session, args): if build_opts.background: #relative to koji.PRIO_DEFAULT priority = 5 - if not source.startswith('cvs://'): + if not (source.startswith('cvs://') or source.startswith('git://')): # only allow admins to perform non-scratch builds from srpm if not opts['scratch'] and not session.hasPerm('admin'): parser.error(_("builds from srpm must use --scratch")) @@ -741,7 +741,7 @@ def handle_chain_build(options, session, args): if build_level: src_list.append(build_level) build_level = [] - elif src.startswith('cvs://'): + elif src.startswith('cvs://') or src.startswith('git://'): build_level.append(src) elif '/' not in src and len(src.split('-')) >= 3: # quick check that it looks like a N-V-R @@ -2432,8 +2432,13 @@ def _parseTaskParams(session, method, task_id): if method == 'buildFromCVS': lines.append("CVS URL: %s" % params[0]) lines.append("Build Target: %s" % params[1]) + elif method == 'buildFromGit': + lines.append("GIT URL: %s" % params[0]) + lines.append("Build Target: %s" % params[1]) elif method == 'buildSRPMFromCVS': lines.append("CVS URL: %s" % params[0]) + elif method == 'buildSRPMFromGit': + lines.append("GIT URL: %s" % params[0]) elif method == 'multiArchBuild': lines.append("SRPM: %s/work/%s" % (options.topdir, params[0])) lines.append("Build Target: %s" % params[1]) diff --git a/koji.spec b/koji.spec index f14bb6e..7d4d643 100644 --- a/koji.spec +++ b/koji.spec @@ -49,7 +49,7 @@ Requires(post): /sbin/service Requires(preun): /sbin/chkconfig Requires(preun): /sbin/service Requires(pre): /usr/sbin/useradd -Requires: cvs +Requires: cvs git-core make Requires: rpm-build Requires: redhat-rpm-config Requires: createrepo >= 0.4.10 diff --git a/koji/__init__.py b/koji/__init__.py index d808126..5789220 100644 --- a/koji/__init__.py +++ b/koji/__init__.py @@ -1517,6 +1517,11 @@ def taskLabel(taskInfo): if source.startswith('cvs://'): source = source[source.rfind('/') + 1:] source = source.replace('#', ':') + elif source.startswith('git://'): + source = source[source.rfind('?') + 1:] + tmp = source[:source.find('#')] + source = source[source.find('#') + 1:] + source = '%s:%s' % (tmp, source[source.rfind('/') + 1:]) else: source = os.path.basename(source) extra = '%s, %s' % (target, source) @@ -1526,6 +1531,12 @@ def taskLabel(taskInfo): url = url[url.rfind('/') + 1:] url = url.replace('#', ':') extra = url + elif method == 'buildSRPMFromGit': + if taskInfo.has_key('request'): + url = taskInfo['request'][0] + url = url[url.rfind('/') + 1:] + url = url.replace('#', ':') + extra = url elif method == 'buildArch': if taskInfo.has_key('request'): srpm, tagID, arch = taskInfo['request'][:3] diff --git a/www/kojiweb/taskinfo.chtml b/www/kojiweb/taskinfo.chtml index 76df4bc..dd5cf72 100644 --- a/www/kojiweb/taskinfo.chtml +++ b/www/kojiweb/taskinfo.chtml @@ -67,6 +67,11 @@ Build Target: $params[1] #elif $task.method == 'buildSRPMFromCVS' CVS URL: $params[0] + #elif $task.method == 'buildFromGit' + GIT URL: $params[0]
+ Build Target: $params[1] + #elif $task.method == 'buildSRPMFromGit' + GIT URL: $params[0] #elif $task.method == 'multiArchBuild' SRPM: $params[0]
Build Target: $params[1]
@@ -276,7 +281,7 @@ $cgi.escape($result.faultString.strip()) $filename
#end for #if $task.state not in ($koji.TASK_STATES.CLOSED, $koji.TASK_STATES.CANCELED, $koji.TASK_STATES.FAILED) and \ - $task.method in ('buildSRPMFromCVS', 'buildArch', 'createrepo') + $task.method in ('buildSRPMFromCVS', 'buildSRPMFromGit', 'buildArch', 'createrepo')
Watch logs #end if diff --git a/www/kojiweb/tasks.chtml b/www/kojiweb/tasks.chtml index 70de06c..32ca6ad 100644 --- a/www/kojiweb/tasks.chtml +++ b/www/kojiweb/tasks.chtml @@ -104,6 +104,7 @@ All + From enrico.scholz at informatik.tu-chemnitz.de Sat Sep 15 19:41:12 2007 From: enrico.scholz at informatik.tu-chemnitz.de (Enrico Scholz) Date: Sat, 15 Sep 2007 21:41:12 +0200 Subject: [PATCH koji] added koji-helper setuid program Message-ID: <20070915194112.12268.93927.stgit@kosh.bigo.ensc.de> This patch adds a 'koji-helper' setuid program which implements the following methods: * koji-helper rmrf removes everything under , inclusive . It does not cross filesystem borders * koji-helper rmtree removes everything under , but not itself. It does not cross filesystem borders Methods above are implemented to replace the python 'safe_rmtree()' method which was never safe, nor will work when 'kojid' is running as non-root. Signed-off-by: Enrico Scholz --- Makefile | 15 ++- builder/kojid | 53 +++-------- koji.spec | 3 - src/koji-helper.c | 260 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 286 insertions(+), 45 deletions(-) diff --git a/Makefile b/Makefile index cd88d4b..2a239fd 100644 --- a/Makefile +++ b/Makefile @@ -2,6 +2,8 @@ NAME=koji SPECFILE = $(firstword $(wildcard *.spec)) SUBDIRS = hub builder koji cli docs util www +sbindir = /usr/sbin + ifdef DIST DIST_DEFINES := --define "dist $(DIST)" endif @@ -52,11 +54,14 @@ ifndef TAG TAG=$(NAME)-$(VERSION)-$(RELEASE) endif -_default: - @echo "read the makefile" +all: src/koji-helper + +src/koji-helper: src/koji-helper.c + $(CC) $(CFLAGS) $< -o $@ clean: rm -f *.o *.so *.pyc *~ koji*.bz2 koji*.src.rpm + rm -f src/koji-helper rm -rf koji-$(VERSION) for d in $(SUBDIRS); do make -s -C $$d clean; done @@ -100,14 +105,16 @@ force-tag:: # @$(MAKE) tag TAG_OPTS="-F $(TAG_OPTS)" DESTDIR ?= / -install: +install: all @if [ "$(DESTDIR)" = "" ]; then \ echo " "; \ echo "ERROR: A destdir is required"; \ exit 1; \ fi - mkdir -p $(DESTDIR) + mkdir -p $(DESTDIR)$(sbindir) + + install -p -m0710 src/koji-helper $(DESTDIR)$(sbindir) for d in $(SUBDIRS); do make DESTDIR=`cd $(DESTDIR); pwd` \ -C $$d install; [ $$? = 0 ] || exit 1; done diff --git a/builder/kojid b/builder/kojid index 6e973fe..5940954 100755 --- a/builder/kojid +++ b/builder/kojid @@ -150,35 +150,17 @@ def log_output(path, args, outfile, uploadpath, cwd=None, logerror=0, append=0, outfd.close() return status[1] -def safe_rmtree(path, unmount=False, strict=True): +def safe_rmtree(path, strict=True, op='rmrf'): logger = logging.getLogger("koji.build") - #safe remove: with -xdev the find cmd will not cross filesystems - # (though it will cross bind mounts from the same filesystem) - if not os.path.exists(path): - logger.debug("No such path: %s" % path) - return - if unmount: - umount_all(path) - #first rm -f non-directories - logger.debug('Scrubbing files in %s' % path) - rv = os.system("find '%s' -xdev \\! -type d -print0 |xargs -0 rm -f" % path) - msg = 'file removal failed (code %r) for %s' % (rv,path) - if rv != 0: - logger.warn(msg) - if strict: - raise koji.GenericError, msg - else: - return rv - #them rmdir directories - #with -depth, we start at the bottom and work up - logger.debug('Scrubbing directories in %s' % path) - rv = os.system("find '%s' -xdev -depth -type d -print0 |xargs -0 rmdir" % path) - msg = 'dir removal failed (code %r) for %s' % (rv,path) - if rv != 0: - logger.warn(msg) - if strict: - raise koji.GenericError, msg - return rv + rc = os.spawnvp(os.P_WAIT, '/usr/sbin/koji-helper', ['/usr/sbin/koji-helper', op, path]) + if rc!=0: + msg = "directory removal failed (code %r) for %s" % (rc,path) + logger.warn(msg) + if strict: + raise koji.GenericError, msg + else: + return rc + return rc def umount_all(topdir): "Unmount every mount under topdir" @@ -635,7 +617,7 @@ class TaskManager(object): if age > 3600*24: #dir untouched for a day self.logger.info("Removing buildroot: %s" % desc) - if topdir and safe_rmtree(topdir, unmount=True, strict=False) != 0: + if topdir and safe_rmtree(topdir, strict=False) != 0: continue #also remove the config try: @@ -644,15 +626,7 @@ class TaskManager(object): self.logger.warn("%s: can't remove config: %s" % (desc, e)) elif age > 120: if rootdir: - try: - flist = os.listdir(rootdir) - except OSError, e: - self.logger.warn("%s: can't list rootdir: %s" % (desc, e)) - continue - if flist: - self.logger.info("%s: clearing rootdir" % desc) - for fn in flist: - safe_rmtree("%s/%s" % (rootdir,fn), unmount=True, strict=False) + safe_rmtree(rootdir, strict=False, op='rmtree') else: self.logger.debug("Recent buildroot: %s: %i seconds" % (desc,age)) self.logger.debug("Local buildroots: %d" % len(local_br)) @@ -1211,8 +1185,7 @@ class BaseTaskHandler(object): def removeWorkdir(self): if self.workdir is None: return - safe_rmtree(self.workdir, unmount=False, strict=True) - #os.spawnvp(os.P_WAIT, 'rm', ['rm', '-rf', self.workdir]) + os.spawnvp(os.P_WAIT, 'rm', ['rm', '-rf', self.workdir]) def wait(self, subtasks=None, all=False, failany=False): """Wait on subtasks diff --git a/koji.spec b/koji.spec index 13d3bf0..f14bb6e 100644 --- a/koji.spec +++ b/koji.spec @@ -16,7 +16,6 @@ Group: Applications/System URL: http://hosted.fedoraproject.org/projects/koji Source: %{name}-%{PACKAGE_VERSION}.tar.bz2 BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n) -BuildArch: noarch Requires: python-krbV >= 1.0.13 Requires: rpm-python Requires: pyOpenSSL @@ -89,6 +88,7 @@ koji-web is a web UI to the Koji system. %setup -q %build +make CFLAGS="$CFLAGS" CC="%__cc" all %install rm -rf $RPM_BUILD_ROOT @@ -125,6 +125,7 @@ rm -rf $RPM_BUILD_ROOT %files builder %defattr(-,root,root) +%attr(4710,root,kojibuilder) %_sbindir/koji-helper %{_sbindir}/kojid %{_initrddir}/kojid %config(noreplace) %{_sysconfdir}/sysconfig/kojid diff --git a/src/koji-helper.c b/src/koji-helper.c new file mode 100644 index 0000000..a3d0921 --- /dev/null +++ b/src/koji-helper.c @@ -0,0 +1,260 @@ +/* --*- c -*-- + * Copyright (C) 2007 Enrico Scholz + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; version 3 of the License. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see . + */ + +#define _GNU_SOURCE + +#ifndef MOCK_ROOT +# define MOCK_ROOT "/var/lib/mock" +#endif + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +static int __attribute__((__nonnull__(1, 2))) +safe_chdir(char const *path, struct stat const *exp_st) +{ + struct stat cur_st; + + if (strchr(path, '/')) { + fprintf(stderr, "safe_chdir(): invalid char in path '%s'\n", path); + return -1; + } + + if (strcmp(path, "..")==0) { + fprintf(stderr, "safe_chdir(): parent dir referred\n"); + return -1; + } + + if (chdir(path) < 0) { + fprintf(stderr, "chdir(%s): %s\n", path, strerror(errno)); + return -1; + } + + if (stat(".", &cur_st) < 0) { + fprintf(stderr, "stat(%s): %s\n", path, strerror(errno)); + return -2; + } + + if (cur_st.st_dev != exp_st->st_dev || + cur_st.st_ino != exp_st->st_ino) { + fprintf(stderr, "RACE: path '%s' changed before chdir()\n", path); + return -2; + } + + return 0; +} + +static int +rmrf_cwd(struct stat *cwd_st) +{ + DIR *cwd = opendir("."); + int rc = -1; + + if (!cwd) { + perror("opendir()"); + return -1; + } + + for (;;) { + struct dirent *ent = readdir(cwd); + struct stat st; + + if (!ent) + break; + + if (ent->d_name[0] == '.' && + (ent->d_name[1] == '\0'|| + (ent->d_name[1] == '.' && ent->d_name[2] == '\0'))) + continue; /* skip '.' and '..' */ + + if (lstat(ent->d_name, &st) < 0) { + fprintf(stderr, "rmrf_cwd: lstat(%s): %s\n", + ent->d_name, strerror(errno)); + continue; + } + + if (cwd_st && cwd_st->st_dev != st.st_dev) + continue; /* do not cross devices */ + else if (S_ISDIR(st.st_mode)) { + switch (safe_chdir(ent->d_name, &st)) { + case -1: continue; + case -2: break; + default: rmrf_cwd(&st); break; + } + + if (fchdir(dirfd(cwd)) < 0) { + perror("rmrf_cwd: fchdir()"); + goto err; + } + + if (rmdir(ent->d_name) < 0) { + fprintf(stderr, "rmrf_cwd: rmdir(%s): %s\n", + ent->d_name, strerror(errno)); + continue; + } + } else if (unlink(ent->d_name) < 0) { + fprintf(stderr, "rmrf_cwd: unlink(%s): %s\n", + ent->d_name, strerror(errno)); + continue; + } + } + + rc = 0; +err: + closedir(cwd); + return rc; +} + +static int +safe_chdir_subpath(char const *path_c, size_t path_len) +{ + char path[path_len+1]; + char *ptr = path; + int rc = 0; + + if (path_len == 0) + return 0; + + strncpy(path, path_c, path_len); + path[path_len] = '\0'; + + while (ptr) { + char *new_ptr = strsep(&ptr, "/"); + struct stat st; + + if (*new_ptr == '\0') + continue; /* skip empty path components + * (e.g. double /) */ + + if (lstat(new_ptr, &st) < 0) { + fprintf(stderr, "stat(%s): %s\n", + new_ptr, strerror(errno)); + rc = -1; + } else if (!S_ISDIR(st.st_mode) || S_ISLNK(st.st_mode)) { + fprintf(stderr, "safe_chdir_subpath(): invalid mode of '%s': %04x\n", + new_ptr, st.st_mode); + rc = -1; + } else + rc = safe_chdir(new_ptr, &st); + + if (rc < 0) + break; + } + + return rc; +} + +/* Usage: rmrf */ +static int +do_rmrf(int argc, char *argv[], bool remove_parent_dir) +{ + char const *dir; + size_t dir_len; + char const *last_path; + int parent_fd; + + if (argc != 2) { + fprintf(stderr, "wrong number of parameters for 'rmrf' operation\n"); + return EXIT_FAILURE; + } + + dir = argv[1]; + + /* strip leading MOCK_ROOT; it's a little bit hacky but required to + * keep backward compatibility */ + if (strncmp(dir, MOCK_ROOT, sizeof(MOCK_ROOT)-1) == 0) + dir += sizeof(MOCK_ROOT)-1; + + while (*dir == '/') + ++dir; /* strip leading '/' */ + + dir_len = strlen(dir); + while (dir_len>0 && dir[dir_len-1] == '/') + --dir_len; /* strip trailing '/' */ + + last_path = dir + dir_len; + while (last_path > dir && last_path[-1] != '/') + --last_path; + + if (dir_len == 0) { + fprintf(stderr, "do_rmrf(): empty path\n"); + return EXIT_FAILURE; + } + if (dir_len > 255) { + fprintf(stderr, "pathname too long\n"); + return EXIT_FAILURE; + } + + + /* real work begins here... */ + + if (chdir(MOCK_ROOT) < 0) { + perror("chdir()"); + return EXIT_FAILURE; + } + + if (last_path > dir && /* else, it would be a noop */ + safe_chdir_subpath(dir, last_path - dir) < 0) + return EXIT_FAILURE; + + parent_fd = open(".", O_RDONLY|O_DIRECTORY); + if (parent_fd < 0) { + perror("open()"); + return EXIT_FAILURE; + } + + if (safe_chdir_subpath(last_path, dir+dir_len - last_path + 1) < 0) + return EXIT_FAILURE; + + /* we are now *in* the given path */ + if (rmrf_cwd(NULL) < 0) + return EXIT_FAILURE; + + if (fchdir(parent_fd) < 0) { + perror("fchdir()"); + return EXIT_FAILURE; + } + + if (remove_parent_dir && + rmdir(last_path) < 0) + return EXIT_FAILURE; + + return EXIT_SUCCESS; +} + +int main(int argc, char *argv[]) +{ + if (argc < 2) { + fprintf(stderr, "not enough parameters\n"); + return EXIT_FAILURE; + } + + if (strcmp(argv[1], "rmrf") == 0) + return do_rmrf(argc-1, argv+1, true); + else if (strcmp(argv[1], "rmtree") == 0) + return do_rmrf(argc-1, argv+1, false); + else + fprintf(stderr, "unknown argument\n"); + + return EXIT_FAILURE; +} From rob.myers at gtri.gatech.edu Mon Sep 17 14:06:04 2007 From: rob.myers at gtri.gatech.edu (rob myers) Date: Mon, 17 Sep 2007 10:06:04 -0400 Subject: [PATCH koji] Added support for building from git repositories In-Reply-To: <20070915193158.11662.35393.stgit@kosh.bigo.ensc.de> References: <20070915193158.11662.35393.stgit@kosh.bigo.ensc.de> Message-ID: <1190037964.22264.68.camel@rxm-581b.stl.gtri.gatech.edu> On Sat, 2007-09-15 at 21:31 +0200, Enrico Scholz wrote: > This patch adds support for building from git:// respositories. It is used > like > TODO: the git and cvs handlers share some common, non trivial code. This > should be generalized. does anyone have any ideas about how to abstract or generalize the SCM backend from koji? the cvs/svn/git SCM backends all perform similar functions and use similar code- what is the best way to make them co-exist with maintainability and extensibility? rob. From jkeating at redhat.com Mon Sep 17 14:12:59 2007 From: jkeating at redhat.com (Jesse Keating) Date: Mon, 17 Sep 2007 10:12:59 -0400 Subject: [PATCH koji] Added support for building from git repositories In-Reply-To: <1190037964.22264.68.camel@rxm-581b.stl.gtri.gatech.edu> References: <20070915193158.11662.35393.stgit@kosh.bigo.ensc.de> <1190037964.22264.68.camel@rxm-581b.stl.gtri.gatech.edu> Message-ID: <20070917101259.6791ec48@localhost.localdomain> On Mon, 17 Sep 2007 10:06:04 -0400 rob myers wrote: > does anyone have any ideas about how to abstract or generalize the SCM > backend from koji? the cvs/svn/git SCM backends all perform similar > functions and use similar code- what is the best way to make them > co-exist with maintainability and extensibility? Probably the best way is to create an "scm_callback" or some such. Each function that deals with scm calls just calls "scm_callback(options)". A config file setting would define which scm_callback to use, either git/svn/hg/cvs/etc... They would all need to take the same options (whether or not they do anything with all the options doesn't matter). This way scm oddities are defined once per scm in the callback definition, and the rest of the koji code just calls generic scm_callback functions. Make sense? -- Jesse Keating Fedora -- All my bits are free, are yours? -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From rob.myers at gtri.gatech.edu Mon Sep 17 15:07:49 2007 From: rob.myers at gtri.gatech.edu (rob myers) Date: Mon, 17 Sep 2007 11:07:49 -0400 Subject: [PATCH koji] Added support for building from git repositories In-Reply-To: <20070917101259.6791ec48@localhost.localdomain> References: <20070915193158.11662.35393.stgit@kosh.bigo.ensc.de> <1190037964.22264.68.camel@rxm-581b.stl.gtri.gatech.edu> <20070917101259.6791ec48@localhost.localdomain> Message-ID: <1190041669.22264.74.camel@rxm-581b.stl.gtri.gatech.edu> On Mon, 2007-09-17 at 10:12 -0400, Jesse Keating wrote: > On Mon, 17 Sep 2007 10:06:04 -0400 > rob myers wrote: > > > does anyone have any ideas about how to abstract or generalize the SCM > > backend from koji? the cvs/svn/git SCM backends all perform similar > > functions and use similar code- what is the best way to make them > > co-exist with maintainability and extensibility? > > Probably the best way is to create an "scm_callback" or some such. > Each function that deals with scm calls just calls > "scm_callback(options)". A config file setting would define which > scm_callback to use, either git/svn/hg/cvs/etc... They would all need > to take the same options (whether or not they do anything with all the > options doesn't matter). This way scm oddities are defined once per > scm in the callback definition, and the rest of the koji code just > calls generic scm_callback functions. > > Make sense? i think so. i don't have time to code this up right now, but i think it is worth doing. maybe i can find some time to develop a patch in the next couple of weeks. if anyone wants to beat me to it, that'd be great. :) rob. From mikem at redhat.com Mon Sep 17 16:20:51 2007 From: mikem at redhat.com (Mike McLean) Date: Mon, 17 Sep 2007 12:20:51 -0400 Subject: [PATCH koji] added koji-helper setuid program In-Reply-To: <20070915194112.12268.93927.stgit@kosh.bigo.ensc.de> References: <20070915194112.12268.93927.stgit@kosh.bigo.ensc.de> Message-ID: <46EEA963.7050806@redhat.com> Enrico Scholz wrote: > This patch adds a 'koji-helper' setuid program which implements the > following methods: > Methods above are implemented to replace the python 'safe_rmtree()' method > which was never safe, nor will work when 'kojid' is running as non-root. It all depends on what you mean by safe, I suppose. The safe_rmtree function protects against the destruction of stray mounts underneath the buildroot. This is a serious risk, though perhaps some folks will not appreciate how serious until they are debugging a buildroot, add a mount, and accidentally delete its contents when the buildroot is cleaned. Your patch seems to remove this protection. I designed kojid to run as root, and I don't see that as a problem. Many daemons run as root and kojid has more need of it than most. I do not like the old mock security model and I consider it flawed. I have no desire to emulate it in koji. From enrico.scholz at informatik.tu-chemnitz.de Mon Sep 17 16:52:24 2007 From: enrico.scholz at informatik.tu-chemnitz.de (Enrico Scholz) Date: Mon, 17 Sep 2007 18:52:24 +0200 Subject: [PATCH koji] added koji-helper setuid program In-Reply-To: <46EEA963.7050806@redhat.com> (Mike McLean's message of "Mon, 17 Sep 2007 12:20:51 -0400") References: <20070915194112.12268.93927.stgit@kosh.bigo.ensc.de> <46EEA963.7050806@redhat.com> Message-ID: <87ps0hp73r.fsf@fc5.bigo.ensc.de> Mike McLean writes: >> This patch adds a 'koji-helper' setuid program which implements the >> following methods: > >> Methods above are implemented to replace the python 'safe_rmtree()' method >> which was never safe, nor will work when 'kojid' is running as non-root. > > It all depends on what you mean by safe Definitively not the racy | find ... | xargs rm ... > The safe_rmtree function protects against the destruction of stray > mounts underneath the buildroot. This is a serious risk, though perhaps > some folks will not appreciate how serious until they are debugging a > buildroot, add a mount, and accidentally delete its contents when the > buildroot is cleaned. > > Your patch seems to remove this protection. no; it does not cross filesystem borders. > I designed kojid to run as root, and I don't see that as a problem. Many > daemons run as root and kojid has more need of it than most. What are these needs? 'kojid' runs perfectly as non-root. > I do not like the old mock security model and I consider it flawed. I > have no desire to emulate it in koji. Yes; mock's helper binary is full of races and broken constraints :( Enrico From aberoham at gmail.com Wed Sep 19 20:34:41 2007 From: aberoham at gmail.com (aberoham at gmail.com) Date: Wed, 19 Sep 2007 13:34:41 -0700 Subject: koji initialization, docs In-Reply-To: <3bdb07840709101003i2c9fb465vd9604dc5033d47a5@mail.gmail.com> References: <3bdb07840709101003i2c9fb465vd9604dc5033d47a5@mail.gmail.com> Message-ID: <3bdb07840709191334u2809d7a3sab446cbf84e2f9fd@mail.gmail.com> What I have figured out about koji initialization so far from talking to folks is attached. My koji, however, isn't "working." I'm reproducing my setup from scratch using these notes in the hopes that I simply messed something up with too much parallel mucking along with way. While I'm retracing my steps, I would be much obliged if any of the koji experts could take a gander that these notes and comment. Thanks, Abe On 9/10/07, aberoham at gmail.com wrote: > > > I have latest trunk version of koji, kojid, kojira, etc setup and running. > But now what? Does any administration documentation exist? To figure out how > to put gas in this engine do I really have to read the source? > > Thanks. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: setup_notes.txt URL: From mikem at redhat.com Thu Sep 20 19:09:58 2007 From: mikem at redhat.com (Mike McLean) Date: Thu, 20 Sep 2007 15:09:58 -0400 Subject: koji initialization, docs In-Reply-To: <3bdb07840709191334u2809d7a3sab446cbf84e2f9fd@mail.gmail.com> References: <3bdb07840709101003i2c9fb465vd9604dc5033d47a5@mail.gmail.com> <3bdb07840709191334u2809d7a3sab446cbf84e2f9fd@mail.gmail.com> Message-ID: <46F2C586.7080802@redhat.com> aberoham at gmail.com wrote: > Note: kojira must be running and must have repo perms (perm_id 3) > > 1. import all srpms from the upstream RHEL/Fedora distribution, use --create-build > koji import --create-build [SRPM] In Koji, builds are named after the srpm, i.e they use the e:n-v-r of the source rpm. Because the SOURCERPM field of the rpm does not include an epoch the system cannot be sure it has the correct data for the build entry. So, by default, it only creates the build entry when importing an srpm. The --create-build option causes import to create a build entry for an rpm even if its srpm is not in the import batch. It will use the n-v-r from the SOURCERPM field, and assume the epoch of the (missing) srpm is the same as the epoch of the rpm. So --create-build should only be necessary when your import is missing srpms. If you're importing Fedora content, I would hope this is not the case. > 2. imprt all binary rpms (for each arch) > koji import [RPM] The current import code in the koji cli will sort the srpms on the command line into groups based on their srpm. It will automatically import the srpms first and if there are rpms that do not have a corresponding srpm or existing build entry (and --create-build is not specified) it will warn and exit before taking any action. So, you should be able to collapse 1 and 2 into one command. I should also point out: if you're importing a large number of [s]rpms that are already on the same filesystem as your koji store, then the --link option may save you a lot of time. It requires that your (local) user have permission to write to the koji work directory. It bypasses the upload and instead simply hardlinks the file into the work directory (where it would have been uploaded to anyway). > 3. create a tag, we'll call it mydist > koji add-tag mydist > koji edit-tag mydist --arches="x86_64 i386" add-tag supports a --arches argument to specify the initial arch list. > 4. add all untagged builds to that tag > (use koji list-untagged to get the list) > koji tag-pkg [package-n-v-r ... package-n-v-r] --force --nowait heh, that's a clever way to generate the n-v-r list during bootstrap :) > 9. add each host listed in "koji list-hosts" to createrepo channel > # koji list-hosts > # koji add-host-to-channel createrepo We put the repo tasks on a special channel because we found that some hosts in our system were particularly slow at it. By leaving those hosts out of the createrepo channel, we found we could improve overall repo regeneration speed. It's also conceivable that you might want to keep some hosts out of the createrepo pool simply to reduce load on them. From oliver at linux-kernel.at Thu Sep 20 19:18:36 2007 From: oliver at linux-kernel.at (Oliver Falk) Date: Thu, 20 Sep 2007 21:18:36 +0200 Subject: koji initialization, docs In-Reply-To: <46F2C586.7080802@redhat.com> References: <3bdb07840709101003i2c9fb465vd9604dc5033d47a5@mail.gmail.com> <3bdb07840709191334u2809d7a3sab446cbf84e2f9fd@mail.gmail.com> <46F2C586.7080802@redhat.com> Message-ID: <46F2C78C.6080204@linux-kernel.at> Mike McLean schrieb: > aberoham at gmail.com wrote: >> Note: kojira must be running and must have repo perms (perm_id 3) >> 1. import all srpms from the upstream RHEL/Fedora distribution, use >> --create-build >> koji import --create-build [SRPM] > > In Koji, builds are named after the srpm, i.e they use the e:n-v-r of > the source rpm. Because the SOURCERPM field of the rpm does not include > an epoch the system cannot be sure it has the correct data for the build > entry. So, by default, it only creates the build entry when importing an > srpm. > > The --create-build option causes import to create a build entry for an > rpm even if its srpm is not in the import batch. It will use the n-v-r > from the SOURCERPM field, and assume the epoch of the (missing) srpm is > the same as the epoch of the rpm. > > So --create-build should only be necessary when your import is missing > srpms. If you're importing Fedora content, I would hope this is not the > case. > >> 2. imprt all binary rpms (for each arch) >> koji import [RPM] > > The current import code in the koji cli will sort the srpms on the > command line into groups based on their srpm. It will automatically > import the srpms first and if there are rpms that do not have a > corresponding srpm or existing build entry (and --create-build is not > specified) it will warn and exit before taking any action. > > So, you should be able to collapse 1 and 2 into one command. > > I should also point out: if you're importing a large number of [s]rpms > that are already on the same filesystem as your koji store, then the > --link option may save you a lot of time. It requires that your (local) > user have permission to write to the koji work directory. It bypasses > the upload and instead simply hardlinks the file into the work directory > (where it would have been uploaded to anyway). > >> 3. create a tag, we'll call it mydist >> koji add-tag mydist >> koji edit-tag mydist --arches="x86_64 i386" > > add-tag supports a --arches argument to specify the initial arch list. > >> 4. add all untagged builds to that tag >> (use koji list-untagged to get the list) >> koji tag-pkg [package-n-v-r ... package-n-v-r] --force --nowait > > heh, that's a clever way to generate the n-v-r list during bootstrap :) > >> 9. add each host listed in "koji list-hosts" to createrepo channel >> # koji list-hosts >> # koji add-host-to-channel createrepo > > We put the repo tasks on a special channel because we found that some > hosts in our system were particularly slow at it. By leaving those hosts > out of the createrepo channel, we found we could improve overall repo > regeneration speed. > > It's also conceivable that you might want to keep some hosts out of the > createrepo pool simply to reduce load on them. You might also want to take a look at http://wiki.linux-kernel.at/index.php/Koji :-) Best, Oliver From Michael_E_Brown at dell.com Tue Sep 25 19:11:12 2007 From: Michael_E_Brown at dell.com (Michael E Brown) Date: Tue, 25 Sep 2007 14:11:12 -0500 Subject: mock improvements Message-ID: <20070925191111.GB2414@humbolt.us.dell.com> So here is the list of things that have been requested lately and I'll be working on a few of these over the next few weeks. If anybody has any input, I'd take it. As I start on each, I'll most likely email the mailing list with the outline of what I'm doing. If anybody has existing patches for these (against current mock git), all the better... :) 1) more reliable mount/umount several people have pointed out instances where mock exits leaving mounts behind (specifically /dev), and the next invokation of mock ends up 'rm -rf' the host machine's /dev. Bad.... 2) caching yum downloads several people have commented that the autocache stuff is great for speeding up builds, others say that it can sometimes be bad for reproducability, and that simply saving the yum cache dir would be better. 3) ccache integration This is a new one that I havent seen before, but should significantly speed up builds for people who often do rebuild-the-entire-distribution-type things. I'm told by some that this is bad for reproducability, but good for speeding up builds when you are just sanity checking or when that small reproducability hit doesnt matter. I've also seen lots of empirical data that ccache should not cause any problems. This will be have to be specifically enabled through a commandline or config file option, so those who care can turn it on/off. 4) distcc integration Pretty much the same case as ccache. Has more things that need thought than the ccache case, above, though. -- Michael From mikeb at redhat.com Tue Sep 25 20:00:58 2007 From: mikeb at redhat.com (Mike Bonnet) Date: Tue, 25 Sep 2007 16:00:58 -0400 Subject: mock improvements In-Reply-To: <20070925191111.GB2414@humbolt.us.dell.com> References: <20070925191111.GB2414@humbolt.us.dell.com> Message-ID: <1190750458.31945.8.camel@maunalani.home> On Tue, 2007-09-25 at 14:11 -0500, Michael E Brown wrote: > So here is the list of things that have been requested lately and I'll > be working on a few of these over the next few weeks. If anybody has any > input, I'd take it. As I start on each, I'll most likely email the > mailing list with the outline of what I'm doing. > > If anybody has existing patches for these (against current mock git), > all the better... :) > > 1) more reliable mount/umount > several people have pointed out instances where mock exits leaving > mounts behind (specifically /dev), and the next invokation of mock > ends up 'rm -rf' the host machine's /dev. Bad.... > > 2) caching yum downloads > several people have commented that the autocache stuff is great for > speeding up builds, others say that it can sometimes be bad for > reproducability, and that simply saving the yum cache dir would be > better. > > 3) ccache integration > This is a new one that I havent seen before, but should significantly > speed up builds for people who often do > rebuild-the-entire-distribution-type things. I'm told by some that > this is bad for reproducability, but good for speeding up builds when > you are just sanity checking or when that small reproducability hit > doesnt matter. I've also seen lots of empirical data that ccache > should not cause any problems. This will be have to be specifically > enabled through a commandline or config file option, so those who care > can turn it on/off. > > 4) distcc integration > Pretty much the same case as ccache. Has more things that need thought > than the ccache case, above, though. A method for cleaning up stale/orphaned processes that get created during a build was proposed a while ago: https://bugzilla.redhat.com/show_bug.cgi?id=221351 I'll leave it to you as to whether or not this is the correct implementation, but I certainly think it's a good idea. I see rogue processes consuming resources on the build machines all the time. The process-cleanup code should probably be run before mock exits, whether the build completed successfully or not. From williams at redhat.com Tue Sep 25 20:10:01 2007 From: williams at redhat.com (Clark Williams) Date: Tue, 25 Sep 2007 15:10:01 -0500 Subject: mock improvements In-Reply-To: <1190750458.31945.8.camel@maunalani.home> References: <20070925191111.GB2414@humbolt.us.dell.com> <1190750458.31945.8.camel@maunalani.home> Message-ID: <46F96B19.1030701@redhat.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Mike Bonnet wrote: > On Tue, 2007-09-25 at 14:11 -0500, Michael E Brown wrote: >> So here is the list of things that have been requested lately and I'll >> be working on a few of these over the next few weeks. If anybody has any >> input, I'd take it. As I start on each, I'll most likely email the >> mailing list with the outline of what I'm doing. >> >> If anybody has existing patches for these (against current mock git), >> all the better... :) >> >> 1) more reliable mount/umount >> several people have pointed out instances where mock exits leaving >> mounts behind (specifically /dev), and the next invokation of mock >> ends up 'rm -rf' the host machine's /dev. Bad.... >> >> 2) caching yum downloads >> several people have commented that the autocache stuff is great for >> speeding up builds, others say that it can sometimes be bad for >> reproducability, and that simply saving the yum cache dir would be >> better. >> >> 3) ccache integration >> This is a new one that I havent seen before, but should significantly >> speed up builds for people who often do >> rebuild-the-entire-distribution-type things. I'm told by some that >> this is bad for reproducability, but good for speeding up builds when >> you are just sanity checking or when that small reproducability hit >> doesnt matter. I've also seen lots of empirical data that ccache >> should not cause any problems. This will be have to be specifically >> enabled through a commandline or config file option, so those who care >> can turn it on/off. >> >> 4) distcc integration >> Pretty much the same case as ccache. Has more things that need thought >> than the ccache case, above, though. > > A method for cleaning up stale/orphaned processes that get created > during a build was proposed a while ago: > > https://bugzilla.redhat.com/show_bug.cgi?id=221351 > > I'll leave it to you as to whether or not this is the correct > implementation, but I certainly think it's a good idea. I see rogue > processes consuming resources on the build machines all the time. The > process-cleanup code should probably be run before mock exits, whether > the build completed successfully or not. > Didn't this go in (orphanskill) in the last round of updates? Clark -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFG+WsZHyuj/+TTEp0RAk9NAJwOhOmF6xmaIhbPDRVpaBKwHV65nwCgjBOU bjyOf43DXXIHJj/AW+jz2Ms= =cx6b -----END PGP SIGNATURE----- From jos at xos.nl Wed Sep 26 10:19:06 2007 From: jos at xos.nl (Jos Vos) Date: Wed, 26 Sep 2007 12:19:06 +0200 Subject: mock improvements In-Reply-To: <20070925191111.GB2414@humbolt.us.dell.com> References: <20070925191111.GB2414@humbolt.us.dell.com> Message-ID: <20070926101906.GA6835@jasmine.xos.nl> On Tue, Sep 25, 2007 at 02:11:12PM -0500, Michael E Brown wrote: > So here is the list of things that have been requested lately and I'll > be working on a few of these over the next few weeks. If anybody has any > input, I'd take it. As I start on each, I'll most likely email the > mailing list with the outline of what I'm doing. Maybe you can also look at bug #235141 (nearly 6 months old), about not removing files with the immutable bit set. The workaround posted there works fine, b.t.w., but I'm not sure if this is the ultimate solution. > 1) more reliable mount/umount > several people have pointed out instances where mock exits leaving > mounts behind (specifically /dev), and the next invokation of mock > ends up 'rm -rf' the host machine's /dev. Bad.... And /dev/pts and /proc. > 2) caching yum downloads > several people have commented that the autocache stuff is great for > speeding up builds, others say that it can sometimes be bad for > reproducability, and that simply saving the yum cache dir would be > better. Only as an extra option, I assume. -- -- Jos Vos -- X/OS Experts in Open Systems BV | Phone: +31 20 6938364 -- Amsterdam, The Netherlands | Fax: +31 20 6948204 From Michael_E_Brown at Dell.com Thu Sep 27 19:55:45 2007 From: Michael_E_Brown at Dell.com (Michael E Brown) Date: Thu, 27 Sep 2007 14:55:45 -0500 Subject: problems with orphanskill feature In-Reply-To: <46FBF6B7.1000105@gmail.com> References: <46FBF6B7.1000105@gmail.com> Message-ID: <20070927195545.GA6471@humbolt.us.dell.com> On Thu, Sep 27, 2007 at 01:30:15PM -0500, Clark Williams wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Michael/Jesse, > > We're seeing odd failures with mock and the orphanskill feature. What seems to happen > is that the '; mock-helper orphanskill ' string is tacked onto a command > which is passed to do_chroot() and after the main command is run, an attempt is made > to run mock-helper (which is not installed in the chroot). So people are seeing a > "File not found" message after a successful command. > > Now it looks like to me that part of the reason for an orphanskill was that the do() > routine might hang until all the child processes are done, so I'm loathe to just run > the orphanskill after the do_chroot() is finished (I suspect twisty lines of logic, > all alike). Seems like we can do a couple of things: > > 1. Copy mock-helper into each chroot, so it's available for orphanskill > 2. Back out the orphanskill logic and try again > > Option #1 is somewhat easy, if kinda ugly (not sure I like the idea of scattering a > setuid-root program into all our build roots). Option #2 requires that we look at the > code in all the do_* and do() routines to make sure that orphanskill runs when we > need it to. Ideally I'd like to insure that orphanskill runs *outside* the chroot and > that it's not needed to keep self.do() from hanging. > > What you guys think? How about we just run two commands in a row? I see the comment but don't really see why. Line 973, we dont need to run orphanskill if it isnt chroot. For the my.do_chroot() on line 975, it looks like we could just do a my.do_chroot() followed by a normal os.system(). The problem it is trying to fix is if the rpmbuild process spawns child processes that fork and never exit. I believe this was seen in some code that was running in the rpmbuild as a unit test? We should also be cc-ing fedora-buildsys-list. (done) I also understand Jesse's sentiment to just back it out. If it is going to take more than a day or two to fix, we could just back it out. -- Michael From williams at redhat.com Thu Sep 27 21:16:33 2007 From: williams at redhat.com (Clark Williams) Date: Thu, 27 Sep 2007 16:16:33 -0500 Subject: problems with orphanskill feature In-Reply-To: <20070927195545.GA6471@humbolt.us.dell.com> References: <46FBF6B7.1000105@gmail.com> <20070927195545.GA6471@humbolt.us.dell.com> Message-ID: <46FC1DB1.3060702@redhat.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Resent from my Red Hat account so the message isn't subject to moderation... Michael E Brown wrote: > On Thu, Sep 27, 2007 at 01:30:15PM -0500, Clark Williams wrote: >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> Michael/Jesse, >> >> We're seeing odd failures with mock and the orphanskill feature. What seems to happen >> is that the '; mock-helper orphanskill ' string is tacked onto a command >> which is passed to do_chroot() and after the main command is run, an attempt is made >> to run mock-helper (which is not installed in the chroot). So people are seeing a >> "File not found" message after a successful command. >> >> Now it looks like to me that part of the reason for an orphanskill was that the do() >> routine might hang until all the child processes are done, so I'm loathe to just run >> the orphanskill after the do_chroot() is finished (I suspect twisty lines of logic, >> all alike). Seems like we can do a couple of things: >> >> 1. Copy mock-helper into each chroot, so it's available for orphanskill >> 2. Back out the orphanskill logic and try again >> >> Option #1 is somewhat easy, if kinda ugly (not sure I like the idea of scattering a >> setuid-root program into all our build roots). Option #2 requires that we look at the >> code in all the do_* and do() routines to make sure that orphanskill runs when we >> need it to. Ideally I'd like to insure that orphanskill runs *outside* the chroot and >> that it's not needed to keep self.do() from hanging. >> >> What you guys think? > > How about we just run two commands in a row? I see the comment but don't > really see why. Line 973, we dont need to run orphanskill if it isnt > chroot. For the my.do_chroot() on line 975, it looks like we could just > do a my.do_chroot() followed by a normal os.system(). That was my first thought, but I was concerned that we might be missing something subtle in the timeout code (hence my email to you :)). If you think we can just run the orphanskill stuff after running the do_chroot() then I say that's the way to go. > > The problem it is trying to fix is if the rpmbuild process spawns child > processes that fork and never exit. I believe this was seen in some code > that was running in the rpmbuild as a unit test? > > We should also be cc-ing fedora-buildsys-list. (done) wups (looks shamefaced) > > I also understand Jesse's sentiment to just back it out. If it is going > to take more than a day or two to fix, we could just back it out. Let's try running it right after the rpmbuild. If that doesn't work right, then we can just comment it out while we take a closer look at it. Clark -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFG/B2xHyuj/+TTEp0RAp39AKCetOBTdrrXkAia7FETgL6Zy7A/SACfUW6g neURvZbXryWInfBOVe9tY6M= =PGG3 -----END PGP SIGNATURE----- From Michael_E_Brown at Dell.com Thu Sep 27 21:21:48 2007 From: Michael_E_Brown at Dell.com (Michael_E_Brown at Dell.com) Date: Thu, 27 Sep 2007 16:21:48 -0500 Subject: problems with orphanskill feature References: <46FBF6B7.1000105@gmail.com> <20070927195545.GA6471@humbolt.us.dell.com> <46FC1C6E.2040801@gmail.com> Message-ID: Do you want to do the patch or me? It will be tomorrow morning before I can get to it. (or very late tonight.) -- Michael -----Original Message----- From: Clark Williams [mailto:clark.williams at gmail.com] Sent: Thu 9/27/2007 4:11 PM To: Brown, Michael E Cc: Jesse Keating; fedora-buildsys-list at redhat.com Subject: Re: problems with orphanskill feature -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Michael E Brown wrote: > On Thu, Sep 27, 2007 at 01:30:15PM -0500, Clark Williams wrote: >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> Michael/Jesse, >> >> We're seeing odd failures with mock and the orphanskill feature. What seems to happen >> is that the '; mock-helper orphanskill ' string is tacked onto a command >> which is passed to do_chroot() and after the main command is run, an attempt is made >> to run mock-helper (which is not installed in the chroot). So people are seeing a >> "File not found" message after a successful command. >> >> Now it looks like to me that part of the reason for an orphanskill was that the do() >> routine might hang until all the child processes are done, so I'm loathe to just run >> the orphanskill after the do_chroot() is finished (I suspect twisty lines of logic, >> all alike). Seems like we can do a couple of things: >> >> 1. Copy mock-helper into each chroot, so it's available for orphanskill >> 2. Back out the orphanskill logic and try again >> >> Option #1 is somewhat easy, if kinda ugly (not sure I like the idea of scattering a >> setuid-root program into all our build roots). Option #2 requires that we look at the >> code in all the do_* and do() routines to make sure that orphanskill runs when we >> need it to. Ideally I'd like to insure that orphanskill runs *outside* the chroot and >> that it's not needed to keep self.do() from hanging. >> >> What you guys think? > > How about we just run two commands in a row? I see the comment but don't > really see why. Line 973, we dont need to run orphanskill if it isnt > chroot. For the my.do_chroot() on line 975, it looks like we could just > do a my.do_chroot() followed by a normal os.system(). That was my first thought, but I was concerned that we might be missing something subtle in the timeout code (hence my email to you :)). If you think we can just run the orphanskill stuff after running the do_chroot() then I say that's the way to go. > > The problem it is trying to fix is if the rpmbuild process spawns child > processes that fork and never exit. I believe this was seen in some code > that was running in the rpmbuild as a unit test? > > We should also be cc-ing fedora-buildsys-list. (done) wups (looks shamefaced) > > I also understand Jesse's sentiment to just back it out. If it is going > to take more than a day or two to fix, we could just back it out. Let's try running it right after the rpmbuild. If that doesn't work right, then we can just comment it out while we take a closer look at it. Clark -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFG/BxuqA4JVb61b9cRAkAIAJ0cusVhhetlhnNJA5RWTkcD4cHG5wCcD5Rb aQo57BZBD1ME0FLDj6nARq4= =itpW -----END PGP SIGNATURE----- From williams at redhat.com Thu Sep 27 21:36:49 2007 From: williams at redhat.com (Clark Williams) Date: Thu, 27 Sep 2007 16:36:49 -0500 Subject: problems with orphanskill feature In-Reply-To: References: <46FBF6B7.1000105@gmail.com> <20070927195545.GA6471@humbolt.us.dell.com> <46FC1C6E.2040801@gmail.com> Message-ID: <46FC2271.9040904@redhat.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Michael_E_Brown at Dell.com wrote: > Do you want to do the patch or me? It will be tomorrow morning before I can get to it. (or very late tonight.) > -- > Michael > > I can do it, although I've somehow managed to hose my git configuration so that I can't seem to push to our fedoraproject repository. I've got a fix for BZ 303791 (just changing the search string for now to match the resolvdep output) and I'd like to get that up before trying the orphanskill fix. Since I never actually got bodhi working last time, I think I should give it another go. I'll make the BZ fix in devel/F-7/FC-6 and when I get that straightened out we can try an orphanskill fix in rawhide. Clark -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFG/CJwHyuj/+TTEp0RAqinAKDZtJS8k2L7qmiNAQ+Kj6AUjHSqZACgidms ydMyyZlburkHY0KN9xkiUxA= =TvvK -----END PGP SIGNATURE-----