From dan at danny.cz Wed Nov 5 09:55:44 2008 From: dan at danny.cz (Dan =?ISO-8859-1?Q?Hor=E1k?=) Date: Wed, 05 Nov 2008 10:55:44 +0100 Subject: caching packages on koji builder Message-ID: <1225878944.3544.27.camel@eagle.danny.cz> Hello, is it possible to somehow cache the packages that are going from koji hub to the builder to create a build root? There is no NFS connection between the builder and hub in s390's koji and so each build requires to download 100 MB via XMLRPC(?) and this is slow. Will squid help here? Dan -- Fedora and Red Hat package maintainer From oliver at linux-kernel.at Wed Nov 5 10:07:10 2008 From: oliver at linux-kernel.at (Oliver Falk) Date: Wed, 05 Nov 2008 11:07:10 +0100 Subject: caching packages on koji builder In-Reply-To: <1225878944.3544.27.camel@eagle.danny.cz> References: <1225878944.3544.27.camel@eagle.danny.cz> Message-ID: <4911704E.4080105@linux-kernel.at> Dan Hor?k wrote: > Hello, > > is it possible to somehow cache the packages that are going from koji > hub to the builder to create a build root? There is no NFS connection > between the builder and hub in s390's koji and so each build requires to > download 100 MB via XMLRPC(?) Newest kojis do not download via XMLRPC. Old kojis 'downloaded' via NFS. > and this is slow. Will squid help here? Maybe... But with a GiB-Connection download of 100MB shouldn't be a problem. Resolving dependencies, unpacking/writing files from packages to the buildroot should take longer, than the download.... -of From dan at danny.cz Wed Nov 5 10:36:06 2008 From: dan at danny.cz (Dan =?ISO-8859-1?Q?Hor=E1k?=) Date: Wed, 05 Nov 2008 11:36:06 +0100 Subject: caching packages on koji builder In-Reply-To: <4911704E.4080105@linux-kernel.at> References: <1225878944.3544.27.camel@eagle.danny.cz> <4911704E.4080105@linux-kernel.at> Message-ID: <1225881366.3544.34.camel@eagle.danny.cz> Oliver Falk p??e v St 05. 11. 2008 v 11:07 +0100: > Dan Hor?k wrote: > > Hello, > > > > is it possible to somehow cache the packages that are going from koji > > hub to the builder to create a build root? There is no NFS connection > > between the builder and hub in s390's koji and so each build requires to > > download 100 MB via XMLRPC(?) > > Newest kojis do not download via XMLRPC. Old kojis 'downloaded' via NFS. > > > and this is slow. Will squid help here? > > Maybe... > > But with a GiB-Connection download of 100MB shouldn't be a problem. > Resolving dependencies, unpacking/writing files from packages to the > buildroot should take longer, than the download.... But the hub and the builders are all at separate locations, so they are only connected via Internet and the lines are a bit slower :-) Dan From oliver at linux-kernel.at Wed Nov 5 11:06:17 2008 From: oliver at linux-kernel.at (Oliver Falk) Date: Wed, 05 Nov 2008 12:06:17 +0100 Subject: caching packages on koji builder In-Reply-To: <1225881366.3544.34.camel@eagle.danny.cz> References: <1225878944.3544.27.camel@eagle.danny.cz> <4911704E.4080105@linux-kernel.at> <1225881366.3544.34.camel@eagle.danny.cz> Message-ID: <49117E29.4060502@linux-kernel.at> Dan Hor?k wrote: > Oliver Falk p??e v St 05. 11. 2008 v 11:07 +0100: >> Dan Hor?k wrote: >>> Hello, >>> >>> is it possible to somehow cache the packages that are going from koji >>> hub to the builder to create a build root? There is no NFS connection >>> between the builder and hub in s390's koji and so each build requires to >>> download 100 MB via XMLRPC(?) >> Newest kojis do not download via XMLRPC. Old kojis 'downloaded' via NFS. >> >> > and this is slow. Will squid help here? >> >> Maybe... >> >> But with a GiB-Connection download of 100MB shouldn't be a problem. >> Resolving dependencies, unpacking/writing files from packages to the >> buildroot should take longer, than the download.... > > But the hub and the builders are all at separate locations, so they are > only connected via Internet and the lines are a bit slower :-) Oh, that's bad. :-( Dunno if you can tell kojid/mock to use a proxy... -of From mikeb at redhat.com Wed Nov 5 15:43:36 2008 From: mikeb at redhat.com (Mike Bonnet) Date: Wed, 05 Nov 2008 10:43:36 -0500 Subject: caching packages on koji builder In-Reply-To: <1225878944.3544.27.camel@eagle.danny.cz> References: <1225878944.3544.27.camel@eagle.danny.cz> Message-ID: <1225899816.18425.8.camel@maunalani.home> On Wed, 2008-11-05 at 10:55 +0100, Dan Hor?k wrote: > Hello, > > is it possible to somehow cache the packages that are going from koji > hub to the builder to create a build root? There is no NFS connection > between the builder and hub in s390's koji and so each build requires to > download 100 MB via XMLRPC(?) and this is slow. Will squid help here? Koji builders have never downloaded packages via XMLRPC. All downloading is done by mock/yum, via http (previously nfs). You could potentially use squid locally to cache downloaded packages. You'd configure pkgurl in koji.conf to point to your local squid instance at "http://localhost:8080/koji/packages" or something similar, and configure squid to pull from the actual http location where /mnt/koji/packages is being served. From tibbs at math.uh.edu Wed Nov 5 15:49:46 2008 From: tibbs at math.uh.edu (Jason L Tibbitts III) Date: 05 Nov 2008 09:49:46 -0600 Subject: caching packages on koji builder In-Reply-To: <1225899816.18425.8.camel@maunalani.home> References: <1225878944.3544.27.camel@eagle.danny.cz> <1225899816.18425.8.camel@maunalani.home> Message-ID: >>>>> "MB" == Mike Bonnet writes: MB> Koji builders have never downloaded packages via XMLRPC. All MB> downloading is done by mock/yum, via http (previously nfs). Well, mock can cache all sorts of things these days. If there are multiple builders at one location then having a single squid cache for them all might be nice, but mock's caching would still help to avoid having to hit the network. - J< From mikeb at redhat.com Wed Nov 5 17:03:32 2008 From: mikeb at redhat.com (Mike Bonnet) Date: Wed, 05 Nov 2008 12:03:32 -0500 Subject: caching packages on koji builder In-Reply-To: References: <1225878944.3544.27.camel@eagle.danny.cz> <1225899816.18425.8.camel@maunalani.home> Message-ID: <1225904612.18425.18.camel@maunalani.home> On Wed, 2008-11-05 at 09:49 -0600, Jason L Tibbitts III wrote: > >>>>> "MB" == Mike Bonnet writes: > > MB> Koji builders have never downloaded packages via XMLRPC. All > MB> downloading is done by mock/yum, via http (previously nfs). > > Well, mock can cache all sorts of things these days. If there are > multiple builders at one location then having a single squid cache for > them all might be nice, but mock's caching would still help to avoid > having to hit the network. Actually mock's caching doesn't really help us. It's all done per-buildroot, and since every build is run in a different buildroot, the caches would never be reused. For this reason Koji disables caching in the mock configs it writes out. A global (per-machine) rpm cache might be useful for reducing network bandwidth. However, because mock/yum would have to lock this global cache while interacting with it, it would become a bottleneck when running concurrent builds. The best approach currently is probably a local squid cache. From mikem at redhat.com Wed Nov 5 18:47:56 2008 From: mikem at redhat.com (Mike McLean) Date: Wed, 05 Nov 2008 13:47:56 -0500 Subject: caching packages on koji builder In-Reply-To: <1225899816.18425.8.camel@maunalani.home> References: <1225878944.3544.27.camel@eagle.danny.cz> <1225899816.18425.8.camel@maunalani.home> Message-ID: <4911EA5C.4030406@redhat.com> Mike Bonnet wrote: > On Wed, 2008-11-05 at 10:55 +0100, Dan Hor?k wrote: >> Hello, >> >> is it possible to somehow cache the packages that are going from koji >> hub to the builder to create a build root? There is no NFS connection >> between the builder and hub in s390's koji and so each build requires to >> download 100 MB via XMLRPC(?) and this is slow. Will squid help here? > > Koji builders have never downloaded packages via XMLRPC. All > downloading is done by mock/yum, via http (previously nfs). This behavior is controlled by kojid options. If you specify the 'topurl' option for kojid, then the mock configs it generates will use an http:// url to point to the repo. Otherwise it will use a file:// url (using the value of the 'topdir' option, which defaults to /mnt/koji). Also, the use of a file:// url doesn't have to mean nfs. You could theoretically use another shared file system. > You could potentially use squid locally to cache downloaded packages. > You'd configure pkgurl in koji.conf to point to your local squid > instance at "http://localhost:8080/koji/packages" or something similar, > and configure squid to pull from the actual http location > where /mnt/koji/packages is being served. This is the approach I would recommend. From dan at danny.cz Wed Nov 5 19:19:15 2008 From: dan at danny.cz (Dan =?ISO-8859-1?Q?Hor=E1k?=) Date: Wed, 05 Nov 2008 20:19:15 +0100 Subject: caching packages on koji builder In-Reply-To: <4911EA5C.4030406@redhat.com> References: <1225878944.3544.27.camel@eagle.danny.cz> <1225899816.18425.8.camel@maunalani.home> <4911EA5C.4030406@redhat.com> Message-ID: <1225912755.3544.58.camel@eagle.danny.cz> Mike McLean p??e v St 05. 11. 2008 v 13:47 -0500: > Mike Bonnet wrote: > > On Wed, 2008-11-05 at 10:55 +0100, Dan Hor?k wrote: > >> Hello, > >> > >> is it possible to somehow cache the packages that are going from koji > >> hub to the builder to create a build root? There is no NFS connection > >> between the builder and hub in s390's koji and so each build requires to > >> download 100 MB via XMLRPC(?) and this is slow. Will squid help here? > > > > Koji builders have never downloaded packages via XMLRPC. All > > downloading is done by mock/yum, via http (previously nfs). > > This behavior is controlled by kojid options. If you specify the > 'topurl' option for kojid, then the mock configs it generates will use > an http:// url to point to the repo. Otherwise it will use a file:// url > (using the value of the 'topdir' option, which defaults to /mnt/koji). > > Also, the use of a file:// url doesn't have to mean nfs. You could > theoretically use another shared file system. > > > You could potentially use squid locally to cache downloaded packages. > > You'd configure pkgurl in koji.conf to point to your local squid > > instance at "http://localhost:8080/koji/packages" or something similar, > > and configure squid to pull from the actual http location > > where /mnt/koji/packages is being served. > > This is the approach I would recommend. Thanks for all your opinions. I will use the url_rewrite* feature of squid. Dan From mikem at redhat.com Wed Nov 5 20:14:43 2008 From: mikem at redhat.com (Mike McLean) Date: Wed, 05 Nov 2008 15:14:43 -0500 Subject: caching packages on koji builder In-Reply-To: <4911EA5C.4030406@redhat.com> References: <1225878944.3544.27.camel@eagle.danny.cz> <1225899816.18425.8.camel@maunalani.home> <4911EA5C.4030406@redhat.com> Message-ID: <4911FEB3.2000808@redhat.com> Mike McLean wrote: > This behavior is controlled by kojid options. If you specify the > 'topurl' option for kojid, then the mock configs it generates will use > an http:// url to point to the repo. Otherwise it will use a file:// url > (using the value of the 'topdir' option, which defaults to /mnt/koji). > > Also, the use of a file:// url doesn't have to mean nfs. You could > theoretically use another shared file system. So this is true, but misleading. The interaction of the topurl and pkgurl options in kojid is complicated. The topurl/topdir options determine how kojid will locate the repo. However, with the current code, the repodata will contain url references for the component rpms. That url is determined when the repo is generated. This happens during a createrepo task on a builder, and the pkgurl (not topurl) option is used. So.. - repodata location determined by topurl/topdir options - rpm location determined by pkgurl option on the builder that created the repo. I admit, this is a bit of a mess. From dan at danny.cz Thu Nov 6 10:55:22 2008 From: dan at danny.cz (Dan =?ISO-8859-1?Q?Hor=E1k?=) Date: Thu, 06 Nov 2008 11:55:22 +0100 Subject: caching packages on koji builder In-Reply-To: <4911FEB3.2000808@redhat.com> References: <1225878944.3544.27.camel@eagle.danny.cz> <1225899816.18425.8.camel@maunalani.home> <4911EA5C.4030406@redhat.com> <4911FEB3.2000808@redhat.com> Message-ID: <1225968922.3541.22.camel@eagle.danny.cz> Mike McLean p??e v St 05. 11. 2008 v 15:14 -0500: > Mike McLean wrote: > > This behavior is controlled by kojid options. If you specify the > > 'topurl' option for kojid, then the mock configs it generates will use > > an http:// url to point to the repo. Otherwise it will use a file:// url > > (using the value of the 'topdir' option, which defaults to /mnt/koji). > > > > Also, the use of a file:// url doesn't have to mean nfs. You could > > theoretically use another shared file system. > > So this is true, but misleading. The interaction of the topurl and > pkgurl options in kojid is complicated. > > The topurl/topdir options determine how kojid will locate the repo. > However, with the current code, the repodata will contain url references > for the component rpms. That url is determined when the repo is > generated. This happens during a createrepo task on a builder, and the > pkgurl (not topurl) option is used. > > So.. > - repodata location determined by topurl/topdir options > - rpm location determined by pkgurl option on the builder that created > the repo. > > I admit, this is a bit of a mess. And I am lost there :-) Squid runs, url_rewriter works and I am still unable to cache the packages. What value is placed into the mock generated yum config? Dan From mikem at redhat.com Thu Nov 6 14:28:23 2008 From: mikem at redhat.com (Mike McLean) Date: Thu, 06 Nov 2008 09:28:23 -0500 Subject: caching packages on koji builder In-Reply-To: <1225968922.3541.22.camel@eagle.danny.cz> References: <1225878944.3544.27.camel@eagle.danny.cz> <1225899816.18425.8.camel@maunalani.home> <4911EA5C.4030406@redhat.com> <4911FEB3.2000808@redhat.com> <1225968922.3541.22.camel@eagle.danny.cz> Message-ID: <4912FF07.3030702@redhat.com> Dan Hor?k wrote: > Mike McLean p??e v St 05. 11. 2008 v 15:14 -0500: >> So.. >> - repodata location determined by topurl/topdir options >> - rpm location determined by pkgurl option on the builder that created >> the repo. >> >> I admit, this is a bit of a mess. > > And I am lost there :-) > > Squid runs, url_rewriter works and I am still unable to cache the > packages. What value is placed into the mock generated yum config? Look at the repo in question, use zless to view primary.xml.gz. Search for 'base=' -- the value of this field is determined by the pkgurl setting of the builder that created the repo. Regardless of how yum gets the repo, once it sees this in the metadata, it will use this base url to download the rpms. You need to make sure that this hits your squid server instead of the remote http server. There are a couple ways you might do this. 1 - change the pkgurl setting on all your builders and regenerate all your repos. 2 - leave pkgurl as-is, but use an iptables redirect on your builders to map it to the local squid proxy. From tibbs at math.uh.edu Fri Nov 7 16:29:26 2008 From: tibbs at math.uh.edu (Jason L Tibbitts III) Date: 07 Nov 2008 10:29:26 -0600 Subject: Trying to figure out some umask issues Message-ID: I do a large number of local mock builds as a part of the package reviews that I do, and one problem I consistently run into is executables and .so files coming out with mode 775, but a scratch build in Fedora's koji instance showing the expected 755 permissions. I thought it might be some local environment leaking into my mock builds, but my personal umask is 022 which should be OK and changing it doesn't alter the result anyway. Init'ing a fresh chroot shows the default umask in it is 0002, which would seem to explain things but wouldn't explain why the Fedora buildsys has different results. Is there customization done there to force the umask somehow? - J< From mikem at redhat.com Fri Nov 7 16:37:24 2008 From: mikem at redhat.com (Mike McLean) Date: Fri, 07 Nov 2008 11:37:24 -0500 Subject: Trying to figure out some umask issues In-Reply-To: References: Message-ID: <49146EC4.3060608@redhat.com> Jason L Tibbitts III wrote: > I do a large number of local mock builds as a part of the package > reviews that I do, and one problem I consistently run into is > executables and .so files coming out with mode 775, but a scratch > build in Fedora's koji instance showing the expected 755 permissions. > > I thought it might be some local environment leaking into my mock > builds, but my personal umask is 022 which should be OK and changing > it doesn't alter the result anyway. > > Init'ing a fresh chroot shows the default umask in it is 0002, which > would seem to explain things but wouldn't explain why the Fedora > buildsys has different results. Is there customization done there to > force the umask somehow? I believe the umask needs to be 002 in order for users in the mock group to be able to use mock effectively. Regardless, if this is affecting your rpms, then your specs are probably broken. Use %defattr in all %files sections. From fbeuserie at gmail.com Mon Nov 10 00:31:13 2008 From: fbeuserie at gmail.com (Frederic Beuserie) Date: Mon, 10 Nov 2008 01:31:13 +0100 Subject: new koji instance: where is the Makefile.common ? Message-ID: <6c22b4810811091631x3403aeecmf67118c48b0cd47b@mail.gmail.com> Hi, just set up a new koji instance for our internal buildsystem (until now, we were using mdvsys/mock/createrepo and some control scripts). i've noticed the use of a Makefile.common in various documentation (http://fedoraproject.org/wiki/PackageMaintainers/UsingKoji, "Building with make targets") i would like to have a look on that. is there any url available ? i've looked at fedora-packager on fedorahosted but the file is not there. another question: i'm using right now a subversion instance for storing all the packages stuff (all sources rpms, sources files (specs, ...) and binary rpms). the layout it the one used by MDVSYS perl module. since i've quite a lot of data in this subversion repo, is there any chance to reuse the old layout with Koji ? is koji capable of working with various SCM layout ? thanks. frederic beuserie From dennis at ausil.us Mon Nov 10 01:53:14 2008 From: dennis at ausil.us (Dennis Gilmore) Date: Sun, 9 Nov 2008 19:53:14 -0600 Subject: new koji instance: where is the Makefile.common ? In-Reply-To: <6c22b4810811091631x3403aeecmf67118c48b0cd47b@mail.gmail.com> References: <6c22b4810811091631x3403aeecmf67118c48b0cd47b@mail.gmail.com> Message-ID: <200811091953.16514.dennis@ausil.us> On Sunday 09 November 2008 06:31:13 pm Frederic Beuserie wrote: > Hi, > > just set up a new koji instance for our internal buildsystem (until > now, we were using mdvsys/mock/createrepo and some control scripts). > > i've noticed the use of a Makefile.common in various documentation > (http://fedoraproject.org/wiki/PackageMaintainers/UsingKoji, > "Building with make targets") it lives in the common module of fedora cvs > i would like to have a look on that. is there any url available ? i've > looked at fedora-packager on fedorahosted but the file is not there. > > > another question: > > i'm using right now a subversion instance for storing all the packages > stuff (all sources rpms, sources files (specs, ...) and binary rpms). > the layout it the > one used by MDVSYS perl module. > > since i've quite a lot of data in this subversion repo, is there any > chance to reuse the old layout with Koji ? is koji capable of working > with various SCM layout ? as long as you pass it the right path/module and make srpm works it should work just fine. Dennis From tibbs at math.uh.edu Mon Nov 10 04:26:26 2008 From: tibbs at math.uh.edu (Jason L Tibbitts III) Date: 09 Nov 2008 22:26:26 -0600 Subject: Trying to figure out some umask issues In-Reply-To: <49146EC4.3060608@redhat.com> References: <49146EC4.3060608@redhat.com> Message-ID: >>>>> "MM" == Mike McLean writes: MM> I believe the umask needs to be 002 in order for users in the mock MM> group to be able to use mock effectively. Which umask? My person one? If mock requires a specific umask value, it should simply set one itself. MM> Regardless, if this is affecting your rpms, then your specs are MM> probably broken. Use %defattr in all %files sections. You'll note that I mentioned that I do many package reviews; these aren't my rpms. Also, %defattr is present in the packages I have seen to have this problem. (Fedora guidelines mandate it.) Is there some specific %defattr value you suggest? "(-.root,root,-)" is quite common. - J< From jkeating at redhat.com Mon Nov 10 07:48:45 2008 From: jkeating at redhat.com (Jesse Keating) Date: Sun, 09 Nov 2008 23:48:45 -0800 Subject: Trying to figure out some umask issues In-Reply-To: References: <49146EC4.3060608@redhat.com> Message-ID: <1226303325.13198.10.camel@luminos.localdomain> On Sun, 2008-11-09 at 22:26 -0600, Jason L Tibbitts III wrote: > "(-.root,root,-)" is quite I do believe that sets it to "whatever owner permissions the file has on the filesystem, root owner, root group, whatever group permissions it has on the filesystem" or something close to that effect. In other words, it's forcing ownership, but not permissions. -- Jesse Keating Fedora -- Freedom? is a feature! identi.ca: http://identi.ca/jkeating -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: From ivazqueznet at gmail.com Mon Nov 10 08:08:20 2008 From: ivazqueznet at gmail.com (Ignacio Vazquez-Abrams) Date: Mon, 10 Nov 2008 03:08:20 -0500 Subject: Trying to figure out some umask issues In-Reply-To: <1226303325.13198.10.camel@luminos.localdomain> References: <49146EC4.3060608@redhat.com> <1226303325.13198.10.camel@luminos.localdomain> Message-ID: <1226304500.16027.17.camel@ignacio.lan> On Sun, 2008-11-09 at 23:48 -0800, Jesse Keating wrote: > On Sun, 2008-11-09 at 22:26 -0600, Jason L Tibbitts III wrote: > > "(-.root,root,-)" is quite > > I do believe that sets it to "whatever owner permissions the file has on > the filesystem, root owner, root group, whatever group permissions it > has on the filesystem" or something close to that effect. In other > words, it's forcing ownership, but not permissions. (file permissions, user ownership, group ownership, directory permissions) -- Ignacio Vazquez-Abrams PLEASE don't CC me; I'm already subscribed -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: From jkeating at redhat.com Mon Nov 10 08:26:18 2008 From: jkeating at redhat.com (Jesse Keating) Date: Mon, 10 Nov 2008 00:26:18 -0800 Subject: Trying to figure out some umask issues In-Reply-To: <1226304500.16027.17.camel@ignacio.lan> References: <49146EC4.3060608@redhat.com> <1226303325.13198.10.camel@luminos.localdomain> <1226304500.16027.17.camel@ignacio.lan> Message-ID: <1226305578.13198.11.camel@luminos.localdomain> On Mon, 2008-11-10 at 03:08 -0500, Ignacio Vazquez-Abrams wrote: > (file permissions, user ownership, group ownership, directory > permissions) That's what I get for answering something after midnight without looking it up in the manual. Thanks! -- Jesse Keating Fedora -- Freedom? is a feature! identi.ca: http://identi.ca/jkeating -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: From tibbs at math.uh.edu Mon Nov 10 14:14:31 2008 From: tibbs at math.uh.edu (Jason L Tibbitts III) Date: 10 Nov 2008 08:14:31 -0600 Subject: Trying to figure out some umask issues In-Reply-To: <1226303325.13198.10.camel@luminos.localdomain> References: <49146EC4.3060608@redhat.com> <1226303325.13198.10.camel@luminos.localdomain> Message-ID: >>>>> "JK" == Jesse Keating writes: JK> I do believe that sets it to "whatever owner permissions the file JK> has on the filesystem, root owner, root group, whatever group JK> permissions it has on the filesystem" or something close to that JK> effect. Well, yes, but obviously (755,root,root,755) doesn't work all that well, because then all files are executable. (-,root,root,755) would be OK, I guess, but of course that wouldn't have any bearing on the problem I'm seeing. - J< From wcang at nav6.org Mon Nov 10 14:20:24 2008 From: wcang at nav6.org (=?UTF-8?B?IkFuZyBXYXkgQ2h1YW5nIDzmtKrkvJ/lo64+Ig==?=) Date: Mon, 10 Nov 2008 23:20:24 +0900 Subject: Minor bug in pungi Message-ID: <49184328.1070308@nav6.org> Hi all, The error message refers to wrong variable. Attached is the patch against the latest code from git. -------------- next part -------------- A non-text attachment was scrubbed... Name: variablefix.diff Type: text/x-diff Size: 556 bytes Desc: not available URL: From jkeating at redhat.com Mon Nov 10 16:47:18 2008 From: jkeating at redhat.com (Jesse Keating) Date: Mon, 10 Nov 2008 08:47:18 -0800 Subject: Trying to figure out some umask issues In-Reply-To: References: <49146EC4.3060608@redhat.com> <1226303325.13198.10.camel@luminos.localdomain> Message-ID: <1226335638.13198.13.camel@luminos.localdomain> On Mon, 2008-11-10 at 08:14 -0600, Jason L Tibbitts III wrote: > Well, yes, but obviously (755,root,root,755) doesn't work all that > well, because then all files are executable. (-,root,root,755) would > be OK, I guess, but of course that wouldn't have any bearing on the > problem I'm seeing. IIRC you can have multiple defatters that you can use per-file for ultimate control over the permissions/ownership of each individual file. Most packages don't do this, and instead set permissions at the %install stage, either by how they call /usr/bin/install or by settings in makefiles for make install. -- Jesse Keating Fedora -- Freedom? is a feature! identi.ca: http://identi.ca/jkeating -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: From mikem at redhat.com Mon Nov 10 18:11:30 2008 From: mikem at redhat.com (Mike McLean) Date: Mon, 10 Nov 2008 13:11:30 -0500 Subject: Trying to figure out some umask issues In-Reply-To: References: <49146EC4.3060608@redhat.com> Message-ID: <49187952.3050503@redhat.com> Jason L Tibbitts III wrote: >>>>>> "MM" == Mike McLean writes: > > MM> I believe the umask needs to be 002 in order for users in the mock > MM> group to be able to use mock effectively. > > Which umask? My person one? If mock requires a specific umask value, > it should simply set one itself. It does. mock calls os.umask(002) at startup (and has for a while). The mock on the Fedora build hosts has this line. I'm not sure why you're seeing a difference. Can you point to a specific Fedora build whose results differ from what you get locally? > MM> Regardless, if this is affecting your rpms, then your specs are > MM> probably broken. Use %defattr in all %files sections. > > You'll note that I mentioned that I do many package reviews; these > aren't my rpms. Also, %defattr is present in the packages I have seen > to have this problem. (Fedora guidelines mandate it.) Is there some > specific %defattr value you suggest? "(-.root,root,-)" is quite > common. Sorry, I guess it gets a bit more complicated with modes. Jesse's comments seem dead-on. If you don't want to patch the package to have it install with proper modes, then you can work around it with multiple %defattr() macros and/or per-file %attr() macros From tibbs at math.uh.edu Mon Nov 10 18:32:24 2008 From: tibbs at math.uh.edu (Jason L Tibbitts III) Date: 10 Nov 2008 12:32:24 -0600 Subject: Trying to figure out some umask issues In-Reply-To: <49187952.3050503@redhat.com> References: <49146EC4.3060608@redhat.com> <49187952.3050503@redhat.com> Message-ID: >>>>> "MM" == Mike McLean writes: MM> It does. mock calls os.umask(002) at startup (and has for a MM> while). The mock on the Fedora build hosts has this line. I'm not MM> sure why you're seeing a difference. Can you point to a specific MM> Fedora build whose results differ from what you get locally? Here's a package from a recent review: http://www.math.uh.edu/~tibbs/rpms/cave9-0.3-2.bog9.src.rpm When build locally, the included file /usr/bin/cave9 has mode 0775. When built in koji (http://koji.fedoraproject.org/koji/taskinfo?taskID=924911) the file has mode 0755. My local machine has mock-0.9.9-1.fc9.noarch. I am using the caching stuff, and my configuration files have been modified to point to local package mirrors and to set basedir to /mock which is a 10G tmpfs with the same permissions as /var/lib/mock. Those permissions happen to be 2775; that's probably coincidental but I guess you never know. - J< From jkeating at redhat.com Mon Nov 10 18:36:55 2008 From: jkeating at redhat.com (Jesse Keating) Date: Mon, 10 Nov 2008 10:36:55 -0800 Subject: Trying to figure out some umask issues In-Reply-To: References: <49146EC4.3060608@redhat.com> <49187952.3050503@redhat.com> Message-ID: <1226342215.12953.6.camel@luminos.localdomain> On Mon, 2008-11-10 at 12:32 -0600, Jason L Tibbitts III wrote: > Here's a package from a recent review: > http://www.math.uh.edu/~tibbs/rpms/cave9-0.3-2.bog9.src.rpm > > When build locally, the included file /usr/bin/cave9 has mode 0775. > When built in koji > (http://koji.fedoraproject.org/koji/taskinfo?taskID=924911) the file > has mode 0755. > > My local machine has mock-0.9.9-1.fc9.noarch. I am using the caching > stuff, and my configuration files have been modified to point to local > package mirrors and to set basedir to /mock which is a 10G tmpfs with > the same permissions as /var/lib/mock. Those permissions happen to be > 2775; that's probably coincidental but I guess you never know. I think the main point to take away from this is that relying on umask of systems to set the permissions of your files correctly is fragile at best, dangerous at worst. Umask can and does change from host to host so the build output is unreliable. Permissions in package builds should be set explicitly at either the %install phase or the %files phase. This likely needs a big sweeping cleanup action on our existing packages, but catching this on new packages is a start. -- Jesse Keating Fedora -- Freedom? is a feature! identi.ca: http://identi.ca/jkeating -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: From mikem at redhat.com Tue Nov 11 18:22:57 2008 From: mikem at redhat.com (Mike McLean) Date: Tue, 11 Nov 2008 13:22:57 -0500 Subject: [PATCH] don't break lines in logOutput (bz#467470) Message-ID: <4919CD81.7010901@redhat.com> Signed-off-by: Mike McLean --- py/mock/util.py | 12 ++++++++++-- 1 files changed, 10 insertions(+), 2 deletions(-) -------------- next part -------------- A non-text attachment was scrubbed... Name: 93642ddc7b9189a4deb4dcde06215fca636ca78d.diff Type: text/x-patch Size: 1248 bytes Desc: not available URL: From mikem at redhat.com Tue Nov 11 18:35:07 2008 From: mikem at redhat.com (Mike McLean) Date: Tue, 11 Nov 2008 13:35:07 -0500 Subject: [PATCH] don't break lines in logOutput (bz#467470) In-Reply-To: <4919CD81.7010901@redhat.com> References: <4919CD81.7010901@redhat.com> Message-ID: <4919D05B.9080707@redhat.com> Mike McLean wrote: > > Signed-off-by: Mike McLean > --- > py/mock/util.py | 12 ++++++++++-- > 1 files changed, 10 insertions(+), 2 deletions(-) Note, I removed the chomp() call since we're already splitting on '\n', hence none of the entries in the split can contain '\n'. We may want to remove the chomp function from util.py as this appears to have been its only use. Also, how would folks feel about removing the "if line == '': continue" bit? Empty lines in the output are still output after all. My main concern here is that the build.log produced by mock should be as accurate as possible. Context: https://bugzilla.redhat.com/show_bug.cgi?id=467470 https://fedorahosted.org/koji/ticket/113 From wacker at octothorp.org Wed Nov 26 00:54:59 2008 From: wacker at octothorp.org (William F. Acker WB2FLW +1 303 722 7209) Date: Tue, 25 Nov 2008 17:54:59 -0700 (MST) Subject: F10 KS file not in latest Pungi. Message-ID: Hi all, Now that Cambridge has gone gold, it seems that I shouldn't be using the rawhide KS file. pungi-2.0.8-1.fc10.noarch has a file for Rawhide and F9, but not F10. TIA. -- Bill in Denver From wacker at octothorp.org Wed Nov 26 01:00:01 2008 From: wacker at octothorp.org (William F. Acker WB2FLW +1 303 722 7209) Date: Tue, 25 Nov 2008 18:00:01 -0700 (MST) Subject: F10 KS file not in latest Pungi. In-Reply-To: References: Message-ID: On Tue, 25 Nov 2008, William F. Acker WB2FLW +1 303 722 7209 wrote: > Hi all, > > Now that Cambridge has gone gold, it seems that I shouldn't be using the > rawhide KS file. pungi-2.0.8-1.fc10.noarch has a file for Rawhide and F9, > but not F10. Maybe it's easier than I thought. I just did a diff on the f9 and rawhide files on my F9 system. The only differences are the repos and the comment at the top. I suppose someone should push a new Pungi for completeness, but I think I'm good to go. -- Bill in Denver From jkeating at redhat.com Wed Nov 26 01:26:59 2008 From: jkeating at redhat.com (Jesse Keating) Date: Tue, 25 Nov 2008 17:26:59 -0800 Subject: F10 KS file not in latest Pungi. In-Reply-To: References: Message-ID: <1227662819.5276.45.camel@luminos.localdomain> On Tue, 2008-11-25 at 18:00 -0700, William F. Acker WB2FLW +1 303 722 7209 wrote: > > Maybe it's easier than I thought. I just did a diff on the f9 and > rawhide files on my F9 system. The only differences are the repos and the > comment at the top. I suppose someone should push a new Pungi for > completeness, but I think I'm good to go. pungi won't be shipping configs for releases anymore, just a reference config for Rawhide. the spin-kickstarts package is where the configs used to produce Fedora will be. -- Jesse Keating Fedora -- Freedom? is a feature! identi.ca: http://identi.ca/jkeating -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: From tmz at pobox.com Thu Nov 27 02:16:31 2008 From: tmz at pobox.com (Todd Zullinger) Date: Wed, 26 Nov 2008 21:16:31 -0500 Subject: [PATCH/mock] Don't disable ccache on epel-5, it is available Message-ID: <20081127021631.GB20204@inocybe.teonanacatl.org> Signed-off-by: Todd Zullinger --- I've wondered why ccache is disabled in the epel-5 mock configs. The ccache package is available in epel 5 now, so it seems reasonable to leave it enabled. Unless I'm missing something obvious, that is. ;) etc/mock/epel-5-i386.cfg | 3 --- etc/mock/epel-5-ppc.cfg | 3 --- etc/mock/epel-5-x86_64.cfg | 3 --- 3 files changed, 0 insertions(+), 9 deletions(-) diff --git a/etc/mock/epel-5-i386.cfg b/etc/mock/epel-5-i386.cfg index 30c66ee..9b9b6ef 100644 --- a/etc/mock/epel-5-i386.cfg +++ b/etc/mock/epel-5-i386.cfg @@ -3,9 +3,6 @@ config_opts['target_arch'] = 'i386' config_opts['chroot_setup_cmd'] = 'install buildsys-build' config_opts['dist'] = 'el5' # only useful for --resultdir variable subst -# ccache not available on epel5 -config_opts['plugin_conf']['ccache_enable'] = False - config_opts['yum.conf'] = """ [main] cachedir=/var/cache/yum diff --git a/etc/mock/epel-5-ppc.cfg b/etc/mock/epel-5-ppc.cfg index b07d136..1bd72a7 100644 --- a/etc/mock/epel-5-ppc.cfg +++ b/etc/mock/epel-5-ppc.cfg @@ -3,9 +3,6 @@ config_opts['target_arch'] = 'ppc' config_opts['chroot_setup_cmd'] = 'install buildsys-build' config_opts['dist'] = 'el5' # only useful for --resultdir variable subst -# ccache not available on epel5 -config_opts['plugin_conf']['ccache_enable'] = False - config_opts['yum.conf'] = """ [main] cachedir=/var/cache/yum diff --git a/etc/mock/epel-5-x86_64.cfg b/etc/mock/epel-5-x86_64.cfg index d0a8cde..0fe89d4 100644 --- a/etc/mock/epel-5-x86_64.cfg +++ b/etc/mock/epel-5-x86_64.cfg @@ -3,9 +3,6 @@ config_opts['target_arch'] = 'x86_64' config_opts['chroot_setup_cmd'] = 'install buildsys-build' config_opts['dist'] = 'el5' # only useful for --resultdir variable subst -# ccache not available on epel5 -config_opts['plugin_conf']['ccache_enable'] = False - config_opts['yum.conf'] = """ [main] cachedir=/var/cache/yum -- 1.6.0.4 -- Todd OpenPGP -> KeyID: 0xBEAF0CE3 | URL: www.pobox.com/~tmz/pgp ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Lazlo's Chinese Relativity Axiom: No matter how great your triumphs or how tragic your defeats -- approximately one billion Chinese couldn't care less. From chenbaozi at gmail.com Sat Nov 29 03:07:10 2008 From: chenbaozi at gmail.com (=?UTF-8?B?6ZmI6bKN5a2c?=) Date: Sat, 29 Nov 2008 11:07:10 +0800 Subject: confused about the relationship between koji and pungi Message-ID: hello, i'm a novice for system building and have read some articles about pungi and koji recently. but i'm a little confused about the relationship between the tools. in my opinion, pungi is focusing on building iso while koji is focusing on building packages. and i'm wondering if the koji has got some similar functions as pungi does, or is it simply that we use koji to build packages and then make tree/iso by pungi? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ivazqueznet at gmail.com Sat Nov 29 03:17:30 2008 From: ivazqueznet at gmail.com (Ignacio Vazquez-Abrams) Date: Fri, 28 Nov 2008 22:17:30 -0500 Subject: confused about the relationship between koji and pungi In-Reply-To: References: Message-ID: <1227928650.3835.49.camel@ignacio.lan> On Sat, 2008-11-29 at 11:07 +0800, ??? wrote: > hello, > i'm a novice for system building and have read some articles about > pungi and koji recently. but i'm a little confused about the > relationship between the tools. in my opinion, pungi is focusing on > building iso while koji is focusing on building packages. and i'm > wondering if the koji has got some similar functions as pungi does, > or is it simply that we use koji to build packages and > then make tree/iso by pungi? The link that you're looking for is mash. -- Ignacio Vazquez-Abrams PLEASE don't CC me; I'm already subscribed -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: From chenbaozi at gmail.com Sun Nov 30 02:50:19 2008 From: chenbaozi at gmail.com (=?UTF-8?B?6ZmI6bKN5a2c?=) Date: Sun, 30 Nov 2008 10:50:19 +0800 Subject: confused about the relationship between koji and pungi (Ignacio Vazquez-Abrams) Message-ID: >> hello, >> i'm a novice for system building and have read some articles about >> pungi and koji recently. but i'm a little confused about the >> relationship between the tools. in my opinion, pungi is focusing on >> building iso while koji is focusing on building packages. and i'm >> wondering if the koji has got some similar functions as pungi does, >> or is it simply that we use koji to build packages and >> then make tree/iso by pungi? > The link that you're looking for is mash. Thank you. So, what do they actually work for? Or in other way, what workflow should we follow when building the distro with koji and pungi? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jkeating at redhat.com Sun Nov 30 03:04:13 2008 From: jkeating at redhat.com (Jesse Keating) Date: Sat, 29 Nov 2008 19:04:13 -0800 Subject: confused about the relationship between koji and pungi (Ignacio Vazquez-Abrams) In-Reply-To: References: Message-ID: <1228014253.3706.9.camel@localhost.localdomain> On Sun, 2008-11-30 at 10:50 +0800, ??? wrote: > Thank you. > So, what do they actually work for? > Or in other way, what workflow should we follow when building the distro > with koji and pungi? Koji is used to build the rpms, mash is used to pull the rpms out of koji and create yum repositories out of them. Pungi uses the yum repositories to make installable trees. -- Jesse Keating Fedora -- Freedom? is a feature! identi.ca: http://identi.ca/jkeating -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: From bryce at zeniv.linux.org.uk Sun Nov 30 10:41:20 2008 From: bryce at zeniv.linux.org.uk (Bryce) Date: Sun, 30 Nov 2008 10:41:20 +0000 Subject: Cockroach task Message-ID: <49326DD0.20009@zeniv.linux.org.uk> (refers to koji-1.2.6-1) -scratch head- Koji seems to have derailed itself and created an unkillable task Although it has built the rpms, it then seems to have done something odd to itself because it's entered a failed state but the koji system still believes that it is still building the package State failed ... Result RetryError: unable to retry call 7 (method host.importChangelog) for session 18591 I've tried koji cancel, and a few of the less savory koji call api commands but it refuses to accept that the task is dead Ideas? I assume that for some reason it's been unable to communicate with the postgres database but I don't see a means to tell it to retry or just plain abandon the whole task Phil =--=