From tmz at pobox.com Thu Jul 10 02:01:50 2008 From: tmz at pobox.com (Todd Zullinger) Date: Wed, 9 Jul 2008 22:01:50 -0400 Subject: DIST_DEFINES duplicated in Makefile.common? Message-ID: <20080710020150.GN7900@inocybe.teonanacatl.org> Greetings, It seems that DIST_DEFINES is duplicated in Makefile.common, being included in RPM_DEFINES and then also being used in places where RPM_DEFINES is used (directly, and as a part of RPM_WITH_DIRS). Does the attached patch look reasonably sane? Disclaimer: I've had a few rum-based beverages this evening and I might be missing some important reason for the duplication. -- Todd OpenPGP -> KeyID: 0xBEAF0CE3 | URL: www.pobox.com/~tmz/pgp ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Religion. A daughter of Hope and Fear, explaining to Ignorance the nature of the Unknowable. -- Ambrose Bierce, The Enlarged Devil's Dictionary, 1906 -------------- next part -------------- Index: Makefile.common =================================================================== RCS file: /cvs/extras/common/Makefile.common,v retrieving revision 1.105 diff -u -p -r1.105 Makefile.common --- Makefile.common 3 Jul 2008 19:15:27 -0000 1.105 +++ Makefile.common 10 Jul 2008 01:47:05 -0000 @@ -94,11 +94,11 @@ ifndef NAME $(error "You can not run this Makefile without having NAME defined") endif ifndef VERSION -VERSION := $(shell rpm $(RPM_DEFINES) $(DIST_DEFINES) -q --qf "%{VERSION}\n" --specfile $(SPECFILE)| head -1) +VERSION := $(shell rpm $(RPM_DEFINES) -q --qf "%{VERSION}\n" --specfile $(SPECFILE)| head -1) endif # the release of the package ifndef RELEASE -RELEASE := $(shell rpm $(RPM_DEFINES) $(DIST_DEFINES) -q --qf "%{RELEASE}\n" --specfile $(SPECFILE)| head -1) +RELEASE := $(shell rpm $(RPM_DEFINES) -q --qf "%{RELEASE}\n" --specfile $(SPECFILE)| head -1) endif # this is used in make patch, maybe make clean eventually. # would be nicer to autodetermine from the spec file... @@ -382,7 +382,7 @@ check: test-srpm ## use this to build an srpm locally srpm: sources $(TARGETS) - $(RPM_WITH_DIRS) $(DIST_DEFINES) --nodeps -bs $(SPECFILE) + $(RPM_WITH_DIRS) --nodeps -bs $(SPECFILE) test-srpm: srpm -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 542 bytes Desc: not available URL: From mikeb at redhat.com Thu Jul 10 16:02:31 2008 From: mikeb at redhat.com (Mike Bonnet) Date: Thu, 10 Jul 2008 12:02:31 -0400 Subject: Supporting EPEL Builds in Koji Message-ID: <1215705751.30991.7.camel@burren.bos.redhat.com> Hi. I've written up a proposal for a way to support EPEL builds in Koji. It's not the only way we could do this, but I think it's doable with a reasonable amount of effort, and has the side-effect of greatly simplifying the Koji setup process for a lot of people (by removing the need to bootstrap/import an entire distro of packages into your private Koji instance). You can view the proposal here: http://fedoraproject.org/wiki/Koji/EPELSupport It's fairly detailed regarding the data model changes necessary, so if you're not familiar with the Koji codebase you can skip those parts. Questions and comments welcome. Thanks, Mike From kanarip at kanarip.com Thu Jul 10 17:12:16 2008 From: kanarip at kanarip.com (Jeroen van Meeuwen) Date: Thu, 10 Jul 2008 19:12:16 +0200 Subject: Supporting EPEL Builds in Koji In-Reply-To: <1215705751.30991.7.camel@burren.bos.redhat.com> References: <1215705751.30991.7.camel@burren.bos.redhat.com> Message-ID: <487642F0.9040901@kanarip.com> Mike Bonnet wrote: > Hi. I've written up a proposal for a way to support EPEL builds in > Koji. It's not the only way we could do this, but I think it's doable > with a reasonable amount of effort, and has the side-effect of greatly > simplifying the Koji setup process for a lot of people (by removing the > need to bootstrap/import an entire distro of packages into your private > Koji instance). You can view the proposal here: > > http://fedoraproject.org/wiki/Koji/EPELSupport > > It's fairly detailed regarding the data model changes necessary, so if > you're not familiar with the Koji codebase you can skip those parts. > Questions and comments welcome. > Hi Mike, good to see you've spend some time on this whereas I have been lazy in Littleton (holiday). I'd like to share a few thoughts on the Wiki page -which is a great start; From the Wiki page: "There is a strong feeling that if a package exists in the Koji-managed local repo (whose contents the Koji admin has full control over) it should always be preferred over the external repo (whose contents the Koji admin may have little or no control over)." The preference koji will have (in using which package in the buildroot), might introduce the problem where customly built package foo-1.0 is used in the buildroot, and upstream updates to foo-1.1 - the running nodes would update to foo-1.1 whereas the buildroot still uses the custom foo-1.0... The point being, that these updates have to managed as they are released. The updates need to managed on the side where said packages are being mashed into a repository (infra side) or applied (client side). You can see the duplicate effort when the updates are managed on either side (infra or client), _and_ in koji, separately. I would like to suggest the koji development team makes the priority setting koji is going to use a configurable item -which in compared to the bigger picture isn't all that much a priority, just something to think about. Additionally, I'd like to comment on / ask about the proposed database changes for the tag_config table; In an attempt to show you what I was thinking, here's a number of questions; From the Wiki page: "At repo creation time, the repodata will be retrieved from the processed url and merged with the local repodata as described above. This single repo will then be used for subsequent builds against the tag" Do I understand correctly one can only give one single repository URL to a certain tag? Does this mean that a tag is created for (example) "dist-el5" with a remote repository URL, and then "dist-el5-updates" with another remote repository URL? This means for the build target used to have dist-el5-updates inherit dist-el5, right? Which then implies either metadata needs to be imported for dist-el5-updates or inheritance can only be applied during build-time... right? The question I guess is basically; how does koji handle tags with a combination of remote urls & inheritance? From the Wiki page: "Right now that (rpminfo) table enforces uniqueness of (name, version, release, arch)." I see that koji does not store complete package nevra which may become a problem in case duplicate nvra occur (which is very much likely the case where rebuilding packages with the release number bumped might collide with upstream doing a release bump -which is where the epoch is often used as upstream has clear guidelines for epoch bumps which -hopefully- make them occur in special circumstances only and thus very much reduces the chance of a colliding nevra). I like the proposed uniqueness of NVRA-namespaces as well, don't get me wrong ;-) The other thing (and probably the last thing for now) I'd like to share is that, for reproducibility purposes, how viable would it be to have koji automatically import the remote RPM (the file and all the data) as it is used from the remote repository? This may or may not be a configurable option, saves work for admins compared to the situation now, and preserves reproducibility under all circumstances, adding the automatically imported RPM to the appropriate tags, storing them for reproducibility whereas upstream only keeps two versions in the repository... Though I understand it 1) consumes space and 2) isn't helpful for the EPEL case, I think this is particularly useful for long-term supported appliance software. Just wondering here ;-) Let me know what you think, Kind regards, Jeroen van Meeuwen -kanarip From kanarip at kanarip.com Thu Jul 10 17:18:13 2008 From: kanarip at kanarip.com (Jeroen van Meeuwen) Date: Thu, 10 Jul 2008 19:18:13 +0200 Subject: Supporting EPEL Builds in Koji In-Reply-To: <487642F0.9040901@kanarip.com> References: <1215705751.30991.7.camel@burren.bos.redhat.com> <487642F0.9040901@kanarip.com> Message-ID: <48764455.9090808@kanarip.com> Jeroen van Meeuwen wrote: > I'd like to share a few thoughts on the Wiki page -which is a great start; > (...) Did I mention my primary concern with aforementioned questions are more related to "make-your-own" private koji instances rather then the one that is going to build EPEL? Sorry for any confusion. Kind regards, Jeroen van Meeuwen -kanarip From mikeb at redhat.com Thu Jul 10 19:49:00 2008 From: mikeb at redhat.com (Mike Bonnet) Date: Thu, 10 Jul 2008 15:49:00 -0400 Subject: Supporting EPEL Builds in Koji In-Reply-To: <487642F0.9040901@kanarip.com> References: <1215705751.30991.7.camel@burren.bos.redhat.com> <487642F0.9040901@kanarip.com> Message-ID: <1215719341.30991.53.camel@burren.bos.redhat.com> On Thu, 2008-07-10 at 19:12 +0200, Jeroen van Meeuwen wrote: > Mike Bonnet wrote: > > Hi. I've written up a proposal for a way to support EPEL builds in > > Koji. It's not the only way we could do this, but I think it's doable > > with a reasonable amount of effort, and has the side-effect of greatly > > simplifying the Koji setup process for a lot of people (by removing the > > need to bootstrap/import an entire distro of packages into your private > > Koji instance). You can view the proposal here: > > > > http://fedoraproject.org/wiki/Koji/EPELSupport > > > > It's fairly detailed regarding the data model changes necessary, so if > > you're not familiar with the Koji codebase you can skip those parts. > > Questions and comments welcome. > > > > Hi Mike, > > good to see you've spend some time on this whereas I have been lazy in > Littleton (holiday). > > I'd like to share a few thoughts on the Wiki page -which is a great start; > > From the Wiki page: "There is a strong feeling that if a package exists > in the Koji-managed local repo (whose contents the Koji admin has full > control over) it should always be preferred over the external repo > (whose contents the Koji admin may have little or no control over)." > > The preference koji will have (in using which package in the buildroot), > might introduce the problem where customly built package foo-1.0 is used > in the buildroot, and upstream updates to foo-1.1 - the running nodes > would update to foo-1.1 whereas the buildroot still uses the custom > foo-1.0... Yes, it's up to the Koji admin to monitor the remote repo, and take appropriate action when their custom local packages are superseded by packages in the remote repo. That may be untagging or blocking the package locally so the newer version can be pulled down from the remote repo. Or it may be rebuilding the custom package based on the updated sources. The point is that the build environment doesn't change unless the Koji admin takes some action to change it. > The point being, that these updates have to managed as they are > released. The updates need to managed on the side where said packages > are being mashed into a repository (infra side) or applied (client side). > > You can see the duplicate effort when the updates are managed on either > side (infra or client), _and_ in koji, separately. There is duplicate effort either way. The difference is that, if highest-nvr-wins is used, and a remote repo updates to a later version of a package that you have a custom build of, there is *no way* for you to revert your build environment to that lower-nvr version without bumping your version higher than their version (without actually changing the source at all) and rebuilding. It encourages this Cold War arms-race of version numbers between your custom packages and the remote repo's packages, and results in the admin having to fake higher version numbers and rebuild constantly *without any source changes* just to keep their custom packages in their build environment. Alternately, if first-match-wins is used (where the first repo is the locally-managed Koji repo), and a remote repo updates to a later version of a package you have a custom version of, nothing happens to your build environment. If you decide you want the newer version from the remote repo, you untag your local package and let it get pulled in from the remote repo. If that newer version has problems, retag your custom version and it will then be available in the build environment again. There is no unnecessary building of packages, no faking version numbers, and no unexpected changes to your build environment. It's the "principle of least surprise", which is why I think it's the right policy to use in a managed build environment like Koji. > I would like to suggest the koji development team makes the priority > setting koji is going to use a configurable item -which in compared to > the bigger picture isn't all that much a priority, just something to > think about. I strongly feel that this isn't something that needs to be configurable, and that first-match-wins is the correct behavior. But if other people agree that there is a valid use-case for making it configurable, and Seth and/or James can make the logic in repomerge configurable, then we can add switch for it to Koji. > Additionally, I'd like to comment on / ask about the proposed database > changes for the tag_config table; In an attempt to show you what I was > thinking, here's a number of questions; > > From the Wiki page: "At repo creation time, the repodata will be > retrieved from the processed url and merged with the local repodata as > described above. This single repo will then be used for subsequent > builds against the tag" > > Do I understand correctly one can only give one single repository URL to > a certain tag? Does this mean that a tag is created for (example) > "dist-el5" with a remote repository URL, and then "dist-el5-updates" > with another remote repository URL? This means for the build target used > to have dist-el5-updates inherit dist-el5, right? Which then implies > either metadata needs to be imported for dist-el5-updates or inheritance > can only be applied during build-time... right? > > The question I guess is basically; how does koji handle tags with a > combination of remote urls & inheritance? Originally you were correct, the proposal only allowed for a single remote repo to be configured. This was mandated by the desire to track packages back to their repository of origin, and the lack of repository data in the rpmdb. jkeating convinced me that this wasn't a very useful implementation, and suggested that we could get information about the origin of a given rpm from the baseurl in the repodata. I've updated the wiki page with a new implementation proposal that will allow for multiple remote repos while still tracking package origin, and specifies how remote repos will interact with the tag inheritance tree. Please take a look and let me know what you think. > From the Wiki page: "Right now that (rpminfo) table enforces uniqueness > of (name, version, release, arch)." > > I see that koji does not store complete package nevra which may become a > problem in case duplicate nvra occur (which is very much likely the case > where rebuilding packages with the release number bumped might collide > with upstream doing a release bump -which is where the epoch is often > used as upstream has clear guidelines for epoch bumps which -hopefully- > make them occur in special circumstances only and thus very much reduces > the chance of a colliding nevra). I like the proposed uniqueness of > NVRA-namespaces as well, don't get me wrong ;-) Koji intentionally ignores epoch when enforcing uniqueness. For better or worse, the epoch is mostly hidden from users, and does not show up in the filename. Having packages with the same NVRA but different epochs was considered harmful when Koji was being designed, and it will prevent this from happening. Note that Koji does *store* the epoch, it just doesn't use it when enforcing uniqueness. In the proposal, local packages exist in one NVRA namespace, and each remote repo (differentiated by URL) exists in a different NVRA namespace. So NVRA much be unique within each repo (local or remote) but not across repos. So NVRA collisions between your local Koji instance and a remote repo will not cause problems at the data model level. Which package gets selected and made available in the buildroots will be handled by the (possibly configurable) package selection policy of createrepo/mergerepo. > The other thing (and probably the last thing for now) I'd like to share > is that, for reproducibility purposes, how viable would it be to have > koji automatically import the remote RPM (the file and all the data) as > it is used from the remote repository? This may or may not be a > configurable option, saves work for admins compared to the situation > now, and preserves reproducibility under all circumstances, adding the > automatically imported RPM to the appropriate tags, storing them for > reproducibility whereas upstream only keeps two versions in the > repository... Though I understand it 1) consumes space and 2) isn't > helpful for the EPEL case, I think this is particularly useful for > long-term supported appliance software. Just wondering here ;-) This sounds much more like the secondary-arch approach, and is separate from what we're trying to accomplish here. I had requested that the secondary-arch daemon support a "same-arch-downstream" mode where it would download and import (rather than rebuild) builds from an upstream Koji as they were completed. However, this is a lot more complicated and requires more detailed policy. If this is a requirement for you, I suggest you take a look at the secondary-arch work. From kevin.linul at gmail.com Wed Jul 16 03:06:44 2008 From: kevin.linul at gmail.com (Linul) Date: Wed, 16 Jul 2008 11:06:44 +0800 Subject: Koji CLI Auth problem Message-ID: <5b123d6c0807152006r25921bfet5700029ae6b98d25@mail.gmail.com> HI: I'm using CentOS 5.2 for my Koji Server, but now I have a problem about Koji CLI auth. According the wiki document in http://fedoraproject.org/wiki/Koji/ServerHowTo , I setup my Koji-hub?Koji-web?postgresql , and have a koji web interface. I also setup my CA Center?and configure the kojiweb.conf?kojihub.conf?/etc/koji.conf. But when i execute the koji command with no username and password, the messages is? Error: [('PEM routines', 'PEM_read_bio', 'no start line'), ('SSL routines', 'SSL_CTX_use_PrivateKey_file', 'PEM lib')] why? thanks..... /etc/koji.conf? [koji] ;configuration for koji cli tool ;url of XMLRPC server ;server = http://koji.fedoraproject.org/kojihub server = http://koji.ossii.com.tw/kojihub ;url of web interface ;weburl = http://koji.fedoraproject.org/koji weburl = http://koji.ossii.com.tw/koji ;url of package download site ;pkgurl = http://koji.fedoraproject.org/packages pkgurl = http://koji.ossii.com.tw/packages ;path to the koji top directory topdir = /mnt/koji ;configuration for SSL athentication ;client certificate ;cert = ~/.fedora.cert cert = /etc/kojid/kojiadmin.crt ;certificate of the CA that issued the client certificate ;ca = ~/.fedora-upload-ca.cert ca = /etc/kojid/kojiadmin.key ;certificate of the CA that issued the HTTP server certificate ;serverca = ~/.fedora-server-ca.cert serverca = /etc/httpd/conf.d/ssl/ossiikojica.crt kojihub.conf? SetHandler mod_python PythonHandler kojixmlrpc PythonOption DBName koji PythonOption DBUser kevin PythonOption DBHost 127.0.0.1 PythonOption KojiDir /mnt/koji # Kerberos auth configuration # PythonOption AuthPrincipal kojihub at EXAMPLE.COM # PythonOption AuthKeytab /etc/koji.keytab # PythonOption ProxyPrincipals kojihub at EXAMPLE.COM # format string for host principals (%s = hostname) # PythonOption HostPrincipalFormat compile/%s at EXAMPLE.COM # end Kerberos auth configuration # SSL client certificate auth configuration # the client username is the common name of the subject of their client certificate PythonOption DNUsernameComponent CN # separate multiple DNs with | # PythonOption ProxyDNs "/C=US/ST=Massachusetts/O=Example Org/OU=Example User/CN=example/emailAddress=example at example.com" PythonOption ProxyDNs "/C=TW/ST=Taiwan/O=OSSII/OU=Koji Hub Server/CN=OSSII Koji Server CA/emailAddress=kevin.lin at ossii.com.tw" # end SSL client certificate auth configuration PythonOption LoginCreatesUser On PythonOption KojiWebURL http://koji.ossii.com.tw/koji # The domain name that will be appended to Koji usernames # when creating email notifications PythonOption EmailDomain example.com # PythonOption KojiDebug On # PythonOption KojiTraceback "extended" # sending tracebacks to the client isn't very helpful for debugging xmlrpc PythonDebug Off # autoreload is mostly useless to us (it would only reload kojixmlrpc.py) PythonAutoReload Off # uncomment this to enable authentication via SSL client certificates SSLOptions +StdEnvVars # these options must be enabled globally (in ssl.conf) SSLVerifyClient require SSLVerifyDepth 10 kojiweb.conf? Alias /koji "/usr/share/koji-web/scripts/" # Config for the publisher handler SetHandler mod_python PythonHandler mod_python.publisher # General settings PythonDebug On PythonOption KojiHubURL http://koji.ossii.com.tw/kojihub PythonOption KojiWebURL http://koji.ossii.com.tw/koji PythonOption KojiPackagesURL http://koji.ossii.com.tw/koji/packages PythonOption WebPrincipal koji/kevin.lin at ossii.com.tw PythonOption WebKeytab /etc/httpd.keytab PythonOption WebCCache /var/tmp/kojiweb.ccache PythonOption WebCert /etc/httpd/conf.d/ssl/kojiweb.crt PythonOption ClientCA /etc/httpd/conf.d/ssl/kojiweb.key PythonOption KojiHubCA /etc/httpd/conf.d/ssl/ossiikojica.crt PythonOption LoginTimeout 72 # This must be changed before deployment PythonOption Secret CHANGE_ME PythonPath "sys.path + ['/usr/share/koji-web/lib']" PythonCleanupHandler kojiweb.handlers::cleanup PythonAutoReload Off SSLOptions +StdEnvVars # these options must be enabled globally (in ssl.conf) SSLVerifyClient require SSLVerifyDepth 10 Alias /koji-static/ "/usr/share/koji-web/static/" Options None AllowOverride None Order allow,deny Allow from all ssl.conf SSLCertificateFile /etc/httpd/conf.d/ssl/kojihub.crt SSLCertificateKeyFile /etc/httpd/conf.d/ssl/kojihub.key SSLCACertificateFile /etc/httpd/conf.d/ssl/ossiikojica.crt SSLVerifyClient require SSLVerifyDepth 10 -- ============================================================================= ??? Linul RedHat Certified Engineer TsLG??????http://www.tslg.idv.tw TsLG?????http://blog.tslg.idv.tw Linul?????http://photo.tslg.idv.tw ???0939797462 E-mail : kevin.linul at gmail.com; linul at tslg.idv.tw ============================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikeb at redhat.com Wed Jul 16 13:29:10 2008 From: mikeb at redhat.com (Mike Bonnet) Date: Wed, 16 Jul 2008 09:29:10 -0400 Subject: Koji CLI Auth problem In-Reply-To: <5b123d6c0807152006r25921bfet5700029ae6b98d25@mail.gmail.com> References: <5b123d6c0807152006r25921bfet5700029ae6b98d25@mail.gmail.com> Message-ID: <1216214950.3599.3.camel@localhost.localdomain> On Wed, 2008-07-16 at 11:06 +0800, Linul wrote: > HI: > > I'm using CentOS 5.2 for my Koji Server, but now I have a problem > about Koji CLI auth. > > According the wiki document in > http://fedoraproject.org/wiki/Koji/ServerHowTo , I setup my Koji-hub? > Koji-web?postgresql > > , and have a koji web interface. > > I also setup my CA Center?and configure the kojiweb.conf? > kojihub.conf?/etc/koji.conf. > > But when i execute the koji command with no username and password, the > messages is? > > Error: [('PEM routines', 'PEM_read_bio', 'no start line'), ('SSL > routines', 'SSL_CTX_use_PrivateKey_file', 'PEM lib')] Your client certificate file (indicated by "cert" in the config file) needs to contain both the certificate and private key. Your private key is missing. > why? > > thanks..... > > > /etc/koji.conf? > > [koji] > > ;configuration for koji cli tool > > ;url of XMLRPC server > ;server = http://koji.fedoraproject.org/kojihub > server = http://koji.ossii.com.tw/kojihub > > ;url of web interface > ;weburl = http://koji.fedoraproject.org/koji > weburl = http://koji.ossii.com.tw/koji > > ;url of package download site > ;pkgurl = http://koji.fedoraproject.org/packages > pkgurl = http://koji.ossii.com.tw/packages > > ;path to the koji top directory > topdir = /mnt/koji > > ;configuration for SSL athentication > > ;client certificate > ;cert = ~/.fedora.cert > cert = /etc/kojid/kojiadmin.crt > > ;certificate of the CA that issued the client certificate > ;ca = ~/.fedora-upload-ca.cert > ca = /etc/kojid/kojiadmin.key > > ;certificate of the CA that issued the HTTP server certificate > ;serverca = ~/.fedora-server-ca.cert > serverca = /etc/httpd/conf.d/ssl/ossiikojica.crt > > > kojihub.conf? > > > SetHandler mod_python > PythonHandler kojixmlrpc > PythonOption DBName koji > PythonOption DBUser kevin > PythonOption DBHost 127.0.0.1 > PythonOption KojiDir /mnt/koji > > # Kerberos auth configuration > # PythonOption AuthPrincipal kojihub at EXAMPLE.COM > # PythonOption AuthKeytab /etc/koji.keytab > # PythonOption ProxyPrincipals kojihub at EXAMPLE.COM > # format string for host principals (%s = hostname) > # PythonOption HostPrincipalFormat compile/%s at EXAMPLE.COM > # end Kerberos auth configuration > > # SSL client certificate auth configuration > # the client username is the common name of the subject of > their client certificate > PythonOption DNUsernameComponent CN > # separate multiple DNs with | > # PythonOption ProxyDNs "/C=US/ST=Massachusetts/O=Example > Org/OU=Example User/CN=example/emailAddress=example at example.com" > PythonOption ProxyDNs "/C=TW/ST=Taiwan/O=OSSII/OU=Koji Hub > Server/CN=OSSII Koji Server CA/emailAddress=kevin.lin at ossii.com.tw" > # end SSL client certificate auth configuration > > PythonOption LoginCreatesUser On > PythonOption KojiWebURL http://koji.ossii.com.tw/koji > > # The domain name that will be appended to Koji usernames > # when creating email notifications > PythonOption EmailDomain example.com > # PythonOption KojiDebug On > # PythonOption KojiTraceback "extended" > # sending tracebacks to the client isn't very helpful for > debugging xmlrpc > PythonDebug Off > # autoreload is mostly useless to us (it would only reload > kojixmlrpc.py) > PythonAutoReload Off > > > # uncomment this to enable authentication via SSL client certificates > > SSLOptions +StdEnvVars > > # these options must be enabled globally (in ssl.conf) > SSLVerifyClient require > SSLVerifyDepth 10 > > kojiweb.conf? > > Alias /koji "/usr/share/koji-web/scripts/" > > > # Config for the publisher handler > SetHandler mod_python > PythonHandler mod_python.publisher > > # General settings > PythonDebug On > PythonOption KojiHubURL http://koji.ossii.com.tw/kojihub > PythonOption KojiWebURL http://koji.ossii.com.tw/koji > PythonOption KojiPackagesURL > http://koji.ossii.com.tw/koji/packages > PythonOption WebPrincipal koji/kevin.lin at ossii.com.tw > PythonOption WebKeytab /etc/httpd.keytab > PythonOption WebCCache /var/tmp/kojiweb.ccache > PythonOption WebCert /etc/httpd/conf.d/ssl/kojiweb.crt > PythonOption ClientCA /etc/httpd/conf.d/ssl/kojiweb.key > PythonOption KojiHubCA /etc/httpd/conf.d/ssl/ossiikojica.crt > PythonOption LoginTimeout 72 > # This must be changed before deployment > PythonOption Secret CHANGE_ME > PythonPath "sys.path + ['/usr/share/koji-web/lib']" > PythonCleanupHandler kojiweb.handlers::cleanup > PythonAutoReload Off > > > SSLOptions +StdEnvVars > > # these options must be enabled globally (in ssl.conf) > SSLVerifyClient require > SSLVerifyDepth 10 > > Alias /koji-static/ "/usr/share/koji-web/static/" > > > Options None > AllowOverride None > Order allow,deny > Allow from all > > > ssl.conf > > SSLCertificateFile /etc/httpd/conf.d/ssl/kojihub.crt > SSLCertificateKeyFile /etc/httpd/conf.d/ssl/kojihub.key > SSLCACertificateFile /etc/httpd/conf.d/ssl/ossiikojica.crt > SSLVerifyClient require > SSLVerifyDepth 10 > > > > > -- > ============================================================================= > ??? > Linul > RedHat Certified Engineer > > TsLG??????http://www.tslg.idv.tw > TsLG?????http://blog.tslg.idv.tw > Linul?????http://photo.tslg.idv.tw > ???0939797462 > E-mail : kevin.linul at gmail.com; linul at tslg.idv.tw > ============================================================================= > -- > Fedora-buildsys-list mailing list > Fedora-buildsys-list at redhat.com > https://www.redhat.com/mailman/listinfo/fedora-buildsys-list From ngavini at jla.rutgers.edu Wed Jul 16 20:53:03 2008 From: ngavini at jla.rutgers.edu (Naveen Gavini) Date: Wed, 16 Jul 2008 16:53:03 -0400 Subject: mod_python error installing Koji Message-ID: <487E5FAF.9000203@jla.rutgers.edu> Hello all, We are trying to setup the Koji build system for our Centos and Fedora repositories. We are getting the errors below after following the setup guide. We have tried numerous different things to attempt to correct the errors and nothing has worked. We initially thought it was an issue of what user it was being run as and we changed users around this did not work. We changed various settings in our apache configuration and still no dice. Here is the errors we are seeing on http://192.168.226.61/koji/: MOD_PYTHON ERROR ProcessId: 9453 Interpreter: '127.0.0.1' ServerName: '127.0.0.1' DocumentRoot: '/var/www/html' URI: '/koji/' Location: None Directory: '/usr/share/koji-web/scripts/' Filename: '/usr/share/koji-web/scripts/index.py' PathInfo: '' Phase: 'PythonHandler' Handler: 'mod_python.publisher' Traceback (most recent call last): File "/usr/lib64/python2.4/site-packages/mod_python/importer.py", line 1537, in HandlerDispatch default=default_handler, arg=req, silent=hlist.silent) File "/usr/lib64/python2.4/site-packages/mod_python/importer.py", line 1229, in _process_target result = _execute_target(config, req, object, arg) File "/usr/lib64/python2.4/site-packages/mod_python/importer.py", line 1128, in _execute_target result = object(arg) File "/usr/lib64/python2.4/site-packages/mod_python/publisher.py", line 213, in handler published = publish_object(req, object) File "/usr/lib64/python2.4/site-packages/mod_python/publisher.py", line 425, in publish_object return publish_object(req,util.apply_fs_data(object, req.form, req=req)) File "/usr/lib64/python2.4/site-packages/mod_python/util.py", line 554, in apply_fs_data return object(**args) File "/usr/share/koji-web/scripts/index.py", line 175, in index start=buildStart, dataName='builds', prefix='build', order=buildOrder, pageSize=10) File "/usr/share/koji-web/lib/kojiweb/util.py", line 109, in paginateMethod totalRows = getattr(server, methodName)(*args, **kw) File "/usr/lib/python2.4/site-packages/koji/__init__.py", line 1077, in __call__ return self.__func(self.__name,args,opts) File "/usr/lib/python2.4/site-packages/koji/__init__.py", line 1302, in _callMethod return proxy.__getattr__(name)(*args) File "/usr/lib64/python2.4/xmlrpclib.py", line 1096, in __call__ return self.__send(self.__name, args) File "/usr/lib64/python2.4/xmlrpclib.py", line 1383, in __request verbose=self.__verbose File "/usr/lib64/python2.4/xmlrpclib.py", line 1137, in request headers ProtocolError: MODULE CACHE DETAILS Accessed: Wed Jul 16 16:47:56 2008 Generation: 1 _mp_0dec3ca8c086f5baed01b0d5504fa2b0 { FileName: '/usr/share/koji-web/scripts/index.py' Instance: 1 Generation: 1 Modified: Fri Dec 14 21:12:36 2007 Imported: Wed Jul 16 16:36:02 2008 } Here is the error we are seeing on http://192.168.226.61/koji/index.chtml: Forbidden You don't have permission to access /koji/index.chtml on this server. Here is the error we are seeing on http://192.168.226.61/kojihub: Internal Server Error blah blah blah http://192.168.226.61/koji-static/ Displays a directory listing of a few files and directories so I am assuming it is working correctly. Anyone have any ideas? Thanks. -- Naveen Gavini Student Systems Programmer OSS/CSS - OIT Rutgers ngavini at jla.rutgers.edu From mikeb at redhat.com Wed Jul 16 21:38:54 2008 From: mikeb at redhat.com (Mike Bonnet) Date: Wed, 16 Jul 2008 17:38:54 -0400 Subject: mod_python error installing Koji In-Reply-To: <487E5FAF.9000203@jla.rutgers.edu> References: <487E5FAF.9000203@jla.rutgers.edu> Message-ID: <1216244334.3599.10.camel@localhost.localdomain> On Wed, 2008-07-16 at 16:53 -0400, Naveen Gavini wrote: > Hello all, > > We are trying to setup the Koji build system for our Centos and Fedora > repositories. We are getting the errors below after following the setup > guide. > We have tried numerous different things to attempt to correct the errors > and nothing has worked. We initially thought it was an issue of what > user it was > being run as and we changed users around this did not work. We changed > various settings in our apache configuration and still no dice. > Here is the errors we are seeing on http://192.168.226.61/koji/: > > MOD_PYTHON ERROR > > ProcessId: 9453 > Interpreter: '127.0.0.1' > > ServerName: '127.0.0.1' > DocumentRoot: '/var/www/html' > > URI: '/koji/' > Location: None > Directory: '/usr/share/koji-web/scripts/' > Filename: '/usr/share/koji-web/scripts/index.py' > PathInfo: '' > > Phase: 'PythonHandler' > Handler: 'mod_python.publisher' > > Traceback (most recent call last): > > File "/usr/lib64/python2.4/site-packages/mod_python/importer.py", line 1537, in HandlerDispatch > default=default_handler, arg=req, silent=hlist.silent) > > File "/usr/lib64/python2.4/site-packages/mod_python/importer.py", line 1229, in _process_target > result = _execute_target(config, req, object, arg) > > File "/usr/lib64/python2.4/site-packages/mod_python/importer.py", line 1128, in _execute_target > result = object(arg) > > File "/usr/lib64/python2.4/site-packages/mod_python/publisher.py", line 213, in handler > published = publish_object(req, object) > > File "/usr/lib64/python2.4/site-packages/mod_python/publisher.py", line 425, in publish_object > return publish_object(req,util.apply_fs_data(object, req.form, req=req)) > > File "/usr/lib64/python2.4/site-packages/mod_python/util.py", line 554, in apply_fs_data > return object(**args) > > File "/usr/share/koji-web/scripts/index.py", line 175, in index > start=buildStart, dataName='builds', prefix='build', order=buildOrder, pageSize=10) > > File "/usr/share/koji-web/lib/kojiweb/util.py", line 109, in paginateMethod > totalRows = getattr(server, methodName)(*args, **kw) > > File "/usr/lib/python2.4/site-packages/koji/__init__.py", line 1077, in __call__ > return self.__func(self.__name,args,opts) > > File "/usr/lib/python2.4/site-packages/koji/__init__.py", line 1302, in _callMethod > return proxy.__getattr__(name)(*args) > > File "/usr/lib64/python2.4/xmlrpclib.py", line 1096, in __call__ > return self.__send(self.__name, args) > > File "/usr/lib64/python2.4/xmlrpclib.py", line 1383, in __request > verbose=self.__verbose > > File "/usr/lib64/python2.4/xmlrpclib.py", line 1137, in request > headers > > ProtocolError: > > > MODULE CACHE DETAILS > > Accessed: Wed Jul 16 16:47:56 2008 > Generation: 1 > > _mp_0dec3ca8c086f5baed01b0d5504fa2b0 { > FileName: '/usr/share/koji-web/scripts/index.py' > Instance: 1 > Generation: 1 > Modified: Fri Dec 14 21:12:36 2007 > Imported: Wed Jul 16 16:36:02 2008 > } > > > Here is the error we are seeing on http://192.168.226.61/koji/index.chtml: > Forbidden > You don't have permission to access /koji/index.chtml on this server. > > Here is the error we are seeing on http://192.168.226.61/kojihub: > Internal Server Error > blah blah blah > > http://192.168.226.61/koji-static/ > Displays a directory listing of a few files and directories so I am > assuming it is working correctly. You should see more detailed error messages in /var/log/httpd/error_log (or ssl_error_log, depending on your setup). I'm guessing the "apache" OS user does not have permission to connect to the "koji" database as the "koji" database user. You need to grant the appropriate access in pg_hba.conf. From mikem at redhat.com Thu Jul 17 17:54:42 2008 From: mikem at redhat.com (Mike McLean) Date: Thu, 17 Jul 2008 13:54:42 -0400 Subject: Supporting EPEL Builds in Koji In-Reply-To: <1215705751.30991.7.camel@burren.bos.redhat.com> References: <1215705751.30991.7.camel@burren.bos.redhat.com> Message-ID: <487F8762.6090007@redhat.com> Mike Bonnet wrote: > http://fedoraproject.org/wiki/Koji/EPELSupport This is mostly in line with what I've been thinking. I do have a few comments/concerns thought... If the remote_repo_url data is going to be inherited (and I tend to think it should be), then I think it should be in a separate table. I'd like to reserve tag_config for data that is local to individual tags. This will also make it easier to represent multiple remote repos. I'm a little concerned about using the rpminfo table. Yes, I know it seems wasteful to introduce another table to track very similar data, but these remote rpms really are differently tracked and handled than the local ones. Also, I'm not sure how I feel about having rpminfo entries will null build_id. Sure, technically the field lacks the 'not null' constraint, but that is more of an oversight. Note, I'm not outright rejecting the idea of using rpminfo this way, but I am concerned. As for the origin field. I think we should track where these external rpms come from, but I'm not sure about including in the uniqueness constraint. I'm not sure that the value of that field is sufficiently well defined (or canonicalizable) for such use. I'd rather see the sigmd5 value (or some abstracting sighash field) used as a unique index. Following are additional ideas relating to this feature. They are perhaps a bit ambitious for the short term, but I'd at least like to keep them in mind with the initial design so we don't paint ourselves into a corner. First, I'd like to be able to support external koji servers (or rather a target or tag from an external koji server) in addition to external repos. Some of the ideas are the same, however an external koji server provides more information and more structure. Second, I'm fond of having a tag /represent/ some external repo/whatever and having the normal inheritance mechanism take care of priority. The trick here is that Koji tag content is by build, but it will be tricky to correctly determine build structure for external rpms -- indeed, external repos might include subpackages from different versions of the same build (the an external koji server would not, at least for its local content). So this will probably be difficult, but if we could manage something like this, I'd feel a lot better about using the rpminfo table. Doing something like this would most likely require Koji to comprehend the external repos instead of just passing them off to a repomerge tool. Third, we may not want to use a repomerge tool. The yum-priorities plugin might serve just as well, and allow us to specify some different yum repo options per external repo. This may conflict with idea#2 though. From dennis at ausil.us Thu Jul 17 21:08:10 2008 From: dennis at ausil.us (Dennis Gilmore) Date: Thu, 17 Jul 2008 16:08:10 -0500 Subject: Supporting EPEL Builds in Koji In-Reply-To: <487F8762.6090007@redhat.com> References: <1215705751.30991.7.camel@burren.bos.redhat.com> <487F8762.6090007@redhat.com> Message-ID: <200807171608.12240.dennis@ausil.us> On Thursday 17 July 2008, Mike McLean wrote: > Mike Bonnet wrote: > > http://fedoraproject.org/wiki/Koji/EPELSupport > > This is mostly in line with what I've been thinking. I do have a few > comments/concerns thought... > > If the remote_repo_url data is going to be inherited (and I tend to > think it should be), then I think it should be in a separate table. I'd > like to reserve tag_config for data that is local to individual tags. > This will also make it easier to represent multiple remote repos. > > I'm a little concerned about using the rpminfo table. Yes, I know it > seems wasteful to introduce another table to track very similar data, > but these remote rpms really are differently tracked and handled than > the local ones. > > Also, I'm not sure how I feel about having rpminfo entries will null > build_id. Sure, technically the field lacks the 'not null' constraint, > but that is more of an oversight. > > Note, I'm not outright rejecting the idea of using rpminfo this way, but > I am concerned. > > > As for the origin field. I think we should track where these external > rpms come from, but I'm not sure about including in the uniqueness > constraint. I'm not sure that the value of that field is sufficiently > well defined (or canonicalizable) for such use. I'd rather see the > sigmd5 value (or some abstracting sighash field) used as a unique index. > > > Following are additional ideas relating to this feature. They are > perhaps a bit ambitious for the short term, but I'd at least like to > keep them in mind with the initial design so we don't paint ourselves > into a corner. > > First, I'd like to be able to support external koji servers (or rather a > target or tag from an external koji server) in addition to external > repos. Some of the ideas are the same, however an external koji server > provides more information and more structure. In addition to external koji servers, id like to support spacewalk servers. and have the ability to push builds back into channels on spacewalk servers. ideally the spacewalk server knows how to pull from koji server rather than duplicating data by importing directly. this way an organisation could build upon fedora/RHEL/CentOS for their own needs. but can also have an easier time doing rel-eng on them. > Second, I'm fond of having a tag /represent/ some external repo/whatever > and having the normal inheritance mechanism take care of priority. The > trick here is that Koji tag content is by build, but it will be tricky > to correctly determine build structure for external rpms -- indeed, > external repos might include subpackages from different versions of the > same build (the an external koji server would not, at least for its > local content). So this will probably be difficult, but if we could > manage something like this, I'd feel a lot better about using the > rpminfo table. i would think there should be a 1-1 mapping of tag external repo using normal inheritence. > Doing something like this would most likely require Koji to comprehend > the external repos instead of just passing them off to a repomerge tool. > > Third, we may not want to use a repomerge tool. The yum-priorities > plugin might serve just as well, and allow us to specify some different > yum repo options per external repo. This may conflict with idea#2 though. I can see a case where this wont work. i have a local tag built on top of F-8 i want it lower than the remote F-9 because some of what i need is now in fedora, but i need other bits from my tag to be inherited so that i can boot strap things to the F-9 level. maybe we would produce 2 local repos and use yum priorites. to fit them together. maybe this case is rare enough not to bother with. but it could be an idea to keep in mind. -- Dennis Gilmore -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part. URL: From mikeb at redhat.com Thu Jul 17 22:48:34 2008 From: mikeb at redhat.com (Mike Bonnet) Date: Thu, 17 Jul 2008 18:48:34 -0400 Subject: Supporting EPEL Builds in Koji In-Reply-To: <487F8762.6090007@redhat.com> References: <1215705751.30991.7.camel@burren.bos.redhat.com> <487F8762.6090007@redhat.com> Message-ID: <1216334914.16888.82.camel@burren.bos.redhat.com> On Thu, 2008-07-17 at 13:54 -0400, Mike McLean wrote: > Mike Bonnet wrote: > > http://fedoraproject.org/wiki/Koji/EPELSupport > > This is mostly in line with what I've been thinking. I do have a few > comments/concerns thought... > > If the remote_repo_url data is going to be inherited (and I tend to > think it should be), then I think it should be in a separate table. I'd > like to reserve tag_config for data that is local to individual tags. > This will also make it easier to represent multiple remote repos. I don't have any problem with this, though it does mean we'll need to duplicate quite a bit of the inheritance-walking code, or make it configurable as to which inheritance it's walking. This new table would also have to be versioned, the same way the tag_config table is. > I'm a little concerned about using the rpminfo table. Yes, I know it > seems wasteful to introduce another table to track very similar data, > but these remote rpms really are differently tracked and handled than > the local ones. The big win here is that the methods and tools that query rpminfo for information about what was present in the buildroot at build time wouldn't have to change, or only change slightly. With minor modification the web UI can continue to show a list of all packages in a buildroot, along with a flag indicating if they were local or remote. The buildroot_listing table would not have to change at all. The majority of XML-RPC calls that interact with the rpminfo or buildroot_listing tables would only need minor modifications. Adding a new table to track remote rpms metadata and which remote rpms end up in a buildroot would add significant effort to this proposal. Also, I think it's more semantically correct to have a single place where we track rpm metadata and buildroot contents, regardless of where they came from. > Also, I'm not sure how I feel about having rpminfo entries will null > build_id. Sure, technically the field lacks the 'not null' constraint, > but that is more of an oversight. Yes, I realize that the "not null" constraint should exist now, and in fact all rpms in the Fedora database do reference builds. However, I think logically having a remote rpm not reference a local build makes sense. The alternative is to create the build object from the srpm info in the repodata (along with some namespacing similar to rpminfo). However, this would significantly clutter the build table with information that is pretty non-essential. > Note, I'm not outright rejecting the idea of using rpminfo this way, but > I am concerned. > > > As for the origin field. I think we should track where these external > rpms come from, but I'm not sure about including in the uniqueness > constraint. I'm not sure that the value of that field is sufficiently > well defined (or canonicalizable) for such use. I'd rather see the > sigmd5 value (or some abstracting sighash field) used as a unique index. I'm open to suggestions on how to modify the uniqueness constraint to handle this case. We care about ensuring that a locally-built rpm doesn't have the same n-v-r as another locally-built rpm. I don't think we care at all about n-v-r uniqueness amongst remote rpms. However, we probably want to avoid creating 2 rpminfo entries when the same remote rpm is used in 2 different buildroots. Using the sigmd5 is a good way to avoid that. However, what happens if a remote rpm with the same n-v-r and sigmd5 gets pulled in from 2 different remote repos? Perhaps the "origin" field should be pushed down to the buildroot_listing table, so the buildroots can reference the same rpminfo object, but indicate that it came from a different repo in each buildroot? Also, what happens when we find 2 remote rpms with the same n-v-r but different sigmd5s? Should that be an error? > Following are additional ideas relating to this feature. They are > perhaps a bit ambitious for the short term, but I'd at least like to > keep them in mind with the initial design so we don't paint ourselves > into a corner. > > First, I'd like to be able to support external koji servers (or rather a > target or tag from an external koji server) in addition to external > repos. Some of the ideas are the same, however an external koji server > provides more information and more structure. I agree that this is a desirable goal. I believe this is more the domain of the Koji secondary-arch daemon. It would be talking directly to an "upstream" Koji server, analyzing what it's doing, and applying some logic to decide what builds to import or replicate, and where/how to do it. This proposal has the much more modest goal of simply consuming static external repos, and is more appropriate for the EPEL and private-standalone-Koji case. > Second, I'm fond of having a tag /represent/ some external repo/whatever > and having the normal inheritance mechanism take care of priority. The > trick here is that Koji tag content is by build, but it will be tricky > to correctly determine build structure for external rpms -- indeed, > external repos might include subpackages from different versions of the > same build (the an external koji server would not, at least for its > local content). So this will probably be difficult, but if we could > manage something like this, I'd feel a lot better about using the > rpminfo table. > > Doing something like this would most likely require Koji to comprehend > the external repos instead of just passing them off to a repomerge tool. The tag content may be managed by build, but when it's time for it to actually get used (in the form of a yum repo) it gets unfolded into a big list of rpms. And what gets associated with a buildroot is simply a big list of rpms. Conceptually I don't really have a problem with the idea of a tag as a big list of rpms, that we happen to group by srpm within Koji because it's more convenient for us. So adding the external repo information to tag_config is just an extension of the big list of rpms model. However, we will already be parsing the remote repodata, which contains information like the srpm name for each rpm, so we could do something more sophisticated here. > Third, we may not want to use a repomerge tool. The yum-priorities > plugin might serve just as well, and allow us to specify some different > yum repo options per external repo. This may conflict with idea#2 though. This was my first thought as well. However, after discussions with Jesse, Seth, and James I was convinced otherwise. The yum-priorities plugin seems very unpopular with yum developers (not quite sure why). I don't think yum-priorities would give us any way to completely block a package from local and remote repos, and configuring multiple repos in the mock config would require Koji to retrieve and parse each remote repodata to determine the origin of a given remote rpm. The repomerge tool seems like it solves the problem better, and would be more useful in general. From kevin.linul at gmail.com Fri Jul 18 02:27:38 2008 From: kevin.linul at gmail.com (Linul) Date: Fri, 18 Jul 2008 10:27:38 +0800 Subject: Koji Web User Login Auth problem Message-ID: <5b123d6c0807171927kbcdc140y583238a98fcfe3f@mail.gmail.com> Hi.. I have already setup my koji-cli? kojid?koji-hub?koji-web?kojira, and test ok,but still have a problem,it is web UI login I have already import the cert. when I login the Web UI, the messages is? Mod_python error: "PythonHandler mod_python.publisher" Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/mod_python/apache.py", line 299, in HandlerDispatch result = object(req) File "/usr/lib/python2.4/site-packages/mod_python/publisher.py", line 213, in handler published = publish_object(req, object) File "/usr/lib/python2.4/site-packages/mod_python/publisher.py", line 412, in publish_object return publish_object(req,util.apply_fs_data(object, req.form, req=req)) File "/usr/lib/python2.4/site-packages/mod_python/util.py", line 439, in apply_fs_data return object(**args) File "/usr/share/koji-web/scripts/index.py", line 144, in login if not _sslLogin(req, session, username): File "/usr/share/koji-web/scripts/index.py", line 56, in _sslLogin proxyuser=username) File "__init__.py", line 1232, in ssl_login File "XMLRPCServerProxy.py", line 74, in __init__ File "SSLCommon.py", line 38, in CreateSSLContext StandardError: /root/genca/certs/kojiweb.pem does not exist or is not readable My kojiweb.conf? PythonDebug On PythonOption KojiHubURL http://koji.ossii.com.tw/kojihub PythonOption KojiWebURL http://koji.ossii.com.tw/koji PythonOption KojiPackagesURL http://koji.ossii.com.tw/koji/packages #PythonOption WebPrincipal koji/kevin.lin at ossii.com.tw #PythonOption WebKeytab /etc/httpd.keytab #PythonOption WebCCache /var/tmp/kojiweb.ccache PythonOption WebCert /root/genca/certs/kojiweb.pem PythonOption ClientCA /root/genca/koji_ca_cert.crt PythonOption KojiHubCA /root/genca/koji_ca_cert.crt My web user is "admin", and have a record in postgresql, also can user koji instruction as normal. and I create the web cert step is? openssl pkcs12 -export -inkey certs/admin.key -in certs/admin.crt -CAfile koji_ca_cert.crt \ -out certs/admin_browser_cert.p12 thanks.. -- ============================================================================= ??? Linul RedHat Certified Engineer TsLG??????http://www.tslg.idv.tw TsLG?????http://blog.tslg.idv.tw Linul?????http://photo.tslg.idv.tw ???0939797462 E-mail : kevin.linul at gmail.com; linul at tslg.idv.tw ============================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From jkeating at redhat.com Fri Jul 18 02:29:20 2008 From: jkeating at redhat.com (Jesse Keating) Date: Thu, 17 Jul 2008 22:29:20 -0400 Subject: Supporting EPEL Builds in Koji In-Reply-To: <1216334914.16888.82.camel@burren.bos.redhat.com> References: <1215705751.30991.7.camel@burren.bos.redhat.com> <487F8762.6090007@redhat.com> <1216334914.16888.82.camel@burren.bos.redhat.com> Message-ID: <1216348160.3391.0.camel@localhost.localdomain> On Thu, 2008-07-17 at 18:48 -0400, Mike Bonnet wrote: > This was my first thought as well. However, after discussions with > Jesse, Seth, and James I was convinced otherwise. The yum-priorities > plugin seems very unpopular with yum developers (not quite sure why). I > don't think yum-priorities would give us any way to completely block a > package from local and remote repos, and configuring multiple repos in > the mock config would require Koji to retrieve and parse each remote > repodata to determine the origin of a given remote rpm. Also you wouldn't be able to prioritize at the srpm level which is what we want (no unwanted subpackages sneaking in). -- Jesse Keating Fedora -- Freedom? is a feature! -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: From kevin.linul at gmail.com Fri Jul 18 02:47:38 2008 From: kevin.linul at gmail.com (Linul) Date: Fri, 18 Jul 2008 10:47:38 +0800 Subject: Koji Web User Login Auth problem In-Reply-To: <5b123d6c0807171927kbcdc140y583238a98fcfe3f@mail.gmail.com> References: <5b123d6c0807171927kbcdc140y583238a98fcfe3f@mail.gmail.com> Message-ID: <5b123d6c0807171947y14aada5aq553da9efa8af4668@mail.gmail.com> Sorry...My setting is error.... PythonOption WebCert /root/genca/certs/kojiweb.pem -> PythonOption WebCert /root/genca/kojiweb.pem but...another messages is feeback.. Mod_python error: "PythonHandler mod_python.publisher" Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/mod_python/apache.py", line 299, in HandlerDispatch result = object(req) File "/usr/lib/python2.4/site-packages/mod_python/publisher.py", line 213, in handler published = publish_object(req, object) File "/usr/lib/python2.4/site-packages/mod_python/publisher.py", line 412, in publish_object return publish_object(req,util.apply_fs_data(object, req.form, req=req)) File "/usr/lib/python2.4/site-packages/mod_python/util.py", line 439, in apply_fs_data return object(**args) File "/usr/share/koji-web/scripts/index.py", line 144, in login if not _sslLogin(req, session, username): File "/usr/share/koji-web/scripts/index.py", line 56, in _sslLogin proxyuser=username) File "__init__.py", line 1233, in ssl_login File "__init__.py", line 1278, in callMethod File "__init__.py", line 1304, in _callMethod AuthError: /C=TW/ST=Taiwan/O=OSSII/CN=web.ossii.com.tw/emailAddress=kevin.lin at ossii.com.tw is not authorized to login other users 2008/7/18 Linul : > Hi.. > > I have already setup my koji-cli? kojid?koji-hub?koji-web?kojira, and test > ok,but still have a problem,it is web UI login > > I have already import the cert. when I login the Web UI, the messages is? > > Mod_python error: "PythonHandler mod_python.publisher" > > Traceback (most recent call last): > > File "/usr/lib/python2.4/site-packages/mod_python/apache.py", line 299, in HandlerDispatch > > result = object(req) > > File "/usr/lib/python2.4/site-packages/mod_python/publisher.py", line 213, in handler > published = publish_object(req, object) > > File "/usr/lib/python2.4/site-packages/mod_python/publisher.py", line 412, in publish_object > > return publish_object(req,util.apply_fs_data(object, req.form, req=req)) > > File "/usr/lib/python2.4/site-packages/mod_python/util.py", line 439, in apply_fs_data > return object(**args) > > File "/usr/share/koji-web/scripts/index.py", line 144, in login > > if not _sslLogin(req, session, username): > > File "/usr/share/koji-web/scripts/index.py", line 56, in _sslLogin > proxyuser=username) > > File "__init__.py", line 1232, in ssl_login > > File "XMLRPCServerProxy.py", line 74, in __init__ > > File "SSLCommon.py", line 38, in CreateSSLContext > > StandardError: /root/genca/certs/kojiweb.pem does not exist or is not readable > > My kojiweb.conf? > > PythonDebug On > PythonOption KojiHubURL http://koji.ossii.com.tw/kojihub > PythonOption KojiWebURL http://koji.ossii.com.tw/koji > PythonOption KojiPackagesURL http://koji.ossii.com.tw/koji/packages > #PythonOption WebPrincipal koji/kevin.lin at ossii.com.tw > #PythonOption WebKeytab /etc/httpd.keytab > #PythonOption WebCCache /var/tmp/kojiweb.ccache > PythonOption WebCert /root/genca/certs/kojiweb.pem > PythonOption ClientCA /root/genca/koji_ca_cert.crt > PythonOption KojiHubCA /root/genca/koji_ca_cert.crt > > My web user is "admin", and have a record in postgresql, also can user koji > instruction as normal. > > and I create the web cert step is? > > openssl pkcs12 -export -inkey certs/admin.key -in certs/admin.crt -CAfile koji_ca_cert.crt \ > > -out certs/admin_browser_cert.p12 > > > thanks.. > > > -- > > ============================================================================= > ??? > Linul > RedHat Certified Engineer > > TsLG??????http://www.tslg.idv.tw > TsLG?????http://blog.tslg.idv.tw > Linul?????http://photo.tslg.idv.tw > ???0939797462 > E-mail : kevin.linul at gmail.com; linul at tslg.idv.tw > ============================================================================= > > -- ============================================================================= ??? Linul RedHat Certified Engineer TsLG??????http://www.tslg.idv.tw TsLG?????http://blog.tslg.idv.tw Linul?????http://photo.tslg.idv.tw ???0939797462 E-mail : kevin.linul at gmail.com; linul at tslg.idv.tw ============================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From dennis at ausil.us Fri Jul 18 02:50:01 2008 From: dennis at ausil.us (Dennis Gilmore) Date: Thu, 17 Jul 2008 21:50:01 -0500 Subject: Koji Web User Login Auth problem In-Reply-To: <5b123d6c0807171927kbcdc140y583238a98fcfe3f@mail.gmail.com> References: <5b123d6c0807171927kbcdc140y583238a98fcfe3f@mail.gmail.com> Message-ID: <200807172150.05211.dennis@ausil.us> On Thursday 17 July 2008, Linul wrote: > Hi.. > > I have already setup my koji-cli? kojid?koji-hub?koji-web?kojira, and test > ok,but still have a problem,it is web UI login > > I have already import the cert. when I login the Web UI, the messages is? > > StandardError: /root/genca/certs/kojiweb.pem does not exist or is not > readable apache runs as user apache it cant read files in roots ~ you need to make sure the webserver can read the certs. I'm betting an authenticated cli command will fail exactly the same way. most cli tasks are unauthenticated. -- Dennis Gilmore From dennis at ausil.us Fri Jul 18 03:17:14 2008 From: dennis at ausil.us (Dennis Gilmore) Date: Thu, 17 Jul 2008 22:17:14 -0500 Subject: Koji Web User Login Auth problem In-Reply-To: <5b123d6c0807171947y14aada5aq553da9efa8af4668@mail.gmail.com> References: <5b123d6c0807171927kbcdc140y583238a98fcfe3f@mail.gmail.com> <5b123d6c0807171947y14aada5aq553da9efa8af4668@mail.gmail.com> Message-ID: <200807172217.21582.dennis@ausil.us> On Thursday 17 July 2008, Linul wrote: > Sorry...My setting is error.... > > AuthError: > /C=TW/ST=Taiwan/O=OSSII/CN=web.ossii.com.tw/emailAddress=kevin.lin at ossii.co >m.tw is not authorized to login other users you need to set PythonOption ProxyDNs "/C=TW/ST=Taiwan/O=OSSII/CN=web.ossii.com.tw/emailAddress=kevin.lin at ossii.com.tw" in /etc/httpd/conf.d/kojihub.conf -- Dennis Gilmore -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part. URL: From sergio at sergiomb.no-ip.org Fri Jul 18 03:23:42 2008 From: sergio at sergiomb.no-ip.org (Sergio Monteiro Basto) Date: Fri, 18 Jul 2008 04:23:42 +0100 Subject: pungi repo priorities Message-ID: <1216351422.26321.39.camel@segulix.localdomain> Hi, how I set pungi repo priorities ? New pungi on fedora 9, give me an error about repo.priorities is a NoneType TIA, -- S?rgio M. B. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 2192 bytes Desc: not available URL: From jkeating at redhat.com Fri Jul 18 15:13:02 2008 From: jkeating at redhat.com (Jesse Keating) Date: Fri, 18 Jul 2008 11:13:02 -0400 Subject: pungi repo priorities In-Reply-To: <1216351422.26321.39.camel@segulix.localdomain> References: <1216351422.26321.39.camel@segulix.localdomain> Message-ID: <1216393982.3391.8.camel@localhost.localdomain> On Fri, 2008-07-18 at 04:23 +0100, Sergio Monteiro Basto wrote: > Hi, how I set pungi repo priorities ? > > New pungi on fedora 9, give me an error about repo.priorities is a > NoneType I thought I pushed an update for that. Check with either F9 updates or F9 updates testing. -- Jesse Keating Fedora -- Freedom? is a feature! -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: From mikem at redhat.com Fri Jul 18 15:38:28 2008 From: mikem at redhat.com (Mike McLean) Date: Fri, 18 Jul 2008 11:38:28 -0400 Subject: Supporting EPEL Builds in Koji In-Reply-To: <1216334914.16888.82.camel@burren.bos.redhat.com> References: <1215705751.30991.7.camel@burren.bos.redhat.com> <487F8762.6090007@redhat.com> <1216334914.16888.82.camel@burren.bos.redhat.com> Message-ID: <4880B8F4.9040701@redhat.com> Mike Bonnet wrote: > On Thu, 2008-07-17 at 13:54 -0400, Mike McLean wrote: >> If the remote_repo_url data is going to be inherited (and I tend to >> think it should be), then I think it should be in a separate table. I'd >> like to reserve tag_config for data that is local to individual tags. >> This will also make it easier to represent multiple remote repos. > > I don't have any problem with this, though it does mean we'll need to > duplicate quite a bit of the inheritance-walking code, or make it > configurable as to which inheritance it's walking. This new table would > also have to be versioned, the same way the tag_config table is. Walking inheritance is just a matter of determining the inheritance order and scanning data on the parent tags in sequence. Currently, nothing scans tag_config in this way because no data in tag_config is inherited. (Well, in a sense tag_changed_since_event() does walk tag_config, but that's a little different.) We need to figure out how we'll deal with multiplicity for the external repos. If tag A uses repo X and inherits from tag B which uses repo Y, then does tag A use both X and Y, or does the X entry override it? A (+repo X) +- B (+repo Y) My inclination is that it should override, because I think we'll want some way to do override that that mechanism seems easiest. Also, I think we'll probably want to allow multiple external repos per tag, something which will be much easier to represent in an external table. We can include an explicit priority field to make a sane uniqueness condition (and to provide a clear ordering for the repo merge). > The big win here is that the methods and tools that query rpminfo for > information about what was present in the buildroot at build time -snip- I see all that, and I'm almost convinced. The flipside is that by default all the code will treat these external rpms the same as the local ones, which will not be correct for a number of cases. Obviously, part of this will involve changing code to behave differently for the external ones, I'm just worried about how much we might have to change, or what we might miss. > Yes, I realize that the "not null" constraint should exist now, and in > fact all rpms in the Fedora database do reference builds. However, I > think logically having a remote rpm not reference a local build makes > sense. The alternative is to create the build object from the srpm info > in the repodata (along with some namespacing similar to rpminfo). > However, this would significantly clutter the build table with > information that is pretty non-essential. The idea of grouping them into builds appeals to me, but I don't think it's possible in general (though maybe we could fake it well enough somehow). The only data we're (mostly) guaranteed to have to work with is the sourcerpm header field. The catch is that in case of an nvr-collision we can't determine which build it belongs to (or indeed if we should create a new build of same nvr). > I'm open to suggestions on how to modify the uniqueness constraint to > handle this case. We care about ensuring that a locally-built rpm > doesn't have the same n-v-r as another locally-built rpm. I don't think > we care at all about n-v-r uniqueness amongst remote rpms. However, we > probably want to avoid creating 2 rpminfo entries when the same remote > rpm is used in 2 different buildroots. Using the sigmd5 is a good way > to avoid that. Agreed. same sigmd5 ==> same rpm. > However, what happens if a remote rpm with the same > n-v-r and sigmd5 gets pulled in from 2 different remote repos? This gets into part of what bugs me about this and why I'm somewhat inclined to keep the ext repo data a step removed. It's so potentially dirty. Koji has all these consistency constraints that an external repo (much less many of them in aggregate) lacks. It's quite possible that an external repo might respin a package keeping the same nvr, so we don't even need 2 external repos to hit this possibility. > Perhaps > the "origin" field should be pushed down to the buildroot_listing table, > so the buildroots can reference the same rpminfo object, but indicate > that it came from a different repo in each buildroot? Interesting. Yeah, I think that is is probably the right answer. Also, I'm thinking we need to have some sort of rpm_origin table so that all these references can be managed cleanly. > Also, what happens when we find 2 remote rpms with the same n-v-r but > different sigmd5s? Should that be an error? Certainly we have to allow the possibility that two origins might have overlapping nvras. Within a single origin, I'm not so sure. I suppose we can get away with some small consistency demands. As long as we're only enforcing unique nvra for local builds and indexing by sigmd5/similar, I don't think we /have/ to make this an error condition. In the same vein, what happens when an external repo has an nvra+sigmd5 matching a /local/ rpm? Maybe it doesn't matter, though I guess technically we want to record the origin properly when it gets into a buildroot via external repo vs internal tag. >> First, I'd like to be able to support external koji servers (or rather a ... > I agree that this is a desirable goal. I believe this is more the > domain of the Koji secondary-arch daemon. It would be talking directly Well, it has some similarities to 2nd arch, but still quite different. The more I think about it, the more I think that supporting an external koji server will probably be much different from from the ext repo business. Most of the issues with rpminfo will carry over, but with a koji server we will be able to determine build data and can probably actually pull off something like "inherit from tag X on koji server Y." > The tag content may be managed by build, but when it's time for it to > actually get used (in the form of a yum repo) it gets unfolded into a > big list of rpms. And what gets associated with a buildroot is simply a > big list of rpms. Conceptually I don't really have a problem with the > idea of a tag as a big list of rpms, that we happen to group by srpm > within Koji because it's more convenient for us. So adding the external > repo information to tag_config is just an extension of the big list of > rpms model. Yeah, I almost wish I hadn't made the build structure quite the way I did. > However, we will already be parsing the remote repodata, which contains > information like the srpm name for each rpm, so we could do something > more sophisticated here. -snipsnip- ... > The repomerge tool seems like it solves the problem better, and would be > more useful in general. If we're going to have our fingers in the repodata, we'll probably want to have them in the merge too. Perhaps we can get createrepo and/or this repomerge tool usefully libified? From kevin.linul at gmail.com Tue Jul 22 09:31:12 2008 From: kevin.linul at gmail.com (Linul) Date: Tue, 22 Jul 2008 17:31:12 +0800 Subject: How init mock in Koji? Message-ID: <5b123d6c0807220231o2515986age6dd4691967d39c2@mail.gmail.com> Hi.. I have already set my koji server and every component is work OK. But not have any document mention howto init the buildroot?(or use mock environment) is any body can tell me?? tks.. -- ============================================================================= ??? Linul RedHat Certified Engineer TsLG??????http://www.tslg.idv.tw TsLG?????http://blog.tslg.idv.tw Linul?????http://photo.tslg.idv.tw ???0939797462 E-mail : kevin.linul at gmail.com; linul at tslg.idv.tw ============================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikem at redhat.com Tue Jul 22 14:48:37 2008 From: mikem at redhat.com (Mike McLean) Date: Tue, 22 Jul 2008 10:48:37 -0400 Subject: How init mock in Koji? In-Reply-To: <5b123d6c0807220231o2515986age6dd4691967d39c2@mail.gmail.com> References: <5b123d6c0807220231o2515986age6dd4691967d39c2@mail.gmail.com> Message-ID: <4885F345.3090001@redhat.com> Linul wrote: > I have already set my koji server and every component is work OK. > > But not have any document mention howto init the buildroot?(or use mock > environment) It's not entirely clear what you're asking, but here goes... mock is used by koji-builder to create buildroots. In normal operation, you submit a build to the system, the builders pick it up and use mock to create a buildroot and run the build. If, on the other hand, you want to run mock manually against the repos that koji maintains (not normal operation, but useful nonetheless), then you should just be able to generate a mock config with the 'koji mock-config' command. You can use that config with a manual run of mock. Sometimes the config may need a little tweaking. If this doesn't answer your question, then perhaps you could provide a bit more context. From ngavini at jla.rutgers.edu Tue Jul 22 19:55:50 2008 From: ngavini at jla.rutgers.edu (Naveen Gavini) Date: Tue, 22 Jul 2008 15:55:50 -0400 Subject: mod_python error installing Koji In-Reply-To: <1216244334.3599.10.camel@localhost.localdomain> References: <487E5FAF.9000203@jla.rutgers.edu> <1216244334.3599.10.camel@localhost.localdomain> Message-ID: <48863B46.3000202@jla.rutgers.edu> Mike Bonnet wrote: > On Wed, 2008-07-16 at 16:53 -0400, Naveen Gavini wrote: > >> Hello all, >> >> We are trying to setup the Koji build system for our Centos and Fedora >> repositories. We are getting the errors below after following the setup >> guide. >> We have tried numerous different things to attempt to correct the errors >> and nothing has worked. We initially thought it was an issue of what >> user it was >> being run as and we changed users around this did not work. We changed >> various settings in our apache configuration and still no dice. >> Here is the errors we are seeing on http://192.168.226.61/koji/: >> >> MOD_PYTHON ERROR >> >> ProcessId: 9453 >> Interpreter: '127.0.0.1' >> >> ServerName: '127.0.0.1' >> DocumentRoot: '/var/www/html' >> >> URI: '/koji/' >> Location: None >> Directory: '/usr/share/koji-web/scripts/' >> Filename: '/usr/share/koji-web/scripts/index.py' >> PathInfo: '' >> >> Phase: 'PythonHandler' >> Handler: 'mod_python.publisher' >> >> Traceback (most recent call last): >> >> File "/usr/lib64/python2.4/site-packages/mod_python/importer.py", line 1537, in HandlerDispatch >> default=default_handler, arg=req, silent=hlist.silent) >> >> File "/usr/lib64/python2.4/site-packages/mod_python/importer.py", line 1229, in _process_target >> result = _execute_target(config, req, object, arg) >> >> File "/usr/lib64/python2.4/site-packages/mod_python/importer.py", line 1128, in _execute_target >> result = object(arg) >> >> File "/usr/lib64/python2.4/site-packages/mod_python/publisher.py", line 213, in handler >> published = publish_object(req, object) >> >> File "/usr/lib64/python2.4/site-packages/mod_python/publisher.py", line 425, in publish_object >> return publish_object(req,util.apply_fs_data(object, req.form, req=req)) >> >> File "/usr/lib64/python2.4/site-packages/mod_python/util.py", line 554, in apply_fs_data >> return object(**args) >> >> File "/usr/share/koji-web/scripts/index.py", line 175, in index >> start=buildStart, dataName='builds', prefix='build', order=buildOrder, pageSize=10) >> >> File "/usr/share/koji-web/lib/kojiweb/util.py", line 109, in paginateMethod >> totalRows = getattr(server, methodName)(*args, **kw) >> >> File "/usr/lib/python2.4/site-packages/koji/__init__.py", line 1077, in __call__ >> return self.__func(self.__name,args,opts) >> >> File "/usr/lib/python2.4/site-packages/koji/__init__.py", line 1302, in _callMethod >> return proxy.__getattr__(name)(*args) >> >> File "/usr/lib64/python2.4/xmlrpclib.py", line 1096, in __call__ >> return self.__send(self.__name, args) >> >> File "/usr/lib64/python2.4/xmlrpclib.py", line 1383, in __request >> verbose=self.__verbose >> >> File "/usr/lib64/python2.4/xmlrpclib.py", line 1137, in request >> headers >> >> ProtocolError: >> >> >> MODULE CACHE DETAILS >> >> Accessed: Wed Jul 16 16:47:56 2008 >> Generation: 1 >> >> _mp_0dec3ca8c086f5baed01b0d5504fa2b0 { >> FileName: '/usr/share/koji-web/scripts/index.py' >> Instance: 1 >> Generation: 1 >> Modified: Fri Dec 14 21:12:36 2007 >> Imported: Wed Jul 16 16:36:02 2008 >> } >> >> >> Here is the error we are seeing on http://192.168.226.61/koji/index.chtml: >> Forbidden >> You don't have permission to access /koji/index.chtml on this server. >> >> Here is the error we are seeing on http://192.168.226.61/kojihub: >> Internal Server Error >> blah blah blah >> >> http://192.168.226.61/koji-static/ >> Displays a directory listing of a few files and directories so I am >> assuming it is working correctly. >> > > You should see more detailed error messages in /var/log/httpd/error_log > (or ssl_error_log, depending on your setup). I'm guessing the "apache" > OS user does not have permission to connect to the "koji" database as > the "koji" database user. You need to grant the appropriate access in > pg_hba.conf. > > We were able to fix this: Our problem was that our pg_hba.conf file was in the wrong order. # TYPE DATABASE USER CIDR-ADDRESS METHOD # "local" is for Unix domain socket connections only local all all ident sameuser # IPv4 local connections: host all all 127.0.0.1/32 ident sameuser # IPv6 local connections: host all all ::1/128 ident sameuser #koji line host koji koji 127.0.0.1/32 trust Moving the koji line to the top instead of the bottom fixed it. Thanks for all the help. -- Naveen Gavini Student Systems Programmer OSS/CSS - OIT Rutgers ngavini at jla.rutgers.edu From ngavini at jla.rutgers.edu Tue Jul 22 20:02:46 2008 From: ngavini at jla.rutgers.edu (Naveen Gavini) Date: Tue, 22 Jul 2008 16:02:46 -0400 Subject: GenericError Repo directory missing Message-ID: <48863CE6.2010506@jla.rutgers.edu> Hello all, We have installed the Koji build system and are able to import RPMS. We are having issues building from SRPMS as it's prompting us that it is unable to find the file once it is uploaded. The other issue is we have seen about 500 tasks fail for newRepo (noarch) with the message: GenericError: Repo directory missing: /mnt/koji/repos/rutgers-centos5-build/538. We have set the permissions to daemon on /mnt/koji (same user as apache). It seems when we try to manually look at the directory it exists, this is also the same for when the srpm is uploaded and it fails prompting us file not found, even though we can find it by looking at that directory on the server. Below is the kojira.log 2008-07-22 19:46:31,358 [DEBUG] koji.repo.manager: Reading current repo data 2008-07-22 19:46:36,855 [DEBUG] koji.repo.manager: Repo data: [{'create_event': 552, 'state': 0, 'tag_id': 2, 'tag_name': 'rutgers-centos5-build', 'create_ts': 1216754980.8750801, 'id': 534}, {'create_event': 553, 'state': 0, 'tag_id': 2, 'tag_name': 'rutgers-centos5-build', 'create_ts': 1216755215.82616, 'id': 535}] 2008-07-22 19:46:42,399 [DEBUG] koji.repo.manager: Needed tags: [2] 2008-07-22 19:46:42,400 [DEBUG] koji.repo.manager: Current tags: [2] 2008-07-22 19:46:42,400 [DEBUG] koji.repo: Checking for changes: [1, 2] 2008-07-22 19:46:47,902 [DEBUG] koji.repo: No tag changes since event 552 2008-07-22 19:46:47,902 [DEBUG] koji.repo: Checking for changes: [1, 2] 2008-07-22 19:46:53,429 [DEBUG] koji.repo: No tag changes since event 553 2008-07-22 19:46:53,430 [DEBUG] koji.repo.manager: order: [] 2008-07-22 19:46:53,430 [DEBUG] koji.repo.manager: repo 534: tag=2, state=INIT 2008-07-22 19:46:53,430 [DEBUG] koji.repo.manager: repo 535: tag=2, state=INIT 2008-07-22 19:46:53,430 [DEBUG] koji.repo.manager: task 548 for tag 2 2008-07-22 19:46:53,430 [DEBUG] koji.repo.manager: Scanning filesystem for repos 2008-07-22 19:47:09,590 [INFO] koji.repo.manager: Problem: newRepo task 548 for tag 2 is FAILED 2008-07-22 19:47:09,591 [DEBUG] koji.repo.manager: Current tasks: {} 2008-07-22 19:47:09,591 [DEBUG] koji.repo.manager: Updating repos 2008-07-22 19:47:09,591 [DEBUG] koji.repo.manager: Reading current repo data 2008-07-22 19:47:15,273 [DEBUG] koji.repo.manager: Repo data: [{'create_event': 552, 'state': 0, 'tag_id': 2, 'tag_name': 'rutgers-centos5-build', 'create_ts': 1216754980.8750801, 'id': 534}, {'create_event': 553, 'state': 0, 'tag_id': 2, 'tag_name': 'rutgers-centos5-build', 'create_ts': 1216755215.82616, 'id': 535}, {'create_event': 554, 'state': 0, 'tag_id': 2, 'tag_name': 'rutgers-centos5-build', 'create_ts': 1216756145.8847401, 'id': 536}] 2008-07-22 19:47:15,273 [INFO] koji.repo.manager: Found repo 536, state=INIT 2008-07-22 19:47:26,421 [DEBUG] koji.repo.manager: Needed tags: [2] 2008-07-22 19:47:26,421 [DEBUG] koji.repo.manager: Current tags: [2] 2008-07-22 19:47:26,421 [DEBUG] koji.repo: Checking for changes: [1, 2] 2008-07-22 19:47:31,905 [DEBUG] koji.repo: No tag changes since event 554 2008-07-22 19:47:31,905 [DEBUG] koji.repo: Checking for changes: [1, 2] Any idea? Thanks. -- Naveen Gavini Student Systems Programmer OSS/CSS - OIT Rutgers ngavini at jla.rutgers.edu From mikeb at redhat.com Tue Jul 22 20:28:06 2008 From: mikeb at redhat.com (Mike Bonnet) Date: Tue, 22 Jul 2008 16:28:06 -0400 Subject: GenericError Repo directory missing In-Reply-To: <48863CE6.2010506@jla.rutgers.edu> References: <48863CE6.2010506@jla.rutgers.edu> Message-ID: <1216758486.20652.15.camel@maunalani.home> On Tue, 2008-07-22 at 16:02 -0400, Naveen Gavini wrote: > Hello all, > > We have installed the Koji build system and are able to import RPMS. We > are having issues building from SRPMS as it's prompting us that it is > unable to find the file once it is uploaded. The other issue is we have > seen about 500 tasks fail for newRepo (noarch) with the message: > GenericError: Repo directory missing: > /mnt/koji/repos/rutgers-centos5-build/538. Does the /mnt/koji/repos directory exist? Is it owned by the same user httpd runs as? Also, have you tried to change topdir to something other than /mnt/koji? There are known problem with this, it needs to be fixed. > We have set the permissions to daemon on /mnt/koji (same user as apache). > > It seems when we try to manually look at the directory it exists, this > is also the same for when the srpm is uploaded and it fails prompting us > file not found, even though we can find it by looking at that directory > on the server. > > Below is the kojira.log > > 2008-07-22 19:46:31,358 [DEBUG] koji.repo.manager: Reading current repo data > 2008-07-22 19:46:36,855 [DEBUG] koji.repo.manager: Repo data: > [{'create_event': 552, 'state': 0, 'tag_id': 2, 'tag_name': > 'rutgers-centos5-build', 'create_ts': 1216754980.8750801, 'id': 534}, > {'create_event': 553, 'state': 0, 'tag_id': 2, 'tag_name': > 'rutgers-centos5-build', 'create_ts': 1216755215.82616, 'id': 535}] > 2008-07-22 19:46:42,399 [DEBUG] koji.repo.manager: Needed tags: [2] > 2008-07-22 19:46:42,400 [DEBUG] koji.repo.manager: Current tags: [2] > 2008-07-22 19:46:42,400 [DEBUG] koji.repo: Checking for changes: [1, 2] > 2008-07-22 19:46:47,902 [DEBUG] koji.repo: No tag changes since event 552 > 2008-07-22 19:46:47,902 [DEBUG] koji.repo: Checking for changes: [1, 2] > 2008-07-22 19:46:53,429 [DEBUG] koji.repo: No tag changes since event 553 > 2008-07-22 19:46:53,430 [DEBUG] koji.repo.manager: order: [] > 2008-07-22 19:46:53,430 [DEBUG] koji.repo.manager: repo 534: tag=2, > state=INIT > 2008-07-22 19:46:53,430 [DEBUG] koji.repo.manager: repo 535: tag=2, > state=INIT > 2008-07-22 19:46:53,430 [DEBUG] koji.repo.manager: task 548 for tag 2 > 2008-07-22 19:46:53,430 [DEBUG] koji.repo.manager: Scanning filesystem > for repos > 2008-07-22 19:47:09,590 [INFO] koji.repo.manager: Problem: newRepo task > 548 for tag 2 is FAILED > 2008-07-22 19:47:09,591 [DEBUG] koji.repo.manager: Current tasks: {} > 2008-07-22 19:47:09,591 [DEBUG] koji.repo.manager: Updating repos > 2008-07-22 19:47:09,591 [DEBUG] koji.repo.manager: Reading current repo data > 2008-07-22 19:47:15,273 [DEBUG] koji.repo.manager: Repo data: > [{'create_event': 552, 'state': 0, 'tag_id': 2, 'tag_name': > 'rutgers-centos5-build', 'create_ts': 1216754980.8750801, 'id': 534}, > {'create_event': 553, 'state': 0, 'tag_id': 2, 'tag_name': > 'rutgers-centos5-build', 'create_ts': 1216755215.82616, 'id': 535}, > {'create_event': 554, 'state': 0, 'tag_id': 2, 'tag_name': > 'rutgers-centos5-build', 'create_ts': 1216756145.8847401, 'id': 536}] > 2008-07-22 19:47:15,273 [INFO] koji.repo.manager: Found repo 536, state=INIT > 2008-07-22 19:47:26,421 [DEBUG] koji.repo.manager: Needed tags: [2] > 2008-07-22 19:47:26,421 [DEBUG] koji.repo.manager: Current tags: [2] > 2008-07-22 19:47:26,421 [DEBUG] koji.repo: Checking for changes: [1, 2] > 2008-07-22 19:47:31,905 [DEBUG] koji.repo: No tag changes since event 554 > 2008-07-22 19:47:31,905 [DEBUG] koji.repo: Checking for changes: [1, 2] > > Any idea? > > Thanks. > From ngavini at jla.rutgers.edu Tue Jul 22 21:19:41 2008 From: ngavini at jla.rutgers.edu (Naveen Gavini) Date: Tue, 22 Jul 2008 17:19:41 -0400 Subject: GenericError Repo directory missing In-Reply-To: <1216758486.20652.15.camel@maunalani.home> References: <48863CE6.2010506@jla.rutgers.edu> <1216758486.20652.15.camel@maunalani.home> Message-ID: <48864EED.6020907@jla.rutgers.edu> Mike Bonnet wrote: > On Tue, 2008-07-22 at 16:02 -0400, Naveen Gavini wrote: > >> Hello all, >> >> We have installed the Koji build system and are able to import RPMS. We >> are having issues building from SRPMS as it's prompting us that it is >> unable to find the file once it is uploaded. The other issue is we have >> seen about 500 tasks fail for newRepo (noarch) with the message: >> GenericError: Repo directory missing: >> /mnt/koji/repos/rutgers-centos5-build/538. >> > > Does the /mnt/koji/repos directory exist? Is it owned by the same user > httpd runs as? > > Also, have you tried to change topdir to something other than /mnt/koji? > There are known problem with this, it needs to be fixed. > > We were able to fix this. The problem seemed to be that we did not have this directory (/mnt/koji) mounted on the builder, it was only mounted on the hub which runs kojira. We thought since kojira was creating the repos directories that would suffice. It also needs to be accessible by the builder. Thanks. >> We have set the permissions to daemon on /mnt/koji (same user as apache). >> >> It seems when we try to manually look at the directory it exists, this >> is also the same for when the srpm is uploaded and it fails prompting us >> file not found, even though we can find it by looking at that directory >> on the server. >> >> Below is the kojira.log >> >> 2008-07-22 19:46:31,358 [DEBUG] koji.repo.manager: Reading current repo data >> 2008-07-22 19:46:36,855 [DEBUG] koji.repo.manager: Repo data: >> [{'create_event': 552, 'state': 0, 'tag_id': 2, 'tag_name': >> 'rutgers-centos5-build', 'create_ts': 1216754980.8750801, 'id': 534}, >> {'create_event': 553, 'state': 0, 'tag_id': 2, 'tag_name': >> 'rutgers-centos5-build', 'create_ts': 1216755215.82616, 'id': 535}] >> 2008-07-22 19:46:42,399 [DEBUG] koji.repo.manager: Needed tags: [2] >> 2008-07-22 19:46:42,400 [DEBUG] koji.repo.manager: Current tags: [2] >> 2008-07-22 19:46:42,400 [DEBUG] koji.repo: Checking for changes: [1, 2] >> 2008-07-22 19:46:47,902 [DEBUG] koji.repo: No tag changes since event 552 >> 2008-07-22 19:46:47,902 [DEBUG] koji.repo: Checking for changes: [1, 2] >> 2008-07-22 19:46:53,429 [DEBUG] koji.repo: No tag changes since event 553 >> 2008-07-22 19:46:53,430 [DEBUG] koji.repo.manager: order: [] >> 2008-07-22 19:46:53,430 [DEBUG] koji.repo.manager: repo 534: tag=2, >> state=INIT >> 2008-07-22 19:46:53,430 [DEBUG] koji.repo.manager: repo 535: tag=2, >> state=INIT >> 2008-07-22 19:46:53,430 [DEBUG] koji.repo.manager: task 548 for tag 2 >> 2008-07-22 19:46:53,430 [DEBUG] koji.repo.manager: Scanning filesystem >> for repos >> 2008-07-22 19:47:09,590 [INFO] koji.repo.manager: Problem: newRepo task >> 548 for tag 2 is FAILED >> 2008-07-22 19:47:09,591 [DEBUG] koji.repo.manager: Current tasks: {} >> 2008-07-22 19:47:09,591 [DEBUG] koji.repo.manager: Updating repos >> 2008-07-22 19:47:09,591 [DEBUG] koji.repo.manager: Reading current repo data >> 2008-07-22 19:47:15,273 [DEBUG] koji.repo.manager: Repo data: >> [{'create_event': 552, 'state': 0, 'tag_id': 2, 'tag_name': >> 'rutgers-centos5-build', 'create_ts': 1216754980.8750801, 'id': 534}, >> {'create_event': 553, 'state': 0, 'tag_id': 2, 'tag_name': >> 'rutgers-centos5-build', 'create_ts': 1216755215.82616, 'id': 535}, >> {'create_event': 554, 'state': 0, 'tag_id': 2, 'tag_name': >> 'rutgers-centos5-build', 'create_ts': 1216756145.8847401, 'id': 536}] >> 2008-07-22 19:47:15,273 [INFO] koji.repo.manager: Found repo 536, state=INIT >> 2008-07-22 19:47:26,421 [DEBUG] koji.repo.manager: Needed tags: [2] >> 2008-07-22 19:47:26,421 [DEBUG] koji.repo.manager: Current tags: [2] >> 2008-07-22 19:47:26,421 [DEBUG] koji.repo: Checking for changes: [1, 2] >> 2008-07-22 19:47:31,905 [DEBUG] koji.repo: No tag changes since event 554 >> 2008-07-22 19:47:31,905 [DEBUG] koji.repo: Checking for changes: [1, 2] >> >> Any idea? >> >> Thanks. >> >> > > -- > Fedora-buildsys-list mailing list > Fedora-buildsys-list at redhat.com > https://www.redhat.com/mailman/listinfo/fedora-buildsys-list > -- Naveen Gavini Student Systems Programmer OSS/CSS - OIT Rutgers ngavini at jla.rutgers.edu From sergio at sergiomb.no-ip.org Tue Jul 22 21:18:59 2008 From: sergio at sergiomb.no-ip.org (Sergio Monteiro Basto) Date: Tue, 22 Jul 2008 22:18:59 +0100 Subject: pungi repo priorities In-Reply-To: <1216393982.3391.8.camel@localhost.localdomain> References: <1216351422.26321.39.camel@segulix.localdomain> <1216393982.3391.8.camel@localhost.localdomain> Message-ID: <1216761539.30688.5.camel@segulix.localdomain> On Fri, 2008-07-18 at 11:13 -0400, Jesse Keating wrote: > On Fri, 2008-07-18 at 04:23 +0100, Sergio Monteiro Basto wrote: > > Hi, how I set pungi repo priorities ? > > > > New pungi on fedora 9, give me an error about repo.priorities is a > > NoneType > > I thought I pushed an update for that. Check with either F9 updates or > F9 updates testing. hum a strange thing happens, from http://koji.fedoraproject.org/koji/packageinfo?packageID=3528 I download pungi-1.2.18.1-1.fc9 and try to update with rpm -Uvh pungi-1.2.18.1-1.fc9 but system says that pungi-1.2.18.1 is already install . I remove pungi-1.2.18.1-1.fc9 and install again. And now works, I verify if gather.py has been change and it is changed . seems that pungi-1.2.18.1-1.fc9 chipped on f9 , is not the really pungi-1.2.18.1-1.fc9 Thanks, -- S?rgio M. B. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 2192 bytes Desc: not available URL: From ngavini at jla.rutgers.edu Tue Jul 22 21:27:48 2008 From: ngavini at jla.rutgers.edu (Naveen Gavini) Date: Tue, 22 Jul 2008 17:27:48 -0400 Subject: How init mock in Koji? In-Reply-To: <4885F345.3090001@redhat.com> References: <5b123d6c0807220231o2515986age6dd4691967d39c2@mail.gmail.com> <4885F345.3090001@redhat.com> Message-ID: <488650D4.4060508@jla.rutgers.edu> Mike McLean wrote: > Linul wrote: > >> I have already set my koji server and every component is work OK. >> >> But not have any document mention howto init the buildroot?(or use mock >> environment) >> > > We are having a similar issue with problems on how to init the buildroot. We generated a config using 'koji mock-config' and copied this to site-defaults.cfg and default.cfg but we get the following error: When manually running mock it seems to init properly (we did not test with a package). This is the error with building from an SRPM via koji build: 2008-07-22 21:14:42,845 [WARNING] koji.build.TaskManager: could not init mock buildroot, mock exited with status 30 Is there any further documentation on how to do this step? Thanks for your help. > It's not entirely clear what you're asking, but here goes... > > mock is used by koji-builder to create buildroots. In normal operation, > you submit a build to the system, the builders pick it up and use mock > to create a buildroot and run the build. > > If, on the other hand, you want to run mock manually against the repos > that koji maintains (not normal operation, but useful nonetheless), then > you should just be able to generate a mock config with the 'koji > mock-config' command. You can use that config with a manual run of mock. > Sometimes the config may need a little tweaking. > > If this doesn't answer your question, then perhaps you could provide a > bit more context. > -- Naveen Gavini Student Systems Programmer OSS/CSS - OIT Rutgers ngavini at jla.rutgers.edu From dledford at redhat.com Tue Jul 22 21:26:57 2008 From: dledford at redhat.com (Doug Ledford) Date: Tue, 22 Jul 2008 17:26:57 -0400 Subject: GenericError Repo directory missing In-Reply-To: <48864EED.6020907@jla.rutgers.edu> References: <48863CE6.2010506@jla.rutgers.edu> <1216758486.20652.15.camel@maunalani.home> <48864EED.6020907@jla.rutgers.edu> Message-ID: <1216762017.5216.8.camel@firewall.xsintricity.com> On Tue, 2008-07-22 at 17:19 -0400, Naveen Gavini wrote: > We thought since kojira was creating the > repos directories that would suffice. Mike corrected me on this same thing once. The deal is that kojira doesn't create repos, it only creates repo creation *tasks* which are then picked up by kojid and carried out by kojid. -- Doug Ledford GPG KeyID: CFBFF194 http://people.redhat.com/dledford Infiniband specific RPMs available at http://people.redhat.com/dledford/Infiniband -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: From mikem at redhat.com Tue Jul 22 21:31:24 2008 From: mikem at redhat.com (Mike McLean) Date: Tue, 22 Jul 2008 17:31:24 -0400 Subject: GenericError Repo directory missing In-Reply-To: <48864EED.6020907@jla.rutgers.edu> References: <48863CE6.2010506@jla.rutgers.edu> <1216758486.20652.15.camel@maunalani.home> <48864EED.6020907@jla.rutgers.edu> Message-ID: <488651AC.9010506@redhat.com> Naveen Gavini wrote: >> Does the /mnt/koji/repos directory exist? Is it owned by the same user >> httpd runs as? >> >> Also, have you tried to change topdir to something other than /mnt/koji? >> There are known problem with this, it needs to be fixed. >> > We were able to fix this. The problem seemed to be that we did not have > this directory (/mnt/koji) mounted on the builder, it was only mounted > on the hub which runs kojira. We thought since kojira was creating the > repos directories that would suffice. It also needs to be accessible by > the builder. It depends on how you configure the builder. If you specify a topurl value, them the builder will access the repos via http. Otherwise it will access them via the filesystem under /mnt/koji. From mikem at redhat.com Tue Jul 22 21:37:34 2008 From: mikem at redhat.com (Mike McLean) Date: Tue, 22 Jul 2008 17:37:34 -0400 Subject: How init mock in Koji? In-Reply-To: <488650D4.4060508@jla.rutgers.edu> References: <5b123d6c0807220231o2515986age6dd4691967d39c2@mail.gmail.com> <4885F345.3090001@redhat.com> <488650D4.4060508@jla.rutgers.edu> Message-ID: <4886531E.9040402@redhat.com> Naveen Gavini wrote: > We are having a similar issue with problems on how to init the buildroot. > We generated a config using 'koji mock-config' and copied this to > site-defaults.cfg and default.cfg but we get the following error: > > When manually running mock it seems to init properly (we did not test > with a package). > > This is the error with building from an SRPM via koji build: > > 2008-07-22 21:14:42,845 [WARNING] koji.build.TaskManager: could not init > mock buildroot, mock exited with status 30 You're asking about koji mock-config, but showing me an error from kojid. This makes no sense. The 'koji mock-config' command is a debug tool that allows users to duplicate the mock configuration outside of kojid. kojid itself generates its own mock configuration on the fly (a new one for every build in fact). You should not need any sort of default mock config on the builders for kojid to function. As for the cause of the mock error 30. You should look at the mock logs under /var/lib/mock. If the error is not there, you might try running kojid in the foreground (--fg) so you can see mock's stdout and stderr. From dledford at redhat.com Tue Jul 22 21:45:09 2008 From: dledford at redhat.com (Doug Ledford) Date: Tue, 22 Jul 2008 17:45:09 -0400 Subject: How init mock in Koji? In-Reply-To: <488650D4.4060508@jla.rutgers.edu> References: <5b123d6c0807220231o2515986age6dd4691967d39c2@mail.gmail.com> <4885F345.3090001@redhat.com> <488650D4.4060508@jla.rutgers.edu> Message-ID: <1216763109.5216.20.camel@firewall.xsintricity.com> On Tue, 2008-07-22 at 17:27 -0400, Naveen Gavini wrote: > Mike McLean wrote: > > Linul wrote: > > > >> I have already set my koji server and every component is work OK. > >> > >> But not have any document mention howto init the buildroot?(or use mock > >> environment) > >> > > > > > We are having a similar issue with problems on how to init the buildroot. > We generated a config using 'koji mock-config' and copied this to > site-defaults.cfg and default.cfg but we get the following error: > > When manually running mock it seems to init properly (we did not test > with a package). > > This is the error with building from an SRPM via koji build: > > 2008-07-22 21:14:42,845 [WARNING] koji.build.TaskManager: could not init > mock buildroot, mock exited with status 30 > > Is there any further documentation on how to do this step? > > Thanks for your help. My understanding is the mock fu is supposed to happen all behind the scenes more or less. Which is to say that A) as long as the mock topdir in /etc/kojid/kojid.conf is valid and accessible as kojibuilder (assuming you are running mock as kojibuilder, again settable in kojid.conf) and B) your build group settings are sane (aka, the package list to be installed is valid and will produce a reasonable build environment), then it should all just work. These are the things I had to make sure were correct to get it working on my setup: in /etc/kojid/kojid.conf: mockdir and mockusr I left at the defaults (/var/lib/mock and kojibuilder) pkgurl must be a valid url on your web server and your wer server must point that url to the root of the koji package tree (when yum runs in the mock buildroot, it needs this to retrieve packages) in /etc/kojira/kojira.conf topdir must be correctly set to the same as KojiDir in /etc/httpd/conf.d/kojihub.conf in the koji database: the build target must be valid the build group for the build target must contain valid package names (trying to use a comps group or meta package didn't work for me, although it might be possible to make it work, I don't know, I was unable to do so) those were the major things that had to be correct as I recall, but I could be missing something along the way... > > It's not entirely clear what you're asking, but here goes... > > > > mock is used by koji-builder to create buildroots. In normal operation, > > you submit a build to the system, the builders pick it up and use mock > > to create a buildroot and run the build. > > > > If, on the other hand, you want to run mock manually against the repos > > that koji maintains (not normal operation, but useful nonetheless), then > > you should just be able to generate a mock config with the 'koji > > mock-config' command. You can use that config with a manual run of mock. > > Sometimes the config may need a little tweaking. > > > > If this doesn't answer your question, then perhaps you could provide a > > bit more context. > > > > -- Doug Ledford GPG KeyID: CFBFF194 http://people.redhat.com/dledford Infiniband specific RPMs available at http://people.redhat.com/dledford/Infiniband -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: From mikeb at redhat.com Tue Jul 22 21:58:28 2008 From: mikeb at redhat.com (Mike Bonnet) Date: Tue, 22 Jul 2008 17:58:28 -0400 Subject: GenericError Repo directory missing In-Reply-To: <48864EED.6020907@jla.rutgers.edu> References: <48863CE6.2010506@jla.rutgers.edu> <1216758486.20652.15.camel@maunalani.home> <48864EED.6020907@jla.rutgers.edu> Message-ID: <1216763908.20652.19.camel@maunalani.home> On Tue, 2008-07-22 at 17:19 -0400, Naveen Gavini wrote: > Mike Bonnet wrote: > > On Tue, 2008-07-22 at 16:02 -0400, Naveen Gavini wrote: > > > >> Hello all, > >> > >> We have installed the Koji build system and are able to import RPMS. We > >> are having issues building from SRPMS as it's prompting us that it is > >> unable to find the file once it is uploaded. The other issue is we have > >> seen about 500 tasks fail for newRepo (noarch) with the message: > >> GenericError: Repo directory missing: > >> /mnt/koji/repos/rutgers-centos5-build/538. > >> > > > > Does the /mnt/koji/repos directory exist? Is it owned by the same user > > httpd runs as? > > > > Also, have you tried to change topdir to something other than /mnt/koji? > > There are known problem with this, it needs to be fixed. > > > > > We were able to fix this. The problem seemed to be that we did not have > this directory (/mnt/koji) mounted on the builder, it was only mounted > on the hub which runs kojira. We thought since kojira was creating the > repos directories that would suffice. It also needs to be accessible by > the builder. It is possible to have a builder that doesn't need access to /mnt/koji if you configure topurl, as Mike McLean suggested, and remove that builder from the "createrepo" channel (so it won't take newRepo tasks). However, you always need to have at least one builder that has (read-only) access to /mnt/koji in the "createrepo" channel, or no repos will ever get created/refreshed. > Thanks. > >> We have set the permissions to daemon on /mnt/koji (same user as apache). > >> > >> It seems when we try to manually look at the directory it exists, this > >> is also the same for when the srpm is uploaded and it fails prompting us > >> file not found, even though we can find it by looking at that directory > >> on the server. > >> > >> Below is the kojira.log > >> > >> 2008-07-22 19:46:31,358 [DEBUG] koji.repo.manager: Reading current repo data > >> 2008-07-22 19:46:36,855 [DEBUG] koji.repo.manager: Repo data: > >> [{'create_event': 552, 'state': 0, 'tag_id': 2, 'tag_name': > >> 'rutgers-centos5-build', 'create_ts': 1216754980.8750801, 'id': 534}, > >> {'create_event': 553, 'state': 0, 'tag_id': 2, 'tag_name': > >> 'rutgers-centos5-build', 'create_ts': 1216755215.82616, 'id': 535}] > >> 2008-07-22 19:46:42,399 [DEBUG] koji.repo.manager: Needed tags: [2] > >> 2008-07-22 19:46:42,400 [DEBUG] koji.repo.manager: Current tags: [2] > >> 2008-07-22 19:46:42,400 [DEBUG] koji.repo: Checking for changes: [1, 2] > >> 2008-07-22 19:46:47,902 [DEBUG] koji.repo: No tag changes since event 552 > >> 2008-07-22 19:46:47,902 [DEBUG] koji.repo: Checking for changes: [1, 2] > >> 2008-07-22 19:46:53,429 [DEBUG] koji.repo: No tag changes since event 553 > >> 2008-07-22 19:46:53,430 [DEBUG] koji.repo.manager: order: [] > >> 2008-07-22 19:46:53,430 [DEBUG] koji.repo.manager: repo 534: tag=2, > >> state=INIT > >> 2008-07-22 19:46:53,430 [DEBUG] koji.repo.manager: repo 535: tag=2, > >> state=INIT > >> 2008-07-22 19:46:53,430 [DEBUG] koji.repo.manager: task 548 for tag 2 > >> 2008-07-22 19:46:53,430 [DEBUG] koji.repo.manager: Scanning filesystem > >> for repos > >> 2008-07-22 19:47:09,590 [INFO] koji.repo.manager: Problem: newRepo task > >> 548 for tag 2 is FAILED > >> 2008-07-22 19:47:09,591 [DEBUG] koji.repo.manager: Current tasks: {} > >> 2008-07-22 19:47:09,591 [DEBUG] koji.repo.manager: Updating repos > >> 2008-07-22 19:47:09,591 [DEBUG] koji.repo.manager: Reading current repo data > >> 2008-07-22 19:47:15,273 [DEBUG] koji.repo.manager: Repo data: > >> [{'create_event': 552, 'state': 0, 'tag_id': 2, 'tag_name': > >> 'rutgers-centos5-build', 'create_ts': 1216754980.8750801, 'id': 534}, > >> {'create_event': 553, 'state': 0, 'tag_id': 2, 'tag_name': > >> 'rutgers-centos5-build', 'create_ts': 1216755215.82616, 'id': 535}, > >> {'create_event': 554, 'state': 0, 'tag_id': 2, 'tag_name': > >> 'rutgers-centos5-build', 'create_ts': 1216756145.8847401, 'id': 536}] > >> 2008-07-22 19:47:15,273 [INFO] koji.repo.manager: Found repo 536, state=INIT > >> 2008-07-22 19:47:26,421 [DEBUG] koji.repo.manager: Needed tags: [2] > >> 2008-07-22 19:47:26,421 [DEBUG] koji.repo.manager: Current tags: [2] > >> 2008-07-22 19:47:26,421 [DEBUG] koji.repo: Checking for changes: [1, 2] > >> 2008-07-22 19:47:31,905 [DEBUG] koji.repo: No tag changes since event 554 > >> 2008-07-22 19:47:31,905 [DEBUG] koji.repo: Checking for changes: [1, 2] > >> > >> Any idea? > >> > >> Thanks. > >> > >> > > > > -- > > Fedora-buildsys-list mailing list > > Fedora-buildsys-list at redhat.com > > https://www.redhat.com/mailman/listinfo/fedora-buildsys-list > > > > From jesusr at redhat.com Wed Jul 23 14:19:13 2008 From: jesusr at redhat.com (Jesus M. Rodriguez) Date: Wed, 23 Jul 2008 10:19:13 -0400 Subject: building from git using custom koji install Message-ID: <48873DE1.4040001@redhat.com> We've installed koji (1.2.5) on F8 and successfully got the buildroots generating and scratch builds of src.rpms working. I'm trying to build packages from our git repo. I gave it the url to our repo: git://git.fedorahosted.org/git?spacewalk.git#HEAD But it complains about not having a spec file. Which makes sense as there is no spec file at the top of the tree. The spec file I want to build is actually located in the web directory. http://tinyurl.com/65hs7b I've tried several versions of the url with no luck. It seems that koji expects the scm repository house a single package which is not the case for us. Has anyone else run into a similar situation? Is there a way to specify the subdirectory to look in after doing the git clone? -- jesus m. rodriguez | jesusr at redhat.com sr. software engineer | irc: zeus red hat network | 919.754.4413 (w) rhce # 805008586930012 | 919.623.0080 (c) +-------------------------------------------+ | "Those who cannot learn from history | | are doomed to repeat it." | | -- George Santayana | +-------------------------------------------+ From mikem at redhat.com Wed Jul 23 16:31:51 2008 From: mikem at redhat.com (Mike McLean) Date: Wed, 23 Jul 2008 12:31:51 -0400 Subject: building from git using custom koji install In-Reply-To: <48873DE1.4040001@redhat.com> References: <48873DE1.4040001@redhat.com> Message-ID: <48875CF7.30805@redhat.com> Jesus M. Rodriguez wrote: > I've tried several versions of the url with no luck. It seems > that koji expects the scm repository house a single package > which is not the case for us. There are innumerable ways that one might build an srpm from an SCM checkout. Koji can only support a limited set. Basically, your SCM layout must support koji, not the other way around. From jesusr at redhat.com Wed Jul 23 18:14:16 2008 From: jesusr at redhat.com (Jesus M. Rodriguez) Date: Wed, 23 Jul 2008 14:14:16 -0400 Subject: building from git using custom koji install In-Reply-To: <48875CF7.30805@redhat.com> References: <48873DE1.4040001@redhat.com> <48875CF7.30805@redhat.com> Message-ID: <488774F8.3000605@redhat.com> Mike McLean wrote: > Jesus M. Rodriguez wrote: >> I've tried several versions of the url with no luck. It seems >> that koji expects the scm repository house a single package >> which is not the case for us. > > There are innumerable ways that one might build an srpm from an SCM > checkout. Koji can only support a limited set. Basically, your SCM > layout must support koji, not the other way around. So am I correct in my assumption that koji expects the scm repository to house a single package? Is there a way to build from src.rpm without doing scratch builds? /me couldn't find anything in the docs about this. Thanks jesus -- jesus m. rodriguez | jesusr at redhat.com sr. software engineer | irc: zeus red hat network | 919.754.4413 (w) rhce # 805008586930012 | 919.623.0080 (c) +-------------------------------------------+ | "Those who cannot learn from history | | are doomed to repeat it." | | -- George Santayana | +-------------------------------------------+ From dledford at redhat.com Wed Jul 23 18:16:51 2008 From: dledford at redhat.com (Doug Ledford) Date: Wed, 23 Jul 2008 14:16:51 -0400 Subject: building from git using custom koji install In-Reply-To: <488774F8.3000605@redhat.com> References: <48873DE1.4040001@redhat.com> <48875CF7.30805@redhat.com> <488774F8.3000605@redhat.com> Message-ID: <1216837011.31637.92.camel@firewall.xsintricity.com> On Wed, 2008-07-23 at 14:14 -0400, Jesus M. Rodriguez wrote: > Mike McLean wrote: > > Jesus M. Rodriguez wrote: > >> I've tried several versions of the url with no luck. It seems > >> that koji expects the scm repository house a single package > >> which is not the case for us. > > > > There are innumerable ways that one might build an srpm from an SCM > > checkout. Koji can only support a limited set. Basically, your SCM > > layout must support koji, not the other way around. > > So am I correct in my assumption that koji expects the scm > repository to house a single package? > > Is there a way to build from src.rpm without doing scratch > builds? /me couldn't find anything in the docs about this. If you are an admin user in the koji system then you can build from srpm without doing scratch...but mere mortals aren't allowed that privilege ;-) -- Doug Ledford GPG KeyID: CFBFF194 http://people.redhat.com/dledford Infiniband specific RPMs available at http://people.redhat.com/dledford/Infiniband -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: From ngavini at jla.rutgers.edu Wed Jul 23 19:56:47 2008 From: ngavini at jla.rutgers.edu (Naveen Gavini) Date: Wed, 23 Jul 2008 15:56:47 -0400 Subject: package upload/import speed Message-ID: <48878CFF.1000701@jla.rutgers.edu> Hello all, When importing or building RPM's from source is there an option that is throttling the upload speed of the package? We are seeing speeds of 10kbps which makes importing/building large packages very slow. Thanks. -- Naveen Gavini Student Systems Programmer OSS/CSS - OIT Rutgers ngavini at jla.rutgers.edu From bryce at zeniv.linux.org.uk Wed Jul 23 20:29:20 2008 From: bryce at zeniv.linux.org.uk (Bryce) Date: Wed, 23 Jul 2008 21:29:20 +0100 Subject: package upload/import speed In-Reply-To: <48878CFF.1000701@jla.rutgers.edu> References: <48878CFF.1000701@jla.rutgers.edu> Message-ID: <488794A0.5010100@zeniv.linux.org.uk> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Naveen Gavini wrote: > Hello all, > > When importing or building RPM's from source is there an option that is > throttling the upload speed of the package? > We are seeing speeds of 10kbps which makes importing/building large > packages very slow. > > Thanks. > On a local machine (where client and server, hub etc all reside) I get 140K/s (about x14 what you're seeing) but that seems to be an artificial cap given that it's invariant no matter what the loading on the machine is. Ie it's not CPU bound. Unfortunately it's a royal PITA for scratch building since that means large src rpms like the kernel can take up to 5 minutes just to be 'uploaded' on the same damned box. To be honest this is something I've been scratching my head over as well. Various slim picked information I've gathered tries to point the finger of slowness at SSL itself for the encryption speed and suggest getting hold of SSL HW accelerators but I don't buy that argument due to the constant transfer speed. I haven't been able to decide if it's a client side issue or a server side issue though 8/ Phil =--= -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFIh5SgPcECCTpcVR0RAjZwAJ9s0giafZQmRHtxdyOefvq+QktiCgCfeFe+ hoOV7CUIucOTbZI1PTtFgDs= =gMPu -----END PGP SIGNATURE----- From ngavini at jla.rutgers.edu Wed Jul 23 20:33:58 2008 From: ngavini at jla.rutgers.edu (Naveen Gavini) Date: Wed, 23 Jul 2008 16:33:58 -0400 Subject: Problems using import --link Message-ID: <488795B6.2050205@jla.rutgers.edu> Hello all, We are trying to bootstrap our new koji setup and when we attempt to import using --link we get the error below. The packages that we are attempting to import are on the hub and have been places in /mnt/koji/import (same volume). The ownership on the packages is daemon (same as /mnt/koji/) - not sure if this matters. sudo koji import --link /mnt/koji/alsa-lib-1.0.14-1.rc4.el5.src.rpm Linking rpm to: /mnt/koji/work/cli-import/1216842099.8955669.PLyYLFZS/alsa-lib-1.0.14-1.rc4.el5.src.rpm importing alsa-lib-1.0.14-1.rc4.el5.src.rpm... Fault: \n parent = None\n config = {\'PythonAutoReload\': \'0\', \'PythonDebug\': \'1\'}\n self = \n cache = \n default_handler = handler\n phase = PythonHandler\n handler = kojixmlrpc\n hlist = {\'handler\':\'kojixmlrpc\',\'directory\':\'/usr/share/koji-hub/\',\'silent\':0}\n aborted = False\n directory = /usr/share/koji-hub/\n root = /usr/share/koji-hub/\n options = {\'KojiTraceback\': \'extended\', \'KojiDebug\': \'On\', \'EmailDomain\': \'192.168.226.61\', \'KojiWebURL\': \'http://192.168.226.61/koji\ ', \'LoginCreatesUser\': \'Off\', \'ProxyDNs\': \'/C=US/ST=New Jersey/O=Open System Solutions/CN=192.168.226.61\', \'DNUsernameComponent\': \'CN\', \'KojiDir\': \'/mnt/koji\', \'DBHost\': \'localhost\', \'DBUser\': \'koji\', \'DBName\': \'koji\'}\n result = 500\nFrame _process_target in /usr/lib64/python2.4/site-packages/mod_python/importer.py at line 1229\n req = \n silent = 0\n default = handler\n handler = kojixmlrpc\n object = \n module = \n directory = /usr/share/koji-hub/\n parts = [\'kojixmlrpc\']\n result = -1\n object_str = handler\n expected = [-1, 0]\n module_name = kojixmlrpc\n path = [\'/usr/share/koji-hub/\']\n config = {\'PythonAutoReload\': \'0\', \'PythonDebug\': \'1\'}\n arg = \nFrame _execute_target in /usr/lib64/python2.4/site-packages/mod_python/importer.py at line 1128\n one_process = False\n object = \n req = \n pdb_debug = 0\n arg = \n config = {\'PythonAutoReload\': \'0\', \'PythonDebug\': \'1\'}\nFrame handler in /usr/share/koji-hub/kojixmlrpc.py at line 291\n functions = <_mp_4e76479ee7b716cdbf29397025ee2356.RootExports object at 0x2b349c6f08d0>\n profiling = False\n h = <_mp_df1c13776110deb1e020020636703560.ModXMLRPCRequestHandler object at 0x2b349c6f0d10>\n req = \n hostFunctions = <_mp_4e76479ee7b716cdbf29397025ee2356.HostExports object at 0x2b349c6f0d90>\n opts = {\'KojiTraceback\': \'extended\', \'KojiDebug\': \'On\', \'EmailDomain\': \'192.168.226.61\', \'KojiWebURL\': \'http://192.168.226.61/koji\ ', \'LoginCreatesUser\': \'Off\', \'ProxyDNs\': \'/C=US/ST=New Jersey/O=Open System Solutions/CN=192.168.226.61\', \'DNUsernameComponent\': \'CN\', \'KojiDir\': \'/mnt/koji\', \'DBHost\': \'localhost\', \'DBUser\': \'koji\', \'DBName\': \'koji\'}\nFrame handle_request in /usr/share/koji-hub/kojixmlrpc.py at line 242\n self = <_mp_df1c13776110deb1e020020636703560.ModXMLRPCRequestHandler object at 0x2b349c6f0d10>\n req = \nFrame _marshaled_dispatch in /usr/share/koji-hub/kojixmlrpc.py at line 112\n e = [Errno 13] Permission denied\n self = <_mp_df1c13776110deb1e020020636703560.ModXMLRPCRequestHandler object at 0x2b349c6f0d10>\n e_class = exceptions.OSError\n start = 1216842100.35\n faultCode = 1\n params = \n tb_str = Traceback (most recent call last):\n File "/usr/share/koji-hub/kojixmlrpc.py", line 86, in _marshaled_dispatch\n response = self._dispatch(method, params)\n File "/usr/share/koji-hub/kojixmlrpc.py", line 156, in _dispatch\n ret = func(*params,**opts)\n File "/usr/share/koji-hub/kojihub.py", line 4110, in importRPM\n import_rpm_file(fn,rpminfo[\'build\'],rpminfo)\n File "/usr/share/koji-hub/kojihub.py", line 2991, in import_rpm_file\n os.rename(fn,final_path)\nOSError: [Errno 13] Permission denied\n\n tb_type = extended\n data = \n\nimportRPM\n\n\ncli-import/1216842099.8955669.PLyYLFZS\n\n\nalsa-lib-1.0.14-1.rc4.el5.src.rpm\n\n\n\n\n method = importRPM\nFrame _dispatch in /usr/share/koji-hub/kojixmlrpc.py at line 156\n self = <_mp_df1c13776110deb1e020020636703560.ModXMLRPCRequestHandler object at 0x2b349c6f0d10>\n start = 1216842100.38\n params = \n func = >\n method = importRPM\n opts = {}\nFrame importRPM in /usr/share/koji-hub/kojihub.py at line 4110\n uploadpath = /mnt/koji/work\n basename = alsa-lib-1.0.14-1.rc4.el5.src.rpm\n rpminfo = {\'build_id\': 20L, \'brootid\': None, \'sourcepackage\': 1, \'name\': \'alsa-lib\', \'buildtime\': 1194704966, \'id\': 28L, \'epoch\': None, \'version\': \'1.0.14\', \'build\': {\'sourcepackage\': 1, \'name\': \'alsa-lib\', \'buildtime\': 1194704966, \'epoch\': None, \'version\': \'1.0.14\', \'release\': \'1.rc4.el5\', \'sourcerpm\': [], \'arch\': \'src\', \'id\': 20L}, \'release\': \'1.rc4.el5\', \'sourcerpm\': [], \'arch\': \'src\', \'payloadhash\': \'8fa4adb63144d52fc1ceb29f6022b7cb\', \'size\': 796510}\n path = cli-import/1216842099.8955669.PLyYLFZS\n self = <_mp_4e76479ee7b716cdbf29397025ee2356.RootExports object at 0x2b349c6f08d0>\n fn = /mnt/koji/work/cli-import/1216842099.8955669.PLyYLFZS/alsa-lib-1.0.14-1.rc4.el5.src.rpm\nFrame import_rpm_file in /usr/share/koji-hub/kojihub.py at line 2991\n final_path = /mnt/koji/packages/alsa-lib/1.0.14/1.rc4.el5/src/alsa-lib-1.0.14-1.rc4.el5.src.rpm\n rpminfo = {\'build_id\': 20L, \'brootid\': None, \'sourcepackage\': 1, \'name\': \'alsa-lib\', \'buildtime\': 1194704966, \'id\': 28L, \'epoch\': None, \'version\': \'1.0.14\', \'build\': {\'sourcepackage\': 1, \'name\': \'alsa-lib\', \'buildtime\': 1194704966, \'epoch\': None, \'version\': \'1.0.14\', \'release\': \'1.rc4.el5\', \'sourcerpm\': [], \'arch\': \'src\', \'id\': 20L}, \'release\': \'1.rc4.el5\', \'sourcerpm\': [], \'arch\': \'src\', \'payloadhash\': \'8fa4adb63144d52fc1ceb29f6022b7cb\', \'size\': 796510}\n buildinfo = {\'sourcepackage\': 1, \'name\': \'alsa-lib\', \'buildtime\': 1194704966, \'epoch\': None, \'version\': \'1.0.14\', \'release\': \'1.rc4.el5\', \'sourcerpm\': [], \'arch\': \'src\', \'id\': 20L}\n fn = /mnt/koji/work/cli-import/1216842099.8955669.PLyYLFZS/alsa-lib-1.0.14-1.rc4.el5.src.rpm\n'> Any ideas? Thanks. -- Naveen Gavini Student Systems Programmer OSS/CSS - OIT Rutgers ngavini at jla.rutgers.edu From mikeb at redhat.com Wed Jul 23 20:34:01 2008 From: mikeb at redhat.com (Mike Bonnet) Date: Wed, 23 Jul 2008 16:34:01 -0400 Subject: package upload/import speed In-Reply-To: <48878CFF.1000701@jla.rutgers.edu> References: <48878CFF.1000701@jla.rutgers.edu> Message-ID: <1216845242.12744.15.camel@maunalani.home> On Wed, 2008-07-23 at 15:56 -0400, Naveen Gavini wrote: > Hello all, > > When importing or building RPM's from source is there an option that is > throttling the upload speed of the package? > We are seeing speeds of 10kbps which makes importing/building large > packages very slow. Package upload happens via xml-rpc, and to avoid holding the entire contents of the file in memory, it's done in chunks. Unfortunately python xml-rpc libs don't support keep-alive, so if you're using SSL authentication you incur the non-trivial SSL connection setup/teardown overhead for every chunk, in addition to the TCP connection setup/teardown. The chunk size in your version is probably very small (64k?). That has been increased in the Koji repo, but not released yet. The next version will improve this situation somewhat. From mikeb at redhat.com Wed Jul 23 20:46:50 2008 From: mikeb at redhat.com (Mike Bonnet) Date: Wed, 23 Jul 2008 16:46:50 -0400 Subject: Problems using import --link In-Reply-To: <488795B6.2050205@jla.rutgers.edu> References: <488795B6.2050205@jla.rutgers.edu> Message-ID: <1216846010.12744.25.camel@maunalani.home> On Wed, 2008-07-23 at 16:33 -0400, Naveen Gavini wrote: > Hello all, > > We are trying to bootstrap our new koji setup and when we attempt to > import using --link we get the error below. > The packages that we are attempting to import are on the hub and have > been places in /mnt/koji/import (same volume). > The ownership on the packages is daemon (same as /mnt/koji/) - not sure > if this matters. > > > sudo koji import --link /mnt/koji/alsa-lib-1.0.14-1.rc4.el5.src.rpm > > Linking rpm to: > /mnt/koji/work/cli-import/1216842099.8955669.PLyYLFZS/alsa-lib-1.0.14-1.rc4.el5.src.rpm > > importing alsa-lib-1.0.14-1.rc4.el5.src.rpm... Fault: 'Traceback (most recent call last):\n File > "/usr/share/koji-hub/kojixmlrpc.py", line 86, in _marshaled_dispatch\n > response = self._dispatch(method, params)\n File > "/usr/share/koji-hub/kojixmlrpc.py", line 156, in _dispatch\n ret = > func(*params,**opts)\n File "/usr/share/koji-hub/kojihub.py", line 4110, > in importRPM\n import_rpm_file(fn,rpminfo[\'build\'],rpminfo)\n File > "/usr/share/koji-hub/kojihub.py", line 2991, in import_rpm_file\n > os.rename(fn,final_path)\nOSError: [Errno 13] Permission denied\nLocals > by frame, innermost last\nFrame HandlerDispatch in > /usr/lib64/python2.4/site-packages/mod_python/importer.py at line 1537\n > req = \n parent = None\n config = > {\'PythonAutoReload\': \'0\', \'PythonDebug\': \'1\'}\n self = > \n cache = WHILE PRINTING VALUE>\n default_handler = handler\n phase = > PythonHandler\n handler = kojixmlrpc\n hlist = > {\'handler\':\'kojixmlrpc\',\'directory\':\'/usr/share/koji-hub/\',\'silent\':0}\n > aborted = False\n directory = /usr/share/koji-hub/\n root = > /usr/share/koji-hub/\n options = {\'KojiTraceback\': \'extended\', > \'KojiDebug\': \'On\', \'EmailDomain\': \'192.168.226.61\', > \'KojiWebURL\': \'http://192.168.226.61/koji\ > ', \'LoginCreatesUser\': \'Off\', > \'ProxyDNs\': \'/C=US/ST=New Jersey/O=Open System > Solutions/CN=192.168.226.61\', \'DNUsernameComponent\': \'CN\', > \'KojiDir\': \'/mnt/koji\', \'DBHost\': \'localhost\', \'DBUser\': > \'koji\', \'DBName\': \'koji\'}\n result = 500\nFrame _process_target in > /usr/lib64/python2.4/site-packages/mod_python/importer.py at line 1229\n > req = \n silent = 0\n default = > handler\n handler = kojixmlrpc\n object = 0x2b34997152a8>\n module = \'_mp_df1c13776110deb1e020020636703560\' from > \'/usr/share/koji-hub/kojixmlrpc.py\'>\n directory = > /usr/share/koji-hub/\n parts = [\'kojixmlrpc\']\n result = -1\n > object_str = handler\n expected = [-1, 0]\n module_name = kojixmlrpc\n > path = [\'/usr/share/koji-hub/\']\n config = {\'PythonAutoReload\': > \'0\', \'PythonDebug\': \'1\'}\n arg = 0x2b3497cbd410>\nFrame _execute_target in > /usr/lib64/python2.4/site-packages/mod_python/importer.py at line 1128\n > one_process = False\n object = \n > req = \n pdb_debug = 0\n arg = > \n config = {\'PythonAutoReload\': > \'0\', \'PythonDebug\': \'1\'}\nFrame handler in > /usr/share/koji-hub/kojixmlrpc.py at line 291\n functions = > <_mp_4e76479ee7b716cdbf29397025ee2356.RootExports object at > 0x2b349c6f08d0>\n profiling = False\n h = > <_mp_df1c13776110deb1e020020636703560.ModXMLRPCRequestHandler object at > 0x2b349c6f0d10>\n req = \n > hostFunctions = <_mp_4e76479ee7b716cdbf29397025ee2356.HostExports object > at 0x2b349c6f0d90>\n opts = {\'KojiTraceback\': \'extended\', > \'KojiDebug\': \'On\', \'EmailDomain\': \'192.168.226.61\', > \'KojiWebURL\': \'http://192.168.226.61/koji\ > ', \'LoginCreatesUser\': \'Off\', > \'ProxyDNs\': \'/C=US/ST=New Jersey/O=Open System > Solutions/CN=192.168.226.61\', \'DNUsernameComponent\': \'CN\', > \'KojiDir\': \'/mnt/koji\', \'DBHost\': \'localhost\', \'DBUser\': > \'koji\', \'DBName\': \'koji\'}\nFrame handle_request in > /usr/share/koji-hub/kojixmlrpc.py at line 242\n self = > <_mp_df1c13776110deb1e020020636703560.ModXMLRPCRequestHandler object at > 0x2b349c6f0d10>\n req = \nFrame > _marshaled_dispatch in /usr/share/koji-hub/kojixmlrpc.py at line 112\n e > = [Errno 13] Permission denied\n self = > <_mp_df1c13776110deb1e020020636703560.ModXMLRPCRequestHandler object at > 0x2b349c6f0d10>\n e_class = exceptions.OSError\n start = 1216842100.35\n > faultCode = 1\n params = \n tb_str = > Traceback (most recent call last):\n File > "/usr/share/koji-hub/kojixmlrpc.py", line 86, in _marshaled_dispatch\n > response = self._dispatch(method, params)\n File > "/usr/share/koji-hub/kojixmlrpc.py", line 156, in _dispatch\n ret = > func(*params,**opts)\n File "/usr/share/koji-hub/kojihub.py", line 4110, > in importRPM\n import_rpm_file(fn,rpminfo[\'build\'],rpminfo)\n File > "/usr/share/koji-hub/kojihub.py", line 2991, in import_rpm_file\n > os.rename(fn,final_path)\nOSError: [Errno 13] Permission denied\n\n > tb_type = extended\n data = version=\'1.0\'?>\n\nimportRPM\n\n\ncli-import/1216842099.8955669.PLyYLFZS\n\n\nalsa-lib-1.0.14-1.rc4.el5.src.rpm\n\n\n\n\n > method = importRPM\nFrame _dispatch in /usr/share/koji-hub/kojixmlrpc.py > at line 156\n self = > <_mp_df1c13776110deb1e020020636703560.ModXMLRPCRequestHandler object at > 0x2b349c6f0d10>\n start = 1216842100.38\n params = VALUE>\n func = <_mp_4e76479ee7b716cdbf29397025ee2356.RootExports object at > 0x2b349c6f08d0>>\n method = importRPM\n opts = {}\nFrame importRPM in > /usr/share/koji-hub/kojihub.py at line 4110\n uploadpath = > /mnt/koji/work\n basename = alsa-lib-1.0.14-1.rc4.el5.src.rpm\n rpminfo > = {\'build_id\': 20L, \'brootid\': None, \'sourcepackage\': 1, \'name\': > \'alsa-lib\', \'buildtime\': 1194704966, \'id\': 28L, \'epoch\': None, > \'version\': \'1.0.14\', \'build\': {\'sourcepackage\': 1, \'name\': > \'alsa-lib\', \'buildtime\': 1194704966, \'epoch\': None, \'version\': > \'1.0.14\', \'release\': \'1.rc4.el5\', \'sourcerpm\': [], \'arch\': > \'src\', \'id\': 20L}, \'release\': \'1.rc4.el5\', \'sourcerpm\': [], > \'arch\': \'src\', \'payloadhash\': > \'8fa4adb63144d52fc1ceb29f6022b7cb\', \'size\': 796510}\n path = > cli-import/1216842099.8955669.PLyYLFZS\n self = > <_mp_4e76479ee7b716cdbf29397025ee2356.RootExports object at > 0x2b349c6f08d0>\n fn = > /mnt/koji/work/cli-import/1216842099.8955669.PLyYLFZS/alsa-lib-1.0.14-1.rc4.el5.src.rpm\nFrame > import_rpm_file in /usr/share/koji-hub/kojihub.py at line 2991\n > final_path = > /mnt/koji/packages/alsa-lib/1.0.14/1.rc4.el5/src/alsa-lib-1.0.14-1.rc4.el5.src.rpm\n > rpminfo = {\'build_id\': 20L, \'brootid\': None, \'sourcepackage\': 1, > \'name\': \'alsa-lib\', \'buildtime\': 1194704966, \'id\': 28L, > \'epoch\': None, \'version\': \'1.0.14\', \'build\': {\'sourcepackage\': > 1, \'name\': \'alsa-lib\', \'buildtime\': 1194704966, \'epoch\': None, > \'version\': \'1.0.14\', \'release\': \'1.rc4.el5\', \'sourcerpm\': [], > \'arch\': \'src\', \'id\': 20L}, \'release\': \'1.rc4.el5\', > \'sourcerpm\': [], \'arch\': \'src\', \'payloadhash\': > \'8fa4adb63144d52fc1ceb29f6022b7cb\', \'size\': 796510}\n buildinfo = > {\'sourcepackage\': 1, \'name\': \'alsa-lib\', \'buildtime\': > 1194704966, \'epoch\': None, \'version\': \'1.0.14\', \'release\': > \'1.rc4.el5\', \'sourcerpm\': [], \'arch\': \'src\', \'id\': 20L}\n fn = > /mnt/koji/work/cli-import/1216842099.8955669.PLyYLFZS/alsa-lib-1.0.14-1.rc4.el5.src.rpm\n'> Embedded in that horribly mangled stack trace was the root of the problem: File "/usr/share/koji-hub/kojihub.py", line 2991, in import_rpm_file os.rename(fn,final_path) OSError: [Errno 13] Permission denied The problem is that "koji import --link", while very useful, is kind of a hack. In order for it to work, is assumes your httpd is running as uid 48 (the "apache" user on Fedora) and chown's the directory under /mnt/koji/work/cli-import to uid 48 so httpd can rename() the file from that dir to it's final location under /mnt/koji/packages successfully. In your case, httpd is running under a different uid, so that rename fails. To fix this, I would do 2 things: 1. In /usr/bin/koji, change the last 2 numbers in the line os.chown(os.path.dirname(dst), 48, 48) #XXX - hack to the uid of the "daemon" user (whatever user httpd runs as). 2. Make sure all packages under /mnt/koji/import are owned by the daemon user before starting the import. Give that a try, and let me know if you're still running in to errors. From dledford at redhat.com Wed Jul 23 20:47:09 2008 From: dledford at redhat.com (Doug Ledford) Date: Wed, 23 Jul 2008 16:47:09 -0400 Subject: Problems using import --link In-Reply-To: <488795B6.2050205@jla.rutgers.edu> References: <488795B6.2050205@jla.rutgers.edu> Message-ID: <1216846029.4504.1.camel@firewall.xsintricity.com> On Wed, 2008-07-23 at 16:33 -0400, Naveen Gavini wrote: > Hello all, > > We are trying to bootstrap our new koji setup and when we attempt to > import using --link we get the error below. > The packages that we are attempting to import are on the hub and have > been places in /mnt/koji/import (same volume). > The ownership on the packages is daemon (same as /mnt/koji/) - not sure > if this matters. > > > sudo koji import --link /mnt/koji/alsa-lib-1.0.14-1.rc4.el5.src.rpm > > Linking rpm to: > /mnt/koji/work/cli-import/1216842099.8955669.PLyYLFZS/alsa-lib-1.0.14-1.rc4.el5.src.rpm > > os.rename(fn,final_path)\nOSError: [Errno 13] Permission denied Did you make sure that the entire tree under /mnt/koji is owned by user apache and group apache (or whatever user/group your web server runs as)? The xmlrpc code runs as the web user, so it needs permissions on this tree. -- Doug Ledford GPG KeyID: CFBFF194 http://people.redhat.com/dledford Infiniband specific RPMs available at http://people.redhat.com/dledford/Infiniband -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: From a.badger at gmail.com Wed Jul 23 21:03:59 2008 From: a.badger at gmail.com (Toshio Kuratomi) Date: Wed, 23 Jul 2008 14:03:59 -0700 Subject: package upload/import speed In-Reply-To: <1216845242.12744.15.camel@maunalani.home> References: <48878CFF.1000701@jla.rutgers.edu> <1216845242.12744.15.camel@maunalani.home> Message-ID: <48879CBF.4060904@gmail.com> Mike Bonnet wrote: > On Wed, 2008-07-23 at 15:56 -0400, Naveen Gavini wrote: >> Hello all, >> >> When importing or building RPM's from source is there an option that is >> throttling the upload speed of the package? >> We are seeing speeds of 10kbps which makes importing/building large >> packages very slow. > > Package upload happens via xml-rpc, and to avoid holding the entire > contents of the file in memory, it's done in chunks. Unfortunately > python xml-rpc libs don't support keep-alive, so if you're using SSL > authentication you incur the non-trivial SSL connection setup/teardown > overhead for every chunk, in addition to the TCP connection > setup/teardown. The chunk size in your version is probably very small > (64k?). That has been increased in the Koji repo, but not released yet. > The next version will improve this situation somewhat. > Any chance we could port to pycurl which does support keep-alive? Or is urllib pretty deeply embedded in the code ATM? -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: OpenPGP digital signature URL: From mikeb at redhat.com Wed Jul 23 23:02:55 2008 From: mikeb at redhat.com (Mike Bonnet) Date: Wed, 23 Jul 2008 19:02:55 -0400 Subject: package upload/import speed In-Reply-To: <48879CBF.4060904@gmail.com> References: <48878CFF.1000701@jla.rutgers.edu> <1216845242.12744.15.camel@maunalani.home> <48879CBF.4060904@gmail.com> Message-ID: <1216854175.12744.29.camel@maunalani.home> On Wed, 2008-07-23 at 14:03 -0700, Toshio Kuratomi wrote: > Mike Bonnet wrote: > > On Wed, 2008-07-23 at 15:56 -0400, Naveen Gavini wrote: > >> Hello all, > >> > >> When importing or building RPM's from source is there an option that is > >> throttling the upload speed of the package? > >> We are seeing speeds of 10kbps which makes importing/building large > >> packages very slow. > > > > Package upload happens via xml-rpc, and to avoid holding the entire > > contents of the file in memory, it's done in chunks. Unfortunately > > python xml-rpc libs don't support keep-alive, so if you're using SSL > > authentication you incur the non-trivial SSL connection setup/teardown > > overhead for every chunk, in addition to the TCP connection > > setup/teardown. The chunk size in your version is probably very small > > (64k?). That has been increased in the Koji repo, but not released yet. > > The next version will improve this situation somewhat. > > > Any chance we could port to pycurl which does support keep-alive? Or is > urllib pretty deeply embedded in the code ATM? I don't know anything about pycurl, but the transport layer is decently separated from the XML-RPC layer in xmlrpclib, so it should be possible to replace that it with a transport that supports SSL and keep-alive. I haven't looked at how much work this might be though. > -Toshio > > -- > Fedora-buildsys-list mailing list > Fedora-buildsys-list at redhat.com > https://www.redhat.com/mailman/listinfo/fedora-buildsys-list From mikem at redhat.com Thu Jul 24 01:50:13 2008 From: mikem at redhat.com (Mike McLean) Date: Wed, 23 Jul 2008 21:50:13 -0400 Subject: building from git using custom koji install In-Reply-To: <488774F8.3000605@redhat.com> References: <48873DE1.4040001@redhat.com> <48875CF7.30805@redhat.com> <488774F8.3000605@redhat.com> Message-ID: <4887DFD5.5070804@redhat.com> Jesus M. Rodriguez wrote: > So am I correct in my assumption that koji expects the scm > repository to house a single package? Yes. Furthermore, koji assumes that the simple command 'make srpm' issued from the checkout will create a single source rpm. There is not enough information passed to koji for it to handle anything otherwise. In your SCM layout with multiple packages, how do you generate the different srpms? Is there a real reason that they are all in one module? > Is there a way to build from src.rpm without doing scratch > builds? /me couldn't find anything in the docs about this. Some of my current work in progress will allow this to be configurable. Look for this to land in the next few weeks. At present, as Doug points out, there is an exception for admins. From jesusr at redhat.com Thu Jul 24 02:27:24 2008 From: jesusr at redhat.com (Jesus Rodriguez) Date: Wed, 23 Jul 2008 22:27:24 -0400 Subject: building from git using custom koji install In-Reply-To: <4887DFD5.5070804@redhat.com> References: <48873DE1.4040001@redhat.com> <48875CF7.30805@redhat.com> <488774F8.3000605@redhat.com> <4887DFD5.5070804@redhat.com> Message-ID: <20080724022724.GB4817@transam.devel.redhat.com> On Wed, Jul 23, 2008 at 09:50:13PM -0400, Mike McLean wrote: > Jesus M. Rodriguez wrote: >> So am I correct in my assumption that koji expects the scm >> repository to house a single package? > > Yes. Furthermore, koji assumes that the simple command 'make srpm' > issued from the checkout will create a single source rpm. There is not > enough information passed to koji for it to handle anything otherwise. > > In your SCM layout with multiple packages, how do you generate the > different srpms? Is there a real reason that they are all in one module? We generate different srpms with the Makefile that exists in each subdirectory. For example, if you want an srpm for the web module cd spacewalk.git/web; make srpm Want one for the java code, then cd spacewalk.git/java. With SVN it seems completely doable because I could give the path to repo/spacewalk/java and have koji checkout just that directory. Seems to be a limitation of GIT in this case (or we're using it wrong) :) In theory, if there was a way I could tell koji to pull from this git repo, and cd into this directory, you'll find the spec file and Makefile there :) Is there a real reason for them to be all in one module? Because this is how we had it in SVN, then switched to git. :) >> Is there a way to build from src.rpm without doing scratch >> builds? /me couldn't find anything in the docs about this. > > Some of my current work in progress will allow this to be configurable. > Look for this to land in the next few weeks. > > At present, as Doug points out, there is an exception for admins. For our purposes, we might go the route of the admins. -- jesus m. rodriguez | jesusr at redhat.com sr. software engineer | irc: zeus red hat network | 919.754.4413 (w) rhce # 805008586930012 | 919.623.0080 (c) +-------------------------------------------+ | "Those who cannot learn from history | | are doomed to repeat it." | | -- George Santayana | +-------------------------------------------+ From dledford at redhat.com Thu Jul 24 03:06:42 2008 From: dledford at redhat.com (Doug Ledford) Date: Wed, 23 Jul 2008 23:06:42 -0400 Subject: building from git using custom koji install In-Reply-To: <20080724022724.GB4817@transam.devel.redhat.com> References: <48873DE1.4040001@redhat.com> <48875CF7.30805@redhat.com> <488774F8.3000605@redhat.com> <4887DFD5.5070804@redhat.com> <20080724022724.GB4817@transam.devel.redhat.com> Message-ID: <1216868802.4504.32.camel@firewall.xsintricity.com> On Wed, 2008-07-23 at 22:27 -0400, Jesus Rodriguez wrote: > On Wed, Jul 23, 2008 at 09:50:13PM -0400, Mike McLean wrote: > > Jesus M. Rodriguez wrote: > >> So am I correct in my assumption that koji expects the scm > >> repository to house a single package? > > > > Yes. Furthermore, koji assumes that the simple command 'make srpm' > > issued from the checkout will create a single source rpm. There is not > > enough information passed to koji for it to handle anything otherwise. > > > > In your SCM layout with multiple packages, how do you generate the > > different srpms? Is there a real reason that they are all in one module? > > We generate different srpms with the Makefile that exists in each > subdirectory. For example, if you want an srpm for the web module > cd spacewalk.git/web; make srpm > > Want one for the java code, then cd spacewalk.git/java. > > With SVN it seems completely doable because I could give the > path to repo/spacewalk/java and have koji checkout just that > directory. Seems to be a limitation of GIT in this case > (or we're using it wrong) :) Well, git was intentionally written to be basically a state machine of the entire project. So, housing more than one project in a single git repo isn't wrong, but it does tie your various projects together at the state machine level. This is why you can't checkout just one subdirectory from git, you have to grab the entire project. I think I heard mumblings of someone wanting to make subprojects work nicely in git, but I haven't heard if it ever got off the ground. Regardless though, koji won't work with what your current SCM layout without modification to the koji code. -- Doug Ledford GPG KeyID: CFBFF194 http://people.redhat.com/dledford Infiniband specific RPMs available at http://people.redhat.com/dledford/Infiniband -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: From mikeb at redhat.com Thu Jul 24 03:51:36 2008 From: mikeb at redhat.com (Mike Bonnet) Date: Wed, 23 Jul 2008 23:51:36 -0400 Subject: building from git using custom koji install In-Reply-To: <1216868802.4504.32.camel@firewall.xsintricity.com> References: <48873DE1.4040001@redhat.com> <48875CF7.30805@redhat.com> <488774F8.3000605@redhat.com> <4887DFD5.5070804@redhat.com> <20080724022724.GB4817@transam.devel.redhat.com> <1216868802.4504.32.camel@firewall.xsintricity.com> Message-ID: <1216871496.12744.38.camel@maunalani.home> On Wed, 2008-07-23 at 23:06 -0400, Doug Ledford wrote: > On Wed, 2008-07-23 at 22:27 -0400, Jesus Rodriguez wrote: > > On Wed, Jul 23, 2008 at 09:50:13PM -0400, Mike McLean wrote: > > > Jesus M. Rodriguez wrote: > > >> So am I correct in my assumption that koji expects the scm > > >> repository to house a single package? > > > > > > Yes. Furthermore, koji assumes that the simple command 'make srpm' > > > issued from the checkout will create a single source rpm. There is not > > > enough information passed to koji for it to handle anything otherwise. > > > > > > In your SCM layout with multiple packages, how do you generate the > > > different srpms? Is there a real reason that they are all in one module? > > > > We generate different srpms with the Makefile that exists in each > > subdirectory. For example, if you want an srpm for the web module > > cd spacewalk.git/web; make srpm > > > > Want one for the java code, then cd spacewalk.git/java. > > > > With SVN it seems completely doable because I could give the > > path to repo/spacewalk/java and have koji checkout just that > > directory. Seems to be a limitation of GIT in this case > > (or we're using it wrong) :) > > Well, git was intentionally written to be basically a state machine of > the entire project. So, housing more than one project in a single git > repo isn't wrong, but it does tie your various projects together at the > state machine level. This is why you can't checkout just one > subdirectory from git, you have to grab the entire project. I think I > heard mumblings of someone wanting to make subprojects work nicely in > git, but I haven't heard if it ever got off the ground. > > Regardless though, koji won't work with what your current SCM layout > without modification to the koji code. While that's true, I'm not sure we're definitely handling things correctly in the git case. The git support in Koji hasn't gotten a lot of testing, and there are a lot of ways to setup a git repo. However, I think that multiple projects in the same repo is probably something we want to support (since we support it in cvs and svn). The SCM URL you pass to Koji has a format of: scheme://[user@]host/path/to/repo?path/to/module#revision_or_tag_identifier With git we combine the /path/to/repo and the path/to/module into a single URL we pass to "git clone", essentially collapsing them both into a full module path (this is also what we do with svn). Would it be more appropriate to treat the /path/to/repo as the location of the git repo which we would pass to "git clone", and /path/to/module as a subdirectory in the repo, which we would use as the base directory of the build, and look there for a .spec file? From oliver at linux-kernel.at Thu Jul 24 06:56:09 2008 From: oliver at linux-kernel.at (Oliver Falk) Date: Thu, 24 Jul 2008 08:56:09 +0200 Subject: package upload/import speed In-Reply-To: <1216845242.12744.15.camel@maunalani.home> References: <48878CFF.1000701@jla.rutgers.edu> <1216845242.12744.15.camel@maunalani.home> Message-ID: <48882789.90906@linux-kernel.at> Mike Bonnet wrote: > On Wed, 2008-07-23 at 15:56 -0400, Naveen Gavini wrote: >> Hello all, >> >> When importing or building RPM's from source is there an option that is >> throttling the upload speed of the package? >> We are seeing speeds of 10kbps which makes importing/building large >> packages very slow. > > Package upload happens via xml-rpc, and to avoid holding the entire > contents of the file in memory, it's done in chunks. Unfortunately > python xml-rpc libs don't support keep-alive, so if you're using SSL > authentication you incur the non-trivial SSL connection setup/teardown > overhead for every chunk, in addition to the TCP connection > setup/teardown. The chunk size in your version is probably very small > (64k?). That has been increased in the Koji repo, but not released yet. > The next version will improve this situation somewhat. Just wanted to write the same... Naveen, you could increase your blocksiize as a workaround. Just search for blocksize in /usr/bin/koji (function do_import)... I've done that here and it helps. BUT do not set to too large, as for every chunk some checksum (it seems) is being build (SSL does that I think) and the larger your chunk the more CPU resources it will use!!! -of From enrico.scholz at informatik.tu-chemnitz.de Thu Jul 24 07:29:44 2008 From: enrico.scholz at informatik.tu-chemnitz.de (Enrico Scholz) Date: Thu, 24 Jul 2008 09:29:44 +0200 Subject: building from git using custom koji install In-Reply-To: <48873DE1.4040001@redhat.com> (Jesus M. Rodriguez's message of "Wed, 23 Jul 2008 10:19:13 -0400") References: <48873DE1.4040001@redhat.com> Message-ID: <873alzbuzb.fsf@sheridan.bigo.ensc.de> "Jesus M. Rodriguez" writes: > git://git.fedorahosted.org/git?spacewalk.git#HEAD > > But it complains about not having a spec file. Which makes sense > as there is no spec file at the top of the tree. We solved a similar issue (building multiple branches from same sources) by passing some information to make; e.g. see http://ensc.de/fedora/0007-pass-some-information-to-the-make-srpm-task.patch 'make' extracts then the branch-id out of the scm revision and set e.g. correct %dist. Should be easy to adapt for your case. Enrico From dledford at redhat.com Thu Jul 24 12:38:15 2008 From: dledford at redhat.com (Doug Ledford) Date: Thu, 24 Jul 2008 08:38:15 -0400 Subject: building from git using custom koji install In-Reply-To: <1216871496.12744.38.camel@maunalani.home> References: <48873DE1.4040001@redhat.com> <48875CF7.30805@redhat.com> <488774F8.3000605@redhat.com> <4887DFD5.5070804@redhat.com> <20080724022724.GB4817@transam.devel.redhat.com> <1216868802.4504.32.camel@firewall.xsintricity.com> <1216871496.12744.38.camel@maunalani.home> Message-ID: <1216903095.4504.37.camel@firewall.xsintricity.com> On Wed, 2008-07-23 at 23:51 -0400, Mike Bonnet wrote: > On Wed, 2008-07-23 at 23:06 -0400, Doug Ledford wrote: > > On Wed, 2008-07-23 at 22:27 -0400, Jesus Rodriguez wrote: > > > On Wed, Jul 23, 2008 at 09:50:13PM -0400, Mike McLean wrote: > > > > Jesus M. Rodriguez wrote: > > > >> So am I correct in my assumption that koji expects the scm > > > >> repository to house a single package? > > > > > > > > Yes. Furthermore, koji assumes that the simple command 'make srpm' > > > > issued from the checkout will create a single source rpm. There is not > > > > enough information passed to koji for it to handle anything otherwise. > > > > > > > > In your SCM layout with multiple packages, how do you generate the > > > > different srpms? Is there a real reason that they are all in one module? > > > > > > We generate different srpms with the Makefile that exists in each > > > subdirectory. For example, if you want an srpm for the web module > > > cd spacewalk.git/web; make srpm > > > > > > Want one for the java code, then cd spacewalk.git/java. > > > > > > With SVN it seems completely doable because I could give the > > > path to repo/spacewalk/java and have koji checkout just that > > > directory. Seems to be a limitation of GIT in this case > > > (or we're using it wrong) :) > > > > Well, git was intentionally written to be basically a state machine of > > the entire project. So, housing more than one project in a single git > > repo isn't wrong, but it does tie your various projects together at the > > state machine level. This is why you can't checkout just one > > subdirectory from git, you have to grab the entire project. I think I > > heard mumblings of someone wanting to make subprojects work nicely in > > git, but I haven't heard if it ever got off the ground. > > > > Regardless though, koji won't work with what your current SCM layout > > without modification to the koji code. > > While that's true, I'm not sure we're definitely handling things > correctly in the git case. The git support in Koji hasn't gotten a lot > of testing, and there are a lot of ways to setup a git repo. However, I > think that multiple projects in the same repo is probably something we > want to support (since we support it in cvs and svn). Just because these are things that are done in cvs and svn doesn't necessarily mean we want to do them in git ;-) Amongst other things, if you put multiple projects into a single repo, then to build *any* of those projects, you have to clone the *entire* repo. In addition, you already have to deal with the fact that you are cloning instead of checking out like you do with cvs. In other words, the overhead of cloning a huge repo grows much more rapidly for git than for cvs, so the pain of a multi-project repo is going to be felt much more acutely. > The SCM URL you pass to Koji has a format of: > > scheme://[user@]host/path/to/repo?path/to/module#revision_or_tag_identifier > > With git we combine the /path/to/repo and the path/to/module into a > single URL we pass to "git clone", essentially collapsing them both into > a full module path (this is also what we do with svn). Would it be more > appropriate to treat the /path/to/repo as the location of the git repo > which we would pass to "git clone", and /path/to/module as a > subdirectory in the repo, which we would use as the base directory of > the build, and look there for a .spec file? I could certainly see that as being a valid interpretation of the url string. -- Doug Ledford GPG KeyID: CFBFF194 http://people.redhat.com/dledford Infiniband specific RPMs available at http://people.redhat.com/dledford/Infiniband -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: From jesusr at redhat.com Thu Jul 24 13:32:34 2008 From: jesusr at redhat.com (Jesus Rodriguez) Date: Thu, 24 Jul 2008 09:32:34 -0400 Subject: building from git using custom koji install In-Reply-To: <1216871496.12744.38.camel@maunalani.home> References: <48873DE1.4040001@redhat.com> <48875CF7.30805@redhat.com> <488774F8.3000605@redhat.com> <4887DFD5.5070804@redhat.com> <20080724022724.GB4817@transam.devel.redhat.com> <1216868802.4504.32.camel@firewall.xsintricity.com> <1216871496.12744.38.camel@maunalani.home> Message-ID: <20080724133234.GA15976@transam.devel.redhat.com> On Wed, Jul 23, 2008 at 11:51:36PM -0400, Mike Bonnet wrote: > On Wed, 2008-07-23 at 23:06 -0400, Doug Ledford wrote: > > On Wed, 2008-07-23 at 22:27 -0400, Jesus Rodriguez wrote: > > > On Wed, Jul 23, 2008 at 09:50:13PM -0400, Mike McLean wrote: > > > > Jesus M. Rodriguez wrote: > > > >> So am I correct in my assumption that koji expects the scm > > > >> repository to house a single package? > > > > > > > > Yes. Furthermore, koji assumes that the simple command 'make srpm' > > > > issued from the checkout will create a single source rpm. There is not > > > > enough information passed to koji for it to handle anything otherwise. > > > > > > > > In your SCM layout with multiple packages, how do you generate the > > > > different srpms? Is there a real reason that they are all in one module? > > > > > > We generate different srpms with the Makefile that exists in each > > > subdirectory. For example, if you want an srpm for the web module > > > cd spacewalk.git/web; make srpm > > > > > > Want one for the java code, then cd spacewalk.git/java. > > > > > > With SVN it seems completely doable because I could give the > > > path to repo/spacewalk/java and have koji checkout just that > > > directory. Seems to be a limitation of GIT in this case > > > (or we're using it wrong) :) > > > > Well, git was intentionally written to be basically a state machine of > > the entire project. So, housing more than one project in a single git > > repo isn't wrong, but it does tie your various projects together at the > > state machine level. This is why you can't checkout just one > > subdirectory from git, you have to grab the entire project. I think I > > heard mumblings of someone wanting to make subprojects work nicely in > > git, but I haven't heard if it ever got off the ground. > > > > Regardless though, koji won't work with what your current SCM layout > > without modification to the koji code. > > While that's true, I'm not sure we're definitely handling things > correctly in the git case. The git support in Koji hasn't gotten a lot > of testing, and there are a lot of ways to setup a git repo. However, I > think that multiple projects in the same repo is probably something we > want to support (since we support it in cvs and svn). > > The SCM URL you pass to Koji has a format of: > > scheme://[user@]host/path/to/repo?path/to/module#revision_or_tag_identifier > > With git we combine the /path/to/repo and the path/to/module into a > single URL we pass to "git clone", essentially collapsing them both into > a full module path (this is also what we do with svn). Would it be more > appropriate to treat the /path/to/repo as the location of the git repo > which we would pass to "git clone", and /path/to/module as a > subdirectory in the repo, which we would use as the base directory of > the build, and look there for a .spec file? Yep, that would actually work in our case. Because then I could do the following: git://git.fedorahosted.org/git/spacewalk.git?web#HEAD web would contain the .spec file. -- jesus m. rodriguez | jesusr at redhat.com sr. software engineer | irc: zeus red hat network | 919.754.4413 (w) rhce # 805008586930012 | 919.623.0080 (c) +-------------------------------------------+ | "Those who cannot learn from history | | are doomed to repeat it." | | -- George Santayana | +-------------------------------------------+ From ajax at redhat.com Tue Jul 29 15:22:11 2008 From: ajax at redhat.com (Adam Jackson) Date: Tue, 29 Jul 2008 11:22:11 -0400 Subject: [PATCH] Add 'lint' target to Makefile.common Message-ID: <1217344931.3747.120.camel@localhost.localdomain> - ajax -------------- next part -------------- A non-text attachment was scrubbed... Name: make-lint.patch Type: text/x-patch Size: 1974 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: From dennis at ausil.us Tue Jul 29 15:41:36 2008 From: dennis at ausil.us (Dennis Gilmore) Date: Tue, 29 Jul 2008 10:41:36 -0500 Subject: [PATCH] Add 'lint' target to Makefile.common In-Reply-To: <1217344931.3747.120.camel@localhost.localdomain> References: <1217344931.3747.120.camel@localhost.localdomain> Message-ID: <200807291041.47399.dennis@ausil.us> On Tuesday 29 July 2008, Adam Jackson wrote: > - ajax applied thanks -- Dennis Gilmore -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part. URL: From martin.langhoff at gmail.com Wed Jul 30 00:09:10 2008 From: martin.langhoff at gmail.com (Martin Langhoff) Date: Wed, 30 Jul 2008 12:09:10 +1200 Subject: Pungi and F7 In-Reply-To: <46a038f90807291706w70921ab7pfadfe920b99da629@mail.gmail.com> References: <46a038f90807291706w70921ab7pfadfe920b99da629@mail.gmail.com> Message-ID: <46a038f90807291709v72570edbm9742c305789d2284@mail.gmail.com> Jesse has mentioned (in an earlier thread on fedora-devel) that the Pungi in F7 has some caveats. At the moment I am considering a migration from livecd-tools to pungi on F7 _before_ I migrate to F9 (this is all for the OLPC School Server, btw). So my questions are roughly: - What known limitations does the F7 pungi have? - Is it known to build a minimal cli distro installer correctly? cheers, martin -- martin.langhoff at gmail.com martin at laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff From martin.langhoff at gmail.com Wed Jul 30 00:20:15 2008 From: martin.langhoff at gmail.com (Martin Langhoff) Date: Wed, 30 Jul 2008 12:20:15 +1200 Subject: Pungi and a minimal F9 - anaconda-runtime/buildinstall errors Message-ID: <46a038f90807291720k4b4c6df4x6d475a7e2e5f4649@mail.gmail.com> Getting my feet wet with pungi, I am tring to build a minimal F9 installer, and it is erroring out, as follows. The ks file is adapted from one from the livecd packages: $ cat livecd-fedora-9-minimal.ks repo --name=release --mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=fedora-9&arch=$basearch repo --name=updates --mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=updates-released-f9&arch=$basearch %packages @core bash kernel passwd policycoreutils chkconfig authconfig rootfiles %end My commandline is $ sudo pungi -c livecd-fedora-9-minimal.ks --nosource --force --ver=F9 and it bails out on me thus: Pungi.Pungi:INFO: Making repodata Warning: Reusing existing destination directory. 450/450 - Packages/notification-daemon-0.3.7-9.fc9.i386.rpmPungi.Pungi:INFO: Running /usr/bin/repoview --quiet --state-dir /var/cache/pungi/repoviewcache --title "Fedora F9 - i386" /home/martin/pungi-tests-9/F9/i386/os Pungi.Pungi:INFO: Running /usr/lib/anaconda-runtime/buildinstall --product Fedora --version F9 --release "Fedora F9" --bugurl http://bugzilla.redhat.com /home/martin/pungi-tests-9/F9/i386/os Pungi.Pungi:ERROR: Got an error from /usr/lib/anaconda-runtime/buildinstall Pungi.Pungi:ERROR: Saving Primary metadata Saving file lists metadata Saving other metadata Generating sqlite DBs Sqlite DBs complete Traceback (most recent call last): File "/usr/bin/pungi", line 189, in main() File "/usr/bin/pungi", line 98, in main mypungi.doBuildinstall() File "/usr/lib/python2.5/site-packages/pypungi/pungi.py", line 168, in doBuildinstall pypungi._doRunCommand(buildinstall, self.logger) #, env={"TMPDIR": self.workdir}) File "/usr/lib/python2.5/site-packages/pypungi/__init__.py", line 70, in _doRunCommand raise OSError, "Got an error from %s: %s" % (command[0], err) OSError: Got an error from /usr/lib/anaconda-runtime/buildinstall: installed versions of the relevant tools are - $ rpm -qa | grep '^\(rpm\|anaconda\|pungi\|repo\)' | sort anaconda-11.4.0.82-1.i386 anaconda-runtime-11.4.0.82-1.i386 pungi-1.2.18.1-1.fc9.noarch repoview-0.6.2-1.fc9.noarch rpm-4.4.2.3-2.fc9.i386 rpm-build-4.4.2.3-2.fc9.i386 rpm-devel-4.4.2.3-2.fc9.i386 rpm-libs-4.4.2.3-2.fc9.i386 rpmlint-0.83-1.fc9.noarch rpm-python-4.4.2.3-2.fc9.i386 (which are up-to-date F9 AFAICT) any hints welcome... cheers, martin -- martin.langhoff at gmail.com martin at laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff From jkeating at redhat.com Wed Jul 30 01:38:42 2008 From: jkeating at redhat.com (Jesse Keating) Date: Tue, 29 Jul 2008 21:38:42 -0400 Subject: Pungi and F7 In-Reply-To: <46a038f90807291709v72570edbm9742c305789d2284@mail.gmail.com> References: <46a038f90807291706w70921ab7pfadfe920b99da629@mail.gmail.com> <46a038f90807291709v72570edbm9742c305789d2284@mail.gmail.com> Message-ID: <1217381922.3151.89.camel@localhost.localdomain> On Wed, 2008-07-30 at 12:09 +1200, Martin Langhoff wrote: > - What known limitations does the F7 pungi have? Only that you need to have anaconda-runtime in your manifest, which drags in a pile of dependencies. There are a couple of other compose needs, I think the config shipped with F7 has them listed under a commented section. > - Is it known to build a minimal cli distro installer correctly? Provided the right things are in your manifest, yes. -- Jesse Keating Fedora -- Freedom? is a feature! -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: From jkeating at redhat.com Wed Jul 30 01:39:18 2008 From: jkeating at redhat.com (Jesse Keating) Date: Tue, 29 Jul 2008 21:39:18 -0400 Subject: Pungi and a minimal F9 - anaconda-runtime/buildinstall errors In-Reply-To: <46a038f90807291720k4b4c6df4x6d475a7e2e5f4649@mail.gmail.com> References: <46a038f90807291720k4b4c6df4x6d475a7e2e5f4649@mail.gmail.com> Message-ID: <1217381958.3151.91.camel@localhost.localdomain> On Wed, 2008-07-30 at 12:20 +1200, Martin Langhoff wrote: > 70, in _doRunCommand > raise OSError, "Got an error from %s: %s" % (command[0], err) > OSError: Got an error from /usr/lib/anaconda-runtime/buildinstall: > > installed versions of the relevant tools are - Please check the .log file in your output/logs/ directory, so that you can get the actual output from buildinstall. -- Jesse Keating Fedora -- Freedom? is a feature! -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: From martin.langhoff at gmail.com Wed Jul 30 04:09:36 2008 From: martin.langhoff at gmail.com (Martin Langhoff) Date: Wed, 30 Jul 2008 16:09:36 +1200 Subject: Pungi and a minimal F9 - anaconda-runtime/buildinstall errors In-Reply-To: <1217381958.3151.91.camel@localhost.localdomain> References: <46a038f90807291720k4b4c6df4x6d475a7e2e5f4649@mail.gmail.com> <1217381958.3151.91.camel@localhost.localdomain> Message-ID: <46a038f90807292109y6079b84fi13202b76d01b1f70@mail.gmail.com> On Wed, Jul 30, 2008 at 1:39 PM, Jesse Keating wrote: > Please check the .log file in your output/logs/ directory, so that > you can get the actual output from buildinstall. Good hint, thanks. Missing anaconda-runtime it seems. Pungi.Pungi.DEBUG: Running buildinstall... /tmp/buildinstall.tree.7NRF8R /tmp No Match for argument anaconda-runtime Nothing to download adding anaconda-runtime makes things work. I end up with a 700MB DVD and 2 CD images, as apparently Gnome gets pulled in. Perhaps @base contains Gnome. I might need to say '--nobase'. Retrying... further hints as to how to build a minimal cli F9 installer ISO welcome... m -- martin.langhoff at gmail.com martin at laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff From martin.langhoff at gmail.com Wed Jul 30 04:59:56 2008 From: martin.langhoff at gmail.com (Martin Langhoff) Date: Wed, 30 Jul 2008 16:59:56 +1200 Subject: Pungi and a minimal F9 - anaconda-runtime/buildinstall errors In-Reply-To: <46a038f90807292109y6079b84fi13202b76d01b1f70@mail.gmail.com> References: <46a038f90807291720k4b4c6df4x6d475a7e2e5f4649@mail.gmail.com> <1217381958.3151.91.camel@localhost.localdomain> <46a038f90807292109y6079b84fi13202b76d01b1f70@mail.gmail.com> Message-ID: <46a038f90807292159j4c607da3j62196df6e08445ec@mail.gmail.com> Getting closer :-) With the following ks I get a 208MB iso that boots into anaconda. Anaconda, however, says "The Fedora disk was not found in any of your CDROM drives. Please insert the Fedora disk and press OK to retry." What does anaconda look for? $ cat kickstarts/pungi-f9-minimal.ks repo --name=release --mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=fedora-9&arch=$basearch repo --name=updates --mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=updates-released-f9&arch=$basearch %packages --nobase ##@core bash kernel passwd policycoreutils chkconfig authconfig rootfiles anaconda-runtime -gnome* -cairo -xorg-x11-server-Xorg %end cheers, martin -- martin.langhoff at gmail.com martin at laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff From martin.langhoff at gmail.com Wed Jul 30 05:31:51 2008 From: martin.langhoff at gmail.com (Martin Langhoff) Date: Wed, 30 Jul 2008 17:31:51 +1200 Subject: livecd-iso-to-disk works on Pungi CDs. Message-ID: <46a038f90807292231m47a29f6dx7a91b4f68aeae414@mail.gmail.com> Welcome surprise - the livecd-iso-to-disk utility included in livecd-tools works well with pungi-created CDs, booting normally into anaconda. Now, I only have to figure out how to get anaconda to accept the repo that's in the CD/USB disk to complete a base install. cheers, m -- martin.langhoff at gmail.com martin at laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff From martin.langhoff at gmail.com Wed Jul 30 07:31:26 2008 From: martin.langhoff at gmail.com (Martin Langhoff) Date: Wed, 30 Jul 2008 19:31:26 +1200 Subject: Pungi and a minimal F9 - anaconda-runtime/buildinstall errors In-Reply-To: <46a038f90807292159j4c607da3j62196df6e08445ec@mail.gmail.com> References: <46a038f90807291720k4b4c6df4x6d475a7e2e5f4649@mail.gmail.com> <1217381958.3151.91.camel@localhost.localdomain> <46a038f90807292109y6079b84fi13202b76d01b1f70@mail.gmail.com> <46a038f90807292159j4c607da3j62196df6e08445ec@mail.gmail.com> Message-ID: <46a038f90807300031h4c746f68gc1bfc52d650fe3ae@mail.gmail.com> On Wed, Jul 30, 2008 at 4:59 PM, Martin Langhoff wrote: > Getting closer :-) With the following ks I get a 208MB iso that boots > into anaconda. Anaconda, however, says "The Fedora disk was not found > in any of your CDROM drives. Please insert the Fedora disk and press > OK to retry." > > What does anaconda look for? After a long tour of the latest anaconda sources from git, and reading how cdinstall.c setupCdrom() - seems to trawl the /images/ directory looking for stage2.img , I find that the /images directory is missing. Pungi/anaconda had bailed out on me due to unresolved dependencies, duly noted in the log/$arch.log file, so I was using a broken, incomplete img. Commenting out the lines that blocked cairo,X.org and gnome from getting into the build gets me a working installer CD that weighs 471MB. So the minimal F9 I can get without starting to splice packages is 471MB, with the following ks: repo --name=release --mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=fedora-9&arch=$basearch repo --name=updates --mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=updates-released-f9&arch=$basearch %packages --nobase bash kernel passwd policycoreutils chkconfig authconfig rootfiles anaconda-runtime %end What approach should I take to get a headless system without breaking anaconda? This task has no use for X, cairo or gnome :-) For example. anaconda and anaconda-runtime pull in - rhpxl, cairo, gtk2, zenity, xorg-x11-font-utils - useful for anaconda's graphical installer - desktop-file-utils -- why? - gnome-python2-gtkhtml2 -- why? About 1/3 or more of the 1st install CD seems to go to Anaconda's graphics lust :-) m -- martin.langhoff at gmail.com martin at laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff From martin.langhoff at gmail.com Wed Jul 30 08:04:20 2008 From: martin.langhoff at gmail.com (Martin Langhoff) Date: Wed, 30 Jul 2008 20:04:20 +1200 Subject: livecd-iso-to-disk works on Pungi CDs. In-Reply-To: <46a038f90807292231m47a29f6dx7a91b4f68aeae414@mail.gmail.com> References: <46a038f90807292231m47a29f6dx7a91b4f68aeae414@mail.gmail.com> Message-ID: <46a038f90807300104x7317fc0t6fb87ac19eb13369@mail.gmail.com> On Wed, Jul 30, 2008 at 5:31 PM, Martin Langhoff wrote: > Welcome surprise - the livecd-iso-to-disk utility included in > livecd-tools works well with pungi-created CDs, booting normally into > anaconda. Time to qualify this. I can get anaconda to start, and with a trivial patch to livecd-iso-to-disk I can get all the files in place. Anaconda doesn't seem to find images/stage2.img however. When it asks "where is your install media" I can point it to /dev/sdb1, and it works. Providing a root=UUID= and rootfstype=vfat does not seem to work. I don't want to write /dev/ in the syslinux.cfg file though. Is there a kernel cmdline that is portable and read by anaconda? If I can set that during the setup of the usb stick, we are sorted. cheers, m -- martin.langhoff at gmail.com martin at laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff From martin.langhoff at gmail.com Wed Jul 30 08:25:46 2008 From: martin.langhoff at gmail.com (Martin Langhoff) Date: Wed, 30 Jul 2008 20:25:46 +1200 Subject: livecd-iso-to-disk works on Pungi CDs. In-Reply-To: <46a038f90807300104x7317fc0t6fb87ac19eb13369@mail.gmail.com> References: <46a038f90807292231m47a29f6dx7a91b4f68aeae414@mail.gmail.com> <46a038f90807300104x7317fc0t6fb87ac19eb13369@mail.gmail.com> Message-ID: <46a038f90807300125x7323deeaoa31facea6a536871@mail.gmail.com> On Wed, Jul 30, 2008 at 8:04 PM, Martin Langhoff wrote: > a trivial patch to livecd-iso-to-disk I can get all the files in place Patch here - http://dev.laptop.org/git?p=projects/xs-livecd;a=commitdiff;h=da6c70eb79d7a9513e9bbabd0acd71bb745d53e5 cheers, martin -- martin.langhoff at gmail.com martin at laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff From katzj at redhat.com Wed Jul 30 13:26:13 2008 From: katzj at redhat.com (Jeremy Katz) Date: Wed, 30 Jul 2008 09:26:13 -0400 Subject: livecd-iso-to-disk works on Pungi CDs. In-Reply-To: <46a038f90807292231m47a29f6dx7a91b4f68aeae414@mail.gmail.com> References: <46a038f90807292231m47a29f6dx7a91b4f68aeae414@mail.gmail.com> Message-ID: <1217424373.16534.18.camel@aglarond.local> On Wed, 2008-07-30 at 17:31 +1200, Martin Langhoff wrote: > Welcome surprise - the livecd-iso-to-disk utility included in > livecd-tools works well with pungi-created CDs, booting normally into > anaconda. The script is for livecds. Trying to keep it working for more variations than that isn't really something that's interesting or sustainable. There are already too many branches and special cases as it is in there. So while it may sort of work today, I don't in any way want to guarantee that it will tomorrow Also, the shell script should hopefully be getting used less by people in preference for lmacken's nicer python util/UI. Jeremy From ajax at redhat.com Wed Jul 30 15:58:17 2008 From: ajax at redhat.com (Adam Jackson) Date: Wed, 30 Jul 2008 11:58:17 -0400 Subject: [PATCHES] Makefile.common speedups Message-ID: <1217433497.1477.21.camel@localhost.localdomain> Attached are two (orthogonal) patches to make evaluation of Makefile.common a bit faster. The first one is possibly contentious. Currently, early-branching works by checking for the existence of the other branch, by using 'cvs rlog'. That kinda sucks, because it means you can't do 'make local' while disconnected, and even when connected it's not fast. The patch changes it to look for the package's name in a new file, common/early-branched-packages. By keeping that file together with Makefile.common we get pretty much the behaviour we're used to: when build targets change, you have to update common/. Note that if we apply this patch we will also need to create that (empty) file. This would change the cvsadmin procedure for early branching, but hopefully not by a burdensome amount. The second one is just a refactoring to only ask rpm for %VERSION and %RELEASE once. In principle, for packages that overrode both VERSION and RELEASE in their Makefile, this would actually make things slower, since now you'd be forcing rpm to run. According to my pkgcvs checkout, no such packages exist. Tested locally on a 3.2GHz P4. 'time make' baseline was 1.3 seconds. First patch dropped that to about 0.6 seconds. Second patch on top of that dropped down to about 0.38 seconds. - ajax -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Redo-early-branching.patch Type: text/x-patch Size: 1221 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0002-Invoke-rpm-less.patch Type: text/x-patch Size: 1562 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: From a.badger at gmail.com Wed Jul 30 17:26:16 2008 From: a.badger at gmail.com (Toshio Kuratomi) Date: Wed, 30 Jul 2008 10:26:16 -0700 Subject: [PATCHES] Makefile.common speedups In-Reply-To: <1217433497.1477.21.camel@localhost.localdomain> References: <1217433497.1477.21.camel@localhost.localdomain> Message-ID: <4890A438.6060004@gmail.com> Adam Jackson wrote: > Attached are two (orthogonal) patches to make evaluation of > Makefile.common a bit faster. > > The first one is possibly contentious. Currently, early-branching works > by checking for the existence of the other branch, by using 'cvs rlog'. > That kinda sucks, because it means you can't do 'make local' while > disconnected, and even when connected it's not fast. The patch changes > it to look for the package's name in a new file, > common/early-branched-packages. By keeping that file together with > Makefile.common we get pretty much the behaviour we're used to: when > build targets change, you have to update common/. Note that if we apply > this patch we will also need to create that (empty) file. > > This would change the cvsadmin procedure for early branching, but > hopefully not by a burdensome amount. > I'm against this one as it moves a kludge from Makefile.common into the scripts that handle branching. Can we achieve the same thing by keeping the information in the devel branch? For instance, each of the other branches has a branch file with the branch name inside. If devel did the same, could we alleviate this problem? (Branch scripts might still need to be updated to write/verify the branch file in the devel directory but that would be more generic than writing into an early branch file) -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: OpenPGP digital signature URL: From dennis at ausil.us Wed Jul 30 17:44:38 2008 From: dennis at ausil.us (Dennis Gilmore) Date: Wed, 30 Jul 2008 12:44:38 -0500 Subject: [PATCHES] Makefile.common speedups In-Reply-To: <1217433497.1477.21.camel@localhost.localdomain> References: <1217433497.1477.21.camel@localhost.localdomain> Message-ID: <200807301244.48589.dennis@ausil.us> On Wednesday 30 July 2008, Adam Jackson wrote: > Attached are two (orthogonal) patches to make evaluation of > Makefile.common a bit faster. > > The first one is possibly contentious. Currently, early-branching works > by checking for the existence of the other branch, by using 'cvs rlog'. > That kinda sucks, because it means you can't do 'make local' while > disconnected, and even when connected it's not fast. The patch changes > it to look for the package's name in a new file, > common/early-branched-packages. By keeping that file together with > Makefile.common we get pretty much the behaviour we're used to: when > build targets change, you have to update common/. Note that if we apply > this patch we will also need to create that (empty) file. > > This would change the cvsadmin procedure for early branching, but > hopefully not by a burdensome amount. > > The second one is just a refactoring to only ask rpm for %VERSION and > %RELEASE once. In principle, for packages that overrode both VERSION > and RELEASE in their Makefile, this would actually make things slower, > since now you'd be forcing rpm to run. According to my pkgcvs checkout, > no such packages exist. > > Tested locally on a 3.2GHz P4. 'time make' baseline was 1.3 seconds. > First patch dropped that to about 0.6 seconds. Second patch on top of > that dropped down to about 0.38 seconds. > > - ajax ive applied the second patch. I think id rather us comment out the early branching check all together when we don't need it. we probably should add some checks to make build and tag so that it does a cvs up on the common dir. as it can be an issue when they are out of date. -- Dennis Gilmore -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part. URL: From ajax at redhat.com Wed Jul 30 18:04:28 2008 From: ajax at redhat.com (Adam Jackson) Date: Wed, 30 Jul 2008 14:04:28 -0400 Subject: [PATCHES] Makefile.common speedups In-Reply-To: <4890A438.6060004@gmail.com> References: <1217433497.1477.21.camel@localhost.localdomain> <4890A438.6060004@gmail.com> Message-ID: <1217441068.4699.9.camel@localhost.localdomain> On Wed, 2008-07-30 at 10:26 -0700, Toshio Kuratomi wrote: > Adam Jackson wrote: > > Attached are two (orthogonal) patches to make evaluation of > > Makefile.common a bit faster. > > > > The first one is possibly contentious. Currently, early-branching works > > by checking for the existence of the other branch, by using 'cvs rlog'. > > That kinda sucks, because it means you can't do 'make local' while > > disconnected, and even when connected it's not fast. The patch changes > > it to look for the package's name in a new file, > > common/early-branched-packages. By keeping that file together with > > Makefile.common we get pretty much the behaviour we're used to: when > > build targets change, you have to update common/. Note that if we apply > > this patch we will also need to create that (empty) file. > > > > This would change the cvsadmin procedure for early branching, but > > hopefully not by a burdensome amount. > > > I'm against this one as it moves a kludge from Makefile.common into the > scripts that handle branching. Can we achieve the same thing by keeping > the information in the devel branch? For instance, each of the other > branches has a branch file with the branch name inside. If devel did > the same, could we alleviate this problem? Sure. Makefile.common could parse the 'branch' file even for devel/, and the magic word "early" or "rawhide" or something would mean "early branch if that's an option, otherwise normal rawhide behaviour". I don't have strong feelings about where the early-branch indicator lives, as long as it's something I can know without asking the CVS server. - ajax -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: From martin.langhoff at gmail.com Wed Jul 30 22:03:39 2008 From: martin.langhoff at gmail.com (Martin Langhoff) Date: Thu, 31 Jul 2008 10:03:39 +1200 Subject: livecd-iso-to-disk works on Pungi CDs. In-Reply-To: <1217424373.16534.18.camel@aglarond.local> References: <46a038f90807292231m47a29f6dx7a91b4f68aeae414@mail.gmail.com> <1217424373.16534.18.camel@aglarond.local> Message-ID: <46a038f90807301503s27616737k4efb902f6e2d5dd9@mail.gmail.com> On Thu, Jul 31, 2008 at 1:26 AM, Jeremy Katz wrote: > On Wed, 2008-07-30 at 17:31 +1200, Martin Langhoff wrote: >> Welcome surprise - the livecd-iso-to-disk utility included in >> livecd-tools works well with pungi-created CDs, booting normally into >> anaconda. > > The script is for livecds. Trying to keep it working for more > variations than that isn't really something that's interesting or > sustainable. No prob. I'll make a fork of it, rename it installcd-iso-to-disk. > Also, the shell script should hopefully be getting used less by people > in preference for lmacken's nicer python util/UI. Google led me to https://fedorahosted.org/liveusb-creator -- looks nice enough, but I need an automatable, text-config-driven build toolchain that runs on headless boxes. cheers, martin -- martin.langhoff at gmail.com martin at laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff From dennis at ausil.us Wed Jul 30 22:30:15 2008 From: dennis at ausil.us (Dennis Gilmore) Date: Wed, 30 Jul 2008 17:30:15 -0500 Subject: [PATCHES] Makefile.common speedups In-Reply-To: <1217441068.4699.9.camel@localhost.localdomain> References: <1217433497.1477.21.camel@localhost.localdomain> <4890A438.6060004@gmail.com> <1217441068.4699.9.camel@localhost.localdomain> Message-ID: <200807301730.19183.dennis@ausil.us> On Wednesday 30 July 2008, Adam Jackson wrote: > On Wed, 2008-07-30 at 10:26 -0700, Toshio Kuratomi wrote: > > Adam Jackson wrote: > > > Attached are two (orthogonal) patches to make evaluation of > > > Makefile.common a bit faster. > > > > > > The first one is possibly contentious. Currently, early-branching > > > works by checking for the existence of the other branch, by using 'cvs > > > rlog'. That kinda sucks, because it means you can't do 'make local' > > > while disconnected, and even when connected it's not fast. The patch > > > changes it to look for the package's name in a new file, > > > common/early-branched-packages. By keeping that file together with > > > Makefile.common we get pretty much the behaviour we're used to: when > > > build targets change, you have to update common/. Note that if we > > > apply this patch we will also need to create that (empty) file. > > > > > > This would change the cvsadmin procedure for early branching, but > > > hopefully not by a burdensome amount. > > > > I'm against this one as it moves a kludge from Makefile.common into the > > scripts that handle branching. Can we achieve the same thing by keeping > > the information in the devel branch? For instance, each of the other > > branches has a branch file with the branch name inside. If devel did > > the same, could we alleviate this problem? > > Sure. Makefile.common could parse the 'branch' file even for devel/, > and the magic word "early" or "rawhide" or something would mean "early > branch if that's an option, otherwise normal rawhide behaviour". > > I don't have strong feelings about where the early-branch indicator > lives, as long as it's something I can know without asking the CVS > server. > > - ajax cat F-9/branch F-9 it should reference the line in branches that tells the Makefiles where to build. this would require changes in the mass-branching, branching scripts to make sure its right. but it should be workable. so when we start early branching we will put F-11 in the branch file and add to branches in common for knowing how to deal with F-11 -- Dennis Gilmore -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part. URL: From katzj at redhat.com Thu Jul 31 02:05:39 2008 From: katzj at redhat.com (Jeremy Katz) Date: Wed, 30 Jul 2008 22:05:39 -0400 Subject: livecd-iso-to-disk works on Pungi CDs. In-Reply-To: <46a038f90807301503s27616737k4efb902f6e2d5dd9@mail.gmail.com> References: <46a038f90807292231m47a29f6dx7a91b4f68aeae414@mail.gmail.com> <1217424373.16534.18.camel@aglarond.local> <46a038f90807301503s27616737k4efb902f6e2d5dd9@mail.gmail.com> Message-ID: <1217469939.16534.87.camel@aglarond.local> On Thu, 2008-07-31 at 10:03 +1200, Martin Langhoff wrote: > On Thu, Jul 31, 2008 at 1:26 AM, Jeremy Katz wrote: > > On Wed, 2008-07-30 at 17:31 +1200, Martin Langhoff wrote: > >> Welcome surprise - the livecd-iso-to-disk utility included in > >> livecd-tools works well with pungi-created CDs, booting normally into > >> anaconda. > > > > The script is for livecds. Trying to keep it working for more > > variations than that isn't really something that's interesting or > > sustainable. > > No prob. I'll make a fork of it, rename it installcd-iso-to-disk. And I have no problem at all with someone maintaining such a script. In fact, I'd encourage it even :-) > > Also, the shell script should hopefully be getting used less by people > > in preference for lmacken's nicer python util/UI. > > Google led me to https://fedorahosted.org/liveusb-creator -- looks > nice enough, but I need an automatable, text-config-driven build > toolchain that runs on headless boxes. The idea is that it should have a command line interface also. If it doesn't, throw stones at Luke until he finishes it ;-) Jeremy From kanarip at kanarip.com Thu Jul 31 14:43:49 2008 From: kanarip at kanarip.com (Jeroen van Meeuwen) Date: Thu, 31 Jul 2008 16:43:49 +0200 Subject: livecd-iso-to-disk works on Pungi CDs. In-Reply-To: <46a038f90807301503s27616737k4efb902f6e2d5dd9@mail.gmail.com> References: <46a038f90807292231m47a29f6dx7a91b4f68aeae414@mail.gmail.com> <1217424373.16534.18.camel@aglarond.local> <46a038f90807301503s27616737k4efb902f6e2d5dd9@mail.gmail.com> Message-ID: <4891CFA5.2070609@kanarip.com> Martin Langhoff wrote: > On Thu, Jul 31, 2008 at 1:26 AM, Jeremy Katz wrote: >> On Wed, 2008-07-30 at 17:31 +1200, Martin Langhoff wrote: >>> Welcome surprise - the livecd-iso-to-disk utility included in >>> livecd-tools works well with pungi-created CDs, booting normally into >>> anaconda. >> The script is for livecds. Trying to keep it working for more >> variations than that isn't really something that's interesting or >> sustainable. > > No prob. I'll make a fork of it, rename it installcd-iso-to-disk. > > Google led me to https://fedorahosted.org/liveusb-creator -- looks > nice enough, but I need an automatable, text-config-driven build > toolchain that runs on headless boxes. Hey, your script is welcome with Revisor, a automatable, text-config-driven build toolchain that runs on headless boxes, as well as a nice GUI for Joe Average users (which we're not). I'll help you write a module to enable your particular use-case if you like. Care to elaborate? Kind regards, Jeroen van Meeuwen -kanarip