From pmeyer at themeyerfarm.com Tue May 1 16:26:59 2007 From: pmeyer at themeyerfarm.com (Phil Meyer) Date: Tue, 01 May 2007 10:26:59 -0600 Subject: Errors F7 updated today and pungi Message-ID: <46376A53.5030906@themeyerfarm.com> This is a fresh install of F7. Updated this morning. Ran a test of the default minimal. All went well. Added custom files, and -- this is what I got. Very un-obvious where my mistake is. The intent was to roll a developer desktop including some things from livna as well as a bunch of development tools. The list of added packages is from my FC6 installs that are used for developers here at work. This is not a 'for public consumption' distro. I am not distributing ... The FC6 pungi worked mostly, but I am following some issues in pungi that are only available on F7, so that is why I am trying to move my build system to F7. A build of FC6 on this system today produced the exact same errors, so I reproduced it from all F7 sources. Copied: comps-f7.xml from F-6.93-i386-DVD.iso to /etc/pungi. Added: /etc/pungi-developer.conf % cat pungi-developer.conf [default] product_name = Fedora ; The name used during install product_path = Fedora ; The directory where RPMS go iso_basename = F ; The first part of the iso file name bugurl = http://bugzilla.redhat.com ; Used for betanag comps = /etc/pungi/comps-f7.xml ; Used to define package groupings and default installs manifest = /etc/pungi/developer-manifest ; Used to determine what to bring in. Supports Kickstart syntax yumconf = /etc/pungi/yum.conf.i386 ; Used to determine where to gather packages from destdir = /srv/pungi/Developer ; Top level compose directory, must be clean cachedir = /srv/pungi/cache ; Cache used for repeat runs arch = i386 ; What arch to compose (must be same arch as system) version = F7 ; Used both in install and part of the dest tree flavor = Custom ; Further define a given cut of the package set discs = 6 ; Number of discs needed to fit data. #cdsize = 4608.0 ; Not used if disc count is 1 getsource = No ; Used to determine if we want source packages or not Added: /etc/developer-manifest % cat developer-manifest aalib anaconda anaconda-runtime apr-devel apr-util-devel aquamarine autossh beryl beryl-core-devel beryl-dbus beryl-gnome beryl-kde beryl-manager beryl-plugins beryl-settings beryl-settings-simple bind-utils busybox-anaconda cdrdao compat-gcc-34 compat-gcc-34-c++ divx4linux expect expect-devel faad2 flash-plugin gkrellm gkrellm-hddtemp gkrellmms gkrellm-themes gkrellm-wireless gnome-libs-devel graphviz gstreamer gstreamer-devel gstreamer-ffmpeg gstreamer-plugins-base gstreamer-plugins-base-devel gstreamer-plugins-farsight gstreamer-plugins-good gstreamer-plugins-good-devel gstreamer-plugins-pulse gstreamer-plugins-ugly gstreamer-plugins-ugly-devel gstreamer-python gstreamer-tools gtweakui heliodor httpd-devel id3lib id3lib-devel imlib Installing ipw2200-firmware jedit julie k3b k3b-extras k3b-extras-nonfree kawa kawa-javadoc kdemultimedia-extras kdewebdev kdewebdev-devel kernel-devel kmymoney2 kmymoney2-devel lame lame-mp3x libc-client libdvdcss libdvdcss-devel libdvdnav libdvdnav-devel libdvdplay libdvdplay-devel libdvdread libdvdread-devel libfame libmad libmad-devel libpostproc libtool-ltdl-devel lzo mplayer mplayer-fonts mplayer-gui mplayer-mencoder mrtg msttcorefonts mysql-devel nautilus-open-terminal phil php php-cli php-common php-imap php-mysql php-pdo php-pear pungi radvd rpmdevtools rpmlint sharutils tcl tidy tk tomcat5-admin-webapps transcode usbutils xine xine-lib xine-lib-devel xine-lib-extras xine-lib-extras-nonfree xmms xmms-flac xmms-mp3 xmms-skins xvidcore Some of these come from livna. Livna is enabled. Linked yum.conf.i386 to /etc/yum.conf to enable all enabled repos. ran: % sudo pungi -c /etc/pungi/pungi-developer.conf --all-stages Traceback (most recent call last): File "/usr/bin/pungi", line 183, in main() File "/usr/bin/pungi", line 115, in main mypungi.doPackageorder() File "/usr/lib/python2.5/site-packages/pypungi/pungi.py", line 146, in doPackageorder self._doRunCommand(pkgorder, output=pkgorderfile) File "/usr/lib/python2.5/site-packages/pypungi/pungi.py", line 77, in _doRunCommand raise OSError, "Got an error from %s" % command[0] OSError: Got an error from /usr/lib/anaconda-runtime/pkgorder End of logfile shows: INFO:pypungi.pungi:Running /usr/lib/anaconda-runtime/pkgorder /srv/pungi/Developer/FC6/Custom/i386/os i386 Fedora ERROR:pypungi.pungi:Got an error from /usr/lib/anaconda-runtime/pkgorder ERROR:pypungi.pungi:Traceback (most recent call last): File "/usr/lib/anaconda-runtime/pkgorder", line 169, in addGroups(ds, ["core", "base", "text-internet"]) File "/usr/lib/anaconda-runtime/pkgorder", line 103, in addGroups map(ds.selectGroup, filter(lambda x: ds.comps.has_group(x), groupLst)) File "/usr/lib/python2.5/site-packages/yum/__init__.py", line 1448, in selectGroup txmbrs = self.install(name = pkg) File "/usr/lib/python2.5/site-packages/yum/__init__.py", line 1800, in install tx_return.extend(self.tsInfo.getMembers(pkgtup=po.pkgtup)) File "/usr/lib/anaconda/sortedtransaction.py", line 33, in getMembers return TransactionData.getMembers(self, pkgtup, output_states) TypeError: getMembers() takes at most 2 arguments (3 given) From jgranado at redhat.com Wed May 2 08:48:06 2007 From: jgranado at redhat.com (Joel Andres Granados) Date: Wed, 02 May 2007 10:48:06 +0200 Subject: Errors F7 updated today and pungi In-Reply-To: <46376A53.5030906@themeyerfarm.com> References: <46376A53.5030906@themeyerfarm.com> Message-ID: <46385046.1030000@redhat.com> Phil Meyer wrote: > This is a fresh install of F7. > > Updated this morning. Ran a test of the default minimal. All went > well. Added custom files, and -- this is what I got. Very un-obvious > where my mistake is. > > The intent was to roll a developer desktop including some things from > livna as well as a bunch of development tools. The list of added > packages is from my FC6 installs that are used for developers here at > work. This is not a 'for public consumption' distro. I am not > distributing ... > > The FC6 pungi worked mostly, but I am following some issues in pungi > that are only available on F7, so that is why I am trying to move my > build system to F7. > A build of FC6 on this system today produced the exact same errors, so > I reproduced it from all F7 sources. > > Copied: > comps-f7.xml from F-6.93-i386-DVD.iso to /etc/pungi. > > Added: > /etc/pungi-developer.conf > > % cat pungi-developer.conf > [default] > product_name = Fedora ; The name used during install > product_path = Fedora ; The directory where RPMS go > iso_basename = F ; The first part of the iso file name > bugurl = http://bugzilla.redhat.com ; Used for betanag > comps = /etc/pungi/comps-f7.xml ; Used to define package groupings and > default installs > manifest = /etc/pungi/developer-manifest ; Used to determine what to > bring in. Supports Kickstart syntax > yumconf = /etc/pungi/yum.conf.i386 ; Used to determine where to gather > packages from > destdir = /srv/pungi/Developer ; Top level compose directory, must be > clean > cachedir = /srv/pungi/cache ; Cache used for repeat runs > arch = i386 ; What arch to compose (must be same arch as system) > version = F7 ; Used both in install and part of the dest tree > flavor = Custom ; Further define a given cut of the package set > discs = 6 ; Number of discs needed to fit data. > #cdsize = 4608.0 ; Not used if disc count is 1 > getsource = No ; Used to determine if we want source packages or not > > Added: > /etc/developer-manifest > > % cat developer-manifest > aalib > anaconda > anaconda-runtime > apr-devel > apr-util-devel > aquamarine > autossh > beryl > beryl-core-devel > beryl-dbus > beryl-gnome > beryl-kde > beryl-manager > beryl-plugins > beryl-settings > beryl-settings-simple > bind-utils > busybox-anaconda > cdrdao > compat-gcc-34 > compat-gcc-34-c++ > divx4linux > expect > expect-devel > faad2 > flash-plugin > gkrellm > gkrellm-hddtemp > gkrellmms > gkrellm-themes > gkrellm-wireless > gnome-libs-devel > graphviz > gstreamer > gstreamer-devel > gstreamer-ffmpeg > gstreamer-plugins-base > gstreamer-plugins-base-devel > gstreamer-plugins-farsight > gstreamer-plugins-good > gstreamer-plugins-good-devel > gstreamer-plugins-pulse > gstreamer-plugins-ugly > gstreamer-plugins-ugly-devel > gstreamer-python > gstreamer-tools > gtweakui > heliodor > httpd-devel > id3lib > id3lib-devel > imlib > Installing > ipw2200-firmware > jedit > julie > k3b > k3b-extras > k3b-extras-nonfree > kawa > kawa-javadoc > kdemultimedia-extras > kdewebdev > kdewebdev-devel > kernel-devel > kmymoney2 > kmymoney2-devel > lame > lame-mp3x > libc-client > libdvdcss > libdvdcss-devel > libdvdnav > libdvdnav-devel > libdvdplay > libdvdplay-devel > libdvdread > libdvdread-devel > libfame > libmad > libmad-devel > libpostproc > libtool-ltdl-devel > lzo > mplayer > mplayer-fonts > mplayer-gui > mplayer-mencoder > mrtg > msttcorefonts > mysql-devel > nautilus-open-terminal > phil > php > php-cli > php-common > php-imap > php-mysql > php-pdo > php-pear > pungi > radvd > rpmdevtools > rpmlint > sharutils > tcl > tidy > tk > tomcat5-admin-webapps > transcode > usbutils > xine > xine-lib > xine-lib-devel > xine-lib-extras > xine-lib-extras-nonfree > xmms > xmms-flac > xmms-mp3 > xmms-skins > xvidcore > > > Some of these come from livna. Livna is enabled. > > Linked yum.conf.i386 to /etc/yum.conf to enable all enabled repos. > > ran: > > % sudo pungi -c /etc/pungi/pungi-developer.conf --all-stages > Traceback (most recent call last): > File "/usr/bin/pungi", line 183, in > main() > File "/usr/bin/pungi", line 115, in main > mypungi.doPackageorder() > File "/usr/lib/python2.5/site-packages/pypungi/pungi.py", line 146, > in doPackageorder > self._doRunCommand(pkgorder, output=pkgorderfile) > File "/usr/lib/python2.5/site-packages/pypungi/pungi.py", line 77, in > _doRunCommand > raise OSError, "Got an error from %s" % command[0] > OSError: Got an error from /usr/lib/anaconda-runtime/pkgorder > > End of logfile shows: > > INFO:pypungi.pungi:Running /usr/lib/anaconda-runtime/pkgorder > /srv/pungi/Developer/FC6/Custom/i386/os i386 Fedora > ERROR:pypungi.pungi:Got an error from /usr/lib/anaconda-runtime/pkgorder > ERROR:pypungi.pungi:Traceback (most recent call last): > File "/usr/lib/anaconda-runtime/pkgorder", line 169, in > addGroups(ds, ["core", "base", "text-internet"]) > File "/usr/lib/anaconda-runtime/pkgorder", line 103, in addGroups > map(ds.selectGroup, filter(lambda x: ds.comps.has_group(x), groupLst)) > File "/usr/lib/python2.5/site-packages/yum/__init__.py", line 1448, > in selectGroup > txmbrs = self.install(name = pkg) > File "/usr/lib/python2.5/site-packages/yum/__init__.py", line 1800, > in install > tx_return.extend(self.tsInfo.getMembers(pkgtup=po.pkgtup)) > File "/usr/lib/anaconda/sortedtransaction.py", line 33, in getMembers > return TransactionData.getMembers(self, pkgtup, output_states) > TypeError: getMembers() takes at most 2 arguments (3 given) > > > > > > > -- > Fedora-buildsys-list mailing list > Fedora-buildsys-list at redhat.com > https://www.redhat.com/mailman/listinfo/fedora-buildsys-list Judging by the pungi traceback this is an anaconda related issue. I would suggest posting the issue in anaconda-devel list to see if someone can give some feedback. Regards From jgranado at redhat.com Fri May 4 14:13:34 2007 From: jgranado at redhat.com (Joel Andres Granados) Date: Fri, 04 May 2007 16:13:34 +0200 Subject: pungi logger Message-ID: <463B3F8E.2000209@redhat.com> Hi list Referring to ticket #34 at https://hosted.fedoraproject.org/projects/pungi/ticket/34 stating that the logger needed a little work so that it didn't depend on the gather.py (or at least thats what I understood :) I propose either a new file (pungiLog.py) located in the pypungi directory or a new function in the "pungi" file that contains the logging stuff. The log services would be started somewhere before the line containing "# Actually do work." of the "pungi" file. The logging root would be called "pungi" and would be called in each file that logging is needed with the logging.getlogger("pungi") command. If "quiet" is specified in the config file the logging will be turned off. *Diff for the pungi file:* 1. Initializes the logger by calling to the new file. 2. specify quiet value. 3. logging function. * Diff for pungi.py file: *1. use the correct logger. * Diff for gather.py file:* change all the if statements for each logging call. Files attached... Comments appreciated. Regards -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: gather.py-diff URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: pungi.py-diff URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: pungi-diff URL: From jgranado at redhat.com Fri May 4 15:39:21 2007 From: jgranado at redhat.com (Joel Andres Granados) Date: Fri, 04 May 2007 17:39:21 +0200 Subject: verbose command option Message-ID: <463B53A9.8040905@redhat.com> Hi list: How about a verbose option "-v"? On the one hand it is very useful for the impatient user that needs some sort of immediate feedback. on the other hand it could be useful for other applications that might need to analyze pungis output (like revisor). The idea is to have the option off by defect and when the user specifies "-v" at the command line, all the logging information is showed. With the output I did a very simple progress bar and every thing seem to work correctly (the progress bar example is not attached, it works but its not pretty :). The verbose option contains the logging patch I posted earlier. The diffs are attached. -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: gather.py-diff URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: pungi.py-diff URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: pungi-diff URL: From jgranado at redhat.com Fri May 4 16:17:57 2007 From: jgranado at redhat.com (Joel Andres Granados) Date: Fri, 04 May 2007 18:17:57 +0200 Subject: Extend verbosity to the scripts and apps that pungi uses Message-ID: <463B5CB5.4070908@redhat.com> hi list: The idea is to allow, via a command line argument, the redirection of the output of the applications and scripts that pungi calls to the stdout of pungi. In this way the user sees all that is happening. including the things that output of the stuff that pungi calls. I havent tested this extensively and I'm just wondering what you guys think of the idea. diffs attached. Cheers. -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: gather.py-diff URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: pungi.py-diff URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: pungi-diff URL: From enrico.scholz at informatik.tu-chemnitz.de Sat May 5 17:03:49 2007 From: enrico.scholz at informatik.tu-chemnitz.de (Enrico Scholz) Date: Sat, 05 May 2007 19:03:49 +0200 Subject: [PATCH mock] Fixup of *pack() methods Message-ID: <20070505170336.20912.27422.stgit@kosh.bigo.ensc.de> Cache operations do not seem to work and use deprecated options; e.g. | DEBUG: Executing tar -zlcf /var/lib/mock/root-cache/fedora-development-i386-core.tar.gz root | tar: Semantics of -l option will change in the future releases. | tar: Please use --one-file-system option instead. | tar: root: Cannot stat: No such file or directory | tar: Error exit delayed from previous errors This patch fixes caching operations and enhances them in some aspects (e.g. do not cache downloaded packages). Signed-off-by: Enrico Scholz --- mock.py | 49 ++++++++++++++++++++++++++++++++++++------------- 1 files changed, 36 insertions(+), 13 deletions(-) diff --git a/mock.py b/mock.py index 573bb6e..3762324 100644 --- a/mock.py +++ b/mock.py @@ -239,15 +239,29 @@ class Root: else: return self._state - def unpack(self): - self.state('unpack_cache') + def __get_tar_compress_opts(self): if self.cache_file.find(".bz2") != -1: - opts = "-jxpf" + return ["-j",] elif self.cache_file.find(".gz") != -1: - opts = "-zxpf" + return ["-z",] else: - opts = "-xpf" - cmd = '%s %s %s %s' % (self.config['unpack_cmd'], opts, self.basedir, self.cache_file) + return [] + + def unpack(self): + self.state('unpack_cache') + + opts=self.__get_tar_compress_opts() + opts.extend([ + '-xpf', + self.cache_file, + '-C', + self.basedir]) + + # the quoting by map(...) is hacky; it would be better to use + # correct datatypes for the do*() methods + cmd = '%s %s' % (self.config['unpack_cmd'], + ' '.join(map(lambda x: '"%s"' % x, opts))) + self.debug("unpacking cache: %s" % cmd) (retval, output) = self.do_elevated(cmd) return retval @@ -255,13 +269,22 @@ class Root: def pack(self): self.state('create_cache') self._ensure_dir(os.path.join(self.config['basedir'], self.config['cache_topdir'])) - if self.cache_file.find(".bz2") != -1: - opts = "-jlcf" - elif self.cache_file.find(".gz") != -1: - opts = "-zlcf" - else: - opts = "-clf" - cmd = '%s %s %s root' % (self.config['pack_cmd'], opts, self.cache_file) + + opts=self.__get_tar_compress_opts() + opts.extend([ + '-cf', + self.cache_file, + '--one-file-system', + '--exclude=__db*', + '--exclude=root/var/cache/yum/*/headers', + '--exclude=root/var/cache/yum/*/packages', + 'root' + ]) + + cmd = 'cd "%s" && %s %s' % (self.basedir, + self.config['pack_cmd'], + ' '.join(map(lambda x: '"%s"' % x, opts))) + self.debug("creating cache: %s" % cmd) (retval, output) = self.do_elevated(cmd) return retval From enrico.scholz at informatik.tu-chemnitz.de Sat May 5 17:07:39 2007 From: enrico.scholz at informatik.tu-chemnitz.de (Enrico Scholz) Date: Sat, 05 May 2007 19:07:39 +0200 Subject: [PATCH mock] Enhancements of the config_opts['macros'] handling Message-ID: <20070505170638.21239.24364.stgit@kosh.bigo.ensc.de> This patch allows filling of ~/.rpmmacros by using dictionaries or lists instead of a single strings. You can write e.g. | config_opts['macros'] = [ | config_opts['macros'], | { | '%dist' : 'foo', | '%packager' : 'bar' | }] This adds old/predefined macros first and then new '%dist' and '%packager' ones. For backward compatibility strings are still supported; to retain a certain order of macros, they can be specified as tuples or lists too. Signed-off-by: Enrico Scholz --- mock.py | 21 +++++++++++++++++++-- 1 files changed, 19 insertions(+), 2 deletions(-) diff --git a/mock.py b/mock.py index e20dbf0..573bb6e 100644 --- a/mock.py +++ b/mock.py @@ -767,6 +767,19 @@ class Root: self.homedir, self.config['chrootuser']) self.do_chroot(cmd, fatal = True) + def _expand_macro_string(self, macros): + res = [] + if isinstance(macros, dict): + for (key,v) in macros.items(): + res.append('%s\t%s' % (key,v)) + elif isinstance(macros, basestring): + res = [macros] + else: + for v in macros: + res.append(self._expand_macro_string(v)) + + return '\n'.join(res) + '\n' + def _build_dir_setup(self): # purge the builddir, if it exists bd_out = '%s%s' % (self.rootdir, self.builddir) @@ -788,8 +801,12 @@ class Root: macrofile_out = '%s%s/.rpmmacros' % (self.rootdir, self.homedir) if not os.path.exists(macrofile_out): rpmmacros = open(macrofile_out, 'w') - self.config['macros'] = self.config['macros'] + "\n%%_rpmlock_path %s/var/lib/rpm/__db.000" % self.basedir - rpmmacros.write(self.config['macros']) + + rpmmacros.write(self._expand_macro_string(self.config['macros'])) + rpmmacros.write(self._expand_macro_string( + {"%_rpmlock_path" : "%s/var/lib/rpm/__db.000" % self.basedir} + )) + rpmmacros.close() From enrico.scholz at informatik.tu-chemnitz.de Sat May 5 17:30:40 2007 From: enrico.scholz at informatik.tu-chemnitz.de (Enrico Scholz) Date: Sat, 05 May 2007 19:30:40 +0200 Subject: [PATCH mock] Bind-mount a precreated /dev filesystem instead of using mknod(2) Message-ID: <20070505173019.23031.6623.stgit@kosh.bigo.ensc.de> This patch adds a 'dev-template' configuration options which points to a directory with a preconfigured /dev directory. This makes it possible to use mock in environments where 'mknod(2)' operations are prohibited (e.g. in vservers). Usually, this directory is a mounted read-only and e.g. a cramfs. The 'create-template.sh' script will create such a cramfs image which must be added to the host's /etc/fstab e.g. as | /usr/share/mock/mock-dev.ramfs /usr/share/mock/dev-template cramfs dev and added to mock configuration as | config_opts['dev-template'] = '/usr/share/mock/dev-template' The changes in the spec file are suggestions only and the cramfs image can not be created during the normal build because it requires superuser capabilities. Signed-off-by: Enrico Scholz --- Makefile | 2 ++ create-template.sh | 29 ++++++++++++++++++++++ mock-dev.ramfs | Bin mock.py | 69 +++++++++++++++++++++++++++++----------------------- mock.spec | 16 ++++++++++++ 5 files changed, 85 insertions(+), 31 deletions(-) diff --git a/Makefile b/Makefile index bee4ad4..a2a5d0b 100644 --- a/Makefile +++ b/Makefile @@ -21,6 +21,8 @@ subdirs: install: install -D mock.py $(DESTDIR)/usr/libexec/mock.py install -D mock-yum $(DESTDIR)/usr/libexec/mock-yum + install -D -m644 mock-dev.ramfs $(DESTDIR)/usr/share/mock/mock-dev.ramfs + install -d -m755 $(DESTDIR)/usr/share/mock/dev-template mkdir -p $(DESTDIR)/var/lib/mock for d in $(SUBDIRS); do make DESTDIR=`cd $(DESTDIR); pwd` -C $$d install; [ $$? = 0 ] || exit 1; done diff --git a/create-template.sh b/create-template.sh new file mode 100755 index 0000000..75e2529 --- /dev/null +++ b/create-template.sh @@ -0,0 +1,29 @@ +#! /bin/bash + +MKNOD=mknod +MKCRAMFS=/sbin/mkfs.cramfs + +set -e + +d=$(mktemp -t -d mockdev.XXXXXX) +trap "rm -rf $d" EXIT + +dev=$d/dev +mkdir -p $dev + +$MKNOD -m666 $dev/null c 1 3 +$MKNOD -m644 $dev/urandom c 1 9 +$MKNOD -m644 $dev/random c 1 8 +$MKNOD -m666 $dev/full c 1 7 +$MKNOD -m666 $dev/ptmx c 5 2 +$MKNOD -m666 $dev/tty c 5 0 +$MKNOD -m666 $dev/zero c 1 5 + +ln -s /proc/self/fd $dev/fd +ln -s /proc/self/fd/0 $dev/stdin +ln -s /proc/self/fd/1 $dev/stdout +ln -s /proc/self/fd/2 $dev/stderr + +mkdir $dev/pts + +$MKCRAMFS $dev "$1" diff --git a/mock-dev.ramfs b/mock-dev.ramfs new file mode 100644 index 0000000..03851ca Binary files /dev/null and b/mock-dev.ramfs differ diff --git a/mock.py b/mock.py index b06156c..3762324 100644 --- a/mock.py +++ b/mock.py @@ -497,6 +497,10 @@ class Root: # mount /proc self._mount('proc', 'proc', 'proc') + # mount /dev + if self.config.get('dev-template'): + self._mount('none --bind -o ro', self.config['dev-template'], 'dev') + # mount /dev/pts self._mount('devpts', 'dev/pts', 'dev/pts') @@ -536,7 +540,38 @@ class Root: # poof, no more file if os.path.exists(mf): os.unlink(mf) + + def __mknodall(self): + # we need stuff + devices = [('null', 'c', '1', '3', '666'), + ('urandom', 'c', '1', '9', '644'), + ('random', 'c', '1', '9', '644'), + ('full', 'c', '1', '7', '666'), + ('ptmx', 'c', '5', '2', '666'), + ('tty', 'c', '5', '0', '666'), + ('zero', 'c', '1', '5', '666')] + + for (dev, devtype, major, minor, perm) in devices: + devpath = os.path.join(self.rootdir, 'dev', dev) + cmd = '%s %s -m %s %s %s %s' % (self.config['mknod'], + devpath, perm, devtype, major, minor) + if not os.path.exists(devpath): + (retval, output) = self.do_elevated(cmd) + if retval != 0: + raise RootError, "could not mknod error was: %s" % output + + # link fd to ../proc/self/fd + devpath = os.path.join(self.rootdir, 'dev/fd') + if not os.path.exists(devpath): + os.symlink('../proc/self/fd', devpath) + fd = 0 + for item in ('stdin', 'stdout', 'stderr'): + devpath = os.path.join(self.rootdir, 'dev', item) + if not os.path.exists(devpath): + fdpath = os.path.join('../proc/self/fd', str(fd)) + os.symlink(fdpath, devpath) + fd += 1 def do(self, command): """execute given command outside of chroot""" @@ -679,39 +714,11 @@ class Root: os.path.join(self.rootdir, 'var/tmp'), os.path.join(self.rootdir, 'etc/yum.repos.d')]: self._ensure_dir(item) - - self._mountall() - # we need stuff - devices = [('null', 'c', '1', '3', '666'), - ('urandom', 'c', '1', '9', '644'), - ('random', 'c', '1', '9', '644'), - ('full', 'c', '1', '7', '666'), - ('ptmx', 'c', '5', '2', '666'), - ('tty', 'c', '5', '0', '666'), - ('zero', 'c', '1', '5', '666')] - - for (dev, devtype, major, minor, perm) in devices: - devpath = os.path.join(self.rootdir, 'dev', dev) - cmd = '%s %s -m %s %s %s %s' % (self.config['mknod'], - devpath, perm, devtype, major, minor) - if not os.path.exists(devpath): - (retval, output) = self.do_elevated(cmd) - if retval != 0: - raise RootError, "could not mknod error was: %s" % output + self._mountall() - # link fd to ../proc/self/fd - devpath = os.path.join(self.rootdir, 'dev/fd') - if not os.path.exists(devpath): - os.symlink('../proc/self/fd', devpath) - - fd = 0 - for item in ('stdin', 'stdout', 'stderr'): - devpath = os.path.join(self.rootdir, 'dev', item) - if not os.path.exists(devpath): - fdpath = os.path.join('../proc/self/fd', str(fd)) - os.symlink(fdpath, devpath) - fd += 1 + if not self.config.get('dev-template'): + self.__mknodall() for item in [os.path.join(self.rootdir, 'etc', 'mtab'), os.path.join(self.rootdir, 'etc', 'fstab'), diff --git a/mock.spec b/mock.spec index db59881..309e290 100644 --- a/mock.spec +++ b/mock.spec @@ -33,6 +33,7 @@ if [ -f fedora-%{fedora}-%{_target_cpu}-core.cfg ]; then fi %endif + # if we haven't created a default link yet, try to do so as devel if [ ! -f default.cfg ]; then if [ -f fedora-development-%{_target_cpu}-core.cfg ]; then @@ -54,6 +55,12 @@ if [ $1 -eq 1 ]; then groupadd -r mock >/dev/null 2>&1 || : fi +%post +mkdir -p -m0755 %_datadir/mock/dev-template + +%preun +test $1 != 0 || rmdir %_datadir/mock/dev-template || : + %files %defattr(-, root, root) %doc README ChangeLog buildsys-build.spec @@ -65,6 +72,15 @@ fi %attr(02775, root, mock) %dir /var/lib/mock %{_libdir}/libselinux-mock.so +# Do not ship the dev-template directory; it might be mounted during +# upgrades which will cpio errors. Adding it to %%_netsharedpath is a +# solution but requires manual editing of /etc/rpm/macros. +# +# Hence, create and remove this directory in scriptlets +%dir %_datadir/mock +%dir %_datadir/mock/*.ramfs +%ghost %dir %_datadir/mock/dev-template + %changelog * Mon Jan 8 2007 Clark Williams - Added Josh Boyer's EPEL config files From enrico.scholz at informatik.tu-chemnitz.de Sat May 5 18:47:49 2007 From: enrico.scholz at informatik.tu-chemnitz.de (Enrico Scholz) Date: Sat, 05 May 2007 20:47:49 +0200 Subject: [PATCH mock] Allow installation when /dev is mounted read-only In-Reply-To: <20070505173019.23031.6623.stgit@kosh.bigo.ensc.de> References: <20070505173019.23031.6623.stgit@kosh.bigo.ensc.de> Message-ID: <20070505184740.4418.39903.stgit@kosh.bigo.ensc.de> This patch bind-mounts a custom /etc/rpm directory to set the %_netsharedpath macro before doing the 'yum install' operation. Else, installation of filesystem package will fail due to a cpio error. Signed-off-by: Enrico Scholz --- mock.py | 12 +++++++++++- 1 files changed, 11 insertions(+), 1 deletions(-) diff --git a/mock.py b/mock.py index 3762324..6061907 100644 --- a/mock.py +++ b/mock.py @@ -500,6 +500,7 @@ class Root: # mount /dev if self.config.get('dev-template'): self._mount('none --bind -o ro', self.config['dev-template'], 'dev') + self._mount('none --bind', os.path.join(self.statedir, 'rpm'), '/etc/rpm') # mount /dev/pts self._mount('devpts', 'dev/pts', 'dev/pts') @@ -698,7 +699,15 @@ class Root: ret = rpmUtils.miscutils.unique(reqlist) self.drop() return ret - + + def __create_rpm_macros_for_install(self): + p = os.path.join(self.statedir, 'rpm') + fname = os.path.join(p, 'macros') + if not os.path.exists(fname): + self._ensure_dir(p) + f = open(fname, 'w') + f.write(self._expand_macro_string({'%_netsharedpath' : '/dev'})) + def _prep_install(self): """prep chroot for installation""" # make chroot dir @@ -715,6 +724,7 @@ class Root: os.path.join(self.rootdir, 'etc/yum.repos.d')]: self._ensure_dir(item) + self.__create_rpm_macros_for_install() self._mountall() if not self.config.get('dev-template'): From enrico.scholz at informatik.tu-chemnitz.de Sat May 5 18:56:17 2007 From: enrico.scholz at informatik.tu-chemnitz.de (Enrico Scholz) Date: Sat, 05 May 2007 20:56:17 +0200 Subject: [PATCH mock] bind-mount the /var/lib/rpm from the chroot into the root directory Message-ID: <20070505185559.4854.26101.stgit@kosh.bigo.ensc.de> This patch should not be needed when rpm would work properly... There was already a try to fix this but setting '%_rpmlock_path'. But this never worked because these macros will not be evaluated at 'yum install' time. Last hunk requires the "Enhancements of the config_opts['macros'] handling" patch. Signed-off-by: Enrico Scholz --- mock.py | 10 +++------- 1 files changed, 3 insertions(+), 7 deletions(-) diff --git a/mock.py b/mock.py index 6061907..df05920 100644 --- a/mock.py +++ b/mock.py @@ -502,6 +502,9 @@ class Root: self._mount('none --bind -o ro', self.config['dev-template'], 'dev') self._mount('none --bind', os.path.join(self.statedir, 'rpm'), '/etc/rpm') + # provide our view of /var/lib/rpm + self._mount('none --bind', os.path.join(self.rootdir, 'var/lib/rpm'), '/var/lib/rpm') + # mount /dev/pts self._mount('devpts', 'dev/pts', 'dev/pts') @@ -834,14 +837,7 @@ class Root: macrofile_out = '%s%s/.rpmmacros' % (self.rootdir, self.homedir) if not os.path.exists(macrofile_out): rpmmacros = open(macrofile_out, 'w') - rpmmacros.write(self._expand_macro_string(self.config['macros'])) - rpmmacros.write(self._expand_macro_string( - {"%_rpmlock_path" : "%s/var/lib/rpm/__db.000" % self.basedir} - )) - - rpmmacros.close() - def _prep_build(self): """prep the chroot for building packages""" From enrico.scholz at informatik.tu-chemnitz.de Mon May 7 06:53:47 2007 From: enrico.scholz at informatik.tu-chemnitz.de (Enrico Scholz) Date: Mon, 7 May 2007 08:53:47 +0200 Subject: [PATCH koji] Made 'mockpath' configurable for kojid Message-ID: <200705070700.l4770Cxa005859@kosh.bigo.ensc.de> Trying to set 'mockpath' in the configuration file results into an | unknown config option: mockpath error. Signed-off-by: Enrico Scholz --- builder/kojid | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/builder/kojid b/builder/kojid index 7a66bfc..6243edd 100755 --- a/builder/kojid +++ b/builder/kojid @@ -338,7 +338,7 @@ class BuildRoot(object): def mock(self, args, skip_setarch=False): """Run mock""" global options - mockpath = getattr(options,"mockpath","/usr/bin/mock") + mockpath = getattr(options,"mockpath") cmd = [mockpath, "-r", self.mockcfg] if not skip_setarch: if self.br_arch.startswith('sparc64'): @@ -2366,6 +2366,7 @@ def get_options(): 'workdir': '/tmp/koji', 'mockdir': '/var/lib/mock', 'mockuser': 'kojibuilder', + 'mockpath': '/usr/bin/mock', 'packager': 'Koji', 'vendor': 'Koji', 'mockhost': 'koji-linux-gnu', -- 1.5.0.6 From Michael_E_Brown at dell.com Mon May 7 17:01:51 2007 From: Michael_E_Brown at dell.com (Michael E Brown) Date: Mon, 7 May 2007 12:01:51 -0500 Subject: [PATCH mock] Fixup of *pack() methods In-Reply-To: <20070505170336.20912.27422.stgit@kosh.bigo.ensc.de> References: <20070505170336.20912.27422.stgit@kosh.bigo.ensc.de> Message-ID: <20070507170151.GF29655@humbolt.us.dell.com> On Sat, May 05, 2007 at 07:03:49PM +0200, Enrico Scholz wrote: > Cache operations do not seem to work and use deprecated options; e.g. > > | DEBUG: Executing tar -zlcf /var/lib/mock/root-cache/fedora-development-i386-core.tar.gz root > | tar: Semantics of -l option will change in the future releases. > | tar: Please use --one-file-system option instead. > | tar: root: Cannot stat: No such file or directory > | tar: Error exit delayed from previous errors this patch looks sane. Clark and I are sorting git issues now. It will be applied soon. -- Michael > > This patch fixes caching operations and enhances them in some aspects > (e.g. do not cache downloaded packages). > > Signed-off-by: Enrico Scholz > --- > > mock.py | 49 ++++++++++++++++++++++++++++++++++++------------- > 1 files changed, 36 insertions(+), 13 deletions(-) > > diff --git a/mock.py b/mock.py > index 573bb6e..3762324 100644 > --- a/mock.py > +++ b/mock.py > @@ -239,15 +239,29 @@ class Root: > else: > return self._state > > - def unpack(self): > - self.state('unpack_cache') > + def __get_tar_compress_opts(self): > if self.cache_file.find(".bz2") != -1: > - opts = "-jxpf" > + return ["-j",] > elif self.cache_file.find(".gz") != -1: > - opts = "-zxpf" > + return ["-z",] > else: > - opts = "-xpf" > - cmd = '%s %s %s %s' % (self.config['unpack_cmd'], opts, self.basedir, self.cache_file) > + return [] > + > + def unpack(self): > + self.state('unpack_cache') > + > + opts=self.__get_tar_compress_opts() > + opts.extend([ > + '-xpf', > + self.cache_file, > + '-C', > + self.basedir]) > + > + # the quoting by map(...) is hacky; it would be better to use > + # correct datatypes for the do*() methods > + cmd = '%s %s' % (self.config['unpack_cmd'], > + ' '.join(map(lambda x: '"%s"' % x, opts))) > + > self.debug("unpacking cache: %s" % cmd) > (retval, output) = self.do_elevated(cmd) > return retval > @@ -255,13 +269,22 @@ class Root: > def pack(self): > self.state('create_cache') > self._ensure_dir(os.path.join(self.config['basedir'], self.config['cache_topdir'])) > - if self.cache_file.find(".bz2") != -1: > - opts = "-jlcf" > - elif self.cache_file.find(".gz") != -1: > - opts = "-zlcf" > - else: > - opts = "-clf" > - cmd = '%s %s %s root' % (self.config['pack_cmd'], opts, self.cache_file) > + > + opts=self.__get_tar_compress_opts() > + opts.extend([ > + '-cf', > + self.cache_file, > + '--one-file-system', > + '--exclude=__db*', > + '--exclude=root/var/cache/yum/*/headers', > + '--exclude=root/var/cache/yum/*/packages', > + 'root' > + ]) > + > + cmd = 'cd "%s" && %s %s' % (self.basedir, > + self.config['pack_cmd'], > + ' '.join(map(lambda x: '"%s"' % x, opts))) > + > self.debug("creating cache: %s" % cmd) > (retval, output) = self.do_elevated(cmd) > return retval > > -- > Fedora-buildsys-list mailing list > Fedora-buildsys-list at redhat.com > https://www.redhat.com/mailman/listinfo/fedora-buildsys-list From Michael_E_Brown at dell.com Mon May 7 17:07:55 2007 From: Michael_E_Brown at dell.com (Michael E Brown) Date: Mon, 7 May 2007 12:07:55 -0500 Subject: [PATCH mock] Enhancements of the config_opts['macros'] handling In-Reply-To: <20070505170638.21239.24364.stgit@kosh.bigo.ensc.de> References: <20070505170638.21239.24364.stgit@kosh.bigo.ensc.de> Message-ID: <20070507170754.GG29655@humbolt.us.dell.com> On Sat, May 05, 2007 at 07:07:39PM +0200, Enrico Scholz wrote: > This patch allows filling of ~/.rpmmacros by using dictionaries or > lists instead of a single strings. You can write e.g. > > | config_opts['macros'] = [ > | config_opts['macros'], > | { > | '%dist' : 'foo', > | '%packager' : 'bar' > | }] > > This adds old/predefined macros first and then new '%dist' and > '%packager' ones. This too, looks entirely sane, except for one question and one nit... > > For backward compatibility strings are still supported; to retain a > certain order of macros, they can be specified as tuples or lists too. > > Signed-off-by: Enrico Scholz > --- > > mock.py | 21 +++++++++++++++++++-- > 1 files changed, 19 insertions(+), 2 deletions(-) > > diff --git a/mock.py b/mock.py > index e20dbf0..573bb6e 100644 > --- a/mock.py > +++ b/mock.py > @@ -767,6 +767,19 @@ class Root: > self.homedir, self.config['chrootuser']) > self.do_chroot(cmd, fatal = True) > > + def _expand_macro_string(self, macros): > + res = [] > + if isinstance(macros, dict): > + for (key,v) in macros.items(): > + res.append('%s\t%s' % (key,v)) > + elif isinstance(macros, basestring): > + res = [macros] > + else: > + for v in macros: > + res.append(self._expand_macro_string(v)) > + > + return '\n'.join(res) + '\n' How does this affect the ability to overwrite config from a config file higher in the heirarchy? > + > def _build_dir_setup(self): > # purge the builddir, if it exists > bd_out = '%s%s' % (self.rootdir, self.builddir) > @@ -788,8 +801,12 @@ class Root: > macrofile_out = '%s%s/.rpmmacros' % (self.rootdir, self.homedir) > if not os.path.exists(macrofile_out): > rpmmacros = open(macrofile_out, 'w') > - self.config['macros'] = self.config['macros'] + "\n%%_rpmlock_path %s/var/lib/rpm/__db.000" % self.basedir > - rpmmacros.write(self.config['macros']) > + > + rpmmacros.write(self._expand_macro_string(self.config['macros'])) > + rpmmacros.write(self._expand_macro_string( > + {"%_rpmlock_path" : "%s/var/lib/rpm/__db.000" % self.basedir} > + )) This should go into the default config section, rather than hidden here. You need to convert the setup functions to use the new format by default, as well as any existing config file references in the default supplied configs. -- Michael From Michael_E_Brown at dell.com Mon May 7 17:31:59 2007 From: Michael_E_Brown at dell.com (Michael E Brown) Date: Mon, 7 May 2007 12:31:59 -0500 Subject: [PATCH mock] Bind-mount a precreated /dev filesystem instead of using mknod(2) In-Reply-To: <20070505173019.23031.6623.stgit@kosh.bigo.ensc.de> References: <20070505173019.23031.6623.stgit@kosh.bigo.ensc.de> Message-ID: <20070507173159.GH29655@humbolt.us.dell.com> On Sat, May 05, 2007 at 07:30:40PM +0200, Enrico Scholz wrote: > This patch adds a 'dev-template' configuration options which points to a > directory with a preconfigured /dev directory. This makes it possible to > use mock in environments where 'mknod(2)' operations are prohibited (e.g. > in vservers). > > Usually, this directory is a mounted read-only and e.g. a cramfs. The > 'create-template.sh' script will create such a cramfs image which must be > added to the host's /etc/fstab e.g. as > > | /usr/share/mock/mock-dev.ramfs /usr/share/mock/dev-template cramfs dev > > and added to mock configuration as > > | config_opts['dev-template'] = '/usr/share/mock/dev-template' 1) Need to add default config option for this in setup_default_config_opts(). 2) need to discuss spec file changes, because, as you say, they cannot go in like that. -- Michael > > > The changes in the spec file are suggestions only and the cramfs image > can not be created during the normal build because it requires superuser > capabilities. > > Signed-off-by: Enrico Scholz > --- > > Makefile | 2 ++ > create-template.sh | 29 ++++++++++++++++++++++ > mock-dev.ramfs | Bin > mock.py | 69 +++++++++++++++++++++++++++++----------------------- > mock.spec | 16 ++++++++++++ > 5 files changed, 85 insertions(+), 31 deletions(-) > > diff --git a/Makefile b/Makefile > index bee4ad4..a2a5d0b 100644 > --- a/Makefile > +++ b/Makefile > @@ -21,6 +21,8 @@ subdirs: > install: > install -D mock.py $(DESTDIR)/usr/libexec/mock.py > install -D mock-yum $(DESTDIR)/usr/libexec/mock-yum > + install -D -m644 mock-dev.ramfs $(DESTDIR)/usr/share/mock/mock-dev.ramfs > + install -d -m755 $(DESTDIR)/usr/share/mock/dev-template > mkdir -p $(DESTDIR)/var/lib/mock > for d in $(SUBDIRS); do make DESTDIR=`cd $(DESTDIR); pwd` -C $$d install; [ $$? = 0 ] || exit 1; done > > diff --git a/create-template.sh b/create-template.sh > new file mode 100755 > index 0000000..75e2529 > --- /dev/null > +++ b/create-template.sh > @@ -0,0 +1,29 @@ > +#! /bin/bash > + > +MKNOD=mknod > +MKCRAMFS=/sbin/mkfs.cramfs > + > +set -e > + > +d=$(mktemp -t -d mockdev.XXXXXX) > +trap "rm -rf $d" EXIT > + > +dev=$d/dev > +mkdir -p $dev > + > +$MKNOD -m666 $dev/null c 1 3 > +$MKNOD -m644 $dev/urandom c 1 9 > +$MKNOD -m644 $dev/random c 1 8 > +$MKNOD -m666 $dev/full c 1 7 > +$MKNOD -m666 $dev/ptmx c 5 2 > +$MKNOD -m666 $dev/tty c 5 0 > +$MKNOD -m666 $dev/zero c 1 5 > + > +ln -s /proc/self/fd $dev/fd > +ln -s /proc/self/fd/0 $dev/stdin > +ln -s /proc/self/fd/1 $dev/stdout > +ln -s /proc/self/fd/2 $dev/stderr > + > +mkdir $dev/pts > + > +$MKCRAMFS $dev "$1" > diff --git a/mock-dev.ramfs b/mock-dev.ramfs > new file mode 100644 > index 0000000..03851ca > Binary files /dev/null and b/mock-dev.ramfs differ > diff --git a/mock.py b/mock.py > index b06156c..3762324 100644 > --- a/mock.py > +++ b/mock.py > @@ -497,6 +497,10 @@ class Root: > # mount /proc > self._mount('proc', 'proc', 'proc') > > + # mount /dev > + if self.config.get('dev-template'): > + self._mount('none --bind -o ro', self.config['dev-template'], 'dev') > + > # mount /dev/pts > self._mount('devpts', 'dev/pts', 'dev/pts') > > @@ -536,7 +540,38 @@ class Root: > # poof, no more file > if os.path.exists(mf): > os.unlink(mf) > + > + def __mknodall(self): > + # we need stuff > + devices = [('null', 'c', '1', '3', '666'), > + ('urandom', 'c', '1', '9', '644'), > + ('random', 'c', '1', '9', '644'), > + ('full', 'c', '1', '7', '666'), > + ('ptmx', 'c', '5', '2', '666'), > + ('tty', 'c', '5', '0', '666'), > + ('zero', 'c', '1', '5', '666')] > + > + for (dev, devtype, major, minor, perm) in devices: > + devpath = os.path.join(self.rootdir, 'dev', dev) > + cmd = '%s %s -m %s %s %s %s' % (self.config['mknod'], > + devpath, perm, devtype, major, minor) > + if not os.path.exists(devpath): > + (retval, output) = self.do_elevated(cmd) > + if retval != 0: > + raise RootError, "could not mknod error was: %s" % output > + > + # link fd to ../proc/self/fd > + devpath = os.path.join(self.rootdir, 'dev/fd') > + if not os.path.exists(devpath): > + os.symlink('../proc/self/fd', devpath) > > + fd = 0 > + for item in ('stdin', 'stdout', 'stderr'): > + devpath = os.path.join(self.rootdir, 'dev', item) > + if not os.path.exists(devpath): > + fdpath = os.path.join('../proc/self/fd', str(fd)) > + os.symlink(fdpath, devpath) > + fd += 1 > > def do(self, command): > """execute given command outside of chroot""" > @@ -679,39 +714,11 @@ class Root: > os.path.join(self.rootdir, 'var/tmp'), > os.path.join(self.rootdir, 'etc/yum.repos.d')]: > self._ensure_dir(item) > - > - self._mountall() > > - # we need stuff > - devices = [('null', 'c', '1', '3', '666'), > - ('urandom', 'c', '1', '9', '644'), > - ('random', 'c', '1', '9', '644'), > - ('full', 'c', '1', '7', '666'), > - ('ptmx', 'c', '5', '2', '666'), > - ('tty', 'c', '5', '0', '666'), > - ('zero', 'c', '1', '5', '666')] > - > - for (dev, devtype, major, minor, perm) in devices: > - devpath = os.path.join(self.rootdir, 'dev', dev) > - cmd = '%s %s -m %s %s %s %s' % (self.config['mknod'], > - devpath, perm, devtype, major, minor) > - if not os.path.exists(devpath): > - (retval, output) = self.do_elevated(cmd) > - if retval != 0: > - raise RootError, "could not mknod error was: %s" % output > + self._mountall() > > - # link fd to ../proc/self/fd > - devpath = os.path.join(self.rootdir, 'dev/fd') > - if not os.path.exists(devpath): > - os.symlink('../proc/self/fd', devpath) > - > - fd = 0 > - for item in ('stdin', 'stdout', 'stderr'): > - devpath = os.path.join(self.rootdir, 'dev', item) > - if not os.path.exists(devpath): > - fdpath = os.path.join('../proc/self/fd', str(fd)) > - os.symlink(fdpath, devpath) > - fd += 1 > + if not self.config.get('dev-template'): > + self.__mknodall() > > for item in [os.path.join(self.rootdir, 'etc', 'mtab'), > os.path.join(self.rootdir, 'etc', 'fstab'), > diff --git a/mock.spec b/mock.spec > index db59881..309e290 100644 > --- a/mock.spec > +++ b/mock.spec > @@ -33,6 +33,7 @@ if [ -f fedora-%{fedora}-%{_target_cpu}-core.cfg ]; then > fi > %endif > > + > # if we haven't created a default link yet, try to do so as devel > if [ ! -f default.cfg ]; then > if [ -f fedora-development-%{_target_cpu}-core.cfg ]; then > @@ -54,6 +55,12 @@ if [ $1 -eq 1 ]; then > groupadd -r mock >/dev/null 2>&1 || : > fi > > +%post > +mkdir -p -m0755 %_datadir/mock/dev-template > + > +%preun > +test $1 != 0 || rmdir %_datadir/mock/dev-template || : > + > %files > %defattr(-, root, root) > %doc README ChangeLog buildsys-build.spec > @@ -65,6 +72,15 @@ fi > %attr(02775, root, mock) %dir /var/lib/mock > %{_libdir}/libselinux-mock.so > > +# Do not ship the dev-template directory; it might be mounted during > +# upgrades which will cpio errors. Adding it to %%_netsharedpath is a > +# solution but requires manual editing of /etc/rpm/macros. > +# > +# Hence, create and remove this directory in scriptlets > +%dir %_datadir/mock > +%dir %_datadir/mock/*.ramfs > +%ghost %dir %_datadir/mock/dev-template > + > %changelog > * Mon Jan 8 2007 Clark Williams > - Added Josh Boyer's EPEL config files > > -- > Fedora-buildsys-list mailing list > Fedora-buildsys-list at redhat.com > https://www.redhat.com/mailman/listinfo/fedora-buildsys-list From Michael_E_Brown at dell.com Mon May 7 17:34:17 2007 From: Michael_E_Brown at dell.com (Michael E Brown) Date: Mon, 7 May 2007 12:34:17 -0500 Subject: [PATCH mock] Allow installation when /dev is mounted read-only In-Reply-To: <20070505184740.4418.39903.stgit@kosh.bigo.ensc.de> References: <20070505173019.23031.6623.stgit@kosh.bigo.ensc.de> <20070505184740.4418.39903.stgit@kosh.bigo.ensc.de> Message-ID: <20070507173416.GI29655@humbolt.us.dell.com> On Sat, May 05, 2007 at 08:47:49PM +0200, Enrico Scholz wrote: > This patch bind-mounts a custom /etc/rpm directory to set the > %_netsharedpath macro before doing the 'yum install' operation. > > Else, installation of filesystem package will fail due to a cpio error. This one is required IFF previous /dev patch is applied, correct? Probably should mention that in the description. Can this be done in .rpmmacros instead? -- Michael > > Signed-off-by: Enrico Scholz > --- > > mock.py | 12 +++++++++++- > 1 files changed, 11 insertions(+), 1 deletions(-) > > diff --git a/mock.py b/mock.py > index 3762324..6061907 100644 > --- a/mock.py > +++ b/mock.py > @@ -500,6 +500,7 @@ class Root: > # mount /dev > if self.config.get('dev-template'): > self._mount('none --bind -o ro', self.config['dev-template'], 'dev') > + self._mount('none --bind', os.path.join(self.statedir, 'rpm'), '/etc/rpm') > > # mount /dev/pts > self._mount('devpts', 'dev/pts', 'dev/pts') > @@ -698,7 +699,15 @@ class Root: > ret = rpmUtils.miscutils.unique(reqlist) > self.drop() > return ret > - > + > + def __create_rpm_macros_for_install(self): > + p = os.path.join(self.statedir, 'rpm') > + fname = os.path.join(p, 'macros') > + if not os.path.exists(fname): > + self._ensure_dir(p) > + f = open(fname, 'w') > + f.write(self._expand_macro_string({'%_netsharedpath' : '/dev'})) > + > def _prep_install(self): > """prep chroot for installation""" > # make chroot dir > @@ -715,6 +724,7 @@ class Root: > os.path.join(self.rootdir, 'etc/yum.repos.d')]: > self._ensure_dir(item) > > + self.__create_rpm_macros_for_install() > self._mountall() > > if not self.config.get('dev-template'): > > -- > Fedora-buildsys-list mailing list > Fedora-buildsys-list at redhat.com > https://www.redhat.com/mailman/listinfo/fedora-buildsys-list From Michael_E_Brown at dell.com Mon May 7 17:36:47 2007 From: Michael_E_Brown at dell.com (Michael E Brown) Date: Mon, 7 May 2007 12:36:47 -0500 Subject: [PATCH mock] bind-mount the /var/lib/rpm from the chroot into the root directory In-Reply-To: <20070505185559.4854.26101.stgit@kosh.bigo.ensc.de> References: <20070505185559.4854.26101.stgit@kosh.bigo.ensc.de> Message-ID: <20070507173647.GJ29655@humbolt.us.dell.com> On Sat, May 05, 2007 at 08:56:17PM +0200, Enrico Scholz wrote: > This patch should not be needed when rpm would work properly... There was > already a try to fix this but setting '%_rpmlock_path'. But this never > worked because these macros will not be evaluated at 'yum install' time. > > Last hunk requires the "Enhancements of the config_opts['macros'] handling" > patch. > > Signed-off-by: Enrico Scholz > --- > > mock.py | 10 +++------- > 1 files changed, 3 insertions(+), 7 deletions(-) > > diff --git a/mock.py b/mock.py > index 6061907..df05920 100644 > --- a/mock.py > +++ b/mock.py > @@ -502,6 +502,9 @@ class Root: > self._mount('none --bind -o ro', self.config['dev-template'], 'dev') > self._mount('none --bind', os.path.join(self.statedir, 'rpm'), '/etc/rpm') > > + # provide our view of /var/lib/rpm > + self._mount('none --bind', os.path.join(self.rootdir, 'var/lib/rpm'), '/var/lib/rpm') Is this the best way to fix this? What are the other options? (open question...) > + > # mount /dev/pts > self._mount('devpts', 'dev/pts', 'dev/pts') > > @@ -834,14 +837,7 @@ class Root: > macrofile_out = '%s%s/.rpmmacros' % (self.rootdir, self.homedir) > if not os.path.exists(macrofile_out): > rpmmacros = open(macrofile_out, 'w') > - > rpmmacros.write(self._expand_macro_string(self.config['macros'])) > - rpmmacros.write(self._expand_macro_string( > - {"%_rpmlock_path" : "%s/var/lib/rpm/__db.000" % self.basedir} > - )) > - > - rpmmacros.close() You just removed the close() on the macros file. This was probably not intended. -- Michael From enrico.scholz at informatik.tu-chemnitz.de Mon May 7 18:13:35 2007 From: enrico.scholz at informatik.tu-chemnitz.de (Enrico Scholz) Date: Mon, 07 May 2007 20:13:35 +0200 Subject: [PATCH mock] Enhancements of the config_opts['macros'] handling In-Reply-To: <20070507170754.GG29655@humbolt.us.dell.com> (Michael E. Brown's message of "Mon, 7 May 2007 12:07:55 -0500") References: <20070505170638.21239.24364.stgit@kosh.bigo.ensc.de> <20070507170754.GG29655@humbolt.us.dell.com> Message-ID: <87mz0gfqnk.fsf@kosh.bigo.ensc.de> Michael E Brown writes: >> + def _expand_macro_string(self, macros): >> + res = [] >> + if isinstance(macros, dict): >> + for (key,v) in macros.items(): >> + res.append('%s\t%s' % (key,v)) >> + elif isinstance(macros, basestring): >> + res = [macros] >> + else: >> + for v in macros: >> + res.append(self._expand_macro_string(v)) >> + >> + return '\n'.join(res) + '\n' > > How does this affect the ability to overwrite config from a config file > higher in the heirarchy? You can write e.g. | -- defaults.cfg -- | config_opts['macros'] = "%foo 1" | | -- my-root.cfg == | config_opts['macros'] = [ | config_opts['macros'], | { | '%bar' : 2 | } | ] Overwriting is not possible/easy when we want to retain backward compatibility. When implemented from scratch, I would have implemented this option completely as a dictionary (I do not think that order of macros is for much interest). > >> + >> def _build_dir_setup(self): >> # purge the builddir, if it exists >> bd_out = '%s%s' % (self.rootdir, self.builddir) >> @@ -788,8 +801,12 @@ class Root: >> macrofile_out = '%s%s/.rpmmacros' % (self.rootdir, self.homedir) >> if not os.path.exists(macrofile_out): >> rpmmacros = open(macrofile_out, 'w') >> - self.config['macros'] = self.config['macros'] + "\n%%_rpmlock_path %s/var/lib/rpm/__db.000" % self.basedir >> - rpmmacros.write(self.config['macros']) >> + >> + rpmmacros.write(self._expand_macro_string(self.config['macros'])) >> + rpmmacros.write(self._expand_macro_string( >> + {"%_rpmlock_path" : "%s/var/lib/rpm/__db.000" % self.basedir} >> + )) > > This should go into the default config section, rather than hidden > here. I just rewrote this hunk of code. In a later version of my mock modifications I removed this completely and replaced it by bind-mounting the chrooted /var/lib/rpm directory into toplevel /var/lib/rpm. > You need to convert the setup functions to use the new format by > default, as well as any existing config file references in the > default supplied configs. See above for statement regarding backward compatibility; as long as you do not change the default string-type, old/existing configurations should still work. When defaults would be dictionaries or vectors, overwriting can fail when people append an string. Enrico -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 480 bytes Desc: not available URL: From enrico.scholz at informatik.tu-chemnitz.de Mon May 7 18:51:12 2007 From: enrico.scholz at informatik.tu-chemnitz.de (Enrico Scholz) Date: Mon, 07 May 2007 20:51:12 +0200 Subject: [PATCH mock] Allow installation when /dev is mounted read-only In-Reply-To: <20070507173416.GI29655@humbolt.us.dell.com> (Michael E. Brown's message of "Mon, 7 May 2007 12:34:17 -0500") References: <20070505173019.23031.6623.stgit@kosh.bigo.ensc.de> <20070505184740.4418.39903.stgit@kosh.bigo.ensc.de> <20070507173416.GI29655@humbolt.us.dell.com> Message-ID: <87fy68fowv.fsf@kosh.bigo.ensc.de> Michael E Brown writes: >> This patch bind-mounts a custom /etc/rpm directory to set the >> %_netsharedpath macro before doing the 'yum install' operation. >> >> Else, installation of filesystem package will fail due to a cpio error. > > This one is required IFF previous /dev patch is applied, correct? yes; I thought that the References: header should make this clear. > Can this be done in .rpmmacros instead? no; there is to differ between two situations: 1. 'yum install' This is executed as root and reads host's /etc/rpm/macros file (and /etc/rpm/platform). Afais, this case is completely ignored in current mock ~/.rpmmacros and other rpm configuration in the chroot will be completely ignored. 2. 'rpmbuild' This is executed as the mock-user in the chroot and reads ~/.rpmmacros >> + self._mount('none --bind', os.path.join(self.statedir, 'rpm'), '/etc/rpm') Enrico -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 480 bytes Desc: not available URL: From enrico.scholz at informatik.tu-chemnitz.de Mon May 7 18:55:06 2007 From: enrico.scholz at informatik.tu-chemnitz.de (Enrico Scholz) Date: Mon, 07 May 2007 20:55:06 +0200 Subject: [PATCH mock] Bind-mount a precreated /dev filesystem instead of using mknod(2) In-Reply-To: <20070507173159.GH29655@humbolt.us.dell.com> (Michael E. Brown's message of "Mon, 7 May 2007 12:31:59 -0500") References: <20070505173019.23031.6623.stgit@kosh.bigo.ensc.de> <20070507173159.GH29655@humbolt.us.dell.com> Message-ID: <87bqgwfoqd.fsf@kosh.bigo.ensc.de> Michael E Brown writes: >> | config_opts['dev-template'] = '/usr/share/mock/dev-template' > > 1) Need to add default config option for this in > setup_default_config_opts(). I am not sure whether it shall be enabled by default. It requires manual configuration (/etc/fstab, perhaps /etc/rpm/macros) and mock will not work out of the box. Then, mounting a dev-fs below /usr/share might be unwanted for some people. Something below /srv will be better but since not covered by FHS, it will be impossible to find a default. Enrico -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 480 bytes Desc: not available URL: From Michael_E_Brown at dell.com Mon May 7 18:58:08 2007 From: Michael_E_Brown at dell.com (Michael E Brown) Date: Mon, 7 May 2007 13:58:08 -0500 Subject: [PATCH mock] Enhancements of the config_opts['macros'] handling In-Reply-To: <87mz0gfqnk.fsf@kosh.bigo.ensc.de> References: <20070505170638.21239.24364.stgit@kosh.bigo.ensc.de> <20070507170754.GG29655@humbolt.us.dell.com> <87mz0gfqnk.fsf@kosh.bigo.ensc.de> Message-ID: <20070507185808.GK29655@humbolt.us.dell.com> On Mon, May 07, 2007 at 08:13:35PM +0200, Enrico Scholz wrote: > Michael E Brown writes: > > >> + def _expand_macro_string(self, macros): > >> + res = [] > >> + if isinstance(macros, dict): > >> + for (key,v) in macros.items(): > >> + res.append('%s\t%s' % (key,v)) > >> + elif isinstance(macros, basestring): > >> + res = [macros] > >> + else: > >> + for v in macros: > >> + res.append(self._expand_macro_string(v)) > >> + > >> + return '\n'.join(res) + '\n' > > > > How does this affect the ability to overwrite config from a config file > > higher in the heirarchy? > > You can write e.g. > > | -- defaults.cfg -- > | config_opts['macros'] = "%foo 1" If something like this goes in, I expect that to become the defualt for everything we ship. In this case: - setup_default_config_opts() would set up defaults using new format. - defaults.cfg probably wouldnt have this option - get rid of the recursive processing. If the option is a string, return it. If it is not a string, use dict processing model. This means: - everything we ship uses new format - old users of this option still work but are encouraged to switch to the new format Additionally: - We need a way to override options in subconfig files. Suggestion: - We should use a dict for the option. If the hash value is None, or some variation thereof, dont defaults.conf: config_opts['macros'] = { '%foo': "xyz", "%bar": "123", "%baz": "quux" } my-root.conf: # completely override all defaults config_opts['macros'] = { "%someopt": "someval" } my-root.conf: # only override some config_opts['macros']['%foo'] = "myvalue" config_opts['macros']['%bar'] = None #dont output this in final # .rpmmacros -- Michael > | > | -- my-root.cfg == > | config_opts['macros'] = [ > | config_opts['macros'], > | { > | '%bar' : 2 > | } > | ] > > Overwriting is not possible/easy when we want to retain backward > compatibility. When implemented from scratch, I would have implemented > this option completely as a dictionary (I do not think that order of > macros is for much interest). > > > > > >> + > >> def _build_dir_setup(self): > >> # purge the builddir, if it exists > >> bd_out = '%s%s' % (self.rootdir, self.builddir) > >> @@ -788,8 +801,12 @@ class Root: > >> macrofile_out = '%s%s/.rpmmacros' % (self.rootdir, self.homedir) > >> if not os.path.exists(macrofile_out): > >> rpmmacros = open(macrofile_out, 'w') > >> - self.config['macros'] = self.config['macros'] + "\n%%_rpmlock_path %s/var/lib/rpm/__db.000" % self.basedir > >> - rpmmacros.write(self.config['macros']) > >> + > >> + rpmmacros.write(self._expand_macro_string(self.config['macros'])) > >> + rpmmacros.write(self._expand_macro_string( > >> + {"%_rpmlock_path" : "%s/var/lib/rpm/__db.000" % self.basedir} > >> + )) > > > > This should go into the default config section, rather than hidden > > here. > > I just rewrote this hunk of code. In a later version of my mock > modifications I removed this completely and replaced it by bind-mounting > the chrooted /var/lib/rpm directory into toplevel /var/lib/rpm. > > > > You need to convert the setup functions to use the new format by > > default, as well as any existing config file references in the > > default supplied configs. > > See above for statement regarding backward compatibility; as long as > you do not change the default string-type, old/existing configurations > should still work. > > When defaults would be dictionaries or vectors, overwriting can fail > when people append an string. > > > > > Enrico > -- > Fedora-buildsys-list mailing list > Fedora-buildsys-list at redhat.com > https://www.redhat.com/mailman/listinfo/fedora-buildsys-list From enrico.scholz at informatik.tu-chemnitz.de Mon May 7 18:59:21 2007 From: enrico.scholz at informatik.tu-chemnitz.de (Enrico Scholz) Date: Mon, 07 May 2007 20:59:21 +0200 Subject: [PATCH mock] bind-mount the /var/lib/rpm from the chroot into the root directory In-Reply-To: <20070507173647.GJ29655@humbolt.us.dell.com> (Michael E. Brown's message of "Mon, 7 May 2007 12:36:47 -0500") References: <20070505185559.4854.26101.stgit@kosh.bigo.ensc.de> <20070507173647.GJ29655@humbolt.us.dell.com> Message-ID: <877irkfoja.fsf@kosh.bigo.ensc.de> Michael E Brown writes: >> - rpmmacros.close() > > You just removed the close() on the macros file. This was probably not > intended. No; it was intended. As a C++ developer I trust into (C)Python's reference counting + stack unwinding. Most python developers are doing this too, or why are they not writing | f = open("...") | try: | ... | finally: | f.close() ;) Enrico -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 480 bytes Desc: not available URL: From Michael_E_Brown at dell.com Mon May 7 19:02:38 2007 From: Michael_E_Brown at dell.com (Michael E Brown) Date: Mon, 7 May 2007 14:02:38 -0500 Subject: [PATCH mock] Bind-mount a precreated /dev filesystem instead of using mknod(2) In-Reply-To: <87bqgwfoqd.fsf@kosh.bigo.ensc.de> References: <20070505173019.23031.6623.stgit@kosh.bigo.ensc.de> <20070507173159.GH29655@humbolt.us.dell.com> <87bqgwfoqd.fsf@kosh.bigo.ensc.de> Message-ID: <20070507190237.GK29614@humbolt.us.dell.com> On Mon, May 07, 2007 at 08:55:06PM +0200, Enrico Scholz wrote: > Michael E Brown writes: > > >> | config_opts['dev-template'] = '/usr/share/mock/dev-template' > > > > 1) Need to add default config option for this in > > setup_default_config_opts(). > > I am not sure whether it shall be enabled by default. It requires manual > configuration (/etc/fstab, perhaps /etc/rpm/macros) and mock will not > work out of the box. Definitely disabled by default. Possibly with a note that it requires manual setup. > Then, mounting a dev-fs below /usr/share might be unwanted for some > people. Something below /srv will be better but since not covered by > FHS, it will be impossible to find a default. I think that /usr/lib/mock is pretty acceptable. It is where we drop everything else. -- Michael From Michael_E_Brown at dell.com Mon May 7 19:05:56 2007 From: Michael_E_Brown at dell.com (Michael E Brown) Date: Mon, 7 May 2007 14:05:56 -0500 Subject: [PATCH mock] bind-mount the /var/lib/rpm from the chroot into the root directory In-Reply-To: <877irkfoja.fsf@kosh.bigo.ensc.de> References: <20070505185559.4854.26101.stgit@kosh.bigo.ensc.de> <20070507173647.GJ29655@humbolt.us.dell.com> <877irkfoja.fsf@kosh.bigo.ensc.de> Message-ID: <20070507190556.GL29614@humbolt.us.dell.com> On Mon, May 07, 2007 at 08:59:21PM +0200, Enrico Scholz wrote: > Michael E Brown writes: > > >> - rpmmacros.close() > > > > You just removed the close() on the macros file. This was probably not > > intended. > > No; it was intended. As a C++ developer I trust into (C)Python's reference > counting + stack unwinding. Most python developers are doing this too, or > why are they not writing > > | f = open("...") > | try: > | ... > | finally: > | f.close() Actually... I believe that this is part of the motivation of the new 'with' keyword. Also, when you use the automatic garbage collection, the file might not be closed until much later. I have seen some discussion on this causing problems, especially on alternative python interpreter implementations... eg. Jython. I would prefer that we keep the close(). Thanks, Michael From enrico.scholz at informatik.tu-chemnitz.de Mon May 7 19:50:29 2007 From: enrico.scholz at informatik.tu-chemnitz.de (Enrico Scholz) Date: Mon, 7 May 2007 21:50:29 +0200 Subject: [PATCH koji] Allow untracked packages in the buildroot Message-ID: <200705071952.l47JqZxD025353@kosh.bigo.ensc.de> When using koji to build a local repository, it is often unwanted to import the >4000 packages of Fedora 6/7. Although direct support for this mode is missing, it is easy to add it (e.g. I use a wrapper around mock which generates a new configuration with additional repositories). Unfortunately, koji will refuse to build packages because buildroot contains untracked packages. This patch makes koji ignore such packages. Signed-off-by: Enrico Scholz --- hub/kojihub.py | 7 ++++++- 1 files changed, 6 insertions(+), 1 deletions(-) diff --git a/hub/kojihub.py b/hub/kojihub.py index 2830ad1..a3bbe5f 100644 --- a/hub/kojihub.py +++ b/hub/kojihub.py @@ -5565,7 +5565,12 @@ class BuildRoot(object): VALUES (%(brootid)s,%(rpm_id)s,%(update)s)""" rpm_ids = [] for an_rpm in rpmlist: - rpm_id = get_rpm(an_rpm, strict=True)['id'] + rpm_id = get_rpm(an_rpm, strict=False) + if rpm_id == None: + #ignore unknown packages (e.g. from untracked repositories) + continue + + rpm_id = rpm_id['id'] if update and current.has_key(rpm_id): #ignore duplicate packages for updates continue -- 1.5.0.6 From mikem at redhat.com Mon May 7 21:43:58 2007 From: mikem at redhat.com (Mike McLean) Date: Mon, 07 May 2007 17:43:58 -0400 Subject: [PATCH koji] Allow untracked packages in the buildroot In-Reply-To: <200705071952.l47JqZxD025353@kosh.bigo.ensc.de> References: <200705071952.l47JqZxD025353@kosh.bigo.ensc.de> Message-ID: <463F9D9E.9060101@redhat.com> Enrico Scholz wrote: > When using koji to build a local repository, it is often unwanted to > import the >4000 packages of Fedora 6/7. Although direct support for this > mode is missing, it is easy to add it (e.g. I use a wrapper around mock > which generates a new configuration with additional repositories). > > Unfortunately, koji will refuse to build packages because buildroot > contains untracked packages. This patch makes koji ignore such packages. Such refusal is deliberate and a design goal of Koji. That goal is reproducibility. Note that the other implicit feature (building from a repository that was not generated by koji) is also contrary to this goal. If you're going to the trouble of generating a custom repo, perhaps you should just skip koji and use mock directly. > Signed-off-by: Enrico Scholz > --- > hub/kojihub.py | 7 ++++++- > 1 files changed, 6 insertions(+), 1 deletions(-) > > diff --git a/hub/kojihub.py b/hub/kojihub.py > index 2830ad1..a3bbe5f 100644 > --- a/hub/kojihub.py > +++ b/hub/kojihub.py > @@ -5565,7 +5565,12 @@ class BuildRoot(object): > VALUES (%(brootid)s,%(rpm_id)s,%(update)s)""" > rpm_ids = [] > for an_rpm in rpmlist: > - rpm_id = get_rpm(an_rpm, strict=True)['id'] > + rpm_id = get_rpm(an_rpm, strict=False) > + if rpm_id == None: > + #ignore unknown packages (e.g. from untracked repositories) > + continue > + > + rpm_id = rpm_id['id'] > if update and current.has_key(rpm_id): > #ignore duplicate packages for updates > continue From paul at city-fan.org Tue May 8 10:19:33 2007 From: paul at city-fan.org (Paul Howarth) Date: Tue, 08 May 2007 11:19:33 +0100 Subject: [PATCH mock] Enhancements of the config_opts['macros'] handling In-Reply-To: <20070507185808.GK29655@humbolt.us.dell.com> References: <20070505170638.21239.24364.stgit@kosh.bigo.ensc.de> <20070507170754.GG29655@humbolt.us.dell.com> <87mz0gfqnk.fsf@kosh.bigo.ensc.de> <20070507185808.GK29655@humbolt.us.dell.com> Message-ID: <46404EB5.8090105@city-fan.org> Michael E Brown wrote: > On Mon, May 07, 2007 at 08:13:35PM +0200, Enrico Scholz wrote: >> Michael E Brown writes: >> >>>> + def _expand_macro_string(self, macros): >>>> + res = [] >>>> + if isinstance(macros, dict): >>>> + for (key,v) in macros.items(): >>>> + res.append('%s\t%s' % (key,v)) >>>> + elif isinstance(macros, basestring): >>>> + res = [macros] >>>> + else: >>>> + for v in macros: >>>> + res.append(self._expand_macro_string(v)) >>>> + >>>> + return '\n'.join(res) + '\n' >>> How does this affect the ability to overwrite config from a config file >>> higher in the heirarchy? >> You can write e.g. >> >> | -- defaults.cfg -- >> | config_opts['macros'] = "%foo 1" > > If something like this goes in, I expect that to become the defualt for > everything we ship. In this case: > - setup_default_config_opts() would set up defaults using new format. > - defaults.cfg probably wouldnt have this option > - get rid of the recursive processing. If the option is a string, > return it. If it is not a string, use dict processing model. > > This means: > - everything we ship uses new format > - old users of this option still work but are encouraged to switch > to the new format > > Additionally: > - We need a way to override options in subconfig files. > > Suggestion: > - We should use a dict for the option. If the hash value is None, or > some variation thereof, dont > > defaults.conf: > config_opts['macros'] = { '%foo': "xyz", "%bar": "123", "%baz": "quux" } > > my-root.conf: # completely override all defaults > config_opts['macros'] = { "%someopt": "someval" } > > my-root.conf: # only override some > config_opts['macros']['%foo'] = "myvalue" > config_opts['macros']['%bar'] = None #dont output this in final > # .rpmmacros > > -- > Michael > > > >> | >> | -- my-root.cfg == >> | config_opts['macros'] = [ >> | config_opts['macros'], >> | { >> | '%bar' : 2 >> | } >> | ] >> >> Overwriting is not possible/easy when we want to retain backward >> compatibility. When implemented from scratch, I would have implemented >> this option completely as a dictionary (I do not think that order of >> macros is for much interest). >> >> >>>> + >>>> def _build_dir_setup(self): >>>> # purge the builddir, if it exists >>>> bd_out = '%s%s' % (self.rootdir, self.builddir) >>>> @@ -788,8 +801,12 @@ class Root: >>>> macrofile_out = '%s%s/.rpmmacros' % (self.rootdir, self.homedir) >>>> if not os.path.exists(macrofile_out): >>>> rpmmacros = open(macrofile_out, 'w') >>>> - self.config['macros'] = self.config['macros'] + "\n%%_rpmlock_path %s/var/lib/rpm/__db.000" % self.basedir >>>> - rpmmacros.write(self.config['macros']) >>>> + >>>> + rpmmacros.write(self._expand_macro_string(self.config['macros'])) >>>> + rpmmacros.write(self._expand_macro_string( >>>> + {"%_rpmlock_path" : "%s/var/lib/rpm/__db.000" % self.basedir} >>>> + )) >>> This should go into the default config section, rather than hidden >>> here. >> I just rewrote this hunk of code. In a later version of my mock >> modifications I removed this completely and replaced it by bind-mounting >> the chrooted /var/lib/rpm directory into toplevel /var/lib/rpm. >> >> >>> You need to convert the setup functions to use the new format by >>> default, as well as any existing config file references in the >>> default supplied configs. >> See above for statement regarding backward compatibility; as long as >> you do not change the default string-type, old/existing configurations >> should still work. >> >> When defaults would be dictionaries or vectors, overwriting can fail >> when people append an string. Would it be possible to move creation of the macros file in the chroot to a point after the unpacking of an autocache cache? That would make it possible to use the same root (and cache) with configurations that differ only in macro values, which would save a lot of disk space for builders serving multiple repos (with different vendor/packager etc. macros). Paul. From Michael_E_Brown at dell.com Tue May 8 12:55:51 2007 From: Michael_E_Brown at dell.com (Michael E Brown) Date: Tue, 8 May 2007 07:55:51 -0500 Subject: [PATCH mock] Enhancements of the config_opts['macros'] handling In-Reply-To: <46404EB5.8090105@city-fan.org> References: <20070505170638.21239.24364.stgit@kosh.bigo.ensc.de> <20070507170754.GG29655@humbolt.us.dell.com> <87mz0gfqnk.fsf@kosh.bigo.ensc.de> <20070507185808.GK29655@humbolt.us.dell.com> <46404EB5.8090105@city-fan.org> Message-ID: <20070508125550.GB22723@humbolt.us.dell.com> On Tue, May 08, 2007 at 11:19:33AM +0100, Paul Howarth wrote: > > Would it be possible to move creation of the macros file in the chroot > to a point after the unpacking of an autocache cache? That would make it > possible to use the same root (and cache) with configurations that > differ only in macro values, which would save a lot of disk space for > builders serving multiple repos (with different vendor/packager etc. > macros). Sounds sane, but you forgot to attach your patch. :) -- Michael From jgranado at redhat.com Wed May 9 16:36:15 2007 From: jgranado at redhat.com (Joel Andres Granados) Date: Wed, 09 May 2007 18:36:15 +0200 Subject: mkisofs VolumeID 32 character limitation In-Reply-To: <461E5B55.9020707@redhat.com> References: <461E58FB.70909@redhat.com> <200704121207.57245.jkeating@redhat.com> <461E5B55.9020707@redhat.com> Message-ID: <4641F87F.1080507@redhat.com> Joel Andres Granados wrote: > Jesse Keating wrote: >> On Thursday 12 April 2007 12:06:19 Joel Andres Granados wrote: >> >>> Possible fix: make sure to snip off the label before creating the >>> isoimage. With the patch I'm trying to cut the id to an allowable >>> size. Although I *do* cut the original name and I don't know what the >>> effects are in the installations. I did run pungi and it ended with no >>> errors. >>> Comments: when the was shortened from the config file pungi executed >>> successfully. comments greatly appreciated. >>> >> >> I've been reluctant to fix this part of the code for a while. I'm >> not really sure what a good thing to do here is, truncation can be >> kind of ugly, but it should have no effect upon the install, it'll >> just effect what the desktop shows as the volume should you mount the >> CD. >> > Thats what I thought as well, I'll do an install test with one of the > truncated IDs and see > if it works Well the truncated ID works just fine, but thinking about the whole aesthetics of the situation I think it would be much better to just change the default value of the version parameter in the "pungi.conf" file. The thing is that it has "development" set for the version parameter and this word is quite long. IMO if it defaults to "devel" instead of "development", we would avoid using 6 characters and pungi would work with no problem when it is installed (Currently when pungi is installed from FC7 repo it will fail with the "ID to long" error if no change is made to the conf file. :( ). If users start to scream about the situation too much the the ID should be truncated. Just an idea. Regards Joel > >> Perhaps this is a good enough solution for now, to prevent strange >> failures. I'll apply it soon, thanks. >> > np > >> >> ------------------------------------------------------------------------ >> >> -- >> Fedora-buildsys-list mailing list >> Fedora-buildsys-list at redhat.com >> https://www.redhat.com/mailman/listinfo/fedora-buildsys-list > > -- > Fedora-buildsys-list mailing list > Fedora-buildsys-list at redhat.com > https://www.redhat.com/mailman/listinfo/fedora-buildsys-list From jkeating at redhat.com Wed May 9 16:49:30 2007 From: jkeating at redhat.com (Jesse Keating) Date: Wed, 9 May 2007 09:49:30 -0700 Subject: mkisofs VolumeID 32 character limitation In-Reply-To: <4641F87F.1080507@redhat.com> References: <461E58FB.70909@redhat.com> <461E5B55.9020707@redhat.com> <4641F87F.1080507@redhat.com> Message-ID: <200705090949.30639.jkeating@redhat.com> On Wednesday 09 May 2007 09:36:15 Joel Andres Granados wrote: > Well the truncated ID works just fine, but thinking about the whole > aesthetics of the situation I think it would be much better to just > change the default value of the version parameter in the "pungi.conf" > file. ?The thing is that it has "development" set for the version > parameter and this word is quite long. ? IMO if it defaults to "devel" > instead of "development", we would avoid using 6 characters and pungi > would work with no problem when it is installed (Currently when pungi is > installed from FC7 repo it will fail with the "ID to long" error if no > change is made to the conf file. :( ). > If users start to scream about the situation too much the the ID should > be truncated. ?Just an idea. > Regards Not a bad idea either (: I'll make that change (if I remember it) when I finally get to the final tweaks to pungi for Fedora 7. -- Jesse Keating Release Engineer: Fedora -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From enrico.scholz at informatik.tu-chemnitz.de Fri May 11 16:48:22 2007 From: enrico.scholz at informatik.tu-chemnitz.de (Enrico Scholz) Date: Fri, 11 May 2007 18:48:22 +0200 Subject: [PATCH koji] Set proper permissions (0666) when creating files Message-ID: <20070511164809.9069.112.stgit@kosh.bigo.ensc.de> Currently, os.open() is used without specifying the third argument. Hence, all files will be created with 0777 perms (minus umask). Because only datafiles will be uploaded, the exec-bit should be removed. Signed-off-by: Enrico Scholz --- builder/kojid | 2 +- hub/kojihub.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/builder/kojid b/builder/kojid index 6243edd..9a82c60 100755 --- a/builder/kojid +++ b/builder/kojid @@ -109,7 +109,7 @@ def log_output(path, args, outfile, uploadpath, cwd=None, logerror=0, append=0, flags = os.O_CREAT | os.O_WRONLY if append: flags |= os.O_APPEND - fd = os.open(outfile, flags) + fd = os.open(outfile, flags, 0666) os.dup2(fd, 1) if logerror: os.dup2(fd, 2) diff --git a/hub/kojihub.py b/hub/kojihub.py index a3bbe5f..03cb76f 100644 --- a/hub/kojihub.py +++ b/hub/kojihub.py @@ -3958,7 +3958,7 @@ class RootExports(object): # elif offset == 0: # #first chunk, so file should not exist yet # raise koji.GenericError, "file already exists: %s" % fn - fd = os.open(fn, os.O_RDWR | os.O_CREAT) + fd = os.open(fn, os.O_RDWR | os.O_CREAT, 0666) # log_error("fd=%r" %fd) try: if offset == 0 or (offset == -1 and size == len(contents)): From mikeb at redhat.com Fri May 11 17:25:01 2007 From: mikeb at redhat.com (Mike Bonnet) Date: Fri, 11 May 2007 13:25:01 -0400 Subject: [PATCH koji] Set proper permissions (0666) when creating files In-Reply-To: <20070511164809.9069.112.stgit@kosh.bigo.ensc.de> References: <20070511164809.9069.112.stgit@kosh.bigo.ensc.de> Message-ID: <1178904302.4400.9.camel@liffey.home> On Fri, 2007-05-11 at 18:48 +0200, Enrico Scholz wrote: > Currently, os.open() is used without specifying the third argument. > Hence, all files will be created with 0777 perms (minus umask). > > Because only datafiles will be uploaded, the exec-bit should be removed. Indeed it should. Patch applied, thanks a lot! > Signed-off-by: Enrico Scholz > --- > > builder/kojid | 2 +- > hub/kojihub.py | 2 +- > 2 files changed, 2 insertions(+), 2 deletions(-) > > diff --git a/builder/kojid b/builder/kojid > index 6243edd..9a82c60 100755 > --- a/builder/kojid > +++ b/builder/kojid > @@ -109,7 +109,7 @@ def log_output(path, args, outfile, uploadpath, cwd=None, logerror=0, append=0, > flags = os.O_CREAT | os.O_WRONLY > if append: > flags |= os.O_APPEND > - fd = os.open(outfile, flags) > + fd = os.open(outfile, flags, 0666) > os.dup2(fd, 1) > if logerror: > os.dup2(fd, 2) > diff --git a/hub/kojihub.py b/hub/kojihub.py > index a3bbe5f..03cb76f 100644 > --- a/hub/kojihub.py > +++ b/hub/kojihub.py > @@ -3958,7 +3958,7 @@ class RootExports(object): > # elif offset == 0: > # #first chunk, so file should not exist yet > # raise koji.GenericError, "file already exists: %s" % fn > - fd = os.open(fn, os.O_RDWR | os.O_CREAT) > + fd = os.open(fn, os.O_RDWR | os.O_CREAT, 0666) > # log_error("fd=%r" %fd) > try: > if offset == 0 or (offset == -1 and size == len(contents)): > > -- > Fedora-buildsys-list mailing list > Fedora-buildsys-list at redhat.com > https://www.redhat.com/mailman/listinfo/fedora-buildsys-list From bugs.michael at gmx.net Sat May 12 14:08:03 2007 From: bugs.michael at gmx.net (Michael Schwendt) Date: Sat, 12 May 2007 16:08:03 +0200 Subject: [PATCH] koji import-comps In-Reply-To: <20070424122422.2d9a8d5d.bugs.michael@gmx.net> References: <20070424122422.2d9a8d5d.bugs.michael@gmx.net> Message-ID: <20070512160803.7320117b.bugs.michael@gmx.net> On Tue, 24 Apr 2007 12:24:22 +0200, Michael Schwendt wrote: > > #for import-comps handler (currently disabled) > > #from rhpl.comps import Comps > > I've ported this to FC6 yum.comps to make it work for the buildroot list, > but it's only rudimentary, because I didn't find anything about the > "biarchonly" attribute. The "basearchonly" attribute, on the other hand, > is not evaluated by yum.comps. Plus, I couldn't find any groupreq and > metapkg support in yum.comps either. If the patch in this thread has been noticed and considered insufficient, please say so. If it has not been seen yet: ping From katzj at redhat.com Mon May 14 15:03:00 2007 From: katzj at redhat.com (Jeremy Katz) Date: Mon, 14 May 2007 11:03:00 -0400 Subject: [PATCH] koji import-comps In-Reply-To: <20070424122422.2d9a8d5d.bugs.michael@gmx.net> References: <20070424122422.2d9a8d5d.bugs.michael@gmx.net> Message-ID: <1179154980.12077.44.camel@aglarond.local> On Tue, 2007-04-24 at 12:24 +0200, Michael Schwendt wrote: > > #for import-comps handler (currently disabled) > > #from rhpl.comps import Comps > > I've ported this to FC6 yum.comps to make it work for the buildroot list, > but it's only rudimentary, because I didn't find anything about the > "biarchonly" attribute. The "basearchonly" attribute, on the other hand, > is not evaluated by yum.comps. Plus, I couldn't find any groupreq and > metapkg support in yum.comps either. So, I'm not actually sure what the import-comps command is actually used for[1], but as for some of the other things. biarchonly, basearchonly, groupreq and metapkg are all elements of the old comps format supported by rhpl.comps and not the new comps format supported directly by yum. Jeremy [1] It looks like the idea is to be able to store the comps info directly in the buildsys so that a comps file could then just be spit out. But I'm not 100% sure there. And it raises some questions on how it works with other parts of the process around comps like translations From jkeating at redhat.com Mon May 14 15:13:34 2007 From: jkeating at redhat.com (Jesse Keating) Date: Mon, 14 May 2007 11:13:34 -0400 Subject: [PATCH] koji import-comps In-Reply-To: <1179154980.12077.44.camel@aglarond.local> References: <20070424122422.2d9a8d5d.bugs.michael@gmx.net> <1179154980.12077.44.camel@aglarond.local> Message-ID: <200705141113.35360.jkeating@redhat.com> On Monday 14 May 2007 11:03:00 Jeremy Katz wrote: > [1] It looks like the idea is to be able to store the comps info > directly in the buildsys so that a comps file could then just be spit > out. ?But I'm not 100% sure there. ?And it raises some questions on how > it works with other parts of the process around comps like translations Not quite right. The comps thing is somewhat of a historical thing with how mock populates the default buildroot. We've created the idea of groups within koji, like the 'build' group, that is used to populate the buildroot. Koji could have the ability to 'import' a written out comps file to get the grouping information, however I've just been doing it by hand since the group is fairly small. It has nothing to do with the comps used for composing a distribution. -- Jesse Keating Release Engineer: Fedora -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From bugs.michael at gmx.net Mon May 14 15:52:12 2007 From: bugs.michael at gmx.net (Michael Schwendt) Date: Mon, 14 May 2007 17:52:12 +0200 Subject: [PATCH] koji import-comps In-Reply-To: <200705141113.35360.jkeating@redhat.com> References: <20070424122422.2d9a8d5d.bugs.michael@gmx.net> <1179154980.12077.44.camel@aglarond.local> <200705141113.35360.jkeating@redhat.com> Message-ID: <20070514175212.3d960185.bugs.michael@gmx.net> On Mon, 14 May 2007 11:13:34 -0400, Jesse Keating wrote: > On Monday 14 May 2007 11:03:00 Jeremy Katz wrote: > > [1] It looks like the idea is to be able to store the comps info > > directly in the buildsys so that a comps file could then just be spit > > out. ?But I'm not 100% sure there. ?And it raises some questions on how > > it works with other parts of the process around comps like translations > > Not quite right. The comps thing is somewhat of a historical thing with how > mock populates the default buildroot. We've created the idea of groups > within koji, like the 'build' group, that is used to populate the buildroot. > Koji could have the ability to 'import' a written out comps file to get the > grouping information, however I've just been doing it by hand since the group > is fairly small. It has nothing to do with the comps used for composing a > distribution. Well, I used the patched koji to import the package-set for the minimal buildroot, which was a requirement for getting a first test-build done. From jkeating at redhat.com Mon May 14 15:58:15 2007 From: jkeating at redhat.com (Jesse Keating) Date: Mon, 14 May 2007 11:58:15 -0400 Subject: [PATCH] koji import-comps In-Reply-To: <20070514175212.3d960185.bugs.michael@gmx.net> References: <20070424122422.2d9a8d5d.bugs.michael@gmx.net> <200705141113.35360.jkeating@redhat.com> <20070514175212.3d960185.bugs.michael@gmx.net> Message-ID: <200705141158.16234.jkeating@redhat.com> On Monday 14 May 2007 11:52:12 Michael Schwendt wrote: > Well, I used the patched koji to import the package-set for the minimal > buildroot, which was a requirement for getting a first test-build done. Right, there is a little known way to get by without importing, using 'koji call' to call a specific API call. There is currently not a cli command to create a group, so I have to use a call to call the groupListAdd API call to create the group, then standard cli commands to add things to said group. -- Jesse Keating Release Engineer: Fedora -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From notting at redhat.com Mon May 14 18:28:59 2007 From: notting at redhat.com (Bill Nottingham) Date: Mon, 14 May 2007 14:28:59 -0400 Subject: [PATCH] koji import-comps In-Reply-To: <1179154980.12077.44.camel@aglarond.local> References: <20070424122422.2d9a8d5d.bugs.michael@gmx.net> <1179154980.12077.44.camel@aglarond.local> Message-ID: <20070514182859.GL5267@nostromo.devel.redhat.com> Jeremy Katz (katzj at redhat.com) said: > biarchonly, basearchonly, groupreq and metapkg are all elements of the > old comps format supported by rhpl.comps and not the new comps format > supported directly by yum. groupreq still exists in current comps. Perhaps that should be 'fixed'? Bill From jkeating at redhat.com Tue May 15 18:13:27 2007 From: jkeating at redhat.com (Jesse Keating) Date: Tue, 15 May 2007 14:13:27 -0400 Subject: mkisofs VolumeID 32 character limitation In-Reply-To: <200705090949.30639.jkeating@redhat.com> References: <461E58FB.70909@redhat.com> <4641F87F.1080507@redhat.com> <200705090949.30639.jkeating@redhat.com> Message-ID: <200705151413.28033.jkeating@redhat.com> On Wednesday 09 May 2007 12:49:30 Jesse Keating wrote: > Not a bad idea either (: ?I'll make that change (if I remember it) when I > finally get to the final tweaks to pungi for Fedora 7. I just made this change. I'm holding off on the other patch regarding this until I started doing Fedora 8 development and I revisit this problem. -- Jesse Keating Release Engineer: Fedora -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From jkeating at redhat.com Tue May 15 18:17:01 2007 From: jkeating at redhat.com (Jesse Keating) Date: Tue, 15 May 2007 14:17:01 -0400 Subject: configuration file not found In-Reply-To: <461E5F6F.1040200@redhat.com> References: <461E5F6F.1040200@redhat.com> Message-ID: <200705151417.01528.jkeating@redhat.com> On Thursday 12 April 2007 12:33:51 Joel Andres Granados wrote: > IMO catching the mistake before the traceback and telling the user what > is happening is the right thing to do. > I don't know about the message though. ?Its the best thing that my > little mind could come up with :) > > The diff: Applied. -- Jesse Keating Release Engineer: Fedora -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From jkeating at redhat.com Tue May 15 18:18:32 2007 From: jkeating at redhat.com (Jesse Keating) Date: Tue, 15 May 2007 14:18:32 -0400 Subject: Pungis work directory. In-Reply-To: <461F86F4.5000903@redhat.com> References: <461F86F4.5000903@redhat.com> Message-ID: <200705151418.32588.jkeating@redhat.com> On Friday 13 April 2007 09:34:44 Joel Andres Granados wrote: > I ran pungi once and created a fedora iso. ?When I ran it again with the > same config file, and forgot to erase the previous work directory > files, ?pungi shows a traceback telling me that some file already exists. > I have two proposals for this situation. > 1. ?create a timestamp directory inside the working directory for each > pungi run. ? The problem with this approach is that the use is left with > a bunch of directories that are named after whatever time.time() spit > out (not very pretty). ?But alas it does away with the ugly traceback This needs more thought as I took in a patch that separates out the different pungi tasks to be called separately, so the work dir _could_ exist in some cases. Will think more on this. -- Jesse Keating Release Engineer: Fedora -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From jkeating at redhat.com Tue May 15 18:20:29 2007 From: jkeating at redhat.com (Jesse Keating) Date: Tue, 15 May 2007 14:20:29 -0400 Subject: pungi logger In-Reply-To: <463B3F8E.2000209@redhat.com> References: <463B3F8E.2000209@redhat.com> Message-ID: <200705151420.29361.jkeating@redhat.com> On Friday 04 May 2007 10:13:34 Joel Andres Granados wrote: > Referring to ticket #34 at > https://hosted.fedoraproject.org/projects/pungi/ticket/34 stating that > the logger needed a little work so that it didn't depend on the > gather.py (or at least thats what I understood :) ?I propose either a > new file (pungiLog.py) located in the pypungi directory or a new > function in the "pungi" file that contains the logging stuff. ?The log > services would be started somewhere before the line containing "# > Actually do work." of the "pungi" file. ?The logging root would be > called "pungi" and would be called in each file that logging is needed > with the logging.getlogger("pungi") command. > If "quiet" is specified in the config file the logging will be turned off. > > *Diff for the pungi file:* > 1. Initializes the logger by calling to the new file. > 2. specify quiet value. > 3. ?logging function. > * > Diff for pungi.py file: > *1. use the correct logger. > * > Diff for gather.py file:* > change all the if statements for each logging call. > > Files attached... > Comments appreciated. Thanks for this. I think this is the right direction. However I'm reluctant to make such a change this late in Fedora 7 development. I'll want to look at this once I start doing Fedora 8 changes. -- Jesse Keating Release Engineer: Fedora -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From jkeating at redhat.com Tue May 15 18:22:44 2007 From: jkeating at redhat.com (Jesse Keating) Date: Tue, 15 May 2007 14:22:44 -0400 Subject: verbose command option In-Reply-To: <463B53A9.8040905@redhat.com> References: <463B53A9.8040905@redhat.com> Message-ID: <200705151422.45136.jkeating@redhat.com> On Friday 04 May 2007 11:39:21 Joel Andres Granados wrote: > How about a verbose option "-v"? > On the one hand it is very useful for the impatient user that needs some > sort of immediate feedback. ?on the other hand it could be useful for > other applications that might need to analyze pungis output (like > revisor). ?The idea is to have the option off by defect and when the > user specifies "-v" at the command line, ?all the logging information is > showed. ? With the output I did a very simple progress bar and every > thing seem to work correctly (the progress bar example is not attached, > it works but its not pretty :). > The verbose option contains the logging patch I posted earlier. I like the idea, but I think a verbose mode should both echo to the cli as well as log to the file. -- Jesse Keating Release Engineer: Fedora -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From jkeating at redhat.com Tue May 15 18:24:30 2007 From: jkeating at redhat.com (Jesse Keating) Date: Tue, 15 May 2007 14:24:30 -0400 Subject: Extend verbosity to the scripts and apps that pungi uses In-Reply-To: <463B5CB5.4070908@redhat.com> References: <463B5CB5.4070908@redhat.com> Message-ID: <200705151424.31006.jkeating@redhat.com> On Friday 04 May 2007 12:17:57 Joel Andres Granados wrote: > The idea is to allow, via a command line argument, the redirection of > the output of the applications and scripts that pungi calls to the > stdout of pungi. ?In this way the user sees all that is happening. ? > including the things that output of the stuff that pungi calls. ?I > havent tested this extensively and I'm just wondering what you guys > think of the idea. > diffs attached. Like my other example, the idea would be that a verbose option to pungi would essentially echo what it puts in the log to the CLI at the same time. Would that satisfy this desire? -- Jesse Keating Release Engineer: Fedora -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From jgranado at redhat.com Wed May 16 08:34:44 2007 From: jgranado at redhat.com (Joel Andres Granados) Date: Wed, 16 May 2007 10:34:44 +0200 Subject: Pungis work directory. In-Reply-To: <200705151418.32588.jkeating@redhat.com> References: <461F86F4.5000903@redhat.com> <200705151418.32588.jkeating@redhat.com> Message-ID: <464AC224.4040600@redhat.com> Jesse Keating wrote: > On Friday 13 April 2007 09:34:44 Joel Andres Granados wrote: > >> I ran pungi once and created a fedora iso. When I ran it again with the >> same config file, and forgot to erase the previous work directory >> files, pungi shows a traceback telling me that some file already exists. >> I have two proposals for this situation. >> 1. create a timestamp directory inside the working directory for each >> pungi run. The problem with this approach is that the use is left with >> a bunch of directories that are named after whatever time.time() spit >> out (not very pretty). But alas it does away with the ugly traceback >> > > This needs more thought as I took in a patch that separates out the different > pungi tasks to be called separately, so the work dir _could_ exist in some > cases. Will think more on this. > > > ------------------------------------------------------------------------ > > -- > Fedora-buildsys-list mailing list > Fedora-buildsys-list at redhat.com > https://www.redhat.com/mailman/listinfo/fedora-buildsys-list Yes, depending on the command line options, the working directory is either used or not. IMHO I think the second solution (Tell the user what is happening) is the best way to go. In this way if the directory is needed and is already there the user will receive a message, but if the directory is not needed pungi will be silent about the situation. I'll revisit this patch today and see if it behaves like expected. I'll post any results on the list. Regards. From jgranado at redhat.com Wed May 16 17:22:52 2007 From: jgranado at redhat.com (Joel Andres Granados) Date: Wed, 16 May 2007 19:22:52 +0200 Subject: Pungis work directory. In-Reply-To: <464AC224.4040600@redhat.com> References: <461F86F4.5000903@redhat.com> <200705151418.32588.jkeating@redhat.com> <464AC224.4040600@redhat.com> Message-ID: <464B3DEC.1090200@redhat.com> Joel Andres Granados wrote: > Jesse Keating wrote: >> On Friday 13 April 2007 09:34:44 Joel Andres Granados wrote: >> >>> I ran pungi once and created a fedora iso. When I ran it again with >>> the >>> same config file, and forgot to erase the previous work directory >>> files, pungi shows a traceback telling me that some file already >>> exists. >>> I have two proposals for this situation. >>> 1. create a timestamp directory inside the working directory for each >>> pungi run. The problem with this approach is that the use is left >>> with >>> a bunch of directories that are named after whatever time.time() spit >>> out (not very pretty). But alas it does away with the ugly traceback >>> >> >> This needs more thought as I took in a patch that separates out the >> different pungi tasks to be called separately, so the work dir >> _could_ exist in some cases. Will think more on this. >> >> >> ------------------------------------------------------------------------ >> >> -- >> Fedora-buildsys-list mailing list >> Fedora-buildsys-list at redhat.com >> https://www.redhat.com/mailman/listinfo/fedora-buildsys-list > Yes, depending on the command line options, the working directory is > either used or not. IMHO I think the second solution (Tell the user > what is happening) is the best way to go. In this way if the > directory is needed and is already there the user will receive a > message, but if the directory is not needed pungi will be silent about > the situation. > I'll revisit this patch today and see if it behaves like expected. > I'll post any results on the list. > > Regards. > > -- > Fedora-buildsys-list mailing list > Fedora-buildsys-list at redhat.com > https://www.redhat.com/mailman/listinfo/fedora-buildsys-list Looked into this a little more and came up with something completely different than my previous post. :) I tested to see what would happen with each if the options (-G -B -P -S -I) and I found out that on G,B and I tracedback, in the other options it didn't. When I tested again (I don't know why I tested two times), I found that the tracebacks where different in G, B and I. So basically the trace backs depend on the state of the directory. If the directory is in one state the traceback happens in one line, if it is in another state it has the possibility of occurring in another. So I thought that the best way to fix it would be to check for the existence of the directory in every directory creation, so I looked for every occurrence of "os.makedirs": /usr/lib/python2.5/site-packages/pypungi/gather.py:64: os.makedirs(logdir) /usr/lib/python2.5/site-packages/pypungi/gather.py:275: os.makedirs(pkgdir) /usr/lib/python2.5/site-packages/pypungi/gather.py:342: os.makedirs(pkgdir) /usr/lib/python2.5/site-packages/pypungi/pungi.py:45: os.makedirs(self.workdir) /usr/lib/python2.5/site-packages/pypungi/pungi.py:165: os.makedirs(docsdir) /usr/lib/python2.5/site-packages/pypungi/pungi.py:263: os.makedirs("%s-disc%d/SRPMS" % (timber.dist_dir, i)) /usr/lib/python2.5/site-packages/pypungi/pungi.py:293: os.makedirs('%s-disc1' % self.topdir) /usr/lib/python2.5/site-packages/pypungi/pungi.py:320: os.makedirs(self.isodir) /usr/bin/pungi:90: os.makedirs(destdir) /usr/bin/pungi:99: os.makedirs(cachedir) I just don't like it... IMO its just not right to check for the existence of the directory in every line!!! So I take what I said in my last post back, and am sticking with the "create a new root directory for each execution" solution. I changed it a little. Instead of using time.time(), I used time.localtime()'s elements. So now the directory would be something like "/srv/pungi/Fedora/2007-5-16-1430/..." IMO it looks much better than what time.time() spits out. The diff is attached. Regards -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: pungi-diff URL: From roland at redhat.com Tue May 22 00:13:14 2007 From: roland at redhat.com (Roland McGrath) Date: Mon, 21 May 2007 17:13:14 -0700 (PDT) Subject: createrepo in packages/n/v/r dir? Message-ID: <20070522001314.ABF281F850F@magilla.localdomain> It would be nice if koji ran createrepo in .../packages/name/version/release when a build finishes, and the same in scratch/user/task dirs. It should be quick and cheap since it's just a few rpms (I'm suggesting one run at the top covering all arch's). Especially for packages with many subpackages, it would be real handy to be able to try a build using a one-off .repo file pointed at http://koji.fedoraproject.org/packages/foo/v/r/ with yum --enablerepo, where one now downloads all the rpms and uses rpm by hand. Thanks, Roland From jgranado at redhat.com Wed May 23 15:22:45 2007 From: jgranado at redhat.com (Joel Andres Granados) Date: Wed, 23 May 2007 17:22:45 +0200 Subject: [Patch][Pungi] log file formating Message-ID: <46545C45.9040104@redhat.com> Hi list: I have noticed that the resulting log file contains some strange looking output. Mainly the output of createrepo and mksquashfs from the buildinstall call. AFAIK the problem is caused because these two applications use the '\r' in there outputs. When using the universal_newlines option the log file improves a little bit. The diff is attached regards. -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: logformat-diff URL: From jkeating at redhat.com Wed May 23 15:28:56 2007 From: jkeating at redhat.com (Jesse Keating) Date: Wed, 23 May 2007 11:28:56 -0400 Subject: [Patch][Pungi] log file formating In-Reply-To: <46545C45.9040104@redhat.com> References: <46545C45.9040104@redhat.com> Message-ID: <200705231128.56410.jkeating@redhat.com> On Wednesday 23 May 2007 11:22:45 Joel Andres Granados wrote: > I have noticed that the resulting log file contains some strange looking > output. ?Mainly the output of createrepo and mksquashfs from the > buildinstall call. ?AFAIK the problem is caused because these two > applications use the '\r' in there outputs. When using the > universal_newlines option the log file improves a little bit. > The diff is attached Thanks. I'm going to rework some of the logging post f7 anyway, so I'll include this too. -- Jesse Keating Release Engineer: Fedora -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From jgranado at redhat.com Wed May 23 16:07:27 2007 From: jgranado at redhat.com (Joel Andres Granados) Date: Wed, 23 May 2007 18:07:27 +0200 Subject: [Patch][Pungi] log file formating In-Reply-To: <200705231128.56410.jkeating@redhat.com> References: <46545C45.9040104@redhat.com> <200705231128.56410.jkeating@redhat.com> Message-ID: <465466BF.8010500@redhat.com> Jesse Keating wrote: > On Wednesday 23 May 2007 11:22:45 Joel Andres Granados wrote: > >> I have noticed that the resulting log file contains some strange looking >> output. Mainly the output of createrepo and mksquashfs from the >> buildinstall call. AFAIK the problem is caused because these two >> applications use the '\r' in there outputs. When using the >> universal_newlines option the log file improves a little bit. >> The diff is attached >> > > Thanks. I'm going to rework some of the logging post f7 anyway, so I'll > include this too. > > > ------------------------------------------------------------------------ > > -- > Fedora-buildsys-list mailing list > Fedora-buildsys-list at redhat.com > https://www.redhat.com/mailman/listinfo/fedora-buildsys-list FYI. I believe this to be temporary fix for the ugliness in the log file. I just don't see a reason to have the progress information expressed in individual lines [1] except for maybe the chance of seeing exactly in what moment of the createrepo or mksquashfs processes failed. Additionally if pungi where to redirect this output to its stdout it would look rather ugly in the CL ( but for know it works for the log file since there is no verbose option). IMO there can be *MAYBE* (Im still very touchy feely here) two solutions: 1. Do away with the createrepo and mksquashfs outputs (createrepo has a quiet options, mksquashfs has a "no progress option" but buildinstall would have to be modified to use this option). In the end the log file and the stdout can be the same without any ugliness (I think :) 2. Create a special object for the file log handler that simple ignores the messages that come from createrepo and mksquashfs and logs everything else. In this way pungis stdout can still have the createrepo and mksquashfs outputs while the log ignores them altogether. Not sure :(. Any feedback is appreciated!!! Regards. [1] The output with patch for the log file: 1/508 - Fedora/libFS-1.0.0-3.1.x86_64.rpm 2/508 - Fedora/kernel-debug-2.6.21-1.3175.fc7.x86_64.rpm 3/508 - Fedora/perl-devel-5.8.8-18.fc7.x86_64.rpm 4/508 - Fedora/avahi-glib-0.6.17-1.fc7.i386.rpm 5/508 - Fedora/rpm-4.4.2-46.fc7.x86_64.rpm 6/508 - Fedora/shared-mime-info-0.20-2.fc7.x86_64.rpm 7/508 - Fedora/grub-0.97-13.x86_64.rpm 8/508 - Fedora/dmraid-1.0.0.rc14-2.fc7.x86_64.rpm 9/508 - Fedora/libdhcp6client-0.10-42.fc7.x86_64.rpm From jkeating at redhat.com Wed May 23 16:22:34 2007 From: jkeating at redhat.com (Jesse Keating) Date: Wed, 23 May 2007 12:22:34 -0400 Subject: [Patch][Pungi] log file formating In-Reply-To: <465466BF.8010500@redhat.com> References: <46545C45.9040104@redhat.com> <200705231128.56410.jkeating@redhat.com> <465466BF.8010500@redhat.com> Message-ID: <200705231222.37761.jkeating@redhat.com> On Wednesday 23 May 2007 12:07:27 Joel Andres Granados wrote: > FYI. ?I believe this to be temporary fix for the ugliness in the log > file. ?I just don't see a reason to have the progress information > expressed in individual lines [1] except for maybe the chance of seeing > exactly in what moment of the createrepo or mksquashfs processes > failed. ?Additionally if pungi where to redirect this output to its > stdout it would look rather ugly in the CL ( but for know it works for > the log file since there is no verbose option). > IMO ?there can be *MAYBE* (Im still very touchy feely here) two solutions: > 1. Do away with the createrepo and mksquashfs outputs (createrepo has a > quiet options, mksquashfs has a "no progress option" but buildinstall > would have to be modified to use this option). ?In the end the log file > and the stdout can be the same without any ugliness (I think :) > 2. Create a special object for the file log handler that simple ignores > the messages that come from createrepo and mksquashfs and logs > everything else. ?In this way pungis stdout can still have the > createrepo and mksquashfs outputs while the log ignores them altogether. > > Not sure :(. ?Any feedback is appreciated!!! I think it is important to gather all the output from things we call and put them into the log. If createrepo barfs on a package I want to know where it was or things like that. Perhaps we just need to fix createrepo and mksquashfs to handle their output better in situations? -- Jesse Keating Release Engineer: Fedora -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From jgranado at redhat.com Wed May 30 16:18:29 2007 From: jgranado at redhat.com (Joel Andres Granados) Date: Wed, 30 May 2007 18:18:29 +0200 Subject: pungi fails in the createiso stage Message-ID: <465DA3D5.2070107@redhat.com> Hi list: Been working on pungi quite a bit and while I was testing some stuff it showed some strange behaviour. In my box it stops in the createiso stage when `mkisofs` is called. The error messages states that there is no isolinux directory ( and when I go look for it where its suppose to be... its not there :) To make sure that it was not what I had done I downloaded a new tar file and installed it in my box. To my surprise it showed the same behavior. I looked at pungi and the anaconda-runtime scripts which I suspect are the cause of all this. To get a better look at things I stopped pungi just before executing the buildinstall command and executed it with the --debug option to see what came out. A bunch of messages popped out ( I will list just a small portion): /.../upd-instroot: line 989: /.../fixmtime.py: No such file or directory /.../upd-instroot: line 989: /.../fixmtime.py: No such file or directory cat: /.../lang-table*: No such file or directory /.../upd-instroot: line 1011: cd: /.../locale: No such file or directory These where the first ones. There are more but I think the root of the problem is here. I looked in the anaconda-runtime scripts for the bug with no success. Finally I changed pungies yum repository to point to a repository that I had lying around for some time (the repository had anaconda-11.2.0.57-1.x86_64.rpm and anaconda-runtime-11.2.0.57-1.x86_64.rpm they were old versions ). It worked like a charm. FYI, the repository that I was using for my previous tests was "mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=rawhide&arch=x86_64" So I diffed the upd-insroot from the 11.2.0.57-1.x86_64 version with the 11.2.0.66-1.x86_64 one and IMHO found no change that would give the behavior that I'm experiencing. Anybody else in this situation??? Comments greatly appreciated Regards -- Joel Andres Granados From jkeating at redhat.com Wed May 30 16:21:29 2007 From: jkeating at redhat.com (Jesse Keating) Date: Wed, 30 May 2007 12:21:29 -0400 Subject: pungi fails in the createiso stage In-Reply-To: <465DA3D5.2070107@redhat.com> References: <465DA3D5.2070107@redhat.com> Message-ID: <200705301221.29896.jkeating@redhat.com> On Wednesday 30 May 2007 12:18:29 Joel Andres Granados wrote: > ?FYI, the repository that I was using for my previous tests was > "mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=rawhide&arch=x >86_64" > > So I diffed the upd-insroot from the 11.2.0.57-1.x86_64 version with the > 11.2.0.66-1.x86_64 one and IMHO found no change that would give the It's probably something else in the package stack. Some other earlier part of buildinstall is most likely failing to produce the expected output and upd-instroot is just noticing that. Rawhide now has Fedora 8 content, all the post-f7 builds so it is quite unsurprising that things failed. -- Jesse Keating Release Engineer: Fedora -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From jgranado at redhat.com Thu May 31 13:38:12 2007 From: jgranado at redhat.com (Joel Andres Granados) Date: Thu, 31 May 2007 15:38:12 +0200 Subject: [PATCH][PUNGI] name of minimal manifest is inconsistent Message-ID: <465ECFC4.5070408@redhat.com> Hi list: The minimal manifest in the config file appears from pungi-0.3.5-1.fc7 with a hyphen "minimal-manifest". The file itself appears with a period "/etc/pungi/minimal.manifest". Just change the config file. or the name of the file :) Regards. --- /etc/pungi/pungi.conf 2007-05-17 15:14:27.000000000 +0200 +++ /etc/pungi/pungi.conf.rpmnew 2007-05-23 17:17:49.000000000 +0200 @@ -8,7 +8,7 @@ iso_basename = F ; The first part of the iso file name bugurl = http://bugzilla.redhat.com ; Used for betanag comps = /etc/pungi/comps-f7.xml ; Used to define package groupings and default installs -manifest = /etc/pungi/minimal.manifest ; Used to determine what to bring in. Supports Kickstart syntax +manifest = /etc/pungi/minimal-manifest ; Used to determine what to bring in. Supports Kickstart syntax yumconf = /etc/pungi/yum.conf.f7.x86_64 ; Used to determine where to gather packages from destdir = /srv/pungi/Fedora ; Top level compose directory, must be clean cachedir = /srv/pungi/cache ; Cache used for repeat runs -- Joel Andres Granados From jgranado at redhat.com Thu May 31 14:25:48 2007 From: jgranado at redhat.com (Joel Andres Granados) Date: Thu, 31 May 2007 16:25:48 +0200 Subject: [PATCH][PUNGI] name of minimal manifest is inconsistent In-Reply-To: <465ECFC4.5070408@redhat.com> References: <465ECFC4.5070408@redhat.com> Message-ID: <465EDAEC.2050105@redhat.com> Joel Andres Granados wrote: > Hi list: > > The minimal manifest in the config file appears from pungi-0.3.5-1.fc7 > with a hyphen "minimal-manifest". The file itself appears with a > period "/etc/pungi/minimal.manifest". Just change the config file. or > the name of the file :) > > Regards. > > --- /etc/pungi/pungi.conf 2007-05-17 15:14:27.000000000 +0200 > +++ /etc/pungi/pungi.conf.rpmnew 2007-05-23 17:17:49.000000000 > +0200 > @@ -8,7 +8,7 @@ > iso_basename = F ; The first part of the iso file name > bugurl = http://bugzilla.redhat.com ; Used for betanag > comps = /etc/pungi/comps-f7.xml ; Used to define package groupings and > default installs > -manifest = /etc/pungi/minimal.manifest ; Used to determine what to > bring in. Supports Kickstart syntax > +manifest = /etc/pungi/minimal-manifest ; Used to determine what to > bring in. Supports Kickstart syntax > yumconf = /etc/pungi/yum.conf.f7.x86_64 ; Used to determine where to > gather packages from > destdir = /srv/pungi/Fedora ; Top level compose directory, must be clean > cachedir = /srv/pungi/cache ; Cache used for repeat runs > Sorry go the diff backwards :( FYI, this is relevant only for people who use /etc/pungi/pungi.conf as their configuration file --- pungi.conf.rpmnew 2007-05-23 17:17:49.000000000 +0200 +++ pungi.conf 2007-05-17 15:14:27.000000000 +0200 @@ -8,7 +8,7 @@ iso_basename = F ; The first part of the iso file name bugurl = http://bugzilla.redhat.com ; Used for betanag comps = /etc/pungi/comps-f7.xml ; Used to define package groupings and default installs -manifest = /etc/pungi/minimal-manifest ; Used to determine what to bring in. Supports Kickstart syntax +manifest = /etc/pungi/minimal.manifest ; Used to determine what to bring in. Supports Kickstart syntax yumconf = /etc/pungi/yum.conf.f7.x86_64 ; Used to determine where to gather packages from destdir = /srv/pungi/Fedora ; Top level compose directory, must be clean cachedir = /srv/pungi/cache ; Cache used for repeat runs regards -- Joel Andres Granados From jgranado at redhat.com Thu May 31 14:39:05 2007 From: jgranado at redhat.com (Joel Andres Granados) Date: Thu, 31 May 2007 16:39:05 +0200 Subject: pungi fails in the createiso stage In-Reply-To: <200705301221.29896.jkeating@redhat.com> References: <465DA3D5.2070107@redhat.com> <200705301221.29896.jkeating@redhat.com> Message-ID: <465EDE09.6010801@redhat.com> > It's probably something else in the package stack. Some other earlier part of > buildinstall is most likely failing to produce the expected output and > upd-instroot is just noticing that. Rawhide now has Fedora 8 content, all > the post-f7 builds so it is quite unsurprising that things failed. > I ran a yum update, everything seems to be back to normal :) regards. -- Joel Andres Granados