From bkearney at redhat.com Tue Sep 2 12:48:36 2008 From: bkearney at redhat.com (Bryan Kearney) Date: Tue, 02 Sep 2008 08:48:36 -0400 Subject: [Thincrust-devel] ec2-creator merge into appliance-tools In-Reply-To: <48B831EB.20808@redhat.com> References: <48B4154B.6070701@redhat.com> <48B431B3.5080302@redhat.com> <26AF7990-D175-4610-BAD1-4D4877E0C7B0@redhat.com> <48B4B220.7010000@redhat.com> <48B56171.80005@redhat.com> <48B56836.6000505@redhat.com> <48B5683B.6080903@redhat.com> <48B57B9A.1090808@redhat.com> <48B82165.1050307@redhat.com> <48B831EB.20808@redhat.com> Message-ID: <48BD3624.6070404@redhat.com> Joey Boggs wrote: > Bryan Kearney wrote: >> Nice work joey. Couple of comments: >> >> - Can you add a -n attribute so I can name it what I want? > Yep can add that in easily Great! >> - is there any way to autodetect the inputtype based on the imagefile? > I'm thinking there is based on using the output of "file $filename" I'm > playing around with glade/pygtk and perhaps adding in a options simple > gui interface in shouldn't be too hard. Dont worry about GUI yet. Lets focus on more commmand line features and the whitelisting. > >> - There is a slight type in the hlp for inputtype. >> > I'll clean that up, the manualdir feature was going to be for someone > that had a directory filled with their custom / contents and just wants > to bundle it into EC2 format, should I still pursue that route? How much extra effort is this? It it is not much, it sounds like a good idea. From jboggs at redhat.com Tue Sep 2 13:10:42 2008 From: jboggs at redhat.com (Joey Boggs) Date: Tue, 02 Sep 2008 09:10:42 -0400 Subject: [Thincrust-devel] ec2-creator merge into appliance-tools In-Reply-To: <48BD3624.6070404@redhat.com> References: <48B4154B.6070701@redhat.com> <48B431B3.5080302@redhat.com> <26AF7990-D175-4610-BAD1-4D4877E0C7B0@redhat.com> <48B4B220.7010000@redhat.com> <48B56171.80005@redhat.com> <48B56836.6000505@redhat.com> <48B5683B.6080903@redhat.com> <48B57B9A.1090808@redhat.com> <48B82165.1050307@redhat.com> <48B831EB.20808@redhat.com> <48BD3624.6070404@redhat.com> Message-ID: <48BD3B52.4090607@redhat.com> Bryan Kearney wrote: > Joey Boggs wrote: >> Bryan Kearney wrote: >>> Nice work joey. Couple of comments: >>> >>> - Can you add a -n attribute so I can name it what I want? >> Yep can add that in easily > > Great! I'll work on that today > >>> - is there any way to autodetect the inputtype based on the imagefile? >> I'm thinking there is based on using the output of "file $filename" >> I'm playing around with glade/pygtk and perhaps adding in a options >> simple gui interface in shouldn't be too hard. > > Dont worry about GUI yet. Lets focus on more commmand line features > and the whitelisting. After spending so much time working on the blacklist/whitelist items, it's gonna be much simpler just to put the list in a file. Based on comments from dlutterkort, the interpreter would most likely need to be written C, as well as passing those directory variables around can be trouble. There are a few C interpreter/compilers framework types out that that can work in our case but it just alot of overhead we would need to add to the image, as well as providing the source that wouldn't be able to compile correctly on gcc etc... The best and most manageable route is to put the list inside of $nameAppliance rpm in a predetermined filename then just run: /path/to/minimizer $filename The file containing the listing can still contain drop/keep commands. Its much cleaner, simpler and easier to keep track of should the list get very granular, and without filling up the kickstart file. > >> >>> - There is a slight type in the hlp for inputtype. >>> >> I'll clean that up, the manualdir feature was going to be for someone >> that had a directory filled with their custom / contents and just >> wants to bundle it into EC2 format, should I still pursue that route? > > How much extra effort is this? It it is not much, it sounds like a > good idea. Very little work, practically copy / paste > > _______________________________________________ > Thincrust-devel mailing list > Thincrust-devel at redhat.com > https://www.redhat.com/mailman/listinfo/thincrust-devel From bkearney at redhat.com Tue Sep 2 14:13:19 2008 From: bkearney at redhat.com (Bryan Kearney) Date: Tue, 02 Sep 2008 10:13:19 -0400 Subject: [Thincrust-devel] ec2-creator merge into appliance-tools In-Reply-To: <48BD3B52.4090607@redhat.com> References: <48B4154B.6070701@redhat.com> <48B431B3.5080302@redhat.com> <26AF7990-D175-4610-BAD1-4D4877E0C7B0@redhat.com> <48B4B220.7010000@redhat.com> <48B56171.80005@redhat.com> <48B56836.6000505@redhat.com> <48B5683B.6080903@redhat.com> <48B57B9A.1090808@redhat.com> <48B82165.1050307@redhat.com> <48B831EB.20808@redhat.com> <48BD3624.6070404@redhat.com> <48BD3B52.4090607@redhat.com> Message-ID: <48BD49FF.4010800@redhat.com> Joey Boggs wrote: > Bryan Kearney wrote: >> Joey Boggs wrote: >>> Bryan Kearney wrote: >>>> Nice work joey. Couple of comments: >>>> >>>> - Can you add a -n attribute so I can name it what I want? >>> Yep can add that in easily >> >> Great! > I'll work on that today >> >>>> - is there any way to autodetect the inputtype based on the imagefile? >>> I'm thinking there is based on using the output of "file $filename" >>> I'm playing around with glade/pygtk and perhaps adding in a options >>> simple gui interface in shouldn't be too hard. >> >> Dont worry about GUI yet. Lets focus on more commmand line features >> and the whitelisting. > After spending so much time working on the blacklist/whitelist items, > it's gonna be much simpler just to put the list in a file. Based on > comments from dlutterkort, the interpreter would most likely need to be > written C, as well as passing those directory variables around can be > trouble. There are a few C interpreter/compilers framework types out > that that can work in our case but it just alot of overhead we would > need to add to the image, as well as providing the source that wouldn't > be able to compile correctly on gcc etc... The best and most manageable > route is to put the list inside of $nameAppliance rpm in a predetermined > filename then just run: /path/to/minimizer $filename The file > containing the listing can still contain drop/keep commands. Its much > cleaner, simpler and easier to keep track of should the list get very > granular, and without filling up the kickstart file. What exactly is the issue? I looked at a simple python program (since we require python to be there) and it seems like the contents of the post section are written to to a file, and that file is passed in as sys.argv[1]. So.. a simple python program could be used to open it, read it, and apply the necessary rules. -- bk From jboggs at redhat.com Tue Sep 2 14:55:15 2008 From: jboggs at redhat.com (Joey Boggs) Date: Tue, 02 Sep 2008 10:55:15 -0400 Subject: [Thincrust-devel] ec2-creator merge into appliance-tools In-Reply-To: <48BD49FF.4010800@redhat.com> References: <48B4154B.6070701@redhat.com> <48B431B3.5080302@redhat.com> <26AF7990-D175-4610-BAD1-4D4877E0C7B0@redhat.com> <48B4B220.7010000@redhat.com> <48B56171.80005@redhat.com> <48B56836.6000505@redhat.com> <48B5683B.6080903@redhat.com> <48B57B9A.1090808@redhat.com> <48B82165.1050307@redhat.com> <48B831EB.20808@redhat.com> <48BD3624.6070404@redhat.com> <48BD3B52.4090607@redhat.com> <48BD49FF.4010800@redhat.com> Message-ID: <48BD53D3.60903@redhat.com> Bryan Kearney wrote: > Joey Boggs wrote: >> Bryan Kearney wrote: >>> Joey Boggs wrote: >>>> Bryan Kearney wrote: >>>>> Nice work joey. Couple of comments: >>>>> >>>>> - Can you add a -n attribute so I can name it what I want? >>>> Yep can add that in easily >>> >>> Great! >> I'll work on that today >>> >>>>> - is there any way to autodetect the inputtype based on the >>>>> imagefile? >>>> I'm thinking there is based on using the output of "file >>>> $filename" I'm playing around with glade/pygtk and perhaps adding >>>> in a options simple gui interface in shouldn't be too hard. >>> >>> Dont worry about GUI yet. Lets focus on more commmand line features >>> and the whitelisting. >> After spending so much time working on the blacklist/whitelist items, >> it's gonna be much simpler just to put the list in a file. Based on >> comments from dlutterkort, the interpreter would most likely need to >> be written C, as well as passing those directory variables around can >> be trouble. There are a few C interpreter/compilers framework types >> out that that can work in our case but it just alot of overhead we >> would need to add to the image, as well as providing the source that >> wouldn't be able to compile correctly on gcc etc... The best and most >> manageable route is to put the list inside of $nameAppliance rpm in a >> predetermined filename then just run: /path/to/minimizer $filename >> The file containing the listing can still contain drop/keep >> commands. Its much cleaner, simpler and easier to keep track of >> should the list get very granular, and without filling up the >> kickstart file. > > What exactly is the issue? I looked at a simple python program (since > we require python to be there) and it seems like the contents of the > post section are written to to a file, and that file is passed in as > sys.argv[1]. So.. a simple python program could be used to open it, > read it, and apply the necessary rules. > > -- bk It's the way the program is being called. If you specify an interpreter it's just running it like a bash script being called. All of the commands should understood inside the script by the shell/interpreter. If you specify /usr/bin/python or some other variant for an interpreter it has to be a real interpreter, since the rest of the file contains DROP commands. It's effectively just copy/pasting the contents on a console and DROP is not a command. %post --interpreter=/myinterpreter DROP /dir1 DROP /dir2 At the above point ^^ the DROP commands are not sys arguments they are commands/keywords > > _______________________________________________ > Thincrust-devel mailing list > Thincrust-devel at redhat.com > https://www.redhat.com/mailman/listinfo/thincrust-devel From dhuff at redhat.com Tue Sep 2 17:49:42 2008 From: dhuff at redhat.com (David Huff) Date: Tue, 02 Sep 2008 13:49:42 -0400 Subject: [Thincrust-devel] appliance-tools now in Raw Hide Message-ID: <48BD7CB6.2040606@redhat.com> The first cut of appliance-tools, appliance-tools-003-3.fc10.noarch.rpm is now in Raw Hide and can be tested on a rawhide box. This first cut has a minimal feature set however it can be used for creating appliances from the command line. For instruction please refer to the following site: - http://thincrust.net/tooling.html Patches can be submitted to this mailing list: - thincrust-devel at redhat.com Bugs can reported in Bugzilla and know issues are reported here: - https://bugzilla.redhat.com/buglist.cgi?component=appliance-tools&product=Fedora -D From dhuff at redhat.com Tue Sep 2 18:27:37 2008 From: dhuff at redhat.com (David Huff) Date: Tue, 02 Sep 2008 14:27:37 -0400 Subject: [Thincrust-devel] appliance-tools now in Raw Hide In-Reply-To: <48BD7CB6.2040606@redhat.com> References: <48BD7CB6.2040606@redhat.com> Message-ID: <48BD8599.9080603@redhat.com> David Huff wrote: > The first cut of appliance-tools, > appliance-tools-003-3.fc10.noarch.rpm is now in Raw Hide and can be > tested on a rawhide box. This first cut has a minimal feature set > however it can be used for creating appliances from the command line. > For instruction please refer to the following site: > > - http://thincrust.net/tooling.html > > Patches can be submitted to this mailing list: > > - thincrust-devel at redhat.com Also might be helpful to note the the source code is available here: http://git.et.redhat.com/?p=act.git;a=summary > > Bugs can reported in Bugzilla and know issues are reported here: > > - > https://bugzilla.redhat.com/buglist.cgi?component=appliance-tools&product=Fedora > > > > -D > > _______________________________________________ > Thincrust-devel mailing list > Thincrust-devel at redhat.com > https://www.redhat.com/mailman/listinfo/thincrust-devel From bkearney at redhat.com Wed Sep 3 12:12:32 2008 From: bkearney at redhat.com (Bryan Kearney) Date: Wed, 03 Sep 2008 08:12:32 -0400 Subject: [Thincrust-devel] appliance-tools now in Raw Hide In-Reply-To: <48BD8599.9080603@redhat.com> References: <48BD7CB6.2040606@redhat.com> <48BD8599.9080603@redhat.com> Message-ID: <48BE7F30.1000003@redhat.com> David Huff wrote: > David Huff wrote: >> The first cut of appliance-tools, >> appliance-tools-003-3.fc10.noarch.rpm is now in Raw Hide and can be >> tested on a rawhide box. This first cut has a minimal feature set >> however it can be used for creating appliances from the command line. >> For instruction please refer to the following site: >> >> - http://thincrust.net/tooling.html >> >> Patches can be submitted to this mailing list: >> >> - thincrust-devel at redhat.com > > Also might be helpful to note the the source code is available here: > > http://git.et.redhat.com/?p=act.git;a=summary Do we need to aos repo any longer? - bk From dhuff at redhat.com Wed Sep 3 14:16:53 2008 From: dhuff at redhat.com (David Huff) Date: Wed, 03 Sep 2008 10:16:53 -0400 Subject: [Thincrust-devel] appliance-tools now in Raw Hide In-Reply-To: <48BE7F30.1000003@redhat.com> References: <48BD7CB6.2040606@redhat.com> <48BD8599.9080603@redhat.com> <48BE7F30.1000003@redhat.com> Message-ID: <48BE9C55.4060200@redhat.com> Bryan Kearney wrote: > David Huff wrote: >> David Huff wrote: >>> The first cut of appliance-tools, >>> appliance-tools-003-3.fc10.noarch.rpm is now in Raw Hide and can be >>> tested on a rawhide box. This first cut has a minimal feature set >>> however it can be used for creating appliances from the command line. >>> For instruction please refer to the following site: >>> >>> - http://thincrust.net/tooling.html >>> >>> Patches can be submitted to this mailing list: >>> >>> - thincrust-devel at redhat.com >> >> Also might be helpful to note the the source code is available here: >> >> http://git.et.redhat.com/?p=act.git;a=summary > > Do we need to aos repo any longer? This is an interesting question, on that I was thinking about yesterday. Reason being that Jeremy just pulled all the kickstart files from livecd-tools. They now only ship one minimal ks example and mark it as docs. I think we will follow suit however may need to keep it aos repo and move files there until we get everything in the official spin kickstart repo. I am also interested to see this new package that is susposto be created that just has the kickstarts in them. -D From jboggs at redhat.com Wed Sep 3 15:28:41 2008 From: jboggs at redhat.com (Joey Boggs) Date: Wed, 03 Sep 2008 11:28:41 -0400 Subject: [Thincrust-devel] appliance-tools now in Raw Hide In-Reply-To: <48BE9C55.4060200@redhat.com> References: <48BD7CB6.2040606@redhat.com> <48BD8599.9080603@redhat.com> <48BE7F30.1000003@redhat.com> <48BE9C55.4060200@redhat.com> Message-ID: <48BEAD29.9090502@redhat.com> We should keep the yum repo just for development access other than the just the git repo. Also what's in rawhide may be a few days/weeks old by the time it gets in there. David Huff wrote: > Bryan Kearney wrote: >> David Huff wrote: >>> David Huff wrote: >>>> The first cut of appliance-tools, >>>> appliance-tools-003-3.fc10.noarch.rpm is now in Raw Hide and can be >>>> tested on a rawhide box. This first cut has a minimal feature set >>>> however it can be used for creating appliances from the command >>>> line. For instruction please refer to the following site: >>>> >>>> - http://thincrust.net/tooling.html >>>> >>>> Patches can be submitted to this mailing list: >>>> >>>> - thincrust-devel at redhat.com >>> >>> Also might be helpful to note the the source code is available here: >>> >>> http://git.et.redhat.com/?p=act.git;a=summary >> >> Do we need to aos repo any longer? > > This is an interesting question, on that I was thinking about > yesterday. Reason being that Jeremy just pulled all the kickstart > files from livecd-tools. They now only ship one minimal ks example > and mark it as docs. I think we will follow suit however may need to > keep it aos repo and move files there until we get everything in the > official spin kickstart repo. I am also interested to see this new > package that is susposto be created that just has the kickstarts in them. > > -D > > _______________________________________________ > Thincrust-devel mailing list > Thincrust-devel at redhat.com > https://www.redhat.com/mailman/listinfo/thincrust-devel From bkearney at redhat.com Wed Sep 3 18:34:32 2008 From: bkearney at redhat.com (Bryan Kearney) Date: Wed, 3 Sep 2008 14:34:32 -0400 Subject: [Thincrust-devel] Whitelisting Message-ID: <1220466873-30962-1-git-send-email-bkearney@redhat.com> Added whitelisting support, and removed the unnecessary portions of post file. From bkearney at redhat.com Wed Sep 3 18:34:33 2008 From: bkearney at redhat.com (Bryan Kearney) Date: Wed, 3 Sep 2008 14:34:33 -0400 Subject: [Thincrust-devel] [PATCH] Added whitelisting and blacklisting support. I also removed the old parts of hte In-Reply-To: <1220466873-30962-1-git-send-email-bkearney@redhat.com> References: <1220466873-30962-1-git-send-email-bkearney@redhat.com> Message-ID: <1220466873-30962-2-git-send-email-bkearney@redhat.com> --- Makefile | 2 + appliance-tools.spec | 1 + config/aos-f8.ks | 6 +-- config/aos-f9.ks | 6 +-- config/aos-rawhide.ks | 6 +-- config/base-post.ks | 97 ++++++++++-------------------------- tools/image-minimizer | 129 +++++++++++++++++++++++++++++++++++++++++++++++++ 7 files changed, 165 insertions(+), 82 deletions(-) create mode 100755 tools/image-minimizer diff --git a/Makefile b/Makefile index 457d7d5..973f466 100644 --- a/Makefile +++ b/Makefile @@ -16,6 +16,7 @@ all: install: $(INSTALL_PROGRAM) -D tools/appliance-creator $(DESTDIR)/usr/bin/appliance-creator + $(INSTALL_PROGRAM) -D tools/image-minimizer $(DESTDIR)/usr/bin/image-minimizer $(INSTALL_DATA) -D README $(DESTDIR)/usr/share/doc/appliance-tools-$(VERSION)/README $(INSTALL_DATA) -D COPYING $(DESTDIR)/usr/share/doc/appliance-tools-$(VERSION)/COPYING mkdir -p $(DESTDIR)/usr/share/appliance-tools/ @@ -26,6 +27,7 @@ install: uninstall: rm -f $(DESTDIR)/usr/bin/appliance-creator + rm -f $(DESTDIR)/usr/bin/image-minimizer rm -rf $(DESTDIR)/usr/lib/appliance-creator rm -rf $(DESTDIR)/usr/share/doc/appliance-tools-$(VERSION) rm -rf $(DESTDIR)/usr/share/appliance-tools diff --git a/appliance-tools.spec b/appliance-tools.spec index 540d912..b7e0bca 100644 --- a/appliance-tools.spec +++ b/appliance-tools.spec @@ -39,6 +39,7 @@ rm -rf $RPM_BUILD_ROOT %doc README %doc COPYING %{_bindir}/appliance-creator +%{_bindir}/imageminimizer %dir %{_datadir}/appliance-tools %{_datadir}/appliance-tools/* %dir %{python_sitelib}/appcreate diff --git a/config/aos-f8.ks b/config/aos-f8.ks index bfb0846..d856578 100644 --- a/config/aos-f8.ks +++ b/config/aos-f8.ks @@ -31,8 +31,6 @@ repo --name=f8-updates --mirrorlist=http://mirrors.fedoraproject.org/mirrorlist? %end # -# Add custom post scripts after the base post. +# Image minimization # -%post - #%include base-post.ks -%end +%include base-post.ks diff --git a/config/aos-f9.ks b/config/aos-f9.ks index 3851911..7e3da1f 100644 --- a/config/aos-f9.ks +++ b/config/aos-f9.ks @@ -31,8 +31,6 @@ repo --name=f9-updates --mirrorlist=http://mirrors.fedoraproject.org/mirrorlist? %end # -# Add custom post scripts after the base post. +# Image minimization # -%post - #%include base-post.ks -%end +%include base-post.ks diff --git a/config/aos-rawhide.ks b/config/aos-rawhide.ks index eeb65ef..f6cd5f6 100644 --- a/config/aos-rawhide.ks +++ b/config/aos-rawhide.ks @@ -30,8 +30,6 @@ repo --name=rawhide --mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?rep %end # -# Add custom post scripts after the base post. +# Image minimization # -%post - #%include base-post.ks -%end +%include base-post.ks diff --git a/config/base-post.ks b/config/base-post.ks index e244865..3a19035 100644 --- a/config/base-post.ks +++ b/config/base-post.ks @@ -1,70 +1,27 @@ -# from katzj's appliance creator script -# cut locale archive down to a smaller set; this should be handled -# automatically based on %packages --installLangs once that's supported -localedef --list-archive |grep -v en_US | xargs localedef --delete-from-archive -mv /usr/lib/locale/locale-archive /usr/lib/locale/locale-archive.tmpl -/usr/sbin/build-locale-archive - -# remove some things that mkinitrd doesn't really need in this case -# we should probably find a way to remove the deps and still have -# ordering work properly -rpm -e --nodeps mkinitrd kpartx dmraid mdadm lvm2 tar - -# fedora-release-notes is required by fedora-release and is pretty -# large, but not really needed for an appliance -rpm -e --nodeps fedora-release-notes - -# here, remove a bunch of files we don't need that are just eating up space. -# it breaks rpm slightly, but it's not too bad - -# FIXME: ug, hard-coded paths. This is going to break if we change to F-9 -# or upgrade certain packages. Not quite sure how to handle it better - -# Added from pmyers image-minimization patch -RM="rm -rf" -# Remove docs and internationalization -$RM /usr/share/omf -$RM /usr/share/gnome -$RM /usr/share/doc -$RM /usr/share/locale -$RM /usr/share/libthai -$RM /usr/share/man -$RM /usr/share/terminfo -$RM /usr/share/X11 -$RM /usr/share/i18n - -find /usr/share/zoneinfo -regextype egrep -type f ! -regex ".*/EST.*|.*/GMT" -exec $RM {} \; - -$RM /usr/lib/locale -$RM /usr/lib/syslinux -$RM /usr/lib64/gconv -$RM /usr/lib64/pango -$RM /usr/lib64/libpango* -$RM /etc/pango -$RM /usr/bin/pango* - -# Remove unnecessary kernel modules - -MODULES="XXX/lib/modules/*/kernel" -$RM $MODULES/sound - -fs_mods="9p affs autofs autofs4 befs bfs cifs coda configfs cramfs dlm \ - ecryptfs efs exportfs freevxfs fuse gfs2 hfs hfsplus jbd jbd2 \ - jffs jfs minix ncpfs ocfs2 qnx4 reiserfs romfs sysv udf ufs xfs" -for dir in $fs_mods ; do - $RM $MODULES/fs/$dir -done - -net_mods="802 8021q 9p appletalk atm ax25 bluetooth dccp decnet \ - ieee80211 ipx irda mac80211 netrom rfkill rose sched \ - sctp tipc wanrouter wireless" -for dir in $net_mods ; do - $RM $MODULES/net/$dir -done - -driver_mods="bluetooth firewire i2c isdn media edac" -for dir in $driver_mods ; do - $RM $MODULES/drivers/$dir -done - - +%post --nochroot --interpreter image-minimizer + drop /usr/share/omf + drop /usr/share/gnome + drop /usr/share/doc + drop /usr/share/libthai + drop /usr/share/man + drop /usr/share/terminfo + drop /usr/share/X11 + drop /usr/share/i18n + + # Remove many of the time zones. + drop /usr/share/zoneinfo + keep /usr/share/zoneinfo/EST + keep /usr/share/zoneinfo/UTC + + drop /usr/lib/locale + drop /usr/lib/syslinux + drop /usr/lib64/gconv + drop /usr/lib64/pango + drop /usr/lib64/libpango* + drop /etc/pango + drop /usr/bin/pango* + + # Nix all locale stuff + drop /usr/share/locale + keep /usr/share/locale/en_US +%end diff --git a/tools/image-minimizer b/tools/image-minimizer new file mode 100755 index 0000000..276f252 --- /dev/null +++ b/tools/image-minimizer @@ -0,0 +1,129 @@ +#!/usr/bin/python +# +# imageminimizer: removes files on the filesystem +# +# Copyright 2007, Red Hat Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Library General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. +import glob +import optparse +import os +import shutil +import sys + +class ImageMinimizer: + filename = '' + dryrun = False + prefix = None + keeps = set() + drops = set() + visited = set() + + def __init__(self, filename, root, dryrun): + self.filename = filename + self.prefix = root + self.dryrun = dryrun + + # Recursively adds all files and directories. + # This is done becuase globbing does not allow + # ** for arbitrary nesting. + def add_directory(self, files, dirname): + self.visited.add(dirname) + for root, dirs, items in os.walk(dirname): + for dir in dirs: + self.visited.add(os.path.join(root, dir)) + for name in items: + files.add(os.path.join(root, name)) + + def add_pattern(self, files, pattern): + if os.path.isdir(pattern): + self.add_directory(files, pattern) + else: + files.update(glob.glob(pattern)) + + # Parses each line in the ifle + def parse_line(self, line): + tup = line.partition(' ') + command = tup[0].lower() + pattern = tup[2].strip() + if not self.prefix == None: + pattern = pattern.lstrip('/') + pattern = os.path.join(self.prefix, pattern) + + # Strip out all the comments and blank lines + if not (command.startswith('#') or command==''): + if command == 'keep': + self.add_pattern(self.keeps, pattern) + elif command == 'drop': + self.add_pattern(self.drops, pattern) + else: + raise Exception ('Unknown Command: ' + command) + + def remove(self, files): + for tag in sorted(files, reverse=True): + self.visited.add(os.path.split(tag)[0]) + if os.path.isdir(tag): + self.visited.add(tag) + else: + if self.dryrun: + print 'rm ' + tag + else: + #print 'rm ' + tag + os.remove(tag) + + #remove all empty directory. Every 8k counts! + for dir in sorted(self.visited, reverse=True): + if len(os.listdir(dir)) == 0: + if self.dryrun: + print 'rm -rf ' + dir + else: + #print 'rm -rf ' + dir + os.rmdir(dir) + + def filter(self): + for line in (open(self.filename).readlines()): + self.parse_line(line.strip()) + final = self.drops.difference(self.keeps) + self.remove(final) + + +def parse_options(args): + usage = "usage: %prog [options] filename" + parser = optparse.OptionParser(usage=usage) + parser.set_defaults(root=os.environ.get('INSTALL_ROOT'), dry_run=False) + + parser.add_option("-i", "--installroot", type="string", dest="root", + help="Root path to prepend to all patterns. Defaults to INSTALL_ROOT") + + parser.add_option("--dryrun", action="store_true", dest="dryrun", + help="If set, no filesystem changes are made.") + + #imgcreate.setup_logging(parser) + + return parser.parse_args() + + +if __name__ == "__main__": + try: + (options, args) = parse_options(sys.argv[1:]) + filename = args[0] + minimizer = ImageMinimizer(filename, options.root, options.dryrun) + minimizer.filter() + except SystemExit, e: + sys.exit(e.code) + except KeyboardInterrupt, e: + print >> sys.stderr, _("Aborted at user request") + except Exception, e: + print e + sys.exit(1) -- 1.5.5.1 From dhuff at redhat.com Wed Sep 3 18:44:15 2008 From: dhuff at redhat.com (David Huff) Date: Wed, 03 Sep 2008 14:44:15 -0400 Subject: [Thincrust-devel] appliance-tools now in Raw Hide In-Reply-To: <48BEAD29.9090502@redhat.com> References: <48BD7CB6.2040606@redhat.com> <48BD8599.9080603@redhat.com> <48BE7F30.1000003@redhat.com> <48BE9C55.4060200@redhat.com> <48BEAD29.9090502@redhat.com> Message-ID: <48BEDAFF.6050600@redhat.com> Joey Boggs wrote: > We should keep the yum repo just for development access other than the > just the git repo. Also what's in rawhide may be a few days/weeks old > by the time it gets in there. > > ok, I have cleaned up the aos repo to mirror what is in the Appliance-tools rpm everything is tested except for the rhel stuff. I am pretty sure that the latest rhewl stuff is in this repo, however i haven't used it in a while. I have also removed all of the of the kickstart files from the appliance-tools rpm as well as the /usr/share/appliance-tools dir to mirror the changes made to livecd-tools . There is one flattened fk included in the rpm however it is now marked as a doc. again this mirrors the changes to livecd-tools. For now we will keep all AOS kickstart definition in the aos repo, with the hopes this will also be included in the spin kickstart pool in the future. http://git.et.redhat.com/?p=aos.git;a=summary -D From bkearney at redhat.com Thu Sep 4 14:24:16 2008 From: bkearney at redhat.com (Bryan Kearney) Date: Thu, 04 Sep 2008 10:24:16 -0400 Subject: [Thincrust-devel] appliance-tools now in Raw Hide In-Reply-To: <48BEDAFF.6050600@redhat.com> References: <48BD7CB6.2040606@redhat.com> <48BD8599.9080603@redhat.com> <48BE7F30.1000003@redhat.com> <48BE9C55.4060200@redhat.com> <48BEAD29.9090502@redhat.com> <48BEDAFF.6050600@redhat.com> Message-ID: <48BFEF90.5070603@redhat.com> David Huff wrote: > Joey Boggs wrote: >> We should keep the yum repo just for development access other than the >> just the git repo. Also what's in rawhide may be a few days/weeks old >> by the time it gets in there. >> >> > > ok, I have cleaned up the aos repo to mirror what is in the > Appliance-tools rpm everything is tested except for the rhel stuff. I > am pretty sure that the latest rhewl stuff is in this repo, however i > haven't used it in a while. > > I have also removed all of the of the kickstart files from the > appliance-tools rpm as well as the /usr/share/appliance-tools dir to > mirror the changes made to livecd-tools . There is one flattened fk > included in the rpm however it is now marked as a doc. again this > mirrors the changes to livecd-tools. > > For now we will keep all AOS kickstart definition in the aos repo, with > the hopes this will also be included in the spin kickstart pool in the > future. > > http://git.et.redhat.com/?p=aos.git;a=summary > > > -D > > _______________________________________________ > Thincrust-devel mailing list > Thincrust-devel at redhat.com > https://www.redhat.com/mailman/listinfo/thincrust-devel OK... ignore my last patch. Lemme pull new versions and resend to you. -- bk From bkearney at redhat.com Thu Sep 4 15:43:18 2008 From: bkearney at redhat.com (Bryan Kearney) Date: Thu, 4 Sep 2008 11:43:18 -0400 Subject: [Thincrust-devel] [PATCH] Added the image minimization to the refactored code Message-ID: <1220542998-25560-1-git-send-email-bkearney@redhat.com> From: root --- Makefile | 2 + appliance-tools.spec | 1 + tools/image-minimizer | 129 +++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 132 insertions(+), 0 deletions(-) create mode 100755 tools/image-minimizer diff --git a/Makefile b/Makefile index cf492eb..d50dffd 100644 --- a/Makefile +++ b/Makefile @@ -16,6 +16,7 @@ all: install: $(INSTALL_PROGRAM) -D tools/appliance-creator $(DESTDIR)/usr/bin/appliance-creator + $(INSTALL_PROGRAM) -D tools/image-minimizer $(DESTDIR)/usr/bin/image-minimizer $(INSTALL_DATA) -D README $(DESTDIR)/usr/share/doc/appliance-tools-$(VERSION)/README $(INSTALL_DATA) -D COPYING $(DESTDIR)/usr/share/doc/appliance-tools-$(VERSION)/COPYING mkdir -p $(DESTDIR)/usr/share/appliance-tools/ @@ -25,6 +26,7 @@ install: uninstall: rm -f $(DESTDIR)/usr/bin/appliance-creator + rm -f $(DESTDIR)/usr/bin/image-minimizer rm -rf $(DESTDIR)/usr/lib/appliance-creator rm -rf $(DESTDIR)/usr/share/doc/appliance-tools-$(VERSION) diff --git a/appliance-tools.spec b/appliance-tools.spec index 97e476c..abc9748 100644 --- a/appliance-tools.spec +++ b/appliance-tools.spec @@ -39,6 +39,7 @@ rm -rf $RPM_BUILD_ROOT %doc README COPYING %doc config/aos-rawhide.ks %{_bindir}/appliance-creator +%{_bindir}/image-minimizer %dir %{python_sitelib}/appcreate %{python_sitelib}/appcreate/*.py %{python_sitelib}/appcreate/*.pyo diff --git a/tools/image-minimizer b/tools/image-minimizer new file mode 100755 index 0000000..276f252 --- /dev/null +++ b/tools/image-minimizer @@ -0,0 +1,129 @@ +#!/usr/bin/python +# +# imageminimizer: removes files on the filesystem +# +# Copyright 2007, Red Hat Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Library General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. +import glob +import optparse +import os +import shutil +import sys + +class ImageMinimizer: + filename = '' + dryrun = False + prefix = None + keeps = set() + drops = set() + visited = set() + + def __init__(self, filename, root, dryrun): + self.filename = filename + self.prefix = root + self.dryrun = dryrun + + # Recursively adds all files and directories. + # This is done becuase globbing does not allow + # ** for arbitrary nesting. + def add_directory(self, files, dirname): + self.visited.add(dirname) + for root, dirs, items in os.walk(dirname): + for dir in dirs: + self.visited.add(os.path.join(root, dir)) + for name in items: + files.add(os.path.join(root, name)) + + def add_pattern(self, files, pattern): + if os.path.isdir(pattern): + self.add_directory(files, pattern) + else: + files.update(glob.glob(pattern)) + + # Parses each line in the ifle + def parse_line(self, line): + tup = line.partition(' ') + command = tup[0].lower() + pattern = tup[2].strip() + if not self.prefix == None: + pattern = pattern.lstrip('/') + pattern = os.path.join(self.prefix, pattern) + + # Strip out all the comments and blank lines + if not (command.startswith('#') or command==''): + if command == 'keep': + self.add_pattern(self.keeps, pattern) + elif command == 'drop': + self.add_pattern(self.drops, pattern) + else: + raise Exception ('Unknown Command: ' + command) + + def remove(self, files): + for tag in sorted(files, reverse=True): + self.visited.add(os.path.split(tag)[0]) + if os.path.isdir(tag): + self.visited.add(tag) + else: + if self.dryrun: + print 'rm ' + tag + else: + #print 'rm ' + tag + os.remove(tag) + + #remove all empty directory. Every 8k counts! + for dir in sorted(self.visited, reverse=True): + if len(os.listdir(dir)) == 0: + if self.dryrun: + print 'rm -rf ' + dir + else: + #print 'rm -rf ' + dir + os.rmdir(dir) + + def filter(self): + for line in (open(self.filename).readlines()): + self.parse_line(line.strip()) + final = self.drops.difference(self.keeps) + self.remove(final) + + +def parse_options(args): + usage = "usage: %prog [options] filename" + parser = optparse.OptionParser(usage=usage) + parser.set_defaults(root=os.environ.get('INSTALL_ROOT'), dry_run=False) + + parser.add_option("-i", "--installroot", type="string", dest="root", + help="Root path to prepend to all patterns. Defaults to INSTALL_ROOT") + + parser.add_option("--dryrun", action="store_true", dest="dryrun", + help="If set, no filesystem changes are made.") + + #imgcreate.setup_logging(parser) + + return parser.parse_args() + + +if __name__ == "__main__": + try: + (options, args) = parse_options(sys.argv[1:]) + filename = args[0] + minimizer = ImageMinimizer(filename, options.root, options.dryrun) + minimizer.filter() + except SystemExit, e: + sys.exit(e.code) + except KeyboardInterrupt, e: + print >> sys.stderr, _("Aborted at user request") + except Exception, e: + print e + sys.exit(1) -- 1.5.5.1 From bkearney at redhat.com Thu Sep 4 15:44:08 2008 From: bkearney at redhat.com (Bryan Kearney) Date: Thu, 4 Sep 2008 11:44:08 -0400 Subject: [Thincrust-devel] [PATCH] Added the image-minimization code to the relocated kickstart files Message-ID: <1220543048-25621-1-git-send-email-bkearney@redhat.com> From: root --- kickstarts/aos-f8.ks | 6 +-- kickstarts/aos-f9.ks | 7 +-- kickstarts/aos-rawhide.ks | 7 +-- kickstarts/base-post.ks | 97 ++++++++++++-------------------------------- 4 files changed, 35 insertions(+), 82 deletions(-) diff --git a/kickstarts/aos-f8.ks b/kickstarts/aos-f8.ks index bfb0846..d856578 100644 --- a/kickstarts/aos-f8.ks +++ b/kickstarts/aos-f8.ks @@ -31,8 +31,6 @@ repo --name=f8-updates --mirrorlist=http://mirrors.fedoraproject.org/mirrorlist? %end # -# Add custom post scripts after the base post. +# Image minimization # -%post - #%include base-post.ks -%end +%include base-post.ks diff --git a/kickstarts/aos-f9.ks b/kickstarts/aos-f9.ks index 3851911..8bc6074 100644 --- a/kickstarts/aos-f9.ks +++ b/kickstarts/aos-f9.ks @@ -31,8 +31,7 @@ repo --name=f9-updates --mirrorlist=http://mirrors.fedoraproject.org/mirrorlist? %end # -# Add custom post scripts after the base post. +# Image minimization # -%post - #%include base-post.ks -%end +%include base-post.ks + diff --git a/kickstarts/aos-rawhide.ks b/kickstarts/aos-rawhide.ks index eeb65ef..f152a25 100644 --- a/kickstarts/aos-rawhide.ks +++ b/kickstarts/aos-rawhide.ks @@ -30,8 +30,7 @@ repo --name=rawhide --mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?rep %end # -# Add custom post scripts after the base post. +# Image minimization # -%post - #%include base-post.ks -%end +%include base-post.ks + diff --git a/kickstarts/base-post.ks b/kickstarts/base-post.ks index e244865..3a19035 100644 --- a/kickstarts/base-post.ks +++ b/kickstarts/base-post.ks @@ -1,70 +1,27 @@ -# from katzj's appliance creator script -# cut locale archive down to a smaller set; this should be handled -# automatically based on %packages --installLangs once that's supported -localedef --list-archive |grep -v en_US | xargs localedef --delete-from-archive -mv /usr/lib/locale/locale-archive /usr/lib/locale/locale-archive.tmpl -/usr/sbin/build-locale-archive - -# remove some things that mkinitrd doesn't really need in this case -# we should probably find a way to remove the deps and still have -# ordering work properly -rpm -e --nodeps mkinitrd kpartx dmraid mdadm lvm2 tar - -# fedora-release-notes is required by fedora-release and is pretty -# large, but not really needed for an appliance -rpm -e --nodeps fedora-release-notes - -# here, remove a bunch of files we don't need that are just eating up space. -# it breaks rpm slightly, but it's not too bad - -# FIXME: ug, hard-coded paths. This is going to break if we change to F-9 -# or upgrade certain packages. Not quite sure how to handle it better - -# Added from pmyers image-minimization patch -RM="rm -rf" -# Remove docs and internationalization -$RM /usr/share/omf -$RM /usr/share/gnome -$RM /usr/share/doc -$RM /usr/share/locale -$RM /usr/share/libthai -$RM /usr/share/man -$RM /usr/share/terminfo -$RM /usr/share/X11 -$RM /usr/share/i18n - -find /usr/share/zoneinfo -regextype egrep -type f ! -regex ".*/EST.*|.*/GMT" -exec $RM {} \; - -$RM /usr/lib/locale -$RM /usr/lib/syslinux -$RM /usr/lib64/gconv -$RM /usr/lib64/pango -$RM /usr/lib64/libpango* -$RM /etc/pango -$RM /usr/bin/pango* - -# Remove unnecessary kernel modules - -MODULES="XXX/lib/modules/*/kernel" -$RM $MODULES/sound - -fs_mods="9p affs autofs autofs4 befs bfs cifs coda configfs cramfs dlm \ - ecryptfs efs exportfs freevxfs fuse gfs2 hfs hfsplus jbd jbd2 \ - jffs jfs minix ncpfs ocfs2 qnx4 reiserfs romfs sysv udf ufs xfs" -for dir in $fs_mods ; do - $RM $MODULES/fs/$dir -done - -net_mods="802 8021q 9p appletalk atm ax25 bluetooth dccp decnet \ - ieee80211 ipx irda mac80211 netrom rfkill rose sched \ - sctp tipc wanrouter wireless" -for dir in $net_mods ; do - $RM $MODULES/net/$dir -done - -driver_mods="bluetooth firewire i2c isdn media edac" -for dir in $driver_mods ; do - $RM $MODULES/drivers/$dir -done - - +%post --nochroot --interpreter image-minimizer + drop /usr/share/omf + drop /usr/share/gnome + drop /usr/share/doc + drop /usr/share/libthai + drop /usr/share/man + drop /usr/share/terminfo + drop /usr/share/X11 + drop /usr/share/i18n + + # Remove many of the time zones. + drop /usr/share/zoneinfo + keep /usr/share/zoneinfo/EST + keep /usr/share/zoneinfo/UTC + + drop /usr/lib/locale + drop /usr/lib/syslinux + drop /usr/lib64/gconv + drop /usr/lib64/pango + drop /usr/lib64/libpango* + drop /etc/pango + drop /usr/bin/pango* + + # Nix all locale stuff + drop /usr/share/locale + keep /usr/share/locale/en_US +%end -- 1.5.5.1 From dhuff at redhat.com Thu Sep 4 20:32:20 2008 From: dhuff at redhat.com (David Huff) Date: Thu, 04 Sep 2008 16:32:20 -0400 Subject: [Thincrust-devel] 3 commits: Added Whitelisting support. and cleaning up kick start post files, Fixed appliance-creator input options master, Removed all the kickstart files in the config dir to mirror livecd-tools Message-ID: <48C045D4.4080706@redhat.com> Added Whitelisting support. and cleaning up kick start post files autor Bryan Kearney Added Whitelisting support. and cleaning up kick start post files added /tools/image-minimizer Fixed appliance-creator input options master author David Huff Thu, 4 Sep 2008 18:29:31 +0000 (14:29 -0400) committer David Huff Thu, 4 Sep 2008 18:54:49 +0000 (14:54 -0400) commit 06fdf1d1d2b5665d31561c5f9f8e82854880a93f tree 029bdf8b571de6463435807327e687ed7a8e1e4b tree | snapshot parent caa95acd5f4f085a684d05beeffb354e1002a4ee commit | diff Fixed appliance-creator input options fixed option parsing group errors fixed wrong output "Wrote: image name" fixed --format, disk image is now converted via qemu-img convert added options to specify vmem and vcpu which is added in image.xml Removed all the kickstart files in the config dir to mirror livecd-tools author David Huff Wed, 3 Sep 2008 18:23:22 +0000 (14:23 -0400) committer David Huff Wed, 3 Sep 2008 18:27:00 +0000 (14:27 -0400) commit caa95acd5f4f085a684d05beeffb354e1002a4ee tree 6b2aa727916b6f2285792871f51c825171e53938 tree | snapshot parent 84869eae7b335468d7b2e3316782f932088b04e4 commit | diff Removed all the kickstart files in the config dir to mirror livecd-tools one configfile is included as an example however is now marked as a doc From jboggs at redhat.com Mon Sep 8 18:00:46 2008 From: jboggs at redhat.com (Joey Boggs) Date: Mon, 08 Sep 2008 14:00:46 -0400 Subject: [Thincrust-devel] [patch] ACT - manual directory option / other cleanups Message-ID: <48C5684E.3090703@redhat.com> -------------- next part -------------- A non-text attachment was scrubbed... Name: 0002-added-in-manual-directory-option-other-cleanups.patch Type: text/x-patch Size: 7730 bytes Desc: not available URL: From dhuff at redhat.com Tue Sep 9 20:45:33 2008 From: dhuff at redhat.com (David Huff) Date: Tue, 09 Sep 2008 16:45:33 -0400 Subject: [Thincrust-devel] [patch] ACT - manual directory option / other cleanups In-Reply-To: <48C5684E.3090703@redhat.com> References: <48C5684E.3090703@redhat.com> Message-ID: <48C6E06D.5010209@redhat.com> Joey Boggs wrote: > > ------------------------------------------------------------------------ > > _______________________________________________ > Thincrust-devel mailing list > Thincrust-devel at redhat.com > https://www.redhat.com/mailman/listinfo/thincrust-devel Committed From jboggs at redhat.com Wed Sep 10 21:31:09 2008 From: jboggs at redhat.com (Joey Boggs) Date: Wed, 10 Sep 2008 17:31:09 -0400 Subject: [Thincrust-devel] [patch] act - ec2 conversion option for appliance-creator Message-ID: <48C83C9D.80608@redhat.com> Only allows 1 disk in a kickstart currently, anymore will fail. Working on the virt-convert scenarios next. -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-added-ec2-conversion-option-to-appliance-creator.patch Type: text/x-patch Size: 4276 bytes Desc: not available URL: From bkearney at redhat.com Thu Sep 11 15:41:20 2008 From: bkearney at redhat.com (Bryan Kearney) Date: Thu, 11 Sep 2008 11:41:20 -0400 Subject: [Thincrust-devel] [patch] act - ec2 conversion option for appliance-creator In-Reply-To: <48C83C9D.80608@redhat.com> References: <48C83C9D.80608@redhat.com> Message-ID: <48C93C20.2070204@redhat.com> Joey Boggs wrote: > Only allows 1 disk in a kickstart currently, anymore will fail. Working > on the virt-convert scenarios next. I testted it and got the error below. Couple of comments 1) If I specify the --appliancetype then is the format, vmem, and vcpu items ignored? Huff, based on our chats yesterday should this be --type (to prep for package, ovf, vmware etc) -- bk [ Minimal BASH-like line editing is supported. For the first word, TAB lists possible command completions. Anywhere else TAB lists the possible completions of a device/filename.] grub> device (hd0) /dev/loop0 grub> root (hd0,0) Filesystem type is ext2fs, partition type 0x83 grub> setup --stage2=/root/cache/imgcreate-gNLr6K/install_root/boot/grub/stage2 --prefix=/boot/grub (hd0) Checking if "/boot/grub/stage1" exists... yes Checking if "/boot/grub/stage2" exists... yes Checking if "/boot/grub/e2fs_stage1_5" exists... yes Running "embed /boot/grub/e2fs_stage1_5 (hd0)"... failed (this is not fatal) Running "embed /boot/grub/e2fs_stage1_5 (hd0,0)"... failed (this is not fatal) Running "install --stage2=/root/cache/imgcreate-gNLr6K/install_root/boot/grub/stage2 /boot/grub/stage1 (hd0) /boot/grub/stage2 p /boot/grub/grub.conf "... succeeded Done. grub> quit TMPDIR: /var/tmp/ec2-convert-lmqjnakw loop0p1 : / Disk Space Required: 423M Creating a new disk image with additional freespace: 699M total dd: opening `/var/tmp/ec2-convert-lmqjnakw-tmpimage/ec2-diskimage.img': No such file or directory mke2fs 1.40.8 (13-Mar-2008) mke2fs: No such file or directory while trying to determine filesystem size /var/tmp/ec2-convert-lmqjnakw-tmpimage/ec2-diskimage.img: No such file or directory Performing rsync on all partitions to new root Unmounting /var/tmp/ec2-convert-lmqjnakw/ Freeing loopdevices openssh-server-5.1p1-2.fc9.i386 is installed curl-7.18.2-1.fc9.i386 is installed MAKEDEV: mkdir: File exists MAKEDEV: mkdir: File exists MAKEDEV: mkdir: File exists MAKEDEV: mkdir: File exists MAKEDEV: mkdir: File exists MAKEDEV: mkdir: File exists MAKEDEV: mkdir: File exists MAKEDEV: mkdir: File exists MAKEDEV: mkdir: File exists MAKEDEV: mkdir: File exists MAKEDEV: mkdir: File exists MAKEDEV: mkdir: File exists * - Updating /etc/fstab * - Creating rc.local configuration * - Creating ssh configuration * - Creating eth0 configuration * - Prevent nosegneg errors Adding EC2 Tools % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 8027k 100 8027k 0 0 1570k 0 0:00:05 0:00:05 --:--:-- 1939k Configure image for accepting the EC2 kernel % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 16.5M 100 16.5M 0 0 741k 0 0:00:22 0:00:22 --:--:-- 838k Preparing... ########################################### [100%] 1:kernel-xen ########################################### [100%] umount: /var/tmp/ec2-convert-lmqjnakw: not mounted Traceback (most recent call last): File "/usr/bin/ec2-converter", line 139, in sys.exit(main()) File "/usr/bin/ec2-converter", line 124, in main shutil.move(fsimage,imagefile) File "/usr/lib/python2.5/shutil.py", line 191, in move os.rename(src, dst) TypeError: coercing to Unicode: need string or buffer, NoneType found From bkearney at redhat.com Thu Sep 11 15:50:10 2008 From: bkearney at redhat.com (Bryan Kearney) Date: Thu, 11 Sep 2008 11:50:10 -0400 Subject: [Thincrust-devel] [PATCH] Added instLangs to the kickstart, and made appliance-creator executable Message-ID: <1221148210-7011-1-git-send-email-bkearney@redhat.com> --- config/aos-rawhide.ks | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) mode change 100644 => 100755 tools/appliance-creator diff --git a/config/aos-rawhide.ks b/config/aos-rawhide.ks index a219cbd..60d2c7b 100644 --- a/config/aos-rawhide.ks +++ b/config/aos-rawhide.ks @@ -26,7 +26,7 @@ part / --fstype="ext3" --ondisk=sda --size=500 --bytes-per-inode=4096 #%include base-post.ks %end -%packages --excludedocs --nobase +%packages --excludedocs --nobase --instLangs=en:fr @core kernel rootfiles diff --git a/tools/appliance-creator b/tools/appliance-creator old mode 100644 new mode 100755 -- 1.5.5.1 From jboggs at redhat.com Thu Sep 11 16:20:53 2008 From: jboggs at redhat.com (Joey Boggs) Date: Thu, 11 Sep 2008 12:20:53 -0400 Subject: [Thincrust-devel] [patch] act - ec2 conversion option for appliance-creator In-Reply-To: <48C93C20.2070204@redhat.com> References: <48C83C9D.80608@redhat.com> <48C93C20.2070204@redhat.com> Message-ID: <48C94565.3080006@redhat.com> I thought for some reason --type was assigned already, I'll make that change. Yes vmem and vcpu are ignored in the EC2 case, for the format option, so long as the created image is an accessible diskimage, the format will not matter. When the loopback unpartitioned image is added in we will most likely need to make some changes to accommodate that. What did you appliance-creator command look like to get the below error? I tested using the ec2-f9 and aos-rawhide kickstarts with no problems when I finished. Bryan Kearney wrote: > Joey Boggs wrote: >> Only allows 1 disk in a kickstart currently, anymore will fail. >> Working on the virt-convert scenarios next. > > I testted it and got the error below. Couple of comments > > 1) If I specify the --appliancetype then is the format, vmem, and vcpu > items ignored? Huff, based on our chats yesterday should this be > --type (to prep for package, ovf, vmware etc) > > -- bk > > > [ Minimal BASH-like line editing is supported. For the first word, TAB > lists possible command completions. Anywhere else TAB lists the > possible > completions of a device/filename.] > grub> device (hd0) /dev/loop0 > grub> root (hd0,0) > Filesystem type is ext2fs, partition type 0x83 > grub> setup > --stage2=/root/cache/imgcreate-gNLr6K/install_root/boot/grub/stage2 > --prefix=/boot/grub (hd0) > Checking if "/boot/grub/stage1" exists... yes > Checking if "/boot/grub/stage2" exists... yes > Checking if "/boot/grub/e2fs_stage1_5" exists... yes > Running "embed /boot/grub/e2fs_stage1_5 (hd0)"... failed (this is not > fatal) > Running "embed /boot/grub/e2fs_stage1_5 (hd0,0)"... failed (this is > not fatal) > Running "install > --stage2=/root/cache/imgcreate-gNLr6K/install_root/boot/grub/stage2 > /boot/grub/stage1 (hd0) /boot/grub/stage2 p /boot/grub/grub.conf "... > succeeded > Done. > grub> quit > > TMPDIR: /var/tmp/ec2-convert-lmqjnakw > loop0p1 : / > > Disk Space Required: 423M > > Creating a new disk image with additional freespace: 699M total > dd: opening > `/var/tmp/ec2-convert-lmqjnakw-tmpimage/ec2-diskimage.img': No such > file or directory > mke2fs 1.40.8 (13-Mar-2008) > mke2fs: No such file or directory while trying to determine filesystem > size > /var/tmp/ec2-convert-lmqjnakw-tmpimage/ec2-diskimage.img: No such file > or directory > > Performing rsync on all partitions to new root > > Unmounting /var/tmp/ec2-convert-lmqjnakw/ > Freeing loopdevices > > > openssh-server-5.1p1-2.fc9.i386 is installed > > curl-7.18.2-1.fc9.i386 is installed > > MAKEDEV: mkdir: File exists > MAKEDEV: mkdir: File exists > MAKEDEV: mkdir: File exists > MAKEDEV: mkdir: File exists > MAKEDEV: mkdir: File exists > MAKEDEV: mkdir: File exists > MAKEDEV: mkdir: File exists > MAKEDEV: mkdir: File exists > MAKEDEV: mkdir: File exists > MAKEDEV: mkdir: File exists > MAKEDEV: mkdir: File exists > MAKEDEV: mkdir: File exists > * - Updating /etc/fstab > > * - Creating rc.local configuration > > * - Creating ssh configuration > > * - Creating eth0 configuration > > * - Prevent nosegneg errors > > Adding EC2 Tools > % Total % Received % Xferd Average Speed Time Time Time > Current > Dload Upload Total Spent Left > Speed > 100 8027k 100 8027k 0 0 1570k 0 0:00:05 0:00:05 > --:--:-- 1939k > Configure image for accepting the EC2 kernel > % Total % Received % Xferd Average Speed Time Time Time > Current > Dload Upload Total Spent Left > Speed > 100 16.5M 100 16.5M 0 0 741k 0 0:00:22 0:00:22 > --:--:-- 838k > Preparing... > ########################################### [100%] > 1:kernel-xen > ########################################### [100%] > umount: /var/tmp/ec2-convert-lmqjnakw: not mounted > Traceback (most recent call last): > File "/usr/bin/ec2-converter", line 139, in > sys.exit(main()) > File "/usr/bin/ec2-converter", line 124, in main > shutil.move(fsimage,imagefile) > File "/usr/lib/python2.5/shutil.py", line 191, in move > os.rename(src, dst) > TypeError: coercing to Unicode: need string or buffer, NoneType found > > _______________________________________________ > Thincrust-devel mailing list > Thincrust-devel at redhat.com > https://www.redhat.com/mailman/listinfo/thincrust-devel From bkearney at redhat.com Thu Sep 11 16:25:47 2008 From: bkearney at redhat.com (Bryan Kearney) Date: Thu, 11 Sep 2008 12:25:47 -0400 Subject: [Thincrust-devel] [patch] act - ec2 conversion option for appliance-creator In-Reply-To: <48C94565.3080006@redhat.com> References: <48C83C9D.80608@redhat.com> <48C93C20.2070204@redhat.com> <48C94565.3080006@redhat.com> Message-ID: <48C9468B.20909@redhat.com> Joey Boggs wrote: > I thought for some reason --type was assigned already, I'll make that > change. Yes vmem and vcpu are ignored in the EC2 case, for the format > option, so long as the created image is an accessible diskimage, the > format will not matter. When the loopback unpartitioned image is added > in we will most likely need to make some changes to accommodate that. > > What did you appliance-creator command look like to get the below error? > > I tested using the ec2-f9 and aos-rawhide kickstarts with no problems > when I finished. I checked out the latest act, made a branch, imported your patch, and exported PYTHONPATH and PYTHONLIB to be /home/bkearney/src/act/ /home/bkearney/src/act/tools/appliance-creator -n ec2 -c ec2Appliance-f9.ks --appliancetype=EC2 -t /root/cache/ -- bk From jboggs at redhat.com Thu Sep 11 16:56:35 2008 From: jboggs at redhat.com (Joey Boggs) Date: Thu, 11 Sep 2008 12:56:35 -0400 Subject: [Thincrust-devel] [patch] act - ec2 conversion option for appliance-creator In-Reply-To: <48C9468B.20909@redhat.com> References: <48C83C9D.80608@redhat.com> <48C93C20.2070204@redhat.com> <48C94565.3080006@redhat.com> <48C9468B.20909@redhat.com> Message-ID: <48C94DC3.502@redhat.com> I can't reproduce, but looking from the output you pasted in, was there a "copying $file to $directory" line like below? If not thats why there's an error, but there shouldnt be any reason for it not to be there unless a portion of the patch didnt take. grub> quit Copying /root/cache/imgcreate-BeUdhE/out/ec2-sda.raw to /var/tmp/ec2-convert-qx6e918t-tmpimage TMPDIR: /var/tmp/ec2-convert-qx6e918t loop0p1 : / Bryan Kearney wrote: > Joey Boggs wrote: >> I thought for some reason --type was assigned already, I'll make that >> change. Yes vmem and vcpu are ignored in the EC2 case, for the format >> option, so long as the created image is an accessible diskimage, the >> format will not matter. When the loopback unpartitioned image is >> added in we will most likely need to make some changes to accommodate >> that. >> >> What did you appliance-creator command look like to get the below error? >> >> I tested using the ec2-f9 and aos-rawhide kickstarts with no problems >> when I finished. > > I checked out the latest act, made a branch, imported your patch, and > exported PYTHONPATH and PYTHONLIB to be /home/bkearney/src/act/ > > /home/bkearney/src/act/tools/appliance-creator -n ec2 -c > ec2Appliance-f9.ks --appliancetype=EC2 -t /root/cache/ > > > > -- bk > > _______________________________________________ > Thincrust-devel mailing list > Thincrust-devel at redhat.com > https://www.redhat.com/mailman/listinfo/thincrust-devel From dhuff at redhat.com Fri Sep 12 16:27:11 2008 From: dhuff at redhat.com (David Huff) Date: Fri, 12 Sep 2008 12:27:11 -0400 Subject: [Thincrust-devel] USER STORIES --type In-Reply-To: <48C93C20.2070204@redhat.com> References: <48C83C9D.80608@redhat.com> <48C93C20.2070204@redhat.com> Message-ID: <48CA985F.3010404@redhat.com> Bryan Kearney wrote: > Joey Boggs wrote: >> Only allows 1 disk in a kickstart currently, anymore will fail. >> Working on the virt-convert scenarios next. > > I testted it and got the error below. Couple of comments > > 1) If I specify the --appliancetype then is the format, vmem, and vcpu > items ignored? Huff, based on our chats yesterday should this be --type > (to prep for package, ovf, vmware etc) > So I started to summarize our conversation from yesterday, where we decided that there should be a --type flag that sets/overrides all other options for that type. Below are some of my thoughts form the discussion..... Valid types would be, KVM, XEN, VMARE, EC2. Overwritten options are, DISK FORMAT, META-DATA FORMAT, VMEM, VCPU, PACKAGE TYPE Package formats can be: TAR, ZIP, OVF, other When I started thinking about implementation I started confusing myself, the existing tools in this arena are: VIRT-CONVERT - converts vmware to libvirt calls qemu-img VIRT-PACK - converts libvirt to vmware[1] QEMU-IMG - converts disk formats EC2-CONVERTER - converts libvirt to ec2 appliance[1] *as far as I know, nothing exist to package appliance. This could be either stuck in appliance-tools or virt-convert I thought it might be helpful to start simple with user stories... User Stories: 1. Build a KVM appliance with 512M and 1VPUS, using a qcow2 formatted disk image, packed with libvirt meta-data in a zip file. 2. Build a VMWARE appliance with 512M and 1VPUS in a OVF bundle 3. Build a XEN appliance that will run on RHEL5, with 512M and 1VPUS, with a RAW disk image, libvirt meta-data packaged in a tar.gz bundle [2] 4. Bild a EC2 appliance and deploy to cloud. (lagre small type???) * In all 4 user stories, the appliance should be described with the same kickstart file, and build using the same appliance-creator command with the exception of --type [1] not sure if or how kernel modules or drives are handled, and my understating is they are needed. [2] not sure if virt convert can handle this and/or what changes need to be make to meta-data This is just me thinking out loud comments welcome... -D From jboggs at redhat.com Fri Sep 12 16:39:16 2008 From: jboggs at redhat.com (Joey Boggs) Date: Fri, 12 Sep 2008 12:39:16 -0400 Subject: [Thincrust-devel] USER STORIES --type In-Reply-To: <48CA985F.3010404@redhat.com> References: <48C83C9D.80608@redhat.com> <48C93C20.2070204@redhat.com> <48CA985F.3010404@redhat.com> Message-ID: <48CA9B34.4050604@redhat.com> comments inline David Huff wrote: > Bryan Kearney wrote: >> Joey Boggs wrote: >>> Only allows 1 disk in a kickstart currently, anymore will fail. >>> Working on the virt-convert scenarios next. >> >> I testted it and got the error below. Couple of comments >> >> 1) If I specify the --appliancetype then is the format, vmem, and >> vcpu items ignored? Huff, based on our chats yesterday should this be >> --type >> (to prep for package, ovf, vmware etc) >> > > So I started to summarize our conversation from yesterday, where we > decided that there should be a --type flag that sets/overrides all > other options for that type. Below are some of my thoughts form the > discussion..... > > Valid types would be, KVM, XEN, VMARE, EC2. > Overwritten options are, DISK FORMAT, META-DATA FORMAT, VMEM, VCPU, > PACKAGE TYPE > Package formats can be: TAR, ZIP, OVF, other > > When I started thinking about implementation I started confusing > myself, the existing tools in this arena are: > VIRT-CONVERT - converts vmware to libvirt calls qemu-img > VIRT-PACK - converts libvirt to vmware[1] I can see what it would take to merge this into virt-convert so we have one less tool to use. > QEMU-IMG - converts disk formats > EC2-CONVERTER - converts libvirt to ec2 appliance[1] > *as far as I know, nothing exist to package appliance. This could be > either stuck in appliance-tools or virt-convert This will be a second cut at it, but requires the ami tools to be inplace and setup properly with a jre. I'll need to test using our supplied jre. I believe it does but I dont remember what I set mine up with. > > > I thought it might be helpful to start simple with user stories... > User Stories: > > 1. Build a KVM appliance with 512M and 1VPUS, using a qcow2 formatted > disk image, packed with libvirt meta-data in a zip file. > > 2. Build a VMWARE appliance with 512M and 1VPUS in a OVF bundle > > 3. Build a XEN appliance that will run on RHEL5, with 512M and 1VPUS, > with a RAW disk image, libvirt meta-data packaged in a tar.gz bundle [2] Can add this into virt-convert, I haven't look at the code in while but it was built to be modular > > 4. Bild a EC2 appliance and deploy to cloud. (lagre small type???) We still run into the kernel issues in bundling, should we bundle against the latest available with amazon or have choices? > > * In all 4 user stories, the appliance should be described with the > same kickstart file, and build using the same appliance-creator > command with the exception of --type > > > [1] not sure if or how kernel modules or drives are handled, and my > understating is they are needed. > [2] not sure if virt convert can handle this and/or what changes need > to be make to meta-data > > > This is just me thinking out loud comments welcome... > > -D > > _______________________________________________ > Thincrust-devel mailing list > Thincrust-devel at redhat.com > https://www.redhat.com/mailman/listinfo/thincrust-devel From bkearney at redhat.com Fri Sep 12 16:54:55 2008 From: bkearney at redhat.com (Bryan Kearney) Date: Fri, 12 Sep 2008 12:54:55 -0400 Subject: [Thincrust-devel] USER STORIES --type In-Reply-To: <48CA985F.3010404@redhat.com> References: <48C83C9D.80608@redhat.com> <48C93C20.2070204@redhat.com> <48CA985F.3010404@redhat.com> Message-ID: <48CA9EDF.5020205@redhat.com> David Huff wrote: > Bryan Kearney wrote: >> Joey Boggs wrote: >>> Only allows 1 disk in a kickstart currently, anymore will fail. >>> Working on the virt-convert scenarios next. >> >> I testted it and got the error below. Couple of comments >> >> 1) If I specify the --appliancetype then is the format, vmem, and vcpu >> items ignored? Huff, based on our chats yesterday should this be --type >> (to prep for package, ovf, vmware etc) >> > > So I started to summarize our conversation from yesterday, where we > decided that there should be a --type flag that sets/overrides all other > options for that type. Below are some of my thoughts form the > discussion..... > > Valid types would be, KVM, XEN, VMARE, EC2. So, type is hypervisor? Is so, then packging should be libvirt? That is kinda wierd since libvirt can (in some cases) abstract out hypervisor. > Overwritten options are, DISK FORMAT, META-DATA FORMAT, VMEM, VCPU, > PACKAGE TYPE > Package formats can be: TAR, ZIP, OVF, other > > When I started thinking about implementation I started confusing myself, > the existing tools in this arena are: > VIRT-CONVERT - converts vmware to libvirt calls qemu-img > VIRT-PACK - converts libvirt to vmware[1] In the latest release notes, it was mentioned that this would go into virt-convert. > QEMU-IMG - converts disk formats > EC2-CONVERTER - converts libvirt to ec2 appliance[1] > *as far as I know, nothing exist to package appliance. This could be > either stuck in appliance-tools or virt-convert > > > I thought it might be helpful to start simple with user stories... > User Stories: > > 1. Build a KVM appliance with 512M and 1VPUS, using a qcow2 formatted > disk image, packed with libvirt meta-data in a zip file. > > 2. Build a VMWARE appliance with 512M and 1VPUS in a OVF bundle > > 3. Build a XEN appliance that will run on RHEL5, with 512M and 1VPUS, > with a RAW disk image, libvirt meta-data packaged in a tar.gz bundle [2] > > 4. Bild a EC2 appliance and deploy to cloud. (lagre small type???) > > * In all 4 user stories, the appliance should be described with the same > kickstart file, and build using the same appliance-creator command with > the exception of --type This is the goal. The question is can we assume/determine enough information to convert to all the target hypervisors (including paravirt ones). I think the user may want the the ueber-use case which is build them all, and let the end customer pick the hypervisor of their choice. So.. i think the question we may want to take to the virt team is how much do they want to do in virt-convirt. Should that layer modify the disk image internals (like the ec2 convereter does) or does that tool stick to metadata and we put a layer on top. > > > [1] not sure if or how kernel modules or drives are handled, and my > understating is they are needed. > [2] not sure if virt convert can handle this and/or what changes need to > be make to meta-data I think we will want to work to add the ability to take in a signed packaged into virt-image. From jboggs at redhat.com Fri Sep 12 17:04:25 2008 From: jboggs at redhat.com (Joey Boggs) Date: Fri, 12 Sep 2008 13:04:25 -0400 Subject: [Thincrust-devel] USER STORIES --type In-Reply-To: <48CA9EDF.5020205@redhat.com> References: <48C83C9D.80608@redhat.com> <48C93C20.2070204@redhat.com> <48CA985F.3010404@redhat.com> <48CA9EDF.5020205@redhat.com> Message-ID: <48CAA119.1060906@redhat.com> Bryan Kearney wrote: > David Huff wrote: >> Bryan Kearney wrote: >>> Joey Boggs wrote: >>>> Only allows 1 disk in a kickstart currently, anymore will fail. >>>> Working on the virt-convert scenarios next. >>> >>> I testted it and got the error below. Couple of comments >>> >>> 1) If I specify the --appliancetype then is the format, vmem, and >>> vcpu items ignored? Huff, based on our chats yesterday should this >>> be --type >>> (to prep for package, ovf, vmware etc) >>> >> >> So I started to summarize our conversation from yesterday, where we >> decided that there should be a --type flag that sets/overrides all >> other options for that type. Below are some of my thoughts form the >> discussion..... >> >> Valid types would be, KVM, XEN, VMARE, EC2. > So, type is hypervisor? Is so, then packging should be libvirt? That > is kinda wierd since libvirt can (in some cases) abstract out hypervisor. >> Overwritten options are, DISK FORMAT, META-DATA FORMAT, VMEM, VCPU, >> PACKAGE TYPE >> Package formats can be: TAR, ZIP, OVF, other >> >> When I started thinking about implementation I started confusing >> myself, the existing tools in this arena are: >> VIRT-CONVERT - converts vmware to libvirt calls qemu-img >> VIRT-PACK - converts libvirt to vmware[1] > > In the latest release notes, it was mentioned that this would go into > virt-convert. > Is anyone already working on this? >> QEMU-IMG - converts disk formats >> EC2-CONVERTER - converts libvirt to ec2 appliance[1] >> *as far as I know, nothing exist to package appliance. This could be >> either stuck in appliance-tools or virt-convert >> >> >> I thought it might be helpful to start simple with user stories... >> User Stories: >> >> 1. Build a KVM appliance with 512M and 1VPUS, using a qcow2 formatted >> disk image, packed with libvirt meta-data in a zip file. >> >> 2. Build a VMWARE appliance with 512M and 1VPUS in a OVF bundle >> >> 3. Build a XEN appliance that will run on RHEL5, with 512M and 1VPUS, >> with a RAW disk image, libvirt meta-data packaged in a tar.gz bundle [2] >> >> 4. Bild a EC2 appliance and deploy to cloud. (lagre small type???) >> >> * In all 4 user stories, the appliance should be described with the >> same kickstart file, and build using the same appliance-creator >> command with the exception of --type > > This is the goal. The question is can we assume/determine enough > information to convert to all the target hypervisors (including > paravirt ones). I think the user may want the the ueber-use case > which is build them all, and let the end customer pick the hypervisor > of their choice. > > So.. i think the question we may want to take to the virt team is how > much do they want to do in virt-convirt. Should that layer modify the > disk image internals (like the ec2 convereter does) or does that tool > stick to metadata and we put a layer on top. > >> >> >> [1] not sure if or how kernel modules or drives are handled, and my >> understating is they are needed. >> [2] not sure if virt convert can handle this and/or what changes need >> to be make to meta-data > > I think we will want to work to add the ability to take in a signed > packaged into virt-image. > > _______________________________________________ > Thincrust-devel mailing list > Thincrust-devel at redhat.com > https://www.redhat.com/mailman/listinfo/thincrust-devel From bkearney at redhat.com Fri Sep 12 17:17:31 2008 From: bkearney at redhat.com (Bryan Kearney) Date: Fri, 12 Sep 2008 13:17:31 -0400 Subject: [Thincrust-devel] USER STORIES --type In-Reply-To: <48CAA119.1060906@redhat.com> References: <48C83C9D.80608@redhat.com> <48C93C20.2070204@redhat.com> <48CA985F.3010404@redhat.com> <48CA9EDF.5020205@redhat.com> <48CAA119.1060906@redhat.com> Message-ID: <48CAA42B.2080006@redhat.com> Joey Boggs wrote: > Bryan Kearney wrote: >> David Huff wrote: >>> Bryan Kearney wrote: >>>> Joey Boggs wrote: >>>>> Only allows 1 disk in a kickstart currently, anymore will fail. >>>>> Working on the virt-convert scenarios next. >>>> >>>> I testted it and got the error below. Couple of comments >>>> >>>> 1) If I specify the --appliancetype then is the format, vmem, and >>>> vcpu items ignored? Huff, based on our chats yesterday should this >>>> be --type >>>> (to prep for package, ovf, vmware etc) >>>> >>> >>> So I started to summarize our conversation from yesterday, where we >>> decided that there should be a --type flag that sets/overrides all >>> other options for that type. Below are some of my thoughts form the >>> discussion..... >>> >>> Valid types would be, KVM, XEN, VMARE, EC2. >> So, type is hypervisor? Is so, then packging should be libvirt? That >> is kinda wierd since libvirt can (in some cases) abstract out hypervisor. >>> Overwritten options are, DISK FORMAT, META-DATA FORMAT, VMEM, VCPU, >>> PACKAGE TYPE >>> Package formats can be: TAR, ZIP, OVF, other >>> >>> When I started thinking about implementation I started confusing >>> myself, the existing tools in this arena are: >>> VIRT-CONVERT - converts vmware to libvirt calls qemu-img >>> VIRT-PACK - converts libvirt to vmware[1] >> >> In the latest release notes, it was mentioned that this would go into >> virt-convert. >> > > Is anyone already working on this? I do not know.. can you follow up with them on that? -- bk From dhuff at redhat.com Fri Sep 12 18:52:38 2008 From: dhuff at redhat.com (David Huff) Date: Fri, 12 Sep 2008 14:52:38 -0400 Subject: [Thincrust-devel] USER STORIES --type In-Reply-To: <48CA9EDF.5020205@redhat.com> References: <48C83C9D.80608@redhat.com> <48C93C20.2070204@redhat.com> <48CA985F.3010404@redhat.com> <48CA9EDF.5020205@redhat.com> Message-ID: <48CABA76.2070004@redhat.com> Bryan Kearney wrote: > David Huff wrote: >> Bryan Kearney wrote: >>> Joey Boggs wrote: >>>> Only allows 1 disk in a kickstart currently, anymore will fail. >>>> Working on the virt-convert scenarios next. >>> >>> I testted it and got the error below. Couple of comments >>> >>> 1) If I specify the --appliancetype then is the format, vmem, and >>> vcpu items ignored? Huff, based on our chats yesterday should this be >>> --type >>> (to prep for package, ovf, vmware etc) >>> >> >> So I started to summarize our conversation from yesterday, where we >> decided that there should be a --type flag that sets/overrides all >> other options for that type. Below are some of my thoughts form the >> discussion..... >> >> Valid types would be, KVM, XEN, VMARE, EC2. > So, type is hypervisor? Is so, then packging should be libvirt? That is > kinda wierd since libvirt can (in some cases) abstract out hypervisor. I don't think its hypervisor more of use case. What the end user wants. Although libvirt does abstract some hypervisors and does not on some. >> Overwritten options are, DISK FORMAT, META-DATA FORMAT, VMEM, VCPU, >> PACKAGE TYPE >> Package formats can be: TAR, ZIP, OVF, other >> >> When I started thinking about implementation I started confusing >> myself, the existing tools in this arena are: >> VIRT-CONVERT - converts vmware to libvirt calls qemu-img >> VIRT-PACK - converts libvirt to vmware[1] > > In the latest release notes, it was mentioned that this would go into > virt-convert. > >> QEMU-IMG - converts disk formats >> EC2-CONVERTER - converts libvirt to ec2 appliance[1] >> *as far as I know, nothing exist to package appliance. This could be >> either stuck in appliance-tools or virt-convert >> >> >> I thought it might be helpful to start simple with user stories... >> User Stories: >> >> 1. Build a KVM appliance with 512M and 1VPUS, using a qcow2 formatted >> disk image, packed with libvirt meta-data in a zip file. >> >> 2. Build a VMWARE appliance with 512M and 1VPUS in a OVF bundle >> >> 3. Build a XEN appliance that will run on RHEL5, with 512M and 1VPUS, >> with a RAW disk image, libvirt meta-data packaged in a tar.gz bundle [2] >> >> 4. Bild a EC2 appliance and deploy to cloud. (lagre small type???) >> >> * In all 4 user stories, the appliance should be described with the >> same kickstart file, and build using the same appliance-creator >> command with the exception of --type > > This is the goal. The question is can we assume/determine enough > information to convert to all the target hypervisors (including paravirt > ones). I think the user may want the the ueber-use case which is build > them all, and let the end customer pick the hypervisor of their choice. > > So.. i think the question we may want to take to the virt team is how > much do they want to do in virt-convirt. Should that layer modify the > disk image internals (like the ec2 convereter does) or does that tool > stick to metadata and we put a layer on top. We also need to bring up the packageing iussue how do we want to package it and where do we what this to happen. > >> >> >> [1] not sure if or how kernel modules or drives are handled, and my >> understating is they are needed. >> [2] not sure if virt convert can handle this and/or what changes need >> to be make to meta-data > > I think we will want to work to add the ability to take in a signed > packaged into virt-image. > > _______________________________________________ > Thincrust-devel mailing list > Thincrust-devel at redhat.com > https://www.redhat.com/mailman/listinfo/thincrust-devel From bkearney at redhat.com Mon Sep 15 12:38:53 2008 From: bkearney at redhat.com (Bryan Kearney) Date: Mon, 15 Sep 2008 08:38:53 -0400 Subject: [Thincrust-devel] [Fwd: [Fedora-spins] Fedora 10 Beta Freeze (Spins)] Message-ID: <48CE575D.9010507@redhat.com> Can you guys please (today) check out the latest code, and execute the release engineering steps [1] to make sure that we have everything set up for rel-eng? Thanks! -- bk [1] https://fedoraproject.org/wiki/Features/ApplianceTools#Release_Engineering -------- Original Message -------- Subject: [Fedora-spins] Fedora 10 Beta Freeze (Spins) Date: Sun, 14 Sep 2008 16:53:02 +0200 From: Jeroen van Meeuwen Reply-To: The Spin Special Interest Group mailing list To: The Spin Special Interest Group mailing list , Discussion list for Fedora Release Engineering Hi there, Fedora 10 has entered it's Beta Freeze stage which in your case is the time to start making absolutely sure your Spins are in perfect shape for being composed by Release Engineering. Things you will need to make sure are, amongst others: - it is not oversized - it composes without troubles - it ends up like you want it - it boots, and runs - it shows no unexpected behavior in applications you include Right now, the following kickstarts are approved by the Spin SIG: = fedora-aos.ks (Fedora AOS) = - SELinux was added back to address the Board's concern with it being removed originally. This should resolve the only issue the Board raised but we've not yet seen explicit approval. - The tool this spin is supposed to be composed with, appliance-tools, is currently at Release Engineering's discretion. As said before, this spin also composes with livecd-tools and spits out a nice minimal live ISO which then again of course also includes the raw ext3 .img file. - Approved by Spin SIG, - Approved by Board - Approved by Rel. Eng. Feature Page: https://fedoraproject.org/wiki/Features/appliance-tools = fedora-livecd-broffice.org.ks (Fedora BrOffice.org) = - Approved by Spin SIG, - Approved by Board, - Approved by Rel. Eng. Feature Page: https://fedoraproject.org/wiki/Features/BrOffice.orgSpin = fedora-livecd-desktop-default.ks (Fedora Desktop) = - One of our permanent spins = fedora-livecd-education-math.ks (Fedora Education Math) = - Approved by Spin SIG, - Approved by Board, - Pending approval from Rel. Eng. Feature Page: https://fedoraproject.org/wiki/Features/EducationMathSpin = fedora-livecd-kde.ks (Fedora KDE) = - One of our permanent spins = fedora-livecd-xfce.ks (Fedora XFCE) = - Approved by Spin SIG, - Approved by Board after a little shuffling with the Feature Page, - Pending approval from Rel. Eng. Feature Page: https://fedoraproject.org/wiki/Features/XfceSpin = fedora-livedvd-developer.ks (Fedora Developer) = - Approved by Spin SIG, - Approved by Board, - Pending approval from Rel. Eng. Feature Page: https://fedoraproject.org/wiki/Features/DeveloperSpin = fedora-livedvd-electronic-lab.ks (Fedora Electronic Lab) = - One of the spins that has been approved previously (Fedora 8) - Approved by Spin SIG, - Approved by Board, - Pending approval from Rel. Eng. (but does not have the right category on it's Feature page to show up on Rel. Eng.'s radar, maybe since the feature page existed for Fedora 8 already). Feature Page: https://fedoraproject.org/wiki/Features/FedoraElectronicLab = fedora-livedvd-games.ks (Fedora Games) = - One of the Spins that have been previously accepted (Fedora 8) and is now re-entering the process. - Approved by Spin SIG, - No Approval from Board requested, but previously approved, - Not yet approved for Fedora 10 by Rel. Eng. (but in the FeatureReadyForFesco category). Feature Page: https://fedoraproject.org/wiki/Features/GamesSpin For the Fedora 10 Beta, a new branch has been created in the spin-kickstarts repository, called F-10-Beta. This branch only contains the files necessary for composing the spins. Note that like the rawhide tree, this branch is supposed to be frozen and as such you should only be applying fixes, no huge changes. To checkout the F-10-Beta branch and start working with it, do: git pull git checkout --track -b F-10-Beta origin/F-10-Beta You should not commit any changes to this branch as this is what Rel. Eng. is going to use to compose the spins from, and we do not know when that is exactly. Fixes go into the master branch and are pulled into the F-10-Beta branch only if necessary. If you have any questions or remarks, please don't hesitate and drop the list a line. Kind regards, Jeroen van Meeuwen -kanarip _______________________________________________ Fedora-spins mailing list Fedora-spins at lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/fedora-spins From dhuff at redhat.com Mon Sep 15 13:41:41 2008 From: dhuff at redhat.com (David Huff) Date: Mon, 15 Sep 2008 09:41:41 -0400 Subject: [Thincrust-devel] [Fwd: [Fedora-spins] Fedora 10 Beta Freeze (Spins)] In-Reply-To: <48CE575D.9010507@redhat.com> References: <48CE575D.9010507@redhat.com> Message-ID: <48CE6615.5080504@redhat.com> Bryan Kearney wrote: > Can you guys please (today) check out the latest code, and execute the > release engineering steps [1] to make sure that we have everything set > up for rel-eng? > > Thanks! > > -- bk > I ran a test from the current ks in the kickstart pool at http://git.fedorahosted.org/git/?p=spin-kickstarts.git, I did hit a space problem and had to bump the size to 550M. Unable to create appliance : Unable to install: [('installing package kernel-2.6.27-0.323.rc6.fc10.x86_64 needs 91MB on the /var/tmp/imgcreate-_aG9G5/install_root filesystem', (9, '/var/tmp/imgcreate-_aG9G5/install_root', 95195136L))] Unable to create appliance : Unable to install: [('installing package I guess adding selinux added some space?? After increasing the partition size in the kickstart file the appliance built successfully. The resulted booted appliance has a / portion of 517M and uses 484M. After zipping the image and xml file the binary image is 114M. I also noticed that the root pw is not set. -D From bkearney at redhat.com Mon Sep 15 15:24:01 2008 From: bkearney at redhat.com (Bryan Kearney) Date: Mon, 15 Sep 2008 11:24:01 -0400 Subject: [Thincrust-devel] [Fwd: [Fedora-spins] Fedora 10 Beta Freeze (Spins)] In-Reply-To: <48CE6615.5080504@redhat.com> References: <48CE575D.9010507@redhat.com> <48CE6615.5080504@redhat.com> Message-ID: <48CE7E11.6090304@redhat.com> David Huff wrote: > Bryan Kearney wrote: >> Can you guys please (today) check out the latest code, and execute the >> release engineering steps [1] to make sure that we have everything set >> up for rel-eng? >> >> Thanks! >> >> -- bk >> > > I ran a test from the current ks in the kickstart pool at > http://git.fedorahosted.org/git/?p=spin-kickstarts.git, I did hit a > space problem and had to bump the size to 550M. > > Unable to create appliance : Unable to install: [('installing package > kernel-2.6.27-0.323.rc6.fc10.x86_64 needs 91MB on the > /var/tmp/imgcreate-_aG9G5/install_root filesystem', (9, > '/var/tmp/imgcreate-_aG9G5/install_root', 95195136L))] > Unable to create appliance : Unable to install: [('installing package > > I guess adding selinux added some space?? After increasing the > partition size in the kickstart file the appliance built successfully. > The resulted booted appliance has a / portion of 517M and uses 484M. > What did you do to build this? I did not get this error. I did the following: - Upgrade an F9 machine to the new keys. - Enable the rawhide repos - install the livecd and appliance tools from rawhide - checked out the F10 kickstart pool - appliance-creator --name fedora-aos --config fedora-aos.ks -- bk From jboggs at redhat.com Mon Sep 15 17:09:39 2008 From: jboggs at redhat.com (Joey Boggs) Date: Mon, 15 Sep 2008 13:09:39 -0400 Subject: [Thincrust-devel] [Fwd: [Fedora-spins] Fedora 10 Beta Freeze (Spins)] In-Reply-To: <48CE7E11.6090304@redhat.com> References: <48CE575D.9010507@redhat.com> <48CE6615.5080504@redhat.com> <48CE7E11.6090304@redhat.com> Message-ID: <48CE96D3.4050700@redhat.com> Worked well for me Might want to update the following though: 6. Use "virsh list" to see if its running and get the IDNUM # vish list fedora-aos virsh instead of vish ^^^ Bryan Kearney wrote: > David Huff wrote: >> Bryan Kearney wrote: >>> Can you guys please (today) check out the latest code, and execute >>> the release engineering steps [1] to make sure that we have >>> everything set up for rel-eng? >>> >>> Thanks! >>> >>> -- bk >>> >> >> I ran a test from the current ks in the kickstart pool at >> http://git.fedorahosted.org/git/?p=spin-kickstarts.git, I did hit a >> space problem and had to bump the size to 550M. >> >> Unable to create appliance : Unable to install: [('installing package >> kernel-2.6.27-0.323.rc6.fc10.x86_64 needs 91MB on the >> /var/tmp/imgcreate-_aG9G5/install_root filesystem', (9, >> '/var/tmp/imgcreate-_aG9G5/install_root', 95195136L))] >> Unable to create appliance : Unable to install: [('installing package >> >> I guess adding selinux added some space?? After increasing the >> partition size in the kickstart file the appliance built >> successfully. The resulted booted appliance has a / portion of 517M >> and uses 484M. >> > > What did you do to build this? I did not get this error. I did the > following: > > - Upgrade an F9 machine to the new keys. > - Enable the rawhide repos > - install the livecd and appliance tools from rawhide > - checked out the F10 kickstart pool > - appliance-creator --name fedora-aos --config fedora-aos.ks > > -- bk > > _______________________________________________ > Thincrust-devel mailing list > Thincrust-devel at redhat.com > https://www.redhat.com/mailman/listinfo/thincrust-devel From jboggs at redhat.com Mon Sep 15 18:17:22 2008 From: jboggs at redhat.com (Joey Boggs) Date: Mon, 15 Sep 2008 14:17:22 -0400 Subject: [Thincrust-devel] [patch] act - ec2 conversion option for appliance-creator In-Reply-To: <48C93C20.2070204@redhat.com> References: <48C83C9D.80608@redhat.com> <48C93C20.2070204@redhat.com> Message-ID: <48CEA6B2.9000606@redhat.com> Updated patch with --type rather than --appliancetype Also added man page entry for the --type option In process of creating a man page for ec2-converter BK, can you try it again and see if you're getting any errors now? Bryan Kearney wrote: > Joey Boggs wrote: >> Only allows 1 disk in a kickstart currently, anymore will fail. >> Working on the virt-convert scenarios next. > > I testted it and got the error below. Couple of comments > > 1) If I specify the --appliancetype then is the format, vmem, and vcpu > items ignored? Huff, based on our chats yesterday should this be > --type (to prep for package, ovf, vmware etc) > > -- bk > > > [ Minimal BASH-like line editing is supported. For the first word, TAB > lists possible command completions. Anywhere else TAB lists the > possible > completions of a device/filename.] > grub> device (hd0) /dev/loop0 > grub> root (hd0,0) > Filesystem type is ext2fs, partition type 0x83 > grub> setup > --stage2=/root/cache/imgcreate-gNLr6K/install_root/boot/grub/stage2 > --prefix=/boot/grub (hd0) > Checking if "/boot/grub/stage1" exists... yes > Checking if "/boot/grub/stage2" exists... yes > Checking if "/boot/grub/e2fs_stage1_5" exists... yes > Running "embed /boot/grub/e2fs_stage1_5 (hd0)"... failed (this is not > fatal) > Running "embed /boot/grub/e2fs_stage1_5 (hd0,0)"... failed (this is > not fatal) > Running "install > --stage2=/root/cache/imgcreate-gNLr6K/install_root/boot/grub/stage2 > /boot/grub/stage1 (hd0) /boot/grub/stage2 p /boot/grub/grub.conf "... > succeeded > Done. > grub> quit > > TMPDIR: /var/tmp/ec2-convert-lmqjnakw > loop0p1 : / > > Disk Space Required: 423M > > Creating a new disk image with additional freespace: 699M total > dd: opening > `/var/tmp/ec2-convert-lmqjnakw-tmpimage/ec2-diskimage.img': No such > file or directory > mke2fs 1.40.8 (13-Mar-2008) > mke2fs: No such file or directory while trying to determine filesystem > size > /var/tmp/ec2-convert-lmqjnakw-tmpimage/ec2-diskimage.img: No such file > or directory > > Performing rsync on all partitions to new root > > Unmounting /var/tmp/ec2-convert-lmqjnakw/ > Freeing loopdevices > > > openssh-server-5.1p1-2.fc9.i386 is installed > > curl-7.18.2-1.fc9.i386 is installed > > MAKEDEV: mkdir: File exists > MAKEDEV: mkdir: File exists > MAKEDEV: mkdir: File exists > MAKEDEV: mkdir: File exists > MAKEDEV: mkdir: File exists > MAKEDEV: mkdir: File exists > MAKEDEV: mkdir: File exists > MAKEDEV: mkdir: File exists > MAKEDEV: mkdir: File exists > MAKEDEV: mkdir: File exists > MAKEDEV: mkdir: File exists > MAKEDEV: mkdir: File exists > * - Updating /etc/fstab > > * - Creating rc.local configuration > > * - Creating ssh configuration > > * - Creating eth0 configuration > > * - Prevent nosegneg errors > > Adding EC2 Tools > % Total % Received % Xferd Average Speed Time Time Time > Current > Dload Upload Total Spent Left > Speed > 100 8027k 100 8027k 0 0 1570k 0 0:00:05 0:00:05 > --:--:-- 1939k > Configure image for accepting the EC2 kernel > % Total % Received % Xferd Average Speed Time Time Time > Current > Dload Upload Total Spent Left > Speed > 100 16.5M 100 16.5M 0 0 741k 0 0:00:22 0:00:22 > --:--:-- 838k > Preparing... > ########################################### [100%] > 1:kernel-xen > ########################################### [100%] > umount: /var/tmp/ec2-convert-lmqjnakw: not mounted > Traceback (most recent call last): > File "/usr/bin/ec2-converter", line 139, in > sys.exit(main()) > File "/usr/bin/ec2-converter", line 124, in main > shutil.move(fsimage,imagefile) > File "/usr/lib/python2.5/shutil.py", line 191, in move > os.rename(src, dst) > TypeError: coercing to Unicode: need string or buffer, NoneType found > > _______________________________________________ > Thincrust-devel mailing list > Thincrust-devel at redhat.com > https://www.redhat.com/mailman/listinfo/thincrust-devel -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-ec2-options.patch Type: text/x-patch Size: 5106 bytes Desc: not available URL: From dhuff at redhat.com Mon Sep 15 18:33:22 2008 From: dhuff at redhat.com (David Huff) Date: Mon, 15 Sep 2008 14:33:22 -0400 Subject: [Thincrust-devel] [Fwd: [Fedora-spins] Fedora 10 Beta Freeze (Spins)] In-Reply-To: <48CE7E11.6090304@redhat.com> References: <48CE575D.9010507@redhat.com> <48CE6615.5080504@redhat.com> <48CE7E11.6090304@redhat.com> Message-ID: <48CEAA72.3050405@redhat.com> Bryan Kearney wrote: > David Huff wrote: >> Bryan Kearney wrote: >>> Can you guys please (today) check out the latest code, and execute >>> the release engineering steps [1] to make sure that we have >>> everything set up for rel-eng? >>> >>> Thanks! >>> >>> -- bk >>> >> >> I ran a test from the current ks in the kickstart pool at >> http://git.fedorahosted.org/git/?p=spin-kickstarts.git, I did hit a >> space problem and had to bump the size to 550M. >> >> Unable to create appliance : Unable to install: [('installing package >> kernel-2.6.27-0.323.rc6.fc10.x86_64 needs 91MB on the >> /var/tmp/imgcreate-_aG9G5/install_root filesystem', (9, >> '/var/tmp/imgcreate-_aG9G5/install_root', 95195136L))] >> Unable to create appliance : Unable to install: [('installing package >> >> I guess adding selinux added some space?? After increasing the >> partition size in the kickstart file the appliance built successfully. >> The resulted booted appliance has a / portion of 517M and uses 484M. >> > > What did you do to build this? I did not get this error. I did the > following: > > - Upgrade an F9 machine to the new keys. > - Enable the rawhide repos > - install the livecd and appliance tools from rawhide > - checked out the F10 kickstart pool > - appliance-creator --name fedora-aos --config fedora-aos.ks > > -- bk I am seeing this on both a fully updated F10 & F9 box building the kickstart file from the ks pool. The only difference i can think of is that I am running 64bit OS on both testboxes. Which may pull in more packages? Once I increase the partition size and get a appliance up there are 167 packages installed. and it uses 484M of disk space. -D From bkearney at redhat.com Mon Sep 15 20:18:07 2008 From: bkearney at redhat.com (Bryan Kearney) Date: Mon, 15 Sep 2008 16:18:07 -0400 Subject: [Thincrust-devel] [Fwd: [Fedora-spins] Fedora 10 Beta Freeze (Spins)] In-Reply-To: <48CEAA72.3050405@redhat.com> References: <48CE575D.9010507@redhat.com> <48CE6615.5080504@redhat.com> <48CE7E11.6090304@redhat.com> <48CEAA72.3050405@redhat.com> Message-ID: <48CEC2FF.6060108@redhat.com> David Huff wrote: > Bryan Kearney wrote: >> David Huff wrote: >>> Bryan Kearney wrote: >>>> Can you guys please (today) check out the latest code, and execute >>>> the release engineering steps [1] to make sure that we have >>>> everything set up for rel-eng? >>>> >>>> Thanks! >>>> >>>> -- bk >>>> >>> >>> I ran a test from the current ks in the kickstart pool at >>> http://git.fedorahosted.org/git/?p=spin-kickstarts.git, I did hit a >>> space problem and had to bump the size to 550M. >>> >>> Unable to create appliance : Unable to install: [('installing package >>> kernel-2.6.27-0.323.rc6.fc10.x86_64 needs 91MB on the >>> /var/tmp/imgcreate-_aG9G5/install_root filesystem', (9, >>> '/var/tmp/imgcreate-_aG9G5/install_root', 95195136L))] >>> Unable to create appliance : Unable to install: [('installing package >>> >>> I guess adding selinux added some space?? After increasing the >>> partition size in the kickstart file the appliance built >>> successfully. The resulted booted appliance has a / portion of 517M >>> and uses 484M. >>> >> >> What did you do to build this? I did not get this error. I did the >> following: >> >> - Upgrade an F9 machine to the new keys. >> - Enable the rawhide repos >> - install the livecd and appliance tools from rawhide >> - checked out the F10 kickstart pool >> - appliance-creator --name fedora-aos --config fedora-aos.ks >> >> -- bk > > > I am seeing this on both a fully updated F10 & F9 box building the > kickstart file from the ks pool. The only difference i can think of is > that I am running 64bit OS on both testboxes. Which may pull in more > packages? > > Once I increase the partition size and get a appliance up there are 167 > packages installed. and it uses 484M of disk space. My guess is that this is caused by the increased package size for the 64 bit rpms. Lemme ask kanarip what he thinks the best approach would be. -- bk From bkearney at redhat.com Tue Sep 16 12:19:49 2008 From: bkearney at redhat.com (Bryan Kearney) Date: Tue, 16 Sep 2008 08:19:49 -0400 Subject: [Thincrust-devel] [Fwd: [Fedora-spins] Fedora 10 Beta Freeze (Spins)] In-Reply-To: <48CE96D3.4050700@redhat.com> References: <48CE575D.9010507@redhat.com> <48CE6615.5080504@redhat.com> <48CE7E11.6090304@redhat.com> <48CE96D3.4050700@redhat.com> Message-ID: <48CFA465.9000701@redhat.com> Joey Boggs wrote: > Worked well for me > > Might want to update the following though: > > 6. Use "virsh list" to see if its running and get the IDNUM > > # vish list fedora-aos > > virsh instead of vish ^^^ Thanks... .done. -- bk From bkearney at redhat.com Tue Sep 16 12:31:24 2008 From: bkearney at redhat.com (Bryan Kearney) Date: Tue, 16 Sep 2008 08:31:24 -0400 Subject: [Thincrust-devel] [Fwd: [Fedora-spins] Fedora 10 Beta Freeze (Spins)] In-Reply-To: <48CEAA72.3050405@redhat.com> References: <48CE575D.9010507@redhat.com> <48CE6615.5080504@redhat.com> <48CE7E11.6090304@redhat.com> <48CEAA72.3050405@redhat.com> Message-ID: <48CFA71C.10600@redhat.com> David Huff wrote: > Bryan Kearney wrote: >> David Huff wrote: >>> Bryan Kearney wrote: > > I am seeing this on both a fully updated F10 & F9 box building the > kickstart file from the ks pool. The only difference i can think of is > that I am running 64bit OS on both testboxes. Which may pull in more > packages? > > Once I increase the partition size and get a appliance up there are 167 > packages installed. and it uses 484M of disk space. > > -D Can you try the latest in the mainline of the kickstarts repo? I pushed it to 550. If you can build with that one, I will ask kanarip to pull it in. --bk From dhuff at redhat.com Tue Sep 16 14:45:52 2008 From: dhuff at redhat.com (David Huff) Date: Tue, 16 Sep 2008 10:45:52 -0400 Subject: [Thincrust-devel] [Fwd: [Fedora-spins] Fedora 10 Beta Freeze (Spins)] In-Reply-To: <48CFA71C.10600@redhat.com> References: <48CE575D.9010507@redhat.com> <48CE6615.5080504@redhat.com> <48CE7E11.6090304@redhat.com> <48CEAA72.3050405@redhat.com> <48CFA71C.10600@redhat.com> Message-ID: <48CFC6A0.9070901@redhat.com> Bryan Kearney wrote: > David Huff wrote: >> Bryan Kearney wrote: >>> David Huff wrote: >>>> Bryan Kearney wrote: >> >> I am seeing this on both a fully updated F10 & F9 box building the >> kickstart file from the ks pool. The only difference i can think of >> is that I am running 64bit OS on both testboxes. Which may pull in >> more packages? >> >> Once I increase the partition size and get a appliance up there are >> 167 packages installed. and it uses 484M of disk space. >> >> -D > Can you try the latest in the mainline of the kickstarts repo? I pushed > it to 550. If you can build with that one, I will ask kanarip to pull it > in. > Testing with that new kickstart seems to work wit no problems. This brings up the topic we had discussed in the past about 32 vs. 64 bit white listing that we may need to revisit in the future. -D From bkearney at redhat.com Tue Sep 16 14:49:39 2008 From: bkearney at redhat.com (Bryan Kearney) Date: Tue, 16 Sep 2008 10:49:39 -0400 Subject: [Thincrust-devel] [Fwd: [Fedora-spins] Fedora 10 Beta Freeze (Spins)] In-Reply-To: <48CFC6A0.9070901@redhat.com> References: <48CE575D.9010507@redhat.com> <48CE6615.5080504@redhat.com> <48CE7E11.6090304@redhat.com> <48CEAA72.3050405@redhat.com> <48CFA71C.10600@redhat.com> <48CFC6A0.9070901@redhat.com> Message-ID: <48CFC783.6000409@redhat.com> David Huff wrote: > Bryan Kearney wrote: >> David Huff wrote: >>> Bryan Kearney wrote: >>>> David Huff wrote: >>>>> Bryan Kearney wrote: >>> >>> I am seeing this on both a fully updated F10 & F9 box building the >>> kickstart file from the ks pool. The only difference i can think of >>> is that I am running 64bit OS on both testboxes. Which may pull in >>> more packages? >>> >>> Once I increase the partition size and get a appliance up there are >>> 167 packages installed. and it uses 484M of disk space. >>> >>> -D >> Can you try the latest in the mainline of the kickstarts repo? I >> pushed it to 550. If you can build with that one, I will ask kanarip >> to pull it in. >> > > > Testing with that new kickstart seems to work wit no problems. This > brings up the topic we had discussed in the past about 32 vs. 64 bit > white listing that we may need to revisit in the future. Agreed... for now this gets us the extra space, and qcow can be used to slim it down for 32bit. -- bk From jboggs at redhat.com Wed Sep 17 17:50:53 2008 From: jboggs at redhat.com (Joey Boggs) Date: Wed, 17 Sep 2008 13:50:53 -0400 Subject: [Thincrust-devel] [patch] act - ec2-converter pod page Message-ID: <48D1437D.7070904@redhat.com> updated makefile as well -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-added-ec2-converter.pod.patch Type: text/x-patch Size: 3478 bytes Desc: not available URL: From jboggs at redhat.com Wed Sep 17 18:00:34 2008 From: jboggs at redhat.com (Joey Boggs) Date: Wed, 17 Sep 2008 14:00:34 -0400 Subject: [Thincrust-devel] [patch] act - ec2-converter pod page In-Reply-To: <48D1437D.7070904@redhat.com> References: <48D1437D.7070904@redhat.com> Message-ID: <48D145C2.5050904@redhat.com> typo in filename, fixed new attachment Joey Boggs wrote: > updated makefile as well > ------------------------------------------------------------------------ > > _______________________________________________ > Thincrust-devel mailing list > Thincrust-devel at redhat.com > https://www.redhat.com/mailman/listinfo/thincrust-devel -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-added-ec2-converter.pod.patch Type: text/x-patch Size: 3488 bytes Desc: not available URL: From jboggs at redhat.com Wed Sep 17 21:24:10 2008 From: jboggs at redhat.com (Joey Boggs) Date: Wed, 17 Sep 2008 17:24:10 -0400 Subject: [Thincrust-devel] appliance-tools-003-6.fc9.testing.noarch.rpm error Message-ID: <48D1757A.6040004@redhat.com> Think we hit a 2GB filesize limit using zip format for appliance-creator. Other than that all else seems to work. One last thing, the man page is missing the package option File "/usr/bin/appliance-creator", line 122, in sys.exit(main()) File "/usr/bin/appliance-creator", line 112, in main creator.package() File "/usr/lib/python2.5/site-packages/imgcreate/creator.py", line 747, in package self._stage_final_image() File "/usr/lib/python2.5/site-packages/appcreate/appliance.py", line 368, in _stage_final_image self._package_image() File "/usr/lib/python2.5/site-packages/appcreate/appliance.py", line 347, in _package_image z.write(file, arcname=os.path.basename(file), compress_type=None) File "/usr/lib/python2.5/zipfile.py", line 561, in write self._writecheck(zinfo) File "/usr/lib/python2.5/zipfile.py", line 533, in _writecheck raise LargeZipFile("Filesize would require ZIP64 extensions") zipfile.LargeZipFile: Filesize would require ZIP64 extensions From dhuff at redhat.com Wed Sep 17 21:32:13 2008 From: dhuff at redhat.com (David Huff) Date: Wed, 17 Sep 2008 17:32:13 -0400 Subject: [Thincrust-devel] appliance-tools-003-6.fc9.testing.noarch.rpm error In-Reply-To: <48D1757A.6040004@redhat.com> References: <48D1757A.6040004@redhat.com> Message-ID: <48D1775D.6050203@redhat.com> Thanks for taking a look... Joey Boggs wrote: > Think we hit a 2GB filesize limit using zip format for > appliance-creator. Other than that all else seems to work. humm do we need to address this or do we think that most people will not need files above 2gig? > zipfile.LargeZipFile: Filesize would require ZIP64 extensions looks like I can add allowZip64=True when the zip file is created however not %100 this would fix the issue and I don't know if there are any drawbacks. Joey can you add allowZip64="True" to line 343 after compression=8 in appliance.py and see if that fixes the issue? > the man page is missing the package option Fixed man page problem. From dhuff at redhat.com Thu Sep 18 15:02:02 2008 From: dhuff at redhat.com (David Huff) Date: Thu, 18 Sep 2008 11:02:02 -0400 Subject: [Thincrust-devel] new appliance-tools build Message-ID: <48D26D6A.8040304@redhat.com> We have added several new features to the appliance-tools RPM and ran a new build, appliance-tools-003-4.fc10 [1] This package should make it in to rawhide in the next couple of days however, I do not think it will make it in to F10-Beta since the freeze was last week. change log below: * Wed Sep 17 2008 David Huff - 003-4 - Removed all the kickstart files in the config dir to mirror livecd-tools - Added the image minimization to the refactored code (BKearney) - multiple interface issue (#460922) - added --format option to specity disk image format - added --package option to specify output, currently only .zip supported - added --vmem and --vcpu options - Merged ec2-converter code (jboggs) [1]http://koji.fedoraproject.org/koji/buildinfo?buildID=63450 From bkearney at redhat.com Thu Sep 18 17:29:35 2008 From: bkearney at redhat.com (Bryan Kearney) Date: Thu, 18 Sep 2008 13:29:35 -0400 Subject: [Thincrust-devel] new appliance-tools build In-Reply-To: <48D26D6A.8040304@redhat.com> References: <48D26D6A.8040304@redhat.com> Message-ID: <48D28FFF.5030007@redhat.com> David Huff wrote: > We have added several new features to the appliance-tools RPM and ran a > new build, appliance-tools-003-4.fc10 [1] Couple of things: 1) You do not validate the format tag until later in the process, but then you default to raw. Dont know if it should error out or now. 2) this command: appliance-creator --cache /root/cache --name fedora-aos --config fedora-aos.ks -f qcow2 --vmem 756 --vcpu 2 Gets me a vm with only 1 cpu defined. 3) Do you want to reference the livecd-tools in the man pages? -- bk From dhuff at redhat.com Thu Sep 18 18:39:44 2008 From: dhuff at redhat.com (David Huff) Date: Thu, 18 Sep 2008 14:39:44 -0400 Subject: [Thincrust-devel] new appliance-tools build In-Reply-To: <48D28FFF.5030007@redhat.com> References: <48D26D6A.8040304@redhat.com> <48D28FFF.5030007@redhat.com> Message-ID: <48D2A070.4090001@redhat.com> Bryan Kearney wrote: > David Huff wrote: >> We have added several new features to the appliance-tools RPM and ran >> a new build, appliance-tools-003-4.fc10 [1] > > > Couple of things: > > 1) You do not validate the format tag until later in the process, but > then you default to raw. Dont know if it should error out or now. The reason that I did it this way is b/c it calls qemu convert under the covers and a-c doesn't know or care what formats it supports. So if more formats are added to qemu convent, or we switch it to useing virt-convert (which we should probly do when virt-convert becomes more mature) there will be minimum code changes. Would be trivial to add extra logic however it seemed a better implementation to me. > 2) this command: > > appliance-creator --cache /root/cache --name fedora-aos --config > fedora-aos.ks -f qcow2 --vmem 756 --vcpu 2 try --vmem=int --vcpu=int i think b/c their integers you need the = sign. > > > Gets me a vm with only 1 cpu defined. > > > 3) Do you want to reference the livecd-tools in the man pages? I guess this would be a good idea since we extend it. However not sure if the users care or not. > > -- bk > > _______________________________________________ > Thincrust-devel mailing list > Thincrust-devel at redhat.com > https://www.redhat.com/mailman/listinfo/thincrust-devel From bkearney at redhat.com Thu Sep 18 21:05:03 2008 From: bkearney at redhat.com (Bryan Kearney) Date: Thu, 18 Sep 2008 17:05:03 -0400 Subject: [Thincrust-devel] new appliance-tools build In-Reply-To: <48D2A070.4090001@redhat.com> References: <48D26D6A.8040304@redhat.com> <48D28FFF.5030007@redhat.com> <48D2A070.4090001@redhat.com> Message-ID: <48D2C27F.6000001@redhat.com> David Huff wrote: > Bryan Kearney wrote: >> David Huff wrote: >>> We have added several new features to the appliance-tools RPM and ran >>> a new build, appliance-tools-003-4.fc10 [1] >> >> >> Couple of things: >> >> 1) You do not validate the format tag until later in the process, but >> then you default to raw. Dont know if it should error out or now. > > The reason that I did it this way is b/c it calls qemu convert under the > covers and a-c doesn't know or care what formats it supports. So if > more formats are added to qemu convent, or we switch it to useing > virt-convert (which we should probly do when virt-convert becomes more > mature) there will be minimum code changes. Would be trivial to add > extra logic however it seemed a better implementation to me. > >> 2) this command: >> >> appliance-creator --cache /root/cache --name fedora-aos --config >> fedora-aos.ks -f qcow2 --vmem 756 --vcpu 2 > > try --vmem=int --vcpu=int i think b/c their integers you need the = sign. I will need to dig into htis more.. but is looks like the xml is set up correctly, but the running machine in virt-manager has only one cpu. I tried changing this from virt-manager and could not. -- bk From bkearney at redhat.com Mon Sep 22 13:17:54 2008 From: bkearney at redhat.com (Bryan Kearney) Date: Mon, 22 Sep 2008 09:17:54 -0400 Subject: [Thincrust-devel] appliance-tools-003-6.fc9.testing.noarch.rpm error In-Reply-To: <48D1775D.6050203@redhat.com> References: <48D1757A.6040004@redhat.com> <48D1775D.6050203@redhat.com> Message-ID: <48D79B02.4020107@redhat.com> David Huff wrote: > Thanks for taking a look... > > Joey Boggs wrote: >> Think we hit a 2GB filesize limit using zip format for >> appliance-creator. Other than that all else seems to work. > > humm do we need to address this or do we think that most people will not > need files above 2gig? I think we need to adress this. -- bk From jboggs at redhat.com Mon Sep 22 13:28:02 2008 From: jboggs at redhat.com (Joey Boggs) Date: Mon, 22 Sep 2008 09:28:02 -0400 Subject: [Thincrust-devel] appliance-tools-003-6.fc9.testing.noarch.rpm error In-Reply-To: <48D79B02.4020107@redhat.com> References: <48D1757A.6040004@redhat.com> <48D1775D.6050203@redhat.com> <48D79B02.4020107@redhat.com> Message-ID: <48D79D62.1000207@redhat.com> It's been fixed as of Friday, but needs to be pushed into koji afaik Bryan Kearney wrote: > David Huff wrote: >> Thanks for taking a look... >> >> Joey Boggs wrote: >>> Think we hit a 2GB filesize limit using zip format for >>> appliance-creator. Other than that all else seems to work. >> >> humm do we need to address this or do we think that most people will >> not need files above 2gig? > > I think we need to adress this. > > -- bk > > > _______________________________________________ > Thincrust-devel mailing list > Thincrust-devel at redhat.com > https://www.redhat.com/mailman/listinfo/thincrust-devel From dhuff at redhat.com Mon Sep 22 15:14:49 2008 From: dhuff at redhat.com (David Huff) Date: Mon, 22 Sep 2008 11:14:49 -0400 Subject: [Thincrust-devel] appliance-tools-003-6.fc9.testing.noarch.rpm error In-Reply-To: <48D79D62.1000207@redhat.com> References: <48D1757A.6040004@redhat.com> <48D1775D.6050203@redhat.com> <48D79B02.4020107@redhat.com> <48D79D62.1000207@redhat.com> Message-ID: <48D7B669.2000003@redhat.com> Joey Boggs wrote: > It's been fixed as of Friday, but needs to be pushed into koji afaik > > Bryan Kearney wrote: >> David Huff wrote: >>> Thanks for taking a look... >>> >>> Joey Boggs wrote: >>>> Think we hit a 2GB filesize limit using zip format for >>>> appliance-creator. Other than that all else seems to work. >>> >>> humm do we need to address this or do we think that most people will >>> not need files above 2gig? >> >> I think we need to adress this. >> >> -- bk I'll make sure we have a bugzilla to track this bug and commit the patch upstream. The next build will have this fix in it however there is no need to do a new build just for this bug. I predict we will do another build closeer to the F10 GA date. In case we discover any addition bugs between now and then. -D From dhuff at redhat.com Thu Sep 25 19:11:46 2008 From: dhuff at redhat.com (David Huff) Date: Thu, 25 Sep 2008 15:11:46 -0400 Subject: [Thincrust-devel] [Fwd: [Fedora Update] [new] appliance-tools-002-3.fc9] Message-ID: <48DBE272.6000401@redhat.com> -------- Original Message -------- Return-Path: updates at fedoraproject.org Received: from zmta01.collab.prod.int.phx2.redhat.com (LHLO zmta01.collab.prod.int.phx2.redhat.com) (10.5.5.31) by mail02.corp.redhat.com with LMTP; Thu, 25 Sep 2008 15:11:25 -0400 (EDT) Received: from localhost (localhost.localdomain [127.0.0.1]) by zmta01.collab.prod.int.phx2.redhat.com (Postfix) with ESMTP id 9702790A66 for ; Thu, 25 Sep 2008 15:11:25 -0400 (EDT) Received: from zmta01.collab.prod.int.phx2.redhat.com ([127.0.0.1]) by localhost (zmta01.collab.prod.int.phx2.redhat.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id dXo9cVSdlOHR for ; Thu, 25 Sep 2008 15:11:25 -0400 (EDT) Received: from int-mx1.corp.redhat.com (int-mx1.corp.redhat.com [172.16.52.254]) by zmta01.collab.prod.int.phx2.redhat.com (Postfix) with ESMTP id 4B80B90A5B for ; Thu, 25 Sep 2008 15:11:25 -0400 (EDT) Received: from mx2.util.phx.redhat.com (mx2.util.phx.redhat.com [10.8.4.102]) by int-mx1.corp.redhat.com (8.13.1/8.13.1) with ESMTP id m8PJBNc8032709 for ; Thu, 25 Sep 2008 15:11:23 -0400 Received: from bastion.fedora.phx.redhat.com (bastion.fedora.phx.redhat.com [10.8.34.50]) by mx2.util.phx.redhat.com (8.13.1/8.13.1) with ESMTP id m8PJBNjf013050 for ; Thu, 25 Sep 2008 15:11:23 -0400 Received: by bastion.fedora.phx.redhat.com (Postfix) id 8D551208D1B; Thu, 25 Sep 2008 19:11:23 +0000 (UTC) Delivered-To: huff at fedoraproject.org Received: by bastion.fedora.phx.redhat.com (Postfix, from userid 404) id 87296208DA2; Thu, 25 Sep 2008 19:11:23 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.2.4 (2008-01-01) on bastion.fedora.phx.redhat.com X-Spam-Level: X-Spam-Status: No, score=-1.4 required=5.0 tests=ALL_TRUSTED,AWL,TO_MALFORMED autolearn=no version=3.2.4 Received: from app2.fedora.phx.redhat.com (app2.fedora.phx.redhat.com [10.8.34.60]) by bastion.fedora.phx.redhat.com (Postfix) with ESMTP id 42902208D1B for ; Thu, 25 Sep 2008 19:11:09 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable From: updates at fedoraproject.org Subject: [Fedora Update] [new] appliance-tools-002-3.fc9 Date: Thu, 25 Sep 2008 19:11:09 +0000 To: huff at fedoraproject.org Message-Id: <20080925191109.42902208D1B at bastion.fedora.phx.redhat.com> X-Scanned-By: MIMEDefang 2.58 on 172.16.52.254 huff has submitted a new update for Fedora 9 ================================================================================ appliance-tools-002-3.fc9 ================================================================================ Release: Fedora 9 Status: pending Type: newpackage Karma: 0 Notes: F9 build of appliance-tools Submitter: huff Submitted: 2008-09-25 19:11:09 http://admin.fedoraproject.org/updates/appliance-tools-002-3.fc9 From dhuff at redhat.com Fri Sep 26 18:52:25 2008 From: dhuff at redhat.com (David Huff) Date: Fri, 26 Sep 2008 14:52:25 -0400 Subject: [Thincrust-devel] [Fwd: appliance-tools-002-3.fc9 successfully moved from dist-f9-updates-candidate into dist-f9-updates by bodhi] Message-ID: <48DD2F69.2030504@redhat.com> An embedded message was scrubbed... From: Koji Build System Subject: appliance-tools-002-3.fc9 successfully moved from dist-f9-updates-candidate into dist-f9-updates by bodhi Date: Fri, 26 Sep 2008 18:49:41 +0000 (UTC) Size: 3015 URL: From bkearney at redhat.com Fri Sep 26 19:06:33 2008 From: bkearney at redhat.com (Bryan Kearney) Date: Fri, 26 Sep 2008 15:06:33 -0400 Subject: [Thincrust-devel] RFC: Customizing an appliance install Message-ID: <48DD32B9.9030605@redhat.com> I am looking for feedback on how to customize an appliance install. As you may be aware, the appliance configuration engine uses puppet as it's configuration syntax. The rational is that this is an up and coming standard syntax, and should be familiar to those who do sysadmin type work (Said another way, we don't need yet another DSL). The trick is how to allow a user (either via the gui or the command line) to customize an appliance recipe. In puppet I have seen 2 ways to push in custom information: 1) External Nodes 2) Facter I am thinking of using the latter, and would like to outline my thoughts. Please feel free to offer up alternative solutions. Goal: Create recipes which can externalize a small set of data which the user needs to provide in order to the appliance to be configured. Implementation A module, or a recipe can deliver one or more "external facts". They will be delivered in the form: name:default_value:query_text before the puppet engine is run, the thincrust tool can iterate over these facts, present them to the user, and record the answer. This can be stored in a file on the appliance. In essence, this is a pre-processor. When puppet is run, a single custom fact will iterate over all the external facts.. and add them to facter. This will allow them to be accessed by all the recipes, modules, and templates. Thoughts? -- bk From bkearney at redhat.com Sat Sep 27 00:26:54 2008 From: bkearney at redhat.com (Bryan Kearney) Date: Fri, 26 Sep 2008 20:26:54 -0400 Subject: [Thincrust-devel] [Fwd: appliance-tools-002-3.fc9 successfully moved from dist-f9-updates-candidate into dist-f9-updates by bodhi] In-Reply-To: <48DD2F69.2030504@redhat.com> References: <48DD2F69.2030504@redhat.com> Message-ID: <48DD7DCE.6050809@redhat.com> David Huff wrote: > 9-updates > > appliance-tools-002-3.fc9 successfully moved from dist-f9-updates-candidate into dist-f9-updates by bodhi great news.. thanks! -- bk From bleanhar at redhat.com Mon Sep 29 12:46:13 2008 From: bleanhar at redhat.com (Brenton Leanhardt) Date: Mon, 29 Sep 2008 08:46:13 -0400 Subject: [Thincrust-devel] Re: RFC: Customizing an appliance install In-Reply-To: <48DD32B9.9030605@redhat.com> References: <48DD32B9.9030605@redhat.com> Message-ID: <20080929124613.GA9573@tumbolia-jboss-dev.usersys.redhat.com> +++ Bryan Kearney [26/09/08 15:06 -0400]: > I am looking for feedback on how to customize an appliance install. As you > may be aware, the appliance configuration engine uses puppet as it's > configuration syntax. The rational is that this is an up and coming > standard syntax, and should be familiar to those who do sysadmin type work > (Said another way, we don't need yet another DSL). > > The trick is how to allow a user (either via the gui or the command line) > to customize an appliance recipe. In puppet I have seen 2 ways to push in > custom information: > > 1) External Nodes > 2) Facter > > I am thinking of using the latter, and would like to outline my thoughts. > Please feel free to offer up alternative solutions. > > Goal: > > Create recipes which can externalize a small set of data which the user > needs to provide in order to the appliance to be configured. > > Implementation > > A module, or a recipe can deliver one or more "external facts". They will > be delivered in the form: > > name:default_value:query_text This seems good to me. In fact similar to how Genome's "machine type" DSL generates parameters for external nodes. One thing I've thought about, that I think could be a better approach, is storing this sort of information in a special syntax (probably embedded in comments JavaDoc style) so that this information would ship with the module as opposed to keeping it in a separate file. When tooling like Thin Crust needs access to this sort of information there could be a Puppet tool that provides a way to collect it (ie, a way that understands how Puppet loads and finds modules). Genome has had a hard time keeping a config file that represents "name:default_value:query_text" type information in sync with a large group of Puppet modules. Especially when 15+ people have their hands in the modules. > > before the puppet engine is run, the thincrust tool can iterate over these > facts, present them to the user, and record the answer. This can be stored > in a file on the appliance. In essence, this is a pre-processor. > > When puppet is run, a single custom fact will iterate over all the external > facts.. and add them to facter. This will allow them to be accessed by all > the recipes, modules, and templates. > > Thoughts? > > -- bk > > From bkearney at redhat.com Mon Sep 29 13:06:47 2008 From: bkearney at redhat.com (Bryan Kearney) Date: Mon, 29 Sep 2008 09:06:47 -0400 Subject: [Thincrust-devel] Re: RFC: Customizing an appliance install In-Reply-To: <20080929124613.GA9573@tumbolia-jboss-dev.usersys.redhat.com> References: <48DD32B9.9030605@redhat.com> <20080929124613.GA9573@tumbolia-jboss-dev.usersys.redhat.com> Message-ID: <48E0D2E7.3030703@redhat.com> Brenton Leanhardt wrote: > +++ Bryan Kearney [26/09/08 15:06 -0400]: >> I am looking for feedback on how to customize an appliance install. As >> you may be aware, the appliance configuration engine uses puppet as >> it's configuration syntax. The rational is that this is an up and >> coming standard syntax, and should be familiar to those who do >> sysadmin type work (Said another way, we don't need yet another DSL). >> >> The trick is how to allow a user (either via the gui or the command >> line) to customize an appliance recipe. In puppet I have seen 2 ways >> to push in custom information: >> >> 1) External Nodes >> 2) Facter >> >> I am thinking of using the latter, and would like to outline my >> thoughts. Please feel free to offer up alternative solutions. >> >> Goal: >> >> Create recipes which can externalize a small set of data which the >> user needs to provide in order to the appliance to be configured. >> >> Implementation >> >> A module, or a recipe can deliver one or more "external facts". They >> will be delivered in the form: >> >> name:default_value:query_text > > This seems good to me. In fact similar to how Genome's "machine type" > DSL generates parameters for external nodes. One thing I've thought > about, that I think could be a better approach, is storing this sort > of information in a special syntax (probably embedded in comments > JavaDoc style) so that this information would ship with the module as > opposed to keeping it in a separate file. Yeah.. I think this is a good way to go. Keeping the number of files to a minimum would be good. Do you have a syntax chart for the genome DSL? > When tooling like Thin Crust needs access to this sort of information > there could be a Puppet tool that provides a way to collect it (ie, a > way that understands how Puppet loads and finds modules). Genome has > had a hard time keeping a config file that represents > "name:default_value:query_text" type information in sync with a large > group of Puppet modules. Especially when 15+ people have their hands > in the modules. That was what I was going to look to facter too. I do not believe this feature would be embraced by the puppet project in general.. so the easiest way to glom on would be the best. -- bk From bleanhar at redhat.com Mon Sep 29 13:31:22 2008 From: bleanhar at redhat.com (Brenton Leanhardt) Date: Mon, 29 Sep 2008 09:31:22 -0400 Subject: [Thincrust-devel] Re: RFC: Customizing an appliance install In-Reply-To: <48E0D2E7.3030703@redhat.com> References: <48DD32B9.9030605@redhat.com> <20080929124613.GA9573@tumbolia-jboss-dev.usersys.redhat.com> <48E0D2E7.3030703@redhat.com> Message-ID: <20080929133122.GC9573@tumbolia-jboss-dev.usersys.redhat.com> +++ Bryan Kearney [29/09/08 09:06 -0400]: > Brenton Leanhardt wrote: >> +++ Bryan Kearney [26/09/08 15:06 -0400]: >>> I am looking for feedback on how to customize an appliance install. As >>> you may be aware, the appliance configuration engine uses puppet as it's >>> configuration syntax. The rational is that this is an up and coming >>> standard syntax, and should be familiar to those who do sysadmin type >>> work (Said another way, we don't need yet another DSL). >>> >>> The trick is how to allow a user (either via the gui or the command line) >>> to customize an appliance recipe. In puppet I have seen 2 ways to push in >>> custom information: >>> >>> 1) External Nodes >>> 2) Facter >>> >>> I am thinking of using the latter, and would like to outline my thoughts. >>> Please feel free to offer up alternative solutions. >>> >>> Goal: >>> >>> Create recipes which can externalize a small set of data which the user >>> needs to provide in order to the appliance to be configured. >>> >>> Implementation >>> >>> A module, or a recipe can deliver one or more "external facts". They will >>> be delivered in the form: >>> >>> name:default_value:query_text >> >> This seems good to me. In fact similar to how Genome's "machine type" >> DSL generates parameters for external nodes. One thing I've thought >> about, that I think could be a better approach, is storing this sort >> of information in a special syntax (probably embedded in comments >> JavaDoc style) so that this information would ship with the module as >> opposed to keeping it in a separate file. > > Yeah.. I think this is a good way to go. Keeping the number of files to a > minimum would be good. Do you have a syntax chart for the genome DSL? I've attached a sample config file. You can see the code here: http://git.fedorahosted.org/git/genome.git?p=genome.git;a=tree;f=tools/genome-dsl;h=63b152ed2fca87bf47ebcae873c6ffe56fa87d96;hb=HEAD > > >> When tooling like Thin Crust needs access to this sort of information >> there could be a Puppet tool that provides a way to collect it (ie, a >> way that understands how Puppet loads and finds modules). Genome has >> had a hard time keeping a config file that represents >> "name:default_value:query_text" type information in sync with a large >> group of Puppet modules. Especially when 15+ people have their hands >> in the modules. > > That was what I was going to look to facter too. I do not believe this > feature would be embraced by the puppet project in general.. so the easiest > way to glom on would be the best. > > -- bk > -------------- next part -------------- # Copyright (C) 2008 Red Hat, Inc # This program is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License # as published by the Free Software Foundation; either version 2 # of the License, or (at your option) any later version. # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # You should have received a copy of the GNU General Public License # a long with this program; if not, write to the Free Software # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. ################################################################################ #Define new facts here. You can later map them to machine types in the # #following section. # # # # One thing to note is the special syntax allowed when setting default values # # for facts. In the default string you are allowed to use "%repo%" or # # "%machine_name%" which will get substituted appropriately whenever this data # # is used (eg, genome-bootstrap uses this for setting context aware default # # values for facts). # # # # If a fact needs to be on all machines types set ":on_all_nodes" # # appropriately. Otherwise in the machine declaration section you need to # # specify what facts will be used for a particular machine. # ################################################################################ # # A couple of example facts. # # newfact("logserver", :on_all_nodes => true) do # set_desc "The machine to send syslog output to" # set_default "%repo%-repo.usersys.redhat.com" # end # # newfact("multiline_description") do # desc = < References: <48DD32B9.9030605@redhat.com> <20080929124613.GA9573@tumbolia-jboss-dev.usersys.redhat.com> <48E0D2E7.3030703@redhat.com> <20080929133122.GC9573@tumbolia-jboss-dev.usersys.redhat.com> Message-ID: <48E0E8F7.6030800@redhat.com> Brenton Leanhardt wrote: > +++ Bryan Kearney [29/09/08 09:06 -0400]: >> Brenton Leanhardt wrote: >>> +++ Bryan Kearney [26/09/08 15:06 -0400]: >>>> I am looking for feedback on how to customize an appliance install. >>>> As you may be aware, the appliance configuration engine uses puppet >>>> as it's configuration syntax. The rational is that this is an up and >>>> coming standard syntax, and should be familiar to those who do >>>> sysadmin type work (Said another way, we don't need yet another DSL). >>>> >>>> The trick is how to allow a user (either via the gui or the command >>>> line) to customize an appliance recipe. In puppet I have seen 2 ways >>>> to push in custom information: >>>> >>>> 1) External Nodes >>>> 2) Facter >>>> >>>> I am thinking of using the latter, and would like to outline my >>>> thoughts. Please feel free to offer up alternative solutions. >>>> >>>> Goal: >>>> >>>> Create recipes which can externalize a small set of data which the >>>> user needs to provide in order to the appliance to be configured. >>>> >>>> Implementation >>>> >>>> A module, or a recipe can deliver one or more "external facts". They >>>> will be delivered in the form: >>>> >>>> name:default_value:query_text >>> >>> This seems good to me. In fact similar to how Genome's "machine type" >>> DSL generates parameters for external nodes. One thing I've thought >>> about, that I think could be a better approach, is storing this sort >>> of information in a special syntax (probably embedded in comments >>> JavaDoc style) so that this information would ship with the module as >>> opposed to keeping it in a separate file. >> >> Yeah.. I think this is a good way to go. Keeping the number of files >> to a minimum would be good. Do you have a syntax chart for the genome >> DSL? > > I've attached a sample config file. You can see the code here: > http://git.fedorahosted.org/git/genome.git?p=genome.git;a=tree;f=tools/genome-dsl;h=63b152ed2fca87bf47ebcae873c6ffe56fa87d96;hb=HEAD What do you think of this. Following the rdoc convention # :default_value Foo # :prompt What should the value of Foo be? $variable = value_from_user() This would result in a custom function which is executed at runtime to pull in the value (not polluting facter) and would feel somewhat rubyish. -- bk From bleanhar at redhat.com Mon Sep 29 15:10:22 2008 From: bleanhar at redhat.com (Brenton Leanhardt) Date: Mon, 29 Sep 2008 11:10:22 -0400 Subject: [Thincrust-devel] Re: RFC: Customizing an appliance install In-Reply-To: <48E0E8F7.6030800@redhat.com> References: <48DD32B9.9030605@redhat.com> <20080929124613.GA9573@tumbolia-jboss-dev.usersys.redhat.com> <48E0D2E7.3030703@redhat.com> <20080929133122.GC9573@tumbolia-jboss-dev.usersys.redhat.com> <48E0E8F7.6030800@redhat.com> Message-ID: <20080929151022.GG9573@tumbolia-jboss-dev.usersys.redhat.com> +++ Bryan Kearney [29/09/08 10:40 -0400]: > Brenton Leanhardt wrote: >> +++ Bryan Kearney [29/09/08 09:06 -0400]: >>> Brenton Leanhardt wrote: >>>> +++ Bryan Kearney [26/09/08 15:06 -0400]: >>>>> I am looking for feedback on how to customize an appliance install. As >>>>> you may be aware, the appliance configuration engine uses puppet as >>>>> it's configuration syntax. The rational is that this is an up and >>>>> coming standard syntax, and should be familiar to those who do sysadmin >>>>> type work (Said another way, we don't need yet another DSL). >>>>> >>>>> The trick is how to allow a user (either via the gui or the command >>>>> line) to customize an appliance recipe. In puppet I have seen 2 ways to >>>>> push in custom information: >>>>> >>>>> 1) External Nodes >>>>> 2) Facter >>>>> >>>>> I am thinking of using the latter, and would like to outline my >>>>> thoughts. Please feel free to offer up alternative solutions. >>>>> >>>>> Goal: >>>>> >>>>> Create recipes which can externalize a small set of data which the user >>>>> needs to provide in order to the appliance to be configured. >>>>> >>>>> Implementation >>>>> >>>>> A module, or a recipe can deliver one or more "external facts". They >>>>> will be delivered in the form: >>>>> >>>>> name:default_value:query_text >>>> >>>> This seems good to me. In fact similar to how Genome's "machine type" >>>> DSL generates parameters for external nodes. One thing I've thought >>>> about, that I think could be a better approach, is storing this sort >>>> of information in a special syntax (probably embedded in comments >>>> JavaDoc style) so that this information would ship with the module as >>>> opposed to keeping it in a separate file. >>> >>> Yeah.. I think this is a good way to go. Keeping the number of files to a >>> minimum would be good. Do you have a syntax chart for the genome DSL? >> >> I've attached a sample config file. You can see the code here: >> http://git.fedorahosted.org/git/genome.git?p=genome.git;a=tree;f=tools/genome-dsl;h=63b152ed2fca87bf47ebcae873c6ffe56fa87d96;hb=HEAD > > > What do you think of this. Following the rdoc convention > > # :default_value Foo > # :prompt What should the value of Foo be? > $variable = value_from_user() > > This would result in a custom function which is executed at runtime to pull > in the value (not polluting facter) and would feel somewhat rubyish. > > -- bk > Did I say JavaDoc? I meant rdoc. :) Yeah. I think I'm following you. The custom function would prompt the user and then store the answer somewhere to keep from asking the same questions again. I'm thinking from the Genome perspective here. I suspect we use Puppet slightly more iteratively than Thin Crust so they ability to keep from asking the same questions every time Puppet runs is nice. It's also nice to be able to figure out all the questions that are going to be asked ahead of time so that the user doesn't have to watch Puppet run. I could see that being a little harder to implement outside of Puppet though. From pmyers at redhat.com Tue Sep 30 22:15:26 2008 From: pmyers at redhat.com (Perry N. Myers) Date: Tue, 30 Sep 2008 18:15:26 -0400 Subject: [Thincrust-devel] Errors w/ qcow output from appliance-tools Message-ID: <48E2A4FE.5090801@redhat.com> David, Here's some info on the problem that I've been having. I used the attached kickstart (foo.ks) and ran: appliance-creator --config foo.ks --name foo -f qcow -d -v The output includes the following: > writing image XML to /var/tmp/imgcreate-223Oxs/tmp-N65fLE/foo.xml > converting /var/tmp/imgcreate-223Oxs/tmp-N65fLE/foo-sda.raw image to /var/tmp/imgcreate-223Oxs/tmp-N65fLE/foo-sda.qcow > qemu-img: Could not open '/var/tmp/imgcreate-223Oxs/tmp-N65fLE/foo-sda.qcow' > Unable to convert disk format to qcow, using raw disk image > moving disks to final location > moving /var/tmp/imgcreate-223Oxs/tmp-N65fLE/foo.xml to /var/tmp/imgcreate-223Oxs/out > moving /var/tmp/imgcreate-223Oxs/tmp-N65fLE/foo-sda.raw to /var/tmp/imgcreate-223Oxs/out > moving /var/tmp/imgcreate-223Oxs/tmp-N65fLE/foo-sda.qcow to /var/tmp/imgcreate-223Oxs/out > done Full output is on: http://ovirt.pastebin.com/d1d01f7cb foo.ks is very basic. Just the min stuff needed to create an appliance really using base-pkgs. These are the versions I am using: appliance-tools-002-3.fc9.noarch appliance-os-001-3.fc9.noarch I've tried this on multiple Fedora 9 hosts and had the same results. Let me know if your investigation using this kickstart/commandline reveals anything. Perry -- |=- Red Hat, Engineering, Emerging Technologies, Boston -=| |=- Email: pmyers at redhat.com -=| |=- Office: +1 412 474 3552 Mobile: +1 703 362 9622 -=| |=- GnuPG: E65E4F3D 88F9 F1C9 C2F3 1303 01FE 817C C5D2 8B91 E65E 4F3D -=| -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: foo.ks URL: