From dimitris at glezos.com Fri Dec 1 00:12:56 2006 From: dimitris at glezos.com (Dimitris Glezos) Date: Fri, 01 Dec 2006 00:12:56 +0000 Subject: L10N tools: Increase quality of translations Message-ID: <456F7388.1040005@glezos.com> Hi all. Some members from the L10N project have identified some issues that need to be solved to make sure the translations of Fedora are of high quality. Some of them are infrastructure-related and at today's (-admin) meeting it was suggested to transfer the discussion here. I'm sure the folks at Red Hat are doing their best to keep the quality of the translations high. But the truth is that Fedora's image in the context of translations is not good; we do hear that a lot, from current and wanna-be members. We can and should work to improve it. Semi-translated applications are U-G-L-Y and shout "low QA" in our ears. Some possible ideas have been listed on the following page: http://fedoraproject.org/wiki/L10N/Tasks Please feel free to chip in and help us out correct whatever doesn't seem reasonable, break tasks into smaller tasks etc. The bigger picture of some of these problems are: * L10N of "local" applications (those listed on [1]) is poor; releases and package updates contain untranslated strings in many languages. This is unacceptable for a fully-localized desktop. * The barrier to contribute to the l10n (be it GUI or Docs) is higher than it should be (compared to other projects). * QA of the translations is difficult with the current tools. Some more concrete ideas to discuss that might concern the InfrProj and the RelEng team are: * Integrate better the handling of translation during a "local" package's lifecycle. Have a flag raised for a package update that introduces new strings so that translators can translate the new strings before the repackaging/updating. Include in the schedule for each release a "string freeze date" and a week later a "translation freeze date" and have all our packages rebuilt after the latter and before the actual release. * Move po files on their own cvsroot on cvs.fedoraproject.org to reduce complexity and maintenance and to increase security (with a new group). * Move the i18n status pages to Fedora servers (Plone/Turbogears?). Include a direct link to the pofiles from there so that new members can have something to work on before getting cvs access. * In the future, use Plone to automate the QA between team members (ie coordinators can review translations etc). * Start working with the complex and tricky path to upstream translations that no distribution has tackled yet in a successful way. Bring our translators closer to the upstream projects. Hope some of the above make sense. :) -d [1]: http://i18n.redhat.com/cgi-bin/i18n-status -- Dimitris Glezos Jabber ID: glezos at jabber.org, GPG: 0xA5A04C3B http://dimitris.glezos.com/ "He who gives up functionality for ease of use loses both and deserves neither." (Anonymous) -- From notting at redhat.com Fri Dec 1 01:41:37 2006 From: notting at redhat.com (Bill Nottingham) Date: Thu, 30 Nov 2006 20:41:37 -0500 Subject: L10N tools: Increase quality of translations In-Reply-To: <456F7388.1040005@glezos.com> References: <456F7388.1040005@glezos.com> Message-ID: <20061201014137.GA4556@nostromo.devel.redhat.com> Dimitris Glezos (dimitris at glezos.com) said: > * Integrate better the handling of translation during a "local" package's > lifecycle. Have a flag raised for a package update that introduces new strings > so that translators can translate the new strings before the > repackaging/updating. Include in the schedule for each release a "string freeze > date" and a week later a "translation freeze date" and have all our packages > rebuilt after the latter and before the actual release. String freezes are easy to do, it's just a matter of discipline in sticking to them. > * Move po files on their own cvsroot on cvs.fedoraproject.org to reduce > complexity and maintenance and to increase security (with a new group). >From a maintainer standpoint, the translations *need* to be in the same SCM system as the code. Without that, you lose features like branching, easy spinning of tarballs, etc. Moving just the po files is not the answer. > * Start working with the complex and tricky path to upstream translations that > no distribution has tackled yet in a successful way. Bring our translators > closer to the upstream projects. Well, the simple answer is if you want to translate upstream, go upstream. For desktop environments, such as GNOME or KDE, this is fairly easy. For other projects, it is more complicated. Bill From linux at elfshadow.net Sun Dec 3 02:02:57 2006 From: linux at elfshadow.net (Jeffrey Tadlock) Date: Sat, 02 Dec 2006 21:02:57 -0500 Subject: IRC Meeting Log - 11/30/2006 Message-ID: <45723051.2070103@elfshadow.net> Sorry, I am a little late with this one. The IRC log from Thursday's meeting has been posted here: http://fedoraproject.org/wiki/Infrastructure/Meetings/2006-11-30 --Jeffrey From linux at elfshadow.net Sun Dec 3 02:15:54 2006 From: linux at elfshadow.net (Jeffrey Tadlock) Date: Sat, 02 Dec 2006 21:15:54 -0500 Subject: glump and config management In-Reply-To: <3237e4410611301240k4bf2affbu8d5a3ecf7742f071@mail.gmail.com> References: <3237e4410611091956l3cecc19eq22b695d4c8466eda@mail.gmail.com> <3237e4410611301240k4bf2affbu8d5a3ecf7742f071@mail.gmail.com> Message-ID: <4572335A.8040502@elfshadow.net> Mike McGrath wrote: > On 11/9/06, Mike McGrath wrote: >> As many of you know we've been looking to make our configuration >> management system a bit more robust. Primarily by trying to find a >> technological solution to actually enforce our config management >> system. > > Anyone had a chance to actually look at this yet? I looked at it a week or two ago. I played with it a bit - mainly reading through the example files and running your test you setup. I like the backups it makes and I like the "enforcement" nature of it. I need to look through it a little more, but it does look like a possible way to handle out config management across a diverse group of people. --Jeffrey From paulo.banon at googlemail.com Mon Dec 4 16:01:15 2006 From: paulo.banon at googlemail.com (Paulo Santos) Date: Mon, 4 Dec 2006 16:01:15 +0000 Subject: Wiki Migration Message-ID: <7a41c4bc0612040801p268216f1pc9053292efb58819@mail.gmail.com> Hi everyone, As you know, after the release of FC6, we had problems with the amount of connections to our servers. This made the wiki unavailable as well as other related problems. With this concern, the Fedora Infrastructure Team, have been working in a way to prevent this from happening again. Our first step, is the wiki migration. We are now ready to test the future wiki platform, and this is why i'm sending this email. URL: http://webtest.fedora.redhat.com/wiki (MoinMoin 1.5.6) Please track all problems under http://fedoraproject.org/wiki/Infrastructure/WikiMigration, edit that page as you see fit, so that we can fix any problems that you guys may encounter. Feel free to contact me if necessary. Many thanks, Paulo -------------- next part -------------- An HTML attachment was scrubbed... URL: From julian_yap at yahoo.com Mon Dec 4 19:02:36 2006 From: julian_yap at yahoo.com (Julian Yap) Date: Mon, 4 Dec 2006 09:02:36 -1000 Subject: Wiki Migration In-Reply-To: <7a41c4bc0612040801p268216f1pc9053292efb58819@mail.gmail.com> References: <7a41c4bc0612040801p268216f1pc9053292efb58819@mail.gmail.com> Message-ID: Paulo, Apart from the change in MoinMoin version from 1.3.6 to 1.5.6, what are the changes implemented in the future wiki platform which should improve things? ~ Julian On 12/4/06, Paulo Santos wrote: > Hi everyone, > > As you know, after the release of FC6, we had problems with the amount of > connections to our servers. This made the wiki unavailable as well as other > related problems. > With this concern, the Fedora Infrastructure Team, have been working in a > way to prevent this from happening again. Our first step, is the wiki > migration. > We are now ready to test the future wiki platform, and this is why i'm > sending this email. > > URL: http://webtest.fedora.redhat.com/wiki (MoinMoin > 1.5.6) > > Please track all problems under > http://fedoraproject.org/wiki/Infrastructure/WikiMigration, > edit that page as you see fit, so that we can fix any problems that you guys > may encounter. > > Feel free to contact me if necessary. > > > Many thanks, > Paulo > _______________________________________________ > Fedora-infrastructure-list mailing list > Fedora-infrastructure-list at redhat.com > https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list > > > From email.ahmedkamal at googlemail.com Mon Dec 4 20:02:16 2006 From: email.ahmedkamal at googlemail.com (Ahmed Kamal) Date: Mon, 4 Dec 2006 22:02:16 +0200 Subject: Wiki Migration In-Reply-To: References: <7a41c4bc0612040801p268216f1pc9053292efb58819@mail.gmail.com> Message-ID: <3da3b5b40612041202m204cb961j6ccd9b13b743096c@mail.gmail.com> basically the older version was totally cache non-friendly (#moin). The newer version has many patches specifically to help caching servers. This should help us under heavy load. On 12/4/06, Julian Yap wrote: > > Paulo, > > Apart from the change in MoinMoin version from 1.3.6 to 1.5.6, what > are the changes implemented in the future wiki platform which should > improve things? > > ~ Julian > > On 12/4/06, Paulo Santos wrote: > > Hi everyone, > > > > As you know, after the release of FC6, we had problems with the amount > of > > connections to our servers. This made the wiki unavailable as well as > other > > related problems. > > With this concern, the Fedora Infrastructure Team, have been working in > a > > way to prevent this from happening again. Our first step, is the wiki > > migration. > > We are now ready to test the future wiki platform, and this is why i'm > > sending this email. > > > > URL: http://webtest.fedora.redhat.com/wiki (MoinMoin > > 1.5.6) > > > > Please track all problems under > > http://fedoraproject.org/wiki/Infrastructure/WikiMigration, > > edit that page as you see fit, so that we can fix any problems that you > guys > > may encounter. > > > > Feel free to contact me if necessary. > > > > > > Many thanks, > > Paulo > > _______________________________________________ > > Fedora-infrastructure-list mailing list > > Fedora-infrastructure-list at redhat.com > > https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list > > > > > > > > _______________________________________________ > Fedora-infrastructure-list mailing list > Fedora-infrastructure-list at redhat.com > https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mmcgrath at fedoraproject.org Mon Dec 4 21:42:24 2006 From: mmcgrath at fedoraproject.org (Mike McGrath) Date: Mon, 4 Dec 2006 15:42:24 -0600 Subject: Wiki Migration In-Reply-To: <3da3b5b40612041202m204cb961j6ccd9b13b743096c@mail.gmail.com> References: <7a41c4bc0612040801p268216f1pc9053292efb58819@mail.gmail.com> <3da3b5b40612041202m204cb961j6ccd9b13b743096c@mail.gmail.com> Message-ID: <3237e4410612041342r6626ec4lc6064769f9d4d2d1@mail.gmail.com> On 12/4/06, Ahmed Kamal wrote: > basically the older version was totally cache non-friendly (#moin). The > newer version has many patches specifically to help caching servers. This > should help us under heavy load. > It's also partly a stepping stone to the next version of moin that comes out. We'll have some of our SoC code implemented in with it and we just figured we'd take the time now to upgrade to something close so the upgrade to the new version is as painless as possible. -Mike From a.badger at gmail.com Tue Dec 5 05:59:11 2006 From: a.badger at gmail.com (Toshio Kuratomi) Date: Mon, 04 Dec 2006 21:59:11 -0800 Subject: Package DB Schema v3 Message-ID: <1165298351.29086.409.camel@localhost.localdomain> I'm attaching my latest incarnation of the PackageDB schema. Here's the run down on what's changed: * Implementation of Collection inheritance/overlays/sets * Add a reviewURL field to Package * Restructure the Package hierarchy slightly: Old: Package Collection | | PackageListing | PackageVersion New: +-Collection Package | | | | | PackageListing PackageVersion | | +----------------->PackageVersionListing The new structure allows a PackageVersion to exist in multiple collections which Jesse says is beneficial when spinning short-term, collections. * PackageInterest revamped as a PackageACL set of tables. Instead of having user roles (comaintainer, watcher, etc) we have functions that the user can perform. The functions I've come up with so far are: - commit: Commit changes to the VCS - build: Request builds from the buildsystem - watchbugzilla: Be informed of bugzilla tickets - watchcommits: Be informed of VCS commits for this package - approveacls: Change these ACLs for this package - checkout: Checkout the package from the VCS. We'll normally set this to true for a group that includes everyone but certain embargoed branches may need to be set to a more restricted group. Owners will implicitly have access to all of these. Sponsors or the equivalent group of trusted contributors under the merged FC/FE will belong to a group with commit-build-approveacls-checkout permissions. Things that still need working on: * Integrating comps/categories. I have some notes in comments. In order to implement this we need to 1) decide if comps is the right thing for this or if something closer to Nicolas Mailhot's suggestion is the right way to go. 2) Give the PackageDB an understanding of binary packages/subpackages as groups and categories won't be the same for each subpackage. [Note unless someone wants to work on this, I'm deferring it to after the first iteration.] * Logging. Make sure we have all the tables we need for logging. Verify that we really need to log on all these areas. Collections, Packages, PackageListing(aka package in Collection), PackageVersion, and PackageACL all have status fields so they should all have logs or we need to evaluate whether we really want status for each of these. * CollectionSets: Overlays can now be tracked in the database but the logic for looking for packages has to be implemented. After talking with Jesse, I think the simplest may be to implement this in application code so that searching for a package will first check the collection, then each of its bases. However, this imposes a performance penalty on each select. It *may* be worthwhile to look into copying the packages to the relevant overlay collections on insert instead. I have some notes as comments in the schema. * Set things up for SQLObject and SQLAlchemy. I'm going to try loading the objects through database introspection and see how the two ORMs fare. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: pkgdb.sql Type: text/x-vhdl Size: 14369 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From kzak at redhat.com Tue Dec 5 13:57:27 2006 From: kzak at redhat.com (Karel Zak) Date: Tue, 5 Dec 2006 14:57:27 +0100 Subject: Package DB Schema v3 In-Reply-To: <1165298351.29086.409.camel@localhost.localdomain> References: <1165298351.29086.409.camel@localhost.localdomain> Message-ID: <20061205135727.GG2582@petra.dvoda.cz> On Mon, Dec 04, 2006 at 09:59:11PM -0800, Toshio Kuratomi wrote: > create table Collection ( > id serial primary key, > name text not null, > version text not null, > status text not null default 'development', > owner integer not null, > publishURLTemplate text null, From PostgreSQL docs: NULL The column is allowed to contain null values. This is the default. ^^^^^^ This clause is only provided for compatibility with non-standard SQL databases. Its use is discouraged in new applications. > pendingURLTemplate text null, > summary text null, > description text null, > unique (name, version), > check (status = 'development' or status = 'active' or status = 'maintanence' > or status = 'EOL' or status = 'rejected') > ); From my point of view the status column is odd. I think you should use an abbreviation or one char only. status text not null default 'D' check ( status IN ('D','A','M','E','R') ) > > create table Branch ( > collectionId integer not null primary key, > branchName varchar(32) not null, ^^^^^^^^^ > distTag varchar(32) not null, ^^^^^^ is it right define duplicate tags and branch names? > parentId integer null, > foreign key (parentId) references Collection(id), > foreign key (collectionId) references Collection(id) > ); Hmm.. here I see 1:1 model (PK=FK). Strange. (It usually means that you should merge the tables to one table only.) Maybe: create table Branch ( id serial primary key, branchName varchar(32) not null unique, distTag varchar(32) not null unique, parentId integer references Collection(id), collectionId integer not null references Collection(id) ); Also, I think there should be defined some FK policy for update and delete. It means "ON DELETE" and "ON UPDATE" definition for the references. > create table Package ( > id serial primary key, > name text not null unique, > summary text not null, > description text null, > reviewURL text null, > status text not null default 'awaitingreview', > check (status = 'awaitingreview' or status = 'underreview' or status = 'approved' or status = 'denied') > ); IMHO, same problem with the status column. Here it is more terrible, because this table will be larger. CHECK( status IN ( 'W', -- wait 'R', -- review 'A', -- approved 'D' -- denied )) > > -- Permissions for who can make various changes to the code. > -- We want to limit the access that a given person may have to edit the package > -- > -- Fields: > -- :id: Primary key > -- :pkgListId: What package in what collection has this value. > -- :acl: The permission being set. > -- :status: Whether this permission is active. > create table PackageACL ( > id serial primary key, > packageListingId integer not null, > acl text not null, > status text not null, > foreign key (packageListingId) references PackageListing(id), > check (status = 'awaitingreview' or status = 'approved' or status = 'denied' > or status = 'obsolete'), > check (acl = 'commit' or acl = 'build' or acl = 'watchbugzilla' > or acl = 'watchcommits' or acl = 'approveacls' or acl = 'checkout') > ); > > -- ACLs that allow a person to do something > -- > -- Fields: > -- :packageACLId: Inherit from an ACL record. > -- :userId: User id from the account system. > create table PersonPackageACL ( > packageACLId integer primary key, > userId integer not null, > foreign key (packageACLId) references PackageACL (id) > ); Again. 1:1 model (PK=FK). I think the ACL model should be: what (ACL), where (package), who (user) you can use one table only (or two if ACL is group of permissions). > -- ACLs that allow a group to do something > -- > -- Fields: > -- :packageACLId: Inherit from an ACL record. > -- :groupId: Group id from the account system. > create table GroupPackagePermissions ( > packageACLId integer primary key, > groupId integer not null, > foreign key (PackageACLId) references PackageACL (id) > ); Again. 1:1 > -- Log a change to the packageDB. > -- > -- Fields: > -- :id: Primary key > -- :userId: Who made the change. > -- :changeTime: Time that the change occurred. > create table Log ( > id serial primary key, > userId integer not null, > changeTime timestamp default now() not null > ); When, Who, ... and where is "What"? (type of change) > -- Log a change made to the Package table. > -- > -- Fields: > -- :logId: The id of the log entry. > -- :packageId: The package that changed. > -- :action: What happened to the package. > -- :description: Additional information about the change. > create table PackageLog ( > logId integer primary key, > packageId integer not null, > action text not null, > description text null, > check (action = 'added' or action = 'removed' or action = 'statuschanged' or > action = 'awaitingreview' or action = 'underreview' or action = 'approved' > or action = 'denied'), > foreign key (logId) references Log(id), > foreign key (packageId) references Package(id) > ); ... ah... here is the "What". Again 1:1 (PK=FK). Don't forget PRIMARY KEY is always UNIQUE. > -- Log changes to packages in collections. > -- > -- Fields: > -- :logId: The id of the log entry. > -- :packageListingId: The packageListing that changed. > -- :action: What happened to the package in the collection. > -- :description: Additional information about the change. > create table PackageListingLog ( > logId integer primary key, > packageListingId integer not null, > action text not null, > description text null, > check (action = 'added' or action = 'removed' or action = 'awaitingreview' > or action = 'awaitingbranch' or action = 'underreview' or > action = 'approved' or action = 'denied'), > foreign key (logId) references Log (id), > foreign key (packageListingId) references PackageListing(id) > ); Again. 1:1. > -- Log changes to built packages. > -- > -- Fields: > -- :logId: The id of the log entry. > -- :packageVersionId: The `PackageVersion` that changed. > -- :action: What happened to the `PackageVersion`. > -- :description: Additional information about the change. > create table PackageVersionLog ( > logId integer primary key, > packageVersionId integer not null, > action text not null, > description text null, > check (action = 'added' or action = 'awaitingdevel' or > action = 'awaitingreview' or action = 'awaitingqa' or > action = 'aaitingpublish' or action = 'approved' or action = 'denied' or > action = 'obsolete'), > foreign key (logId) references Log (id), > foreign key (packageVersionId) references PackageVersion(id) > ); Again. 1:1. > -- Log changes to built package ACLs. > -- > -- Fields: > -- :logId: The id of the log entry. > -- :packageVersionId: The `PackageACL` that changed. > -- :action: What happened to the ACLs for the package. > -- :description: Additional information about the change. > create table PackageACLLog ( > logId integer primary key, > packageACLId integer not null, > action text not null, > description text null, > check (action = 'added' or action = 'awaitingreview' > or action = 'awaitingbranch' or action = 'underreview' or > action = 'approved' or action = 'denied' or action = 'obsolete'), > foreign key (logId) references Log (id), > foreign key (packageACLId) references PackageACL(id) > ); Again. 1:1. Karel -- Karel Zak From mmcgrath at fedoraproject.org Tue Dec 5 14:09:59 2006 From: mmcgrath at fedoraproject.org (Mike McGrath) Date: Tue, 5 Dec 2006 08:09:59 -0600 Subject: OpenID on Slashdot Message-ID: <3237e4410612050609h7e57bem360c0e90c3832483@mail.gmail.com> http://it.slashdot.org/it/06/12/05/139204.shtml -Mike From jeff at ocjtech.us Tue Dec 5 16:19:37 2006 From: jeff at ocjtech.us (Jeffrey C. Ollie) Date: Tue, 05 Dec 2006 10:19:37 -0600 Subject: Glump in Fedora Extras? Message-ID: <1165335577.3408.12.camel@lt21223.campus.dmacc.edu> It looks like Glump might be the perfect tool for a project here at work. Is there anyone @ Duke that wanted to maintain the package in Fedora Extras? If not, I'm willing to be the maintainer. Jeff -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From dimitris at glezos.com Tue Dec 5 19:03:48 2006 From: dimitris at glezos.com (Dimitris Glezos) Date: Tue, 05 Dec 2006 19:03:48 +0000 Subject: Package DB Schema v3 In-Reply-To: <20061205135727.GG2582@petra.dvoda.cz> References: <1165298351.29086.409.camel@localhost.localdomain> <20061205135727.GG2582@petra.dvoda.cz> Message-ID: <4575C294.7090103@glezos.com> O/H Karel Zak ??????: > On Mon, Dec 04, 2006 at 09:59:11PM -0800, Toshio Kuratomi wrote: >> pendingURLTemplate text null, >> summary text null, >> description text null, >> unique (name, version), >> check (status = 'development' or status = 'active' or status = 'maintanence' >> or status = 'EOL' or status = 'rejected') >> ); > > From my point of view the status column is odd. I think you should > use an abbreviation or one char only. > > status text not null default 'D' > check ( status IN ('D','A','M','E','R') ) It would be best to keep open the possibility that these (and other similar) flags might change in the future, happen to have conflicting first letters etc. -d -- Dimitris Glezos Jabber ID: glezos at jabber.org, GPG: 0xA5A04C3B http://dimitris.glezos.com/ "He who gives up functionality for ease of use loses both and deserves neither." (Anonymous) -- From a.badger at gmail.com Tue Dec 5 19:32:29 2006 From: a.badger at gmail.com (Toshio Kuratomi) Date: Tue, 05 Dec 2006 11:32:29 -0800 Subject: Package DB Schema v3 In-Reply-To: <20061205135727.GG2582@petra.dvoda.cz> References: <1165298351.29086.409.camel@localhost.localdomain> <20061205135727.GG2582@petra.dvoda.cz> Message-ID: <1165347149.21163.38.camel@localhost.localdomain> Thanks for the feedback! On Tue, 2006-12-05 at 14:57 +0100, Karel Zak wrote: > On Mon, Dec 04, 2006 at 09:59:11PM -0800, Toshio Kuratomi wrote: > > create table Collection ( > > id serial primary key, > > name text not null, > > version text not null, > > status text not null default 'development', > > owner integer not null, > > publishURLTemplate text null, > > From PostgreSQL docs: > > NULL > The column is allowed to contain null values. This is the > default. > ^^^^^^ > > This clause is only provided for compatibility with > non-standard SQL databases. Its use is discouraged in new > applications. > Okay. I'll make this implicit. > > > pendingURLTemplate text null, > > summary text null, > > description text null, > > unique (name, version), > > check (status = 'development' or status = 'active' or status = 'maintanence' > > or status = 'EOL' or status = 'rejected') > > ); > > From my point of view the status column is odd. I think you should > use an abbreviation or one char only. > > status text not null default 'D' > check ( status IN ('D','A','M','E','R') ) > What's the justification? Here's my reasoning: [Single letter code] + Harder to misspell + Takes less storage space [Full words] + Less cryptic We could solve both concerns by having a foreign key constraint into a table with the valid status phrases for each table that needs statuses but that makes things more complex. This would allow us to select the list of valid statuses from a database table which is a plus. But it would also require a join for the common case of giving a human readable name for the status. So I'd like to hear your justification. > > > > create table Branch ( > > collectionId integer not null primary key, > > branchName varchar(32) not null, > ^^^^^^^^^ > > distTag varchar(32) not null, > ^^^^^^ > is it right define duplicate tags and branch names? > distTag and branchName are different pieces of data. From the comments: -- :branchName: Name of the branch in the VCS ("FC-3", "devel") -- :distTag: DistTag used in the buildsystem (".fc3", ".fc6") It may be unfortunate that we don't use the same names for both, but it's the way things are. > > parentId integer null, > > foreign key (parentId) references Collection(id), > > foreign key (collectionId) references Collection(id) > > ); > > Hmm.. here I see 1:1 model (PK=FK). Strange. (It usually means that > you should merge the tables to one table only.) Maybe: > > create table Branch ( > id serial primary key, > branchName varchar(32) not null unique, > distTag varchar(32) not null unique, > parentId integer references Collection(id), > collectionId integer not null references Collection(id) > ); > We're actually modeling a 1:[0 1] relationship. For programmers, this would be inheritance. > Also, I think there should be defined some FK policy for update and > delete. It means "ON DELETE" and "ON UPDATE" definition for the > references. > You're correct. It's something I forgot to add to my list of todo items at the bottom. [snip lots of comments on 1:1 and succinct status fields] If you can show that inheritance is bad or that there's a compelling reason to make status fields a single character, I can change all the instances. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From kzak at redhat.com Tue Dec 5 22:10:14 2006 From: kzak at redhat.com (Karel Zak) Date: Tue, 5 Dec 2006 23:10:14 +0100 Subject: Package DB Schema v3 In-Reply-To: <1165347149.21163.38.camel@localhost.localdomain> References: <1165298351.29086.409.camel@localhost.localdomain> <20061205135727.GG2582@petra.dvoda.cz> <1165347149.21163.38.camel@localhost.localdomain> Message-ID: <20061205221014.GJ2582@petra.dvoda.cz> On Tue, Dec 05, 2006 at 11:32:29AM -0800, Toshio Kuratomi wrote: > > status text not null default 'D' > > check ( status IN ('D','A','M','E','R') ) > > > What's the justification? Here's my reasoning: > [Single letter code] > + Harder to misspell > + Takes less storage space + Takes less space for data transfer between client and DB engine > [Full words] > + Less cryptic cryptic? Who will be typical client for your DB? Human or script? ;-) > We could solve both concerns by having a foreign key constraint into a > table with the valid status phrases for each table that needs statuses Yes. It's a good way *in case* you need modify/add status types, otherwise it's over engineering. (FK is nothing cheap for DB engine.) > but that makes things more complex. This would allow us to select the > list of valid statuses from a database table which is a plus. But it > would also require a join for the common case of giving a human readable > name for the status. So I'd like to hear your justification. > > > > > > > create table Branch ( > > > collectionId integer not null primary key, > > > branchName varchar(32) not null, > > ^^^^^^^^^ > > > distTag varchar(32) not null, > > ^^^^^^ > > is it right define duplicate tags and branch names? > > > distTag and branchName are different pieces of data. > From the comments: > -- :branchName: Name of the branch in the VCS ("FC-3", "devel") > -- :distTag: DistTag used in the buildsystem (".fc3", ".fc6") > > It may be unfortunate that we don't use the same names for both, but > it's the way things are. Well, the question is: is it expected that there will be same DistTag for different branches? If not.. define distTag as UNIQUE. > > > parentId integer null, > > > foreign key (parentId) references Collection(id), > > > foreign key (collectionId) references Collection(id) > > > ); > > > > Hmm.. here I see 1:1 model (PK=FK). Strange. (It usually means that > > you should merge the tables to one table only.) Maybe: > > > > create table Branch ( > > id serial primary key, > > branchName varchar(32) not null unique, > > distTag varchar(32) not null unique, > > parentId integer references Collection(id), > > collectionId integer not null references Collection(id) > > ); > > > We're actually modeling a 1:[0 1] relationship. For programmers, this > would be inheritance. Hint: never think about programmers and DB usage when you work on DB design. Think about data and relatioships between data only :-) I don't have time to study data and real relatioships for this DB, but 1:1 is usually very strange. (and 1:0 = PK:NULL in normal table). Karel -- Karel Zak From a.badger at gmail.com Tue Dec 5 23:04:13 2006 From: a.badger at gmail.com (Toshio Kuratomi) Date: Tue, 05 Dec 2006 15:04:13 -0800 Subject: Package DB Schema v3 In-Reply-To: <20061205221014.GJ2582@petra.dvoda.cz> References: <1165298351.29086.409.camel@localhost.localdomain> <20061205135727.GG2582@petra.dvoda.cz> <1165347149.21163.38.camel@localhost.localdomain> <20061205221014.GJ2582@petra.dvoda.cz> Message-ID: <1165359853.21163.145.camel@localhost.localdomain> On Tue, 2006-12-05 at 23:10 +0100, Karel Zak wrote: > On Tue, Dec 05, 2006 at 11:32:29AM -0800, Toshio Kuratomi wrote: > > > status text not null default 'D' > > > check ( status IN ('D','A','M','E','R') ) > > > > > What's the justification? Here's my reasoning: > > [Single letter code] > > + Harder to misspell > > + Takes less storage space > > + Takes less space for data transfer between client and DB engine > > > [Full words] > > + Less cryptic > > cryptic? Who will be typical client for your DB? Human or script? ;-) > Humans :-) If we only keep codes in the database, then application level code has to translate that into something human readable. Then you end up with duplicate code when more than one application is utilizing the database. It's much better to keep the information somewhere in the database. > > We could solve both concerns by having a foreign key constraint into a > > table with the valid status phrases for each table that needs statuses > > Yes. It's a good way *in case* you need modify/add status types, > otherwise it's over engineering. (FK is nothing cheap for DB engine.) > Right. I'm not projecting needing to modify status types too frequently, so I figured that FK would be overkill. Jeffrey Ollie brought up some l10n and i18n ideas wrt this, though. I'll pull that discussion back into this thread. > > but that makes things more complex. This would allow us to select the > > list of valid statuses from a database table which is a plus. But it > > would also require a join for the common case of giving a human readable > > name for the status. So I'd like to hear your justification. > > > > > > > > > > create table Branch ( > > > > collectionId integer not null primary key, > > > > branchName varchar(32) not null, > > > ^^^^^^^^^ > > > > distTag varchar(32) not null, > > > ^^^^^^ > > > is it right define duplicate tags and branch names? > > > > > distTag and branchName are different pieces of data. > > From the comments: > > -- :branchName: Name of the branch in the VCS ("FC-3", "devel") > > -- :distTag: DistTag used in the buildsystem (".fc3", ".fc6") > > > > It may be unfortunate that we don't use the same names for both, but > > it's the way things are. > > Well, the question is: is it expected that there will be same DistTag > for different branches? If not.. define distTag as UNIQUE. > Ah. I understand you now. Yes, both distTag and branchName can be unique. I'll change that. > > > > parentId integer null, > > > > foreign key (parentId) references Collection(id), > > > > foreign key (collectionId) references Collection(id) > > > > ); > > > > > > Hmm.. here I see 1:1 model (PK=FK). Strange. (It usually means that > > > you should merge the tables to one table only.) Maybe: > > > > > > create table Branch ( > > > id serial primary key, > > > branchName varchar(32) not null unique, > > > distTag varchar(32) not null unique, > > > parentId integer references Collection(id), > > > collectionId integer not null references Collection(id) > > > ); > > > > > We're actually modeling a 1:[0 1] relationship. For programmers, this > > would be inheritance. > > Hint: never think about programmers and DB usage when you work on DB > design. Think about data and relatioships between data only :-) > Generally good. But in this case we're working with an Object Relational Mapper so we have to pay some attention to the programmer as well. Additionally, we *are* dealing with relationships between data here, it's just that it's 1:[0 1] which is covered much less frequently than 1:1 (mostly avoid), 1:many, and many:many (transform into 1:many) > I don't have time to study data and real relatioships for this DB, but > 1:1 is usually very strange. (and 1:0 = PK:NULL in normal table). > Right, 1:1 is usually not necessary. 1:0 makes no sense :-) This is neither of those. 1:[0 1] does not make PK = NULL. It means that there will be some records in Collection for which there is also a record in Branch and some for which no branch record exists at all. PostgreSQL and SQL-1999 both have ways of modeling this. But PostgreSQL's has some caveats that we might run into and neither PostgreSQL nor MySQL support the SQL-1999 construct. Here's some documentation on PostgreSQL's inheritance feature: http://www.postgresql.org/docs/8.1/interactive/ddl-inherit.html and some documentation on the interface with the ORM: http://www.sqlalchemy.org/docs/adv_datamapping.myt#advdatamapping_inheritance -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From a.badger at gmail.com Tue Dec 5 23:19:38 2006 From: a.badger at gmail.com (Toshio Kuratomi) Date: Tue, 05 Dec 2006 15:19:38 -0800 Subject: Package DB Schema v3 In-Reply-To: <1165356840.3408.32.camel@lt21223.campus.dmacc.edu> References: <1165298351.29086.409.camel@localhost.localdomain> <20061205135727.GG2582@petra.dvoda.cz> <1165347149.21163.38.camel@localhost.localdomain> <1165347803.3408.23.camel@lt21223.campus.dmacc.edu> <1165350940.21163.69.camel@localhost.localdomain> <1165356840.3408.32.camel@lt21223.campus.dmacc.edu> Message-ID: <1165360778.21163.157.camel@localhost.localdomain> On Tue, 2006-12-05 at 16:14 -0600, Jeffrey C. Ollie wrote: > On Tue, 2006-12-05 at 12:35 -0800, Toshio Kuratomi wrote: > > On Tue, 2006-12-05 at 13:43 -0600, Jeffrey C. Ollie wrote: > > > On Tue, 2006-12-05 at 11:32 -0800, Toshio Kuratomi wrote: > > > > > > > > We could solve both concerns by having a foreign key constraint into a > > > > table with the valid status phrases for each table that needs statuses > > > > but that makes things more complex. This would allow us to select the > > > > list of valid statuses from a database table which is a plus. But it > > > > would also require a join for the common case of giving a human readable > > > > name for the status. So I'd like to hear your justification. > > > > > > You could include localized versions of the status names in the database > > > table rather than doing the localization in the front-end (of which > > > there might be several). > > > > Okay. What do you think about doing it like this: > > > > create table StatusCode ( > > id serial primary key, > > ); > > > > create table Translations ( > > statusCodeId integer references StatusCode(id), > > language text not null default 'C', > > statusName text not null, > > primary key (statusCodeId, language) > > ); > > Yeah, that looks like what I had in mind, except that I would call the > table "StatusCodeTranslations." > That would be fine. > > create view CollectionStatusCode as select id from StatusCode where id = 1 or > > id = 3 or id = 7; > > > > create table Collection ( > > [...] > > status integer references CollectionStatusCode(id) > > ); > > > > Overengineered or good? > > Yeah, looks good to me. > > > (The view allows us to query which statuses are available for this > > table. I don't know of a sane way to do that with a check constraint.) > > Why not make CollectionStatusCode a table like this: > > create table CollectionStatusCode ( > id integer primary key references StatusCode(id) > ); That should be fine. With a dataset as small as the possible status codes I don't know that there would be much difference between the two implementations. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From jeff at ocjtech.us Wed Dec 6 06:40:40 2006 From: jeff at ocjtech.us (Jeffrey C. Ollie) Date: Wed, 06 Dec 2006 00:40:40 -0600 Subject: Glump in Fedora Extras? In-Reply-To: <1165335577.3408.12.camel@lt21223.campus.dmacc.edu> References: <1165335577.3408.12.camel@lt21223.campus.dmacc.edu> Message-ID: <1165387240.8726.4.camel@lt21223.campus.dmacc.edu> On Tue, 2006-12-05 at 10:19 -0600, Jeffrey C. Ollie wrote: > It looks like Glump might be the perfect tool for a project here at > work. Is there anyone @ Duke that wanted to maintain the package in > Fedora Extras? If not, I'm willing to be the maintainer. https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=218577 Jeff -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From sopwith at gmail.com Wed Dec 6 14:19:55 2006 From: sopwith at gmail.com (Elliot Lee) Date: Wed, 6 Dec 2006 09:19:55 -0500 Subject: Package DB Schema v3 In-Reply-To: <1165347149.21163.38.camel@localhost.localdomain> References: <1165298351.29086.409.camel@localhost.localdomain> <20061205135727.GG2582@petra.dvoda.cz> <1165347149.21163.38.camel@localhost.localdomain> Message-ID: On Dec 5, 2006, at 2:32 PM, Toshio Kuratomi wrote: >> status text not null default 'D' >> check ( status IN ('D','A','M','E','R') ) >> > What's the justification? Here's my reasoning: > [Single letter code] > + Harder to misspell > + Takes less storage space > [Full words] You're sticking the status into a column of type 'TEXT', though. TEXT is not even like varchar which defaults to 1024 characters or so of storage. I believe TEXT tells the database 'be ready to store between zero and 4 billion bytes in this column'. If you want to cut down on storage space, check the field types and specify maximum sizes. In this case, varchar(1) would be a more appropriate type. Since the database will wind up storing this field using at least 4 bytes of space for alignment purposes, might as well take it to varchar(3) or so to also allow for future expansion (and maybe a bit more readability). To balance fast indexing/comparison, small storage space, and room for expansion, it's often easier to use INTEGER for these type of columns, and then assign meaning to those values elsewhere (either through a separate table that translates values to strings, or by #define-like constants in the source code). :) Best, -- Elliot From sbranden at redhat.com Wed Dec 6 16:41:35 2006 From: sbranden at redhat.com (Stacy J Brandenburg) Date: Wed, 06 Dec 2006 11:41:35 -0500 Subject: new 1950 Message-ID: <4576F2BF.7020205@redhat.com> I received and installed a new 1950 from Dell... What is the purpose of this machine going to be so that I can label it correctly? -- ======================================================== = Stacy J. Brandenburg Red Hat Inc. = = Manager, Network Operations sbranden at redhat.com = = 919-754-4313 http://www.redhat.com = ======================================================== From dennis at ausil.us Wed Dec 6 16:46:16 2006 From: dennis at ausil.us (Dennis Gilmore) Date: Wed, 6 Dec 2006 10:46:16 -0600 Subject: new 1950 In-Reply-To: <4576F2BF.7020205@redhat.com> References: <4576F2BF.7020205@redhat.com> Message-ID: <200612061046.16824.dennis@ausil.us> On Wednesday 06 December 2006 10:41 am, Stacy J Brandenburg wrote: > I received and installed a new 1950 from Dell... > > What is the purpose of this machine going to be so that I can label it > correctly? I think thats the new builder its hostname will be xen4 it will have a xenguest of xenbuilder2. Can you set it up so i can get access to do a fc6 install. -- Dennis Gilmore, RHCE Proud Australian From sbranden at redhat.com Wed Dec 6 17:02:11 2006 From: sbranden at redhat.com (Stacy J Brandenburg) Date: Wed, 06 Dec 2006 12:02:11 -0500 Subject: new 1950 In-Reply-To: <200612061046.16824.dennis@ausil.us> References: <4576F2BF.7020205@redhat.com> <200612061046.16824.dennis@ausil.us> Message-ID: <4576F793.1000400@redhat.com> you betcha. It is on the network now, and is available at xen4 or tty02 on the cycaldes. Bios redirection was turned on as well. It is also available on the KVM as xen4. Dennis Gilmore wrote: > On Wednesday 06 December 2006 10:41 am, Stacy J Brandenburg wrote: >> I received and installed a new 1950 from Dell... >> >> What is the purpose of this machine going to be so that I can label it >> correctly? > I think thats the new builder > its hostname will be xen4 it will have a xenguest of xenbuilder2. Can you > set it up so i can get access to do a fc6 install. -- ======================================================== = Stacy J. Brandenburg Red Hat Inc. = = Manager, Network Operations sbranden at redhat.com = = 919-754-4313 http://www.redhat.com = ======================================================== From a.badger at gmail.com Wed Dec 6 19:19:29 2006 From: a.badger at gmail.com (Toshio Kuratomi) Date: Wed, 06 Dec 2006 11:19:29 -0800 Subject: Package DB Schema v3 In-Reply-To: References: <1165298351.29086.409.camel@localhost.localdomain> <20061205135727.GG2582@petra.dvoda.cz> <1165347149.21163.38.camel@localhost.localdomain> Message-ID: <1165432770.21163.246.camel@localhost.localdomain> On Wed, 2006-12-06 at 09:19 -0500, Elliot Lee wrote: > On Dec 5, 2006, at 2:32 PM, Toshio Kuratomi wrote: > >> status text not null default 'D' > >> check ( status IN ('D','A','M','E','R') ) > >> > > What's the justification? Here's my reasoning: > > [Single letter code] > > + Harder to misspell > > + Takes less storage space > > [Full words] > > You're sticking the status into a column of type 'TEXT', though. TEXT > is not even like varchar which defaults to 1024 characters or so of > storage. I believe TEXT tells the database 'be ready to store between > zero and 4 billion bytes in this column'. > varchar and text have had equivalent performance and storage in postgreSQL for a long time. According to the 8.2 docs[1]_, char is currently no better (and it space pads your entries so there's probably no reason to use it if you aren't planning on switching databases) (This change appears to have been part of 7.1 [2]_) This assumes you store the same values in each column. The value of varchar is it allows you to specify a limit which can prevent the user from using 'supercalifragilisticexpialadocious' as a value. But in most of our use cases, allowing room for expansion seems fine. In this particular case, we're using a constraint to prevent random values so we wouldn't be providing any additional limitations on the user's ability to enter long data. [1]_ http://www.postgresql.org/docs/8.2/interactive/datatype-character.html [2]_ http://www.postgresql.org/docs/7.2/static/release-7-1.html > If you want to cut down on storage space, check the field types and > specify maximum sizes. In this case, varchar(1) would be a more > appropriate type. Since the database will wind up storing this field > using at least 4 bytes of space for alignment purposes, might as well > take it to varchar(3) or so to also allow for future expansion (and > maybe a bit more readability). > I'm not too concerned with storage space bloat from the status field. Karel's clarification about db-to-client transfer makes sense. However, I'm in the camp that believes the status strings belong in the database so I think the typical usage will transfer the same amount of data. (If the statusTranslations table is cached this might be better, though... more room for optimization if we have separate tables.) > To balance fast indexing/comparison, small storage space, and room > for expansion, it's often easier to use INTEGER for these type of > columns, and then assign meaning to those values elsewhere (either > through a separate table that translates values to strings, or by > #define-like constants in the source code). > I think we'll do this with INTEGER and a separate table because Jeffrey Ollie's proposal to allow translations to the status codes makes sense. I really hate having constants in source code because it hinders the database's ability to enforce integrity. The source code of the application has to stay in sync with the database. If there's more than one consumer, each one has to stay in sync. Enforcing this is a problem that the database was designed to solve. > :) I thought about this as I modified the schema, really! ;-) -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From jeroen.janssen at gmail.com Wed Dec 6 21:53:28 2006 From: jeroen.janssen at gmail.com (Jeroen Janssen) Date: Wed, 6 Dec 2006 22:53:28 +0100 Subject: Update on Plague/Brew status? Message-ID: Hello, Any information on what's going to happen with Plague/Brew yet? Best regards, Jeroen Janssen From email.ahmedkamal at googlemail.com Wed Dec 6 22:43:50 2006 From: email.ahmedkamal at googlemail.com (Ahmed Kamal) Date: Thu, 7 Dec 2006 00:43:50 +0200 Subject: Web server torturing Message-ID: <3da3b5b40612061443k6385e18ck871081b072b0386e@mail.gmail.com> Hi, Due to the web server hit faced with FC6 release, some actions are being taken to minimize the chance of facing such issues again. One of the steps is to stress test our web-server infrastructure to measure our current and future capabilities. I'd like to run some tests on fp.o web-server, please let me know your thoughts/comments. Here are some details. Test Targets: 1- Measure our bare (no caching) maximum serving rate 2- Measure our cached serving rate, to assess the implemented caching efficiency 3- Gather numbers like (When do we get CPU saturated, RAM requirements ..). Possibly draw graphs (everyone thinks graphs are cool), the numbers should help us base future calculations on a solid basis 4- Future: Possibly implement a mechanism to cap the maximum connected clients to a specific server, to the maximum it can handle gracefully, to avoid killing a server Test Plan: 1- A script was written which uses apache's ab tool to stress the server. Script will run on the web-server host. 2- The script fires a total number of connections equal to ten times the maximum concurrency rate (to get good average, and avoid transients) 3- The concurrency rate is sweeped between 10 and 400 (my 1G-RAM machine swaps at about 100 connections)? any suggestions? 4- All ab output is recorded for future analysis 5- A monitoring thread is fired before ab is launched. The monitoring uses "top" to record load/cpu/ram/process information in log files as well 6- Tests are repeated with "ab -k" for enabling the HTTP keep-alive option. Not sure if this is needed, or if it will make much difference! comments? 7- Tests are done once with caching enabled and one more time without caching Please let me know your thoughts about the testing setup, should we be recording more data? should we be stressing the server in a different way, should we be testing some apache config options ... etc ? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: webtorture.sh Type: application/x-sh Size: 1696 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: webtorture.sh.sig Type: application/octet-stream Size: 189 bytes Desc: not available URL: From kzak at redhat.com Wed Dec 6 23:00:44 2006 From: kzak at redhat.com (Karel Zak) Date: Thu, 7 Dec 2006 00:00:44 +0100 Subject: Package DB Schema v3 In-Reply-To: <1165432770.21163.246.camel@localhost.localdomain> References: <1165298351.29086.409.camel@localhost.localdomain> <20061205135727.GG2582@petra.dvoda.cz> <1165347149.21163.38.camel@localhost.localdomain> <1165432770.21163.246.camel@localhost.localdomain> Message-ID: <20061206230044.GM2582@petra.dvoda.cz> On Wed, Dec 06, 2006 at 11:19:29AM -0800, Toshio Kuratomi wrote: > On Wed, 2006-12-06 at 09:19 -0500, Elliot Lee wrote: > > On Dec 5, 2006, at 2:32 PM, Toshio Kuratomi wrote: > > >> status text not null default 'D' > > >> check ( status IN ('D','A','M','E','R') ) > > >> > > > What's the justification? Here's my reasoning: > > > [Single letter code] > > > + Harder to misspell > > > + Takes less storage space > > > [Full words] > > > > You're sticking the status into a column of type 'TEXT', though. TEXT > > is not even like varchar which defaults to 1024 characters or so of > > storage. I believe TEXT tells the database 'be ready to store between > > zero and 4 billion bytes in this column'. > > > > varchar and text have had equivalent performance and storage in Definitely true. All internal PostgreSQL routines uses same type (text) for almost all operations. The varchar(n) makes sense if you want to restrict something in your data model, but it's not important for performance and storage. > > To balance fast indexing/comparison, small storage space, and room > > for expansion, it's often easier to use INTEGER for these type of > > columns, and then assign meaning to those values elsewhere (either > > through a separate table that translates values to strings, or by > > #define-like constants in the source code). > > > I think we'll do this with INTEGER and a separate table because Jeffrey > Ollie's proposal to allow translations to the status codes makes sense. It depends on number of status codes. You can use 1 or 2 chars as a primary key for status table (instead integers). It's better for humans, because simple selects (without join to status table) are still readable. Karel -- Karel Zak From mmcgrath at fedoraproject.org Thu Dec 7 00:02:19 2006 From: mmcgrath at fedoraproject.org (Mike McGrath) Date: Wed, 6 Dec 2006 18:02:19 -0600 Subject: Web server torturing In-Reply-To: <3da3b5b40612061443k6385e18ck871081b072b0386e@mail.gmail.com> References: <3da3b5b40612061443k6385e18ck871081b072b0386e@mail.gmail.com> Message-ID: <3237e4410612061602h21819dcco6c2466d351509fad@mail.gmail.com> On 12/6/06, Ahmed Kamal wrote: > Hi, > Due to the web server hit faced with FC6 release, some actions are being > taken to minimize the chance of facing such issues again. One of the steps > is to stress test our web-server infrastructure to measure our current and > future capabilities. I'd like to run some tests on fp.o web-server, please > let me know your thoughts/comments. Here are some details. > > Test Targets: > > 1- Measure our bare (no caching) maximum serving rate > 2- Measure our cached serving rate, to assess the implemented caching > efficiency > 3- Gather numbers like (When do we get CPU saturated, RAM requirements ..). > Possibly draw graphs (everyone thinks graphs are cool), the numbers should > help us base future calculations on a solid basis > 4- Future: Possibly implement a mechanism to cap the maximum connected > clients to a specific server, to the maximum it can handle gracefully, to > avoid killing a server > > Test Plan: > 1- A script was written which uses apache's ab tool to stress the server. > Script will run on the web-server host. > 2- The script fires a total number of connections equal to ten times the > maximum concurrency rate (to get good average, and avoid transients) > 3- The concurrency rate is sweeped between 10 and 400 (my 1G-RAM machine > swaps at about 100 connections)? any suggestions? > 4- All ab output is recorded for future analysis > 5- A monitoring thread is fired before ab is launched. The monitoring uses > "top" to record load/cpu/ram/process information in log files as well > 6- Tests are repeated with "ab -k" for enabling the HTTP keep-alive option. > Not sure if this is needed, or if it will make much difference! comments? > 7- Tests are done once with caching enabled and one more time without > caching > > Please let me know your thoughts about the testing setup, should we be > recording more data? should we be stressing the server in a different way, > should we be testing some apache config options ... etc ? > Thanks Thanks for sending this out Ahmed, you and Paulo have been doing a great job with all of this. Before we get started I'm hoping to get cacti set up for a good baseline. One thing I'd like to note to the rest of the list is that paulo and ahmed are both timezoned around GMT, so they can do this easily during off hours as not to cause a disruption. -Mike From a.badger at gmail.com Thu Dec 7 00:21:22 2006 From: a.badger at gmail.com (Toshio Kuratomi) Date: Wed, 06 Dec 2006 16:21:22 -0800 Subject: Package DB Schema v3 In-Reply-To: <20061206230044.GM2582@petra.dvoda.cz> References: <1165298351.29086.409.camel@localhost.localdomain> <20061205135727.GG2582@petra.dvoda.cz> <1165347149.21163.38.camel@localhost.localdomain> <1165432770.21163.246.camel@localhost.localdomain> <20061206230044.GM2582@petra.dvoda.cz> Message-ID: <1165450882.21163.295.camel@localhost.localdomain> On Thu, 2006-12-07 at 00:00 +0100, Karel Zak wrote: > On Wed, Dec 06, 2006 at 11:19:29AM -0800, Toshio Kuratomi wrote: > > On Wed, 2006-12-06 at 09:19 -0500, Elliot Lee wrote: > > > To balance fast indexing/comparison, small storage space, and room > > > for expansion, it's often easier to use INTEGER for these type of > > > columns, and then assign meaning to those values elsewhere (either > > > through a separate table that translates values to strings, or by > > > #define-like constants in the source code). > > > > > I think we'll do this with INTEGER and a separate table because Jeffrey > > Ollie's proposal to allow translations to the status codes makes sense. > > It depends on number of status codes. You can use 1 or 2 chars as a > primary key for status table (instead integers). It's better for > humans, because simple selects (without join to status table) are > still readable. Somewhat. We have 15 status codes right now. Even with three letter abbreviations, you have to have a pretty good idea what the possible statuses are in order to make use of it in "raw" form. Here are a few of the problems: + 7 of the statuses begin with "a" + Many of our statuses are two words + We have status's like "Under Review" and "Awaiting Review" which prevents simplifying to a mnemonic for the most significant word. Not sure that it's worthwhile to try to make these status codes directly human readable or not. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From sbranden at redhat.com Thu Dec 7 02:42:25 2006 From: sbranden at redhat.com (Stacy J. Brandenburg) Date: Wed, 06 Dec 2006 21:42:25 -0500 Subject: Web server torturing In-Reply-To: <3237e4410612061602h21819dcco6c2466d351509fad@mail.gmail.com> References: <3da3b5b40612061443k6385e18ck871081b072b0386e@mail.gmail.com> <3237e4410612061602h21819dcco6c2466d351509fad@mail.gmail.com> Message-ID: <45777F91.7050308@redhat.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I need to know well in advance of any thing that will "stress test" systems on the 209.132.176.0/23 network. I want to ensure the LB can handle it. Note, incoming BW can exceed 2 Gb/s to the systems in the Red Hat DC. Thanks Mike McGrath wrote: > On 12/6/06, Ahmed Kamal wrote: >> Hi, >> Due to the web server hit faced with FC6 release, some actions are being >> taken to minimize the chance of facing such issues again. One of the >> steps >> is to stress test our web-server infrastructure to measure our current >> and >> future capabilities. I'd like to run some tests on fp.o web-server, >> please >> let me know your thoughts/comments. Here are some details. >> >> Test Targets: >> >> 1- Measure our bare (no caching) maximum serving rate >> 2- Measure our cached serving rate, to assess the implemented caching >> efficiency >> 3- Gather numbers like (When do we get CPU saturated, RAM requirements >> ..). >> Possibly draw graphs (everyone thinks graphs are cool), the numbers >> should >> help us base future calculations on a solid basis >> 4- Future: Possibly implement a mechanism to cap the maximum connected >> clients to a specific server, to the maximum it can handle gracefully, to >> avoid killing a server >> >> Test Plan: >> 1- A script was written which uses apache's ab tool to stress the server. >> Script will run on the web-server host. >> 2- The script fires a total number of connections equal to ten times the >> maximum concurrency rate (to get good average, and avoid transients) >> 3- The concurrency rate is sweeped between 10 and 400 (my 1G-RAM machine >> swaps at about 100 connections)? any suggestions? >> 4- All ab output is recorded for future analysis >> 5- A monitoring thread is fired before ab is launched. The monitoring >> uses >> "top" to record load/cpu/ram/process information in log files as well >> 6- Tests are repeated with "ab -k" for enabling the HTTP keep-alive >> option. >> Not sure if this is needed, or if it will make much difference! comments? >> 7- Tests are done once with caching enabled and one more time without >> caching >> >> Please let me know your thoughts about the testing setup, should we be >> recording more data? should we be stressing the server in a different >> way, >> should we be testing some apache config options ... etc ? >> Thanks > > Thanks for sending this out Ahmed, you and Paulo have been doing a > great job with all of this. Before we get started I'm hoping to get > cacti set up for a good baseline. One thing I'd like to note to the > rest of the list is that paulo and ahmed are both timezoned around > GMT, so they can do this easily during off hours as not to cause a > disruption. > > -Mike > > _______________________________________________ > Fedora-infrastructure-list mailing list > Fedora-infrastructure-list at redhat.com > https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list - -- ======================================================== = Stacy J. Brandenburg Red Hat Inc. = = Manager, Network Operations sbranden at redhat.com = = 919-754-4313 http://www.redhat.com = ======================================================== Fingerprint 03F7 43BE 1150 CCFA F57B 54DD AEDB 1C27 1828 D94D -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFFd3+QrtscJxgo2U0RAvPtAKDPKZWMzr+6m3/VUI5yYzKraTa7kQCfcmbR 2l8jIA7QuF1fHqnxIJeg9rE= =zIMR -----END PGP SIGNATURE----- From mmcgrath at fedoraproject.org Thu Dec 7 03:00:02 2006 From: mmcgrath at fedoraproject.org (Mike McGrath) Date: Wed, 6 Dec 2006 21:00:02 -0600 Subject: Web server torturing In-Reply-To: <45777F91.7050308@redhat.com> References: <3da3b5b40612061443k6385e18ck871081b072b0386e@mail.gmail.com> <3237e4410612061602h21819dcco6c2466d351509fad@mail.gmail.com> <45777F91.7050308@redhat.com> Message-ID: <3237e4410612061900g38ddae0em6a6901e09579512c@mail.gmail.com> On 12/6/06, Stacy J. Brandenburg wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > I need to know well in advance of any thing that will "stress test" > systems on the 209.132.176.0/23 network. > No problem, we'll be fully coordinating this with everyone. -Mike From jeff at ocjtech.us Thu Dec 7 03:07:40 2006 From: jeff at ocjtech.us (Jeffrey C. Ollie) Date: Wed, 06 Dec 2006 21:07:40 -0600 Subject: Web server torturing In-Reply-To: <3237e4410612061602h21819dcco6c2466d351509fad@mail.gmail.com> References: <3da3b5b40612061443k6385e18ck871081b072b0386e@mail.gmail.com> <3237e4410612061602h21819dcco6c2466d351509fad@mail.gmail.com> Message-ID: <1165460860.5157.7.camel@lt21223.campus.dmacc.edu> On Wed, 2006-12-06 at 18:02 -0600, Mike McGrath wrote: > Before we get started I'm hoping to get > cacti set up for a good baseline. Not to denigrate Cacti but you might want to take a look at Zabbix... It was recently included in FE (I did the review). It's a little bit trickier to set up but the polling is done using a server & agent written in C, as opposed to Cacti which is largely PHP. I've been working on a Zabbix install at work and I poll a number of variables every 5 sec. Jeff -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From skvidal at linux.duke.edu Thu Dec 7 03:12:53 2006 From: skvidal at linux.duke.edu (seth vidal) Date: Wed, 06 Dec 2006 22:12:53 -0500 Subject: Web server torturing In-Reply-To: <1165460860.5157.7.camel@lt21223.campus.dmacc.edu> References: <3da3b5b40612061443k6385e18ck871081b072b0386e@mail.gmail.com> <3237e4410612061602h21819dcco6c2466d351509fad@mail.gmail.com> <1165460860.5157.7.camel@lt21223.campus.dmacc.edu> Message-ID: <1165461173.25300.33.camel@cutter> On Wed, 2006-12-06 at 21:07 -0600, Jeffrey C. Ollie wrote: > On Wed, 2006-12-06 at 18:02 -0600, Mike McGrath wrote: > > Before we get started I'm hoping to get > > cacti set up for a good baseline. > > Not to denigrate Cacti but you might want to take a look at Zabbix... It > was recently included in FE (I did the review). It's a little bit > trickier to set up but the polling is done using a server & agent > written in C, as opposed to Cacti which is largely PHP. I've been > working on a Zabbix install at work and I poll a number of variables > every 5 sec. > cacti in php doesn't matter. It's the snmp interface in cacti which does the tricks and that's all done using snmp libs. It also has the advantage of just being another use of snmp instead of being it's own whole thing. -sv From nils at breun.nl Thu Dec 7 08:33:35 2006 From: nils at breun.nl (Nils Breunese) Date: Thu, 7 Dec 2006 09:33:35 +0100 Subject: Web server torturing In-Reply-To: <1165461173.25300.33.camel@cutter> References: <3da3b5b40612061443k6385e18ck871081b072b0386e@mail.gmail.com> <3237e4410612061602h21819dcco6c2466d351509fad@mail.gmail.com> <1165460860.5157.7.camel@lt21223.campus.dmacc.edu> <1165461173.25300.33.camel@cutter> Message-ID: <14BCFE43-7846-43F8-9B46-452E2CD2F5C7@breun.nl> seth vidal wrote: > On Wed, 2006-12-06 at 21:07 -0600, Jeffrey C. Ollie wrote: >> On Wed, 2006-12-06 at 18:02 -0600, Mike McGrath wrote: >>> Before we get started I'm hoping to get >>> cacti set up for a good baseline. >> >> Not to denigrate Cacti but you might want to take a look at >> Zabbix... It >> was recently included in FE (I did the review). It's a little bit >> trickier to set up but the polling is done using a server & agent >> written in C, as opposed to Cacti which is largely PHP. I've been >> working on a Zabbix install at work and I poll a number of variables >> every 5 sec. > > cacti in php doesn't matter. It's the snmp interface in cacti which > does > the tricks and that's all done using snmp libs. It also has the > advantage of just being another use of snmp instead of being it's own > whole thing. ZABBIX also uses the snmp-libs for the optional SNMP agents. I use ZABBIX to monitor all of my servers and it suits me pretty nicely. The ZABBIX web frontend is in PHP, just like Cacti's. See http:// www.zabbix.com/manual/v1.1/install_requirements_software.php for more info on the software requirements for ZABBIX. Nils Breunese. -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 186 bytes Desc: Dit deel van het bericht is digitaal ondertekend URL: From damian.myerscough at gmail.com Thu Dec 7 10:12:19 2006 From: damian.myerscough at gmail.com (Damian Myerscough) Date: Thu, 7 Dec 2006 10:12:19 +0000 Subject: Web server torturing In-Reply-To: <14BCFE43-7846-43F8-9B46-452E2CD2F5C7@breun.nl> References: <3da3b5b40612061443k6385e18ck871081b072b0386e@mail.gmail.com> <3237e4410612061602h21819dcco6c2466d351509fad@mail.gmail.com> <1165460860.5157.7.camel@lt21223.campus.dmacc.edu> <1165461173.25300.33.camel@cutter> <14BCFE43-7846-43F8-9B46-452E2CD2F5C7@breun.nl> Message-ID: <8c9e56490612070212k5d12c08cj837b74b2fdc91ea2@mail.gmail.com> Hi, > 1- A script was written which uses apache's ab tool to stress the server. Script will run on the web-server host. There is a tool available for benchmarking apache called: ab or ab2 which is apache benchmark to use it you simply issue: ./ab -n 1000 -c 100 http://www.fedoraproject.org/wiki/index.htm The above will simulate 100 users accessing the website 1000 times On 07/12/06, Nils Breunese wrote: > seth vidal wrote: > > > On Wed, 2006-12-06 at 21:07 -0600, Jeffrey C. Ollie wrote: > >> On Wed, 2006-12-06 at 18:02 -0600, Mike McGrath wrote: > >>> Before we get started I'm hoping to get > >>> cacti set up for a good baseline. > >> > >> Not to denigrate Cacti but you might want to take a look at > >> Zabbix... It > >> was recently included in FE (I did the review). It's a little bit > >> trickier to set up but the polling is done using a server & agent > >> written in C, as opposed to Cacti which is largely PHP. I've been > >> working on a Zabbix install at work and I poll a number of variables > >> every 5 sec. > > > > cacti in php doesn't matter. It's the snmp interface in cacti which > > does > > the tricks and that's all done using snmp libs. It also has the > > advantage of just being another use of snmp instead of being it's own > > whole thing. > > ZABBIX also uses the snmp-libs for the optional SNMP agents. I use > ZABBIX to monitor all of my servers and it suits me pretty nicely. > The ZABBIX web frontend is in PHP, just like Cacti's. See http:// > www.zabbix.com/manual/v1.1/install_requirements_software.php for more > info on the software requirements for ZABBIX. > > Nils Breunese. > > > _______________________________________________ > Fedora-infrastructure-list mailing list > Fedora-infrastructure-list at redhat.com > https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list > > > > From paulo.banon at googlemail.com Thu Dec 7 10:24:05 2006 From: paulo.banon at googlemail.com (Paulo Santos) Date: Thu, 7 Dec 2006 10:24:05 +0000 Subject: Web server torturing In-Reply-To: <8c9e56490612070212k5d12c08cj837b74b2fdc91ea2@mail.gmail.com> References: <3da3b5b40612061443k6385e18ck871081b072b0386e@mail.gmail.com> <3237e4410612061602h21819dcco6c2466d351509fad@mail.gmail.com> <1165460860.5157.7.camel@lt21223.campus.dmacc.edu> <1165461173.25300.33.camel@cutter> <14BCFE43-7846-43F8-9B46-452E2CD2F5C7@breun.nl> <8c9e56490612070212k5d12c08cj837b74b2fdc91ea2@mail.gmail.com> Message-ID: <7a41c4bc0612070224p7c00377arae14578b18b523b3@mail.gmail.com> Hi Damian, The script written by Ahmed, already uses ab. He is using the script to gather some other information, besides the ab output. Please take a look into webtorture.sh, and you'll see what i mean. Paulo On 12/7/06, Damian Myerscough wrote: > > Hi, > > > 1- A script was written which uses apache's ab tool to stress the > server. Script will run on the web-server host. > > There is a tool available for benchmarking apache called: ab or ab2 > which is apache benchmark to use it you simply issue: > > ./ab -n 1000 -c 100 http://www.fedoraproject.org/wiki/index.htm > > The above will simulate 100 users accessing the website 1000 times > > On 07/12/06, Nils Breunese wrote: > > seth vidal wrote: > > > > > On Wed, 2006-12-06 at 21:07 -0600, Jeffrey C. Ollie wrote: > > >> On Wed, 2006-12-06 at 18:02 -0600, Mike McGrath wrote: > > >>> Before we get started I'm hoping to get > > >>> cacti set up for a good baseline. > > >> > > >> Not to denigrate Cacti but you might want to take a look at > > >> Zabbix... It > > >> was recently included in FE (I did the review). It's a little bit > > >> trickier to set up but the polling is done using a server & agent > > >> written in C, as opposed to Cacti which is largely PHP. I've been > > >> working on a Zabbix install at work and I poll a number of variables > > >> every 5 sec. > > > > > > cacti in php doesn't matter. It's the snmp interface in cacti which > > > does > > > the tricks and that's all done using snmp libs. It also has the > > > advantage of just being another use of snmp instead of being it's own > > > whole thing. > > > > ZABBIX also uses the snmp-libs for the optional SNMP agents. I use > > ZABBIX to monitor all of my servers and it suits me pretty nicely. > > The ZABBIX web frontend is in PHP, just like Cacti's. See http:// > > www.zabbix.com/manual/v1.1/install_requirements_software.php for more > > info on the software requirements for ZABBIX. > > > > Nils Breunese. > > > > > > _______________________________________________ > > Fedora-infrastructure-list mailing list > > Fedora-infrastructure-list at redhat.com > > https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list > > > > > > > > > > _______________________________________________ > Fedora-infrastructure-list mailing list > Fedora-infrastructure-list at redhat.com > https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list > -------------- next part -------------- An HTML attachment was scrubbed... URL: From damian.myerscough at gmail.com Thu Dec 7 11:01:57 2006 From: damian.myerscough at gmail.com (Damian Myerscough) Date: Thu, 7 Dec 2006 11:01:57 +0000 Subject: Web server torturing In-Reply-To: <7a41c4bc0612070224p7c00377arae14578b18b523b3@mail.gmail.com> References: <3da3b5b40612061443k6385e18ck871081b072b0386e@mail.gmail.com> <3237e4410612061602h21819dcco6c2466d351509fad@mail.gmail.com> <1165460860.5157.7.camel@lt21223.campus.dmacc.edu> <1165461173.25300.33.camel@cutter> <14BCFE43-7846-43F8-9B46-452E2CD2F5C7@breun.nl> <8c9e56490612070212k5d12c08cj837b74b2fdc91ea2@mail.gmail.com> <7a41c4bc0612070224p7c00377arae14578b18b523b3@mail.gmail.com> Message-ID: <8c9e56490612070301k707fe766k9666f2a462ae1065@mail.gmail.com> ok. On 07/12/06, Paulo Santos wrote: > Hi Damian, > > The script written by Ahmed, already uses ab. He is using the script to > gather some other information, besides the ab output. > Please take a look into webtorture.sh, and you'll see what i mean. > > > Paulo > > > > On 12/7/06, Damian Myerscough wrote: > > Hi, > > > > > 1- A script was written which uses apache's ab tool to stress the > server. Script will run on the web-server host. > > > > There is a tool available for benchmarking apache called: ab or ab2 > > which is apache benchmark to use it you simply issue: > > > > ./ab -n 1000 -c 100 > http://www.fedoraproject.org/wiki/index.htm > > > > The above will simulate 100 users accessing the website 1000 times > > > > On 07/12/06, Nils Breunese < nils at breun.nl> wrote: > > > seth vidal wrote: > > > > > > > On Wed, 2006-12-06 at 21:07 -0600, Jeffrey C. Ollie wrote: > > > >> On Wed, 2006-12-06 at 18:02 -0600, Mike McGrath wrote: > > > >>> Before we get started I'm hoping to get > > > >>> cacti set up for a good baseline. > > > >> > > > >> Not to denigrate Cacti but you might want to take a look at > > > >> Zabbix... It > > > >> was recently included in FE (I did the review). It's a little bit > > > >> trickier to set up but the polling is done using a server & agent > > > >> written in C, as opposed to Cacti which is largely PHP. I've been > > > >> working on a Zabbix install at work and I poll a number of variables > > > >> every 5 sec. > > > > > > > > cacti in php doesn't matter. It's the snmp interface in cacti which > > > > does > > > > the tricks and that's all done using snmp libs. It also has the > > > > advantage of just being another use of snmp instead of being it's own > > > > whole thing. > > > > > > ZABBIX also uses the snmp-libs for the optional SNMP agents. I use > > > ZABBIX to monitor all of my servers and it suits me pretty nicely. > > > The ZABBIX web frontend is in PHP, just like Cacti's. See http:// > > > > www.zabbix.com/manual/v1.1/install_requirements_software.php > for more > > > info on the software requirements for ZABBIX. > > > > > > Nils Breunese. > > > > > > > > > _______________________________________________ > > > Fedora-infrastructure-list mailing list > > > Fedora-infrastructure-list at redhat.com > > > > https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list > > > > > > > > > > > > > > > > _______________________________________________ > > Fedora-infrastructure-list mailing list > > Fedora-infrastructure-list at redhat.com > > > https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list > > > > From email.ahmedkamal at googlemail.com Thu Dec 7 14:07:33 2006 From: email.ahmedkamal at googlemail.com (Ahmed Kamal) Date: Thu, 7 Dec 2006 16:07:33 +0200 Subject: Web server torturing In-Reply-To: <8c9e56490612070301k707fe766k9666f2a462ae1065@mail.gmail.com> References: <3da3b5b40612061443k6385e18ck871081b072b0386e@mail.gmail.com> <3237e4410612061602h21819dcco6c2466d351509fad@mail.gmail.com> <1165460860.5157.7.camel@lt21223.campus.dmacc.edu> <1165461173.25300.33.camel@cutter> <14BCFE43-7846-43F8-9B46-452E2CD2F5C7@breun.nl> <8c9e56490612070212k5d12c08cj837b74b2fdc91ea2@mail.gmail.com> <7a41c4bc0612070224p7c00377arae14578b18b523b3@mail.gmail.com> <8c9e56490612070301k707fe766k9666f2a462ae1065@mail.gmail.com> Message-ID: <3da3b5b40612070607p7ef3ffffq6e7c916f0188ab5e@mail.gmail.com> AFAIK, cacti uses the php interface to net-snmp libs in their poller code, so they are a bit slower. For that they made another poller in C just for performance. http://www.cacti.net/cactid_info.php Stacy, all tests planned will be on the LAN only. No incoming bandwidth will be used (not that I have the kind of bandwidth that would stress the servers anyway ;) Anyway, as Mike said, we will of course be co-ordinating with you. Anyway, please let's dont let the thread slip into a Zabbix-vs-Cacti debate, since it's difficult to get a chance to stress test the servers, I hope to get all relevant information the first time. Kindly, do let me know your suggestions regarding: - Test setup/conditions - Interesting apache configuration options we might need to test under (currently no config sweeping is planned) - A kind of load monitoring tool we could/should use (other than "top") Best Regards On 12/7/06, Damian Myerscough wrote: > > ok. > > On 07/12/06, Paulo Santos wrote: > > Hi Damian, > > > > The script written by Ahmed, already uses ab. He is using the script to > > gather some other information, besides the ab output. > > Please take a look into webtorture.sh, and you'll see what i mean. > > > > > > Paulo > > > > > > > > On 12/7/06, Damian Myerscough wrote: > > > Hi, > > > > > > > 1- A script was written which uses apache's ab tool to stress the > > server. Script will run on the web-server host. > > > > > > There is a tool available for benchmarking apache called: ab or ab2 > > > which is apache benchmark to use it you simply issue: > > > > > > ./ab -n 1000 -c 100 > > http://www.fedoraproject.org/wiki/index.htm > > > > > > The above will simulate 100 users accessing the website 1000 times > > > > > > On 07/12/06, Nils Breunese < nils at breun.nl> wrote: > > > > seth vidal wrote: > > > > > > > > > On Wed, 2006-12-06 at 21:07 -0600, Jeffrey C. Ollie wrote: > > > > >> On Wed, 2006-12-06 at 18:02 -0600, Mike McGrath wrote: > > > > >>> Before we get started I'm hoping to get > > > > >>> cacti set up for a good baseline. > > > > >> > > > > >> Not to denigrate Cacti but you might want to take a look at > > > > >> Zabbix... It > > > > >> was recently included in FE (I did the review). It's a little > bit > > > > >> trickier to set up but the polling is done using a server & agent > > > > >> written in C, as opposed to Cacti which is largely PHP. I've > been > > > > >> working on a Zabbix install at work and I poll a number of > variables > > > > >> every 5 sec. > > > > > > > > > > cacti in php doesn't matter. It's the snmp interface in cacti > which > > > > > does > > > > > the tricks and that's all done using snmp libs. It also has the > > > > > advantage of just being another use of snmp instead of being it's > own > > > > > whole thing. > > > > > > > > ZABBIX also uses the snmp-libs for the optional SNMP agents. I use > > > > ZABBIX to monitor all of my servers and it suits me pretty nicely. > > > > The ZABBIX web frontend is in PHP, just like Cacti's. See http:// > > > > > > www.zabbix.com/manual/v1.1/install_requirements_software.php > > for more > > > > info on the software requirements for ZABBIX. > > > > > > > > Nils Breunese. > > > > > > > > > > > > _______________________________________________ > > > > Fedora-infrastructure-list mailing list > > > > Fedora-infrastructure-list at redhat.com > > > > > > https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > Fedora-infrastructure-list mailing list > > > Fedora-infrastructure-list at redhat.com > > > > > https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list > > > > > > > > > _______________________________________________ > Fedora-infrastructure-list mailing list > Fedora-infrastructure-list at redhat.com > https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeff at ocjtech.us Thu Dec 7 14:18:44 2006 From: jeff at ocjtech.us (Jeffrey C. Ollie) Date: Thu, 07 Dec 2006 08:18:44 -0600 Subject: Web server torturing In-Reply-To: <3da3b5b40612070607p7ef3ffffq6e7c916f0188ab5e@mail.gmail.com> References: <3da3b5b40612061443k6385e18ck871081b072b0386e@mail.gmail.com> <3237e4410612061602h21819dcco6c2466d351509fad@mail.gmail.com> <1165460860.5157.7.camel@lt21223.campus.dmacc.edu> <1165461173.25300.33.camel@cutter> <14BCFE43-7846-43F8-9B46-452E2CD2F5C7@breun.nl> <8c9e56490612070212k5d12c08cj837b74b2fdc91ea2@mail.gmail.com> <7a41c4bc0612070224p7c00377arae14578b18b523b3@mail.gmail.com> <8c9e56490612070301k707fe766k9666f2a462ae1065@mail.gmail.com> <3da3b5b40612070607p7ef3ffffq6e7c916f0188ab5e@mail.gmail.com> Message-ID: <1165501124.3535.7.camel@lt21223.campus.dmacc.edu> On Thu, 2006-12-07 at 16:07 +0200, Ahmed Kamal wrote: > > - A kind of load monitoring tool we could/should use (other than > "top") If we could coordinate things with the crew at the hosting site, it would be interesting to capture all of the packets between the server and clients during the test. There are some tools in wireshark that would be useful for analyzing the results of the test. For best performance you'd want to have a separate system capturing traffic that was being mirrored by the switch. Second best would be to dump the packet captures to the web server's disk, but I would think that the I/O caused by that would affect the test. Jeff -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From email.ahmedkamal at googlemail.com Thu Dec 7 14:36:57 2006 From: email.ahmedkamal at googlemail.com (Ahmed Kamal) Date: Thu, 7 Dec 2006 16:36:57 +0200 Subject: Web server torturing In-Reply-To: <1165501124.3535.7.camel@lt21223.campus.dmacc.edu> References: <3da3b5b40612061443k6385e18ck871081b072b0386e@mail.gmail.com> <3237e4410612061602h21819dcco6c2466d351509fad@mail.gmail.com> <1165460860.5157.7.camel@lt21223.campus.dmacc.edu> <1165461173.25300.33.camel@cutter> <14BCFE43-7846-43F8-9B46-452E2CD2F5C7@breun.nl> <8c9e56490612070212k5d12c08cj837b74b2fdc91ea2@mail.gmail.com> <7a41c4bc0612070224p7c00377arae14578b18b523b3@mail.gmail.com> <8c9e56490612070301k707fe766k9666f2a462ae1065@mail.gmail.com> <3da3b5b40612070607p7ef3ffffq6e7c916f0188ab5e@mail.gmail.com> <1165501124.3535.7.camel@lt21223.campus.dmacc.edu> Message-ID: <3da3b5b40612070636w29feada0l2ccb865b5936be49@mail.gmail.com> Please note that there will only be one client connecting to the web-server. I guess (if we can't get a separate dump machine), it would be better to dump on the client machine, than on the server machine where the disk will probably be active. OTOH, I'm not sure what kind of information would be interesting to us from a packet dump in this case? On 12/7/06, Jeffrey C. Ollie wrote: > > On Thu, 2006-12-07 at 16:07 +0200, Ahmed Kamal wrote: > > > > - A kind of load monitoring tool we could/should use (other than > > "top") > > If we could coordinate things with the crew at the hosting site, it > would be interesting to capture all of the packets between the server > and clients during the test. There are some tools in wireshark that > would be useful for analyzing the results of the test. For best > performance you'd want to have a separate system capturing traffic that > was being mirrored by the switch. Second best would be to dump the > packet captures to the web server's disk, but I would think that the I/O > caused by that would affect the test. > > Jeff > > > > _______________________________________________ > Fedora-infrastructure-list mailing list > Fedora-infrastructure-list at redhat.com > https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dennis at ausil.us Thu Dec 7 15:29:04 2006 From: dennis at ausil.us (Dennis Gilmore) Date: Thu, 7 Dec 2006 09:29:04 -0600 Subject: Web server torturing In-Reply-To: <3da3b5b40612061443k6385e18ck871081b072b0386e@mail.gmail.com> References: <3da3b5b40612061443k6385e18ck871081b072b0386e@mail.gmail.com> Message-ID: <200612070929.05177.dennis@ausil.us> On Wednesday 06 December 2006 16:43, Ahmed Kamal wrote: > Hi, > Due to the web server hit faced with FC6 release, some actions are being > taken to minimize the chance of facing such issues again. One of the steps > is to stress test our web-server infrastructure to measure our current and > future capabilities. I'd like to run some tests on fp.o web-server, please > let me know your thoughts/comments. Here are some details. > > Test Targets: > > 1- Measure our bare (no caching) maximum serving rate > 2- Measure our cached serving rate, to assess the implemented caching > efficiency > 3- Gather numbers like (When do we get CPU saturated, RAM requirements ..). > Possibly draw graphs (everyone thinks graphs are cool), the numbers should > help us base future calculations on a solid basis > 4- Future: Possibly implement a mechanism to cap the maximum connected > clients to a specific server, to the maximum it can handle gracefully, to > avoid killing a server I think this is great. I think to get things done right we will need to do it in a distributed manner. > Test Plan: > 1- A script was written which uses apache's ab tool to stress the server. > Script will run on the web-server host. Is that a fair test? i tired the script on a box i have which while very different to the boxes in use i could not get it to break a sweat. admittedly i was only serving the default welcome page. I will try it again with a wiki setup and see how it goes then > 2- The script fires a total number of connections equal to ten times the > maximum concurrency rate (to get good average, and avoid transients) > 3- The concurrency rate is sweeped between 10 and 400 (my 1G-RAM machine > swaps at about 100 connections)? any suggestions? i had up to 2000 connections in an effort to get thinks choked up and did it in steps of 100 > 4- All ab output is recorded for future analysis I had some that did not get captured for some reason (1900 and 2000) also i was left with alot of top processes running > 5- A monitoring thread is fired before ab is launched. The monitoring uses > "top" to record load/cpu/ram/process information in log files as well > 6- Tests are repeated with "ab -k" for enabling the HTTP keep-alive option. > Not sure if this is needed, or if it will make much difference! comments? > 7- Tests are done once with caching enabled and one more time without > caching > > Please let me know your thoughts about the testing setup, should we be > recording more data? should we be stressing the server in a different way, > should we be testing some apache config options ... etc ? > Thanks -- ?,-._|\ ? ?Dennis Gilmore, RHCE / ? ? ?\ ? Proud Australian \_.--._/ ? | Aurora | Fedora | ? ? ? v ? ? From paulo.banon at googlemail.com Thu Dec 7 15:36:17 2006 From: paulo.banon at googlemail.com (Paulo Santos) Date: Thu, 7 Dec 2006 15:36:17 +0000 Subject: Web server torturing In-Reply-To: <200612070929.05177.dennis@ausil.us> References: <3da3b5b40612061443k6385e18ck871081b072b0386e@mail.gmail.com> <200612070929.05177.dennis@ausil.us> Message-ID: <7a41c4bc0612070736s68de3e9dj5882c70985666b32@mail.gmail.com> Mike just deployed cacti on the folowing hosts: - proxy1 - proxy2 - app1 - app2 I think we can drop top for now. What you guys think ? Paulo On 12/7/06, Dennis Gilmore wrote: > > On Wednesday 06 December 2006 16:43, Ahmed Kamal wrote: > > Hi, > > Due to the web server hit faced with FC6 release, some actions are being > > taken to minimize the chance of facing such issues again. One of the > steps > > is to stress test our web-server infrastructure to measure our current > and > > future capabilities. I'd like to run some tests on fp.o web-server, > please > > let me know your thoughts/comments. Here are some details. > > > > Test Targets: > > > > 1- Measure our bare (no caching) maximum serving rate > > 2- Measure our cached serving rate, to assess the implemented caching > > efficiency > > 3- Gather numbers like (When do we get CPU saturated, RAM requirements > ..). > > Possibly draw graphs (everyone thinks graphs are cool), the numbers > should > > help us base future calculations on a solid basis > > 4- Future: Possibly implement a mechanism to cap the maximum connected > > clients to a specific server, to the maximum it can handle gracefully, > to > > avoid killing a server > I think this is great. I think to get things done right we will need to > do it > in a distributed manner. > > > Test Plan: > > 1- A script was written which uses apache's ab tool to stress the > server. > > Script will run on the web-server host. > Is that a fair test? i tired the script on a box i have which while very > different to the boxes in use i could not get it to break a sweat. > admittedly i was only serving the default welcome page. I will try it > again > with a wiki setup and see how it goes then > > > 2- The script fires a total number of connections equal to ten times the > > maximum concurrency rate (to get good average, and avoid transients) > > 3- The concurrency rate is sweeped between 10 and 400 (my 1G-RAM machine > > swaps at about 100 connections)? any suggestions? > i had up to 2000 connections in an effort to get thinks choked up and did > it > in steps of 100 > > > 4- All ab output is recorded for future analysis > I had some that did not get captured for some reason (1900 and 2000) also > i > was left with alot of top processes running > > 5- A monitoring thread is fired before ab is launched. The monitoring > uses > > "top" to record load/cpu/ram/process information in log files as well > > 6- Tests are repeated with "ab -k" for enabling the HTTP keep-alive > option. > > Not sure if this is needed, or if it will make much difference! > comments? > > 7- Tests are done once with caching enabled and one more time without > > caching > > > > Please let me know your thoughts about the testing setup, should we be > > recording more data? should we be stressing the server in a different > way, > > should we be testing some apache config options ... etc ? > > Thanks > > -- > ,-._|\ Dennis Gilmore, RHCE > / \ Proud Australian > \_.--._/ | Aurora | Fedora | > v > > _______________________________________________ > Fedora-infrastructure-list mailing list > Fedora-infrastructure-list at redhat.com > https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lmacken at redhat.com Thu Dec 7 15:46:06 2006 From: lmacken at redhat.com (Luke Macken) Date: Thu, 7 Dec 2006 10:46:06 -0500 Subject: Meetings Message-ID: <20061207154606.GE4290@tomservo.rh.rit.edu> Hey guys, Due to poor scheduling[0] on my part, I will not be able to make any of the meetings for the next 10 weeks. I just couldn't resist and had to sign up for a class on Zen Buddhism at that time. As far as things are going now, I've been doing much hackery on the new UpdatesSystem[1], and should have a functional system shortly. Right now I have a local tree of test builds which I am using to push out to a test-stage. After I write up a bunch of nose tests for it, I'll commit a package or so from the test tree as well. I poked around at publictest2 last night, which doesn't seem to be public at all. I also was not able to get any traffic out. If anyone has a free second, could you take a look at this? Thanks, luke [0]: http://schedule.csh.rit.edu/?s=JxWp7Uj [1]: http://fedoraproject.org/wiki/Infrastructure/UpdatesSystem From email.ahmedkamal at googlemail.com Thu Dec 7 17:20:59 2006 From: email.ahmedkamal at googlemail.com (Ahmed Kamal) Date: Thu, 7 Dec 2006 19:20:59 +0200 Subject: Web server torturing In-Reply-To: <200612070929.05177.dennis@ausil.us> References: <3da3b5b40612061443k6385e18ck871081b072b0386e@mail.gmail.com> <200612070929.05177.dennis@ausil.us> Message-ID: <3da3b5b40612070920i55d1c945pc22491bdc21fe9a7@mail.gmail.com> > Test Plan: > > 1- A script was written which uses apache's ab tool to stress the > server. > > Script will run on the web-server host. > Is that a fair test? i tired the script on a box i have which while very > different to the boxes in use i could not get it to break a sweat. > admittedly i was only serving the default welcome page. I will try it > again > with a wiki setup and see how it goes then Changing the original plan. The script should run from a client host (not the web server). I think one client host is enough to hammer the server. Using multi-hosts will only complicate data gathering un-necessarily Serving static pages is waaay faster. Just try it on a wiki/cms style server. My 2Ghz CPU is at 100% just at 10 concurrent connections to joomla! I had some that did not get captured for some reason (1900 and 2000) also i > was left with alot of top processes running umm ok, will try to fix this -------------- next part -------------- An HTML attachment was scrubbed... URL: From email.ahmedkamal at googlemail.com Thu Dec 7 17:31:00 2006 From: email.ahmedkamal at googlemail.com (Ahmed Kamal) Date: Thu, 7 Dec 2006 19:31:00 +0200 Subject: Web server torturing In-Reply-To: <7a41c4bc0612070736s68de3e9dj5882c70985666b32@mail.gmail.com> References: <3da3b5b40612061443k6385e18ck871081b072b0386e@mail.gmail.com> <200612070929.05177.dennis@ausil.us> <7a41c4bc0612070736s68de3e9dj5882c70985666b32@mail.gmail.com> Message-ID: <3da3b5b40612070931l7e724e77s72a63d1048d4ee20@mail.gmail.com> I guess I agree, if I am not running the script on the web-server, top is useless. Does anyone know how fast cacti polls the server, we should need something < 5 sec I guess? Also, it would have been nice to get per process swap/ram info, guess cacti cant do that. Anyway, I can still launch one long-running top job as well as cacti. On 12/7/06, Paulo Santos wrote: > > Mike just deployed cacti on the folowing hosts: > > - proxy1 > - proxy2 > - app1 > - app2 > > I think we can drop top for now. What you guys think ? > > > Paulo > > > On 12/7/06, Dennis Gilmore wrote: > > > > On Wednesday 06 December 2006 16:43, Ahmed Kamal wrote: > > > Hi, > > > Due to the web server hit faced with FC6 release, some actions are > > being > > > taken to minimize the chance of facing such issues again. One of the > > steps > > > is to stress test our web-server infrastructure to measure our current > > and > > > future capabilities. I'd like to run some tests on fp.o web-server, > > please > > > let me know your thoughts/comments. Here are some details. > > > > > > Test Targets: > > > > > > 1- Measure our bare (no caching) maximum serving rate > > > 2- Measure our cached serving rate, to assess the implemented caching > > > efficiency > > > 3- Gather numbers like (When do we get CPU saturated, RAM requirements > > ..). > > > Possibly draw graphs (everyone thinks graphs are cool), the numbers > > should > > > help us base future calculations on a solid basis > > > 4- Future: Possibly implement a mechanism to cap the maximum connected > > > clients to a specific server, to the maximum it can handle gracefully, > > to > > > avoid killing a server > > I think this is great. I think to get things done right we will need to > > do it > > in a distributed manner. > > > > > Test Plan: > > > 1- A script was written which uses apache's ab tool to stress the > > server. > > > Script will run on the web-server host. > > Is that a fair test? i tired the script on a box i have which while > > very > > different to the boxes in use i could not get it to break a sweat. > > admittedly i was only serving the default welcome page. I will try it > > again > > with a wiki setup and see how it goes then > > > > > 2- The script fires a total number of connections equal to ten times > > the > > > maximum concurrency rate (to get good average, and avoid transients) > > > 3- The concurrency rate is sweeped between 10 and 400 (my 1G-RAM > > machine > > > swaps at about 100 connections)? any suggestions? > > i had up to 2000 connections in an effort to get thinks choked up and > > did it > > in steps of 100 > > > > > 4- All ab output is recorded for future analysis > > I had some that did not get captured for some reason (1900 and > > 2000) also i > > was left with alot of top processes running > > > 5- A monitoring thread is fired before ab is launched. The monitoring > > uses > > > "top" to record load/cpu/ram/process information in log files as well > > > 6- Tests are repeated with "ab -k" for enabling the HTTP keep-alive > > option. > > > Not sure if this is needed, or if it will make much difference! > > comments? > > > 7- Tests are done once with caching enabled and one more time without > > > caching > > > > > > Please let me know your thoughts about the testing setup, should we be > > > recording more data? should we be stressing the server in a different > > way, > > > should we be testing some apache config options ... etc ? > > > Thanks > > > > -- > > ,-._|\ Dennis Gilmore, RHCE > > / \ Proud Australian > > \_.--._/ | Aurora | Fedora | > > v > > > > _______________________________________________ > > Fedora-infrastructure-list mailing list > > Fedora-infrastructure-list at redhat.com > > https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list > > > > > _______________________________________________ > Fedora-infrastructure-list mailing list > Fedora-infrastructure-list at redhat.com > https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeff at ocjtech.us Thu Dec 7 17:58:22 2006 From: jeff at ocjtech.us (Jeffrey C. Ollie) Date: Thu, 07 Dec 2006 11:58:22 -0600 Subject: Web server torturing In-Reply-To: <3da3b5b40612070931l7e724e77s72a63d1048d4ee20@mail.gmail.com> References: <3da3b5b40612061443k6385e18ck871081b072b0386e@mail.gmail.com> <200612070929.05177.dennis@ausil.us> <7a41c4bc0612070736s68de3e9dj5882c70985666b32@mail.gmail.com> <3da3b5b40612070931l7e724e77s72a63d1048d4ee20@mail.gmail.com> Message-ID: <1165514302.3535.21.camel@lt21223.campus.dmacc.edu> On Thu, 2006-12-07 at 19:31 +0200, Ahmed Kamal wrote: > > Does anyone know how fast cacti polls the server, we should need > something < 5 sec I guess? Cacti by default polls every 5 minutes. It's adjustable, but unfortunately the polling is controlled by cron so at best you could do 1 minute intervals. > Also, it would have been nice to get per process swap/ram info, guess > cacti cant do that. There's some stuff in the host resources SNMP mib that will show you the total amount of memory allocated to a process, but it isn't separated into separate resident/virtual set numbers. > Anyway, I can still launch one long-running top job as well as cacti. Can't hurt... Jeff -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From linux at elfshadow.net Thu Dec 7 20:52:29 2006 From: linux at elfshadow.net (Jeffrey Tadlock) Date: Thu, 07 Dec 2006 15:52:29 -0500 Subject: IRC Meeting Log Posted - 12/07/2006 Message-ID: <45787F0D.3010404@elfshadow.net> The notes from today's Infrastructure meeting on IRC have been posted to the Wiki: http://fedoraproject.org/wiki/Infrastructure/Meetings/2006-12-07 --Jeffrey From jeff at ocjtech.us Thu Dec 7 21:38:22 2006 From: jeff at ocjtech.us (Jeffrey C. Ollie) Date: Thu, 07 Dec 2006 15:38:22 -0600 Subject: OpenID Libraries Message-ID: <1165527502.21380.12.camel@lt21223.campus.dmacc.edu> I've submitted for review a Python library for OpenID and some dependencies that weren't already in FE. Here are the relevant links: python-pycurl https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=218787 python-urljr https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=218831 pyflakes https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=218839 python-yadis https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=218844 python-openid https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=218852 Jeff -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From skvidal at linux.duke.edu Thu Dec 7 16:11:03 2006 From: skvidal at linux.duke.edu (seth vidal) Date: Thu, 07 Dec 2006 11:11:03 -0500 Subject: Glump in Fedora Extras? In-Reply-To: <1165387240.8726.4.camel@lt21223.campus.dmacc.edu> References: <1165335577.3408.12.camel@lt21223.campus.dmacc.edu> <1165387240.8726.4.camel@lt21223.campus.dmacc.edu> Message-ID: <1165507863.25300.95.camel@cutter> On Wed, 2006-12-06 at 00:40 -0600, Jeffrey C. Ollie wrote: > On Tue, 2006-12-05 at 10:19 -0600, Jeffrey C. Ollie wrote: > > It looks like Glump might be the perfect tool for a project here at > > work. Is there anyone @ Duke that wanted to maintain the package in > > Fedora Extras? If not, I'm willing to be the maintainer. > > https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=218577 > It shouldn't be heavy to maintain. I'll see about putting up a review. thanks, -sv From linux at elfshadow.net Sun Dec 10 13:29:12 2006 From: linux at elfshadow.net (Jeffrey Tadlock) Date: Sun, 10 Dec 2006 08:29:12 -0500 Subject: publictest2 net traffic In-Reply-To: <20061207154606.GE4290@tomservo.rh.rit.edu> References: <20061207154606.GE4290@tomservo.rh.rit.edu> Message-ID: <457C0BA8.3030004@elfshadow.net> Luke Macken wrote: > I poked around at publictest2 last night, which doesn't seem to be > public at all. I also was not able to get any traffic out. If anyone > has a free second, could you take a look at this? It looks like the default gateway was set incorrectly on publictest2. I corrected that and it seems to be working fine now. Since it's public, I also installed denyhosts and set it to start automatically. I also set sshd to only accept public key logins for now. --Jeffrey From nils at breun.nl Sun Dec 10 13:38:14 2006 From: nils at breun.nl (Nils Breunese) Date: Sun, 10 Dec 2006 14:38:14 +0100 Subject: publictest2 net traffic In-Reply-To: <457C0BA8.3030004@elfshadow.net> References: <20061207154606.GE4290@tomservo.rh.rit.edu> <457C0BA8.3030004@elfshadow.net> Message-ID: <0B359D3D-476E-4D4E-8FBE-DA560B6EDC67@breun.nl> Jeffrey Tadlock wrote: > Luke Macken wrote: >> I poked around at publictest2 last night, which doesn't seem to be >> public at all. I also was not able to get any traffic out. If >> anyone >> has a free second, could you take a look at this? > > It looks like the default gateway was set incorrectly on > publictest2. I > corrected that and it seems to be working fine now. > > Since it's public, I also installed denyhosts and set it to start > automatically. I also set sshd to only accept public key logins > for now. I used to run DenyHosts on all my servers, but switched to Fail2ban [0] because it creates (temporary) iptables rules instead of entries in /etc/hosts.deny. Is there any particular reason you're chosing DenyHosts over others? Do you use it's synchronization feature? Nils. [0] http://fail2ban.sourceforge.net/ -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 186 bytes Desc: Dit deel van het bericht is digitaal ondertekend URL: From mmcgrath at fedoraproject.org Sun Dec 10 16:03:51 2006 From: mmcgrath at fedoraproject.org (Mike McGrath) Date: Sun, 10 Dec 2006 10:03:51 -0600 Subject: publictest2 net traffic In-Reply-To: <0B359D3D-476E-4D4E-8FBE-DA560B6EDC67@breun.nl> References: <20061207154606.GE4290@tomservo.rh.rit.edu> <457C0BA8.3030004@elfshadow.net> <0B359D3D-476E-4D4E-8FBE-DA560B6EDC67@breun.nl> Message-ID: <3237e4410612100803g142e9385xf1ecbab89845ba5a@mail.gmail.com> On 12/10/06, Nils Breunese wrote: > Jeffrey Tadlock wrote: > I used to run DenyHosts on all my servers, but switched to Fail2ban > [0] because it creates (temporary) iptables rules instead of entries > in /etc/hosts.deny. Is there any particular reason you're chosing > DenyHosts over others? > DenyHosts is already in extras. -Mike From nils at breun.nl Sun Dec 10 16:49:35 2006 From: nils at breun.nl (Nils Breunese) Date: Sun, 10 Dec 2006 17:49:35 +0100 Subject: publictest2 net traffic In-Reply-To: <3237e4410612100803g142e9385xf1ecbab89845ba5a@mail.gmail.com> References: <20061207154606.GE4290@tomservo.rh.rit.edu> <457C0BA8.3030004@elfshadow.net> <0B359D3D-476E-4D4E-8FBE-DA560B6EDC67@breun.nl> <3237e4410612100803g142e9385xf1ecbab89845ba5a@mail.gmail.com> Message-ID: <3CBD2EDD-9648-4C4C-B688-34D9086C8BCD@breun.nl> Mike McGrath wrote: > On 12/10/06, Nils Breunese wrote: >> Jeffrey Tadlock wrote: >> I used to run DenyHosts on all my servers, but switched to Fail2ban >> [0] because it creates (temporary) iptables rules instead of entries >> in /etc/hosts.deny. Is there any particular reason you're chosing >> DenyHosts over others? > > DenyHosts is already in extras. I have never packaged anything, but maybe someone could package Fail2ban for Extras? They already create their own rpms (http:// fail2ban.sourceforge.net/rpms/) that work just fine on my Fedora and CentOS boxes. Nils Breunese. -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 186 bytes Desc: Dit deel van het bericht is digitaal ondertekend URL: From lmacken at redhat.com Sun Dec 10 20:29:00 2006 From: lmacken at redhat.com (Luke Macken) Date: Sun, 10 Dec 2006 15:29:00 -0500 Subject: publictest2 net traffic In-Reply-To: <457C0BA8.3030004@elfshadow.net> References: <20061207154606.GE4290@tomservo.rh.rit.edu> <457C0BA8.3030004@elfshadow.net> Message-ID: <20061210202900.GA26324@tomservo.rh.rit.edu> On Sun, Dec 10, 2006 at 08:29:12AM -0500, Jeffrey Tadlock wrote: > Luke Macken wrote: > > I poked around at publictest2 last night, which doesn't seem to be > > public at all. I also was not able to get any traffic out. If anyone > > has a free second, could you take a look at this? > > It looks like the default gateway was set incorrectly on publictest2. I > corrected that and it seems to be working fine now. > > Since it's public, I also installed denyhosts and set it to start > automatically. I also set sshd to only accept public key logins for now. Awesome, thanks for your help Jeffrey! luke From mmcgrath at fedoraproject.org Tue Dec 12 04:24:12 2006 From: mmcgrath at fedoraproject.org (Mike McGrath) Date: Mon, 11 Dec 2006 22:24:12 -0600 Subject: Fwd: [Ticket#2006121110000016] New ticket notification! (Fedora planet RSS [...]) In-Reply-To: <1165822515.739084.436695739@admin.fedoraproject.org> References: <1165822515.739084.436695739@admin.fedoraproject.org> Message-ID: <3237e4410612112024u6b1b9f16l25b003a598f7b13b@mail.gmail.com> Anyone familiar enough with our rss system that can take a look at this? If not I'll have a look tomorow. -Mike ---------- Forwarded message ---------- From: Fedora Ticket Notification Date: Dec 11, 2006 1:35 AM Subject: Re: [Ticket#2006121110000016] New ticket notification! (Fedora planet RSS [...]) To: mmcgrath at fedoraproject.org Hello, there is a new ticket in "Websites"! Aurelien Bompard wrote: > Hi, > > The Fedora Planet RSS feed seem to be broken : > http://fedoraproject.org/people/rss20.xml > > It seems to be broken by a link in David Lutterkort's entry > about "Kickstarting into puppet". > > There are errors using xmllint: > > $ xmllint rss20.xml > rss20.xml:647: parser error : EntityRef: expecting ';' > > http://watzmann.net/blog/index.php?title=kickstarting_into_puppet&more=1& > > ^ https://admin.fedoraproject.org/tickets/index.pl?Action=AgentZoom&TicketID=518 Users who would like to request access to be an agent in the ticketing system please see: http://fedoraproject.org/wiki/Infrastructure/Tickets Your OTRS Notification Master From lmacken at redhat.com Tue Dec 12 04:45:16 2006 From: lmacken at redhat.com (Luke Macken) Date: Mon, 11 Dec 2006 23:45:16 -0500 Subject: Fedora Updates System In-Reply-To: <20061124110958.2002c691.bugs.michael@gmx.net> References: <20061112214220.GE7378@tomservo.rh.rit.edu> <20061124110958.2002c691.bugs.michael@gmx.net> Message-ID: <20061212044516.GB26324@tomservo.rh.rit.edu> On Fri, Nov 24, 2006 at 11:09:58AM +0100, Michael Schwendt wrote: [...] > repoview > -------- > > Running it for all of Fedora Extras takes a lot of time. So long, that > even if you let the push script do its work in a background terminal, it > is still painful to see how long it takes to complete. > > For unknown reasons we also create repoview pages for the "debug" > repositories. If it were just my own decision, I would stop doing that, > since I doubt those web pages are popular enough. Who really browses > repoview for debuginfo packages? We should expect debuginfo packages to be > available for every relevant package in the repository. Yeah, repoview for debuginfo packages seems unnecessary. We don't even run repoview for updates{,-testing} at the moment, but we can easily integrate it with the new system if we want. > createrepo < 0.4.5 is unable to handle "unknown" files in its repodata > directory. Therefore it conflicts with repoview and runs into a fatal > error condition with a premature program termination, leaving behind a > temporary ".olddir". This is especially ugly, since an administrator would > need to recover from that manually and either move back files from > ".olddir" or delete it. But when deleted, the repoview tree is lost and is > created from scratch. I've been told that this is a problem for mirrors, > where the several thousands of files are re-examined for changes just > because of the fresh time stamps. So, as of a few weeks ago, the push > script works around that successfully with a repodata backup strategy > outside of createrepo. The old updates system code (and the fedora-updates-clean cronjob) has hacks around this, as well, when dealing with the updateinfo.xml.gz. I haven't checked, but how does createrepo >= 0.4.5 deal with unknown files now? > For createrepo and repoview to be run in the background as a scheduled > job, it needs a local lock on the repository. Most likely _not_ > fine-grained locking on every arch-specific sub-repository, because every > locking comes at a cost (especially when there are multiple jobs of > different priority waiting). Yeah, there definitely needs to be mutual exclusion with the updates staging repository. > repoclosure > ----------- > > Running this takes even more time. Currently, it examine all of Fedora > Extras + Core + Updates + Legacy in a background job after packages have > been published. The time it takes is approximately the difference between > the time stamps of the build report and the broken deps report. I'd like to run repoclosure on the updates before pushing them out at all (sure we have updates-testing, but I don't want that becoming another rawhide). I spoke with skvidal over the summer speeding up the closure process, and seem to remember mention of caching the provides/requires to gain some speed. We'll definitely have to look into this, as this process is indeed dreadfully slow. > > ''' Pushing ''' > > * Moves packages to proper updates stage > > More "stages" which are understood by plague would be good. We only use > one stage, needsign, which is the build-results repository known to the > build servers. Well, once we get the Brew situation figured out, we can use its detached gpg signatures to help with the push process. Whatever stages we want (pending/testing/needsign/pushed) should probably be implemented in the updates system itself, to allow the buildsystem to only worry about spitting out builds. Althought, my Brew-fu is weak, so we'll have to see what it offers once/if it is released. [..] > > * updates repo cleaner > > * remove old packages > > So far, the script I named repoprune is much faster than repomanage and > simplifies Fedora Extras repository maintenance a lot, since it gets rid > of orphaned and out-of-date sub-packages automatically, too. Awesome, I will see what I can do about integrating this tool into the new system. luke From lmacken at redhat.com Tue Dec 12 05:08:24 2006 From: lmacken at redhat.com (Luke Macken) Date: Tue, 12 Dec 2006 00:08:24 -0500 Subject: Fedora Updates System In-Reply-To: <20061112214220.GE7378@tomservo.rh.rit.edu> References: <20061112214220.GE7378@tomservo.rh.rit.edu> Message-ID: <20061212050824.GC26324@tomservo.rh.rit.edu> Hey guys, I updated the UpdatesSystem[0] wiki last night with a bunch of tasks that need to get done, and a couple of wishlist items. For those looking to help out, feel free to grab anything on there (even if my name is on it), or add something you would like to see, and start hacking. Don't hesitate to speak up on IRC or onlist with any question/comments/concerns. Development has been going fairly smooth (except for a fun little exception thrown after trying to bump SQLObject that I still need to investigate). Since the buildsystem stuff is currently in the air, I setup the development environment to use a LocalTest Buildsystem (see buildsys.py), which just points to a `pkg/ver/rel/arch` hierarchy for pulling in built updates. Once I polish up the push code, I'm going to write a bunch of test cases and commit my test-build tree as well, with a couple of RPMS to be able to run the tests and be able to push updates 'out of the box'. I'm not really in a rush to get this running on publictest2 at the moment, because you can get full functionality by checking out the code locally and running `./start-updatesssystem`. However, I will be giving publictest2 some love in the near future, as I plan on reading the 'Deployment' chapter in my TurboGears book very soon. luke [0]: http://fedoraproject.org/wiki/Infrastructure/UpdatesSystem From skvidal at linux.duke.edu Tue Dec 12 12:17:37 2006 From: skvidal at linux.duke.edu (seth vidal) Date: Tue, 12 Dec 2006 07:17:37 -0500 Subject: Fwd: [Ticket#2006121110000016] New ticket notification! (Fedora planet RSS [...]) In-Reply-To: <3237e4410612112024u6b1b9f16l25b003a598f7b13b@mail.gmail.com> References: <1165822515.739084.436695739@admin.fedoraproject.org> <3237e4410612112024u6b1b9f16l25b003a598f7b13b@mail.gmail.com> Message-ID: <1165925857.23462.1.camel@cutter> On Mon, 2006-12-11 at 22:24 -0600, Mike McGrath wrote: > Anyone familiar enough with our rss system that can take a look at > this? If not I'll have a look tomorow. > I looked at it. there's nothing that can be done. Some arbitrary feed has a garbage character which corrupts the whole feed in terms of the validator. There's no way to prune those in planet other than removing the feed from the configurations. Users of the feed have bitched about all sorts of things but they don't seem to understand how the feeds work. -sv From mmcgrath at fedoraproject.org Tue Dec 12 14:27:56 2006 From: mmcgrath at fedoraproject.org (Mike McGrath) Date: Tue, 12 Dec 2006 08:27:56 -0600 Subject: Fwd: [Ticket#2006121110000016] New ticket notification! (Fedora planet RSS [...]) In-Reply-To: <1165925857.23462.1.camel@cutter> References: <1165822515.739084.436695739@admin.fedoraproject.org> <3237e4410612112024u6b1b9f16l25b003a598f7b13b@mail.gmail.com> <1165925857.23462.1.camel@cutter> Message-ID: <3237e4410612120627o68483a96y4ada4ff0154ccb26@mail.gmail.com> On 12/12/06, seth vidal wrote: > On Mon, 2006-12-11 at 22:24 -0600, Mike McGrath wrote: > > Anyone familiar enough with our rss system that can take a look at > > this? If not I'll have a look tomorow. > > > > I looked at it. > > there's nothing that can be done. Some arbitrary feed has a garbage > character which corrupts the whole feed in terms of the validator. > There's no way to prune those in planet other than removing the feed > from the configurations. > > Users of the feed have bitched about all sorts of things but they don't > seem to understand how the feeds work. > > -sv Is there a way to provide contact information to the people using the feed so they can contact this person and have them correct it, even in future cases? -Mike From skvidal at linux.duke.edu Tue Dec 12 14:31:29 2006 From: skvidal at linux.duke.edu (seth vidal) Date: Tue, 12 Dec 2006 09:31:29 -0500 Subject: Fwd: [Ticket#2006121110000016] New ticket notification! (Fedora planet RSS [...]) In-Reply-To: <3237e4410612120627o68483a96y4ada4ff0154ccb26@mail.gmail.com> References: <1165822515.739084.436695739@admin.fedoraproject.org> <3237e4410612112024u6b1b9f16l25b003a598f7b13b@mail.gmail.com> <1165925857.23462.1.camel@cutter> <3237e4410612120627o68483a96y4ada4ff0154ccb26@mail.gmail.com> Message-ID: <1165933889.23462.11.camel@cutter> On Tue, 2006-12-12 at 08:27 -0600, Mike McGrath wrote: > On 12/12/06, seth vidal wrote: > > On Mon, 2006-12-11 at 22:24 -0600, Mike McGrath wrote: > > > Anyone familiar enough with our rss system that can take a look at > > > this? If not I'll have a look tomorow. > > > > > > > I looked at it. > > > > there's nothing that can be done. Some arbitrary feed has a garbage > > character which corrupts the whole feed in terms of the validator. > > There's no way to prune those in planet other than removing the feed > > from the configurations. > > > > Users of the feed have bitched about all sorts of things but they don't > > seem to understand how the feeds work. > > > > -sv > > Is there a way to provide contact information to the people using the > feed so they can contact this person and have them correct it, even in > future cases? The contact info is in the feed. Click on the link to the post and you'll get the person's blog site. If they have contact info there then that's all you need to do. However, I don't think we want to post feed contributors' email addresses onto the fedora people site, that would be, umm, bad. -sv From dlutter at redhat.com Tue Dec 12 19:08:34 2006 From: dlutter at redhat.com (David Lutterkort) Date: Tue, 12 Dec 2006 11:08:34 -0800 Subject: Fwd: [Ticket#2006121110000016] New ticket notification! (Fedora planet RSS [...]) In-Reply-To: <1165925857.23462.1.camel@cutter> References: <1165822515.739084.436695739@admin.fedoraproject.org> <3237e4410612112024u6b1b9f16l25b003a598f7b13b@mail.gmail.com> <1165925857.23462.1.camel@cutter> Message-ID: <1165950515.32758.6.camel@localhost.localdomain> On Tue, 2006-12-12 at 07:17 -0500, seth vidal wrote: > On Mon, 2006-12-11 at 22:24 -0600, Mike McGrath wrote: > > Anyone familiar enough with our rss system that can take a look at > > this? If not I'll have a look tomorow. > > > > I looked at it. > > there's nothing that can be done. Some arbitrary feed has a garbage > character which corrupts the whole feed in terms of the validator. Since I am the offending party in this case, I'd be very much interested in fixing things on my end; yet I am not convinced hte problem is on my end: > wget -O rss2.xml http://watzmann.net/blog/xmlsrv/rss2.php?blog=2 > xmllint --noout rss2.xml > echo $? 0 And visual inspection shows that in rss2.xml all the '&' are properly escaped as '&' David From bugs.michael at gmx.net Wed Dec 13 19:09:15 2006 From: bugs.michael at gmx.net (Michael Schwendt) Date: Wed, 13 Dec 2006 20:09:15 +0100 Subject: Fedora Updates System In-Reply-To: <20061212044516.GB26324@tomservo.rh.rit.edu> References: <20061112214220.GE7378@tomservo.rh.rit.edu> <20061124110958.2002c691.bugs.michael@gmx.net> <20061212044516.GB26324@tomservo.rh.rit.edu> Message-ID: <20061213200915.79a37ee3.bugs.michael@gmx.net> On Mon, 11 Dec 2006 23:45:16 -0500, Luke Macken wrote: > > createrepo < 0.4.5 > The old updates system code (and the fedora-updates-clean cronjob) has hacks > around this, as well, when dealing with the updateinfo.xml.gz. I > haven't checked, but how does createrepo >= 0.4.5 deal with unknown > files now? It moves old files and restores them when it's done. Still, with a CLI push-script, something like Ctrl+C would interrupt createrepo in an unwanted way and require manual cleanup. Hence a backup/restore mechanism outside of it is good. It seems createrepo 0.4.6 from 11-Aug-2006 is the only version after 0.4.4 (FC6). From mmcgrath at fedoraproject.org Wed Dec 13 20:29:21 2006 From: mmcgrath at fedoraproject.org (Mike McGrath) Date: Wed, 13 Dec 2006 14:29:21 -0600 Subject: Web server torturing In-Reply-To: <3da3b5b40612070931l7e724e77s72a63d1048d4ee20@mail.gmail.com> References: <3da3b5b40612061443k6385e18ck871081b072b0386e@mail.gmail.com> <200612070929.05177.dennis@ausil.us> <7a41c4bc0612070736s68de3e9dj5882c70985666b32@mail.gmail.com> <3da3b5b40612070931l7e724e77s72a63d1048d4ee20@mail.gmail.com> Message-ID: <3237e4410612131229t6178fb48ma44701d03e4abbd6@mail.gmail.com> On 12/7/06, Ahmed Kamal wrote: Are we ready to pick a date and time for the first test? -Mike From paulo.banon at googlemail.com Wed Dec 13 22:22:50 2006 From: paulo.banon at googlemail.com (Paulo Santos) Date: Wed, 13 Dec 2006 22:22:50 +0000 Subject: Web server torturing In-Reply-To: <3237e4410612131229t6178fb48ma44701d03e4abbd6@mail.gmail.com> References: <3da3b5b40612061443k6385e18ck871081b072b0386e@mail.gmail.com> <200612070929.05177.dennis@ausil.us> <7a41c4bc0612070736s68de3e9dj5882c70985666b32@mail.gmail.com> <3da3b5b40612070931l7e724e77s72a63d1048d4ee20@mail.gmail.com> <3237e4410612131229t6178fb48ma44701d03e4abbd6@mail.gmail.com> Message-ID: <7a41c4bc0612131422r56a1e7dcv53b60fc65fe02642@mail.gmail.com> I am ready from my side. Paulo On 12/13/06, Mike McGrath wrote: > > On 12/7/06, Ahmed Kamal wrote: > > Are we ready to pick a date and time for the first test? > > -Mike > > _______________________________________________ > Fedora-infrastructure-list mailing list > Fedora-infrastructure-list at redhat.com > https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mmcgrath at fedoraproject.org Thu Dec 14 04:23:47 2006 From: mmcgrath at fedoraproject.org (Mike McGrath) Date: Wed, 13 Dec 2006 22:23:47 -0600 Subject: Web server torturing In-Reply-To: <7a41c4bc0612131422r56a1e7dcv53b60fc65fe02642@mail.gmail.com> References: <3da3b5b40612061443k6385e18ck871081b072b0386e@mail.gmail.com> <200612070929.05177.dennis@ausil.us> <7a41c4bc0612070736s68de3e9dj5882c70985666b32@mail.gmail.com> <3da3b5b40612070931l7e724e77s72a63d1048d4ee20@mail.gmail.com> <3237e4410612131229t6178fb48ma44701d03e4abbd6@mail.gmail.com> <7a41c4bc0612131422r56a1e7dcv53b60fc65fe02642@mail.gmail.com> Message-ID: <3237e4410612132023h453999f2l39daf2fe28df1cdf@mail.gmail.com> On 12/13/06, Paulo Santos wrote: > I am ready from my side. > > > Paulo > Someone in IRC posted this link tonight. Its pretty useful - http://www.ircache.net/cgi-bin/cacheability.py?query=http%3A%2F%2Ffedoraproject.org%2Fwiki%2FFC6ReleaseSummary&descend=on -Mike From rob.kearey at gmail.com Thu Dec 14 04:41:51 2006 From: rob.kearey at gmail.com (Rob Kearey) Date: Thu, 14 Dec 2006 15:41:51 +1100 Subject: Web server torturing In-Reply-To: <3237e4410612132023h453999f2l39daf2fe28df1cdf@mail.gmail.com> References: <3da3b5b40612061443k6385e18ck871081b072b0386e@mail.gmail.com> <200612070929.05177.dennis@ausil.us> <7a41c4bc0612070736s68de3e9dj5882c70985666b32@mail.gmail.com> <3da3b5b40612070931l7e724e77s72a63d1048d4ee20@mail.gmail.com> <3237e4410612131229t6178fb48ma44701d03e4abbd6@mail.gmail.com> <7a41c4bc0612131422r56a1e7dcv53b60fc65fe02642@mail.gmail.com> <3237e4410612132023h453999f2l39daf2fe28df1cdf@mail.gmail.com> Message-ID: Can we do some side-by-side comparisons with lighttpd whilst we're at it? It'd be trivial to do at the same time. I've had good experience with lighttpd in the past, and it's in extras already. -- Rob K http://motk.blogspot.com From email.ahmedkamal at googlemail.com Thu Dec 14 09:51:10 2006 From: email.ahmedkamal at googlemail.com (Ahmed Kamal) Date: Thu, 14 Dec 2006 11:51:10 +0200 Subject: Web server torturing In-Reply-To: <3237e4410612131229t6178fb48ma44701d03e4abbd6@mail.gmail.com> References: <3da3b5b40612061443k6385e18ck871081b072b0386e@mail.gmail.com> <200612070929.05177.dennis@ausil.us> <7a41c4bc0612070736s68de3e9dj5882c70985666b32@mail.gmail.com> <3da3b5b40612070931l7e724e77s72a63d1048d4ee20@mail.gmail.com> <3237e4410612131229t6178fb48ma44701d03e4abbd6@mail.gmail.com> Message-ID: <3da3b5b40612140151nb2a4710wb613db5823fd4546@mail.gmail.com> Generally I am ready. I'm just facing some problems with my ISP, so I'd like to wait a couple of days On 12/13/06, Mike McGrath wrote: > > On 12/7/06, Ahmed Kamal wrote: > > Are we ready to pick a date and time for the first test? > > -Mike > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mmcgrath at fedoraproject.org Thu Dec 14 13:39:44 2006 From: mmcgrath at fedoraproject.org (Mike McGrath) Date: Thu, 14 Dec 2006 07:39:44 -0600 Subject: Web server torturing In-Reply-To: <3da3b5b40612140151nb2a4710wb613db5823fd4546@mail.gmail.com> References: <3da3b5b40612061443k6385e18ck871081b072b0386e@mail.gmail.com> <200612070929.05177.dennis@ausil.us> <7a41c4bc0612070736s68de3e9dj5882c70985666b32@mail.gmail.com> <3da3b5b40612070931l7e724e77s72a63d1048d4ee20@mail.gmail.com> <3237e4410612131229t6178fb48ma44701d03e4abbd6@mail.gmail.com> <3da3b5b40612140151nb2a4710wb613db5823fd4546@mail.gmail.com> Message-ID: <3237e4410612140539x945de43ha2f50a347ac35bb0@mail.gmail.com> On 12/14/06, Ahmed Kamal wrote: > Generally I am ready. I'm just facing some problems with my ISP, so I'd like > to wait a couple of days > Works for me, can you pick a start time, end time, and date so we can get Stacy's approval? -MIke From linux at elfshadow.net Fri Dec 15 18:03:34 2006 From: linux at elfshadow.net (Jeffrey Tadlock) Date: Fri, 15 Dec 2006 13:03:34 -0500 Subject: IRC Meeting Log Posted - 12/14/2006 Message-ID: <4582E376.40806@elfshadow.net> The notes from yesterday's Infrastructure meeting on IRC have been posted to the Wiki: http://fedoraproject.org/wiki/Infrastructure/Meetings/2006-12-14 --Jeffrey From wtogami at redhat.com Fri Dec 15 21:39:49 2006 From: wtogami at redhat.com (Warren Togami) Date: Fri, 15 Dec 2006 16:39:49 -0500 Subject: gary_lerhaupt@dell.com Re: Undelivered Mail Returned to Sender In-Reply-To: <20061215213305.1FA9D15213F@buildsys.fedoraproject.org> References: <20061215213305.1FA9D15213F@buildsys.fedoraproject.org> Message-ID: <45831625.9050202@redhat.com> Matt, is this person no longer at Dell? Who do we reassign the package ownership to? Warren Togami wtogami at redhat.com Mail Delivery System wrote: > This is the Postfix program at host buildsys.fedoraproject.org. > > I'm sorry to have to inform you that your message could not > be delivered to one or more recipients. It's attached below. > > For further assistance, please send mail to > > If you do so, please include this problem report. You can > delete your own text from the attached returned message. > > The Postfix program > > : host smtp.ins.dell.com[143.166.83.183] said: 501 > #5.1.1 bad address gary_lerhaupt at dell.com (in reply to RCPT TO command) > > > ------------------------------------------------------------------------ > > Reporting-MTA: dns; buildsys.fedoraproject.org > X-Postfix-Queue-ID: AEB04152140 > X-Postfix-Sender: rfc822; buildsys at fedoraproject.org > Arrival-Date: Fri, 15 Dec 2006 16:32:59 -0500 (EST) > > Final-Recipient: rfc822; gary_lerhaupt at dell.com > Action: failed > Status: 5.0.0 > Diagnostic-Code: X-Postfix; host smtp.ins.dell.com[143.166.83.183] said: 501 > #5.1.1 bad address gary_lerhaupt at dell.com (in reply to RCPT TO command) > > > ------------------------------------------------------------------------ > > Subject: > Broken dependencies in Fedora Extras - 2006-12-15 > From: > Fedora Extras repoclosure > Date: > Fri, 15 Dec 2006 21:32:59 -0000 > To: > gary_lerhaupt at dell.com, bugs.michael at gmx.net > > To: > gary_lerhaupt at dell.com, bugs.michael at gmx.net > > > This is an automated mail created by an experimental script. > Your following packages in the repository contain broken dependencies: > > package: dkms - 2.0.13-1.fc6.noarch from fedora-extras-development-ppc > unresolved deps: > kernel-devel > From Matt_Domsch at dell.com Fri Dec 15 23:48:41 2006 From: Matt_Domsch at dell.com (Matt Domsch) Date: Fri, 15 Dec 2006 17:48:41 -0600 Subject: gary_lerhaupt@dell.com Re: Undelivered Mail Returned to Sender In-Reply-To: <45831625.9050202@redhat.com> References: <20061215213305.1FA9D15213F@buildsys.fedoraproject.org> <45831625.9050202@redhat.com> Message-ID: <20061215234841.GB23781@humbolt.us.dell.com> On Fri, Dec 15, 2006 at 04:39:49PM -0500, Warren Togami wrote: > Matt, is this person no longer at Dell? Who do we reassign the package > ownership to? I'm the owner, but he was on the cc:. I just fixed owners.list. Thanks, Matt -- Matt Domsch Software Architect Dell Linux Solutions linux.dell.com & www.dell.com/linux Linux on Dell mailing lists @ http://lists.us.dell.com From mmcgrath at fedoraproject.org Mon Dec 18 02:24:47 2006 From: mmcgrath at fedoraproject.org (Mike McGrath) Date: Sun, 17 Dec 2006 20:24:47 -0600 Subject: TurboGears and version control Message-ID: <3237e4410612171824p707f9dbap2934593391e06196@mail.gmail.com> So many of us have started using TurboGears for Fedora infrastructure projects. My question is how should we be version controlling the source? Do we check in the entire project or just the related kid templates, configs, controller files, etc. When you create a project with one version of TurboGears does it work with other versions? How do we prevent mismatches? -Mike From lmacken at redhat.com Mon Dec 18 05:20:44 2006 From: lmacken at redhat.com (Luke Macken) Date: Mon, 18 Dec 2006 00:20:44 -0500 Subject: TurboGears and version control In-Reply-To: <3237e4410612171824p707f9dbap2934593391e06196@mail.gmail.com> References: <3237e4410612171824p707f9dbap2934593391e06196@mail.gmail.com> Message-ID: <4586252C.1080802@redhat.com> Mike McGrath wrote: > So many of us have started using TurboGears for Fedora infrastructure > projects. My question is how should we be version controlling the > source? Do we check in the entire project or just the related kid > templates, configs, controller files, etc. When you create a project > with one version of TurboGears does it work with other versions? How > do we prevent mismatches? The projects don't actually contain TurboGears, or any other associated module. The default `tg-admin quickstart` provides the controllers, model, and configuration for dealing with all of the pieces that make up TurboGears. Pythons egg entry-points then make it dead simple to replace/plugin a variety of different modules in your project (like replaceing kid with Cheetah, SQLObject with SQLAlchemy). When newer versions contain configuration/api changes, TurboGears provides `tg-admin update`[0], which will update your project configuration for you. luke [0]: http://www.turbogears.org/download/upgrade.html From jeff at ocjtech.us Tue Dec 19 04:23:31 2006 From: jeff at ocjtech.us (Jeffrey C. Ollie) Date: Mon, 18 Dec 2006 22:23:31 -0600 Subject: bcfg2 Message-ID: <1166502211.5602.4.camel@lt21223.campus.dmacc.edu> Another option to look into for configuration management: http://mricon.livejournal.com/360529.html Jeff -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From Axel.Thimm at ATrpms.net Tue Dec 19 13:14:58 2006 From: Axel.Thimm at ATrpms.net (Axel Thimm) Date: Tue, 19 Dec 2006 14:14:58 +0100 Subject: bcfg2 In-Reply-To: <1166502211.5602.4.camel@lt21223.campus.dmacc.edu> References: <1166502211.5602.4.camel@lt21223.campus.dmacc.edu> Message-ID: <20061219131458.GA19154@neu.nirvana> On Mon, Dec 18, 2006 at 10:23:31PM -0600, Jeffrey C. Ollie wrote: > Another option to look into for configuration management: Looks interesting, I'd go ahead and put it into extras so we can play with it, unless you want to do so? In the latter case I can do the review. -- Axel.Thimm at ATrpms.net -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From jeff at ocjtech.us Tue Dec 19 14:36:26 2006 From: jeff at ocjtech.us (Jeffrey C. Ollie) Date: Tue, 19 Dec 2006 08:36:26 -0600 Subject: bcfg2 In-Reply-To: <20061219131458.GA19154@neu.nirvana> References: <1166502211.5602.4.camel@lt21223.campus.dmacc.edu> <20061219131458.GA19154@neu.nirvana> Message-ID: <1166538986.3537.9.camel@lt21223.campus.dmacc.edu> On Tue, 2006-12-19 at 14:14 +0100, Axel Thimm wrote: > On Mon, Dec 18, 2006 at 10:23:31PM -0600, Jeffrey C. Ollie wrote: > > Another option to look into for configuration management: > > Looks interesting, I'd go ahead and put it into extras so we can play > with it, unless you want to do so? In the latter case I can do the > review. I've begun to package it up, since it looks useful in for my work as well. There's at least one dependency that isn't in FE yet so I'm working on packaging that as well. I'll try and get the review requests up later today. Jeff -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From orion at cora.nwra.com Tue Dec 19 15:58:40 2006 From: orion at cora.nwra.com (Orion Poplawski) Date: Tue, 19 Dec 2006 08:58:40 -0700 Subject: bcfg2 In-Reply-To: <1166538986.3537.9.camel@lt21223.campus.dmacc.edu> References: <1166502211.5602.4.camel@lt21223.campus.dmacc.edu> <20061219131458.GA19154@neu.nirvana> <1166538986.3537.9.camel@lt21223.campus.dmacc.edu> Message-ID: <45880C30.4070704@cora.nwra.com> Jeffrey C. Ollie wrote: > On Tue, 2006-12-19 at 14:14 +0100, Axel Thimm wrote: >> On Mon, Dec 18, 2006 at 10:23:31PM -0600, Jeffrey C. Ollie wrote: >>> Another option to look into for configuration management: Has anyone looked at puppet? http://reductivelabs.com/projects/puppet/ It's in Extras. Or cfengine for that matter, though I'm getting dissatisfied with it myself. -- Orion Poplawski Technical Manager 303-415-9701 x222 NWRA/CoRA Division FAX: 303-415-9702 3380 Mitchell Lane orion at cora.nwra.com Boulder, CO 80301 http://www.cora.nwra.com From jeff at ocjtech.us Tue Dec 19 16:15:07 2006 From: jeff at ocjtech.us (Jeffrey C. Ollie) Date: Tue, 19 Dec 2006 10:15:07 -0600 Subject: bcfg2 In-Reply-To: <45880C30.4070704@cora.nwra.com> References: <1166502211.5602.4.camel@lt21223.campus.dmacc.edu> <20061219131458.GA19154@neu.nirvana> <1166538986.3537.9.camel@lt21223.campus.dmacc.edu> <45880C30.4070704@cora.nwra.com> Message-ID: <1166544907.3537.21.camel@lt21223.campus.dmacc.edu> On Tue, 2006-12-19 at 08:58 -0700, Orion Poplawski wrote: > Jeffrey C. Ollie wrote: > > On Tue, 2006-12-19 at 14:14 +0100, Axel Thimm wrote: > >> On Mon, Dec 18, 2006 at 10:23:31PM -0600, Jeffrey C. Ollie wrote: > >>> Another option to look into for configuration management: > > Has anyone looked at puppet? > http://reductivelabs.com/projects/puppet/ I haven't looked at Puppet in depth, but one con is that it's written in Ruby (not that there's anything wrong with that). But there may be license issues with bcfg2 so that may be an option as well. > Or cfengine for that matter, though I'm getting dissatisfied with it myself. I haven't looked at cfengine yet either, but from what I've seen it's cryptic configuration is a major con. Jeff -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From a.badger at gmail.com Tue Dec 19 16:44:40 2006 From: a.badger at gmail.com (Toshio Kuratomi) Date: Tue, 19 Dec 2006 08:44:40 -0800 Subject: bcfg2 In-Reply-To: <1166544907.3537.21.camel@lt21223.campus.dmacc.edu> References: <1166502211.5602.4.camel@lt21223.campus.dmacc.edu> <20061219131458.GA19154@neu.nirvana> <1166538986.3537.9.camel@lt21223.campus.dmacc.edu> <45880C30.4070704@cora.nwra.com> <1166544907.3537.21.camel@lt21223.campus.dmacc.edu> Message-ID: <1166546680.21769.53.camel@localhost.localdomain> On Tue, 2006-12-19 at 10:15 -0600, Jeffrey C. Ollie wrote: > On Tue, 2006-12-19 at 08:58 -0700, Orion Poplawski wrote: > > Jeffrey C. Ollie wrote: > > > On Tue, 2006-12-19 at 14:14 +0100, Axel Thimm wrote: > > >> On Mon, Dec 18, 2006 at 10:23:31PM -0600, Jeffrey C. Ollie wrote: > > >>> Another option to look into for configuration management: > > > > Has anyone looked at puppet? > > http://reductivelabs.com/projects/puppet/ > > I haven't looked at Puppet in depth, but one con is that it's written in > Ruby (not that there's anything wrong with that). But there may be > license issues with bcfg2 so that may be an option as well. > skvidal was able to articulate a good reason to not use ruby for the config stack: Anytime we add an interpreted language to the dependencies we need to get a system up and running we're adding another language stack we need to confirm works with our setup before performing an upgrade. That aside, I think the stateless people have been using puppet and generally like it. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From skvidal at linux.duke.edu Tue Dec 19 17:14:28 2006 From: skvidal at linux.duke.edu (seth vidal) Date: Tue, 19 Dec 2006 12:14:28 -0500 Subject: bcfg2 In-Reply-To: <1166544907.3537.21.camel@lt21223.campus.dmacc.edu> References: <1166502211.5602.4.camel@lt21223.campus.dmacc.edu> <20061219131458.GA19154@neu.nirvana> <1166538986.3537.9.camel@lt21223.campus.dmacc.edu> <45880C30.4070704@cora.nwra.com> <1166544907.3537.21.camel@lt21223.campus.dmacc.edu> Message-ID: <1166548468.19477.78.camel@cutter> On Tue, 2006-12-19 at 10:15 -0600, Jeffrey C. Ollie wrote: > On Tue, 2006-12-19 at 08:58 -0700, Orion Poplawski wrote: > > Jeffrey C. Ollie wrote: > > > On Tue, 2006-12-19 at 14:14 +0100, Axel Thimm wrote: > > >> On Mon, Dec 18, 2006 at 10:23:31PM -0600, Jeffrey C. Ollie wrote: > > >>> Another option to look into for configuration management: > > > > Has anyone looked at puppet? > > http://reductivelabs.com/projects/puppet/ > > I haven't looked at Puppet in depth, but one con is that it's written in > Ruby (not that there's anything wrong with that). But there may be > license issues with bcfg2 so that may be an option as well. > > > Or cfengine for that matter, though I'm getting dissatisfied with it myself. > > I haven't looked at cfengine yet either, but from what I've seen it's > cryptic configuration is a major con. > What was wrong with glump and friends? It's simple, no cryptic formatting of files or craziness. The scripting language that runs on the hosts is whatever you want it to be. -sv From Axel.Thimm at ATrpms.net Tue Dec 19 17:15:14 2006 From: Axel.Thimm at ATrpms.net (Axel Thimm) Date: Tue, 19 Dec 2006 18:15:14 +0100 Subject: bcfg2 In-Reply-To: <1166548468.19477.78.camel@cutter> References: <1166502211.5602.4.camel@lt21223.campus.dmacc.edu> <20061219131458.GA19154@neu.nirvana> <1166538986.3537.9.camel@lt21223.campus.dmacc.edu> <45880C30.4070704@cora.nwra.com> <1166544907.3537.21.camel@lt21223.campus.dmacc.edu> <1166548468.19477.78.camel@cutter> Message-ID: <20061219171514.GA2556@neu.nirvana> On Tue, Dec 19, 2006 at 12:14:28PM -0500, seth vidal wrote: > On Tue, 2006-12-19 at 10:15 -0600, Jeffrey C. Ollie wrote: > > On Tue, 2006-12-19 at 08:58 -0700, Orion Poplawski wrote: > > > Jeffrey C. Ollie wrote: > > > > On Tue, 2006-12-19 at 14:14 +0100, Axel Thimm wrote: > > > >> On Mon, Dec 18, 2006 at 10:23:31PM -0600, Jeffrey C. Ollie wrote: > > > >>> Another option to look into for configuration management: > > > > > > Has anyone looked at puppet? > > > http://reductivelabs.com/projects/puppet/ > > > > I haven't looked at Puppet in depth, but one con is that it's written in > > Ruby (not that there's anything wrong with that). But there may be > > license issues with bcfg2 so that may be an option as well. > > > > > Or cfengine for that matter, though I'm getting dissatisfied with it myself. > > > > I haven't looked at cfengine yet either, but from what I've seen it's > > cryptic configuration is a major con. > > > > What was wrong with glump and friends? > > It's simple, no cryptic formatting of files or craziness. The scripting > language that runs on the hosts is whatever you want it to be. Do you have a URL for glump? -- Axel.Thimm at ATrpms.net -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From jeff at ocjtech.us Tue Dec 19 17:20:15 2006 From: jeff at ocjtech.us (Jeffrey C. Ollie) Date: Tue, 19 Dec 2006 11:20:15 -0600 Subject: bcfg2 In-Reply-To: <20061219171514.GA2556@neu.nirvana> References: <1166502211.5602.4.camel@lt21223.campus.dmacc.edu> <20061219131458.GA19154@neu.nirvana> <1166538986.3537.9.camel@lt21223.campus.dmacc.edu> <45880C30.4070704@cora.nwra.com> <1166544907.3537.21.camel@lt21223.campus.dmacc.edu> <1166548468.19477.78.camel@cutter> <20061219171514.GA2556@neu.nirvana> Message-ID: <1166548815.3537.37.camel@lt21223.campus.dmacc.edu> On Tue, 2006-12-19 at 18:15 +0100, Axel Thimm wrote: > > Do you have a URL for glump? http://linux.duke.edu/projects/mini/glump/ https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=218577 Jeff -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From skvidal at linux.duke.edu Tue Dec 19 17:23:44 2006 From: skvidal at linux.duke.edu (seth vidal) Date: Tue, 19 Dec 2006 12:23:44 -0500 Subject: bcfg2 In-Reply-To: <20061219171514.GA2556@neu.nirvana> References: <1166502211.5602.4.camel@lt21223.campus.dmacc.edu> <20061219131458.GA19154@neu.nirvana> <1166538986.3537.9.camel@lt21223.campus.dmacc.edu> <45880C30.4070704@cora.nwra.com> <1166544907.3537.21.camel@lt21223.campus.dmacc.edu> <1166548468.19477.78.camel@cutter> <20061219171514.GA2556@neu.nirvana> Message-ID: <1166549024.19477.80.camel@cutter> On Tue, 2006-12-19 at 18:15 +0100, Axel Thimm wrote: > On Tue, Dec 19, 2006 at 12:14:28PM -0500, seth vidal wrote: > > On Tue, 2006-12-19 at 10:15 -0600, Jeffrey C. Ollie wrote: > > > On Tue, 2006-12-19 at 08:58 -0700, Orion Poplawski wrote: > > > > Jeffrey C. Ollie wrote: > > > > > On Tue, 2006-12-19 at 14:14 +0100, Axel Thimm wrote: > > > > >> On Mon, Dec 18, 2006 at 10:23:31PM -0600, Jeffrey C. Ollie wrote: > > > > >>> Another option to look into for configuration management: > > > > > > > > Has anyone looked at puppet? > > > > http://reductivelabs.com/projects/puppet/ > > > > > > I haven't looked at Puppet in depth, but one con is that it's written in > > > Ruby (not that there's anything wrong with that). But there may be > > > license issues with bcfg2 so that may be an option as well. > > > > > > > Or cfengine for that matter, though I'm getting dissatisfied with it myself. > > > > > > I haven't looked at cfengine yet either, but from what I've seen it's > > > cryptic configuration is a major con. > > > > > > > What was wrong with glump and friends? > > > > It's simple, no cryptic formatting of files or craziness. The scripting > > language that runs on the hosts is whatever you want it to be. > > Do you have a URL for glump? http://linux.duke.edu/projects/mini/glump/ and mike posted a note about it in some detail a few weeks, maybe a month, back. -sv From jeff at ocjtech.us Tue Dec 19 17:30:34 2006 From: jeff at ocjtech.us (Jeffrey C. Ollie) Date: Tue, 19 Dec 2006 11:30:34 -0600 Subject: bcfg2 In-Reply-To: <1166548468.19477.78.camel@cutter> References: <1166502211.5602.4.camel@lt21223.campus.dmacc.edu> <20061219131458.GA19154@neu.nirvana> <1166538986.3537.9.camel@lt21223.campus.dmacc.edu> <45880C30.4070704@cora.nwra.com> <1166544907.3537.21.camel@lt21223.campus.dmacc.edu> <1166548468.19477.78.camel@cutter> Message-ID: <1166549434.3537.47.camel@lt21223.campus.dmacc.edu> On Tue, 2006-12-19 at 12:14 -0500, seth vidal wrote: > > What was wrong with glump and friends? > > It's simple, no cryptic formatting of files or craziness. The scripting > language that runs on the hosts is whatever you want it to be. There's nothing "wrong" with glump. It does an excellent job at what it was designed to do. I think that the issue here is that {cfengine, bcfg2, puppet} were designed to do more that serve out customized versions of config files, like checking ownership/permissions of files, the status of servcies, and whether packages are installed. We just need to look at the alternatives, figure out which one does what we want it to do, and move on from there. I'm not even sure that we've agreed on what we want a configuration management system to do for us yet. Personally, I'm going to be using glump to manage the configuration files for my umpteen bazillion wireless access points. Jeff -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From skvidal at linux.duke.edu Tue Dec 19 17:43:25 2006 From: skvidal at linux.duke.edu (seth vidal) Date: Tue, 19 Dec 2006 12:43:25 -0500 Subject: bcfg2 In-Reply-To: <1166549434.3537.47.camel@lt21223.campus.dmacc.edu> References: <1166502211.5602.4.camel@lt21223.campus.dmacc.edu> <20061219131458.GA19154@neu.nirvana> <1166538986.3537.9.camel@lt21223.campus.dmacc.edu> <45880C30.4070704@cora.nwra.com> <1166544907.3537.21.camel@lt21223.campus.dmacc.edu> <1166548468.19477.78.camel@cutter> <1166549434.3537.47.camel@lt21223.campus.dmacc.edu> Message-ID: <1166550205.19477.90.camel@cutter> On Tue, 2006-12-19 at 11:30 -0600, Jeffrey C. Ollie wrote: > On Tue, 2006-12-19 at 12:14 -0500, seth vidal wrote: > > > > What was wrong with glump and friends? > > > > It's simple, no cryptic formatting of files or craziness. The scripting > > language that runs on the hosts is whatever you want it to be. > > There's nothing "wrong" with glump. It does an excellent job at what it > was designed to do. I think that the issue here is that {cfengine, > bcfg2, puppet} were designed to do more that serve out customized > versions of config files, like checking ownership/permissions of files, > the status of servcies, and whether packages are installed. So what we do at duke with glump is have it serve out custom versions of cron jobs. we have a cron job that runs hourly and nightly that requests its jobs via glump. glump puts together the shell script for that host and hands it back. so if we want to check ownerships or update packages it would be: chown user.group /path/to/file yum -d0 -e0 -y install your_pkg_set That's why we don't need the other features, we implement them within what glump can do. -sv From dlutter at redhat.com Tue Dec 19 22:45:15 2006 From: dlutter at redhat.com (David Lutterkort) Date: Tue, 19 Dec 2006 14:45:15 -0800 Subject: bcfg2 In-Reply-To: <1166550205.19477.90.camel@cutter> References: <1166502211.5602.4.camel@lt21223.campus.dmacc.edu> <20061219131458.GA19154@neu.nirvana> <1166538986.3537.9.camel@lt21223.campus.dmacc.edu> <45880C30.4070704@cora.nwra.com> <1166544907.3537.21.camel@lt21223.campus.dmacc.edu> <1166548468.19477.78.camel@cutter> <1166549434.3537.47.camel@lt21223.campus.dmacc.edu> <1166550205.19477.90.camel@cutter> Message-ID: <1166568315.4534.45.camel@localhost.localdomain> On Tue, 2006-12-19 at 12:43 -0500, seth vidal wrote: > On Tue, 2006-12-19 at 11:30 -0600, Jeffrey C. Ollie wrote: > > On Tue, 2006-12-19 at 12:14 -0500, seth vidal wrote: > > > > > > What was wrong with glump and friends? > > > > > > It's simple, no cryptic formatting of files or craziness. The scripting > > > language that runs on the hosts is whatever you want it to be. > > > > There's nothing "wrong" with glump. It does an excellent job at what it > > was designed to do. I think that the issue here is that {cfengine, > > bcfg2, puppet} were designed to do more that serve out customized > > versions of config files, like checking ownership/permissions of files, > > the status of servcies, and whether packages are installed. > > > So what we do at duke with glump is have it serve out custom versions of > cron jobs. Correct me if I am wrong, but my impression is that glump is mostly a template-expansion tool with a custom language expressed in XML. The two most important features that full-blown config mgmt tools add to that are * direct control over individual entries in database-like config files (like /etc/hosts, /etc/passwd etc.) * flexible grouping of config settings that is flexible enough to express variations with little effort > we have a cron job that runs hourly and nightly that requests its jobs > via glump. > > glump puts together the shell script for that host and hands it back. How do you handle security ? E.g., how do you keep host A getting its hands on the config for host B ? That is important when you manage security-sensitive parts of a machine's config with the tool. > so if we want to check ownerships or update packages it would be: > > > chown user.group /path/to/file > yum -d0 -e0 -y install your_pkg_set How do you deal with failures ? Logging ? Do you know whether the chown actually changed anything ? (Which might be cause for concern) ? > That's why we don't need the other features, we implement them within > what glump can do. Don't get me wrong - glump might be the right tool for the Fedora infrastructure, but you should be conscious about the issues it does _not_ address compared to a full-fledged config mgmt tool. David From dlutter at redhat.com Tue Dec 19 22:52:05 2006 From: dlutter at redhat.com (David Lutterkort) Date: Tue, 19 Dec 2006 14:52:05 -0800 Subject: bcfg2 In-Reply-To: <1166548468.19477.78.camel@cutter> References: <1166502211.5602.4.camel@lt21223.campus.dmacc.edu> <20061219131458.GA19154@neu.nirvana> <1166538986.3537.9.camel@lt21223.campus.dmacc.edu> <45880C30.4070704@cora.nwra.com> <1166544907.3537.21.camel@lt21223.campus.dmacc.edu> <1166548468.19477.78.camel@cutter> Message-ID: <1166568725.4534.51.camel@localhost.localdomain> On Tue, 2006-12-19 at 12:14 -0500, seth vidal wrote: > What was wrong with glump and friends? > > It's simple, no cryptic formatting of files or craziness. The scripting > language that runs on the hosts is whatever you want it to be. To my untrained eye, the glue.xml looks very much like its own language (sure, a simple one, but just because it's XML doesn't mean that it's not its own language. David From skvidal at linux.duke.edu Wed Dec 20 06:58:44 2006 From: skvidal at linux.duke.edu (seth vidal) Date: Wed, 20 Dec 2006 01:58:44 -0500 Subject: bcfg2 In-Reply-To: <1166568725.4534.51.camel@localhost.localdomain> References: <1166502211.5602.4.camel@lt21223.campus.dmacc.edu> <20061219131458.GA19154@neu.nirvana> <1166538986.3537.9.camel@lt21223.campus.dmacc.edu> <45880C30.4070704@cora.nwra.com> <1166544907.3537.21.camel@lt21223.campus.dmacc.edu> <1166548468.19477.78.camel@cutter> <1166568725.4534.51.camel@localhost.localdomain> Message-ID: <1166597924.25450.8.camel@cutter> On Tue, 2006-12-19 at 14:52 -0800, David Lutterkort wrote: > On Tue, 2006-12-19 at 12:14 -0500, seth vidal wrote: > > What was wrong with glump and friends? > > > > It's simple, no cryptic formatting of files or craziness. The scripting > > language that runs on the hosts is whatever you want it to be. > > To my untrained eye, the glue.xml looks very much like its own language > (sure, a simple one, but just because it's XML doesn't mean that it's > not its own language. In comparison to cfengine it is not it's own language. it's a file format, sure, but I wouldn't call it a language. That's all the comparison I was making. -sv From skvidal at linux.duke.edu Wed Dec 20 07:07:59 2006 From: skvidal at linux.duke.edu (seth vidal) Date: Wed, 20 Dec 2006 02:07:59 -0500 Subject: bcfg2 In-Reply-To: <1166568315.4534.45.camel@localhost.localdomain> References: <1166502211.5602.4.camel@lt21223.campus.dmacc.edu> <20061219131458.GA19154@neu.nirvana> <1166538986.3537.9.camel@lt21223.campus.dmacc.edu> <45880C30.4070704@cora.nwra.com> <1166544907.3537.21.camel@lt21223.campus.dmacc.edu> <1166548468.19477.78.camel@cutter> <1166549434.3537.47.camel@lt21223.campus.dmacc.edu> <1166550205.19477.90.camel@cutter> <1166568315.4534.45.camel@localhost.localdomain> Message-ID: <1166598479.25450.18.camel@cutter> On Tue, 2006-12-19 at 14:45 -0800, David Lutterkort wrote: > Correct me if I am wrong, but my impression is that glump is mostly a > template-expansion tool with a custom language expressed in XML. The two > most important features that full-blown config mgmt tools add to that > are > * direct control over individual entries in database-like config > files (like /etc/hosts, /etc/passwd etc.) > * flexible grouping of config settings that is flexible enough to > express variations with little effort Which we cover in the set of scripts (just shell scripts) that I sent along with our glump configs to mike mcgrath when he was looking at it. > > we have a cron job that runs hourly and nightly that requests its jobs > > via glump. > > > > glump puts together the shell script for that host and hands it back. > > How do you handle security ? E.g., how do you keep host A getting its > hands on the config for host B ? That is important when you manage > security-sensitive parts of a machine's config with the tool. Not terribly important when all the machines are managed by the same people. > > so if we want to check ownerships or update packages it would be: > > > > > > chown user.group /path/to/file > > yum -d0 -e0 -y install your_pkg_set > > How do you deal with failures ? Logging ? Do you know whether the chown > actually changed anything ? (Which might be cause for concern) ? That's up to how you want to write the shell script. If you need that level of caution, do so. If you don't, don't. You can report via logging or alerts, pretty much whatever function you want to run in your shell script. The point is you're not learning a new set of skills and a new layout of things to get the job done. You're just extending the skills you already have. Glump just lets you deploy them in a logical sets to lots of various systems. > Don't get me wrong - glump might be the right tool for the Fedora > infrastructure, but you should be conscious about the issues it does > _not_ address compared to a full-fledged config mgmt tool. glump isn't trying to be everything for everyone. It's just a very straightforward mechanism for keeping track of systems. copyfile and other such tools are just an easy way of deploying file updates for them. my concerns about puppet are the new language for things: class passwordsync { # remotefile is syntactic sugar - see the defintion in the docs remotefile { "/etc/passwd": server => 'server', source => 'passwd', } } and of course the concern I issued before is that it ties us into yet another scripting language for systems-maintenance tasks. -sv From dlutter at redhat.com Wed Dec 20 19:53:09 2006 From: dlutter at redhat.com (David Lutterkort) Date: Wed, 20 Dec 2006 11:53:09 -0800 Subject: bcfg2 In-Reply-To: <1166598479.25450.18.camel@cutter> References: <1166502211.5602.4.camel@lt21223.campus.dmacc.edu> <20061219131458.GA19154@neu.nirvana> <1166538986.3537.9.camel@lt21223.campus.dmacc.edu> <45880C30.4070704@cora.nwra.com> <1166544907.3537.21.camel@lt21223.campus.dmacc.edu> <1166548468.19477.78.camel@cutter> <1166549434.3537.47.camel@lt21223.campus.dmacc.edu> <1166550205.19477.90.camel@cutter> <1166568315.4534.45.camel@localhost.localdomain> <1166598479.25450.18.camel@cutter> Message-ID: <1166644389.18520.79.camel@localhost.localdomain> On Wed, 2006-12-20 at 02:07 -0500, seth vidal wrote: > On Tue, 2006-12-19 at 14:45 -0800, David Lutterkort wrote: > > > Correct me if I am wrong, but my impression is that glump is mostly a > > template-expansion tool with a custom language expressed in XML. The two > > most important features that full-blown config mgmt tools add to that > > are > > * direct control over individual entries in database-like config > > files (like /etc/hosts, /etc/passwd etc.) > > * flexible grouping of config settings that is flexible enough to > > express variations with little effort > > Which we cover in the set of scripts (just shell scripts) that I sent > along with our glump configs to mike mcgrath when he was looking at it. So it's not just glump, but also a whole bunch of custom shell scripts that effectively implement (part of) the functionality of a config mgmt tool without calling it that ? How are bugs getting addressed in this model ? Are the Duke and the Fedora Infrastructure deployments effectively forks of the same thing ? > > How do you handle security ? E.g., how do you keep host A getting its > > hands on the config for host B ? That is important when you manage > > security-sensitive parts of a machine's config with the tool. > > Not terribly important when all the machines are managed by the same > people. What happens when one of the machines gets compromised ? How much easy access does the config mgmt tool give you when you break into one machine ? For example, can you use that to get your hands on ssh keys for another machine ? > The point is you're not learning a new set of skills and a new layout of > things to get the job done. You're just extending the skills you already > have. Glump just lets you deploy them in a logical sets to lots of > various systems. It sounds to me that instead of reading a few webpages about your config mgmt tool of choice, you have to read and understand a bunch of shell scripts and/or shell libraries (assuming common concerns like logging are addressed by those scripts); you're just trading one kind of learning with another. > glump isn't trying to be everything for everyone. It's more an issue that if you try to centrally manage configs for machines you quickly run into most of the issues a tool like puppet addresses; from what you've been saying, glump definitely had to address a lot of them. > It's just a very straightforward mechanism for keeping track of systems. I can't really judge that without having seen the shell scripts that are necessary to make glump more than a template-expansion mechanism. What bothers me about the whole config mgmt space is that even though lots of people encounter those problems, and even though config mgmt is an important problem in practice, there's no tools that are commonly used for it. That's partially because the incumbent (cfengine) falls short on many fronts, and partially because it's so easy to get started with sth really simple and site-specific like glump w/o supporting shell scripts. By the time you realize that your own tool has to address a lot of the harder problems that tools like puppet wrestle with it's too late since migrating to another tool has become a major task. I've spent quite some time looking for a good config mgmt system, and IMHO, puppet is by far the most promising of the lot (including bcfg2). I'd much rather Fedora settled on a relatively widespread and mature tool and worked with its community to address whatever shortcomigns it has than go off and do another custom config tool; and in my experience, puppet is pretty easy to learn and work with (and I am willing to put money where my mouth is if somebody could point me to what kind of prototype setup would help them understand better what puppet is and isn't) > my concerns about puppet are the new language for things: > > class passwordsync { > # remotefile is syntactic sugar - see the defintion in the docs > remotefile { "/etc/passwd": > server => 'server', > source => 'passwd', > } > } True, it has its own language, though the language is straightforard and simple: http://reductivelabs.com/projects/puppet/documentation/languagetutorial.html As to why it has its own language: check question four on the FAQ (http://reductivelabs.com/projects/puppet/faq.html) > and of course the concern I issued before is that it ties us into yet > another scripting language for systems-maintenance tasks. What exactly is that saying to people who use the ruby that we ship in Fedora ? I understand that there is some concern that another language might cause upgrade problems, but that's true for _any_ additonal software that is used; upgrade problems aren't restricted to language interpreters. And ruby _is_ a mature language that has been around for a pretty long time, i.e. the ruby interpreter poses as much risk for random breakage as any other package you might want to use to maintain your system. David From mmcgrath at fedoraproject.org Wed Dec 20 20:28:26 2006 From: mmcgrath at fedoraproject.org (Mike McGrath) Date: Wed, 20 Dec 2006 14:28:26 -0600 Subject: Load test 1 - Sat Dec 23rd Message-ID: <3237e4410612201228ka35d259l7bcb0e9edb07ff4@mail.gmail.com> So the load test for the proxy servers is scheduled for this Saturday, Dec 23rd at 10:00 a.m. (GMT). It will involve at most, proxy1, proxy2, app1, and app2. Though will most likely not use both app servers. One of the proxy servers will be used as a client so none of the other RH systems (including the balancer for now) should be affected. Depending on the results of this test we will proceed with another test at a later date. The test should take no longer than 2 hours but may go 3. Stacy, if this doesn't work for you let me know, this test sounds pretty sane and shouldn't have any impact on anything other than fedora servers. It'll also happen during non-peak hours which is nice. -Mike From Axel.Thimm at ATrpms.net Wed Dec 20 20:42:14 2006 From: Axel.Thimm at ATrpms.net (Axel Thimm) Date: Wed, 20 Dec 2006 21:42:14 +0100 Subject: bcfg2 In-Reply-To: <1166644389.18520.79.camel@localhost.localdomain> References: <20061219131458.GA19154@neu.nirvana> <1166538986.3537.9.camel@lt21223.campus.dmacc.edu> <45880C30.4070704@cora.nwra.com> <1166544907.3537.21.camel@lt21223.campus.dmacc.edu> <1166548468.19477.78.camel@cutter> <1166549434.3537.47.camel@lt21223.campus.dmacc.edu> <1166550205.19477.90.camel@cutter> <1166568315.4534.45.camel@localhost.localdomain> <1166598479.25450.18.camel@cutter> <1166644389.18520.79.camel@localhost.localdomain> Message-ID: <20061220204214.GK4374@neu.nirvana> On Wed, Dec 20, 2006 at 11:53:09AM -0800, David Lutterkort wrote: > > and of course the concern I issued before is that it ties us into yet > > another scripting language for systems-maintenance tasks. > > What exactly is that saying to people who use the ruby that we ship in > Fedora ? I understand that there is some concern that another language > might cause upgrade problems No, that's not the issue Seth addresses (I think), and also no attribute against the quality of ruby as language and implementation. The question is more like what language are the current and future maintainers of the fedora infrastructure comfortable with and would be able to do some bug hunting, fixing, changes if required. And of course any python solution will have a small bonus here. -- Axel.Thimm at ATrpms.net -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From dlutter at redhat.com Thu Dec 21 00:44:47 2006 From: dlutter at redhat.com (David Lutterkort) Date: Wed, 20 Dec 2006 16:44:47 -0800 Subject: bcfg2 In-Reply-To: <20061220204214.GK4374@neu.nirvana> References: <20061219131458.GA19154@neu.nirvana> <1166538986.3537.9.camel@lt21223.campus.dmacc.edu> <45880C30.4070704@cora.nwra.com> <1166544907.3537.21.camel@lt21223.campus.dmacc.edu> <1166548468.19477.78.camel@cutter> <1166549434.3537.47.camel@lt21223.campus.dmacc.edu> <1166550205.19477.90.camel@cutter> <1166568315.4534.45.camel@localhost.localdomain> <1166598479.25450.18.camel@cutter> <1166644389.18520.79.camel@localhost.localdomain> <20061220204214.GK4374@neu.nirvana> Message-ID: <1166661887.18520.116.camel@localhost.localdomain> On Wed, 2006-12-20 at 21:42 +0100, Axel Thimm wrote: > On Wed, Dec 20, 2006 at 11:53:09AM -0800, David Lutterkort wrote: > > > and of course the concern I issued before is that it ties us into yet > > > another scripting language for systems-maintenance tasks. > > > > What exactly is that saying to people who use the ruby that we ship in > > Fedora ? I understand that there is some concern that another language > > might cause upgrade problems > > No, that's not the issue Seth addresses (I think), and also no > attribute against the quality of ruby as language and > implementation. > > The question is more like what language are the current and future > maintainers of the fedora infrastructure comfortable with and would be > able to do some bug hunting, fixing, changes if required. And of > course any python solution will have a small bonus here. That is understandable (though a little different from the reasons given a few days ago here) - I hope that phear of ruby won't keep people from having a look at puppet and comparing its features to bcfg2. David From skvidal at linux.duke.edu Thu Dec 21 06:37:37 2006 From: skvidal at linux.duke.edu (seth vidal) Date: Thu, 21 Dec 2006 01:37:37 -0500 Subject: bcfg2 In-Reply-To: <1166644389.18520.79.camel@localhost.localdomain> References: <1166502211.5602.4.camel@lt21223.campus.dmacc.edu> <20061219131458.GA19154@neu.nirvana> <1166538986.3537.9.camel@lt21223.campus.dmacc.edu> <45880C30.4070704@cora.nwra.com> <1166544907.3537.21.camel@lt21223.campus.dmacc.edu> <1166548468.19477.78.camel@cutter> <1166549434.3537.47.camel@lt21223.campus.dmacc.edu> <1166550205.19477.90.camel@cutter> <1166568315.4534.45.camel@localhost.localdomain> <1166598479.25450.18.camel@cutter> <1166644389.18520.79.camel@localhost.localdomain> Message-ID: <1166683057.32180.28.camel@cutter> On Wed, 2006-12-20 at 11:53 -0800, David Lutterkort wrote: > So it's not just glump, but also a whole bunch of custom shell scripts > that effectively implement (part of) the functionality of a config mgmt > tool without calling it that ? It's a bunch of shell scripts. Learning a new language that you will never use somewhere else is the problem I'm citing with items like puppet and cfengine. Furthering the already existent shell scripting skills that most sysadmins already have is what I was talking about. > How are bugs getting addressed in this > model ? Are the Duke and the Fedora Infrastructure deployments > effectively forks of the same thing ? That really depends on what we want to do. > What happens when one of the machines gets compromised ? How much easy > access does the config mgmt tool give you when you break into one > machine ? For example, can you use that to get your hands on ssh keys > for another machine ? I don't deploy ssh keys that way. It's a chicken-and-egg problem. You have to have a special key in order to get a special key, so how do you get the first special key? > It sounds to me that instead of reading a few webpages about your config > mgmt tool of choice, you have to read and understand a bunch of shell > scripts and/or shell libraries (assuming common concerns like logging > are addressed by those scripts); you're just trading one kind of > learning with another. not really. You're extending your knowledge that already exists (every sysadmin is implicitly capable of moderate to advanced shell scripting) rather than capturing a specialized knowledge set for a single use. > > glump isn't trying to be everything for everyone. > > It's more an issue that if you try to centrally manage configs for > machines you quickly run into most of the issues a tool like puppet > addresses; from what you've been saying, glump definitely had to address > a lot of them. and we've addressed a lot of them. > True, it has its own language, though the language is straightforard and > simple: > http://reductivelabs.com/projects/puppet/documentation/languagetutorial.html > As to why it has its own language: check question four on the FAQ > (http://reductivelabs.com/projects/puppet/faq.html) This is just it. I don't care to learn another one-off language. There are enough of those as it is. > What exactly is that saying to people who use the ruby that we ship in > Fedora ? I understand that there is some concern that another language > might cause upgrade problems, but that's true for _any_ additonal > software that is used; upgrade problems aren't restricted to language > interpreters. And ruby _is_ a mature language that has been around for a > pretty long time, i.e. the ruby interpreter poses as much risk for > random breakage as any other package you might want to use to maintain > your system. I'm not bad-mouthing ruby. This doesn't have to do with it being ruby. Right now in order to maintain a system running for admin tasks we require: python - stay consistent and reliable for yum and friends bash - stay consistent and reliable for all the init scripts, etc If we start tying down ruby or any other language as we move along we end up having more and more pieces of the OS that we cannot deploy new versions of w/o fear of breaking our administrative infrastructure. So if puppet needs a version of ruby or some number of gems in order to run, but we want to have a ruby install for rails that needs a different version then we have to screw around with multiple version installs of ruby and keep up with those installs for security patching. That's what I mean. I'd like to keep our administrative language requirements to a minimum. Since anaconda, yum and most of the system-config-* tools are in python, it makes since to tie down python for administrative uses, but if we keep adding components in more languages we end up with progressively more problems. it has nothing to do with the language itself, just the proliferation of ones we rely on for admin tools. -sv From skvidal at linux.duke.edu Thu Dec 21 06:38:55 2006 From: skvidal at linux.duke.edu (seth vidal) Date: Thu, 21 Dec 2006 01:38:55 -0500 Subject: bcfg2 In-Reply-To: <1166661887.18520.116.camel@localhost.localdomain> References: <20061219131458.GA19154@neu.nirvana> <1166538986.3537.9.camel@lt21223.campus.dmacc.edu> <45880C30.4070704@cora.nwra.com> <1166544907.3537.21.camel@lt21223.campus.dmacc.edu> <1166548468.19477.78.camel@cutter> <1166549434.3537.47.camel@lt21223.campus.dmacc.edu> <1166550205.19477.90.camel@cutter> <1166568315.4534.45.camel@localhost.localdomain> <1166598479.25450.18.camel@cutter> <1166644389.18520.79.camel@localhost.localdomain> <20061220204214.GK4374@neu.nirvana> <1166661887.18520.116.camel@localhost.localdomain> Message-ID: <1166683135.32180.31.camel@cutter> On Wed, 2006-12-20 at 16:44 -0800, David Lutterkort wrote: > On Wed, 2006-12-20 at 21:42 +0100, Axel Thimm wrote: > > On Wed, Dec 20, 2006 at 11:53:09AM -0800, David Lutterkort wrote: > > > > and of course the concern I issued before is that it ties us into yet > > > > another scripting language for systems-maintenance tasks. > > > > > > What exactly is that saying to people who use the ruby that we ship in > > > Fedora ? I understand that there is some concern that another language > > > might cause upgrade problems > > > > No, that's not the issue Seth addresses (I think), and also no > > attribute against the quality of ruby as language and > > implementation. > > > > The question is more like what language are the current and future > > maintainers of the fedora infrastructure comfortable with and would be > > able to do some bug hunting, fixing, changes if required. And of > > course any python solution will have a small bonus here. > > That is understandable (though a little different from the reasons given > a few days ago here) - I hope that phear of ruby won't keep people from > having a look at puppet and comparing its features to bcfg2. > it's not a fear of ruby at all. it's a fear of eventually having admin tools that require specific versions of: python bash ruby java php perl etc etc etc and not being able to more flexibly deploy applications that need to run on those systems. -sv From smooge at gmail.com Thu Dec 21 06:57:08 2006 From: smooge at gmail.com (Stephen John Smoogen) Date: Wed, 20 Dec 2006 23:57:08 -0700 Subject: bcfg2 In-Reply-To: <1166683135.32180.31.camel@cutter> References: <20061219131458.GA19154@neu.nirvana> <1166548468.19477.78.camel@cutter> <1166549434.3537.47.camel@lt21223.campus.dmacc.edu> <1166550205.19477.90.camel@cutter> <1166568315.4534.45.camel@localhost.localdomain> <1166598479.25450.18.camel@cutter> <1166644389.18520.79.camel@localhost.localdomain> <20061220204214.GK4374@neu.nirvana> <1166661887.18520.116.camel@localhost.localdomain> <1166683135.32180.31.camel@cutter> Message-ID: <80d7e4090612202257q3e4b6b04i585b9c2316182362@mail.gmail.com> On 12/20/06, seth vidal wrote: > On Wed, 2006-12-20 at 16:44 -0800, David Lutterkort wrote: > > On Wed, 2006-12-20 at 21:42 +0100, Axel Thimm wrote: > > > On Wed, Dec 20, 2006 at 11:53:09AM -0800, David Lutterkort wrote: > it's not a fear of ruby at all. it's a fear of eventually having admin > tools that require specific versions of: > > python > bash > ruby > java > php > perl That has been my biggest fear of deploying puppet myself. I wish I had the skill to make a 'portage' of it to python as its meta language is what I want in a system. -- Stephen J Smoogen. -- CSIRT/Linux System Administrator How far that little candle throws his beams! So shines a good deed in a naughty world. = Shakespeare. "The Merchant of Venice" From sbranden at redhat.com Thu Dec 21 14:06:04 2006 From: sbranden at redhat.com (Stacy J Brandenburg) Date: Thu, 21 Dec 2006 09:06:04 -0500 Subject: Load test 1 - Sat Dec 23rd In-Reply-To: <3237e4410612201228ka35d259l7bcb0e9edb07ff4@mail.gmail.com> References: <3237e4410612201228ka35d259l7bcb0e9edb07ff4@mail.gmail.com> Message-ID: <458A94CC.9010802@redhat.com> ok by me Mike McGrath wrote: > So the load test for the proxy servers is scheduled for this Saturday, > Dec 23rd at 10:00 a.m. (GMT). It will involve at most, proxy1, > proxy2, app1, and app2. Though will most likely not use both app > servers. One of the proxy servers will be used as a client so none of > the other RH systems (including the balancer for now) should be > affected. > > Depending on the results of this test we will proceed with another > test at a later date. The test should take no longer than 2 hours but > may go 3. > > Stacy, if this doesn't work for you let me know, this test sounds > pretty sane and shouldn't have any impact on anything other than > fedora servers. It'll also happen during non-peak hours which is > nice. > > -Mike -- ======================================================== = Stacy J. Brandenburg Red Hat Inc. = = Manager, Network Operations sbranden at redhat.com = = 919-754-4313 http://www.redhat.com = ======================================================== From dlutter at redhat.com Thu Dec 21 23:22:01 2006 From: dlutter at redhat.com (David Lutterkort) Date: Thu, 21 Dec 2006 15:22:01 -0800 Subject: bcfg2 In-Reply-To: <1166683057.32180.28.camel@cutter> References: <1166502211.5602.4.camel@lt21223.campus.dmacc.edu> <20061219131458.GA19154@neu.nirvana> <1166538986.3537.9.camel@lt21223.campus.dmacc.edu> <45880C30.4070704@cora.nwra.com> <1166544907.3537.21.camel@lt21223.campus.dmacc.edu> <1166548468.19477.78.camel@cutter> <1166549434.3537.47.camel@lt21223.campus.dmacc.edu> <1166550205.19477.90.camel@cutter> <1166568315.4534.45.camel@localhost.localdomain> <1166598479.25450.18.camel@cutter> <1166644389.18520.79.camel@localhost.localdomain> <1166683057.32180.28.camel@cutter> Message-ID: <1166743321.26878.97.camel@localhost.localdomain> This has gone way longer than I wanted to; just to make sure: the main points I was trying to make was (1) for Fedora infrastructure there should be a very good understanding how complex things need to be and (2) that the choice of tool should be based on matching those needs to the features of the tools available. A couple comments to Seth's post: On Thu, 2006-12-21 at 01:37 -0500, seth vidal wrote: > On Wed, 2006-12-20 at 11:53 -0800, David Lutterkort wrote: > > > So it's not just glump, but also a whole bunch of custom shell scripts > > that effectively implement (part of) the functionality of a config mgmt > > tool without calling it that ? > > It's a bunch of shell scripts. Learning a new language that you will > never use somewhere else is the problem I'm citing with items like > puppet and cfengine. Like everythign else, it's a tradeoff; there are problems for which domain-specific languages are appropriate and others for which they aren't. It's a matter of degrees, and what you have to invest in learning a new language you hopefully gain in clarity, abstraction, security etc. At the same time, puppet's (and even more so cfengine's) language are very simple and by design very restricted in what you can and can't do. > > What happens when one of the machines gets compromised ? How much easy > > access does the config mgmt tool give you when you break into one > > machine ? For example, can you use that to get your hands on ssh keys > > for another machine ? > > I don't deploy ssh keys that way. It's a chicken-and-egg problem. You > have to have a special key in order to get a special key, so how do you > get the first special key? You didn't really answer how you deal with security breeches. As for ssh keys: they are (by design) not very useful for establishing trust in a setting with a central authority. SSL (X509, really) certs are much better for that, and the way you do that is by giving each client it's own cert; you still have the issue of how to get that cert onto the client initially, and the answer depends on your level of paranoia - a very paranoid solution would be to generate the signed cert and private key on a USB stick, carry it to the client and clean out the USB stick after copying the keypair onto the client. Basing trust on SSL certs buys you at least two things over ssh keys: (1) trust is ultimately derived from the fact that the certs are signed by the central authority, (2) certs can be revoked individually and centrally if they should be compromised. With certs in place, you can send ssh keys securely from your central server to each client, and _know_ that you are really talking to the right guy. The reason ssh keys even come up is that if you have to rebuild a machine you would usually want it to keep its host keys so that you don't have > I'm not bad-mouthing ruby. This doesn't have to do with it being ruby. > Right now in order to maintain a system running for admin tasks we > require: > > python - stay consistent and reliable for yum and friends > bash - stay consistent and reliable for all the init scripts, etc And, in reality also perl; at least on my system, 'yum erase perl' wants to uninstall stuff like bind, php(!), postfix, psutils, a number of system-config- things, samba, squid, subversion, and, worst of all, gaim. > If we start tying down ruby or any other language as we move along we > end up having more and more pieces of the OS that we cannot deploy new > versions of w/o fear of breaking our administrative infrastructure. There are many ways to break a system on upgrade or otherwise, and the tools around a specific language are pretty low on that list. From your experience with yum, how much breakage is caused by python bugs, how much by bugs in components that yum relies on, how much by bugs in yum itself and how much by user error ? I would assume the lion's share of these problems, as with any tool, is bugs in the tool itself. Actually, I assume user error is the biggest cuase here; and some user errors definitely fall into teh category of user interface bugs of the tool. > So if puppet needs a version of ruby or some number of gems in order to > run, but we want to have a ruby install for rails that needs a different > version then we have to screw around with multiple version installs of > ruby and keep up with those installs for security patching. How does that have anything to do with implementation language ? This exact same argument works for any sizeable component, and it's a danger everytime more than one important tool relies on that component. You could rewrite the above paragraph and turn it into an argument why yum shouldn't use sqlite or expat. BTW, puppet's requirements are by design very simple, especially for the client: ruby and facter, a ruby library developed in concert with puppet (which could really be bundled with puppet) And I would not even look at a config management tool that wants rails on the client ;) > That's what I mean. I'd like to keep our administrative language > requirements to a minimum. Since anaconda, yum and most of the > system-config-* tools are in python, it makes since to tie down python > for administrative uses, but if we keep adding components in more > languages we end up with progressively more problems. Again, that argument is only valid if there's some evidence that language-related problems are a significant part of the overall problems a sysadmin encounters. And you need to contrast that with the problems that are caused by choosing an inadequate tool - or worse, no tool - for the wrong reasons. I hope we can all agree that for config management there is a lot of room for improvement in any which way. Clearly, adding another language is of some concern, but I think it's far from the most important issue to look at; things like functionality, match of the features to what's needed, viability of the tool's community, development method (are bugs fixed responsively? patches accepted? are there automated tests?) etc. deserve serious attention. David From ask at develooper.com Fri Dec 22 00:10:01 2006 From: ask at develooper.com (=?ISO-8859-1?Q?Ask_Bj=F8rn_Hansen?=) Date: Fri, 22 Dec 2006 01:10:01 +0100 Subject: Introduction Message-ID: Hi everyone! I've written another mail to the list I'm about to send, but I figured it'd be polite to introduce myself first. I think it's very nice how most contributors to this list have been doing that. I'm Ask Bj?rn Hansen. My little consulting company develops neat application for companies large and small (lately mostly larger ones) and occasionally at conferences and to companies I give advice on how to architect web applications to scale[1]. More relevant for the work you guys are doing here, then I've been running many of the perl.org services for the Perl community since 1999 or so. We usually use RHEL (3 and 4), but lately a bit of Fedora too in addition to a FreeBSD system here and there. I also have a cabinet with Fedora servers for a startup I am, uh, starting. We might change those to RHEL5 at some point, but more likely then we'll get the Stateless Linux magic working instead and use that for managing all the systems (virtual and physical). I joined this list to absorb any tips and ideas for running a cluster of RH/Fedora boxes -- and ideally to offer my help, but of course I haven't had any time for that. - ask [1] http://develooper.com/talks/ -- http://develooper.com/ - http://askask.com/ From ask at develooper.com Fri Dec 22 00:13:37 2006 From: ask at develooper.com (=?ISO-8859-1?Q?Ask_Bj=F8rn_Hansen?=) Date: Fri, 22 Dec 2006 01:13:37 +0100 Subject: bcfg2 In-Reply-To: <1166683057.32180.28.camel@cutter> References: <1166502211.5602.4.camel@lt21223.campus.dmacc.edu> <20061219131458.GA19154@neu.nirvana> <1166538986.3537.9.camel@lt21223.campus.dmacc.edu> <45880C30.4070704@cora.nwra.com> <1166544907.3537.21.camel@lt21223.campus.dmacc.edu> <1166548468.19477.78.camel@cutter> <1166549434.3537.47.camel@lt21223.campus.dmacc.edu> <1166550205.19477.90.camel@cutter> <1166568315.4534.45.camel@localhost.localdomain> <1166598479.25450.18.camel@cutter> <1166644389.18520.79.camel@localhost.localdomain> <1166683057.32180.28.camel@cutter> Message-ID: On Dec 21, 2006, at 7:37, seth vidal wrote: > If we start tying down ruby or any other language as we move along we > end up having more and more pieces of the OS that we cannot deploy new > versions of w/o fear of breaking our administrative infrastructure. I'm sorry, that's just FUD. Generally: There are lots of components that, if you were truly paranoid, you couldn't upgrade without fear of breaking any sort of mildly complex infrastructure anyway. Yet most of us go along alright with frequent-ish "yum upgrade" or (of course) "up2date -u". More specifically: I don't know the change history of python well, but with for instance perl then any sort of breakage is extremely rare. Maybe I have selective memory, but I don't recall any actually. You should see the lengths they go to to preserve obscure side-effects of bugs and undocumented "features" as they still rapidly are developing and enhancing perl 5. I've only been using Ruby casually for a few years, but I haven't gotten the impression that it's much different in the Ruby community. As someone else pointed out: It's really not reasonable to say "oh, we don't trust we won't break our own packages so let's not use them". Dog food and all. In particular not if the worst case scenario is to login to each box manually to downgrade a bad RPM to get the administration infrastructure going again. Your other argument: "Few here programs in Ruby and critically depending on something we can't fix sucks" is much much better. :-) As a, future, counter argument then the stateless guys are planning to use puppet as a part of that system. If so then there's a good chance it'll be very well integrated with Fedora and RHEL. - ask -- http://develooper.com/ - http://askask.com/ From Axel.Thimm at ATrpms.net Fri Dec 22 00:57:20 2006 From: Axel.Thimm at ATrpms.net (Axel Thimm) Date: Fri, 22 Dec 2006 01:57:20 +0100 Subject: bcfg2 In-Reply-To: <1166743321.26878.97.camel@localhost.localdomain> References: <45880C30.4070704@cora.nwra.com> <1166544907.3537.21.camel@lt21223.campus.dmacc.edu> <1166548468.19477.78.camel@cutter> <1166549434.3537.47.camel@lt21223.campus.dmacc.edu> <1166550205.19477.90.camel@cutter> <1166568315.4534.45.camel@localhost.localdomain> <1166598479.25450.18.camel@cutter> <1166644389.18520.79.camel@localhost.localdomain> <1166683057.32180.28.camel@cutter> <1166743321.26878.97.camel@localhost.localdomain> Message-ID: <20061222005720.GA11417@neu.nirvana> On Thu, Dec 21, 2006 at 03:22:01PM -0800, David Lutterkort wrote: > From your experience with yum, how much breakage is caused by python > bugs, how much by bugs in components that yum relies on, how much by > bugs in yum itself and how much by user error ? I would assume the > lion's share of these problems, as with any tool, is bugs in the > tool itself. That's an argument in favour of a tool written in a language everyone around here is comfortable with, or not? Anyway I think we should start looking at the tools themselves and consider more parameters than the mere language they are written in. A healthy developer and user community is probably more important, as well as usability and scalability to more complex setups. -- Axel.Thimm at ATrpms.net -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From mmcgrath at fedoraproject.org Fri Dec 22 01:08:25 2006 From: mmcgrath at fedoraproject.org (Mike McGrath) Date: Thu, 21 Dec 2006 19:08:25 -0600 Subject: Introduction In-Reply-To: References: Message-ID: <3237e4410612211708y59879c7aw9d84d0353495b65d@mail.gmail.com> On 12/21/06, Ask Bj?rn Hansen wrote: > More relevant for the work you guys are doing here, then I've been > running many of the perl.org services for the Perl community since > 1999 or so. We usually use RHEL (3 and 4), but lately a bit of > Fedora too in addition to a FreeBSD system here and there. I also > have a cabinet with Fedora servers for a startup I am, uh, starting. > We might change those to RHEL5 at some point, but more likely then > we'll get the Stateless Linux magic working instead and use that for > managing all the systems (virtual and physical). Welcome Bj?rn, good to have you aboard. -Mike From skvidal at linux.duke.edu Fri Dec 22 04:03:04 2006 From: skvidal at linux.duke.edu (seth vidal) Date: Thu, 21 Dec 2006 23:03:04 -0500 Subject: bcfg2 In-Reply-To: References: <1166502211.5602.4.camel@lt21223.campus.dmacc.edu> <20061219131458.GA19154@neu.nirvana> <1166538986.3537.9.camel@lt21223.campus.dmacc.edu> <45880C30.4070704@cora.nwra.com> <1166544907.3537.21.camel@lt21223.campus.dmacc.edu> <1166548468.19477.78.camel@cutter> <1166549434.3537.47.camel@lt21223.campus.dmacc.edu> <1166550205.19477.90.camel@cutter> <1166568315.4534.45.camel@localhost.localdomain> <1166598479.25450.18.camel@cutter> <1166644389.18520.79.camel@localhost.localdomain> <1166683057.32180.28.camel@cutter> Message-ID: <1166760184.7782.2.camel@cutter> On Fri, 2006-12-22 at 01:13 +0100, Ask Bj?rn Hansen wrote: > On Dec 21, 2006, at 7:37, seth vidal wrote: > > > If we start tying down ruby or any other language as we move along we > > end up having more and more pieces of the OS that we cannot deploy new > > versions of w/o fear of breaking our administrative infrastructure. > > I'm sorry, that's just FUD. You're using FUD like it is its own thing. Fear Uncertainty and Doubt are things that a sysadmin must weigh when evaluating tools. > As someone else pointed out: It's really not reasonable to say "oh, > we don't trust we won't break our own packages so let's not use > them". Dog food and all. In particular not if the worst case > scenario is to login to each box manually to downgrade a bad RPM to > get the administration infrastructure going again. If you can login. But let's be clear, I don't really care enough about this argument to continue it. If y'all want to use puppet, do so or something. -sv From mdehaan at redhat.com Fri Dec 22 18:25:09 2006 From: mdehaan at redhat.com (Michael DeHaan) Date: Fri, 22 Dec 2006 13:25:09 -0500 Subject: Introduction / Xen Stuff Message-ID: <458C2305.10705@redhat.com> Hi, My name is Michael DeHaan, and I work in Red Hat's emerging technologies group in Raleigh, NC. I joined this list because (A) I really want to get more involved in the Fedora project beyond maintaining some extras packages, (B) I'm hoping that I can contribute some on the Xen and systems-management side of things, since I've heard infrastructure is going to start to do more with virtual machines. I also have a good deal of experience with other companies build systems (but not Red Hat's or Fedora's) and possibly I might be able to contribute some there also, or at least learn more about Fedora's setup. Mainly I'm interested in systems management and virtualization stuff though. Xen-wise, two things. I wrote and maintain a provisioning application called cobbler (http://cobbler.et.redhat.com), which potentially could be useful for provisioning Xen boxes in the infrastructure environment. Maybe. If not, maybe it can be extended, and feedback is awesome. If it's not useful, that's ok. The other thing is that I'm looking at ways to deploy Xen in easy ways for Fedora users, basically an open way of managing a /lot/ of Xen machines with integrated configuration management, central interfaces, and not a lot of setup or technical knowledge needed. That's all pretty new, though if some of that would be useful for infrastructure as it evolves, that would be pretty neat. Either way, I figure I can learn from ways that Fedora is using Xen just the same. Anyhow, hi. --Michael From ssrat at ticon.net Fri Dec 22 19:03:55 2006 From: ssrat at ticon.net (David Douthitt) Date: Fri, 22 Dec 2006 13:03:55 -0600 Subject: Introduction (and cfengine) Message-ID: <458C2C1B.2050301@ticon.net> Hello! I am a UNIX System Administrator from the midwest (Wisconsin), and have worked with multiple open source and proprietary UNIX and Linux environments. (I'll skip over the UNIX and non-Red Hat experiences unless there is interest.) I've used Linux on non-Intel machines as well as Intel machines (PARISC and PowerPC specifically). I've used BSD systems as well, including on non-Intel architectures. I'm also certified: RHCE, SCSA, Linux+, and LPI Level 1. First time I used Linux was Slackware on a 386SX-16MHz machine with 4M of RAM; my oldest Red Hat is Red Hat 2 on a Linux User Group CD I still have around here. I also have 4.4BSD Lite and FreeBSD 2.2.6 and OpenBSD 2.6 on CDs..... I had my own floppy-based Linux distribution (Oxygen) based on the Linux Router Project. I have, in the past, also "built" my own Red Hat Enterprise systems without the benefit of an installer, and also in some cases from source RPMs alone (rather than binary RPMs). As far as cfengine goes, I contribute sporadically to the cfengine documentation, mostly proofreading and copyediting. I will be embarking on a book about cfengine shortly (after I complete the GNU screen book). I have run cfengine in a production environment and tend to be rather a voracious supporter of cfengine :-) I've not used puppet, but am a Ruby programmer and would like to learn more about it - it was designed after using cfengine and was designed to use the power of Ruby. I'm also a long-time Korn shell scripter - I used ksh for scripting before I used ksh as my shell! (I used csh then, but programmed in ksh.) I've also programmed in Perl. I've used Perl since the MSDOS 4.0 port, and Ruby since version 1.4. I also tend to be something of a programmer and programming languages nut; I've used FORTH (extensively and comprehensively), Smalltalk (dabble), LISP (moderate), and BASIC, C, Pascal, C++, COBOL, SNOBOL, all moderate to extensively. I contributed to the LCDd project briefly (mostly revamping the system to use GNU getopts). I'll be joining the documentation group at the same time. I'll also post a cfengine review shortly. -- David Douthitt HP-UX, Unixware, Linux, FreeBSD RHCE, SCSA, Linux+, LPIC-1 http://www.lulu.com/ssrat From ssrat at ticon.net Fri Dec 22 19:08:34 2006 From: ssrat at ticon.net (David Douthitt) Date: Fri, 22 Dec 2006 13:08:34 -0600 Subject: Mailing List Troubles? Message-ID: <458C2D32.6080403@ticon.net> I sent my signup to the list using the *old-fashioned* way: I sent an email to fedora-infrastructure-list-request at redhat.com (and to fedora-documentation-list-request at redhat.com) with "subscribe" in the subject and mail with a standard signature (marked by dash-dash-space on a line). Both replied twice (for whatever reason!). I also seem to have sent off an HTML mail for my intro! Ack! I expected it to ask (at which point I'd use text only) but natch.... -- David Douthitt HP-UX, Unixware, Linux, FreeBSD RHCE, SCSA, Linux+, LPIC-1 http://www.lulu.com/ssrat From mmcgrath at fedoraproject.org Fri Dec 22 19:17:03 2006 From: mmcgrath at fedoraproject.org (Mike McGrath) Date: Fri, 22 Dec 2006 13:17:03 -0600 Subject: Introduction / Xen Stuff In-Reply-To: <458C2305.10705@redhat.com> References: <458C2305.10705@redhat.com> Message-ID: <3237e4410612221117s781baee3r749c409aff4c934b@mail.gmail.com> On 12/22/06, Michael DeHaan wrote: > Hi, > > My name is Michael DeHaan, and I work in Red Hat's emerging technologies > group in Raleigh, NC. I joined this list because (A) I really want to > get more involved in the Fedora project beyond maintaining some extras > packages, (B) I'm hoping that I can contribute some on the Xen and > systems-management side of things, since I've heard infrastructure is > going to start to do more with virtual machines. I also have a good > deal of experience with other companies build systems (but not Red Hat's > or Fedora's) and possibly I might be able to contribute some there also, > or at least learn more about Fedora's setup. Mainly I'm interested in > systems management and virtualization stuff though. Welcome Michael. Our current Xen infrastructure is pretty small and we pretty much have no "management" system for the actual machines though as we grow that need will become more apparent. I'm particularly interested in giving trusted parties[1] the ability to deploy some of our infrastructure if need be. A management system or at least a start of tools would be great for that. At present we have 3 xen servers with many guest OS's. Pretty much all machines built from here on out will be in a xen guest as upgrades happen or new machines get installed. The two exceptions to this may be the CVS stuff (which is still up in the air on the best way to do that) and the backup server. Seeing as we're becoming so dependant on Xen its good to have another xen guy on the team. -Mike [1] trusted parties - We should figure out how to define this. We'll wait till the full timer gets on to have more time to examine options for expanding our infrastructure beyond duke and the phx colo. From mmcgrath at fedoraproject.org Fri Dec 22 19:19:58 2006 From: mmcgrath at fedoraproject.org (Mike McGrath) Date: Fri, 22 Dec 2006 13:19:58 -0600 Subject: Introduction (and cfengine) In-Reply-To: <458C2C1B.2050301@ticon.net> References: <458C2C1B.2050301@ticon.net> Message-ID: <3237e4410612221119i4628c1ecm7418a5ce1c2e873c@mail.gmail.com> On 12/22/06, David Douthitt wrote: > I'll be joining the documentation group at the same time. I'll also > post a cfengine review shortly. excellent. Looks like we'll have a puppet expert, cfengine expert and a glump expert on hand to help us make the decision on what to use. At least we know we won't be missing anything :-) Good have you aboard David. -Mike From mgalgoci at redhat.com Fri Dec 22 19:26:16 2006 From: mgalgoci at redhat.com (Matthew Galgoci) Date: Fri, 22 Dec 2006 14:26:16 -0500 Subject: Introduction (and cfengine) In-Reply-To: <3237e4410612221119i4628c1ecm7418a5ce1c2e873c@mail.gmail.com> References: <458C2C1B.2050301@ticon.net> <3237e4410612221119i4628c1ecm7418a5ce1c2e873c@mail.gmail.com> Message-ID: > > I'll be joining the documentation group at the same time. I'll also > > post a cfengine review shortly. > > excellent. Looks like we'll have a puppet expert, cfengine expert and > a glump expert on hand to help us make the decision on what to use. > At least we know we won't be missing anything :-) > > Good have you aboard David. I've also got some cfengine experience, among other things :) -- Matthew Galgoci GIS Production Operations Red Hat, Inc 919.754.3700 x44155 From mmcgrath at fedoraproject.org Fri Dec 22 19:31:21 2006 From: mmcgrath at fedoraproject.org (Mike McGrath) Date: Fri, 22 Dec 2006 13:31:21 -0600 Subject: Introduction (and cfengine) In-Reply-To: References: <458C2C1B.2050301@ticon.net> <3237e4410612221119i4628c1ecm7418a5ce1c2e873c@mail.gmail.com> Message-ID: <3237e4410612221131n7fc4b21gc1202ea2cda3998c@mail.gmail.com> On 12/22/06, Matthew Galgoci wrote: > > I've also got some cfengine experience, among other things :) > Thumbs up or down? -Mike From mgalgoci at redhat.com Fri Dec 22 19:32:24 2006 From: mgalgoci at redhat.com (Matthew Galgoci) Date: Fri, 22 Dec 2006 14:32:24 -0500 Subject: Introduction (and cfengine) In-Reply-To: <3237e4410612221131n7fc4b21gc1202ea2cda3998c@mail.gmail.com> References: <458C2C1B.2050301@ticon.net> <3237e4410612221119i4628c1ecm7418a5ce1c2e873c@mail.gmail.com> <3237e4410612221131n7fc4b21gc1202ea2cda3998c@mail.gmail.com> Message-ID: > Date: Fri, 22 Dec 2006 13:31:21 -0600 > From: Mike McGrath > To: Matthew Galgoci > Cc: David Douthitt , fedora-infrastructure-list at redhat.com > Subject: Re: Introduction (and cfengine) > > On 12/22/06, Matthew Galgoci wrote: > > > > I've also got some cfengine experience, among other things :) > > > > Thumbs up or down? It does the job. I've not evaluated puppet. I've been told puppet picks up where cfengine left off. -- Matthew Galgoci GIS Production Operations Red Hat, Inc 919.754.3700 x44155 From ssrat at ticon.net Fri Dec 22 19:39:16 2006 From: ssrat at ticon.net (David Douthitt) Date: Fri, 22 Dec 2006 13:39:16 -0600 Subject: cfengine overview - pros and cons Message-ID: <458C3464.6030206@ticon.net> I've used cfengine in a production environment, and found it to be very useful and powerful. I'll just list the features (pro and con) below. PROS ---- * Distributed operations * Well-supported and open-source leader in its field * Widely-used * Supports many "selection critera" such as hour of day, hostname, IP address, network, cfengine version, operating system, kernel version" * Battle-tested with environments numbering in thousands (including that most hostile of environments, the college campus) * Integrates well with other systems such as CVS, RCS, et al * Works well in isolation as well in distributed fashion - and can keep system protected while server is offline * Extremely flexible * Comprehensive documentation * Can replace cron entirely (if one has a notion to...) * Can keep excess files from cluttering up /tmp or /var/tmp * Can keep unwanted files or processes from appearing at all (such as .rhosts, etc). * Can "edit" files as well as maintain complete files * Utilizes public-key encryption to identify clients (encrypted links available) * "Selection criteria" (classes) can be set programmatically by scripts * Can be used in place of samhain or tripwire (and *reacts*!) * Works well with NFS-mounted home directories * Works under Windows as well * Can manage processes - including "must be present" and "must *not* be present" and more * Active mailing list for support * Can be used to configure new systems from startup (using a minimal configuration) CONS ---- * Documentation - comprehensive but can be hard to know where to start with new installations * Configuration is unlike anything you've ever seen * The "editfiles" section of the configuration is also unlike anything you've ever seen - and is different than any other configuration section (looks a lot like a computer language without reasonable syntax) * The customizability of the configuration can be overwhelming * Doesn't necessarily "play nice" with file integrity checkers like samhain or tripwire - i.e., if cfengine restores a file to its original state or changes the permissions samhain may flag it as being changed. * Inclusion in configuration files ("include file") is counter-intuitive: "included files" are actually concatenated to currently scanned file * "Regexes" in the EditFiles configuration section match the entire line, not a substring (unless using proper EditFiles command) Most of the down-side to cfengine revolves around the unique configuration file syntax (and the EditFiles section most of all) and the comprehensive documentation (which does not provide for an oft-requested 1-2-3 steps to get started). The latter problem will be solved with an upcoming book ;-) -- David Douthitt HP-UX, Unixware, Linux, FreeBSD RHCE, SCSA, Linux+, LPIC-1 http://www.lulu.com/ssrat From dlutter at redhat.com Sat Dec 23 00:17:45 2006 From: dlutter at redhat.com (David Lutterkort) Date: Fri, 22 Dec 2006 16:17:45 -0800 Subject: puppet overview - pros and cons Message-ID: <1166833065.15663.77.camel@localhost.localdomain> Since introductions are all the vogue today, here's some background about me: I work in Red Hat's Emerging Technologies group on systems management things; a little over a year ago I got interested in configuration management and started looking around for a tool that could fill the gaps left by the current tool chain that people use (well, a very short chain generally, mostly made up of package management and a little bit of source control) During that time, I looked at pretty much all the config mgmt tools out there, and found that puppet has the most promise of the lot, both for straightup config mgmt and for pushing the envelope of what can be done there (e.g., distributing detailed configurations in a reusable way[1]) Before that, I worked on RHN for a while, and before that I did a lot of consulting work for Red Hat, mostly around J2EE web applications. I used to know TCL, but the rehab really helped. In the interest of full disclosure, I actively contribute to puppet and work on stuff building on it. I will be out of town until Jan. 2nd, with unclear email access, so I might be a little sluggish with responses - but you can always ask questions on puppet-dev at reductivelabs.com or #puppet on freenode. Puppet ====== The references like [N] are at the end and lead to docs/additonal info PROS ---- * Project lead (Luke Kanies) is experienced as sysadmin and consultant around system administratio, makes his living exclusively off consulting around puppet * Designed and implemented in direct response to experiences with other (and no) config mgmt systems like cfengine [5], isconf, sth proprietary etc. * Architecture * Clients connect to central server (but all sane cfg mgmt tools do that) * Clients report facts about themselves (OS/kernel version/release, MAC/IP address, basic HW info) to central server, which uses them to make decisions about client's config; the fact mechanism is pluggable and can be easily extended with custom facts * Server assembles config for client from sitewide description (manifest) * Can also be used standalone with cmd line tool for testing (or dirt simple single machine setups) * Use 'native' tools for all config tasks in the backend (e.g., yum for pkg mgmt on RH-derived systems) * Security * Thorough security model (each client has its own SSL cert) Puppet comes with tools to make basic SSL setup and cert generation very painless (puppetca) * Each client only gets to see the part of the site config that applies to it, not the whole site config * Builtin file server where file access can be secured per-client (e.g. only hostX gets access to hostX/ssh_host_key) * Cross-platform, works on most flavors of Unix (Fedora/RHEL/Debian/Gentoo, Solaris, OS X, some sort of *BSD IIRC) * Domain-specific language for manifest [2] * Clean abstraction from messy details of changing config * Describe desired config of system, puppet figures out how to get there (e.g., you say 'need user X with homedir /foo and uid N', puppet figures out appropriate calls to useradd/usermod depending on whether user exists and fixes attributes that are out of sync) * Abstraction: describe config in high-level terms (user, service, package, mount) for common config objects [3] * Templating support for things that can't/don't need to be described as objects; or distribute complete files * Group config items logically with classes: can describe that a webserver has to have latest httpd package, service httpd enabled and running, and custom httpd.conf file from location X (that's not possible with at least one of the other config mgmt tools) * Override mechanism for classes to allow for simple one-off (or hundred-off) tweaks, e.g. to take webserver class from above but use with different httpd.conf * Clean definition of what inputs can influence a client's config * Language makes config easily readable and comprehensible IMHO * Emphasis on practical usability, not research * Good set of unit tests * No EditFiles ;) * Cron-like support for scheduling actions during maintenance windows (on a per-config object basis, if need be, though in reality you want to keep that simple for your own sanity) * Tie-in with kickstart: provision basic system with ks (including puppet client), complete config with puppet [4] * RH interested in furthering it for other reasons, too * Active community, Luke is very responsive both with developer and user issues/questions * Beginnings of task-oriented user docs on a Wiki [6] * GPL CONS ---- * Not everybody is familiar with puppet's implementation language (Ruby) * Evolves rapidly * Some of the more esoteric features (like comprehensive reporting) are immature * Need to learn puppet's language to describe site config * Scalability in very large deployments unknown (there are production deployments in the low hundreds of machines) * Language is mostly declarative, but has 'exec' loophole for running arbitrary commands on the client for practical reasons More info --------- Puppet's website (http://reductivelabs.com/projects/puppet/) has lots of more info; if you want to get more of an impression, I would start with the following, in this order: 1. http://reductivelabs.com/projects/puppet/faq.html 2. Luke's BayLISA presentation from last year (http://video.google.co.uk/videosearch?q=Kanies+puppet) - the ones from August '06 are also very good but _long_ 3. The high-level introduction (http://reductivelabs.com/projects/puppet/documentation/introduction.html) 4. Luke's puppet/cfengine comparison (http://reductivelabs.com/projects/puppet/documentation/notcfengine.html) and his blog post about BCFG2 (http://www.madstop.com/articles/2006/08/08/puppet-vs-bcfg2) - gives some more insight into the why's and how's of puppet and how the main author contrasts it with what's out there. 5. The language tutorial http://reductivelabs.com/projects/puppet/documentation/languagetutorial.html David [1] http://people.redhat.com/dlutter/puppet-app.html [2] http://reductivelabs.com/projects/puppet/documentation/languagetutorial.html [3] http://reductivelabs.com/projects/puppet/documentation/typedocs.html [4] http://watzmann.net/blog/index.php/2006/12/05/kickstarting_into_puppet [5] http://reductivelabs.com/projects/puppet/documentation/notcfengine.html [6] http://reductivelabs.com/cookbook/ From Axel.Thimm at ATrpms.net Sat Dec 23 02:25:04 2006 From: Axel.Thimm at ATrpms.net (Axel Thimm) Date: Sat, 23 Dec 2006 03:25:04 +0100 Subject: puppet overview - pros and cons In-Reply-To: <1166833065.15663.77.camel@localhost.localdomain> References: <1166833065.15663.77.camel@localhost.localdomain> Message-ID: <20061223022504.GJ1625@neu.nirvana> Thanks for the puppet overview. Maybe we should create a wiki page with a product/feature matrix? On Fri, Dec 22, 2006 at 04:17:45PM -0800, David Lutterkort wrote: > PROS > ---- > * Override mechanism for classes to allow for simple > one-off (or hundred-off) tweaks, e.g. to take webserver > class from above but use with different httpd.conf That's nice. > * Tie-in with kickstart: provision basic system with ks (including > puppet client), complete config with puppet [4] That, of course, defeats lots of the security items, but there is no otherway. One just needs to be aware of that. What about diffing support, e.g. compare what's there and what we'd like it to become, kind of a dry-run session to check deviations between spec and reality? -- Axel.Thimm at ATrpms.net -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From smooge at gmail.com Sat Dec 23 18:06:02 2006 From: smooge at gmail.com (Stephen John Smoogen) Date: Sat, 23 Dec 2006 11:06:02 -0700 Subject: cfengine overview - pros and cons In-Reply-To: <458C3464.6030206@ticon.net> References: <458C3464.6030206@ticon.net> Message-ID: <80d7e4090612231006p70bc5721p40cad137b9ee6e50@mail.gmail.com> On 12/22/06, David Douthitt wrote: > I've used cfengine in a production environment, and found it to be very > useful and powerful. I'll just list the features (pro and con) below. > > PROS > ---- > * Distributed operations * Self contained binary. > > CONS > ---- > * Documentation - comprehensive but can be hard to know where to start > with new installations > * Configuration is unlike anything you've ever seen > * The "editfiles" section of the configuration is also unlike anything > you've ever seen - and is different than any other configuration section Actually almost every section has its own variants of the syntax. The second con is that this is a research project for the author and not much else.. this can make dealing with problems a bit of a headache when he has completely theoretical issues he wants to try out. -- Stephen J Smoogen. -- CSIRT/Linux System Administrator How far that little candle throws his beams! So shines a good deed in a naughty world. = Shakespeare. "The Merchant of Venice" From email.ahmedkamal at googlemail.com Sat Dec 23 20:19:57 2006 From: email.ahmedkamal at googlemail.com (Ahmed Kamal) Date: Sat, 23 Dec 2006 22:19:57 +0200 Subject: Web torture results Message-ID: <3da3b5b40612231219h47fd26c2oe59e4a3d2f99c8f2@mail.gmail.com> Hi, Paulo and me (kim0) have been working on testing a caching setup for the wiki. A test migration to Moin 1.5 is complete, squid is now configured as a reverse caching proxy. We've done some stress tests on the current setup. Attached are some extracted results that (I think) are of interest. Mainly the number of requests served per second, and the average time for serving a request. Also, paulo pointed that caching differs per file type, the tests have been done on three different file types (html, png image, and css) Test Setup: ========= 1- All connections were initiated from proxy1 2- Proxy2 had squid caching turned on 3- Testing for html/png/css done, sweeping the number of concurrent connections 4- Turn off squid caching on proxy2 5- Testing for html/png/css done, sweeping the number of concurrent connections again Interesting notes: ============ 1- Serving PNG is 10X faster than html 2- Serving CSS is 10X faster than PNG! 3- Serving html is really the bottleneck. Unfortunately, Moin developers acknowledged current version (1.5) is not cache friendly. Work for making 1.6 cache friendly is undergoing 4- Using squid currently only seems to double our PNG serving rate, nothing else 5- The application server hits swapping (about 0.5GB) at full load (~300 concurrent connections), for some reason the requests/second served is still high!! (Is our cache disks that fast) 6- The test did not stress the server bad enough to run out of swap space, not sure if this is needed though! I can send the full results date if anyone is interested. Best Regards -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: results.ods Type: application/vnd.oasis.opendocument.spreadsheet Size: 43867 bytes Desc: not available URL: From paulo.banon at googlemail.com Sat Dec 23 21:49:14 2006 From: paulo.banon at googlemail.com (Paulo Santos) Date: Sat, 23 Dec 2006 21:49:14 +0000 Subject: Web torture results In-Reply-To: <3da3b5b40612231219h47fd26c2oe59e4a3d2f99c8f2@mail.gmail.com> References: <3da3b5b40612231219h47fd26c2oe59e4a3d2f99c8f2@mail.gmail.com> Message-ID: <7a41c4bc0612231349s7863808eva3a89f087c171334@mail.gmail.com> Hi all, The PNG link i gave kim0 was just an example of an image. Currently we are caching all general type of images (gif, jpg, png). As Ahmed pointed, current version of Moin isn't still html friendly, hopefully this will be addressed at Moin 1.6.x (our current version is Moin 1.3.4 and our testing was based on Moin 1.5.6). Moin 1.5.6, still bring alot of improvements, so in my opinion, even though that Moin html is not cache friendly we should still be thinking in the migration. Also as a workaround, we can still create a static website that with the help of mod_headers, it could be forced to be cached. Last but not least, let's not forget that this was a local test. A more real stress test, is still going to be prepared and should include a server requesting to the future architecture (including load balancer and both proxies), this will improve even more our ability to deliver the desired content. We would like some coments on this tests (past and future), so that we can improve what needs to be improved. As a final note, i would like all of you to start testing the future platform[1]. Just play with Moin, and give some feedback on the WikiMigration page[2]. [1] http://webtest.fedora.redhat.com/wiki [2] http://fedoraproject.org/wiki/Infrastructure/WikiMigration Once again thanks all for the help and feedback... Paulo On 12/23/06, Ahmed Kamal wrote: > > Hi, > Paulo and me (kim0) have been working on testing a caching setup for the > wiki. A test migration to Moin 1.5 is complete, squid is now configured as > a reverse caching proxy. We've done some stress tests on the current setup. > Attached are some extracted results that (I think) are of interest. Mainly > the number of requests served per second, and the average time for serving a > request. Also, paulo pointed that caching differs per file type, the tests > have been done on three different file types (html, png image, and css) > > Test Setup: > ========= > 1- All connections were initiated from proxy1 > 2- Proxy2 had squid caching turned on > 3- Testing for html/png/css done, sweeping the number of concurrent > connections > 4- Turn off squid caching on proxy2 > 5- Testing for html/png/css done, sweeping the number of concurrent > connections again > > Interesting notes: > ============ > 1- Serving PNG is 10X faster than html > 2- Serving CSS is 10X faster than PNG! > 3- Serving html is really the bottleneck. Unfortunately, Moin developers > acknowledged current version ( 1.5) is not cache friendly. Work for making > 1.6 cache friendly is undergoing > 4- Using squid currently only seems to double our PNG serving rate, > nothing else > 5- The application server hits swapping (about 0.5GB) at full load (~300 > concurrent connections), for some reason the requests/second served is still > high!! (Is our cache disks that fast) > 6- The test did not stress the server bad enough to run out of swap space, > not sure if this is needed though! > > I can send the full results date if anyone is interested. > > Best Regards > > _______________________________________________ > Fedora-infrastructure-list mailing list > Fedora-infrastructure-list at redhat.com > https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dlutter at redhat.com Sat Dec 23 23:31:19 2006 From: dlutter at redhat.com (David Lutterkort) Date: Sat, 23 Dec 2006 15:31:19 -0800 Subject: puppet overview - pros and cons In-Reply-To: <20061223022504.GJ1625@neu.nirvana> References: <1166833065.15663.77.camel@localhost.localdomain> <20061223022504.GJ1625@neu.nirvana> Message-ID: <1166916679.15663.98.camel@localhost.localdomain> On Sat, 2006-12-23 at 03:25 +0100, Axel Thimm wrote: > Thanks for the puppet overview. Maybe we should create a wiki page > with a product/feature matrix? > > On Fri, Dec 22, 2006 at 04:17:45PM -0800, David Lutterkort wrote: > > > PROS > > ---- > > > > * Override mechanism for classes to allow for simple > > one-off (or hundred-off) tweaks, e.g. to take webserver > > class from above but use with different httpd.conf > > That's nice. One thing I forgot to mention: puppet lets you also express dependencies between config elements, so that you can say that the httpd package should be installed/upgraded first before your custom httpd.conf is copied from the central server. A similar mechanism also lets you say that service X should be restarted whenever puppet needs to make a change to file Y (useful to keep service restarts to the absolute minimum needed) > > * Tie-in with kickstart: provision basic system with ks (including > > puppet client), complete config with puppet [4] > > That, of course, defeats lots of the security items, but there is no > otherway. One just needs to be aware of that. Yeah, it's not entirely watertight, but the holes that it leaves are fairly esoteric (somebody posing as a client while you are installing the client; somebody posing as the server while you are installing a client), and you'll discover both breaches quickly (when the client you are installing gets to the point where it asks the server for a cert; the request will either fail or never show up on your server) > What about diffing support, e.g. compare what's there and what we'd > like it to become, kind of a dry-run session to check deviations > between spec and reality? Both the command-line tool and the client daemon have a 'noop' option, that goes through the motions w/o making changes. After changes to configs, I usually test them by running # /usr/sbin/puppetd -v --onetime --noop which produces output like notice: Starting configuration run notice: //common/lemon/autofs-server/configfile[/etc/sysconfig/autofs]/file=/etc/sysconfig/autofs/source: is {md5}6cd456ce35b212ea429e63bf892d2535, should be {md5}3ed0678eb6af27c0a264522705e79a08 (noop) notice: //common/lemon/autofs-server/configfile[/etc/auto.homes]/file=/etc/auto.homes/owner: is build, should be root (noop) notice: //common/lemon/users/user=build/home: is /home/build, should be /homes/build (noop) notice: Finished configuration run in 4.56 seconds The lines are telling me that * /etc/sysconfig/autofs has different content from what it should be * /etc/auto.homes has the wrong owner * the user 'build' has the wrong home directory Of course, puppet would fix all these if I ran that without --noop David From mmcgrath at fedoraproject.org Sun Dec 24 03:35:11 2006 From: mmcgrath at fedoraproject.org (Mike McGrath) Date: Sat, 23 Dec 2006 21:35:11 -0600 Subject: Web torture results In-Reply-To: <7a41c4bc0612231349s7863808eva3a89f087c171334@mail.gmail.com> References: <3da3b5b40612231219h47fd26c2oe59e4a3d2f99c8f2@mail.gmail.com> <7a41c4bc0612231349s7863808eva3a89f087c171334@mail.gmail.com> Message-ID: <3237e4410612231935x5c3ec65eob61f7cfe541317c3@mail.gmail.com> On 12/23/06, Paulo Santos wrote: > > 3- Serving html is really the bottleneck. Unfortunately, Moin developers > acknowledged current version ( 1.5) is not cache friendly. Work for making It sounds like we really won't get much of a benefit till 1.6. Our biggest issue last time appeard to be the dynamic generation of content, converting them to static content helped cpu load on fpserv quite a bit. How did the appserver handle the stress test? How about the proxy server? Not just in code generation and such but actual cpuload, that type of thing? -Mike From ssrat at ticon.net Sun Dec 24 03:45:00 2006 From: ssrat at ticon.net (David Douthitt) Date: Sat, 23 Dec 2006 21:45:00 -0600 Subject: cfengine overview - pros and cons In-Reply-To: <80d7e4090612231006p70bc5721p40cad137b9ee6e50@mail.gmail.com> References: <458C3464.6030206@ticon.net> <80d7e4090612231006p70bc5721p40cad137b9ee6e50@mail.gmail.com> Message-ID: <458DF7BC.3070808@ticon.net> Stephen John Smoogen wrote: > On 12/22/06, David Douthitt wrote: >> I've used cfengine in a production environment, and found it to be very >> useful and powerful. I'll just list the features (pro and con) below. >> CONS >> ---- >> * Documentation - comprehensive but can be hard to know where to start >> with new installations >> * Configuration is unlike anything you've ever seen >> * The "editfiles" section of the configuration is also unlike anything >> you've ever seen - and is different than any other configuration section > > Actually almost every section has its own variants of the syntax. The syntax is at least visually and apparently similar and nearly consistent, though EditFiles is completely unusual. > The second con is that this is a research project for the author and > not much else.. this can make dealing with problems a bit of a > headache when he has completely theoretical issues he wants to try > out. I've heard this mentioned before, but I don't really see it. As one reads the documentation it also becomes apparent that the author is a campus system administrator (in some fashion), and has to deal with system administration problems as well as anyone. My thought was that the EditFiles sections begs for a complete miniature language of its own (like awk or lua or guile) but provides nothing of the sort, and does not provide a consistent language at all. The other mentioned "one-off" pro for puppet is a cfengine FAQ, but the usual answer is: don't create "one-off" syntax settings; define the *state* to be attained and let the system maintain the state. -- David Douthitt HP-UX, Unixware, Linux, FreeBSD RHCE, SCSA, Linux+, LPIC-1 http://www.lulu.com/ssrat From ssrat at ticon.net Sun Dec 24 04:02:59 2006 From: ssrat at ticon.net (David Douthitt) Date: Sat, 23 Dec 2006 22:02:59 -0600 Subject: puppet overview - pros and cons In-Reply-To: <1166916679.15663.98.camel@localhost.localdomain> References: <1166833065.15663.77.camel@localhost.localdomain> <20061223022504.GJ1625@neu.nirvana> <1166916679.15663.98.camel@localhost.localdomain> Message-ID: <458DFBF3.3070102@ticon.net> David Lutterkort wrote: >> What about diffing support, e.g. compare what's there and what we'd >> like it to become, kind of a dry-run session to check deviations >> between spec and reality? >> > Both the command-line tool and the client daemon have a 'noop' option, > that goes through the motions w/o making changes. After changes to > configs, I usually test them by running > # /usr/sbin/puppetd -v --onetime --noop Cfengine also has a "dry run" option, though only on the "client" side - then it is only the client that can make changes: cfagent -v --no-splay --dry-run Splay is the client's "run time delay" in order to spread out the load onto the server. Low-tech (in my opinion) and not completely scalable, but decent enough nonetheless. -- David Douthitt HP-UX, Unixware, Linux, FreeBSD RHCE, SCSA, Linux+, LPIC-1 http://www.lulu.com/ssrat From a.badger at gmail.com Fri Dec 22 21:22:27 2006 From: a.badger at gmail.com (Toshio Kuratomi) Date: Fri, 22 Dec 2006 13:22:27 -0800 Subject: Package Database Schema v0.4 Message-ID: <1166822547.19928.77.camel@localhost.localdomain> Here's the latest version of the package DB schema. Thanks to Karel, Jeff, and Sopwith for the comments on the last version! Some of the things that still need to be worked on: - I've added several triggers. These need to be tested. - The relationship between PackageBuild, PackageListing, PackageBuildListing, and Package is ugly. * PackageListing tells that a Package is present in a Collection * PackageBuild is a specific build of a package (PackageId, EVR make these records unique) * PackageBuildListing combines these two (as a Build may belong to more than one Collection.) We want the PackageBuild to belong to one or more Collections that the Package belongs to. I'm currently using a trigger to try to enforce that but I have the nagging feeling I've just designed something wrong. - Package Groups (for collections) and Package Categories (for packages) should now be possible. Need to implement them. - Review grant statements once we've finalized the tables we'll be providing. Here's a short ChangeLog: Move the StatusCodes into their own table to make translations easier. This involves several new tables: - StatusCode holds the status codes. - StatusCodeTranslation holds the translated strings for each status code. This is prefilled with the C translations. - *StatusCode are tables that hold a subset of the StatusCode table. These are used in foreign key relations to limit the status codes that can be used on those tables. (For instance, Collection.status is a foreign key of CollectionStatusCode.) - *LogStatusCode are tables that holds all the status codes plus other statuses that belong to logs. Since logs record status changes plus "Added" and "Removed", these are generated from the StatusCode. There is also a trigger to keep the *LogStatusCode tables in sync with their *StatusCode tables. (Thanks to jcollie for the idea to do this) Branch: Make distTag and branchName unique values. (Thanks Karel) Add on delete and on update clauses to all foreign keys. (Thanks Karel) CollectionSet: Add a priority field to specify the search order when overlaying collections. (Thanks f13) Rename PackageVersion* to PackageBuild* Trigger to make sure PackageBuildListing references PackageBuilds and PackageListings with the same Package. Restructure PackageACL, PersonPackageACL, and GroupPackageACL. The ACL list can now have one record for every ACL-Package combination. *PackageACL tables add users and groups to the relevant ACL. (Thanks Karel) Trigger to make changing the acl field illegal. This prevents possible abuse. Add a description field to Log for possible extra information. Add some grant statements to give out permissions for pkgdbadmin to do useful things in the db. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: pkgdb.sql Type: text/x-vhdl Size: 27580 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From dlutter at redhat.com Sun Dec 24 05:15:41 2006 From: dlutter at redhat.com (David Lutterkort) Date: Sun, 24 Dec 2006 05:15:41 +0000 Subject: cfengine overview - pros and cons In-Reply-To: <458DF7BC.3070808@ticon.net> References: <458C3464.6030206@ticon.net> <80d7e4090612231006p70bc5721p40cad137b9ee6e50@mail.gmail.com> <458DF7BC.3070808@ticon.net> Message-ID: <1166937341.31616.5.camel@localhost.localdomain> On Sat, 2006-12-23 at 21:45 -0600, David Douthitt wrote: > The other mentioned "one-off" pro for puppet is a cfengine FAQ, but the > usual answer is: don't create "one-off" syntax settings; define the > *state* to be attained and let the system maintain the state. I don't understand ... is that in reference to my remark about overrides and how they are useful to modify a standard set of configs ? When you override, you still specify the desired state; and syntactically it's not 'one-off' - it's just useful when you have to deal with sth that is the same across most machines, and you need a way to tweak that config for a few machines that need some small changes. David From ssrat at ticon.net Sun Dec 24 18:47:02 2006 From: ssrat at ticon.net (David Douthitt) Date: Sun, 24 Dec 2006 12:47:02 -0600 Subject: cfengine, TemplateTree II, and other research Message-ID: <458ECB26.1040900@ticon.net> Anyone looked at TemplateTree II? It's a cfengine configuration preprocessor from Tobias Oetiker (the fellow who created RRDTool). It was discussed at LISA '01 by the author: Paper: http://www.usenix.org/events/lisa01/tech/full_papers/oetiker/oetiker.pdf Home: http://isg.ee.ethz.ch/tools/isgtc/index.cgi?page=module_pod;module=tetre2;pod=docs/overview.pod There is also the System Configuration booklet from SAGE (Booklet #14) for a discussion, as well as a PDF from Gridweaver examining the various possibilities: http://www.gridweaver.org/WP1/report1.pdf -- David Douthitt HP-UX, Unixware, Linux, FreeBSD RHCE, SCSA, Linux+, LPIC-1 http://www.lulu.com/ssrat From email.ahmedkamal at googlemail.com Sun Dec 24 22:55:19 2006 From: email.ahmedkamal at googlemail.com (Ahmed Kamal) Date: Mon, 25 Dec 2006 00:55:19 +0200 Subject: Web torture results In-Reply-To: <3237e4410612231935x5c3ec65eob61f7cfe541317c3@mail.gmail.com> References: <3da3b5b40612231219h47fd26c2oe59e4a3d2f99c8f2@mail.gmail.com> <7a41c4bc0612231349s7863808eva3a89f087c171334@mail.gmail.com> <3237e4410612231935x5c3ec65eob61f7cfe541317c3@mail.gmail.com> Message-ID: <3da3b5b40612241455n378ceea7x44dea44b278991f@mail.gmail.com> Under full load (~200 concurrent connections), this was the status Proxy2 machine =========== 1- RAM 100% used, machine not swapping though (Linux ram utilization is a bit complex though) 2- Squid was saturating the CPU (80~90%) 3- Lots and lots of httpd processess, each taking 0% cpu, and minimal ram ~0.7% 4- load average: 1.06, 0.94, 0.75. top is showing squid cpu utilization > 85%, while total cpu utilization = 7%!! This seems to be a bug?! So, I cant trust the load numbers either App1 server ========= 1- RAM 100% used, 0.5GB swapping 2- CPU saturated by large number of httpd processes 3- load average: 150.35, 143.94, 104.04. Cacti showing load peaking to 400, while top doesnt go that high! https://admin.fedoraproject.org/cacti/graph.php?action=zoom&local_graph_id=68&rra_id=0&view_type=tree&graph_start=1166864400&graph_end=1166878800 Best Regards On 12/24/06, Mike McGrath wrote: > > On 12/23/06, Paulo Santos wrote: > > > 3- Serving html is really the bottleneck. Unfortunately, Moin > developers > > acknowledged current version ( 1.5) is not cache friendly. Work for > making > > It sounds like we really won't get much of a benefit till 1.6. Our > biggest issue last time appeard to be the dynamic generation of > content, converting them to static content helped cpu load on fpserv > quite a bit. > > How did the appserver handle the stress test? How about the proxy > server? Not just in code generation and such but actual cpuload, that > type of thing? > > -Mike > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ask at develooper.com Mon Dec 25 15:13:11 2006 From: ask at develooper.com (=?ISO-8859-1?Q?Ask_Bj=F8rn_Hansen?=) Date: Mon, 25 Dec 2006 16:13:11 +0100 Subject: Web torture results In-Reply-To: <3da3b5b40612241455n378ceea7x44dea44b278991f@mail.gmail.com> References: <3da3b5b40612231219h47fd26c2oe59e4a3d2f99c8f2@mail.gmail.com> <7a41c4bc0612231349s7863808eva3a89f087c171334@mail.gmail.com> <3237e4410612231935x5c3ec65eob61f7cfe541317c3@mail.gmail.com> <3da3b5b40612241455n378ceea7x44dea44b278991f@mail.gmail.com> Message-ID: On Dec 24, 2006, at 23:55, Ahmed Kamal wrote: > 1- RAM 100% used, 0.5GB swapping You should configure the web server for fewer concurrent connections. Having it swap is most certainly bad for performance (unless it's swapping unrelated and inactive programs). - ask -- http://develooper.com/ - http://askask.com/ From ask at develooper.com Mon Dec 25 15:16:32 2006 From: ask at develooper.com (=?ISO-8859-1?Q?Ask_Bj=F8rn_Hansen?=) Date: Mon, 25 Dec 2006 16:16:32 +0100 Subject: Web torture results In-Reply-To: <3da3b5b40612231219h47fd26c2oe59e4a3d2f99c8f2@mail.gmail.com> References: <3da3b5b40612231219h47fd26c2oe59e4a3d2f99c8f2@mail.gmail.com> Message-ID: Instead of Squid it might be worthwhile looking at Varnish for the reverse-proxy cache. http://phk.freebsd.dk/pubs/varnish_tech.pdf http://varnish.linpro.no/ - ask From sopwith at gmail.com Tue Dec 26 03:04:32 2006 From: sopwith at gmail.com (Elliot Lee) Date: Mon, 25 Dec 2006 22:04:32 -0500 Subject: Web torture results In-Reply-To: <3da3b5b40612231219h47fd26c2oe59e4a3d2f99c8f2@mail.gmail.com> References: <3da3b5b40612231219h47fd26c2oe59e4a3d2f99c8f2@mail.gmail.com> Message-ID: Hey Ahmed, It's neat to see these results! A few comments: I don't see anywhere where variables such as the file size, request size, HTTP pipelining, etc. are taken into account. Optimally, the test would have cases that exercise points along all these dimensions. E.g. a load test of serving an empty file would help us understand what the absolute best case performance is within the constraints the current systems. The other thing that'd be useful to have is a well-defined capacity goal that helps everyone understand where things ought to be. For a straw-man example, let's say 20 home page loads/second, including all associated image/CSS/JS files, by logged-in users. I don't know what the actual needs are, but a look at the web logs during the FC6 release should help. From what's mentioned below, I also didn't quite understand whether squid was sitting in front of apache serving straight HTML/PNG/CSS, or whether it was moinmoin that contained all the content. Hopefully that's just something I missed in an earlier message. My current favorite treatise on optimizing overall web performance - http://www.die.net/musings/page_load_time/ Hope this helps, -- Elliot On Dec 23, 2006, at 3:19 PM, Ahmed Kamal wrote: > Hi, > Paulo and me (kim0) have been working on testing a caching setup > for the wiki. A test migration to Moin 1.5 is complete, squid is > now configured as a reverse caching proxy. We've done some stress > tests on the current setup. Attached are some extracted results > that (I think) are of interest. Mainly the number of requests > served per second, and the average time for serving a request. > Also, paulo pointed that caching differs per file type, the tests > have been done on three different file types (html, png image, and > css) > > Test Setup: > ========= > 1- All connections were initiated from proxy1 > 2- Proxy2 had squid caching turned on > 3- Testing for html/png/css done, sweeping the number of concurrent > connections > 4- Turn off squid caching on proxy2 > 5- Testing for html/png/css done, sweeping the number of concurrent > connections again > > Interesting notes: > ============ > 1- Serving PNG is 10X faster than html > 2- Serving CSS is 10X faster than PNG! > 3- Serving html is really the bottleneck. Unfortunately, Moin > developers acknowledged current version ( 1.5) is not cache > friendly. Work for making 1.6 cache friendly is undergoing > 4- Using squid currently only seems to double our PNG serving rate, > nothing else > 5- The application server hits swapping (about 0.5GB) at full load > (~300 concurrent connections), for some reason the requests/second > served is still high!! (Is our cache disks that fast) > 6- The test did not stress the server bad enough to run out of swap > space, not sure if this is needed though! > > I can send the full results date if anyone is interested. > > Best Regards > > _______________________________________________ > Fedora-infrastructure-list mailing list > Fedora-infrastructure-list at redhat.com > https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list From a.badger at gmail.com Thu Dec 28 08:12:18 2006 From: a.badger at gmail.com (Toshio Kuratomi) Date: Thu, 28 Dec 2006 00:12:18 -0800 Subject: May not be making the meeting Message-ID: <1167293538.19928.191.camel@localhost.localdomain> Hey all, I may not be able to make the FESCo Meeting tomorrow as I have to drop some family off at the airport. I'll probably be back by the time the Infrastructure Meeting starts but I'm not 100% certain of that. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From rzarouali at gmail.com Thu Dec 28 08:46:16 2006 From: rzarouali at gmail.com (Zarouali Rachid) Date: Thu, 28 Dec 2006 09:46:16 +0100 Subject: my introduction Message-ID: hy all, My name is Rachid Zarouali, I'm a linux sysadmin for the french cctld registry. i use linux since 1996 (redhat 4.0 as far as i remember). i used several distributions like Redhat, Suse, Debian my personal linux server is on Ubuntu ;) (no no there's no troll ). i have skills in system administration, network, security, a little bit in databases like mysql and postgresql ... i'm a linux network/sys admin for now almost 6 years, i've been working on different infrastructures , homogenous (*nix only) and heterogenous (win/*nix), wich had different size i've also been a solaris sysadmin for 2 years (Solaris 7). right now i'm professionally focused on projects like, wide deployement system (win/linux), monitoring , email infrastructure ... i put an eye on the schedule page and seen some tasks i could help on like xen, config management.... well, i hope this little introduction is enough ;) sincerely, Rachid Zarouali -------------- next part -------------- An HTML attachment was scrubbed... URL: From mmcgrath at fedoraproject.org Thu Dec 28 16:05:20 2006 From: mmcgrath at fedoraproject.org (Mike McGrath) Date: Thu, 28 Dec 2006 10:05:20 -0600 Subject: my introduction In-Reply-To: References: Message-ID: <3237e4410612280805j11850e70j5f299d45128d5eb4@mail.gmail.com> On 12/28/06, Zarouali Rachid wrote: > i put an eye on the schedule page and seen some tasks i could help on like > xen, config management.... Sounds good, take a look at threads started recently on configuration management, in particular with regards to puppet, glump and cfengine to get up to date on what has already been discussed, use the list archives. -Mike From a.badger at gmail.com Fri Dec 29 20:26:40 2006 From: a.badger at gmail.com (Toshio Kuratomi) Date: Fri, 29 Dec 2006 12:26:40 -0800 Subject: PackageDB progress Message-ID: <1167424000.19928.240.camel@localhost.localdomain> As a followup to yesterday's meeting, I've finished converting the owners.py script that imports owners.list and cvs information into the database. A sync against what's currently in owners.list has been added to the postgres database on test3 (where we're doing the packagedb testing.) To people who'd like to help out with the backend here's the tasks we have: 1) look over the db schema posted last week and propose changes. 2) Enhance the owners.py script to convert RHL and EPEL branches as well as FC branches. 3) c4chris has been working on getting review requests and approval information into the database. There are a pair of files APPROVED_trim.txt and REQUESTS_trim.txt on the server that have information from the mailing list approvals at the beginning of Extras. A build_list perl script that c4chris is working on transforms these into a csv file. Someone can work on pushing this data into the database. 4) Storing Groups and Categories for comps generation could be worked on. I have some ideas for the database schema but nothing solid yet. I'm gong to start working on a minimal TurboGears front end next with the idea of being able to take over from owners.list as a first milestone. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From mmcgrath at fedoraproject.org Sat Dec 30 05:32:08 2006 From: mmcgrath at fedoraproject.org (Mike McGrath) Date: Fri, 29 Dec 2006 23:32:08 -0600 Subject: PackageDB progress In-Reply-To: <1167424000.19928.240.camel@localhost.localdomain> References: <1167424000.19928.240.camel@localhost.localdomain> Message-ID: <3237e4410612292132g4d5d6850m4dc011124aa1cecd@mail.gmail.com> On 12/29/06, Toshio Kuratomi wrote: > As a followup to yesterday's meeting, I've finished converting the > owners.py script that imports owners.list and cvs information into the > database. A sync against what's currently in owners.list has been added > to the postgres database on test3 (where we're doing the packagedb > testing.) > > To people who'd like to help out with the backend here's the tasks we > have: > 1) look over the db schema posted last week and propose changes. > 2) Enhance the owners.py script to convert RHL and EPEL branches as well > as FC branches. > 3) c4chris has been working on getting review requests and approval > information into the database. There are a pair of files > APPROVED_trim.txt and REQUESTS_trim.txt on the server that have > information from the mailing list approvals at the beginning of Extras. > A build_list perl script that c4chris is working on transforms these > into a csv file. Someone can work on pushing this data into the > database. > 4) Storing Groups and Categories for comps generation could be worked > on. I have some ideas for the database schema but nothing solid yet. > > I'm gong to start working on a minimal TurboGears front end next with > the idea of being able to take over from owners.list as a first > milestone. > > -Toshio Awesome Toshio, seriously. For those of you out there looking to get involved, this is a high potential and very visible project to get involved with. Any takers? -Mike From keld at dkuug.dk Fri Dec 1 16:18:31 2006 From: keld at dkuug.dk (Keld =?iso-8859-1?Q?J=F8rn?= Simonsen) Date: Fri, 01 Dec 2006 16:18:31 -0000 Subject: L10N tools: Increase quality of translations In-Reply-To: <20061201014137.GA4556@nostromo.devel.redhat.com> References: <456F7388.1040005@glezos.com> <20061201014137.GA4556@nostromo.devel.redhat.com> Message-ID: <20061201154705.GB16226@rap.rap.dk> I would like a better way to distribute corrections to translations, after the release date. Many times the translations did not make it in time, for the relaese. This can be both for fedora/red hat programs, but also for other programs like kde, gnome and what-have-you. I note that, as Linux becomes more and more popular, end users are much less likely to understand English, and the translations are then mandatory for the programs to be usable. Could this be done as separate packages, or reissues of packages with more complete translations? and do we need some delta mechanisms for rpm packages to avoid waisting bnetwork bandwidth on small translations updates. Or is there a need to have several levels of updating, incliding translation updates, security updates, fixes, and facility updates, or should people just always be current? Do we have the necessary tools to make such things? best regards keld On Thu, Nov 30, 2006 at 08:41:37PM -0500, Bill Nottingham wrote: > Dimitris Glezos (dimitris at glezos.com) said: > > * Integrate better the handling of translation during a "local" package's > > lifecycle. Have a flag raised for a package update that introduces new strings > > so that translators can translate the new strings before the > > repackaging/updating. Include in the schedule for each release a "string freeze > > date" and a week later a "translation freeze date" and have all our packages > > rebuilt after the latter and before the actual release. > > String freezes are easy to do, it's just a matter of discipline > in sticking to them. > > > * Move po files on their own cvsroot on cvs.fedoraproject.org to reduce > > complexity and maintenance and to increase security (with a new group). > > >From a maintainer standpoint, the translations *need* to > be in the same SCM system as the code. Without that, you lose features > like branching, easy spinning of tarballs, etc. Moving just the > po files is not the answer. > > > * Start working with the complex and tricky path to upstream translations that > > no distribution has tackled yet in a successful way. Bring our translators > > closer to the upstream projects. > > Well, the simple answer is if you want to translate upstream, > go upstream. For desktop environments, such as GNOME or KDE, > this is fairly easy. For other projects, it is more complicated. > > Bill > > -- > Fedora-trans-list mailing list > Fedora-trans-list at redhat.com > https://www.redhat.com/mailman/listinfo/fedora-trans-list From nicolas.mailhot at laposte.net Tue Dec 5 08:16:25 2006 From: nicolas.mailhot at laposte.net (Nicolas Mailhot) Date: Tue, 05 Dec 2006 08:16:25 -0000 Subject: Package DB Schema v3 In-Reply-To: <1165298351.29086.409.camel@localhost.localdomain> References: <1165298351.29086.409.camel@localhost.localdomain> Message-ID: <22662.192.54.193.51.1165306561.squirrel@rousalka.dyndns.org> Le Mar 5 d?cembre 2006 06:59, Toshio Kuratomi a ?crit : > * Integrating comps/categories. I have some notes in comments. In > order to implement this we need to 1) decide if comps is the right thing > for this or if something closer to Nicolas Mailhot's suggestion is the > right way to go. Actually, I'd be fine if someone else's suggestion is taken as long as the grouping and sorting problem is led to rest. With a working packagedb one could hide the categorization in the db and only export final groups, but that would create two classes of repositories : those with RW access to the db and those without. On a related issue I'd like to know if I should follow up on the comps ralx-ng schema I posted on the other day, work on a schema for some new comps format, or if the large round of silence that followed the last schema posting means no one's really interested. Regards, -- Nicolas Mailhot From sundaram at fedoraproject.org Wed Dec 27 08:56:00 2006 From: sundaram at fedoraproject.org (Rahul Sundaram) Date: Wed, 27 Dec 2006 08:56:00 -0000 Subject: Official mirror requirements Message-ID: <45923466.7030406@fedoraproject.org> Hi What are the requirements other than bandwidth for the official mirrors listed at http://fedora.redhat.com/Download/mirrors.html. In particular do we require them to carry the complete copy including source images and packages? If not we should probably enforce that or list the mirrors which only have binary packages as partial mirrors. We can run some routine checks in a period basis automatically to cross verify this. Rahul