Meeting Log - 2009-06-11

[Ricky Zhou]

Also avaliable at:
Minutes: http://www.scrye.com/~kevin/fedora/fedora-meeting/2009/fedora-meeting.2009-06-11-20.00.html
Log: http://www.scrye.com/~kevin/fedora/fedora-meeting/2009/fedora-meeting.2009-06-11-20.00.log.html

20:00 < mmcgrath> #startmeeting
20:00 < fedbot> Meeting started Thu Jun 11 20:00:09 2009 UTC.  The chair is mmcgrath.
20:00 < fedbot> Information about MeetBot at http://wiki.debian.org/MeetBot , Useful Commands: #action #agreed #halp #info #idea #link #topic.
20:00  * ricky 
20:00 < mmcgrath> #topic Infrastructure -- Who's here?
20:00 -!- ChitleshGoorah1 [n=chitlesh at d54C48C73.access.telenet.be] has joined #fedora-meeting
20:00 -!- fedbot changed the topic of #fedora-meeting to: Infrastructure -- Who's here?
20:00  * ricky 
20:00  * LinuxCode 
20:00  * SmootherFrOgZ is
20:00  * lmacken 
20:00  * mmcgrath is here
20:00  * nirik is lurking around. 
20:01 < mmcgrath> Ok, well lets get started
20:01 < mmcgrath> #topic Infrastructure -- F11 Release
20:01 -!- fedbot changed the topic of #fedora-meeting to: Infrastructure -- F11 Release
20:01 -!- sijis [n=sijis at adsl-70-131-122-68.dsl.emhril.sbcglobal.net] has joined #fedora-meeting
20:01 < mmcgrath> .ticket 1395
20:01 < zodbot> mmcgrath: #1395 (Lessons Learned) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/1395
20:01 < mmcgrath> So what went wrong with this release?
20:02 < mmcgrath> All in all I think it went very well.
20:02 < nirik> very little I think. ;)
20:02 < mmcgrath> The early bit flip (thanks f13) provided a 75% mirror rate when the anouncement went out.
20:02  * herlo is lurking
20:02 < mmcgrath> thats much better then for F10.
20:02 < ricky> I didn't expect docs links to change, so we had to do some last-minute updates that made us a bit late
20:02 < ricky> We'll make sure to double check that better next time around.
20:02 -!- ChitleshGoorah2 [n=chitlesh at 123.5-241-81.adsl-dyn.isp.belgacom.be] has joined #fedora-meeting
20:02 < mmcgrath> Also, as mdomsch has pointed out many times, the torrents were screaming.  The fastest torrent dl's I've ever had during a release.
20:03  * sijis is here.
20:03 < mmcgrath> 15-20 minutes for 4 isos compared to 8 or 9 hours previously.
20:03  * ianweller 
20:03  * abadger1999 here
20:03 < nirik> yeah, due to them opening early we got a lot more seeders at release time.
20:03 -!- ldimagg__ [n=ldimaggi at pool-96-252-122-35.bstnma.fios.verizon.net] has joined #fedora-meeting
20:03 < mmcgrath> ricky: should we make that a docs task or an us task?
20:03 < ricky> That'll be a websites task, I'll send links to the test page to list in advance.
20:04  * johe_ here too, by the way
20:04 < mmcgrath> k
20:04 < mmcgrath> The other thing that went kind of wacky was the bit leaks we had the thursday or friday before hand.
20:04 < mmcgrath> I think f13 is working on those though.
20:05 < mmcgrath> Oh, one other thing.
20:05 < mmcgrath> Proxy3 was hurting.
20:05 < mmcgrath> And I think I know why but I need to do a bit more research.
20:05 < mmcgrath> long story short it was doing too much.
20:05 < mmcgrath> I think it's just because it's one of our slower boxes in terms of cpu and disk.
20:05 < ricky> Did the x86_64 vs. i386 difference also come into play?
20:06 < mmcgrath> ricky: not this time.  But proxy1 and proxy2 are kind of odd because they're behind a balancer.
20:06  * mmcgrath needs to fix that.
20:06 < ricky> Ah
20:06 < mmcgrath> But once the release was out the door....  Things went great.
20:06 < mmcgrath> Anyone have anything else on this topic?
20:06 < rjune_wrk> why are they behind the balancer, but the slowest box has to face the cold cruel world all alone?
20:07 < mmcgrath> rjune_wrk: well, we've got 4 sites, only one of which has a balancer.
20:07 < mmcgrath> so we've got 5 proxy servers, 2 behind the balancer, the others are all at the various sites.
20:07 -!- che [n=che at redhat/che] has joined #fedora-meeting
20:07 < rjune_wrk> ah, ok.
20:07 < mmcgrath> Ok, anyone have anything else?
20:07 < mmcgrath> Solid
20:08 < mmcgrath> #topic Infrastructure -- Fedora Community
20:08 -!- fedbot changed the topic of #fedora-meeting to: Infrastructure -- Fedora Community
20:08 < lmacken> We deployed our first TurboGears2 application this week :)
20:08 < mmcgrath> So fedora community shipped.
20:08 < mmcgrath> lmacken: you want to talk a bit about all of it?
20:08 < lmacken> Sure
20:08 < SmootherFrOgZ> \o/
20:08 < lmacken> Software speaking, not much went wrong.
20:08  * mmcgrath is talking about https://admin.fedoraproject.org/community/
20:08 < lmacken> I hit some insane traceback during staging that was caused by pointing to the unresolvabl
20:08 < lmacken> e db1.stg, instead of just db1.  Other than that, everything went very smooth.
20:09 < lmacken> It's using both of our new memecached servers, using postgres on db2 for the feed store, and currently load balancing to mod_wsgi on app1 and 2
20:09 < lmacken> and it's the snappiest I've ever seen the app as of yet
20:09 -!- sseiersen|away is now known as sseiersen
20:09 < lmacken> but anyway, I did another update today to fix a variety of bugs, and we're going to keep cranking on it
20:09 < mmcgrath> It also means we can, at our leasure, convert our apps to TG2.
20:09 < lmacken> I want to plan a post-mortem when spot gets back from vacation
20:10 < lmacken> so we can talk about what happened, and how we can do it better
20:10 < mmcgrath> lmacken: is the whole tg2 stack in Fedora yet?  I need to install and start playing with it.
20:10 < lmacken> the whole TurboGears2/FedoraCommunity/Moksha stack is in our fedora-infrastructure repo, and a chunk of it is still under review
20:10 < ricky> Does our tg2 support mean close-to automatic support for pylons as well?
20:10 < lmacken> ricky: yes
20:10 < ricky> Awesome
20:11 < lmacken> pylons should work fine now
20:11 < mmcgrath> excellent.
20:11 < mmcgrath> Anyone have any questions for Luke on fedora community?
20:11 < lmacken> I will start writing up a TurboGears2 SOP
20:11 < abadger1999> I'd like to get together some questions to ask spot wrt licensing.
20:12 < ricky> I guess that can be discussed at the post-mortem as soon as spot is back - can that be on IRC?
20:12 < abadger1999> That way we can have less round trips to legal.
20:12 < abadger1999> IRC +1
20:12 < mmcgrath> <nod> I just want to be better educated about it before I copy and paste :)
20:12 < lmacken> yes, it will be public, probably on irc or asterisk
20:13 < rjune_wrk> IRC+1
20:13 -!- ks3 [n=ks3 at 74.203.195.1] has joined #fedora-meeting
20:13 < mmcgrath> Anyone have anything else on this?
20:13 < abadger1999> asterisk -1; IRC is better for public stuff.
20:13 < lmacken> mmcgrath: when do you wanna spread to app3-6?
20:13 < mmcgrath> lmacken: good work, thanks for the time you've put in on it.
20:13 < lmacken> abadger1999: agreed
20:14 < mmcgrath> lmacken: today or tomorrow.  I know you did some updates, were they straight rpm updates?
20:14 < lmacken> mmcgrath: thank you.  Thanks for your help as well, couldn't have done it without you guys
20:14 < lmacken> mmcgrath: yep, straight rpm updates
20:14 < mmcgrath> coolz.
20:14 < mmcgrath> ok so next topic
20:15 < mmcgrath> #topic Infrastructure -- Memcached
20:15 -!- fedbot changed the topic of #fedora-meeting to: Infrastructure -- Memcached
20:15 < mmcgrath> we've been slipping down the slippery slope that is memcached.
20:15 < mmcgrath> a month ago we were using it for sessions on the wiki.
20:15 -!- ldimaggi_ [n=ldimaggi at pool-96-252-122-35.bstnma.fios.verizon.net] has quit Read error: 110 (Connection timed out)
20:15 -!- phunster [n=lanny at pool-74-105-133-148.nwrknj.fios.verizon.net] has quit "Ex-Chat"
20:15 < mmcgrath> starting today we've got 3 apps using it and dedicated memcached servers.
20:16 < mmcgrath> the fas memcached stuff is going to exist just on the fas servers for security reasons.
20:16 < mmcgrath> everything else is on memcached[1-2]
20:16 < mmcgrath> ricky: do you want to remove memcached from puppet on app[1-2] or do you want me to
20:16 < mmcgrath> ?
20:16 < ricky> I'll get that after the meeting
20:16 < mmcgrath> solid
20:16 < mmcgrath> Nothing major here, just mentioning it.  Any questions?
20:16 -!- ChitleshGoorah [n=chitlesh at fedora/ChitleshGoorah] has quit Read error: 113 (No route to host)
20:16 < ricky> Speaking of security, I just want to let everybody know that load access into memcached = arbitrary command execution with python-memcached
20:17 < ricky> So be careful about that :-)
20:17 < mmcgrath> <nod>
20:17 -!- djf_jeff [n=jeff at modemcable026.33-70-69.static.videotron.ca] has quit "I quit"
20:17 < mmcgrath> k
20:18 < mmcgrath> #topic Infrastructure -- fas update
20:18 -!- fedbot changed the topic of #fedora-meeting to: Infrastructure -- fas update
20:18 < mmcgrath> So we just got done (well, are still doing) the fas updates.
20:18 < mmcgrath> This was a pretty major change as changes go.
20:18 < mmcgrath> Especially in how our shell accounts and aliases are generated.
20:18 < mmcgrath> So keep your eyes out for oddities.
20:19 < mmcgrath> So far things have been going fine, we've been running in staging for a while now without issue.
20:19 -!- ChitleshGoorah1 [n=chitlesh at d54C48C73.access.telenet.be] has quit Connection timed out
20:19 < mmcgrath> I'm going to be moving our fas-sync to run more often on some of our more public servers
20:19 < mmcgrath> fedorapeople, fedorahosted, cvs, etc.
20:19 < mmcgrath> Any questions there?
20:19 -!- rdieter [n=rdieter at fedora/rdieter] has quit Remote closed the connection
20:19 < ricky> Woohoo :-)  My goal is <= 5 minutes sync time
20:20 < mmcgrath> :)
20:20 < mmcgrath> Ok
20:20 < mmcgrath> #topic Infrastructure -- Outage this morning
20:20 -!- fedbot changed the topic of #fedora-meeting to: Infrastructure -- Outage this morning
20:20 < mmcgrath> So we had some doom this morning for those of you hanging out in #sysadmin-noc
20:20 < mmcgrath> Not sure what happened yet but I opened a bug about it
20:20 < mmcgrath> https://bugzilla.redhat.com/505352
20:20 < mmcgrath> xen13 up and died.
20:20 < mmcgrath> and took db2 with it, twice.
20:20 < mmcgrath> I moved db2 to a different host, hasn't happened since.
20:21 < mmcgrath> at the moment, nothing new on this so I'll move on.
20:21 < mmcgrath> #topic Infrastructure -- Pkgdb update
20:21 -!- fedbot changed the topic of #fedora-meeting to: Infrastructure -- Pkgdb update
20:21 < mmcgrath> abadger1999: we're on for a pkgdb rollout on Monday?
20:21 < abadger1999> that's right.
20:21 < mmcgrath> and this is a wsgi conversion?
20:21 < mmcgrath> any other changes?
20:21 < abadger1999> It's a major update that gets us using wsgi for the packagedb, all the bugfixes that accrued during change freeze,
20:22 < mmcgrath> I've still got smolt to convert.
20:22 < abadger1999> moving to usernames instead of userids i nthe public API.
20:22 < mmcgrath> after that I *think* all of our tg stuff will be good to go.
20:22 < ricky> Afdter pkgdb and smolt, will all of our apps be on haproxy + wsgi?
20:22 < abadger1999> Switching package status from the orphan owner to the status field.
20:22 < abadger1999> Lots of stuff changing.
20:23 -!- moixs [n=moixs at 81.13.143.9] has quit Remote closed the connection
20:23 < lmacken> I'll get a new bodhi ready to go at that time
20:23 < lmacken> I should probably get a new one into staging now so we can test bodhi#319
20:23 < mmcgrath> ricky: I believe so.
20:24 < abadger1999> lmacken: Excellent
20:24 < mmcgrath> lmacken: sounds good.
20:24 < mmcgrath> Anything else on this?
20:24 < abadger1999> Test out admin.stg.fedoraproject.org/pkgdb ! ;-)
20:24 < mmcgrath> hehehe
20:24 < mmcgrath> #topic Infrastructure -- releng1.stg
20:24 -!- fedbot changed the topic of #fedora-meeting to: Infrastructure -- releng1.stg
20:24 < abadger1999> I'm sure there's more bugs since this has major changes.
20:24 < ricky> After the pkgdb update, let's go through staging
20:24 < mmcgrath> speaking of staging
20:24 < mmcgrath> abadger1999: yeah :)
20:25 < mmcgrath> releng1.stg is nearing rediness.
20:25 < ricky> And make sure to get all the commits we want out of it before restarting the branch.
20:25 < mmcgrath> lmacken: I need to find out from you and jwb what is missing from it.  I think it's going to take a close eye to make sure we know what we're putting in staging.
20:25 < mmcgrath> cron jobs and things that might fail, that sort of thing.
20:25 < lmacken> mmcgrath: readonly mount of /mnt/koji?
20:25 < mmcgrath> lmacken: it'll have that.
20:26 < lmacken> ok, to test mashing and whatnot, we'll have to tweak some paths
20:26 < mmcgrath> lmacken: and I don't think anything mash like is installed yet.
20:26 < mmcgrath> but I figure we'll work through it one boy one until it's  usable for you and jwb.
20:27 < lmacken> ok, cool
20:27 < mmcgrath> I also assume you're going to need some sort of $TEMP_STORAGE
20:27 < lmacken> yeah
20:27 < mmcgrath> that's fairly large.  we'll figure that out when the time comes.
20:27 -!- knurd is now known as knurd_afk
20:27 < mmcgrath> was jwb at FAD?
20:27 < lmacken> yep
20:27 < mmcgrath> probably traveling / busy
20:27 < mmcgrath> we'll get ahold of him.
20:27 < mmcgrath> ok, ricky :)
20:27 < mmcgrath> #topic Infrastructure -- Staging
20:27 -!- fedbot changed the topic of #fedora-meeting to: Infrastructure -- Staging
20:28 < mmcgrath> So we need to refresh the staging branch at some point soon.
20:28 < mmcgrath> and make sure all of our staging commits get merged back with master.
20:28 < mmcgrath> ricky: what are a rundown of some of the commits left?
20:28 -!- cassmodiah [n=cass at fedora/cassmodiah] has quit "Verlassend"
20:28 < mmcgrath> I know pkgdb had a bunch.
20:29 < ricky> Here's the full diff: http://ricky.fedorapeople.org/staging.diff
20:29 -!- phunster [n=lanny at pool-74-105-133-148.nwrknj.fios.verizon.net] has joined #fedora-meeting
20:29 < ricky> It's mostly pkgdb, I thnk
20:29 < ricky> **think
20:29 < ricky> It's not that bad, so we might get away with not blowing it away
20:30 < ricky> But I guess that's something to look at once pkgdb changes have been merged.
20:30 < abadger1999> pkgdb will be synced after Monday, so that's good.
20:30 < mmcgrath> Yeah
20:30 < ricky> Cool.
20:30 < abadger1999> Do we have a good recipe for merging the changes yet?
20:30 < ricky> And thanks to mmcgrath for starting to merge often this week
20:30 < mmcgrath> All in all, considering we don't really monitor this interaction much... things are in better shape then they otherwise could be.
20:30 < mmcgrath> ricky: figured i'd make a habit of it
20:31 < ricky> abadger1999: It'll probably consist mostly of finding out what we want to keep or throw away
20:32 < ricky> The process of merging isn't that painful at all, just getting updates on half-finished or abandoned work in staging.
20:32 < abadger1999> ricky: I'm just wondering, can I copy changes from staging to production, commit, and git will figure it out?  Or do I need to use some git command to pick out specific changesets?
20:33 -!- Sonar_Guy [n=Who at fedora/sonarguy] has joined #fedora-meeting
20:33 < ricky> It's cleaner to cherry-pick, but I *think* hand-copying will work as well.  I haven't tried doing work that way though, so I'm not sure how well it gets handled.
20:33 < ricky> I think the diff is short enough for us to go through it manually though.
20:33 < mmcgrath> I've hand-copied with success in the past.
20:33 < mmcgrath> but try to use git just so I'm mo'betta with it.
20:35 < abadger1999> Okay.  We'll see what it looks like after Monday's merge ;-)
20:35 < mmcgrath> Ok, anything else on that?
20:35 < mmcgrath> yeah
20:35 -!- Sonar_Guy [n=Who at fedora/sonarguy] has quit Client Quit
20:35 < mmcgrath> With that, we'll open the floor
20:35 -!- Gremlin_ [n=lkrzemin at 77-97-161-32.cable.ubr04.donc.blueyonder.co.uk] has joined #fedora-meeting
20:35 < mmcgrath> #topic Infrastructure -- Open Floor
20:35 -!- fedbot changed the topic of #fedora-meeting to: Infrastructure -- Open Floor
20:35 < mmcgrath> Anyone have anything they'd like to discuss?
20:35 < LinuxCode> apart from a million questions I have ?
20:35 < LinuxCode> lol
20:35 < ricky> Ask away :-)
20:36 < LinuxCode> I will have to figure out what is done for what purpose the enxt weeks
20:36 -!- Gremlin_ [n=lkrzemin at 77-97-161-32.cable.ubr04.donc.blueyonder.co.uk] has quit Client Quit
20:36 -!- Sonar_Guy [n=Who at fedora/sonarguy] has joined #fedora-meeting
20:36 < LinuxCode> ricky, I ask you guys the next weeks, or we would be here all night
20:36 < LinuxCode> lol
20:36 < ricky> feel free to ask in #fedora-admin after the meeting as well
20:36 < LinuxCode> Id like to figure out how everything fits together
20:36 < LinuxCode> sure
20:36 < LinuxCode> I have one question though, the fas update was to kill bugs ?
20:37 < LinuxCode> it said so in the email I had
20:37 < abadger1999> LinuxCode: For some definition of bugs, yes.
20:37 < LinuxCode> abadger1999, I take it to make sure that people participate and not just sign up then ?
20:37 < abadger1999> Ther were bugs and there was also performance problems that were causing failures to occur.
20:37 < LinuxCode> ohh ok
20:37 < LinuxCode> i see
20:38 < ricky> The extra @fp.o aliases being created was a bug too
20:38 < LinuxCode> ricky, thats what I meant
20:38 < ricky> Unfortunately, it went unnoticed for a loong time.
20:38 < LinuxCode> would have been nice to give people a week though to sort themselves out
20:38 < ricky> The policy hasn't changed, it was just a bug that we weren't enforcing it.
20:38 < LinuxCode> I had the email today saying about the bug
20:39 < LinuxCode> hmm k
20:39 -!- sseiersen is now known as Video_juegos_not
20:39 < LinuxCode> well it doesnt matter now
20:39 < LinuxCode> for me anyway
20:39 < LinuxCode> ;-}
20:39 < mmcgrath> heheh
20:39 < mmcgrath> Anyone have anything else to discuss?  If not we'll close the meeting early.
20:39 -!- Video_juegos_not is now known as sseiersen
20:40 < mmcgrath> Closing in 30
20:40 < mmcgrath> 10
20:40 < mmcgrath> #endmeeting
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/fedora-infrastructure-list/attachments/20090611/24cd8b24/attachment.sig>