Meeting Log - 2008-08-07

Ricky Zhou ricky at
Thu Aug 7 20:50:30 UTC 2008

20:00 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Who's here?
20:00  * ianweller 
20:00 < mmcgrath> Howdy all, who's about?
20:00  * gregdek !
20:00  * ianweller !!!!!!!1
20:00 < G> mooo
20:00  * londo is here
20:00 -!- wolfy [n=lonewolf at fedora/wolfy] has left #fedora-meeting ["I can please only one person per day. Today is not your day."]
20:00 < themayor> im around
20:00  * ianweller starts screaming at his USB for not booting his eeepc
20:01 -!- cjb [n=cjb at] has joined #fedora-meeting
20:01 -!- m_stone [n=mstone at] has joined #fedora-meeting
20:01 < mmcgrath> themayor: gregdek: welcome, don't see you guys at the infra meetings that much ;-)
20:01 < mmcgrath> First up, oustanding tickets
20:01 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- tickets
20:01 < mmcgrath>
20:01  * ricky 
20:01 < zodbot> <> (at
20:01 < themayor> im just happy to finally have internet connection again, ill join anything at this point ;)
20:01  * kanarip !
20:02 < mmcgrath> Sorry, thats the wrong link...  Here's the right one
20:02  * dgilmore is here
20:02 < mmcgrath>
20:02 < zodbot> <> (at
20:02 < mmcgrath> First ticket
20:02 < mmcgrath> .ticket 732
20:02 < zodbot> mmcgrath: #732 (Investigate New System Monitoring Methods) - Fedora Infrastructure - Trac -
20:02 < mmcgrath> G: want to take this one?  I figure a summary of the last week would be good.  Its coming along much more quickly then even I had anticipated.
20:03  * dgilmore would like to say that when doing things like restarting servers and services.  please be kind and schedule downtime in nagios so we dont get alerts at 1am
20:03 < G> Yeah I'm actually on a slow-mo mode on this right now, got a few things I have to do, -noc'ers should be getting some e-mails soon though
20:03 < dgilmore> regardless of what the monitoring system is
20:04 < ricky> dgilmore: I think those were false ones last night :-(
20:04 < ricky> Or a VPN blip or something
20:04 < G> the base is up, and appears to relatively stable
20:04 < mmcgrath> ricky: I looked throught he logs and sar last night, I do believe it was a vpn blip.  Got into a funky mode and fixed itself.
20:04 < G> that was noc2 (tummy) all last night)
20:04 < ricky> Ah
20:04 < ricky> noc1 gave alerts for stuff on the VPN too :-(
20:04 < G> so yeah, it's progressing
20:05 < mmcgrath> G: good work, really.  I suspect we'll be fully converted soon.  I look forward to the -noc emails.
20:05 < mmcgrath> G: anything else?  If not we'll move on.
20:05 < G> oh for full conversion...
20:05 < G> I'd sooner hold out to September when 1.6.x is out if possible
20:06 < ricky> Hopefully, we can use the time in between to get a good baseline for the alert settings
20:06 < G> It provides several 'missing links'
20:06 < mmcgrath> G: any specific features we're blocking on?
20:06 < G> err hold on one sec, brain is a bit slow, and DNS just died
20:07  * mmcgrath notes we're talking about for those who aren't familiar with the actual site.
20:07 < G> okay...
20:08 -!- themayor [n=jack at] has quit Read error: 104 (Connection reset by peer)
20:08 < G> 1.6 offers us: better distributed monitoring, ability to proxy results, encryption on the agent, able to cache DB stuff
20:08 -!- themayor [n=jack at] has joined #fedora-meeting
20:08 < ricky> Encryption sounds like a cool one
20:09 < mmcgrath> k, well in the meantime that just gives us all the opportunity to make sure that its done right the first time.
20:09 < mmcgrath> G: again, good work.
20:09 < ricky> I'm also curious as to how well distributed monitoring will work.  Can we get things setup so that not every single host has to be on the VPN?
20:09 < mmcgrath> ricky: depends on how we want to monitor it.
20:09 < mmcgrath> Anywho, next ticket :)
20:09 < mmcgrath> .ticket 395
20:09 < zodbot> mmcgrath: #395 (Audio Streaming of Fedora Board Conference Calls) - Fedora Infrastructure - Trac -
20:09 < mmcgrath> jcollie: any progress here?
20:09 < jcollie> nope
20:09 < mmcgrath> k, next ticket.
20:10 < mmcgrath> .ticket 446
20:10 < zodbot> mmcgrath: #446 (Possibility to add external links on spins page) - Fedora Infrastructure - Trac -
20:10 < mmcgrath> dgilmore: ^^^
20:10 < mmcgrath> is that done?  do we even need that link anymore?
20:10  * mmcgrath thinks dgilmore is pretty busy today with other meetings.
20:10 < mmcgrath> dgilmore: we'll go back to that
20:10 < dgilmore> mmcgrath: we have a wiki page that needs some legal love.
20:11 < dgilmore> but i dont know if its needed any longer
20:11 < mmcgrath> dgilmore: ah, k.  has that request been forwarded on to legal?
20:11 < mmcgrath> yeah, the whole spins thing has kind of been re-designed since that request was created.
20:11 < mmcgrath> dgilmore: anywho, we'll move on.
20:11 < dgilmore> mmcgrath: no i need to do that.  I need a sentance to say that fedora doesnt provide or endorse the referenced spins
20:11 < dgilmore> yeah
20:11 < mmcgrath> .ticket 576
20:11 < zodbot> mmcgrath: #576 (Infrastructure Contact Information) - Fedora Infrastructure - Trac -
20:11 < mmcgrath> dgilmore: ahh, k.
20:12 < mmcgrath> So this one's blocking on me.
20:12 < mmcgrath> I need to get up, put the thing together and hand out directions on how it works.
20:12 < mmcgrath> .ticket 740
20:12 < zodbot> mmcgrath: #740 (Loaning out system time to OLPC participants) - Fedora Infrastructure - Trac -
20:12 < mmcgrath> this one's new to me.
20:13 < mmcgrath> ahh,
20:13 < mmcgrath> gregdek: this one's yours.
20:13 < mmcgrath> gregdek: so the specific use case is giving people the ability to build for OLPC machines using our infrastructure somehow?
20:13 < gregdek> That's the idea.
20:13 < mmcgrath> So there's a couple of ways we could do this.
20:13 < dgilmore> gregdek: build what?
20:14 -!- hhardy_ [n=hhardy at] has joined #fedora-meeting
20:14 < mmcgrath> 1) is to just use koji.  My concern there is that I'm not aware of anyone using non-RH/Fedora OS's to build against koji.
20:14 < gregdek> Updated packages, as I understand it, when they don't have their own Fedora boxes.
20:14 -!- epithumia [n=tibbs at fedora/tibbs] has joined #fedora-meeting
20:14 < mmcgrath> and 2) is to use a 3rd party who's wiling to donate their machine but allow FAS to build on it.
20:14 < ricky> What buildsystem does olpc currently use?
20:14 < mmcgrath> gregdek: once the package is built, they'd distribute it somehow or just download it themselves?
20:15 < mmcgrath> ricky: I bet dgilmore knows ;-)
20:15 < gregdek> These are questions I do not readily have the answers to.
20:15 < G> mmcgrath: what about re-rack a couple of old machines, install fedora on them, lock them down etc
20:15 < dgilmore> gregdek: so we are best off getting koji packaged for debian/ubuntu/SuSE/mandriva etc
20:15 < mmcgrath> gregdek: sure thing.
20:15 < dgilmore> ricky: fedora's koji
20:15 < themayor> yeah
20:15 < ricky> Oh
20:15 < mmcgrath> G: we could do that, but we're still tight on space in PHX.  it is an option though.
20:15 < gregdek> It's my understanding that we've tried getting koji into other distros with limited success...?
20:15 < G> mmcgrath: ahhh rack space
20:15 < mmcgrath> dgilmore: do you happen to know if its possible to build OLPC packages on SuSE's Open Build System?
20:16 < dgilmore> mmcgrath: i have no idea.  i looked at the crack it is and cried
20:16 < mmcgrath> heh
20:16 < mmcgrath> gregdek: not sure, there's two parts to koji (obviously) the client and the server.
20:16 < mmcgrath> I'd think the client could be put in other distros fairly easily.
20:16 < mmcgrath> the server part though, not sure.
20:16 -!- tibbs|h [n=tibbs at fedora/tibbs] has quit Nick collision from services.
20:16 -!- epithumia is now known as tibbs|h
20:16 < dgilmore> im guessing whats really wanted here is somewhere that people could log in, run mock and do development builds
20:16 < mmcgrath> dgilmore: whats your opinion on that?
20:17 < ricky> So they're currently building OLPC packages on Fedora, and they need to build it on their distro instead?
20:17 < mmcgrath> gregdek: did you see them as having signed the CLA and considered contributors?
20:17 < dgilmore> mmcgrath: client should be no problems i would think
20:17 < gregdek> mmcgrath: I think that's reasonable.
20:17 < gregdek> Given the easy CLA process.
20:17 < dgilmore> ricky: there is olpc specific tags that they can scratch build in
20:18 < dgilmore> and mock is in debian,  which worked last i tested
20:18 < gregdek> People are currently using mock to build in Debian, yes.
20:18 < gregdek> AIUI.
20:18 < ricky> Wow, cool
20:18 < mmcgrath> gregdek: dgilmore: lets get some questions together and put in that ticket and make sure they get answered then meet up again next week.
20:18 < mmcgrath> how's that sound?
20:18 < gregdek> Brilliant.
20:18 < dgilmore> mmcgrath: sounds fine.
20:18 < mmcgrath> schweet.
20:19 < mmcgrath> Ok. So thats the end of the tickets, I've got a couple of items to discuss
20:19 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- PPC builders
20:19 < mmcgrath> I've added a few ppc builders, ppc6 is on the way hopefully.
20:19 < mmcgrath> dgilmore: how'd they look?
20:19 < mmcgrath> .builders
20:19 < zodbot> mmcgrath: Enabled: 12 Ready: 12 Disabled: 11
20:19 < mmcgrath> .building ppc4
20:19 < zodbot> mmcgrath: ppc4 - tasks/765415/eclipse-3.4.0-18.fc10.src.rpm:ppc64
20:19 < mmcgrath> pretty quiet right now
20:19 < dgilmore> mmcgrath: poking at it now.  it seems ok other than disk
20:20 < mmcgrath> yeah
20:20 < mmcgrath> Just so everyone knows, our bladecenter builders came with lots of power and memory, not much disk.  Its our biggest limiter especially on the PPC hosts.  I'm hoping some firmware upgrades will give us access to the tools we need to disable the hardware raid and enable software raid.
20:20 < mmcgrath> that'd give us the ability to do raid0 and thus, get more disk space (and faster disk) out of those boxes.
20:21 < mmcgrath> I'm just happy they've booted and have an OS on them.
20:21 < mmcgrath> So thats really all there.
20:21 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- CVS boxes
20:21 < dgilmore> .building ppc5
20:21 < zodbot> dgilmore: ppc5 - Not doing anything
20:21 < mmcgrath> As some of you have seen I've built cvs1 and cvs2.  I'm going to be doing some proof of concepts on them but soon the current will get replaced with something more... modern.
20:21 < ricky> :-D
20:21 < mmcgrath> And far less scary.
20:22 < abadger1999> yay!
20:22 < dgilmore> mmcgrath: its not that scary
20:22 < ricky> chroots.
20:22 < mmcgrath> So expect that soon.  I'll coordinate with releng.  I suspect this will be ready long before F10 ships, but probably won't go live until after that just to make sure we don't break F10 needlessly.
20:22 < mmcgrath> Next topic
20:22 -!- lxo [n=aoliva at] has joined #fedora-meeting
20:22 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- haproxy
20:23 < mmcgrath> I've been evaluating a few proxy / HA balancers and haproxy seems to fit best with our use case.  Its got a decent base and all my tests have... well just worked.  no fuss.
20:23 < mmcgrath> We're looking to use this to replace mod_proxy_balancer which just isn't cutting it with us anymore.  Its not as feature rich nor as mature as what our environment now needs.
20:23 -!- lx0 [n=aoliva at] has quit Read error: 110 (Connection timed out)
20:23 < mmcgrath> it will also play well when we start to abstract our caching servers and add geoDNS.
20:23 < dgilmore> mmcgrath: what else was considered?
20:24 < ricky> What will we put in front of it for SSL?
20:24 < G> mmcgrath: will this change the basic workflow on how webapps operate?
20:24 < mmcgrath> If the plan goes right, it will mean faster load times for our users as content will be served geographically close to them.
20:24 < mmcgrath> ricky: nope, behind SSL.
20:24 < mmcgrath> G: for the first rollout no
20:24 < ricky> mmcgrath: But what will we use for the SSL part?
20:25 < mmcgrath> dgilmore: I've looked at mod_proxy_balancer (what we currently use), piranha, and Pound
20:25 < mmcgrath> ricky: apache
20:25 < mmcgrath> ricky: this will literally replace the balancer:// bits in our configs.  Other then that the proxy servers, for now, remain the same.
20:26 < G> mmcgrath: balancer:// becomes...?
20:26 < ricky> Oh, so it'll be web app -> haproxy -> apache?
20:26 < mmcgrath> balancer will likely become http://localhost:haproxyPort/
20:26 < ricky> Or maybe I'm misinterpreting this page.
20:26 < G> mmcgrath: gotcha
20:26 < mmcgrath> well it'll be browser -> apache -> haproxy -> app  whereas right now its browser -> apache -> mod_proxy_balancer(apache) -> app
20:27 < ricky> Ahh
20:27 < mmcgrath> haproxy has far better metrics and a great stats page to see wtf is going on when things start to bork.
20:27  * mmcgrath gets the screenshot
20:27 < dgilmore> mmcgrath: :)
20:28 < mmcgrath> <-- Older version'
20:28 < ricky> Very descriptive website :-)
20:28 < mmcgrath> but you get the idea.
20:29 < mmcgrath> The stats include things like when the worker just came back, and when the last time the actual farm was down.
20:29 < mmcgrath> All very useful and just not things we're getting out of mod_proxy_balancer, even with the mod_proxy_balancer stats.
20:29 < ricky> Nice
20:29 < mmcgrath> Anyone have any questions on that?  If not I'll open the floor.
20:29 < ricky> So what do you see the "final" setup as being?
20:30 < ricky> Will we still use apache for caching, or will we look at squid or something at some point?
20:30 < mmcgrath> ricky: not sure yet, it depends on what we decide our caching strategy is and what we use to cache.
20:30 < ricky> All right
20:30 < mmcgrath> well, for now yes.  But ultimately we'll likely be implementing squid or maybe even invest more time in memcached.
20:31 < mmcgrath> lots more research is needed and we'll have to re-examine how our applications work, where the expensive network calls are (IE: from the app to DB, or from the app to the user) that type of thing
20:31 < mdomsch> mmcgrath, wildcard certs yet?
20:31 < G> ohh yeah...
20:31 < mmcgrath> mdomsch: yes, actually we do have a * cert as of last week.  I sent an email to I think sysadmin-web-members.
20:31 < G> thats the only blocker on the new wikis basically
20:31  * mmcgrath goes ahead and opens the floor
20:31 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Open Floor
20:31 < G> mmcgrath: I don't recall getting one
20:31 < lmacken> mmcgrath: How is the appFc/appRhel merge going ?
20:32 < mmcgrath> G: boo, well I have it and its ready to be deployed if you or ricky or someone else wants to do it.  If not I'll probably hit it up next week.
20:32 < G> I had two things for Open Floor, but I've forgotten one
20:32 < mmcgrath> lmacken: its going ok.  I hit some snags but I expect to be done with it early this week late last week.
20:32 < mdomsch> at some point in the next month or so we'll want to enable https for mirrors.fp.o
20:32 < lmacken> mmcgrath: cool
20:32 < mmcgrath> mdomsch: sure thing, that should be a low-cost operation now :)
20:32 < G> mmcgrath: yeah, we can't do it for fp.o until we are using *.fp.o
20:32 < mdomsch> but right now yum doesn't do anything useful with https (except encrypt, no validation), so it's of limited value
20:32 < G> (well not really)
20:33 < ricky> mdomsch: By the way, could use some love, when you have some time
20:33 < zodbot> <> (at
20:33 < mdomsch> ok
20:33 < G> Anyone, I did have the whole l10n wiki stuff :)
20:33 < mdomsch> next, serving yum metadata from Fedora-owned servers...
20:33 < mmcgrath> <nod>  well if anyone wants to get that setup let me know.  I have a few kind of strict requrements for its use (IE, don't let it get on a test server)
20:33 < mmcgrath> <nod>  well if anyone wants to get that setup let me know.  I have a few kind of strict requrements for its use (IE, don't let it get on a test se
20:34 < mmcgrath> oof
20:34 < G> As the wildcart certs a ready, it just needs some TLC from our translators (I'll poke couf today)
20:34 < ricky> What will be the translator workflow for the l10ned wiki?
20:34 < mmcgrath> ricky: I'm a little curious about that myself
20:34 < mmcgrath> G: what two things did you have to discuss?
20:35 < G> mmcgrath: wiki, and as I said, I forgot the other
20:35 < mmcgrath> ah, k.
20:35 < ricky> It's an interesting tradeoff between ease-of-implementation (for me) and letting translators keep their processes (for fedora-web, at least)
20:35 < mmcgrath> ricky: yeah.  I'll be curious to see how it actually goes when the time comes.
20:35 < G> really the workflow wouldn't be much different to how it was with the old wiki
20:36 < mmcgrath> Ok, well kind of a short meeting this time around, still have plenty of time left.  Anyone have anything else they'd like to discuss?
20:36 < mmcgrath> or re-discuss even?
20:36 < G> Also we have Mediawiki talking to FAS via json :)
20:36 < abadger1999>  rh bz3
20:36 < mmcgrath> G: we'll be gitting rid of the http auth plugin completely after that gets deployed, right?
20:37 < G> if someone (anyone) wants to take a look at bug 458220 it'd be much appreciated
20:37 < buggbot> Bug medium, medium, ---, nobody at, NEW, Review Request: php-pecl-json - PECL library to implement JSON in PHP
20:37 < G> mmcgrath: correct!
20:37 < mmcgrath> solid
20:37 < mdomsch> how bad would it suck if we started pushing the yum sqlite databases from mirrors.fp.o instead of from the mirrors themselves?
20:37 < mmcgrath> mdomsch: the sqlite db's as they currently are?
20:37 < mmcgrath> or will the format / size change?
20:37 < mdomsch> yeah; all 30MB of each of them
20:38 < mmcgrath> mdomsch: not sure, go ahead and submit a ticket and we'll get an estimate together.
20:38 < abadger1999> mdomsch: Why not just repomd.xml?
20:38 < ivazquez> Wouldn't that cause sync issues galore?
20:38 < mdomsch> (opensuse pushes all repository metadata from their owned servers; only packages are downloaded via redirects to the mirrors)
20:38 < mmcgrath> mdomsch: and you're sure from mirrors.fp.o and not ?
20:38 < mdomsch> i'm looking
20:38 < mdomsch> yeah, maybe d.f.r.c
20:38 < mdomsch> not sure yet
20:38 < mmcgrath> mdomsch: if we can do it from d.f.rh.c or even alias some secure.mirrors.fp.o or something I think that'd be good.
20:39 < mmcgrath> AFAIK those mirros aren't getting that much traffic now that they're not in the public view.
20:39 < G> ivazquez: yum would fall onto a new server if the content wasn't yet on one
20:39 -!- vwbusguy [n=scott at] has quit "Leaving"
20:39 < abadger1999> G: But right after a push, everyone will be out of date.
20:39 < mmcgrath> G: it is a good point though, d.f.r.c or mirrors.fp.o would have the content first.
20:39 < G> abadger1999: yeah
20:39 < mmcgrath> could be a couple of hours where people would think all mirrors are bad.
20:39 < G> what about...
20:39 < mmcgrath> mdomsch: how do we protect against that?
20:39 < mdomsch> yeah, d.f.r.c may wind up being the last mirror listed
20:40 < abadger1999> mdomsch: Not going to try to do versioned repodata?
20:40 < mdomsch> so it gets used only if none of the other mirrors are current
20:40 < G> how about just distribute hashes of all the sqlite dbs like the mirrorlists are
20:40 < mdomsch> abadger1999, yeah, probably will do versioned repodata, at least timestamped
20:40 < mmcgrath> mdomsch: might be time again to look into pushed updates debian style
20:40 < G> i.e.
20:40 < ivazquez> It would be nice if there was some sort of prioqueue that mirrors could get dumped on, then read the mirrors down that way.
20:40 < G> Current: <hash>
20:40 < mdomsch> mmcgrath, we _absolutely_ need push mirroring
20:40 < G> Old: <hash>
20:40 -!- fchiulli [i=824c4012 at gateway/web/ajax/] has joined #fedora-meeting
20:40 < mdomsch> I haven't had time to do anything about push mirroring though
20:40 < mdomsch> volunteers? :-)
20:41 < G> that way yum can use either or, but it can warn you if it's an old hash
20:41 < mdomsch> g: yes exactly
20:41 < G> (i.e. "Heck, the mirror doesn't have the latest, you might want to check back in a few hours"
20:41 < mdomsch> I've started working on this a couple weeks ago
20:42 < mmcgrath> mdomsch: how serious of a risk do you estimate we're actually in.
20:42 < mmcgrath> both in reality, and compared to the other major distros.
20:42 < mdomsch> adding in metalink support, which gets us a common format for <file><checksum type=sha1/>...
20:42 < mdomsch> the only real problem is maliciously stale mirrors targeting specific netblocks
20:42 -!- cassmodiah [n=cass at] has quit " | auf freenode"
20:43 < G> mdomsch: yeah might pay to offer two or three checksums for each file though
20:43 < mdomsch> and by malicious, I mean "one that lies to MM and says it's up-to-date" when it's not
20:43 < mmcgrath> <nod>
20:43 < ivazquez> Prompt for some checksum.
20:43 < mdomsch> g: yeah; that's an extension to metalinks I need to add
20:44 < mdomsch> I've started that thread with the metalink spec authors
20:44 < G> mdomsch: ahh great
20:44 < mdomsch> but, when we have it, we get metalinks for ISOs etc for free
20:44 < kanarip> it could be as simple as letting yum always grab the headers from at least two different mirrors, and if they differ, take a third and drop the non-matching
20:44 < mmcgrath> mdomsch: well it sounds like we have options there to examine.  I don't see any blockers though really.
20:44 < mdomsch> I've wanted to avoid serving the metadata itself (sqlite dbs & large XML files) from our own servers
20:45 < mdomsch> but if push comes to shove, good to have that as an option
20:45 < mdomsch> that's all for now
20:45 < mmcgrath> mdomsch: if we didn't do that, are we talking about bad as in someone submits a bug abouta  bad mirror, or we get torn a new one on lwn?
20:45 < mdomsch> worst case, the latter
20:46 < mmcgrath> k
20:46 < abadger1999> not necessarily justified though....
20:46 < mmcgrath> mdomsch: yeah, go ahead and create a ticket and we'll get it figured out.
20:46 < mmcgrath> abadger1999: same old song and dance then ;-)
20:46 < abadger1999> Yep :-/
20:46 < abadger1999> One FYI: red hat bugzilla was upgraded last weekend.  So far it looks like they did a really good job on compat... only fas's bugzilla sync script appears to be broken.
20:47 < G> mmcgrath: is our Wildcard cert chained?
20:47 < mmcgrath> G: "chained" ?
20:47 < mmcgrath> abadger1999: is it still borked?
20:47 < G> mmcgrath: yeah, one issuers chain the certs together so there is basically 3 certs
20:47 < abadger1999> Until dkl implements an equivalent to updatePerms() in the xmlrpc API, I have to sync fas => bugzilla manually.  If someone complains, ping me to update the tables.
20:48 < mmcgrath> G: oh, not sure.  I haven't even opened the attachment yet to look at it.
20:48 < abadger1999> (OTherwise I'm doing it once a day)
20:48 < mmcgrath> abadger1999: will do, thanks for the heads up.
20:48 < mmcgrath> G: ping me after the meeting, I'll take a look.
20:48 < mmcgrath> Anyone have anything else they'd like to discuss?  If not we'll close the meeting in 30 seconds?
20:48 < G> the site cert, the issuing cert (chained to the site cert) and the issuing cert of the issurer thats in the browsers etc
20:48 < ricky> I think you just need to look at who signed it
20:48 < mmcgrath> 15
20:49 < mmcgrath> 5
20:49 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Meeting end
20:49 < mmcgrath> Thanks for coming everyone!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <>

More information about the Fedora-infrastructure-list mailing list