Meeting Log - 2008-05-01

[Ricky Zhou]

16:00 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Meeting, who's here
16:01 -!- maploin [n=oin at 86.123.200.16] has joined #fedora-meeting
16:01  * jima , barely
16:01  * skvidal is
16:01 < mmcgrath> abadger1999: dgilmore f13 ianweller ivazquez lmacken marek mdomsch paulobanon skvidal: ping
16:02 < ivazquez> Pong.
16:02 < mmcgrath> ricky: ?
16:02  * abadger1999 here
16:02  * lmacken is around
16:03 < mmcgrath> aaalrighty.
16:03 < mmcgrath> lets get started.
16:03 < mmcgrath> this should be _real_ quick since we're in a change freeze.
16:03 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- F9 Release.
16:03 < mdomsch> yep
16:04 < ianweller> hi
16:04 < mmcgrath> .tiny https://fedorahosted.org/fedora-infrastructure/query?status=new&status=assigned&status=reopened&milestone=Fedora+9
16:04 < zodbot> mmcgrath: http://tinyurl.com/25vzyu
16:04 < mmcgrath> So thats what we've got left to do for the F9 launch
16:04 < marek> hey all
16:04 < mmcgrath> skvidal: is it your opinion we should wait until after F9 to switch to torrent.fedoraproject.org?
16:04  * ianweller just got home  :D
16:04 < mmcgrath> marek: howdy
16:05 < skvidal> mmcgrath: at this point, yes
16:05  * mmcgrath agrees.
16:05 < skvidal> mmcgrath: even if we get the new ips - we've got no way of knowing if that actually solves it until more tests :-/
16:05 < mmcgrath> it might make it now, but I have no reason to believe that the current torrent box will flake out in the next two weeks :)
16:05 < mmcgrath> .ticket 231
16:05 < zodbot> mmcgrath: #231 (post master directory space usage for mirrors) - Fedora Infrastructure - Trac - https://fedorahosted.org/projects/fedora-infrastructure/ticket/231
16:05 < mmcgrath> mdomsch: were you happy with the outcome on that enough that I can close the ticket?
16:06 < mdomsch> mmcgrath, yes
16:06 < mdomsch> mmcgrath, does that get run occasionally?
16:06 < mdomsch> e.g. cronjob?
16:06 < mmcgrath> mdomsch: yeah its a cron job.
16:06  * mmcgrath verifies its actually running
16:06 < mdomsch> perfect
16:07 < mmcgrath> mdomsch: it is - 30 19 * * *
16:07 < mdomsch> great
16:07 < mmcgrath> .any yingbull
16:07 < zodbot> mmcgrath: yingbull was last seen in #fedora-meeting 4 hours, 11 minutes, and 21 seconds ago: *** yingbull has quit IRC ("ChatZilla 0.9.78.1 [Firefox 2.0.0.13/2008031114]")
16:07 < mmcgrath> f13: you around?
16:08 < mmcgrath> we'll go back to that.
16:08 < mmcgrath> .ticket 333
16:08 < zodbot> mmcgrath: #333 (Add spam headers to bastion (smtp)) - Fedora Infrastructure - Trac - https://fedorahosted.org/projects/fedora-infrastructure/ticket/333
16:08 < mmcgrath> .ticket 54
16:08 < zodbot> mmcgrath: #54 (Postfix Server) - Fedora Infrastructure - Trac - https://fedorahosted.org/projects/fedora-infrastructure/ticket/54
16:08 -!- bzbot is now known as buggbot
16:08 < mmcgrath> those configs are ready to be deployed but there were concerns on the list about deploying this close to the release.
16:08 -!- hughsie [n=hughsie at host86-158-145-201.range86-158.btcentralplus.com] has left #fedora-meeting ["Ex-Chat"]
16:08 < mmcgrath> So shortly after the release we'll deploy.
16:08 < mmcgrath> .272 can be closed
16:09 < mmcgrath> .any susmit
16:09 < zodbot> mmcgrath: susmit was last seen in #fedora-meeting 1 week, 5 days, 7 hours, 41 minutes, and 11 seconds ago: *** susmit has parted #fedora-meeting ()
16:09 -!- smooge [n=smooge at canopus.unm.edu] has quit "-ENOCAFFEINE"
16:09 < mmcgrath> I'm not sure whats going on with
16:09 < mmcgrath> .389
16:09 < mmcgrath> we haven't had an issue in a while but it'd be nice to have that shortly after F9
16:09 < mmcgrath> .ticket 389
16:09 < zodbot> mmcgrath: #389 (Monitor primary mirror) - Fedora Infrastructure - Trac - https://fedorahosted.org/projects/fedora-infrastructure/ticket/389
16:09 -!- petreu [n=peter at fedora/Standby] has quit "( www.nnscript.de :: NoNameScript 4.02 :: www.XLhost.de )"
16:10 < mmcgrath> So really there's nothing left to discuss on the F9 release.
16:10 < mmcgrath> One thing I'm looking at doing is finding the cheapest operation for our common pages via load tests here in my lab at home.
16:10 < mmcgrath> it turns out there's some fun stuff that goes on with mod_cache and mod_negotiation (which we use for translated websites)
16:10 < mmcgrath> there's a meeting at the end of the week to determine exactly what links we'll be giving out.
16:11 < mmcgrath> I had planned on setting the proxy servers to auto-cache just those pages in memory.
16:11 < mmcgrath> and their requisites (images, css, etc)
16:11 -!- sharkcz [n=dan at plz1-v-4-17.static.adsl.vol.cz] has joined #fedora-meeting
16:11 < mmcgrath> Anyone have any questions or concerns about the F9 release?
16:12 < mmcgrath> I'll take that as a no.
16:12 < mdomsch> mmcgrath, keeping the wiki online?
16:12 < ianweller> heh
16:12 < mdomsch> we won't have the new mediawiki up yet
16:13 < mmcgrath> mdomsch: yeah I'll be getting that list of pages during the meeting this week and creating static pages for the links we give out.
16:13 < mdomsch> even if it's r/o for release day
16:13 < mdomsch> ok
16:13 < mmcgrath> also, in theory, for this release we can add more readers.  But I think creating static pages of our official links is probably the best strategy.
16:13 < ianweller> +1
16:13 < mdomsch> agreed
16:14 < mdomsch> does it make sense to add readers, and stop all writers for the day?
16:14 < mdomsch> or is that too hard?
16:14 < marek> I think we should leave few core writers to be able to edit the wiki, just in case of wrong info or typos
16:14 < mmcgrath> shouldn't be too hard.
16:15 < mmcgrath> marek: I'm actually more inclined to make them come to us first.  In the past when the wiki has gotten "busy", writing makes it infinitly worse for some reason.
16:15 < mmcgrath> al though, in theory, we have the wiki setup in a way we never have before (and didn't have last time)
16:15 < mmcgrath> so ew can dedicate a writer and create a few readers.
16:15 -!- GeroldKa [n=GeroldKa at fedora/geroldka] has joined #fedora-meeting
16:16 < mmcgrath> I'll run a couple of tests and send a note to the list.
16:16 < mmcgrath> Ok, anyone else have any questions?
16:16 < mdomsch> mmcgrath, you mean we rewrite the edit URLs and point them at the same box?
16:16 < mdomsch> i'm good
16:16 < mmcgrath> mdomsch: well, right now all reads are going to app1, all writes are going to app2.  We can make all reads go to app1,3,4 and leave all reads alone.
16:17 < mmcgrath> which isn't bad because app2 is a pretty busy box compared to the rest.
16:17 < mdomsch> nice
16:17 < mmcgrath> err "and leave all writes alone"
16:17 < mmcgrath> ok, we'll scoot along for now.
16:17 < ianweller> so we have separate read and write servers for moin?
16:17 < mmcgrath> ianweller: correct.
16:17 < ianweller> hmm.
16:17 < ianweller> ok!
16:17 < mmcgrath> ianweller: which is the most stable we've ever had it :)
16:17 < ianweller> tee hee
16:18 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Outage last week.
16:18 < mmcgrath> So we had two interesting outages this last week.
16:18 < mmcgrath> one of them, came and went on its own.  DB2's load went way up, then came back down.
16:18 < mmcgrath> but the one before that, on a Friday or Sat. evening was very strange.
16:18 < mmcgrath> basically xen6 lost access to its storage subsystem.
16:18 < mmcgrath> I haven't had time to go through the logs yet but its not something we've seen before but is something we need to keep a closer eye on.
16:19 < mmcgrath> the good news is that because of the work susmit and I have been up to, our nagios false positives are way down.
16:19 -!- couf [n=bart at fedora/couf] has quit "leaving"
16:19 < mmcgrath> And the last thing on our agenda this week...
16:19 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Outage this Sat.
16:19 < mmcgrath> So PHX is going to be offline.
16:20 < skvidal> I'm sure nothing will break
16:20 < mmcgrath> They said they'll keep the outage to a minimum, but still, unlike nfs, we can't have our iscsi shares going up and down for 5 hours.
16:20 < mmcgrath> so I think we should just shut them all down.
16:20 < mmcgrath> So the basic plan is this.
16:20 < marek> I should be back at home and online to report the progress from network team
16:21 < mmcgrath> marek: thanks, that will be most helpful.
16:21 < mmcgrath> marek: do you know if they'll be switching out our switch?
16:21 < marek> I'm in Paris now and I hope I'll get back home till Friday
16:21 < mmcgrath> its EOL AFAIK.
16:21 < marek> mmcgrath: don't know, good question
16:21 < marek> mmcgrath: let me ask quick
16:22 < mmcgrath> marek: thanks.
16:22 < mmcgrath> So we'll be running all traffic through tummy.com for that outage.
16:22 < mmcgrath> we'll setup proxy3 to balance mirrormanager between app5 and app6.
16:23 < mmcgrath> since app2 is on local storage, it'll be going up and down as the network goes up and down, with no real harm.
16:23 < mmcgrath> so, with the exception of db2 (and therefore auth) the wiki at least and smolt will stay up when the network is available for them to do s.
16:23 < mmcgrath> unfortunately everything else will just be down.
16:24 < mdomsch> can tummy handle that?
16:24 < mdomsch> nirik was concerned that the secondary content rsync was eating their bandwidth
16:24 < mmcgrath> mdomsch: yeah, tummy handled all of our traffic during the F8 launch for a few hours.
16:24 -!- SmootherFrOgZ [n=Smoother at ASte-Genev-Bois-152-1-83-163.w86-218.abo.wanadoo.fr] has quit Remote closed the connection
16:24 < mdomsch> ok, as long as they're ok with it
16:25 < nirik> yeah, that transfer was coming in on the much slower of our links for some reason...
16:25 < nirik> and it was for a long time. ;)
16:25 < mmcgrath> I don't think they'll even notice.  I'll try to get an actual estimate of bw and talk to them ahead of time and see if we need to throttle or something.
16:25 < mmcgrath> nirik: right now though, can you see how much traffic is going through proxy3?  It'll double.
16:25  * nirik goes to look
16:25 < mmcgrath> .dns fedoraproject.org
16:25 < zodbot> mmcgrath: 66.35.62.162
16:26 < nirik> Current Traffic:	117.3KiB/sec in/165.1KiB/sec out
16:26 < nirik> no biggie at all
16:26 < mmcgrath> so if we shutdown PHX and sent all traffic through you, that number would double.
16:26 < mmcgrath> yeah, thats what I thought.  Good to verify it though.
16:26 < mmcgrath> Ok, Also during that outage, I'm going to reboot xen1 and give it the new 5.2 kernel.
16:27 < mmcgrath> we never did that during the last outage because the branching took much longer then we thought.
16:27 < mmcgrath> Anyone else have any concerns for this weekend?
16:27 < nirik> I don't think that would be any problem...
16:27 < marek> mmcgrath: ah, mgalgoci is out for training, I'll try to find more details tomorrow and let you know
16:28 < mdomsch> marek, see if malgoci got his internet2 netblock for download1 pls
16:29 < mdomsch> that'd be nice to have, but not critical
16:29 < skvidal> hey
16:29 < marek> mdomsch: afaik there's download-i2.fedora.redhat.com for that no?
16:29 < skvidal> mmcgrath just dropped off the net
16:29 < skvidal> he called me
16:29 < mdomsch> marek, don't know
16:29 < skvidal> anything else on the meeting?
16:30 < marek> mdomsch: from what I remember from mails, download1.fedora -> download3.fedora so people can download normalny
16:30 < marek> and download-i2.fedora.redhat.com is the inet2
16:30 < marek> skvidal: not for me
16:30 < mdomsch> marek, you've been out of the loop for a bit...
16:30 < mmcgrath> oh!  I'm back.
16:30 < skvidal> marek: and ibiblio is getting things through th i2 connection
16:30 < skvidal> oh look
16:30 < skvidal> we have an mmcgrath again
16:30 < mmcgrath> marek: thanks.
16:30 < mdomsch> marek, I'll fill you in after the meeting
16:30 < marek> mdomsch: that's passible, passing my days in paris now :/
16:31 < marek> k
16:31 < mmcgrath> Lets open the floor and close the meeting :)
16:31 < mdomsch> mmcgrath, good to prove our strategy for losing a data center
16:31 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Open Floor
16:32 < mdomsch> just before a release
16:32 < mmcgrath> Does anyone have anything else to discuss?
16:32 < mmcgrath> mdomsch: no doubt :)  I hope for the next release we'll have the alternate vpn server up so the changes are automatic.
16:32 < abadger1999> skvidal: Are you holding off on the package-owners at fp.o aliases until after release?
16:32 < skvidal> abadger1999: yah - I'll tie it to postfix
16:33 < skvidal> abadger1999: b/c it'll be less effort (sorry, lazy)
16:34 < mmcgrath> skvidal: its also "the smart thing" to do :)
16:34 < skvidal> laziness frequently is
16:34 < skvidal> :)
16:34 < abadger1999> :-)
16:34 < mmcgrath> hah
16:34 < mmcgrath> Alrighty, anyone have anything else to discuss?
16:34 < mmcgrath> If not I'll close the meeting in 30
16:34 -!- rdieter is now known as rdieter_away
16:34 < mmcgrath> 10
16:35 < mmcgrath> 5
16:35 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Meeting end!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/fedora-infrastructure-list/attachments/20080501/895a68a0/attachment.sig>