IRC Log for Fedora Infrastructure Meeting (2007-06-14)

[Jeffrey C. Ollie]

[15:06] mmcgrath has set the subject to fedora-infrastructure meeting --  Who's here?
[15:06] xDamox: Me
[15:06] mmcgrath: PING ALL: who's here?
[15:06] G: I'm here
[15:06] mbonnet: yo
[15:06] cemc has joined the group chat (n=gimre at voy.narancs.net)
[15:06] G: I'm gonna have to disappear in 15-20
[15:06] jcollie: howdy!
[15:06] f13: I'm here.
[15:06] mmcgrath: abadger1999: you around?
[15:06] abadger1999: Yep.
[15:06] mmcgrath: allrighty then
[15:07] * lmacken !
[15:07] * wolfy grabs a viewing seat
[15:07] mmcgrath has set the subject to abadger1999 - Package Database -------------
[15:07] mmcgrath: abadger1999: whats the word?
[15:07] abadger1999: We have progress 
[15:07] abadger1999: Script to import pkgs is written and I'm finding all sorts of problem with owners.list.
[15:08] mmcgrath: thats good at least 
[15:08] mmcgrath: how bad is it?
[15:08] abadger1999: Mostly lack of EPEL owner information for EPEL branches
[15:08] lmacken: abadger1999: is there an owners API yet?  If not, want some help?
[15:08] abadger1999: lmacken: I'm using nottings owners.py.
[15:09] lmacken: abadger1999: ah, where does that code live ?
[15:09] lmacken: maybe i can stuff that into bodhi in the mean time
[15:09] abadger1999: Since the pkgdb will get rid of owners.list I'm not too worried about an API.
[15:09] G: abadger1999: in that case dgilmore would be the person to poke I think
[15:09] glezos has joined the group chat (n=glezos at fedora/glezos)
[15:09] lmacken: well, bodhi needs to know who owns what
[15:09] abadger1999: cvs-int:/cvs/extras/CVSROOT/admin/owners.py
[15:09] lmacken: thanks
[15:09] mmcgrath: abadger1999: if you do get together a comprehensive list of stuff missing from EPEL send it my way.
[15:09] abadger1999: I have a few changes to what's checked in that I'll have to add later.
[15:10] abadger1999: Will do.
[15:10] abadger1999: warren is going to work on koji syncing.
[15:10] tibbs has left ("Konversation terminated!" (n=tibbs at fedora/tibbs))
[15:10] abadger1999: I'm going to do bugzilla sync and cvs acls sync this week.
[15:10] abadger1999: G (Nigel Jones) has started looking at the code and made some changes to the way it looks.
[15:11] mmcgrath: cool
[15:11] G: (Minor to start with, just getting to grips with Turbogears and how it works)
[15:11] abadger1999: So we're on track for next week or the week after.
[15:11] mmcgrath: abadger1999: anything else?
[15:11] abadger1999: That's about it.
[15:11] mmcgrath: abadger1999: the package db only contacts the database, correct?  Does it need any write access to a file system like koji or cvs?
[15:12] MrBawb has joined the group chat (i=abob at guppy.drown.org)
[15:12] abadger1999: i think cvs acls can pull from the packagedb for now instead of pkgdb pushing to cvs-int.
[15:12] abadger1999: koji I'm not sure about.  warren's looking into it.
[15:13] mmcgrath: cool.
[15:13] mmcgrath: pull is better than push for security reasons.
[15:13] mmcgrath: same reason bodhi is deployed on app5 instead of in our cluster.
[15:13] warren: you rather koji pull from packagedb?
[15:13] mbonnet: that's not really possible
[15:13] mbonnet: unless we have a cronjob that does the sync periodically
[15:13] mmcgrath: warren: if its filesystem stuff, yes.  if its db stuff then no.
[15:14] warren: abadger1999, it is all db right?
[15:14] abadger1999: mbonnet: How do you currently sync owners.list?
[15:14] warren: abadger1999, see /cvs/pkgs/CVSROOT/admin/owners-sync.py
[15:14] mbonnet: I believe the script is run by hand right now
[15:14] f13: yes, by hand
[15:15] * dgilmore is here
[15:15] f13: when we make changes to owners.list we run the sync script
[15:15] f13: less than ideal, but functional
[15:15] mmcgrath: <nod>
[15:15] abadger1999: Hmm...  Is it just pushing into koji's db?
[15:15] dgilmore: abadger1999: EPEL owner information is in owners.epel.list
[15:16] abadger1999: if so it can be a cronjob or we can write a callback that pushes the information from the package db to koji when it's updated.
[15:16] rdieter has left (Remote closed the connection (n=rdieter at sting.unl.edu))
[15:17] mmcgrath: abadger1999: we can figure it out.  We should move on for the meeting though.
[15:17] f13: abadger1999: it uses the koji API to do ownership adds/changes.
[15:17] f13: doesn't talk to the db directly
[15:17] abadger1999: f13: Sounds like it won't be a problem then.
[15:17] abadger1999: mmcgrath: And won't need to write to any filesystems.
[15:17] mmcgrath: abadger1999: excellent
[15:17] mmcgrath: k, moving on
[15:18] mmcgrath has set the subject to Config Management - mmcgrath
[15:18] mmcgrath: nothing majorly new here.  I'm going through and making sure our xen dom0's are setup properly.
[15:18] mmcgrath: I've also started forcing some packages to uninstall and some services not to start (cups, gpm, etc)
[15:18] mmcgrath has set the subject to VCS - jcollie
[15:18] mmcgrath: jcollie: ping?
[15:18] jcollie: yo
[15:19] mmcgrath: jcollie: Are you still playing with VCS solutions?
[15:19] jcollie: i think that the discussion last week on -devel and -infra was good
[15:19] f13: abadger1999: it needs to be made smarter, like knowing about different owners for different tags and all that, but that's just details (:
[15:19] jcollie: i just need to sit down and write up a more concrete proposal
[15:19] mmcgrath: jcollie: me too, its gotten more interest this time around then 6 mo. ago or so
[15:20] mmcgrath: jcollie: solid, make sure to get some good input from the jeremy's and jesse's in the world 
[15:20] mmcgrath: k, moving next
[15:20] jcollie: i think it'll be a mix... there'll be a repository that looks a lot like we have now, but with some meta-language or -tags to pull patches out of a "exploded tree" repo
[15:20] * mdomsch joins belatedly
[15:20] mmcgrath has set the subject to Firewall System Rewrite - lmacken skvidal
[15:20] mmcgrath: mdomsch: yo
[15:21] mmcgrath: jcollie: excellent, thanks for getting that stuff together.
[15:21] mmcgrath: lmacken: ping
[15:21] mmcgrath: skvidal: ping?
[15:21] lmacken: no updates on this from my end.. have we decided to abandon pyroman ?
[15:21] mmcgrath: lmacken: Not sure, I know xDamox has some opinions on it.
[15:21] mmcgrath: xDamox: ping?
[15:21] xDamox: yo
[15:21] mmcgrath: you had some items to discuss regarding the Firewall System?
[15:21] skvidal: mmcgrath: I think we should just go with simple iptables files in /etc/sysconfig
[15:21] xDamox: Yea, I updated the template we were using and neaten it up a little
[15:21] mbonnet: question: does our firewall system have some kind of NAT/conntrack limit?
[15:22] mmcgrath: skvidal: I agree, what about boxes that have different firewall needs though?
[15:22] xDamox: I can help 100% with the iptables writing.
[15:22] skvidal: mmcgrath: that's what puppet is for
[15:22] lmacken: taking the strict & simple rule approach sounds good to me
[15:22] mbonnet: I'm wondering exceeding that limit might be the cause of the intermittent connection drops people see connecting to koji.fp.o
[15:22] skvidal: mmcgrath: distribute files out based on host
[15:22] fchiulli has joined the group chat (i=824c4010 at gateway/web/cgi-irc/ircatwork.com/x-04ce31ce5397d4ea)
[15:22] mmcgrath: mbonnet: Both the host based and hardware firewalls can do it but only the proxy servers actually do do it now.
[15:22] mmcgrath: skvidal: ahh, yes.  A puppet template would work well for that I think.
[15:23] mmcgrath: mbonnet: I'll verify that we're not rate limiting in any way on the hardware firewall.
[15:23] xDamox: mmcgrath, do we have an up to date list of services running on each box
[15:23] xDamox: and their ports?
[15:23] mmcgrath: xDamox: we're pretty close.
[15:23] mmcgrath: skvidal: do you have a link to the iptables rules you'd suggested on the list?
[15:24] skvidal: yes
[15:24] skvidal: uno momento
[15:24] skvidal: http://linux.duke.edu/~skvidal/misc/iptables-template
[15:24] xDamox: Ok. If you could give me a copy, I could do a sample firewall for some boxes maybe and have skvidal and lmacken check it over?
[15:24] lmacken: xDamox: sounds good to me
[15:24] xDamox: that good with you too skvidal ?
[15:24] skvidal: xDamox: fine - I already have those on a couple of the boxes due to the release
[15:25] skvidal: iirc they're on proxy1 and 2
[15:25] mmcgrath: xDamox: remember KISS 
[15:25] xDamox: Ok, yep 
[15:25] xDamox: Ill make it a simple as possible
[15:25] G: Have fun with the rest of meeting, I'm out
[15:25] mmcgrath: G: later, thanks for coming
[15:26] xDamox: also I am sure skvidal and lmacken will be able to simplify it more 
[15:26] mmcgrath: xDamox: cool, take what skvidal has at http://linux.duke.edu/~skvidal/misc/iptables-template and give it a good lookover.
[15:26] mmcgrath: I'll create an erb (puppet template) out of it and see how it goes.
[15:26] xDamox: yep  will do
[15:27] dgilmore: mbonnet: i dont know what on our firewalls as far as that goes
[15:27] dgilmore: mbonnet: we dont control the nat part of it
[15:28] mmcgrath: k, xDamox when you're done send'er to the list and we can get this all underway.
[15:28] mmcgrath has set the subject to Server Upgrades - mmcgrath
[15:28] xDamox: OK mmcgrath,
[15:28] mmcgrath: So I'm trying to get some additional RAM in some of our servers.
[15:28] mmcgrath: but we have more pressing issues.. .namely a lot of our newer boxes don't have warrantys.
[15:29] mmcgrath: so I'm trying to figure out where money should come from to pay for that.
[15:29] mmcgrath: additionally we have a lot of boxes that are reaching the end of their natural life and should be replaced.
[15:29] mmcgrath: Fortunately if we stick with high capacity devices, this will allow us to use our rack more efficiently.
[15:29] mmcgrath: The major limiting factor being cost, heat and power.
[15:29] mmcgrath: just letting everyone know whats going on there.
[15:30] mmcgrath has set the subject to Xen Conversion - mmcgrath
[15:30] mmcgrath: So I've started doing some work with iscsi
[15:30] dgilmore: 
[15:30] mmcgrath: It's actually going quite well.
[15:30] * mmcgrath digs up a bonnie run
[15:30] warren: what will serve iscsi?
[15:30] mmcgrath: warren: the netapp already is.
[15:31] mmcgrath: grr pastebin
[15:31] dgilmore: mbonnet: how much storage do we have?  how much did you use for iscsi?
[15:31] dgilmore: mmcgrath: http://paste.ausil.us
[15:32] dgilmore: mmcgrath: ^^^^^^^^^^^^^^  meant you not mbonnet
[15:32] mmcgrath: dgilmore: already on it 
[15:32] mmcgrath: ok, here's an iscsi run on publictest9
[15:32] mmcgrath: http://paste.ausil.us/161
[15:32] mmcgrath: dgilmore: right now 500G
[15:32] mmcgrath: all in all I've been quite pleased with it.
[15:33] mmcgrath: I've kickstarted a few boxes with iscsi, the package install portion (about 400 packages) takes about 2 minutes.
[15:33] londo: mmcgrath: random access seems slow to me
[15:33] dgilmore: mmcgrath: live migration is easy to do?
[15:33] mmcgrath: dgilmore: yep, so far its just worked for me.  There's a brief network blip I need to work on.  The box itself doesn't experience it that bad but I think there's some arp issues.
[15:34] Karl_le_Rouge has joined the group chat (n=RedKarl at ALyon-257-1-149-122.w81-251.abo.wanadoo.fr)
[15:35] dgilmore: awesome
[15:35] mmcgrath: londo: I've seen random seek as high as 705.2.
[15:35] mmcgrath: The larger the test was on iscsi the slower that got though, always a good excuse to tweak and test though 
[15:36] mmcgrath: All in all I think iscsi will work very well for us.  We just need to watch carefully network utilization and overall netapp utilization.
[15:36] londo: mmcgrath: numbers from tiobench will be nice if you can get them
[15:36] mmcgrath: londo: is it in extras?
[15:36] londo: mmcgrath: yeap
[15:36] mmcgrath: londo: cool, I'll run it then.
[15:36] mmcgrath: k, moving on
[15:37] mmcgrath has set the subject to Bacula
[15:37] mmcgrath: So I've been testing out bacula on xen6 and publictest[3-4]
[15:37] mmcgrath: everything's been working great.
[15:37] f13: hurray!
[15:37] dgilmore: mmcgrath: how much total disk do we need to backup?
[15:37] mmcgrath: We're just blocking on https://bugzilla.redhat.com/230344
[15:37] f13: a scary amount
[15:37] f13: (if you count /mnt/koji)
[15:38] mmcgrath: dgilmore: wellllll, depends, do you con't /mnt/koji or not?
[15:38] dgilmore: welll  we really should backup /mnt/koji
[15:38] f13: mmcgrath: btw, did the new disk shelf show up in phx?
[15:38] mmcgrath: dgilmore: the plan right now is to do a backup of everything on xen6's local storage which is 378G.  I'm working on getting a tape backup for everything though (including koji)
[15:38] * dgilmore needs a cloning machine 
[15:39] mmcgrath: f13: I've not heard one way or the other but I was under the impression that it should be there by now.  I'll send an emil.
[15:39] dgilmore: mmcgrath:   ok
[15:39] mmcgrath: dgilmore: I've got the tape drive as a priority2 thing after our warranty issue with the soc.
[15:39] mmcgrath: all in all though, ixs says he'll have more time in the comming days for us to do a formal review.
[15:40] dgilmore: mmcgrath:   yeah  we would probably want LT)2 or 3  with at least 10 slots
[15:40] mmcgrath: For those that haven't used it Bacula is really slick.
[15:40] skvidal: is it wicked slick?
[15:40] londo: if you are going to move things on netapp/iscsi is it possible to do the backup there (if a tape drive is available)
[15:40] mmcgrath: super wicked slick.
[15:40] abadger1999: skvidal: wykd
[15:40] dgilmore: i need to find time to get it reviewed
[15:40] mmcgrath: londo: thats the problem, we had 3 netapps to deal with now we have 1 super netapp and I'm not comfortable with backing up to itself.
[15:41] mmcgrath: londo: sorry, I missed your (if tape drive) comment.
[15:41] dgilmore: mmcgrath: i agree
[15:41] f13: I loved bacula when I was using it.
[15:41] mmcgrath: k, moving on
[15:41] mmcgrath has set the subject to Translators stuff -
[15:42] f13: seriously hot stuff
[15:42] mmcgrath: glezos: has been working on this.  Its now at http://publictest4.fedora.redhat.com/
[15:42] mmcgrath: this will be a very big deal when we start moving stuff to it.
[15:42] RedKarl has left (Connection timed out (n=RedKarl at ALyon-257-1-39-129.w90-14.abo.wanadoo.fr))
[15:42] JSchmitt has left ("Konversation terminated!" (n=s4504kr at fedora/JSchmitt))
[15:42] mmcgrath: so all keep your eyes out for it and help out because all parties involved can use it.
[15:42] mmcgrath has set the subject to account system -
[15:42] mmcgrath: Nothing new here.  If anyone is interested in working on it with me that would be good.
[15:43] mmcgrath has set the subject to Project Hosted - f13
[15:43] mmcgrath: f13: ?
[15:43] f13: nothing new.  Trac git plugin sucks.
[15:44] mmcgrath: <nod>
[15:44] f13: Oh, I created a script to create trac projects, but havne't put it in scm any where or documented it
[15:44] abadger1999: f13: Could you give me access to the hosted box?
[15:44] f13: sure.
[15:44] abadger1999: Thanks.
[15:44] f13: at some point it should be FAS'd but...
[15:44] mmcgrath: <nod>
[15:45] mmcgrath: next
[15:45] mmcgrath has set the subject to FedoraPeople.org - skvidal
[15:45] mmcgrath: skvidal: anything new?
[15:45] skvidal: nothing
[15:45] warren: Is that planned for shell and web?
[15:45] skvidal: yes
[15:45] dgilmore: mmcgrath: just thought of something ill switch off plague on June 29  for FC-5
[15:46] dgilmore: skvidal: anyidea when you will get to rebuild the box?
[15:46] skvidal: dgilmore: not this week and probably not beginning of next since I'll be in orientation, etc
[15:46] mmcgrath: dgilmore: <nod>
[15:47] skvidal: but I'll be working again come next week
[15:47] skvidal: so it's a start
[15:47] skvidal: and I should be able to spend the time
[15:47] mmcgrath: cool
[15:48] mmcgrath has set the subject to Ibiblio Mirror - On hold
[15:48] mmcgrath: The ibiblio mirror is on hold for probably about a week while we hook don up with direct I2 access to our mirror in RDU.
[15:48] mdomsch: mmcgrath, pick set up the static route already
[15:48] mmcgrath: mdomsch: hmm, I'll have to check with don, he was under the impression he needed to wait a bit.
[15:49] mmcgrath: Ok, thats all I've got.
[15:49] mmcgrath has set the subject to Open Floor ----------------
[15:49] lmacken: word
[15:49] lmacken: I was wondering what you guys thought about having some sort of development environment for our webapps.
[15:49] lmacken: So, there are a handfull of people that are interested in hacking on bodhi, but due it's dependencies on koji and mash, it's extremely difficult to develop it anywhere other than PHX.  I've currently been doing all of my development on publictest2, which has been working out great.
[15:49] lmacken: So a possibility for this is to have some Xen guest with a read-only mount of /mnt/koji and blocked out from the rest of PHX.
[15:49] mdomsch: lmacken, +1; /me misses publictest7
[15:50] lmacken: yeah, and honestly.. i have no idea how to start hacking on mirrormanager, smolt, etc
[15:50] lmacken: i think if we opened the doors a bit, our infrastructure could improve vastly
[15:50] lmacken: mdomsch: feel free to hack on publictest2 for now 
[15:50] mmcgrath: lmacken: the main limiting facter on that is RAM, but we can set something up.
[15:51] lmacken: mmcgrath: cool
[15:51] mmcgrath: lmacken: we should probably setup more shared xen instances.
[15:51] abadger1999: lmacken: +1
[15:51] dgilmore: im going to start work on enabling secondary archs if anyone wants to help feel fee to talk to me
[15:52] dgilmore: mmcgrath: can we possibly get another vlan?
[15:52] jcollie: mmcgrath, could i get a xen guest for testing the git/vcs stuff?
[15:52] lmacken: mmcgrath: cool.. so what is the next action to getting this ready?  creating a group for infrahackers and granting access on a restricted guest ?
[15:52] dgilmore: mmcgrath: so we can seperate the some guestd  for this kind of thing
[15:52] mmcgrath: lmacken: well I'll need to find where we have RAM avaiable for the instances.  Its item "Server Upgrades" on the wiki.
[15:52] mmcgrath: dgilmore: we should.
[15:53] wolfy has left ("When you are down and out something always turns up-and it is usually the noses of your friends." (n=lonewolf at fedora/wolfy))
[15:53] lmacken: mmcgrath: ok.. well, publictest2 has been my playground for the past few months.. any reason not to just start using that ?
[15:53] mmcgrath: jcollie: I think we can setup something, it'll be a bit
[15:53] mmcgrath: lmacken: not sure, I think it only has 512M ram right?
[15:53] lmacken: mmcgrath: i'm not sure
[15:54] dgilmore: mmcgrath: im pretty sure thats all it ahs
[15:54] dgilmore: has
[15:54] lmacken: mmcgrath: also, i noticed that you setup the security guest.. does bressers know about it yet ?
[15:54] mmcgrath: k, I'll try to find ways to consolidate some of our lesser machines into a bigger, sort of super machine.
[15:54] lmacken: mmcgrath: cool
[15:54] mmcgrath: lmacken: I think dgilmore did that
[15:55] lmacken: ah
[15:55] lmacken: dgilmore: is the security guest ready to go ?
[15:56] dgilmore: lmacken: not yet
[15:56] lmacken: dgilmore: k, just checking 
[15:56] dgilmore: i need to add the security group to get shell access
[15:58] mmcgrath: solid
[15:58] mmcgrath: so anyone have anything else?  If not I'll close the meeting in 30 seconds?
[15:58] mmcgrath: 10
[15:59] mmcgrath has set the subject to Meeting End -----------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://listman.redhat.com/archives/fedora-infrastructure-list/attachments/20070614/8f98223a/attachment.sig>