<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns="http://www.w3.org/TR/REC-html40"> <head> <META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii"> <meta name=Generator content="Microsoft Word 11 (filtered medium)">  <title>Re: [Fedora-xen] Fedora Core 8 + Xenbr0 + network bridging?</title> <style>  </style> </head> <body lang=EN-US link=blue vlink=blue> <div class=Section1> Hi Mark:<o:p></o:p> <o:p> </o:p> Well, <o:p></o:p> <o:p> </o:p> I carefully went through your PDF document for setting up a bonded pair on RHEL 5.1 and at the end of the day this does not work for me. This is the procedure more or less I have tried:<o:p></o:p> <o:p> </o:p> <ul style='margin-top:0in' type=disc> <li class=MsoNormal style='color:navy;mso-list:l1 level1 lfo1'>Make sure eth0 and eth1 work as normal eth devices without LACP<o:p></o:p></li> <li class=MsoNormal style='color:navy;mso-list:l1 level1 lfo1'>Edit /etc/modprobe.conf to add the bonding kernel module<o:p></o:p></li> <li class=MsoNormal style='color:navy;mso-list:l1 level1 lfo1'>Edit /etc/sysconfig/network-scripts/ifcfg-eth0 and ifcfg-eth1 to be slave devices with the master as bond0<o:p></o:p></li> <li class=MsoNormal style='color:navy;mso-list:l1 level1 lfo1'>Create /etc/sysconfig/network-scripts/ifcfg-bond0 with static IP info<o:p></o:p></li> <li class=MsoNormal style='color:navy;mso-list:l1 level1 lfo1'>Plug into LACP enabled ports on switch and reboot the box<o:p></o:p></li> <li class=MsoNormal style='color:navy;mso-list:l1 level1 lfo1'>At this point bond0 works fine, as expected, we have been doing this all day long on non-xen kernels for years<o:p></o:p></li> <li class=MsoNormal style='color:navy;mso-list:l1 level1 lfo1'>NOW…following your document<o:p></o:p></li> <li class=MsoNormal style='color:navy;mso-list:l1 level1 lfo1'>Remove the /etc/libvirt/qemu/networks/autostart/* so that the pointless 192.168.122.x NAT network does not get created<o:p></o:p></li> <li class=MsoNormal style='color:navy;mso-list:l1 level1 lfo1'>Grab the network-scripts-bridge-bond-vlan script from the link in the message and put into /etc/xen/scripts<o:p></o:p></li> <li class=MsoNormal style='color:navy;mso-list:l1 level1 lfo1'>Edit /etc/xen/xend-config.sxp and replace (network-script network-bridge) with (network-script network-bridge-bond-vlan)<o:p></o:p></li> <li class=MsoNormal style='color:navy;mso-list:l1 level1 lfo1'>Restart networking<o:p></o:p></li> <li class=MsoNormal style='color:navy;mso-list:l1 level1 lfo1'>At this point networking for the physical host in invariably completely hosed, and there is no xen networking at all<o:p></o:p></li> <li class=MsoNormal style='color:navy;mso-list:l1 level1 lfo1'>It is noteworthy to add that I did not make any modifications to the network-scripts-bridge-bond-vlan script as you suggest since I am on a single vlan (so far) and the default setup in the script seems sane for a single NON-VLAN bond0 configuration<o:p></o:p></li> </ul> <o:p> </o:p> So in a nutshell the it seems that libvirt interface just does not seem to use the xen scripts…rather it’s own (undocumented?) crap. After a week of head-banging on this we really have no choice but to abandon Redhat in favor of Suse 10.3, which incidentally works right out of the box with bridge, NAT, bond, you name it. I suggest RH staff install a copy of this to see how they did it, you don’t even have to jump through hoops to configure a bond from the command line, you can do this with their stock GUI tools if you so choose. I am not knocking RedHat, but we need something that “just works” and so far for xen + bond + RH has been a non-starter for us after much effort. By the way, to get networking un-foobar’ed on the physical machine I have to run system-config-network and remove all Ethernet devices, then reboot, then redefine eth0 and eth1 as as normal DHCP devices. I can then go re-setup the bond0 device from the command line.<o:p></o:p> <o:p> </o:p> If anyone has step-by-step directions on how to do this with RHEL 5.1 I am happy to try again at some future point. The goals are simple:<o:p></o:p> <o:p> </o:p> <ul style='margin-top:0in' type=disc> <li class=MsoNormal style='color:navy;mso-list:l0 level1 lfo2'>Step one: create a bond0 from eth0 and eth1 that can be used as a bridged device by xen on a single VLAN<o:p></o:p></li> <li class=MsoNormal style='color:navy;mso-list:l0 level1 lfo2'>Step two: create a bond0 (aliases?) that can be assigned multiple VLANs, AND have these VLAN devices available to be used by xen<o:p></o:p></li> </ul> <o:p> </o:p> I would settle for item #1 really.<o:p></o:p> <o:p> </o:p> /Christian<o:p></o:p> <o:p> </o:p> <o:p> </o:p> <o:p> </o:p> <o:p> </o:p> <o:p> </o:p> <div> <div class=MsoNormal align=center style='text-align:center'> <hr size=2 width="100%" align=center tabindex=-1> </div> From: Christian Lahti Sent: Saturday, December 01, 2007 3:56 PM To: mnielsen@redhat.com; fedora-xen@redhat.com Subject: RE: [Fedora-xen] Fedora Core 8 + Xenbr0 + network bridging?<o:p></o:p> </div> <o:p> </o:p> <div id=idOWAReplyText29821> <div> Hi Mark:<o:p></o:p> </div> <div> <o:p></o:p> </div> <div> Thank you very much for your response, I did indeed read the original poster as Dale by mistake :) So what you are saying makes perfect sense to me and sounds like exactly what we are after, I will have 3 vlans to bridge myself ultimately. My next question is the relative merits of RHEL5.1 as compared to Fedora 8. Obviously I would prefer the stable enterprise release rather than bleeding edge Fedora, but has fully virtualized windows performance been fixed in this release? At any rate I am looking forward to getting this up and running tomorrow!<o:p></o:p> </div> <div> <o:p></o:p> </div> <div> /Christian<o:p></o:p> </div> <div> <o:p></o:p> </div> </div> <div> <o:p> </o:p> <div class=MsoNormal align=center style='text-align:center'> <hr size=2 width="100%" align=center tabIndex=-1> </div> From: Mark Nielsen [mailto:mnielsen@redhat.com] Sent: Sat 12/1/2007 3:19 PM To: Christian Lahti Subject: Re: [Fedora-xen] Fedora Core 8 + Xenbr0 + network bridging?<o:p></o:p> </div> <div> hmm, did you mean "Hi Mark" ?? I have 8 Dell 2950s running RHEL 5.1 (new libvirt with that funky NAT they added). I have 4 NICs in each; 2 copper, 2 fiber. I bond the 2 copper (eth0 and eth1) and call it bond0. bond0 carries my "private" IP for cluster suite communications on the dom0 (physical) cluster. Then I bond eth2 and eth3 (fiber) in to bond1. I lay down the public network for the dom0 cluster on bond1.100 (for example, that would be VLAN 100). I also add many (up to 10 or so now) VLANs on bond1 (bond1.20, bond1.21, bond1.22, etc). Then I create xen bridges to each of these bond/VLAN devices. This allows me to put any particular VM on any particular (or combination up to 3) of these xen bridged bonded VLAN device. My document explains, in detail, how to do all of this :) The only added step is that I have to "undefine" (virsh net-undefine default) the default network that the new libvirt creates (virbr0). Even with this new NAT thing they added, I've been told (by our devs) that the preferred way to do static network configurations is with the method I lay out. NAT is more for dynamic networks (cable modems, dial-up, wifi, etc). I'm pretty sure there weren't any significant changes in Fedora 8 (we've dropped the word "core" now, btw) that don't exist in RHEL 5.1 with respects to the network. 5.0 -> 5.1 is when that NAT change came down the pipe. Mark p.s. I'm happy to answer any other questions you may have about my document. I'm quite certain that, if you follow it, you'll have what you're looking for. Christian Lahti wrote: > Hi Dale: > > I work with David who posted the original question to the mailing > list. I think we need to give a bit more background info on what we > are trying to do. We are running a mixed environment of mostly CentOS > 3, 4and 5, we do have a few windows servers and XP systems as well. > We are looking to virtualize all these platforms. Normally we have a > bonded pair of NICs for the physical hosts, we were able to get this > running using CentOS 5 x86_64 with no problems, the guest machines use > the bonded pair in bridged mode as expected after a bit of tweaking. > The biggest issue we found with EL5 is that windows guest performace > is dismal at best, hence our decision to have a look at Fedora Core 8 > x86_64. I am happy to report that performance for all of our guest > platforms is *very* good with FC8, but it seems that libvirt changed > the way networking is setup for Xen. The default NAT configuration is > pretty useless for production server environment. Thanks to the > mailing list we are now able to bridge a single NIC on FC8 (like eth0 > for example), but we cannot figure out how to get a bridge for bond0 > (comprised of eth0 and eth1) defined and available to Xen. All the > tweaks that worked find on EL5 have not worked so far on FC8. I am > going to review your document tomorrow and give it a try, but any idea > on whether your methodology will work on FC8 and libvirt? I am > willing to blow a Sunday to get this worked out once and for all :) > > Basically we are after good performance on both para and fully > virtualized guests using a bonded pair of GB NICs for speed and > redundancy. If this can be achieved with enterprise linux then that > would be preferable, but we will go FC8 if the bonding thing can be > sorted out. By the way Xensource 4.x looks to be a respin of RHEL5 > and has pretty good performance but their free version is limited to > 32bit (and hence 4GB ram). Adding the clustering failover is the next > step of course :) > > Thanks again for the help so far. > > /Christian > > > > >>>>>>>>>>> > just FYI for the list, I have a how-to for a bonded and VLAN tagged network. > > <a href="http://www.certifried.com">http://www.certifried.com</a> > > ODT and PDF formats available. > > > It might not be the best way, but I've sent it out to my colleagues > several times and have never received any negative feedback. > Mark > > > > Dale Bewley wrote: > > > I haven't done bonding, but you should be able to bond them and then compose a bridge on top of this bonded device I would think. > > -- > Dale Bewley - Unix Administrator - Shields Library - UC Davis > GPG: 0xB098A0F3 0D5A 9AEB 43F4 F84C 7EFD 1753 064D 2583 B098 A0F3 > > -- > Fedora-xen mailing list > Fedora-xen redhat com > <a href="https://www.redhat.com/mailman/listinfo/fedora-xen">https://www.redhat.com/mailman/listinfo/fedora-xen</a> > > > **************************************************************************** > Checked by MailWasher server (www.Firetrust.com) > WARNING. No FirstAlert account found. > To reduce spam further activate FirstAlert. > This message can be removed by purchasing a FirstAlert Account. > **************************************************************************** > > ------------------------------------------------------------------------ > > -- > Fedora-xen mailing list > Fedora-xen@redhat.com > <a href="https://www.redhat.com/mailman/listinfo/fedora-xen">https://www.redhat.com/mailman/listinfo/fedora-xen</a> > <o:p></o:p> </div> </div> </body> </html>