[Ovirt-devel] [PATCH] Add support for vendor hooks during ovirt-early start()

[Ryan Harper]

* Perry Myers <pmyers at redhat.com> [2009-11-16 17:11]:
> On 11/16/2009 05:14 PM, Ryan Harper wrote:
> > * Perry Myers <pmyers at redhat.com> [2009-11-16 15:48]:
> >> On 11/16/2009 04:40 PM, Ryan Harper wrote:
> >>> * Perry Myers <pmyers at redhat.com> [2009-11-16 14:56]:
> >>>> On 11/16/2009 02:21 PM, Ryan Harper wrote:
> >>>>> Add a kernel parameter, vendor= which takes a path to a script
> >>>>> embedded in the image.  If this script is executable, we will then
> >>>>> source it during ovirt-early start() after command line processing
> >>>>> and before mounting of /config.  We also call a post hook at the end of
> >>>>> ovirt-early start().
> >>>>>
> >>>>> Also include a sample vendor script.
> >>>>
> >>>> If you're using edit-livecd to embed a vendor specific script in the oVirt
> >>>> ISO image, why wouldn't you just change the init process to call that
> >>>> script instead of using a kernel cmdline parameter like this?
> >>>>
> >>>> i.e. edit-livecd to add a new init script called vendor-foo to
> >>>> /etc/init.d, then chkconfig that script on.  Now during startup the script
> >>>> will automatically execute.
> >>>>
> >>>> I suppose maybe some information about the use case that is driving this
> >>>> request might help to explain.
> >>>
> >>> Sure, one of the use cases we need is using an iscsi target as "local"
> >>> disk.  This requires bringing up networking and iscsi prior to
> >>> ovirt-early trying to mount the /config partition.  Previously, had just
> >>> injected code around the area where the pre hook is now.  Recently moved
> >>> that to it's own function so we could conditionally call it.. then
> >>> looking at how to merge any of that upstream and it made more sense to
> >>> keep the very vendory specific changes out of ovirt-early altogether.  I
> >>> hadn't thought to do a completely separate file since some of the logic
> >>> triggers on other boot parameters that ovirt-early processes (like
> >>> firstboot, and storage_init).
> >>
> >> jboggs was working on providing iscsi as local disk and the plan was to
> >> integrate that as fully supported functionality for the base oVirt ISO
> >> image.  So I'd prefer in this particular case that we work to get it
> >> integrated into the base oVirt ISO image rather than put it in as a vendor
> >> specific script.
> > 
> > I can share what we've had to do to get it working so far; not the
> > prettiest bit of code.  
> > 
> > Networking must be brough up prior to iscsi connection and one can't
> > access /config.  Ideally, we could pass a simple CONFIG_IP_PNP parameter
> > to the kernel to get networking autoconfigured but neither RHEL nor
> > Fedora enable IP_PNP.  Passing the config via command is doable, but
> > makes having a common boot image/config troublesome since each server
> > has to have a unique config file.  Additionally, we wanted to configure
> > a bonded bridge prior to establishing the iscsi connection since once
> > you mount /config over iscsi one cannot restart networking without
> > dropping your filesystem into RO mode.  The per-server config issue
> > ended up being resolved by embedding additional config (iscsi server,
> > initiatorname, and nfs server path) via DHCP vendor options.  
> > Here is what we're doing to bring up networking and connect to an iscsi 
> > target prior to mounting /config; this code runs where vendor_pre_hook()
> > would.
> 
> The ovirt-early script uses the standard ip= and ipv6= kernel command line
> options to bring up autoconfigured network interfaces.  The interface to
> bring up can be specified using the standard BOOTIF= kernel cmdline arg.
> So what you're talking about should work fine, as both ip= and ipv6= both
> support dhcp as an argument.

Sure, except AFAICT, this won't bring up networking until after we've
attempted to mount /config ... which is of course too late if we're
using an iscsi "local" disk.

> 
> But yeah, doing automatic bonding is a different story.  There's no way to
> do that automagically right now.  You could add some additional kernel
> cmdline options similar to BOOTIF...  i.e.
> STORAGE_IFS=MAC:MAC:MAC
> STORAGE_BONDING_TYPE=
> 
> Gets a little complicated though so perhaps this is not the best way to do
> it...  BOOTIF is nice since it is set to the MAC addr of the iface used to
> pxe boot the server.  But there's no simple way to reliably define which
> ifaces you'd want to bond other than by MACaddr.  Or...  maybe you could
> do the following...
> 
> rely in BOOTIF to tell you which device is the primary connection.  Then
> bring up each other iface on the host in sequence and test to see what
> network they're on.  The ones that are on the same network as the BOOTIF
> network iface would automatically all be bonded together.  That would be
> pretty slick...  Then all you need to do is pass as a kernel cmdline arg
> what bonding style you want to use.

Oddly enough, we pxe from our 1g nics, but want to bond the 10g nics...
I'd prefer specifying input for creating a bond, or bonds...

BONDIF=bond0,bond1
bond0_MACS=$primary_mac,$secondary_mac
bond0_OPTIONS="mode=6 miimon=100 use_carrier=1"
bond0_PROTO="dhcp"
bond0_ONBOOT="yes"
bond1_MACS=$primary_mac,$secondary_mac
bond1_OPTIONS="mode=6 miimon=100 use_carrier=1"

We'd also need bridging parameters so we can configure a bridge with the
bond port...
BRIDGEIF=brbond0

and ... this gets ugly rather quickly; which is why we ended up with a
vendor script instead... How far do you want to go with these kernel
parameters for custom networking config?


> 
> > 
> > # don't let networking autostart later
> > chkconfig --del network
> > 
> > # generate custom ifcfg network files that
> > #   - build a bonded interface over 10g nics
> > #   - create a bridge with the bond0 as port
> > configure_network_files
> > 
> > # disable any current networking
> > service network restart
> > 
> > # look for custom DHCP paramters if they exist
> > # and extract iscsi server IP and initator name
> > echo "InitiatorName=${iscsi_initiator}" > /etc/iscsi/initiatorname.iscsi
> > 
> > # disable iscsi autostart
> > chkconfig --del iscsi
> > chkconfig --del iscsd
> > 
> > # bring up iscsi
> > service iscsi start
> > 
> > # discover iscsi targets on iscsi server
> > TARGET=$(iscsiadm -m discovery -t sendtargets -p ${iscsi_server})
> > 
> > # restart iscsi server
> > server iscsi restart
> > 
> > # wait for devices to populate /dev/disk/by-id
> > l=0
> > count=`find /dev/disk/by-id -mindeth 1 2>/dev/null | wc -l`
> > while [ $l -lt 5 ]; do
> >     curr=`find /dev/disk/by-id -mindeth 1 2>/dev/null | wc -l`
> >     log "by-id now has $curr entries"
> >     [ $curr -gt $count ] && break
> >     l=$(($l + 1))
> >     sleep 1;
> > done
> > 
> > # if not firstboot, expect LVs 
> > if [ "$firstboot" != "1" ]; then
> >     # it's not the firstboot
> >     log "Scanning for lvm config"
> >     l=0
> >     while [ $l -lt 3 ]; do
> >         echo "lvm scan count=$l" >> /tmp/lvm.log
> >         pvscan >> /tmp/lvm.log
> >         sleep 1
> >         vgscan >> /tmp/lvm.log
> >         sleep 1
> >         lvscan >> /tmp/lvm.log
> >         sleep 1
> >         vgchange -ay >> /tmp/lvm.log
> >         [ -e /dev/HostVG/Config ] && break;
> >         l=$(($l + 1))
> >     done
> >     if [ ! -e /dev/HostVG/Config ] ; then
> >         log "Rescue mode requested, starting emergency shell"
> >         stop_log
> >         bash < /dev/console
> >         start_log
> >     fi
> > fi
> > 
> > At this point, we've made an iscsi connection and repopulated LVM and we can mount /config.
> 
> Seems reasonable.  I'm sure with a little tweaking this could be included
> somehow, but I'll defer to Alan and Joey on the specifics :)
> 
> > 
> >>
> >> apevec, since jboggs is out of the office for a bit can you follow up here?
> >>
> >>> I'll have to look at seeing if we could move that into it's own init
> >>> script; though there might be some duplication of code (command parsing,
> >>> and such).  With the vendor approach, it executes in the same context as
> >>> ovirt-early so one can potentially override function implementations and
> >>> exert control over the flow in the remainder of ovirt-early; that
> >>> wouldn't be doable in a separate init script.  
> >>>
> >>> One example that we rely upon that can't be done from separate script is
> >>> appending boot parameters to the params list to be saved.
> >>
> >> ... But regardless, it is probably worth creating an easy to use interface
> >> for vendors to add their own init scripts in without needing to duplicate
> >> code, and trampling on ovirt-early.  Perhaps an
> >> /etc/insert-your-vendor-script-here.d directory that ovirt-early and
> >> ovirt-post reads and executes functions from?  That would provide an easy
> >> way to drop in custom functionality and would also not require a new
> >> kernel cmdline param.  Thoughts?
> > 
> > That does seem cleaner; though I do still like having a way to choose
> > which set of vendor scripts to run via kernel parameter.  We'd like to
> > have a single iso image that we could support different types of
> > startups.
> 
> Yeah, I have no opposition to the kernel cmdline parameter used for that.

OK.

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
ryanh at us.ibm.com