[Ovirt-devel] [PATCH] Add support for vendor hooks during ovirt-early start()

[Perry Myers]

On 11/16/2009 05:14 PM, Ryan Harper wrote:
> * Perry Myers <pmyers at redhat.com> [2009-11-16 15:48]:
>> On 11/16/2009 04:40 PM, Ryan Harper wrote:
>>> * Perry Myers <pmyers at redhat.com> [2009-11-16 14:56]:
>>>> On 11/16/2009 02:21 PM, Ryan Harper wrote:
>>>>> Add a kernel parameter, vendor= which takes a path to a script
>>>>> embedded in the image.  If this script is executable, we will then
>>>>> source it during ovirt-early start() after command line processing
>>>>> and before mounting of /config.  We also call a post hook at the end of
>>>>> ovirt-early start().
>>>>>
>>>>> Also include a sample vendor script.
>>>>
>>>> If you're using edit-livecd to embed a vendor specific script in the oVirt
>>>> ISO image, why wouldn't you just change the init process to call that
>>>> script instead of using a kernel cmdline parameter like this?
>>>>
>>>> i.e. edit-livecd to add a new init script called vendor-foo to
>>>> /etc/init.d, then chkconfig that script on.  Now during startup the script
>>>> will automatically execute.
>>>>
>>>> I suppose maybe some information about the use case that is driving this
>>>> request might help to explain.
>>>
>>> Sure, one of the use cases we need is using an iscsi target as "local"
>>> disk.  This requires bringing up networking and iscsi prior to
>>> ovirt-early trying to mount the /config partition.  Previously, had just
>>> injected code around the area where the pre hook is now.  Recently moved
>>> that to it's own function so we could conditionally call it.. then
>>> looking at how to merge any of that upstream and it made more sense to
>>> keep the very vendory specific changes out of ovirt-early altogether.  I
>>> hadn't thought to do a completely separate file since some of the logic
>>> triggers on other boot parameters that ovirt-early processes (like
>>> firstboot, and storage_init).
>>
>> jboggs was working on providing iscsi as local disk and the plan was to
>> integrate that as fully supported functionality for the base oVirt ISO
>> image.  So I'd prefer in this particular case that we work to get it
>> integrated into the base oVirt ISO image rather than put it in as a vendor
>> specific script.
> 
> I can share what we've had to do to get it working so far; not the
> prettiest bit of code.  
> 
> Networking must be brough up prior to iscsi connection and one can't
> access /config.  Ideally, we could pass a simple CONFIG_IP_PNP parameter
> to the kernel to get networking autoconfigured but neither RHEL nor
> Fedora enable IP_PNP.  Passing the config via command is doable, but
> makes having a common boot image/config troublesome since each server
> has to have a unique config file.  Additionally, we wanted to configure
> a bonded bridge prior to establishing the iscsi connection since once
> you mount /config over iscsi one cannot restart networking without
> dropping your filesystem into RO mode.  The per-server config issue
> ended up being resolved by embedding additional config (iscsi server,
> initiatorname, and nfs server path) via DHCP vendor options.  
> Here is what we're doing to bring up networking and connect to an iscsi 
> target prior to mounting /config; this code runs where vendor_pre_hook()
> would.

The ovirt-early script uses the standard ip= and ipv6= kernel command line
options to bring up autoconfigured network interfaces.  The interface to
bring up can be specified using the standard BOOTIF= kernel cmdline arg.
So what you're talking about should work fine, as both ip= and ipv6= both
support dhcp as an argument.

But yeah, doing automatic bonding is a different story.  There's no way to
do that automagically right now.  You could add some additional kernel
cmdline options similar to BOOTIF...  i.e.
STORAGE_IFS=MAC:MAC:MAC
STORAGE_BONDING_TYPE=

Gets a little complicated though so perhaps this is not the best way to do
it...  BOOTIF is nice since it is set to the MAC addr of the iface used to
pxe boot the server.  But there's no simple way to reliably define which
ifaces you'd want to bond other than by MACaddr.  Or...  maybe you could
do the following...

rely in BOOTIF to tell you which device is the primary connection.  Then
bring up each other iface on the host in sequence and test to see what
network they're on.  The ones that are on the same network as the BOOTIF
network iface would automatically all be bonded together.  That would be
pretty slick...  Then all you need to do is pass as a kernel cmdline arg
what bonding style you want to use.

> 
> # don't let networking autostart later
> chkconfig --del network
> 
> # generate custom ifcfg network files that
> #   - build a bonded interface over 10g nics
> #   - create a bridge with the bond0 as port
> configure_network_files
> 
> # disable any current networking
> service network restart
> 
> # look for custom DHCP paramters if they exist
> # and extract iscsi server IP and initator name
> echo "InitiatorName=${iscsi_initiator}" > /etc/iscsi/initiatorname.iscsi
> 
> # disable iscsi autostart
> chkconfig --del iscsi
> chkconfig --del iscsd
> 
> # bring up iscsi
> service iscsi start
> 
> # discover iscsi targets on iscsi server
> TARGET=$(iscsiadm -m discovery -t sendtargets -p ${iscsi_server})
> 
> # restart iscsi server
> server iscsi restart
> 
> # wait for devices to populate /dev/disk/by-id
> l=0
> count=`find /dev/disk/by-id -mindeth 1 2>/dev/null | wc -l`
> while [ $l -lt 5 ]; do
>     curr=`find /dev/disk/by-id -mindeth 1 2>/dev/null | wc -l`
>     log "by-id now has $curr entries"
>     [ $curr -gt $count ] && break
>     l=$(($l + 1))
>     sleep 1;
> done
> 
> # if not firstboot, expect LVs 
> if [ "$firstboot" != "1" ]; then
>     # it's not the firstboot
>     log "Scanning for lvm config"
>     l=0
>     while [ $l -lt 3 ]; do
>         echo "lvm scan count=$l" >> /tmp/lvm.log
>         pvscan >> /tmp/lvm.log
>         sleep 1
>         vgscan >> /tmp/lvm.log
>         sleep 1
>         lvscan >> /tmp/lvm.log
>         sleep 1
>         vgchange -ay >> /tmp/lvm.log
>         [ -e /dev/HostVG/Config ] && break;
>         l=$(($l + 1))
>     done
>     if [ ! -e /dev/HostVG/Config ] ; then
>         log "Rescue mode requested, starting emergency shell"
>         stop_log
>         bash < /dev/console
>         start_log
>     fi
> fi
> 
> At this point, we've made an iscsi connection and repopulated LVM and we can mount /config.

Seems reasonable.  I'm sure with a little tweaking this could be included
somehow, but I'll defer to Alan and Joey on the specifics :)

> 
>>
>> apevec, since jboggs is out of the office for a bit can you follow up here?
>>
>>> I'll have to look at seeing if we could move that into it's own init
>>> script; though there might be some duplication of code (command parsing,
>>> and such).  With the vendor approach, it executes in the same context as
>>> ovirt-early so one can potentially override function implementations and
>>> exert control over the flow in the remainder of ovirt-early; that
>>> wouldn't be doable in a separate init script.  
>>>
>>> One example that we rely upon that can't be done from separate script is
>>> appending boot parameters to the params list to be saved.
>>
>> ... But regardless, it is probably worth creating an easy to use interface
>> for vendors to add their own init scripts in without needing to duplicate
>> code, and trampling on ovirt-early.  Perhaps an
>> /etc/insert-your-vendor-script-here.d directory that ovirt-early and
>> ovirt-post reads and executes functions from?  That would provide an easy
>> way to drop in custom functionality and would also not require a new
>> kernel cmdline param.  Thoughts?
> 
> That does seem cleaner; though I do still like having a way to choose
> which set of vendor scripts to run via kernel parameter.  We'd like to
> have a single iso image that we could support different types of
> startups.

Yeah, I have no opposition to the kernel cmdline parameter used for that.