[Ovirt-devel] Ovirt Host Tasks

[Hugh O. Brock]

On Sun, Mar 16, 2008 at 11:43:02PM -0400, Perry N. Myers wrote:

Wow that's a big mail <g>.

I have a couple of general comments based on the way we *have been*
thinking about this stuff (this of course doesn't mean we can't
change). 

First off, we have been imagining the oVirt host image as stateless
except for whatever we use to store its identity (if we even do
that). This means we simply don't support static addressing of oVirt
hosts, at all. The appealing thing about a stateless oVirt host image
to me is that it means the host is immediately and easily upgradeable;
there is 0 local state to worry about aside from the Kerberos
keytab. It also means that a reboot will immediately restore a
compromised machine FWIW. 

Having said all that, if there is a compelling reason to move away
from the stateless idea I'm all ears.

Further comments in-line below...
>
> 1. Image Creation:
>
> Right now the Ovirt Host is created using scripts in the Ovirt host 
> repository that utilize livecd creator to make ISO, USB and PXE formats.
>
> I think it makes sense to start with the tools that Chris L. has already 
> been working with and migrate new functionality in, rather than starting 
> with other tools.
>
> a. Integrate whitelist functionality into the image creation
>    process to reduce image size.
> b. Determine the minimal set of files that the host image needs to
>    construct a baseline whitelist.
> c. Determine minimal set of kernel modules to support wide platform base
>    (i.e. keep disk, net modules.  remove things like sound, isdn, etc...)
> d. Construct a repeatable build process so that images can be built
>    daily for testing.
> e. Construct a utility that detects the minimal set of kernel modules
>    for a given platform and generates a config file that can be fed to
>    the build system for generating hardware platform specific images.
> f. Construct a method for taking the generic host image and removing all
>    kernel modules not applicable for a specific platform.
> g. Construct a method for taking an image and adding site specific
>    configuration information to it (IP addresses of Ovirt Mgmt Server and
>    FreeIPA Server).  This is only necessary to support hosts that require
>    static IP configuration -and- do not have persistent storage to store
>    configuration information.  (If we decide to support static IP config
>    we could make presence of persistent storage a requirement...)

You could add "Kerberos keytab" to "site specific configuration
information" and the host image would be completely
self-contained. I can imagine a library of unique images under
/tftpboot...? 
>
> 2. Installation
>
> We have to account for two methods of distributing Ovirt: pre-installation 
> by OEMs and install by IT admins.  Pre-install by OEM implies that the 
> platform has flash or a reserved partition on the primary hard disk. 
> Install by IT can be done by setting up PXE boot, adding an external 
> bootable USB disk, booting to CD or installing to onboard flash/disk.
>
> The livecd/liveusb disk should allow either booting and running directly or 
> installation onto platform flash/disk.

That's a nice addition, yeah. Although again I'm in favor of the
running image itself remaining stateless.

> 3. Persistent Configuration Storage
>
> Since the host is non-persistent and may or may not have access to 
> persistent storage, this could become tricky.  It's further complicated by 
> the need to support DHCP and static addressing of the host.
>
> The following things need to be stored for the host:
> a. Kerberos keytab or SSL Certificate
> b. IP configuration (if using static configuration)
> c. IP address of FreeIPA server and Ovirt Management Server (if using
>    static configuration)

> Here's where things could be much simpler if DHCP were required.  Without 
> static addressing, b is not necessary and c can be transmitted as custom 
> DHCP fields (iirc).  Then the only thing that needs to be cached on the 
> host on persistent storage is the auth credentials.  (NOTE: Does 
> Microsoft's DHCP server allow custom DHCP fields???)

Actually we've talked about moving away from overloading DHCP for this
stuff and instead handling it with mdns (let zeroconf be
zeroconf). 

There is another interesting possibility we might want to look at in the
medium term. We are actively working on ways to use amqp as part of
our infrastructure. If a machine has an identity via a keytab and an
IP address via DHCP, I imagine it's possible to retrieve the remaining
necessary config information by sending a message to a known alias?
This relieves the oVirt host of having to know the precise address of
the management server. Something to think about at least.

> 4. Initial Configuration
>
> When the host comes up the first time it needs to know the IP address and 
> password of the Ovirt Mgmt server so it can get its auth credentials set 
> up.  Here's how a typical setup might look:
>
> a. Cable up the new hardware and get MAC addrs for each iface.  Need
>    to record which MAC is for:
>    * management network (NOTE: should be only iface that PXE boots
>      if PXE is being used)
>    * storage network
>    * normal networks
>    (these all don't have to be separate, there could be just a single
>    interface on the box used for management, storage and normal traffic)
> b. Set up DNS/DHCP servers.  If using static addressing, skip DHCP setup
>    (This is a manual step that IT admin would do, i.e. we're not trying
>    to automate this process)
> c. Boot Ovirt Host for first time.  Kernel cmdline takes options like:
>    * ovirt_net=static (to indicate that static ip config should be
>      prompted for during boot)
>    * ovirt_auth_pw=<password> (password to use when connecting to the
>      Ovirt Mgmt server to register the host and retrieve auth information)
>    * ovirt_init (this is specified to let the host know that initial
>      config needs to happen.  Otherwise normal boot occurs)

Getting back to the stateless idea, I think the host image should
always boot exactly the same way and be smart enough to know whether
it needs config information or not and, if so, ask for it? Having said
that the "password" idea is reasonable except that you would always
need to be there to enter the password for a reboot...

> f. The host is checked for persistent storage and TPM devices.  If TPM
>    is found it is initialized so that auth info can be stored there.
>    * If either USB or hard disk is found it is checked for a partition
>      with a label called "OVIRT".  If that exists it is used for
>      persistent storage.  If it does not, an "OVIRT" partition is created.
>      on available unpartitioned space. (how big should it be?)
>    * USB/disk is also checked for swap partition.  If none is found, a
>      swap partition is created using unpartitioned space.

Hadn't thought about using TPM for the keytab, that's a neat idea. I'm
a bit leery of using any local HDD on the machine for it though.

> 7. Performance Monitoring/Auditing/Health
>
> * collectd is used presently to send performance stats to management
>   server.  Is this the right solution?  I don't have a better one, but
>   suggestions are appreciated.

There is some really interesting stuff going on around monitoring with
amqp. At a first step we need to try to get collectd hooked into
that. 

> * Need to create a health monitoring daemon that communicates to mgmt
>   server:
>   - Sends heartbeat at configurable interval
>   - Sends status changes of host (including machine check exceptions,
>     and other errors)
>   - Sends VM state changes

This is where the amqp stuff might be particularly useful, although we
need something in place before July (which is the earliest we can
really start looking at AMQP, it sounds like).

> 8. Clustering
>
> This probably needs a lot of refinement.  So someone who knows clustering 
> better than I do, should feel free to help out here...

I'm afraid that someone would not be me <g>...

Take care,
--Hugh