xen1 outage

[Mike McGrath]

On Sun, 20 Jan 2008, Jon Stanley wrote:

> I'm out watching football atm, can't really look at my mail archives,
> but has xen1 exhibited these issues since downgrading to RHEL5 GA
> kernel? And is there the common signature of iSCSI issue -> crash
> here? As someone who has been in the hosting business for a long time,
> I hate to make two separate issues the same when there is not data to
> support that conclusion. Feel free to tell me to shut up :-)
>
>


xen1 started exhibiting these issues when we moved from FC6 to RHEL5 GA.
It was assumed to be hardware issue because of how sporatic the issues
were and because we actually do have RHEL5 on other xen hosts.

The iscsi issue may or may not be a red herring but some of the reports
listed in the ticket (and I haven't done exhaustive research on this yet)
suggests a kernel / poweredge bug that we may be hitting.  We had moved
all non-redundant guests off of xen1 onto the more stable xen2 box.  After
upgrading xen2 to RHEL5 we started seeing the same problems with it.
There's a few things we can try, I'll be doing so on xen1 with proxy4 as
our test host since its competely redundant and has, in the past, crashed
that box.

For now I'm going to see if the FC6 kernel and xen libs keep xen2 happy
while we're testing / probing at xen1.

	-Mike