xen15 outage - sorry :-(

[Ricky Zhou]

Hey, here's a quick report of what just happened on xen15 and its
guests:

At around 3:00 UTC, I saw a puppet error email complaining about an
error on running /sbin/service iscsi start on xen15.  I logged onto
xen15 and stupidly ran a /etc/init.d/iscsi restart, realizing what I had
actually done a moment later.

I saw disk errors on the consoles of db2 and relepel1, which were on
iscsi.  At this point, I tried to shut all iscsi guests on that machine
down (db2, relepel1, sign-bridge1) with xm shutdown.  db2 and relepel1
both finished shutting down, but I had to xm destroy sign-bridge1 as it
hung at the end.

After all guests were down, I followed the Logging Out procedure at
https://fedoraproject.org/wiki/ISCSI_Infrastructure_SOP, ran
/etc/init.d/iscsi restart, then followed the Logging In procedure.

At that point, I was able to xm create all three guests again, and they
seem back up now (although an unrelated VPN outage is still keeping web
apps down at the moment).

If anybody sees any issues on any of these hosts - particularly with
data on db2, it could be related to this.  Sorry about this, I'll
definitely think more before I run a command like that next time.

Thanks,
Ricky
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/fedora-infrastructure-list/attachments/20090811/2b142986/attachment.sig>