[rhelv6-list] KVM issues post RHEL6-1->6.2 update

[Robin Price II]

Ben,

I went ahead and did a 'virsh xmldump' of my RHEL6.1 guest I upgraded to 
RHEL6.2 from yesterday for you.

http://people.redhat.com/rprice/RHEL6.xml

Not sure if that will help but maybe reviewing some of the stanzas, you 
can re-add them as needed.


HTHs,

~rp


On 12/08/2011 04:31 AM, Ben wrote:
> Just thought I'd share my experiences of updating a KVM host and guests
> this morning. I'll acknowledge up front that I didn't do things in the
> right order so the mistakes were mine.
>
> Start: RHEL6.1 KVM host, x2 RHEL6.1 guests using .img files (LVM
> partitions inside). Fully up to date as of just before the RHEL6.2
> errata release.
>
> I did "yum clean all ; yum update" on both the host and the guests at
> the same time (yeah, I know). In my defence, a seemingly identical setup
> I did this on yesterday worked without issues.
>
> At the point at which the host was completing its cleanup this happened
> in /var/log/messages:
>
> Dec 8 07:14:47 frazil libvirtd: 07:14:47.926: 14778: warning :
> qemudDispatchSignalEvent:403 : Shutting down on signal 15
> Dec 8 07:14:49 frazil yum[1235]: Updated: libvirt-0.9.4-23.el6_2.1.x86_64
>
> and further down
>
> Dec 8 07:15:00 frazil kernel: br1: port 2(vnet1) entering disabled state
> Dec 8 07:15:00 frazil kernel: device vnet1 left promiscuous mode
> Dec 8 07:15:00 frazil kernel: br1: port 2(vnet1) entering disabled state
> Dec 8 07:15:02 frazil ntpd[2194]: Deleting interface #23 vnet1,
> fe80::fc54:ff:fe01:6b3b#123, interface stats: received=0, sent=0,
> dropped=0, active_time=7241352 secs
> Dec 8 07:15:05 frazil kernel: br0: port 2(vnet0) entering disabled state
> Dec 8 07:15:05 frazil kernel: device vnet0 left promiscuous mode
> Dec 8 07:15:05 frazil kernel: br0: port 2(vnet0) entering disabled state
> Dec 8 07:15:07 frazil ntpd[2194]: Deleting interface #25 vnet0,
> fe80::fc54:ff:fe49:fae6#123, interface stats: received=0, sent=0,
> dropped=0, active_time=7238050 secs
>
> At this point I lost connection to the guests, which (according to the
> SSH connections I had open to them) had apparently finished cleaning up
> after the yum update (according to the right-hand side X/Y counter) but
> hadn't returned a prompt yet so were obviously still busy doing stuff.
>
> I guess the restart of the libvirtd service dropped the guests (except
> the same lines appear in the messages file of the server on which the
> guests didn't get killed).
>
> Given I was rebooting the host anyway I didn't bother to bring the
> guests back up again and rebooted the host (yeah, I know). On reboot
> neither of the guests autostarted, so I logged in to the host and tried
> to start them with "virsh start <domain>". Both complained that
>
> error: internal error unable to reserve PCI address 0:0:2.0
>
> and didn't start. Checking the .xml files for both guests I noted that
>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
>
> was listed for the 'disk' device. I also noticed that the following
> lines were missing
>
> <input type='mouse' bus='ps2'/>
> <graphics type='vnc' port='5901' autoport='no'/>
> <video>
> <model type='cirrus' vram='9216' heads='1'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
> </video>
>
> whereas they were in place for the KVM setup/host and guests which had
> successfully updated. I added in the lines, made the 'disk' PCI ID
> something else and after restarting libvirtd tried booting the guests
> again. Still no joy. Still the same error. In the end I commented out
> the "address type='pci'" line for 'video' and attempted to boot again.
> This time I got failures booting the newly installed kernel at the point
> at which the root LVM mount was attempted. It recommended I look at the
> "root=" part of the boot line, but didn't give me suggestions as to what
> to put there.
>
> At this point I tried mounting the guests' disk images to see if the
> update of the kernel hadn't worked fully and the grub.conf was in a mess:
>
> # losetup /dev/loop0 foo.img
> # kpartx -av /dev/loop0
> # mount /dev/mapper/loop0p1 /mnt
> ...
> # umount /mnt
> # kpartx -dv /dev/loop0
> # losetup -d /dev/loop0
>
> Once inside the image I looked at the grub.conf files and couldn't see
> any issues. I umounted the image and tried booting into an older kernel
> and the guests booted successfully. "yum update" indicated an incomplete
> transaction so I ran "yum-complete-transaction" and then "yum update
> kernel" and rebooted both guests successfully into the new kernel. All
> now seems well. Phew.
>
> My questions are:
>
> 1) Is it a bad idea to patch the host's libvirtd while guests are running?
> 2) Should libvirtd have killed the guests like that?
> 3) With this update to KVM/qemu/libvird are "address type='pci'" now
> unnecessary and removable from /etc/libvirt/qemu/<domain>.xml files as
> PCI IDs are now dynamically assigned?
>
> Ben


-- 
+-----------------------------[ robin at redhat.com ]----+
| Robin Price II - RHCE,RHCDS,RHCVA                   |
| Technical Account Manager                           |
| Red Hat, Inc.                                       |
| w: +1 (919) 754 4412                                |
| c: +1 (252) 474 3525                                |
|                                                     |
+---------[ Dissenters will inevitably abhor. ]-------+