[Linux-cluster] Problem deleting running VM from rgmanager

[Digimer]

Hi all,

   I've got a problem where I deleted a running VM from the cluster using;

ccs -h localhost --activate --sync --password "secret" --rmvm vm01-win7

   This kind of worked, in that the VM was removed from cluster.conf, 
but 'clustat' still shows it. The logs from the call are:

=====
Jul 26 20:19:01 an-c05n01 ricci[18020]: Executing '/usr/bin/virsh nodeinfo'
Jul 26 20:19:01 an-c05n01 ricci[18066]: Executing '/usr/bin/virsh nodeinfo'
Jul 26 20:19:01 an-c05n01 ricci[18069]: Executing '/usr/bin/virsh nodeinfo'
Jul 26 20:19:01 an-c05n01 ricci[18071]: Executing 
'/usr/libexec/ricci/ricci-worker -f /var/lib/ricci/queue/1428781577'
Jul 26 20:19:01 an-c05n01 ricci[18075]: Executing '/usr/bin/virsh nodeinfo'
Jul 26 20:19:01 an-c05n01 ricci[18077]: Executing '/usr/bin/virsh nodeinfo'
Jul 26 20:19:01 an-c05n01 ricci[18080]: Executing '/usr/bin/virsh nodeinfo'
Jul 26 20:19:01 an-c05n01 ricci[18082]: Executing 
'/usr/libexec/ricci/ricci-worker -f /var/lib/ricci/queue/881799278'
Jul 26 20:19:03 an-c05n01 ricci[18088]: Executing '/usr/bin/virsh nodeinfo'
Jul 26 20:19:03 an-c05n01 ricci[18090]: Executing '/usr/bin/virsh nodeinfo'
Jul 26 20:19:03 an-c05n01 ricci[18093]: Executing '/usr/bin/virsh nodeinfo'
Jul 26 20:19:03 an-c05n01 ricci[18095]: Executing 
'/usr/libexec/ricci/ricci-worker -f /var/lib/ricci/queue/439919971'
Jul 26 20:19:03 an-c05n01 modcluster: Updating cluster.conf
Jul 26 20:19:03 an-c05n01 corosync[3479]:   [QUORUM] Members[2]: 1 2
Jul 26 20:19:03 an-c05n01 ricci[18140]: Executing '/usr/bin/virsh nodeinfo'
Jul 26 20:19:03 an-c05n01 ricci[18170]: Executing '/usr/bin/virsh nodeinfo'
Jul 26 20:19:03 an-c05n01 rgmanager[3710]: Reconfiguring
Jul 26 20:19:03 an-c05n01 ricci[18194]: Executing '/usr/bin/virsh nodeinfo'
Jul 26 20:19:03 an-c05n01 ricci[18234]: Executing 
'/usr/libexec/ricci/ricci-worker -f /var/lib/ricci/queue/446527166'
Jul 26 20:19:04 an-c05n01 ricci[18457]: Executing '/usr/bin/virsh nodeinfo'
Jul 26 20:19:04 an-c05n01 ricci[18496]: Executing '/usr/bin/virsh nodeinfo'
Jul 26 20:19:04 an-c05n01 ricci[18528]: Executing '/usr/bin/virsh nodeinfo'
Jul 26 20:19:04 an-c05n01 ricci[18560]: Executing 
'/usr/libexec/ricci/ricci-worker -f /var/lib/ricci/queue/1207456461'
Jul 26 20:19:04 an-c05n01 modcluster: Updating cluster.conf
Jul 26 20:19:04 an-c05n01 corosync[3479]:   [QUORUM] Members[2]: 1 2
Jul 26 20:19:05 an-c05n01 kernel: vbr2: port 4(vnet2) entering disabled 
state
Jul 26 20:19:05 an-c05n01 kernel: device vnet2 left promiscuous mode
Jul 26 20:19:05 an-c05n01 kernel: vbr2: port 4(vnet2) entering disabled 
state
Jul 26 20:19:06 an-c05n01 rgmanager[3710]: vm:vm01-win7 removed from the 
config, but I am not stopping it.
Jul 26 20:19:06 an-c05n01 rgmanager[3710]: Reconfiguring
Jul 26 20:19:07 an-c05n01 ntpd[2794]: Deleting interface #16 vnet2, 
fe80::fc54:ff:fea5:37ea#123, interface stats: received=0, sent=0, 
dropped=0, active_time=135 secs
=====

However, clustat still shows;

=====
Cluster Status for an-cluster-05 @ Fri Jul 26 20:37:17 2013
Member Status: Quorate

  Member Name                             ID   Status
  ------ ----                             ---- ------
  an-c05n01.alteeve.ca                        1 Online, rgmanager
  an-c05n02.alteeve.ca                        2 Online, Local, rgmanager

  Service Name                   Owner (Last)                   State
  ------- ----                   ----- ------                   -----
  service:storage_n01            an-c05n01.alteeve.ca           started
  service:storage_n02            an-c05n02.alteeve.ca           started
  vm:vm01-win7                   an-c05n02.alteeve.ca           started
  vm:vm02-rhel6                  an-c05n02.alteeve.ca           started
  vm:vm03-debian7                an-c05n01.alteeve.ca           started
  vm:vm04-solaris11              an-c05n02.alteeve.ca           started
  vm:vm05-win2008r2              an-c05n02.alteeve.ca           started
  vm:vm06-win8                   an-c05n01.alteeve.ca           started
  vm:vm07-win2012                an-c05n02.alteeve.ca           started
  vm:vm08-freebsd9               an-c05n01.alteeve.ca           started
  vm:vm09-suse11                 an-c05n01.alteeve.ca           started
=====

Trying to stop it produces;

=====
an-c05n02:~# clusvcadm -d vm:vm01-win7
Local machine disabling vm:vm01-win7...Failure
=====

CentOS 6.4, fully up to date; rgmanager-3.0.12.1-17.el6.x86_64

-- 
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without 
access to education?