[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

[libvirt-users] libvirtd does not update VM .xml configuration on filesystem after virsh blockcommit



Hi,

Recently We've upgraded some KVM hosts from Fedora 29 to Fedora 30 and
now experience broken VM configurations on filesystem after virsh blockcommit.

Commands "virsh dumpxml ..." and "virsh dumpxml --inactive ..." is showing diffrent configuration than the one on filesystem.
In case of restart libvirtd or system reboot, there are broken VM xml configurations on filesystem.

Everything is OK on Fedora 29 KVM hosts!


0. XML configurations before snapshot is taken (all good, nothing found)

[root server1 ~]# cat /etc/libvirt/qemu/somedomain.com.ncloud.xml| grep BACK
[root server1 ~]# cat /etc/libvirt/qemu/somedomain.com.ncloud.xml| grep backingStore
[root server1 ~]# /usr/bin/virsh --quiet  dumpxml somedomain.com.ncloud|grep BACK
[root server1 ~]# /usr/bin/virsh --quiet  dumpxml somedomain.com.ncloud|grep backingStore
[root server1 ~]# /usr/bin/virsh --quiet  dumpxml --inactive somedomain.com.ncloud|grep BACK
[root server1 ~]# /usr/bin/virsh --quiet  dumpxml --inactive somedomain.com.ncloud|grep backingStore



1. When VM snapshot is taken with (all OK)

/usr/bin/virsh --quiet  snapshot-create-as --domain somedomain.com.ncloud ....

there is a change of configuration, active, inactive and filesystem:

- active (virsh dumpxml)

[root server1 ~]# /usr/bin/virsh --quiet  dumpxml somedomain.com.ncloud|grep BACK
      <source file='/Virtualization/linux/somedomain.com/somedomain.com.ncloud.qcow2-BACKUPING_NOW'/>
      <source file='/Virtualization/linux/somedomain.com/somedomain.com.ncloud-swap.qcow2-BACKUPING_NOW'/>
      <source file='/Virtualization/linux/somedomain.com/somedomain.com.ncloud-data1.qcow2-BACKUPING_NOW'/>
      <source file='/Virtualization/linux/somedomain.com/somedomain.com.ncloud-data2.qcow2-BACKUPING_NOW'/>

[root server1 ~]# /usr/bin/virsh --quiet  dumpxml somedomain.com.ncloud|grep backingStore
      <backingStore type='file'>
      </backingStore>
      <backingStore type='file'>
      </backingStore>
      <backingStore type='file'>
      </backingStore>
      <backingStore type='file'>
      </backingStore>

- inactive (virsh dumpxml --inactive)

[root server1 ~]# /usr/bin/virsh --quiet  dumpxml --inactive somedomain.com.ncloud|grep BACK
      <source file='/Virtualization/linux/somedomain.com/somedomain.com.ncloud.qcow2-BACKUPING_NOW'/>
      <source file='/Virtualization/linux/somedomain.com/somedomain.com.ncloud-swap.qcow2-BACKUPING_NOW'/>
      <source file='/Virtualization/linux/somedomain.com/somedomain.com.ncloud-data1.qcow2-BACKUPING_NOW'/>
      <source file='/Virtualization/linux/somedomain.com/somedomain.com.ncloud-data2.qcow2-BACKUPING_NOW'/>

[root server1 ~]# /usr/bin/virsh --quiet  dumpxml --inactive somedomain.com.ncloud|grep backingStore
      <backingStore type='file'>
      </backingStore>
      <backingStore type='file'>
      </backingStore>
      <backingStore type='file'>
      </backingStore>
      <backingStore type='file'>
      </backingStore>


- XML configuration on filesystem has changed

[root server1 ~]# ls -al /etc/libvirt/qemu/somedomain.com.ncloud.xml
-rw-------. 1 root root 6260 Jun 18 23:00 /etc/libvirt/qemu/somedomain.com.ncloud.xml

[root server1 ~]# cat /etc/libvirt/qemu/somedomain.com.ncloud.xml |grep BACK
      <source file='/Virtualization/linux/somedomain.com/somedomain.com.ncloud.qcow2-BACKUPING_NOW'/>
      <source file='/Virtualization/linux/somedomain.com/somedomain.com.ncloud-swap.qcow2-BACKUPING_NOW'/>
      <source file='/Virtualization/linux/somedomain.com/somedomain.com.ncloud-data1.qcow2-BACKUPING_NOW'/>
      <source file='/Virtualization/linux/somedomain.com/somedomain.com.ncloud-data2.qcow2-BACKUPING_NOW'/>

[root server1 ~]# cat /etc/libvirt/qemu/somedomain.com.ncloud.xml |grep backingStore
      <backingStore type='file'>
      </backingStore>
      <backingStore type='file'>
      </backingStore>
      <backingStore type='file'>
      </backingStore>
      <backingStore type='file'>
      </backingStore>



2. When VM backup is done, data merged with virsh backcommit ..." and snapshot deleted (NOT OK!!!)

...
/usr/bin/virsh --quiet  blockcommit somedomain.com.ncloud sdd --active --pivot
/usr/bin/virsh --quiet  snapshot-delete --domain somedomain.com.ncloud somedomain.com.ncloud-SNAPSHOT --metadata


there is a following state of VM configurations:


- active (virsh dumpxml),

[root server1 ~]# /usr/bin/virsh --quiet  dumpxml  somedomain.com.ncloud|grep BACK ;;; OK

[root server1 ~]# /usr/bin/virsh --quiet  dumpxml  somedomain.com.ncloud|grep backingStore ;;; why is there empty backingStore left ???
      <backingStore/>
      <backingStore/>
      <backingStore/>
      <backingStore/>

- inactive

[root server1 qemu]# virsh dumpxml --inactive somedomain.com.ncloud |grep BACK          ;;; OK
[root server1 qemu]# virsh dumpxml --inactive somedomain.com.ncloud |grep backingStore  ;;; OK


- XML on filesystem (.xml file on filesystem has not changed/reverted since snapshot has been taken - NOT OK!!!!, should be cleared of snapshot source file and backingStore)

[root server1 ~]# ls -al /etc/libvirt/qemu/somedomain.com.ncloud.xml
-rw-------. 1 root root 6260 Jun 18 23:00 /etc/libvirt/qemu/somedomain.com.ncloud.xml

[root server1 qemu]# cat somedomain.com.ns2.xml |grep BACK
      <source file='/Virtualization/linux/somedomain.com/somedomain.com.ns2.qcow2-BACKUPING_NOW'/>

[root server1 qemu]# cat somedomain.com.ns2.xml |grep backingStore
      <backingStore type='file' index='1'>
      </backingStore>

##############################################################################################################################################################################

- Fedora 29 has libvirt 4.7.0 and qemu 3.0.1:

[root solaris1 ~]# rpm -qa |grep libvirt-daemon-kvm
libvirt-daemon-kvm-4.7.0-3.fc29.x86_64

[root solaris1 ~]# rpm -qa |grep qemu-system-x86
qemu-system-x86-core-3.0.1-3.fc29.x86_64
qemu-system-x86-3.0.1-3.fc29.x86_64

- Fedora 30 has libvirt 5.1.0 and qemu 3.1.0:

[root server1 ~]# rpm -qa |grep libvirt-daemon-kvm
libvirt-daemon-kvm-5.1.0-8.fc30.x86_64

[root server1 ~]# rpm -qa |grep qemu-system-x86
qemu-system-x86-3.1.0-8.fc30.x86_64
qemu-system-x86-core-3.1.0-8.fc30.x86_64


For every VM from "virsh list" we do following steps (in script) for VM backup:

/usr/bin/virsh --quiet  domblklist somedomain.com.ncloud
/usr/bin/virsh --quiet  dumpxml --inactive somedomain.com.ncloud > /Backuping/VMs/Daily/somedomain.com.ncloud.xml

/usr/bin/virsh --quiet  snapshot-create-as --domain somedomain.com.ncloud somedomain.com.ncloud-SNAPSHOT  --diskspec sda,file=/Virtualization/linux/somedomain.com/somedomain.com.ncloud.qcow2-BACKUPING_NOW --diskspec sdb,file=/Virtualization/linux/somedomain.com/somedomain.com.ncloud-swap.qcow2-BACKUPING_NOW --diskspec sdc,file=/Virtualization/linux/somedomain.com/somedomain.com.ncloud-data1.qcow2-BACKUPING_NOW --diskspec sdd,file=/Virtualization/linux/somedomain.com/somedomain.com.ncloud-data2.qcow2-BACKUPING_NOW --disk-only --atomic --quiesce

/usr/bin/virsh --quiet  snapshot-list somedomain.com.ncloud 

/usr/bin/scp -p server1.somedomain.us:/Virtualization/linux/somedomain.com/somedomain.com.ncloud.qcow2 somedomain.com.ncloud.qcow2
/usr/bin/scp -p server1.somedomain.us:/Virtualization/linux/somedomain.com/somedomain.com.ncloud-swap.qcow2 somedomain.com.ncloud-swap.qcow2
/usr/bin/scp -p server1.somedomain.us:/Virtualization/linux/somedomain.com/somedomain.com.ncloud-data1.qcow2 somedomain.com.ncloud-data1.qcow2
/usr/bin/scp -p server1.somedomain.us:/Virtualization/linux/somedomain.com/somedomain.com.ncloud-data2.qcow2 somedomain.com.ncloud-data2.qcow2

/usr/bin/virsh --quiet  blockcommit somedomain.com.ncloud sda --active --pivot
/usr/bin/virsh --quiet  blockcommit somedomain.com.ncloud sdb --active --pivot
/usr/bin/virsh --quiet  blockcommit somedomain.com.ncloud sdc --active --pivot
/usr/bin/virsh --quiet  blockcommit somedomain.com.ncloud sdd --active --pivot
/usr/bin/virsh --quiet  snapshot-delete --domain somedomain.com.ncloud somedomain.com.ncloud-SNAPSHOT --metadata

/usr/bin/ssh server1.somedomain.us "/usr/bin/rm /Virtualization/linux/somedomain.com/somedomain.com.ncloud.qcow2-BACKUPING_NOW"
/usr/bin/ssh server1.somedomain.us "/usr/bin/rm /Virtualization/linux/somedomain.com/somedomain.com.ncloud-swap.qcow2-BACKUPING_NOW"
/usr/bin/ssh server1.somedomain.us "/usr/bin/rm /Virtualization/linux/somedomain.com/somedomain.com.ncloud-data1.qcow2-BACKUPING_NOW"
/usr/bin/ssh server1.somedomain.us "/usr/bin/rm /Virtualization/linux/somedomain.com/somedomain.com.ncloud-data2.qcow2-BACKUPING_NOW"

/usr/bin/pigz --best --rsyncable somedomain.com.ncloud.qcow2
/usr/bin/pigz --best --rsyncable somedomain.com.ncloud-swap.qcow2
/usr/bin/pigz --best --rsyncable somedomain.com.ncloud-data1.qcow2
/usr/bin/pigz --best --rsyncable somedomain.com.ncloud-data2.qcow2


There is no error on script (commands) execution:


[root server1 ~]# /Backuping/bin/simple_KVM_backup.pl --depth 7 --path Daily --hosting_server server1.somedomain.us

###Backuping VM somedomain.com.ncloud:

+ Moving somedomain.com.ncloud.xml.06 to somedomain.com.ncloud.xml.07.
+ Moving somedomain.com.ncloud.qcow2.gz.06 to somedomain.com.ncloud.qcow2.gz.07.
...

+ Disk snapshot somedomain.com.ncloud-SNAPSHOT for somedomain.com.ncloud ... Domain snapshot somedomain.com.ncloud-SNAPSHOT created
 Name                       Creation Time               State
-----------------------------------------------------------------------
 somedomain.com.ncloud-SNAPSHOT   2019-06-18 23:00:04 +0200   disk-snapshot

Done!

+ Time scp start : 2019-06-18 23:00:05
somedomain.com.ncloud.qcow2                                                                                                                                                 100%   20GB 401.3MB/s   00:51    
Done!
somedomain.com.ncloud-swap.qcow2                                                                                                                                            100% 4097MB 483.5MB/s   00:08    
Done!
somedomain.com.ncloud-data1.qcow2                                                                                                                                           100%  100GB 400.7MB/s   04:15    
Done!
somedomain.com.ncloud-data2.qcow2                                                                                                                                                   100%  100GB 440.6MB/s   03:52    
Done!
+ Time scp end   : 2019-06-18 23:09:14

+ Time blockcommit start : 2019-06-18 23:09:14
Successfully pivoted

Successfully pivoted

Successfully pivoted

Successfully pivoted

+ Time blockcommit end   : 2019-06-18 23:09:16

+ Snapshot delete ... Domain snapshot somedomain.com.ncloud-SNAPSHOT deleted

Done!
+ Delete snapshot files ... Done!

+ Time compress start : 2019-06-18 23:09:17


Regards,
saso


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]