[libvirt] virsh domjobinfo during storage migration

[Jiri Denemark]

On Thu, Nov 24, 2016 at 22:54:10 +0100, Ruben Kerkhof wrote:
> Hi all,
> 
> Running virsh domjobinfo on my CentOS 6 systems during a migration
> with --copy-storage-all, the output used to look like this:
> 
> Job type:         Unbounded
> Time elapsed:     1830632      ms
> Data processed:   37.212 GiB
> Data remaining:   1.025 GiB
> Data total:       16.016 GiB
> Memory processed: 37.212 GiB
> Memory remaining: 1.025 GiB
> Memory total:     16.016 GiB
> Memory bandwidth: 100.018 MiB/s
> Constant pages:   618279
> Normal pages:     9734623
> Normal data:      37.135 GiB
> Expected downtime: 1118         ms
> Setup time:       61           ms
> 
> But when I run the same command on CentOS 7, libvirt-1.2.17-13 and
> qemu-kvm-1.5.3-105, I just get this:
> 
> $ sudo virsh domjobinfo 58358fec-c35c-4a7a-a4dd-18ec2e1327bf
> Job type:         Unbounded
> Time elapsed:     209616       ms
> 
> I tried to debug this myself, but I'm a bit stuck. I'd expected
> libvirt to send query-migrate commands to qemu every 50 ms or so. To
> verify I attached gdb and put a breakpoint at
> qemuMonitorJSONGetMigrationStatus, but the breakpoint only hits after
> the complete disk is mirrored.

That's expected since QEMU finally implemented a migration event, which
allowed us to stop polling every 50 ms. Libvirt will only call
query-migrate at the end of a migration or when virsh domjobinfo is
called.

The reason why you don't see anything interesting in the output is NBD
storage migration. With new enough QEMU libvirt will first migrate
storage using QEMU's integrated NBD server and then "migrate" QMP
command will called. With old QEMU storage migration was done by the
"migrate" command itself. Thus calling query-migrate while storage is
migrated (i.e., before migrate was called) does not provide anything. I
guess it should be possible to also check the progress of all running
block jobs so that we can report statistics about ongoing storage
migration even when NBD is used.

Jirka