[libvirt] virsh domjobinfo during storage migration

[Ruben Kerkhof]

Hi Jiri,

On Fri, Nov 25, 2016 at 9:26 AM, Jiri Denemark <jdenemar at redhat.com> wrote:
> On Thu, Nov 24, 2016 at 22:54:10 +0100, Ruben Kerkhof wrote:
>> Hi all,
>>
>> Running virsh domjobinfo on my CentOS 6 systems during a migration
>> with --copy-storage-all, the output used to look like this:
>>
>> Job type:         Unbounded
>> Time elapsed:     1830632      ms
>> Data processed:   37.212 GiB
>> Data remaining:   1.025 GiB
>> Data total:       16.016 GiB
>> Memory processed: 37.212 GiB
>> Memory remaining: 1.025 GiB
>> Memory total:     16.016 GiB
>> Memory bandwidth: 100.018 MiB/s
>> Constant pages:   618279
>> Normal pages:     9734623
>> Normal data:      37.135 GiB
>> Expected downtime: 1118         ms
>> Setup time:       61           ms
>>
>> But when I run the same command on CentOS 7, libvirt-1.2.17-13 and
>> qemu-kvm-1.5.3-105, I just get this:
>>
>> $ sudo virsh domjobinfo 58358fec-c35c-4a7a-a4dd-18ec2e1327bf
>> Job type:         Unbounded
>> Time elapsed:     209616       ms
>>
>> I tried to debug this myself, but I'm a bit stuck. I'd expected
>> libvirt to send query-migrate commands to qemu every 50 ms or so. To
>> verify I attached gdb and put a breakpoint at
>> qemuMonitorJSONGetMigrationStatus, but the breakpoint only hits after
>> the complete disk is mirrored.
>
> That's expected since QEMU finally implemented a migration event, which
> allowed us to stop polling every 50 ms. Libvirt will only call
> query-migrate at the end of a migration or when virsh domjobinfo is
> called.
>
> The reason why you don't see anything interesting in the output is NBD
> storage migration. With new enough QEMU libvirt will first migrate
> storage using QEMU's integrated NBD server and then "migrate" QMP
> command will called. With old QEMU storage migration was done by the
> "migrate" command itself. Thus calling query-migrate while storage is
> migrated (i.e., before migrate was called) does not provide anything.

That makes sense, thanks for the explanation.

> I guess it should be possible to also check the progress of all running
> block jobs so that we can report statistics about ongoing storage
> migration even when NBD is used.

That would be great, since I won't be able to upgrade to a newer
version of QEMU yet.
How do you want to handle this, would you like me to open an issue in bugzilla?

>
> Jirka

Kind regards,

Ruben Kerkhof