[libvirt] Tunnelled post-copy live migration - support of bidirectional communication over the tunnel

[Vojtech Cima]

Hello,

thank you for the clarification.

However, right now I still cannot see any traffic going back from the 
stream (stream doesn't become readable on the source side). I still 
struggle with the overall tunnelling design - especially on the 
destination side. I suppose following scheme where unidirectional pipes 
were replaced by sockets:

(souce QEMU) sock1 ----- sock2 === A TUNNEL (STREAM) B === sock3 ----- 
sock4(destination QEMU)

I think the source side is quite clear Sock1 and sock2 are socket pair 
on the source host. Data from src to dest are sent just by reading from 
sock2 and and sending to the A-end of the stream.

Sock3 and sock4 is the socketpair on the destination side. The part that 
confuses me is the destination side. How is the traffic handled there? 
In the 'qemuMigrationPrepareAny' around here [1]. The fd of sock3 is 
given to the stream and the fd of the sock4 to the destination QEMU .
Does it mean, that All traffic sent to the A-end of the stream goes all 
the way to the destination QEMU (sock4) with any other handling? (At 
least it seems to be that case.)
And the same question for the other direction. Does it mean that all 
data written by the destination QEMU to the sock4 should automatically 
appear on the A-end of the tunnel? Or is there a need to add a logic to 
add code that reads from sock3 and send it to B-end of the stream? 
Overally I am not very clear with the relationship of B-end of the 
tunnel and fd of sock(3).

[1]http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/qemu/qemu_migration.c;h=0acbb5795e930e27157c580a3236701acd91bbaf;hb=e3435caf6af41748204e542dee13ede8441d88c0#l3011

Thank you very much for any suggestions.

Regards,
Vojtech

On 15.01.2015 14:56, Daniel P. Berrange wrote:
> On Thu, Jan 15, 2015 at 02:45:13PM +0100, Vojtech Cima wrote:
>> Hello everyone,
>>
>> we are very interested about the post-copy live migration mechanism in
>> libvirt [1] using QEMU. Mentioning post-copy in this context means the setup
>> with the first full iteration of standard pre-copy and then switching to the
>> post-copy. VIR_MIGRATE_ENABLE_POSTCOPY, VIR_MIGRATE_POSTCOPY_AFTER_PRECOPY
>> together with VIR_MIGRATE_TUNNELLED flag.
>>
>> Recently we have noticed the issue that post-copy live migration doesn't
>> work over the tunnel since since current tunnel implementation in libvirt
>> supports only uni-directional communication. I spent some time trying to
>> enable bidirectional communication over the tunnel but still facing several
>> issues and I would appreciate any hints how to move forward.
>>
>> The idea is to add a function (probably a callback function) responsible for
>> handling the traffic in the opposite direction (dest->src) reading from the
>> virStream and writing to the QEMU [2]. To be able to do so I replaced the
>> current 'pipe' calls with the 'socketpair' calls to create bidirectional
>> paths.
>>
>> Now, trying to use tunnelled post-copy live migration, it gets initiated,
>> there is some traffic going from source to destination (obviously the
>> pre-copy phase) but at some point it gets stuck on the 'saferead' function
>> [3] which is now blocking. Can this issue issue be linked with the default
>> socket blocking policy?
>> Trying to use standard 'read' function instead of 'saferead' results that
>> the lower data transfers and it seems it's missing significant part of
>> information.
> Since the code was historically unidirectional, the qemuMigrationIOFunc()
> method used the virStream object in blocking I/O mode, and used a simple
> loop to read from the pipe / write to the stream. If you want to do
> bi-directional transfer that won't be possible - you'll need to put the
> stream into fully non-blocking mode and use event callbacks to deal with
> all I/O. This also means you can't use saferead either. The code in
> tools/virsh-console.c is a reasonably good example of how to use a
> stream in non-blocking mode doing  bi-directional I/O between the stream
> and regular file descriptors. The virsh-console uses stdin/stdout but
> you'd just use your socketpair instead.
>
>
>> Another point is that we suppose that the virStream on its own supports
>> bidirectional communication, is that correct?
> Yes, absolutely.
>
>> [1] https://github.com/orbitfp7/qemu/tree/wp3-postcopy
>>
>> [2] http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/qemu/qemu_migration.c;h=0acbb5795e930e27157c580a3236701acd91bbaf;hb=e3435caf6af41748204e542dee13ede8441d88c0#l3595
>>
>> [3] http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/qemu/qemu_migration.c;h=0acbb5795e930e27157c580a3236701acd91bbaf;hb=e3435caf6af41748204e542dee13ede8441d88c0#l3644
>>
>> Thank you very much for any ideas on this topic.
> Regards,
> Daniel