[libvirt] [PATCH 1/2] rpc: client: fix race on stream error and stream creation

[Nikolay Shirokovskiy]

On 22.02.2019 18:15, John Ferlan wrote:
> 
> 
> On 2/15/19 9:10 AM, Nikolay Shirokovskiy wrote:
>> Message of API call that creates stream and stream itself have
>> same rpc serial. This can lead to issues. If stream got error
>> immediately after creation then notification can be delivered
>> before API call reply arrived. This is possible because the
>> reply and the error message are sent from different threads
>> (rpc worker thread and event loop thread respectively). As
>> we don't check for message type in virNetClientCallCompleteAllWaitingReply
>> we complete the API call which leads to API call error [1]
>> as there is no actual reply. Later when reply arrives connection
>> will be closed due to protocol error (see check in virNetClientCallDispatchReply).
>> Let's fix virNetClientCallCompleteAllWaitingReply.
>>
>> Queue inspection on arriving VIR_NET_CONTINUE message in
>> virNetClientCallDispatchStream is safe because there we check for
>> status field also. Queue inspection on arriving VIR_NET_OK is safe
>> too as this message can not arrive before we call virFinishAbort(Finish)
>> which is not possible before API call that creates streams returns.
>> But just to be sure let's add checking message type in these places too.
>>
>> [1] error: internal error: Unexpected message type 0
>>
>> Signed-off-by: Nikolay Shirokovskiy <nshirokovskiy at virtuozzo.com>
>> ---
>>  src/rpc/virnetclient.c | 9 ++++++---
>>  1 file changed, 6 insertions(+), 3 deletions(-)
>>
> 
> The commit message was hard to read, but I know it's an english is not
> the first language type thing.
> 
>> diff --git a/src/rpc/virnetclient.c b/src/rpc/virnetclient.c
>> index 64855fb..0393587 100644
>> --- a/src/rpc/virnetclient.c
>> +++ b/src/rpc/virnetclient.c
>> @@ -1163,7 +1163,8 @@ static void virNetClientCallCompleteAllWaitingReply(virNetClientPtr client)
>>      virNetClientCallPtr call;
>>  
>>      for (call = client->waitDispatch; call; call = call->next) {
>> -        if (call->msg->header.prog == client->msg.header.prog &&
>> +        if (call->msg->header.type == VIR_NET_STREAM &&
> 
> What about VIR_NET_STREAM_HOLE ?

Here and in the other places we care only on "dummy recv"/finish/abort messages
(only such messages have expectReply flag set) and they all have type VIR_NET_STREAM.

> 
> Hopefully someone with greater working knowledge of the processing of
> this code can help out here. I guess I'm just concerned about some odd
> processing case that may have expected to be called, but now wouldn't be
> because of this change.
> 
> BTW: any thoughts to perhaps extracting out the common parts between the
> three places changed into it's own helper/method? e.g.:
> 
>  thecall->msg->header.prog == client->msg.header.prog &&
>  thecall->msg->header.vers == client->msg.header.vers &&
>  thecall->msg->header.serial == client->msg.header.serial &&
>  thecall->expectReply
> 
> and then the follow-up patch to add the additional '.type' checks?
> 

I'm ok.

Nikolay

> 
>> +            call->msg->header.prog == client->msg.header.prog &&
>>              call->msg->header.vers == client->msg.header.vers &&
>>              call->msg->header.serial == client->msg.header.serial &&
>>              call->expectReply)
>> @@ -1207,7 +1208,8 @@ static int virNetClientCallDispatchStream(virNetClientPtr client)
>>  
>>          /* Find oldest dummy message waiting for incoming data. */
>>          for (thecall = client->waitDispatch; thecall; thecall = thecall->next) {
>> -            if (thecall->msg->header.prog == client->msg.header.prog &&
>> +            if (thecall->msg->header.type == VIR_NET_STREAM &&
>> +                thecall->msg->header.prog == client->msg.header.prog &&
>>                  thecall->msg->header.vers == client->msg.header.vers &&
>>                  thecall->msg->header.serial == client->msg.header.serial &&
>>                  thecall->expectReply &&
>> @@ -1225,7 +1227,8 @@ static int virNetClientCallDispatchStream(virNetClientPtr client)
>>      case VIR_NET_OK:
>>          /* Find oldest abort/finish message. */
>>          for (thecall = client->waitDispatch; thecall; thecall = thecall->next) {
>> -            if (thecall->msg->header.prog == client->msg.header.prog &&
>> +            if (thecall->msg->header.type == VIR_NET_STREAM &&
>> +                thecall->msg->header.prog == client->msg.header.prog &&
>>                  thecall->msg->header.vers == client->msg.header.vers &&
>>                  thecall->msg->header.serial == client->msg.header.serial &&
>>                  thecall->expectReply &&
>>