[libvirt] QMP; unsigned 64-bit ints; JSON standards compliance

[Daniel P. Berrangé]

The QEMU  QMP service is based on JSON which is nice because that is a
widely supported "standard" data format.....

....except QEMU's implementation (and indeed most impls) are not strictly
standards compliant.

Specifically the problem is around representing 64-bit integers, whether
signed or unsigned.

The JSON standard declares that largest integer is 2^53-1 and the
likewise the smallest is -(2^53-1):

  http://www.ecma-international.org/ecma-262/6.0/index.html#sec-number.max_safe_integer

A crazy limit inherited from its javascript origins IIUC.

QEMU, and indeed many applications, want to handle 64-bit integers.
The C JSON library impls have traditionally mapped integers to the
data type 'long long int' which gives a min/max of  -(2^63) / 2^63-1.

QEMU however /really/ needs 64-bit unsigned integers, ie a max 2^64-1.

Libvirt has historically used the YAJL library which uses 'long long int'
and thus can't officially go beyond 2^63-1 values. Fortunately it lets
libvirt get at the raw json string, so libvirt can re-parse the value
to get an 'unsigned long long'.

We recently tried to switch to Jansson because YAJL has a dead upstream
for many years and countless unanswered bugs & patches. Unfortunately we
forgot about this need for 2^64-1 max, and Jansson also uses 'long long int'
and raises a fatal parse error for unsigned 64-bit values above 2^63-1. It
also provides no backdoor for libvirt todo its own integer parsing. Thus
we had to abort our switch to jansson as it broke parsing QEMU's JSON:

  https://bugzilla.redhat.com/show_bug.cgi?id=1614569

Other JSON libraries we've investigated have similar problems. I imagine
the same may well be true of non-C based JOSN impls, though I've not
investigated in any detail.

Essentially libvirt is stuck with either using the dead YAJL library
forever, or writing its own JSON parser (most likely copying QEMU's
JSON code into libvirt's git).

This feels like a very unappealing situation to be in as not being
able to use a JSON library of our choice is loosing one of the key
benefits of using a standard data format.

Thus I'd like to see a solution to this to allow QMP to be reliably
consumed by any JSON library that exists.

I can think of some options:

  1. Encode unsigned 64-bit integers as signed 64-bit integers.

     This follows the example that most C libraries map JSON ints
     to 'long long int'. This is still relying on undefined
     behaviour as apps don't need to support > 2^53-1.

     Apps would need to cast back to 'unsigned long long' for
     those QMP fields they know are supposed to be unsigned.


  2. Encode all 64-bit integers as a pair of 32-bit integers.
    
     This is fully compliant with the JSON spec as each half
     is fully within the declared limits. App has to split or
     assemble the 2 pieces from/to a signed/unsigned 64-bit
     int as needed.


  3. Encode all 64-bit integers as strings

     The application has todo all parsing/formatting client
     side.


None of these changes are backwards compatible, so I doubt we could make
the change transparently in QMP.  Instead we would have to have a
QMP greeting message capability where the client can request enablement
of the enhanced integer handling.

Any of the three options above would likely work for libvirt, but I
would have a slight preference for either 2 or 3, so that we become
100% standards compliant.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|