[Libguestfs] [PATCH libnbd] info: Add --map --totals sub-mode to display summary of map

[Eric Blake]

On Sat, Jun 26, 2021 at 05:19:47PM +0100, Richard W.M. Jones wrote:
> This is similar to "qemu-img measure".  Some examples:
> 
> $ nbdkit -r file fedora-33.img --run 'nbdinfo --map --totals $uri'
> 1226113024  19.0   0 data
> 5216337920  81.0   3 hole,zero

Is it worth a human-legible header line?  Or maybe a % character in
the second column?

> 
> $ nbdkit -r file fedora-33.img --run 'nbdinfo --map --totals --json $uri | jq'
> [
>   {
>     "size": 1226113024,
>     "percent": 19.0318,
>     "type": 0,
>     "description": "data"
>   },
>   {
>     "size": 5216337920,
>     "percent": 80.9682,
>     "type": 3,
>     "description": "hole,zero"
>   }
> ]

Output looks nice.

> 
> $ nbdkit sparse-random 6G --run 'nbdinfo --map --totals $uri'
>  941551616  14.6   0 data
> 5500899328  85.4   3 hole,zero
> ---
...
> +++ b/info/info-map-totals-json.sh
> +
> +test $( jq -r '.[0].size' < $out ) -eq 65536
> +test $( jq -r '.[0].percent' < $out ) = "6.25"
> +test $( jq -r '.[0].type' < $out ) -eq 0
> +test $( jq -r '.[0].description' < $out ) = "data"
> +
> +test $( jq -r '.[1].size' < $out ) -eq 983040
> +test $( jq -r '.[1].percent' < $out ) = "93.75"
> +test $( jq -r '.[1].type' < $out ) -eq 3
> +test $( jq -r '.[1].description' < $out ) = "hole,zero"
...
> diff --git a/info/info-map-totals.sh b/info/info-map-totals.sh
> new file mode 100755
> index 0000000..12c1263
> --- /dev/null
> +++ b/info/info-map-totals.sh
> @@ -0,0 +1,43 @@

> +
> +cat $out
> +
> +if [ "$(tr -s ' ' < $out)" != " 65536 6.2 0 data
> + 983040 93.8 3 hole,zero" ]; then

So this is the human-readable counterpart of info-map-totals-json.sh.
I'm trying to figure out why 6.25 rounded down and 93.75 rounded up.
I guess it's related to round-to-even?  But it's also fine if the
rounded values don't sum back up to 100%; I don't know if it's worth a
documentation disclaimer.

> +++ b/info/map.c

> +/* --map --totals suboption */
> +static void
> +print_totals (uint32_vector *entries, int64_t size)
> +{
> +  uint32_t type;
> +  bool comma = false;
> +
> +  if (json_output) fprintf (fp, "[\n");
> +
> +  /* In the outer loop assume we have already printed all entries with
> +   * entry type < type.  Count all instances of type and at the same
> +   * time find the next type that exists > type.
> +   */
> +  type = 0;
> +  for (;;) {
> +    uint64_t next_type = (uint64_t)UINT32_MAX + 1;
> +    uint64_t c = 0;
> +    size_t i;
> +
> +    for (i = 0; i < entries->size; i += 2) {
> +      uint32_t t = entries->ptr[i+1];
> +
> +      if (t == type)
> +        c += entries->ptr[i];
> +      else if (type < t && t < next_type)
> +        next_type = t;
> +    }

I see what you mean about a malicious server being able to force us to
do a LOT of work, but I think this is fine for a first cut (we can
worry about rewriting it to do a single pass utilizing a hash table if
it proves to be noticeable in practice).

> +++ b/info/nbdinfo.pod
> @@ -8,7 +8,7 @@ nbdinfo - display information and metadata about NBD servers and exports
>  
>   nbdinfo --size [--json] NBD-URI
>  
> - nbdinfo --map [--json] NBD-URI
> + nbdinfo --map [--totals] [--json] NBD-URI
>  
>   nbdinfo -L|--list [--json] NBD-URI
>  
> @@ -119,6 +119,35 @@ other maps too:
>  For more information on NBD maps, see I<Metadata querying> in the NBD
>  protocol.
>  
> +=head2 Map totals
> +
> +Using S<I<--map --totals>> performs the same operation as I<--map> but
> +displays a summary of the total size of each type of allocation, in
> +bytes and as a percentage (of the virtual size of the export).  This
> +is useful for estimating how much real storage is used on the server,
> +or might be required when copying a sparse image with L<nbdcopy(1)>.
> +
> +In the example below, half (50.0%) of the disk is allocated data and
> +half is unallocated:
> +
> + $ nbdinfo --map --totals nbd://localhost/
> + 1048576  50.0  0  data
> + 1048576  50.0  3  hole,zero
> +
> +The fields are: total size in bytes, percentage of the virtual size,
> +type, description (optional).
> +
> +You can also get the same information in parseable form using I<--json>:
> +
> + $ nbdinfo --map --totals --json nbd://localhost/
> + [{ "size": 1048576, "percent": 50,
> +    "type": 0, "description": "data" },
> +  { "size": 1048576, "percent": 50,
> +    "type": 3, "description": "hole,zero" }]
> +
> +As with the I<--map> option, by default this shows the
> +C<"base:allocation"> map, but you can show the summary for other maps.

I'd still like to someday have a mode where we can read all advertised
maps at once, rather than making the command-line user have to
manually invoke nbdinfo once per map spelled by the user, but that can
still go on top of this.

Overall, looks good to me!

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org