[Libguestfs] [libnbd PATCH] common/utils/vector: Better overflow handling

[Eric Blake]

Check newcap * itemsize for overflow prior to calling realloc, so that
we don't accidentally truncate an existing array.  Set errno to ENOMEM
on all failure paths, rather than leaving it indeterminate on
overflow.  The patch works by assuming a prerequisite that
v->cap*itemsize is always smaller than size_t.

Fixes: 985dfa72ae (common/utils/vector.c: Optimize vector append)
---

Unless you see problems in this, I'll push this and also port it to
nbdkit.

 common/utils/vector.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/common/utils/vector.c b/common/utils/vector.c
index a4b43ce..c37f0c3 100644
--- a/common/utils/vector.c
+++ b/common/utils/vector.c
@@ -1,5 +1,5 @@
 /* nbdkit
- * Copyright (C) 2018-2020 Red Hat Inc.
+ * Copyright (C) 2018-2021 Red Hat Inc.
  *
  * Redistribution and use in source and binary forms, with or without
  * modification, are permitted provided that the following conditions are
@@ -44,12 +44,14 @@ generic_vector_reserve (struct generic_vector *v, size_t n, size_t itemsize)
   size_t reqcap, newcap;

   reqcap = v->cap + n;
-  if (reqcap < v->cap)
+  if (reqcap * itemsize < v->cap * itemsize) {
+    errno = ENOMEM;
     return -1; /* overflow */
+  }

   newcap = (v->cap * 3 + 1) / 2;

-  if (newcap < reqcap)
+  if (newcap * itemsize < reqcap * itemsize)
     newcap = reqcap;

   newptr = realloc (v->ptr, newcap * itemsize);
-- 
2.33.1