[Libguestfs] bug in gnutls_init()

[Eric Blake]

I just fixed a bug in nbdkit for incorrectly calling
free(gnutls_session_t) after gnutls_init(&session, ...) fails:
https://gitlab.com/nbdkit/nbdkit/-/commit/40faf3dfb20c06b9c5faa0a122607e3ae7c6202a

But in the process, I was browsing the source code to gnutls_init() to
see why Coverity wasn't flagging free(opaque_type) as fishy, and found
that there is a nasty lurking bug:

int gnutls_init(gnutls_session_t * session, unsigned int flags)
{
	int ret;

	FAIL_IF_LIB_ERROR;

	*session = gnutls_calloc(1, sizeof(struct gnutls_session_int));

Note that *session is left uninitialized if FAIL_IF_LIB_ERROR; causes
an early return GNUTLS_E_LIB_IN_ERROR_STATE.  If a caller (properly)
treats gnutls_session_t as an opaque type, and does not try to
zero-initialize it (as there is no way to know that 0 is a safe value
for an opaque type), then writing:

  gnutls_session_t session;
  int err = gnutls_init (&session, GNUTLS_SERVER);
  if (err < 0)
    gnutls_deinit (session);

is a bug waiting to happen, because it WILL cause gnutls_deinit() to
attempt to dereference an uninitialized pointer if session remains
uninitialized because of an earlier library error.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org