[libvirt] [PATCH 0/3 RFC] Demonstrating a new API for spawning processes

[Daniel P. Berrange]

We have had great success with our APIs for memory allocation and
string buffer management, removing most of the potential for incorrect
API usage, thus avoiding many common errors. It is time todo the same
for command execution.

This patch series is a proof of concept for a set of APIs I've
been thinking about for many months. They are intended to replace
all current usage of virExec and virRun with a more flexible and
less error prone API. In particular they improve of OOM handling,
prevent more memory leaks & simplify alot of code that used horrible
macros

Patch 1 contains the actual implementation and comprehensive test
suite. Patch 2 contains HTML docs. Patch 3 ports two current methods
to the new APIs to demonstrate the improved code clarity.

What follows is a plain text rendering of the HTML docs from the
second patch describing these APIs. 

               Spawning processes / commands from libvirt drivers

   This page describes the usage of libvirt APIs for spawning processes /
   commands from libvirt drivers. All code is required to use these APIs

Problems with standard POSIX APIs

   The POSIX specification includes a number of APIs for spawning processes /
   commands, but they suffer from a number of flaws

     * fork+exec: The lowest & most flexible level, but very hard to use
       correctly / safely. It is easy to leak file descriptors, have
       unexpected signal handler behaviour and not handle edge cases
     * system: Convenient if you don't care about capturing command output,
       but has the serious downside that the command string is interpreted by
       the shell. This makes it very dangerous to use, because improperly
       validated user input can lead to exploits via shell meta characters.
     * posix_spawn: A half-way house between simplicity of system() and the
       flexibility of fork+exec. It does not allow for a couple of important
       features though, such as running a hook between the fork+exec stage,
       or closing all open file descriptors.

   Due to the problems mentioned with each of these, libvirt driver code must
   not use any of the above APIs. Historically libvirt provided a higher
   level API known as virExec. This was wrapper around fork+exec, in a
   similar style to posix_spawn, but with a few more features.

   This wrapper still suffered from a number of problems. Handling command
   cleanup via waitpid() is overly complex & error prone for most usage.
   Building up the argv[] + env[] string arrays is quite cumbersome and error
   prone, particularly wrt memory leak / OOM handling.

The libvirt command execution API

   There is now a high level API that provides a safe and flexible way to
   spawn commands, which prevents the most common errors & is easy to code
   against. This code is provided in the src/util/command.h header which can
   be imported using #include "command.h"

  Defining commands in libvirt

   The first step is to declare what command is to be executed. The command
   name can be either a fully qualified path, or a bare command name. In the
   latter case it will be resolved wrt the $PATH environment variable.

       virCommandPtr cmd = virCommandNew("/usr/bin/dnsmasq");


   There is no need to check for allocation failure after virCommandNew. This
   will be detected and reported at a later time.

  Adding arguments to the command

   There are a number of APIs for adding arguments to a command. To add a
   direct string arg

       virCommandAddArg(cmd, "-strict-order");


   If an argument takes an attached value of the form -arg=val, then this can
   be done using

       virCommandAddArgPair(cmd, "--conf-file", "/etc/dnsmasq.conf");


   To add a entire NULL terminated array of arguments in one go

       const char *const args[] = {
           "--strict-order", "--except-interface",
           "lo", "--domain", "localdomain", NULL
       };
       virCommandAddArgSet(cmd, args);


   This latter option can also be used at time of initial construction of the
   virCommandPtr object

       const char *const args[] = {
           "--strict-order", "--except-interface",
           "lo", "--domain", "localdomain", NULL
       };
       virCommandPtr cmd = virCommandNewArgs(cmd, args);


  Setting up the environment

   By default a command will inherit all environment variables from the
   current process. Generally this is not desirable and a customized
   environment will be more suitable. Any customization done via the
   following APIs will prevent inheritance of any existing environment
   variables unless explicitly allowed. The first step is usually to pass
   through a small number of variables from the current process.

       virCommandAddEnvPassCommon(cmd);


   This has now set up a clean environment for the child, passing through
   PATH, LD_PRELOAD, LD_LIBRARY_PATH, HOME, USER, LOGNAME and TMPDIR.
   Furthermore it will explicitly set LC_ALL=C to avoid unexpected
   localization of command output. Further variables can be passed through
   from parent explicitly:

       virCommandAddEnvPass(cmd, "DISPLAY");
       virCommandAddEnvPass(cmd, "XAUTHORITY");


   To define an environment variable in the child with an separate key /
   value:

       virCommandAddEnvPair(cmd, "TERM", "xterm");


   If the key/value pair is pre-formatted in the right format, it can be set
   directly

       virCommandAddEnvString(cmd, "TERM=xterm");


  Miscellaneous other options

   Normally the spawned command will retain the current process as its
   parent. If the current process dies, the child will then (usually) be
   terminated too. If this cleanup is not desired, then the command should be
   marked as daemonized:

       virCommandDaemonize(cmd);


   When daemonizing a command, the PID visible from the caller will be that
   of the intermediate process, not the actual damonized command. If the PID
   of the real command is required then a pidfile can be requested

       virCommandSetPidFile(cmd, "/var/run/dnsmasq.pid");


   This PID file is guaranteed to be written before the intermediate process
   exits.

  Reducing command privileges

   Normally a command will inherit all privileges of the current process. To
   restrict what a command can do, it is possible to request that all its
   capabilities are cleared. With this done it will only be able to access
   resources for which it has explicit DAC permissions

       virCommandClearCaps(cmd);


  Managing file handles

   To prevent unintended resource leaks to child processes, all open file
   handles will be closed in the child, and its stdin/out/err all connected
   to /dev/null. It is possible to allow an open file handle to be passed
   into the child:

       virCommandPreserveFD(cmd, 5);


   With this file descriptor 5 in the current process remains open as file
   descriptor 5 in the child. For stdin/out/err it is usually neccessary to
   map a file handle. To attach file descriptor 7 in the current process to
   stdin in the child:

       virCommandSetInputFD(cmd, 7);


   Equivalently to redirect stdout or stderr in the child, pass in a pointer
   to the desired handle

       int outfd = open("out.log", "w+");
       int errfd = open("err.log", "w+");
       virCommandSetOutputFD(cmd, &outfd);
       virCommandSetErrorFD(cmd, &errfd);


   Alternatively it is possible to request that a pipe be created to fetch
   stdout/err in the parent, by initializing the FD to -1.

       int outfd = -1;
       int errfd = -1
       virCommandSetOutputFD(cmd, &outfd);
       virCommandSetErrorFD(cmd, &errfd);


   Once the command is running, outfd and errfd will be initialized with
   valid file handles that can be read from.

  Feeding & capturing strings to/from the child

   Often dealing with file handles for stdin/out/err is unneccessarily
   complex. It is possible to specify a string buffer to act as the data
   source for the child's stdin

       const char *input = "Hello World\n";
       virCommandSetInputBuffer(cmd, input);


   Similarly it is possible to request that the child's stdout/err be
   redirected into a string buffer

       char *output = NULL, *errors = NULL;
       virCommandSetOutputBuffer(cmd, &output);
       virCommandSetErrorBuffer(cmd, &errors);


   Once the command has finished executing, these buffers will contain the
   output. It is the callers responsibility to free these buffers.

  Running commands synchronously

   For most commands, the desired behaviour is to spawn the command, wait for
   it to complete & exit and then check that its exit status is zero.

       if (virCommandRun(cmd, NULL) < 0)
          return -1;


   Note: if the command has been daemonized this will only block & wait for
   the intermediate process, not the real command. virCommandRun will report
   on any errors that have occured upon this point with all previous API
   calls. If the command fails to run, or exits with non-zero status an error
   will be reported via normal libvirt error infrastructure. If a non-zero
   exit status can represent a success condition, it is possible to request
   the exit status and perform that check manually instead of letting
   virCommandRun raise the error

       int status;
       if (virCommandRun(cmd, &status) < 0)
          return -1;

       if (WEXITSTATUS(status) ...) {
         ...do stuff...
       }


  Running commands asynchronously

   In certain complex scenarios, particularly special I/O handling is
   required for the child's stdin/err/out it will be neccessary to run the
   command asynchronously and wait for completion separately.

       pid_t pid;
       if (virCommandRunAsync(cmd, &pid) < 0)
          return -1;

       ... do something while pid is running ...

       int status;
       if (virCommandWait(cmd, &status) < 0)
          return -1;

       if (WEXITSTATUS(status)...) {
          ..do stuff..
       }


   As with virCommandRun, the status arg for virCommandWait can be omitted,
   in which case it will validate that exit status is zero and raise an error
   if not.

  Releasing resources

   Once the command has been executed, or if execution has been abandoned, it
   is neccessary to release resources associated with the virCommandPtr
   object. This is done with:

       virCommandFree(cmd);


   There is no need to check if cmd is NULL before calling virCommandFree.
   This scenario is handled automatically. If the command is still running,
   it will be forcably killed and cleaned up (via waitpid).

Complete examples

   This shows a complete example usage of the APIs roughly using the libvirt
   source src/util/hooks.c

     int runhook(const char *drvstr, const char *id,
                 const char *opstr, const char *subopstr,
                 const char *extra) {
       int ret;
       char *path;
       virCommandPtr cmd;

       ret = virBuildPath(&path, LIBVIRT_HOOK_DIR, drvstr);
       if ((ret < 0) || (path == NULL)) {
           virHookReportError(VIR_ERR_INTERNAL_ERROR,
                              _("Failed to build path for %s hook"),
                              drvstr);
           return -1;
       }

       cmd = virCommandNew(path);
       VIR_FREE(path);

       virCommandAddEnvPassCommon(cmd);

       virCommandAddArg(cmd, id);
       virCommandAddArg(cmd, opstr);
       virCommandAddArg(cmd, subopstr);
       virCommandAddArg(cmd, extra);

       virCommandSetInputBuffer(cmd, input);

       ret = virCommandRun(cmd, NULL);

       virCommandFree(cmd);

       return ret;


   In this example, the command is being run synchronously. A pre-formatted
   string is being fed to the command as its stdin. The command takes four
   arguments, and has a minimal set of environment variables passed down. In
   this example, the code does not require any error checking. All errors are
   reported by the virCommandRun method, and the exit status from this is
   returned to the caller to handle as desired.