[libvirt] [PATCH 00/12] Structured syslog (Lumberjack/CEE) support

[Miloslav Trmač]

This patch set adds structured syslog support to libvirt.

In short, the idea is to embed JSON in a syslog record.  This format
is used in a new log output type called "syslog/json".

Example "syslog/json" record (line-wrapped for readability):

Sep 20 19:59:40 kulicka libvirt: @cee: {"msg":"libvirt version: 0.10.1",
	"category":"../../src/test/test_driver.c","priority":"info",
	"funcname":"virLogVMessage","line":769,
	"timestamp":"2012-09-20 17:59:40.853+0000"}

compared with existing "syslog" record format:

Sep 20 20:02:24 kulicka libvirt: 5896: info : libvirt version: 0.10.1


This JSON embedding format is defined by the Lumberjack project led
by the major Open Source syslog implementations
(https://fedorahosted.org/lumberjack/).  The format is also a base
for a future MITRE CEE standard.  Some of the fields are specified
at https://fedorahosted.org/lumberjack/wiki/FieldList , but applications
are free to use whatever fields make sense for them.

The JSON structure makes easy to process logs by programs without writing
code that understands the specific format used by each application,
or using unreliable regexps (https://fedorahosted.org/ceelog/ is a
companion project that allows processing such logs).


Current systems will store and these records unmodified and present them
to users unmodified, but tools (and probably /var/log/messages) will
be able to show only the "msg" field; this allows us to add more detailed
information without worsening the user experience (the example above
already shows added information that is currently being thrown away when
logging to syslog).


The patch set, aside from building the required infrastructure, has two
main components:

* Add the "syslog/json" output type, and use it to record all information
  passed to virLogMessage.  The example result is shown above.  This primarily
  makes it easier to parse the message.

* Allow callers of virLogMessage to add arbitrary additional JSON fields,
  and use it in virRaiseErrorFull and its callers.  Because the additional
  fields for errors include virErrorDomain and virErrorNumber, this allows
  _programs_ to _automatically_ detect the event and react to it, perhaps
  alerting the user about a situation that requires immediate attention,
  or automatically suggesting a solution.

  This has not been possible until now because the current logs contain
  only a translated error message, so the log record doesn't contain
  anything stable that could be detected by a program.

  It is assumed that other parts of libvirt may add more event-specific
  JSON fields in the future, based on demand by applications that process
  the logs; I have, however, no immediate plans to add more.

Miloslav Trmač (12):
  Drop unused return value of virLogOutputFunc
  Add JSON API for preallocated objects.
  Split yajl_gen allocation into a separate function.
  Split adding object properties from virJSONValueToStringOne
  Add a JSON generator API.
  Pass the "raw" log message to each virLogOutputFunc
  Split priority conversion from virLogOutputToSyslog
  Add "syslog/json" log output format.
  Add a JSON properties parameter to virLog{,V}Message
  Add a JSON properties parameter to virLogOutputFunc
  Add detailed error information to JSON syslog
  Add a test for the new error reporting mechanism.

 docs/logging.html.in     |   3 +
 src/libvirt_private.syms |   4 +
 src/uml/uml_conf.c       |   2 +-
 src/util/json.c          | 232 +++++++++++++++++++++++++++++++++++++++++-----
 src/util/json.h          |  26 ++++++
 src/util/logging.c       | 233 ++++++++++++++++++++++++++++++++++-------------
 src/util/logging.h       |  35 +++----
 src/util/viraudit.c      |   4 +-
 src/util/virterror.c     |  32 ++++++-
 tests/Makefile.am        |   6 +-
 tests/errorjsontest.c    | 211 ++++++++++++++++++++++++++++++++++++++++++
 tests/testutils.c        |   7 +-
 12 files changed, 684 insertions(+), 111 deletions(-)
 create mode 100644 tests/errorjsontest.c

-- 
1.7.11.4