How to exercise the lwatch code:

1) Build lwatch

% cd sssd
% autoreconf -i -f
% make
% cd lwatch/src

2) Have lwatch tell you what it thinks are log files available to be watched
under /var/log

% sudo ./lwatch -f -r /var/log

The -f arg tells lwatch to "find" log files, the -r says to do it
recursively. Note, the list of file omits files it thinks are rotated backups
and only reports the primary log file.

3) Create default directory for lwatch sqlite database (needed for watching)

% sudo mkdir /var/lib/lwatch
% sudo chmod 777 /var/lib/lwatch

4) Run lwatch and ask it to watch ./tmp/tmp.log
 Note, the directory ./tmp does not need to exist, lwatch knows how to watch for
 the creation or destruction of any ancestor in the path and adjust accordingly.

% ./lwatch tmp/tmp.log

5) In a separate shell (shell2) create the ./tmp directory

% mkdir ./tmp

Notice lwatch reports it discovered a directory previously absent, but essential
to one of its watch targets came into existence.

6) In shell2 create the log file and put some data in it.

% date >> tmp/tmp.log

Notice that lwatch discovered the file it was watching came into existence.
Periodically lwatch wakes up and asks if any file its watching needs have its
newly added data collected and sent to the central store, this is called a
"reap".

Notice that lwatch will report something like this:

REAP "/home/jdennis/src/sssd/sssd/lwatch/src/tmp/tmp.log" [0:29]

the numbers in the brackets are the begininng and end offsets in the file which
need to be reaped.

7) In shell2 append some more data to the log file

% date >> tmp/tmp.log

After lwatch wakes up to process it's reaps you see something like this:

REAP "/home/jdennis/src/sssd/sssd/lwatch/src/tmp/tmp.log" [29:58]

Notice the file offsets indicate only the newly added data needs to be collected
since the earlier data was already reaped.

8) Observe how lwatch behaves when logrotate rotates the log file.
Create a logrotate.conf file

% cat << EOF >> logrotate.conf
/home/jdennis/src/sssd/sssd/lwatch/src/tmp/tmp.log {
    missingok
    rotate 5
    size=25
}
EOF

note, you must use the full path name of the tmp.log file and adjust it for your
system. This logrotate files says keep 5 rotated backups and rotate the file
whenever it's size grows beyond 25 bytes.

Now invoke logrotate to rotate the file:

% logrotate logrotate.conf

Notice that lwatch recognized the file was renamed to tmp.log.1. It is also
important to notice that lwatch is now watching BOTH tmp.log and tmp.log.1. Why?
Because if a process had tmp.log open for writing before the rename and didn't
close and reopen the log file then it will continue to write into what is now
tmp.log.1, it might also open a new tmp.log, therefore we must watch both.

Now create and add data to the new tmp.log file

% date >> tmp/tmp.log

After waking up to process any pending reaps you'll see:

REAP "/home/jdennis/src/sssd/sssd/lwatch/src/tmp/tmp.log" [0:29]

Notice how the file offsets have been reset, lwatch knows this is now a
different version of the same file and has assigned it a new version number.
Everytime lwatch believes the file contents have been replaced it assigns a new
version number to the watched file.

9) Have lwatch dump the contents of its persistent database and observe some
important facets. The -l command line arg of lwatch "lists" the watch database.

% ./lwatch -l

path: /home/jdennis/src/sssd/sssd/lwatch/src/tmp/tmp.log
    flags:             [TARGET,STAT_DATA_VALID]
    version:           2
    reap_time:         2009-07-22T13:00:01Z (1248267601)
    reap_position:     29
    dev:               64770
    inode:             1594525
    mode:              -rw-rw-r-- (0100664)
    uid:gid:           jdennis:jdennis (3596:3596)
    size:              29
    access_time:       2009-07-22T12:59:50Z (1248267590)
    modification time: 2009-07-22T12:59:50Z (1248267590)
    change_time:       2009-07-22T12:59:50Z (1248267590)

    path: /home/jdennis/src/sssd/sssd/lwatch/src/tmp/tmp.log.1
        original path:     /home/jdennis/src/sssd/sssd/lwatch/src/tmp/tmp.log
        flags:             [BACKUP,STAT_DATA_VALID]
        version:           1
        reap_time:         2009-07-22T12:18:06Z (1248265086)
        reap_position:     58
        dev:               64770
        inode:             1594524
        mode:              -rw-rw-r-- (0100664)
        uid:gid:           jdennis:jdennis (3596:3596)
        size:              58
        access_time:       2009-07-22T12:17:39Z (1248265059)
        modification time: 2009-07-22T12:18:04Z (1248265084)
        change_time:       2009-07-22T12:18:04Z (1248265084)

Notice how there are two files being watched after the rotate and now tmp.log.1
is a child of tmp.log (because it's indented). This indicates there is a
derivative relationship between tmp.log and tmp.log.1. Also notice how the
version numbers of the two files are different. What is now tmp.log.1 started as
tmp.log and was assigned version 1. After the rotation tmp.log is assigned the
next version number. Also notice how tmp.log.1 is marked as a BACKUP of tmp.log
(the BACKUP flag in the flags and the presence of the "original path" value).

10) Observe how lwatch can resynchronize itself if it hasn't been running and
the filesystem changed.

Stop the running lwatch process by typing control-C.

Now add some new data to tmp.log while lwatch is NOT running.

% date >> tmp/tmp.log

Start the lwatch process again.

% ./lwatch ./tmp/tmp.log

After waking up on it's reap schedule you'll see something like this:

REAP "/home/jdennis/src/sssd/sssd/lwatch/src/tmp/tmp.log" [29:58]

This indicates that lwatch recognized in it's absense that bytes 29-58 were
appended to the file and needs to be collected.