[RFC] Testcase Scenarios for Auditfs Code

[Stephen Smalley]

On Tue, 2005-04-26 at 10:38 -0500, Loulwa F Salem wrote:
> These are the testcases I am writing to test the code Tim is providing
> for the audit filesystem. The testcases will be included in LTP. 
> These cases are meant to cover the CAPP requirements. there is some
> FVT testing covered as well, but that still needs to be expanded to
> test the permissions and boundary cases (watch fields lengths, invalid
> fields ... etc). 
> Please give feedback if you can think of any scenarios that I have not
> considered, or modifications that I need to implement. 

I just ran through these testcases by hand (just using normal utilities
from a shell) using Tim's #7U3 patch against 2.6.12-rc2-mm1.  It might
be helpful to distinguish which testcases would yield different
behaviors if one were to try using the inode-based syscall filters
rather than the name-based watches and think about how you might explain
to users when they should use one or the other method and how they
differ.  Some observations:

> Test 1 : Access watched file by opening it (file already exists) 
>                Expected: 1 watch records for file access/open 

Should be equivalent to putting an inode-based syscall filter on the
file, aside from the specific output fields.
      
> Test 2 : Create a watched file (file created after watch is inserted) 
>                Expected: 1 watch records for open() with regards to
> file creation 

Only makes sense for the name-based watches.  I think that the closest
equivalent for inode-based syscall filter would be to monitor file
creation syscalls on the parent directory.

> Test 3 : Create hard link to a watched file and then access it by
> opening it for read 
>                Expected: 1 watch record for open() on original file

This also generates a record for the link(2) call on the watched file.
That's what I would expect, but you didn't list it above.  Note that an
inode-based syscall filter would yield the same result for this
particular test, but the inode-based filter would persist across reboots
(if loaded by auditd on each boot) unlike the watch (which was only on
the original file path, unless you explicitly add the new link name to
the watch list).  Hence, watch-based auditing differs from inode-based
filtering in its persistence of tracking of hard links.  And actually,
the watch-based auditing of access via the hard link would also cease if
the incore inode is evicted under memory pressure and then brought
incore again later via the link name, right?  So how do you explain to
users that the watch _may_ be triggered by subsequent accesses via the
hard link (until reboot) but can be lost at any time due to memory
pressure?  Is that the behaviour you want?

> Test 4 : Create hard link to an unwatched file, watch file, then
> access hard link by opening it for read 
>                 Expected: 1 watch record for open() on original file 

Should be equivalent to an inode-based filter on the file, as the hard
link will share the same inode.

> Test 5 : Create hard link to watched file, delete watched file, then
> access hard link by opening it for read 
>                 Expected: No watch record 

This will also generate records for the link(2) call and the unlink(2)
call (and if using rm to delete, it will also generate one for access(2)
to check file existence and permissions prior to unlinking).  Note that
an inode-based syscall filter would generate an audit record upon the
access to the hard link since the inode is unchanged.  Which behavior
will most users want?

> Test 6 : Recreate a watched file that has been deleted previously 
>                 Expected: 1 watch record for unlink() with regards to
> file deletion 
>                 Expected: 1 watch record for open() with regards to
> file recreation 

You said "deleted previously" but included the unlink in your expected
output.  Regardless, if I include the deletion as part of the operation,
I get the expected results as well as the access(2) call due to using rm
(1).  Naturally, this kind of test (creation of a file at a given name)
is specific to the watches, but inode-based syscall filter could be used
on file creation syscalls on the parent directory.

> Test 7 : Recreate a watched file that has been deleted previously 
>                (creating hardlink to it first, so on recreation we are
> certain we get new inode number) 
>                Expected: 1 watch record for unlink() with regards to
> file deletion 
>                Expected: 1 watch record for open() with regards to
> file recreation 

You included the unlink in your expected output, but not the link(2)
call for creating the hard link in the first place.  As before, inode-
based syscall filter could only be used on the parent directory.

> Test 8 : Delete a watched file 
>               Expected: 1 watch record for file deletion 

Plus access(2) if using rm(1) to perform the deletion.

> Test 10 : copy a watched file to unwatched name (copy out) - access
> through new name 
>                Expected: 1 watch record for original file access/open 
>                Expected: No record for accessing through new name  

Should be equivalent to an inode-based syscall filter on the file.

> Test 11 : copy a file to a watched name (copy in) - access through new
> name 
>                Expected: 1 record for file creation in new name 
>                Expected: 1 record for new file access/open 

Distinctive to the watch-based approach; inode-based syscall filter
could catch the file creation on the parent directory, but would require
some kind of userspace assist to set up the filter on the new file.

> Test 12 : Move a watched file out of a watched location and access by
> opening it for read 
>                Expected: 1 watch record for rename() with regards to
> moving out of a watched location.         
>                Expected: No watch record for new file open() 

An inode-based syscall filter would continue auditing on the file.  What
will the users want?  

> Test 13 : Move a watched file into a watched location and access by
> opening it for read 
>                Expected: 1 record for rename() with regards to moving
> out of a watched location.         
>                Expected: 1 watch record for open() in new location 

Should be equivalent to an inode-based syscall filter on the file, since
the inode is unchanged.

> Test 14 : Move a directory having a watched file - access file in new
> location 

An inode-based syscall filter would continue auditing access to the
file.

What concerns me is unclear/unstable semantics and a lack of a clear
subdivision between this mechanism and the inode-based syscall filters:
- Auditing may or may not be preserved on hard links when using the
watches depending on memory pressure, reboots, or whether the watched
name is unlinked; is always preserved for inode-based watches.
- Auditing is never preserved for renames when using the watches; is
always preserved for inode-based watches.
- Auditing is automatically enabled for new files when they are created
in watched locations when using watches; requires userspace modification
to achieve with inode-based watches.
 
-- 
Stephen Smalley <sds at tycho.nsa.gov>
National Security Agency