Security testing: need for a security policy, and a security-critical package process

[Adam Williamson]

Hi, everyone. I'm sending this email as a result of a discussion in the
Fedora QA meeting this morning. You can find a log of the meeting here:

http://meetbot.fedoraproject.org/fedora-meeting/2009-11-23/fedora-meeting.2009-11-23-16.00.log.html

the discussion takes place from 16:14:09 onwards.

We discussed the recent PackageKit kerfuffle from a QA perspective,
which means we talked about how we could have meaningful security
testing. We came to some basic conclusions about this which require
co-ordination with the security and development groups.

We can't do any meaningful security testing without knowing exactly what
we should be testing for, in which packages. I believe Seth Vidal's
upcoming proposal for covering 'major changes' may touch on this, but I
doubt they'll cover exactly the same ground.

So, if we are to have meaningful security testing in future releases -
which QA believes would be a good thing - we need the project to define
a security policy. We believe there's a genuine need for this anyway, as
the introduction and widespread adoption of PolicyKit will likely lead
to much more complex and significant potential changes in security
posture than any previous change.

It's not QA's role to define exactly what the security policy should
look like or what it should cover, but from the point of view of
testing, what we really need are concrete requirements. The policy does
not have to be immediately comprehensive - try and cover every possible
security-related issue - to be valuable. Something as simple as spot's
proposed list of things an unprivileged user must not be able to do -
http://spot.livejournal.com/312216.html - would serve a valuable purpose
here.

The second thing QA would require, aside from a policy with concrete and
testable requirements, is a list of security-sensitive components to
test. Obviously we couldn't test every package in the entire
distribution for compliance with even such a simple list as spot's, and
it would be a waste of time to try. 

Focussing on the relatively simple issues for now, we believe it would
be reasonably simple to generate a list of all packages in the
distribution that attempt privilege escalation. We believe this would be
a list of packages that contain suid binaries, that invoke su, sudo or
consolehelper, or that contain PolicyKit policies. This list of packages
would be what the QA team would test with regard to the security policy.
We also believe there ought to be a process for maintaining this list,
and additions to the packaging guidelines for any new package which
would be on this list or any existing package for which a proposed
change would add it to this list. We could also hook AutoQA into this
process, to run additional tests on security-sensitive packages or alert
us when a package change was submitted which added security-sensitive
elements to an existing package.

Will Woods has indicated he is prepared to help work on the tools
necessary to generate the security-sensitive package list. The QA group
as a whole is happy to contribute what input we can to any discussion of
a general security policy. Mostly, we wanted to make it clear that we
believe security testing would be of benefit to the distribution, but
these things need to be in place before any meaningful such testing
could be done.

-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net