Recovery when disk_full_action=HALT

[Steve Grubb]

On Thursday, April 16, 2015 08:29:23 AM Andrew Ruch wrote:
> Hello,
> 
> We have a RHEL6 system with the disk_full_action set to HALT. I'm
> working on procedures for what to do if this case occurs. When the log
> partition fills up, the system shuts down. However, the system will
> not boot after this because as soon as auditd tries to start, the
> system immediately shuts down again. What are the options for
> recovering after this happens? I've come up with two:

Normally, I would think that system maintenance for a situation like this is 
to boot the computer into Single User Mode. You should have switched the 
system over to using sulogin as the shell for single user mode. This way its 
password protected. Then once in, do what you need to archive and make room 
again.


> 1) Stop the boot process at grub and disable audit by adding a kernel
> parameter 'audit=0'.

If you don't use single user mode, then there is the risk of someone doing 
something while the audit system can't record anything. You probably don't 
want that possibility either.


> 2) If grub timeout is 0, use a live CD to access the audit partition.

This would work also, but Single User Mode is so much easier. :-)
 
 
> I'm sure there are some variations on option 1 using an interactive
> boot. Are there any other options I missed, especially if grub timeout
> has been set to 0?

I wouldn't set it to 0. You can make it short like 2 or 3. But you need to be 
able to get into the editor to tell it 'S' for single user mode.

-Steve