authentication hangs when disk > 80% full

[Alfred Hovdestad]

The problem is that auditd "suspends" activities when /var falls below 
20% free.



 From /etc/audit/audit.conf

output {
         mode            = bin;
         num-files       = 4;
         file-size       = 20M;
         file-name       = "/var/log/audit.d/bin";
         notify          = "/usr/sbin/audbin -S /var/log/audit.d/save.%u 
-C -T 20
%";
#  AUDBIN THRESHOLDS:
#       The above notify will cause auditd to enter 'suspend' mode when
#       free space on the /var/ filesystem falls below 20%.




You can change the activity that auditd takes when the disk with /var 
reaches 80%, or you can turn auditd off:

service audit stop
chkconfig audit off


    Alfred Hovdestad, RHCE
    University of Saskatchewan



Stephen Jascourt wrote:
> I discovered that PAM authentication hangs in at least 4 different 
> kernels of Enterprise Linux, including one installed last week, when the 
> partition containing /var is more than 80% full (e.g. less than 20% 
> available).
> 
> The following appears in the /var/log/messages file:
> audbin[2673]: saving binary audit log /var/log/audit.d/bin.2
> audbin[2673]: threshold 20.00 exceeded for filesystem /var/log/audit.d/. 
> - free blocks down to 16.24%
> auditd[2130]: Notify command /usr/sbin/audbin -S 
> /var/log/audit.d/save.%u -C -T 20% exited with status 1
> auditd[2130]: output error
> auditd[2130]: output error
> auditd[2130]: output error; suspending execution
> 
> [numbers in brackets are probably process id numbers or something, they 
> change each time; all other parts of the message repeat on each boot 
> attempt]
> 
> I don't know what auditd does or what it's connection to authentication 
> is. What I do know is that when this happens at a time when you have 
> open xterm windows, you can still execute commands and use the system, 
> but you cannot ssh or sftp or otherwise enter from outside - you can 
> connect, but upon entering your password, the connection will hang. 
> Similarly, trying to change user from an open window or go to root via 
> "su" also hangs upon entering password. Other open windows and processes 
> will be unaffected. I don't have any cron scripts, but if they involve 
> authentication, they probably would also hang. However, once you end 
> your session and have no open windows into the machine, you are locked 
> out entirely. A regular boot proceeds normally aside from the above 
> messages in the message log, but it does not startx, the screen with the 
> color login box (control-alt-F7) does not appear, and logging in from 
> the text login screen (such as the screen on control-alt-F1) hangs upon 
> entering password, as do attempts to login from outside. Same result for 
> logging in as root.
> 
> The problem is resolved by reducing disk usage under 80% for the 
> partition containing /var. You need to find a bootable disk, mount your 
> hard drives, set permissions to write, and delete some files. After 
> getting under 80%, everything is fine, shut down and do a normal boot 
> and you're back in business.
> 
> I am not a system manager or IT person, nor am I a linux expert, just a 
> linux user who needed to access his work PC to do work when our local 
> linux experts are on their holiday vacation. I could not find any web 
> messages or other documentation on this problem, so apparently it has 
> not been documented before or it is documented in a place that does not 
> show up in web searches.
> 
> The problem had been happening intermittently for a while and would 
> clear up upon rebooting. I guess I may have been near 80% and booting 
> cleaned up all the temporary files bringing me under 80%. I didn't 
> troubleshoot the problem, didn't have any clues, so I didn't know about 
> the magic 80% threshold. Then I downloaded some large data files and got 
> locked out until someone suggested I use a bootable CD, mount my hard 
> drives, and someone else suggested looking in /var/log/messages, where I 
> found the above lines and tried deleting files. Once I got under 80%, it 
> was fine.
> 
> There may be some parameter in some configuration file that sets the 20% 
> threshold for available disk space required, but I don't know where that 
> would be. If so, I could lower it to say 1% or something small. Another 
> solution is when configuring a new system, you could partition /var by 
> itself with only the amount of space it will need and run a daily script 
> to monitor space in /var and send a warning message when space available 
> starts getting close to 20%. But it would be much better if folks who 
> work on linux source code could just fix the problem.
> 
> Stephen Jascourt
> UCAR/COMET meteorologist at National Weather Service Headquarters
> 
> 
> 
> 
>