[K12OSN] Random system crashes: Linux gurus, what would you do?

ssanders at coin.org ssanders at coin.org
Wed Dec 21 06:03:45 UTC 2005

On Tue, 2005-12-20 at 20:51 -0800, Carl Keil wrote:

> My k12ltsp 4.2.1 server has been frozen every night when I wake up in 
> the morning. 

If this were a crucial machine, you should take it offline, clone the
drives and work on a copy. If you suspect it has been rooted, the only
ultimate solution may be to copy data off and reinstall, but I know
about recreating a custom setup that has grown over time. I can't advise
you so much on the specific crash messages, but there are several things
you can do to minimize outside intrusion. In no particular order: 

Change the root password of course. This may not be a complete solution,
because if you have a trojaned or vulnerable process, he can exploit
that again.

Block that IP, if you have attempts from one IP. Contact the church, and
see if there is a timeframe and check if they can see who might have
access to computers there. Configure an upstream firewall if present to
blackhole any connections, or edit /etc/hosts.deny or learn more about
netfilter/iptables to block (I need to take my own advice on iptables).

Check your users. Check who is logged in now with the simple w command.
Sophisticated intruders that change the logs can be hard to find, but
don't overlook the obvious. There can be users without home directories,
so ls /home won't give you a complete picture, but try it quickly
anyway. See any users you don't recognize? Check to see when they last
logged in with last <username>

Dig further into all users with cat /etc/passwd Wow, a pretty big list.
Lots of those users are system-supplied and cannot login anyway. You
could check to see when they have logged in with the last command, but
first it may be easier to narrow it down a bit. By piping a few commands
together, you can get a more realistic list:

  cat /etc/passwd |grep "/home" |cut -d: -f1

See what is listening and talking to the outside. 
nmap localhost -p 1- 
will check all 65k ports and be fast on localhost. If someone was able
to get netcat installed, the machine could be connecting a root shell
out to a netcat listener, usually bypassing any firewall rules. Play
around with the netstat command options and see if there is any errant
communication. You are using cron, check for any strange entries that
might be starting something that you didn't intend.

Force your local users' passwords to expire tomorrow with:

chage -M 1 username

Watch and see who all changes them when the system prompts them upon
their next login. A smart attacker wouldn't touch this with a ten foot
pole, but many attackers are not smart.

These are just some basic system and communication things to check.
There are many lower-level system problems that could be hard to find.
If you reinstall, consider something like Tripwire, so you can later
verify system files.

Hope this helps!

More information about the K12OSN mailing list