Ack! I've been rooted...

[Rick Stevens]

On Thu, 2007-02-01 at 17:58 -0600, Chris Mohler wrote:
> Well, through no one's fault but my own our file server has been compromised.
> 
> It looks like the SHV5 kit.  I plan a reformat/reinstall tomorrow and
> I was wondering if anyone had advice.  I discovered that some of the
> coreutils had been replaced with compromised versions, so I (stupidly)
> downloaded the coreutils RPM, then did 'rpm -ev coreutils' and tried
> 'rpm --Uvh coreutils'.  Should have researched that a bit, because (as
> root) I don't have permission to remove/rename the hacked binaries!

The standard rootkit thing is to "chattr +us" the files so you can't
delete them in a standard way.  To fix it, boot off the first CD in
rescue mode and let the system mount your / volume.  Then:

	cd /mnt/sysimage
	chattr -R -us sbin/*
	chattr -R -us bin/*
	mount (path to your /usr partition) usr
	cd usr
	chattr -R -us sbin/*
	chattr -R -us bin/*

That should allow you to delete the hacked versions of the various
files.

Next, take a REAL CAREFUL look at the /mnt/sysimage/etc/rc.d directory
tree and make SURE there's no items in there that start up the hidden
binaries.  Often, those binaries are buried in the /dev directory and
are given non-displayable names (non-graphic characters, names that
start with ".", etc.)

> Oops. For the time being, I've (physically) removed the server's
> network connection.

Good call!

> So - the plan:
> 1. telinit 1
> 2. try to reinstall coreutils
> 3. telinit 3

I suggest doing what I mentioned above.  Then reboot the machine in
single user mode (append "single" to the "kernel" line in grub),
reinstall the coreutils and THEN "telinit 3" if you must.  There's
really no need to telinit 3--you can get the network up with a simple
"service network start" from single user mode.

> 4. rsync the last week's worth of data to another machine
> 5. reformat/reinstall
> 6. create new home dirs
> 7. rsync the data back - do a recursive chown/chmod
> 8. run rkhunter
> 
> Any thoughts on this plan of attack are welcome.

You have mine.

> And of course the moral of all of this is UPDATE and DON'T RUN
> UNNEEDED WEB SERVICES.  This happened on a FC2 server (I know ;) ),
> and possibly via the SWAT or phpMyAdmin web interfaces.

Evil!  I always iptable the hell out a box with any form of outside
management stuff.  Also, once you've got the machine up and BEFORE you
plug in the net cable:

1. Take a good hard look at what's running with "ps ax".  Turn off
ANYTHING you don't need (sendmail, nfs, snmpd, samba, etc.).

2. Do a "netstat -pan" and look at what network ports are active and
make sure you know what they are and that they should be there.

3. Run an nmap against the primary IP address of the machine (not
127.0.0.1, but the one that ties it to the network) and make sure there
aren't any open ports you don't know about.

4. Set up iptables and firewall the hell out of the machine.  Only allow
webmin access to the machines you use to manage it.  Block ssh except
from your management machines.  Set up tripwire.  You get the idea.

As someone once observed, "The Internet is rough.  Wear a cup!"

----------------------------------------------------------------------
- Rick Stevens, Senior Systems Engineer     rstevens at vitalstream.com -
- VitalStream, Inc.                       http://www.vitalstream.com -
-                                                                    -
-        Hard work has a future payoff. Laziness pays off now.       -
----------------------------------------------------------------------