Tech Mavens

Home

Build Your Own Incident Response Toolkit

by Ross Oliver

It's 4am when the cell phone rings.  Your UNIX server has been hacked. You fire up your PC, dial in, and log on.  You find some files that don't belong.  The format of the "ps" output is different than it used to be.  Someone has messed with your system, and you need to clean it up, but how to do it without a complete wipe and re-install?  And how can you find out if the invaders have breached more of your  systems?

The most important part of the toolkit is a set of known-good system utilities. Intruders often install their own versions of system utilities such as ps and netstat that are modified to hide the intruder's presence.  Your toolkit should contain the basic operating system tools and utilities to examine the system, determine what changes are needed, and implement the repairs.  Here is a list of utilities that should be included.

Basic command set
sh
ls
lsattr (Linux)
cat
more


File manipulation
mv
rm
mkdir
rmdir
find
vi
chmod
chgrp
setfacl (Solaris)
chattr (Linux)


System information and administration
ps
top
netstat
ifconfig
route
who
finger
passwd
su
fsck
mount
telinit
halt
reboot
shutdown


Networking and communications
telnet
rlogin
rsh
ftp
minicom (Linux)
ypbind
ypwhich
ypcat

Key deamons
inetd
syslogd
crond
mgetty (Linux)
getty (Solaris)

Investigation tools
traceroute
snoop (Solaris)
tcpdump (Linux)
lsof
nslookup
whois
sum
md5
od
strings
nc

In addition to system binaries, your toolkit should also have backup copies of important system configuration files:

/etc/passwd
/etc/shadow
/etc/hosts
/etc/inittab
/etc/vfstab (Solaris)
/etc/fstab (Linux)
/etc/inetd.conf
/etc/syslog.conf

Also include configuration files for any services, such as DNS, NIS, NFS, DHCP, sendmail, etc.

In addition to system utilities, you will need listings of all files that are supposed to be on the system, including ownership, permissions, sizes, and checksums/MD5 hash values. For Linux systems, this data is maintained in the RPM database for all files installed by RPM.  For Solaris systems, you will have to generate this data yourself:

rpm -V -a

This command will check the integrity of all files in all RPM packages against the RPM database (stored in /var/lib/rpm), and output a list of any discrepancies in this format:

S.5....T c /etc/hosts.allow
S.5....T c /etc/hosts.deny
S.5....T c /etc/services
S.5....T c /etc/localtime
S.5....T c /etc/nsswitch.conf

In this example, rpm reported that all 5 files differ in size (S), MD5 hash (5), and modification time (T).  Since these are configuration files, it is normal for these to be different from their original installed state.  However, you should verify their contents to be sure they contain the correct data for your configuration.  On the other hand:

..5..... /bin/ps

Any discrepancies with system binaries should be highly suspicious.

NOTE: the verify option of rpm will report discrepancies of only those files that were installed using RPM.  It will not report any changes to files installed using other methods.

Solaris does not have a built-in MD5 hash-generating utility that comes standard with Linux. However, it is very simple to build one by compiling the sample MD5 implementation code contained in the MD5 RFC, RFC1321.  MD5 hashes are much more difficult to forge than the simpler CRC checksums used by the sum utility, so MD5 hashes are a much better indicator of file modifications.

Solaris does not maintain an extensive database of file information as RPM does for Linux. So, you must build your own file lists for your toolkit.  This command line will produce a list of every file, directory, and special device on a Solaris system:

find / -ls | grep -v '/proc/' | sort -k 1.66

The -ls option to find causes it to write its output in a format corresponding to "ls -l" This format includes all important information about a file, including its ownership, permissions, size, and modification time. The grep command excludes the /proc directory, containing page files for currently running processes.  Since these change constantly, they should be excluded from the list. The sort command sorts the list on the file's path name, so the order of files is reproducible. Save the output of this command in your toolkit to give you your file baseline data. Then run it again on any suspect machine, and run a diff on the two data sets to identify any differences.

The command to generate either checksums or MD5 hash values is more complex:

find / -type f -print | grep -v '^/proc/' \
     | xargs -n 1 md5 \
     | sed -e 's/^MD5 (//' -e 's/)//' -e 's/ = / /' \
     | sort

This command will produce a list of files and MD5 hashes that can be saved and compared with the same data from a suspect machine.

Once you have assembled your utilities and data, burn them on to a CD-ROM. Then you will have a complete toolkit ready to pop into any suspect machine.