thanks.txt on my Play Machine

On my SE Linux Play Machine I have a file in the root home directory named thanks.txt_append_only_dont_edit_with_vi which users can append random comments to. It kept slowly growing from the time of Fedora Core 2 to today, here is the text. Any text within brackets is my response to a question.

you can send messages to the owner through this file
should I be able to see dmesg output?
Lon was here
Is this a virtual machine? [at that time it wasn’t, it is now]

nice toy here :)
cool stuff – will you be posting instructions on how to lock down a machine like this? [yes]

Had fun poking around
Impressive stuff, though I’m not exactly a security expert ;)

I guess it’s a bit better than LIDS. I’ll give it a try
Does there even have to be a root user? could it have been a ‘John’ instead with no impact on the fedora system? [the user name was never an issue, changing a Unix system to have “John” map to UID 0 is no big deal]
nice toy…
This is my first look at SElinux, very secure but seems broken from a desktop usability standpoint. Is FC2’s policy to be more liberal than this? [SE Linux has been continually improving]
Out of curiosity are you running exec-shield as well [sometimes yes, sometimes no, depends on the distro]

This machine is a little bit more permissive than the Gentoo machine,
I can actually read the security policy files! [by design, you can look and learn]
Thanx and have a nice day
I was able to coredump bash and read some history enries. see ./coredumptest Is this expected behaviour? kenny @ [you could have just read ~/.bash_history or run the “history” command]
exec-shield what is that? When I ran this command It gives a error: -bash: exec-shield: command not found [exec-shield is a kernel patch to prevent some application exploits which rely on writable and executable memory]
Where are the security policy files? Excelent job here! Thank you for the public root account ;-p

Very interesting.
Russel ! Thank You for work, Thank You for this box. SELinux Rulz ! [s/Russel/Russell/ :-)]
I was able to fill up the filesystem to 100% (/tmp) and I was able to terminate the shells of other root users
[Filling the root fs is a DOS attack, read the MOTD.]
[Killing the shells of other users is expected behavior, they are all using the same account as you!]

The tar program sure gets upset. I untar something that was originally tarred up as UID 1000, and it gets changed to that. Then I try to untar a second portion of the data, and I get all sorts of errors. Had the UID change been blocked, the errors wouldn’t happen when the second tar tries to write to the directories again. Errors look like this:

tar: procps-3.2.1/test/ps/thread-nosort-L/header: Cannot open: No such file or directory
tar: procps-3.2.1/test/ps/thread-nosort-default: Cannot mkdir: No such file or directory
tar: procps-3.2.1/test/ps/thread-nosort-default/setup: Cannot open: No such file or direc

You’re seriously short on RAM. Only about 9 MB are free. Nothing I can view is eating it. Programs are crashing due to lack of memory. [you don’t have permission to see most processes]

can’t wait for fedora core 2. this is one sweet security setup. hopefully a howto will come out, plus maybe a gui for the windows folks.

thanks. you’ve inspired me to install fedora. cool stuff.

Thanks very much for setting this box up. It is a great learning tool

I note that I can’t ping, traceroute or telnet off the box. Is this intentional? Is this part of the lockdown to show me that I can’t do things I expect to be able to do with uid 0? My initial impression is that without those functions it is not very useful to have a system. [in the early days I allowed such things, but they were abused too often]

Have you updated the kernel with the information in this
post? Have you tried whether that might be a real exploitable vulnerability?
Sorry about the formatting of the url. [there are kernel vulnerabilities all the time, I keep updating it to the latest kernel]
Its very interesting. Thank you.
bagus juga pengamanan boxnya. salam dari indonesia
##\n thanks from me too\n##
##/nD’oh’/n thanks from a Windows Luser too/n##
hello althepcman was here
Thanks very much for setting this box up. I’ll try the SELinux on Fedora Core 2.
thank your for your great job, Fedora is great
thanks, from argentina, i really dont like fedora…in fact im a debian or gentoo user…but i think that fedora its kind a cool thing
nice small server with fine security patch. thx for the try-out. greetings from hannover/germany

Thanks from Brazil. I’m studying selinux and ids integration and probably I’m gonna come back here. marciorg at
is it correctly that root can sudo ?

-bash-3.00# ps auxw
root 20860 0.0 0.5 5576 1432 pts/42 Ss+ 07:44 0:00 -bash
root 20910 0.0 0.5 4852 1296 pts/43 Ss+ 08:02 0:00 bash -i
root 21033 0.0 0.5 5092 1436 pts/45 Ss+ 08:29 0:00 -bash
root 21105 0.0 0.5 4860 1460 pts/46 Ss 08:39 0:00 -bash
root 21219 0.0 0.2 2708 756 pts/46 R+ 08:55 0:00 ps auxw
-bash-3.00# sudo -u mysql ps auxw
root 20860 0.0 0.5 5576 1432 pts/42 Ss+ 07:44 0:00 -bash
root 20910 0.0 0.5 4852 1296 pts/43 Ss+ 08:02 0:00 bash -i
root 21033 0.0 0.5 5092 1436 pts/45 Ss+ 08:29 0:00 -bash
root 21105 0.0 0.5 4860 1460 pts/46 Ss 08:39 0:00 -bash
mysql 21220 0.0 0.0 2476 252 pts/46 R+ 08:56 0:00 sesh /bin/ps auxw
mysql 21221 0.0 0.2 3844 752 pts/46 R+ 08:56 0:00 /bin/ps auxw

and is it realy sudo? AFAIK mysqld was started on this system, but sudo -u mysql ps auxw doesn’t show me other mysql processes…
[sudo doesn’t change to the mysqld_t domain…]


i’ve written script: i tried to kill hidden pids from /proc using sudo -u rjc kill -9 $pid.
does rjc has 2 roles ? why i couldn’t kill his shell ?
[rjc has 2 roles, neither of which is user_r, so neither of them has the domain user_t that you can kill]


thx in advance :)

rjc, check this out: /root/ls_rjc_home_:)
[fixed – thanks for that, it was due to a bug in locate]
and don’t forget about sudo plz :)
Thanks for the effort to let us experiment with SELinux/Fedora
Thanks alot for this publicly accessible machine! I recenly snagged a RH-specific file for my Debian GNU/Linux-based server :)
thanks for this nice playbox :)

That’s great! I’ve just typed rm -rf / under root and nothing happened! Fantastic! Still can’t believe it!
Thanks for the opportunity to let Linux enthusiasts learn SELINux hands-on!
Very cool! Good work!
Now Thats Cool stuff man. (masud.sp at
COOL! thanks for your work :)
Great service, thanks
Nice :)

Excelent\!I will try this at home.

Nice demo, I’ll pass this on to the secuity team to show the concepts
I have been pushing for more SE Linux deployments but policy managment is a big cost

Nice works\!
SELinux is interesting.
Dear Russell Coker,

Thanks to providing Play machine, this concept is mind blowing.

This will help community to grow.

Thanks again with regards,

Deepak Mahajan
Head – Internet
Jain Irrigation Systems Ltd.
Jalgaon – India
thanks – very interesting – perfect to get a first look at the selinux features
Thanks for the demo system, very cool!
This is just cool….Great wok Russell ………. anspuli

very good!
thanks. my work have a new possibilit now!

thanks for the access – rgds rhp
killroy was here
this really is amazing. thanks for the demo. – db
how do you do normal root admin stuff on a selinux system with strict policies in force? [you do it as sysadm_r:sysadm_t]
Thanks for the access.
thank you — a brave man indead !
Nice box.
Thanks for sharing it with us!
—-[ The OOM Killer ]—-
root can still eat up all memory and the next process that requests memory will be killed by the kernel. that could be something important like apache on a server, or the “top” of the admin trying to figure out what’s going on, etc
memory usage should be limited
[Limiting the number of processes root can use is impossible, therefore trying to limit memory use is not going to be very productive. So I just make the conditions of use include that DOS attacks are not acceptable. For real servers don’t give the root account to hostile users and use SE Linux to help prevent hostile users getting root.]
mcgrof: how about limiting number of open binds maybe?
mcgrof: anyway, thanks , this is cool
mcgrof: I logged out and the listening ports are still here
mcgrof: I killed them for you
[Again, SE Linux isn’t about resource limitation. Note that you can’t bind to a port that’s reserved for some other purpose.]

this server sucks, cant even do a simple rm -rf / :)

mcgrof: re: binds — yeah, makes sense, thanks anyway, this is great


Nice to have a hands-on SE Linux demo available! What has been bugging me for a long time: How does SE Linux compare to RSBAC? I read the mailing list discussions^H^H^Hflamewars, but didn’t get any useful information out of it.

HOW-TO make demo SeLinux machine? Tambov ,Russia
Hallo Welt
Nice Try
Thanks for this test system. I just copied the thanks.txt_append_only_dont_edit_with_vi file to a different name, which it allowed me to do. It appeared to have the same permissions as the original file. [“ls -lZ” shows the SE Linux contexts of files, the file you copied had a different “type”] I couldn’t delete the original file, but it allowed me to delete the copy. I also tried to shutdown the system and was denied. Good demonstration of SELinux.

thanks, very cool

sweet! with the help of your configuration I managed to set up my Debian box; didn’t try to break it though, looks pretty hopeless concerning my security background. i’ll be back to learn more; thanks

I’m Sorry.I’ve executed it programming continuousness fork.But It’s not being malicious.Sorry really [don’t worry, that happens all the time]
Very impressive, thanks for the demonstration
Thanks Russell, xor007 from South Africa
thanks for showing off your excellent work ~Alicia
Kool. a very intrestig demo
iCanMakeAFile in my home directory.
good that root can still do this.
pretty wacky, see what else is around here for me to try to muck up.
Ooh, root can make files in its homedir.

thx! linio


Thanks for setting up a machine like this! Are there any newer packages installed than what comes with Debian Etch? Or can I build myself a machine like this using nothing but the etch packages? [during Etch I had my own repository for updated packages, now I’m doing the same for Lenny]

neat. – folken from CH
eat meat
Thanks… — Philipp Kern ()(DD)

Nice one Russ.

mlh 2007 11 05 13:48

Very cool. Thanks a lot
From Russia with Fun! Thx u. skynerve
16 nov 2007
Funny to allow strangers root access to your computer, but still be safe. :-)
Still I think a little more documentation for SELinux-newbies could be very useful…


great. just fucking great. russel FTW\!\!\!11one

That’s pretty cool.

cool do you have an apparmor play machine too? [it would be possible to run an apparmor Play Machine, but no-one bothered]
Thanks for this nice setup, i’m not a security expert but the few things I tried where not allowed ;), way to go
The fact that you feel secure even after giving out the root password has motivated me to finally dive into SELinux – thanks!
Nice to meet you! I am from a university of China.
It’s strange playing on a machine in the future; I’m on the other side of
the international dateline.
amazing do i can do it on my debian too?:)
i will try second time with selinux maybe is not too diffcult for me.
Kelaz was herels

cool! gonna install this on my laptop. ^.^

Hello Mon Jan 21 15:32:59 EST 2008
nice… having an open box like this is a ballsy move i really respect that.
if you don’t see the fnords they can’t eat you
nice, you can’t even ls /etc/shadow ;)
Nice one


Hi there,

Thanks for the server, the best I can do so far is to have the box connect to itself continuously through ssh port so no one can log in.


ohls -lsa! i can change passwordls -lsals -lsa [I stuffed up there]
This is pretty cool. Unfortunately, this is only the second time I’m logging into a remote shell so I’m just basking in the novelty and not really contributing anything of worth -George
PRRV-Test from Austria
well, thanx ;) .. i’ll read and learn about selinux i come back ;) .. bodik civ zcu cz
I not able to delete /root/.ssh/authorized_keys, but was able to overwrite it. Should this have been allowed? [no]
sorry for the forkbombs!!!!
Thanks for the peek inside!
I noticed some crashes in the last logins, what caused the crash?
I am internet famousls – Murray.
Would be internet famous if I could spell
[crashes are usually caused by DOS attacks]
i was here
Pretty neat! Thanks Russel
root:user_r:user_t:-s0:c0.c100@play:~# hostname test
hostname: you must be root to change the host name
Nice :)
Mon Jul 7 05:48:55 EST 2008
Thanks for giving me the opportunity to test this machine
SELINUX student from INDIA
Hi there!

Nice security. This convinces me to have a beter look at SELinux


JL Lacroix from Belgium
Wed Jul 23 16:15:35 EST 2008

SELINUX is really enormous!
pretty cool setup / Henrik
Thx, nice demo!
format:c dont work, maybe a bug
Thanks for the really amusing demo! -e

thanks for this stuff. it is a good starting point for SELINUX.
thanks a lot for the opportunity to try this. a big THX from MDQ, Argentina ;) ….zer0
Thanks for a great demo\nMichel van Deventer, Netherlands\

Interesting. I’ve always had reservations about SE Linux, because it introduces another security layer on top of the standard posix model – even with the “normal” model you can sometimes accidentally miss things. I’d be interested to hear how SE Linux has an impact on the daily life of an administrator.

Anyways, thanks.

— Random person from Belgium

Someone wrote “very impressive” in banner art.


Thanks for the demostration\!\!
I really need to learn more about SELinux
Great job\!

My name is Alexandre Stefani
What you do is really cool. I_m learning SELinux and will install it on my Debian
Thanks a lot. I_ll purchase a T-Shirt soon.

Thanks for the demo.
please install iptraf and mc. it would be real fun. thanks!
Hello. Leave your handle here:
Malformation – 27/10/08
I don’t remember writing that! -Malformation

Hello from SELinux course from Austria
SO, root is no more the boss now, \n but you do have a boss i.e SE admin \n root is a normal handicap user on this machine

Ahoy from around the world! This is an amazing demonstration! Are the files in /selinux supposed to be world readable (even though the parent directory restricts access)? Seems to me that a tiny privacy issue exists with concurrent play users and their /proc/${SESSIONPROCNUM}/environ file. Then again, I am a newb…. Thanks & feel free to reply to my comments at! [When two users login with the same UID and context then they can mess with each other, the privacy issue of the environ file is just the tip of the iceberg.]

…. what I meant to say was world-writable… lol.. later -peritus

Cool.Best regards!
best regards from Poland:) Nice work here. When will be the demo how to create this kind of machine? ^_^
Greetings from Chicago, i’m very much interested inlearning SELinux. Thanks for kindly providing this resource
Thanks from pl.
thank you for providing this. I really want to learn selinux.

Thanks Russell.

[update Oct 2009]
It would be nice if you explained how to setup such a play machine.
[that’s on my todo list]
when i grow up ill build such mashines for educational puropses. Necessary docs, tutorials, and an ability to tune the system during one paticular session. And of course – tests: are you sucsessfull. Such a system could be a wonderful alternative to e.g. LPI exams: show me. Communications inside one particular computer system. When i grow up – i’ll know English better =)
seLinux is fun.
Hi, All.
the point is to break the machine?
[the point is to discover security flaws]
Interesting, going to read up on this and maybe set up a VM… sounds like fun! Thanks :)
Hola. Archivo de pruebas.
nice setup
hi all
oru kundhoom nadakkunnilla
Thanks for making this available – I’m just starting to look into SELinux in the hopes that it offers a usably simple security model…
I am fascinated by the fact that I can append to this file, but not remo
ve or truncate it. I like the fine-grained opermissions!
bla bla bla
selinux looks very cool. thank you for providing this.
Hello <3 selinux
win, or WIN.
all your base are belong to us
[Section 2 of the MOTD clearly says that DOS attacks are out of scope]
Hello Kind Sir,
I am Dr. Adamu Salaam, the the bank manager of bank of africa (BOA) Burkina Faso West
I am sending you this message about the $3.14159 million dollars in bank
account number 2718281828450945. I will give you this money in exchange
for the password to the ‘bofh’ account.
[Thanks for the amusing offer. I’ve been offered stolen credit cards and other
junk for the password, Pi million dollars in the account numbered “e” is a
refreshing change.]
Can you recommend any textbooks that teach selinux? Presumably targeted at a Linux SA.
weird stuff this. doesn’t feel like being root :)
Why no /proc/mtrr ? I want to run exploit!
[/proc/mtrr doesn’t exist in a Xen DomU, there wouldn’t be much point in it]
SQL injection doesn’t work on flat files
Hello, boys! :)
Really good
pretty cool…gonna be learning this reall soon. — Glitch
good job! Is this a custom build of selinux policy?
[Custom configuration, but the main policy package is the same one that everyone else should be using]
Great setup, Mr Coker. :)
Cool. Thanks for the opportunity to play with this.
good job SELINUX is really great :)
congratulations Sir it’s really good fun to play with Your server.. SELINUX rules cat thanks.txt_append_only_dont_edit_with_vi ! ~kawooem
seems untouchable… please post your SELinux recepies
Also thanks for this Testmachine, i could test my ISP if he was allowing ssh over cable network.
Greets JacksOn
thanks… interesting
thanks.. interesting CANARIS
@ CANARIS: Yes, just what I was going to say :) ~gmatht
mmm4m5m: Nice. Thanks. I was here.
Managed to get the server to reboot with your tight selinux … ;)
18:56:43 up 1 min, 1 user, load average: 0.08, 0.06, 0.02
[That was the watchdog responding to your DOS attack. NB DOS attacks are out of scope.]
David Jacobson
From South Africa – Down under! []
09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0
I’m curious about malicious commands, i.e. do you consider malicious commands such as:
rm -rf / or using mkfs on / or using a fork bomb liek :(){:|:&};: is considered a security flaw or a type of DOS, didn’t want to try them just incase.
[A fork bomb is a DOS attack, rm -rf and mkfs are legitimate tests of the
security of the system. I encourage you to use rm and mkfs to test the system.]
I’m also curious, if you log on to the console, not ssh, but physical, as root, are the SEL restrestrictions lifted?
[No, the restrictions are based on the context not the terminal. It is
possible to have pam restrict which accounts can login via various methods, so
accounts that allow higher levels of access could be denied ssh logins.
Also there is a boolean to determine whether the administrator can login via
ssh, I have that turned on but for best security you would turn it off.]
Thanks for letting us play on this box. It is a good demo. Perhaps I should not discount SELinux as just a pain in the butt like I traditionally have.
impressive indeed. -reablettoz
This is really cool stuff, thanks for the demo! Gotta say, the real
“wow” moment for me was when I ran top and couldn’t see any procs but my own.
BTW ssh is a bit laggy for me when logged into this box, moreso that most
machines I ssh into. Would selinux have anything to do with this, or have I
just ended up with a slow/laggy link?
— Daniel Gnoutcheff <>
Sun Jul 19 23:24:51 UTC 2009
[I was in the middle of doing a big file transfer when you logged in. But even if I wasn’t the link is a SOHO grade connection so you don’t expect the same quality as a proper data-center.]
Nice, I’ll have to look into this. Thanks for the demo\!
Herro people :3
nice one
Sorry bout the fork-bomb yesterday :3
you know it works when your instinct is to rerun with sudo before realizeing youralready root lol
test test
Wow, this is cool! SELinux rules! I got to try this on my own machine
Hello from Russia
Hello from San Juan, Puerto Rico!
I just found out about this server by reading the SELinux book from O’Reilly. The book is pretty old (2004) and I’m glad to know the URL provided on the book still works!
All the best,
22:09:47 up 21:34, 1 user, load average: 0.00, 0.00, 0.00
Great job with this one, i’ve tried a number of things –
attemtping to get cron to run the files as bofh (no luck, cron transitions to the context im in)
attempting to put hard links in /root so that it relabels key files (no luck, /root is on a different partition)
attempting to mknod a block device (no luck, nodev is set in the mount options and there isnt many places I can write to anyway)
attempting to signal a coredump of “chage” (which doesnt complain when i run it by the way!) so I can read shadow.
attempting to perform sigstop on chage so i can ouput the file descriptor (no luck, chage transitions, i cant read its proc entry nor can i signal it anyway)
attempting to chroot a new environment (no luck, no chroot process privilege)
I think the closest i got was trying to manipulate chage, but i was far far off then. That or being able to write to bofh crontab.
The most effective way to get around the selinux restrictuions would probably be to get read access to /dev/hdc then run debugfs on it to dump the shadow file. But I spent too long on this now anyway!
Great work!

Installing SE Linux on Debian/Lenny

Currently Debian/Lenny contains all packages needed to run SE Linux. Development continues so there are periodic updates which sit in Unstable for a while before migrating to Lenny (testing).

I have set up my own APT repository for SE Linux packages. This has packages that need newer versions than in Lenny but which will be in Lenny eventually (which includes the latest policy packages) as well as my own modified packages to fix bugs that won’t be fixed in Lenny. After Lenny is released I will maintain the repository for i386 and AMD64 for bug fixes and new features above what is in Lenny.

gpg --keyserver hkp:// --recv-key F5C75256
gpg -a --export F5C75256 | apt-key add –

To enable the use of my repository you must first run the above two commands to retrieve and install my GPG key (take appropriate measures to verify that you have the correct key).

deb lenny selinux

Then add the above line to /etc/apt/sources.list and run “apt-get update” to download the list of packages.

Next run the command “apt-get install selinux-policy-default selinux-basics” to install all the necessary packages. After that is done you need the file /.autorelabel to exist for the next boot to cause the filesystems to be labeled. The file /boot/grub/menu.lst needs to have “selinux=1” on the end of the line which starts with “# kopt=” (and the kernel command-lines for each kernel). You can do this manually but the recommended thing to do is to run the command selinux-activate, if given no parameters it will apply all the necessary tweaks to enable SE Linux (it changes PAM configuration files, GRUB configuration, and creates /.autorelabel.

Note that if you use gdm then the file /etc/pam.d/gdm needs to have the line moved to before the GNOME key lines. I need to update the selinux-basics package for this.

Then reboot and the filesystems will be relabeled. The relabel process will cause a second automatic reboot of the machine (it needs to be rebooted so that init gets the correct context). After that is finished the machine will be running in “permissive mode“, this means that SE Linux will log the actions that it would deny, but they will still be performed.

To put the machine in “enforcing mode” you can run the command “setenforce 1“, this means that SE Linux actually controls access to the machine. When you are confident that the machine is working correctly you can edit the file /etc/selinux/config and change the SELINUX= line to specify that it is in “enforcing” mode. The script selinux-config-enforcing will do this for you (with no parameters if configures SE Linux to be in enforcing mode at the next boot). If you need to override this (for example if critical files get the wrong labels and prevent booting) then the kernel command-line option enforcing=0 will override it. I will add a new command selinux-config-enforcing to the selinux-basics package to manage this (it will hopefully be there for Lenny).

If you use Postfix then you need to run it without chroot, the command postfix-nochroot will configure Postfix to not use chroot and will restart it. This script is included in the selinux-basics package but will hopefully be in Postfix for Lenny+1 (I think that many people who don’t use SE Linux will be able to use it).

In summary here are the commands you need:
apt-get install selinux-policy-default selinux-basics

SE Linux Magic

Here is a complete list of entries for /etc/magic related to SE Linux.

# SE Linux policy database for Fedora versions less than 5, RHEL 4, and Debian before Etch
0      lelong  0xf97cff8c      SE Linux policy
>16    lelong  x              v%d
>20    lelong  1      MLS
>24    lelong  x      %d symbols
>28    lelong  x      %d ocons

# SE Linux policy modules *.pp reference policy for Fedora 5 to 9,
# RHEL5, and Debian Etch and Lenny.
0      lelong  0xf97cff8f      SE Linux modular policy
>4      lelong  x      version %d,
>8      lelong  x      %d sections,
>>(12.l) lelong 0xf97cff8d
>>>(12.l+27) lelong x          mod version %d,
>>>(12.l+31) lelong 0          Not MLS,
>>>(12.l+31) lelong 1          MLS,
>>>(12.l+23) lelong 2
>>>>(12.l+47) string >\0        module name %s
>>>(12.l+23) lelong 1          base

# for SE Linux policy source for reference policy
0      string  policy_module(  SE Linux policy module source
1      string  policy_module(  SE Linux policy module source
2      string  policy_module(  SE Linux policy module source

0      string ##\ <summary>    SE Linux policy interface source

0      search  gen_context(    SE Linux policy file contexts

0        search        gen_sens(        SE Linux policy MLS constraints source

SE Linux Saves

Here are links to some instances when SE Linux prevented exploits from working or mitigated their damage:

Here are links to some instances when SE Linux prevented exploits from working or mitigated their damage:

Going Live with a Linux Server

Based on past mistakes by myself and others, here is a check-list before putting a Linux (or other Unix) server online:

  1. Run memtest86+ (or an equivalent program for other architectures) before going live, ideally run it before installing the OS. Run it again every time you upgrade the RAM.
  2. Reboot the machine after every significant change. EG if you install a new daemon then reboot it to make sure that the daemon starts correctly. It’s better to have 5 minutes of down-time for a scheduled reboot than a few hours of down-time after something goes wrong at 2AM.
  3. Make sure that every account that is used for cron jobs has it’s email directed somewhere that a human will see it. Make sure that root has it’s mail sent somewhere useful even if you don’t plan to have any root cron jobs.
  4. Make sure that ntpd is running and has at least two servers to look at. If you have a big site then run two NTP servers yourself and have each of them look to two servers in the outside world or one server and a GPS.
  5. Make sure that you have some sort of daily cron job doing basic log analysis. The Red Hat logwatch program is quite effective, then you need to have some way of making sure that you notice if an email stops being sent (getting 11 instead of 12 messages from logwatch in the morning won’t be noticed by most people).
  6. Make sure that when (not if) a hard drive in your RAID array dies then you will notice it.

Any suggestions on other things I can add?