Notes

I presented this paper at the 2003 Ottawa Linux Symposium (OLS).

http://lsm.immunix.org/ is defunct, since about 2004, so I removed the link.

The NSA changed the URLs on their web site, so this version of the paper has the new ones.

The SE Linux kernel interfaces have changed, now it’s all through the proc and selinuxfs filesystems and there are no SE Linux specific system calls. Equivalent functionality is provided.

With significant changes to the code base (kernel, policy, and tools) the amounts of memory used will differ. But the methods of saving memory will remain the same.

Abstract

In the first part of this paper I will describe how I ported SE Linux to User-Mode-Linux and to the ARM CPU. I will focus on providing information that is useful to people who are porting to other platforms as well. In the second part I will describe the changes necessary to applications and security policy to run on small devices. This will be focussed on hand-held devices but can also be used for embedded applications such as router or firewall type devices, and any machine that has limited memory and storage.

Introduction

SE Linux offers significant benefits for security. It accomplishes this by adding another layer of security in addition to the default Unix permissions model. This is achieved by firstly assigning a type to every file, device, network socket, etc. Then every process has a domain and the level of access permitted to a type is determined by the domain of the process that is attempting the access (in addition to the usual Unix permission checks). Domains may only be changed at process execution time. The domain may automatically be changed when a process is executed based on the type of the executable program file and the domain of the process that is executing it, or a privileged process may specify the new domain for the child process.

In addition to the use of domains and types for access control SE Linux tracks the identity of the user (which will be system_u for processes that are part of the operating system or the Unix user-name) and the role. Each identity will have a list of roles that it is permitted to assume, and each role will have a list of domains that it may use. This gives a high level of control over the actions of a user which is tracked through the system. When the user runs SUID or SGID programs the original identity will still be tracked and their privileges in the SE security scheme will not change. This is very different to the standard Unix permissions where after a SUID program runs another SUID program it’s impossible to determine who ran the original process. Also of note is the fact that operations that are denied by the security [smalley] have the identity of the process in question logged.

I often run SE Linux demonstration machines on the Internet which provide root access to the world and an invitation to try and break the security [play-machine].

For a detailed description of how SE Linux works I recommend reading the paper Peter Loscocco presented at OLS in 2001 [ols2001:loscocco-smalley].

SE Linux has been shown to provide significant security benefits for little overhead on servers, desktop workstations, and laptops. However it has not had much use in embedded devices yet.

Some people believe that SE Linux is only needed for server systems. I think that is incorrect, and I believe that in many situations laptops and hand-held devices need more protection than servers. A server will usually have a firewall protecting it, with a small number of running applications which are well maintained and easy to upgrade. Portable computers are often used in hostile environments that servers do not experience, they have no firewalls to protect them, and often they are connected to routers operated by potentially negligent or hostile organizations.

But there are two main factors that cause an increased need for security on portable devices. One is that it is usually extremely difficult and expensive to upgrade them if a new security fix is needed. This means that in commercial use portable computers tend to never have security fixes applied. Another factor is that often the person in posession of a hand-held computer is not authorised to access all the data it contains, and may even be hostile to the owner of the machine.

Naturally for a full security solution for portable computers a strong encryption system will need to be used for all persistent file systems. There are various methods of doing this, but all aspects of such encryption are outside the scope of this project and can be implemented independently.

Kernel Porting

The current stable series of SE Linux is based on the 2.4.x kernels and uses the Linux Security Modules (LSM) lsm interface. The current LSM interface has a single sys_security() system call that is used to multiplex all the system calls for all of the security modules. SE Linux uses 52 different system calls through this interface. Due to problems in porting the kernel code to some platforms (particularly those that have a mixed 32 and 64bit memory model) the decision was made to change the LSM interface for kernel 2.6.0. The new interface will make the code fully portable and remove the painful porting work that is currently required. However I needed to have SE Linux working with the 2.4.x kernels so I couldn’t wait for kernel 2.6.0.

The main difficulty in porting the code is the system call execve_secure() which is used to specify the security context for the new process. This calls the kernel funtion do_exec() to perform the execution, and do_exec() needs a pointer to the stack, thus requiring architecture specific code in the sys_execve_secure() function. The sys_security_selinux_worker() function (which determines which SE Linux system call is desired and passes the appropriate parameters to it) calls sys_execve_secure() and therefore also needs architecture specific code, and so does the main system call sys_security_selinux().

My first port of SE Linux was to User-Mode Linux [uml]. This was a practice effort for the main porting work. It is quite easy to debug kernel code under UML, and as it uses the i386 system call interface I could port the kernel code without any need to port application code.

The main architecture dependent code is in the source file security/selinux/arch/i386/wrapper.c, which has code to look on the stack for the contents of particular registers. This needs to be changed for platforms with different register names, and for UML which does not permit such direct access of registers.

The solution in the case of UML was to not have a wrapper function, as the current structure had a pointer to the stack anyway that could be used inside the sys_execve_secure() function. So I renamed the sys_security_selinux_worker() function to sys_security_selinux() for the UML port and entirely removed all reference to the wrapper. Then I moved the implementation of sys_execve_secure() into the platform specific directory and implemented a different version for each port.

This was essentially all that was required to complete the port, the core code of SE Linux was all cleanly written and could just be compiled. The only other work involved getting the Makefile’s correctly configured, and adding a hook to sys_ptrace().

One thing I did differently with my port to the ARM architecture was that I removed the code to replace the system call entry. When the SE Linux kernel code loads on UML and i386 it replaces the system call with a direct call to the SE Linux code (rather than using the option for LSM to multiplex between different modules). As there is currently no support for having SE Linux be a loadable module there seems to be no benefit in this, and it seems that on ARM there will be more overhead for adding an extra level of indirection for this. So I made the SE Linux patch hard-code the SE system call into the sys-call table.

iPaQ Design Constraints

The CompaQ/HP iPaQ [ipaq] computers are small hand-held devices. The most powerful iPaQ machines on sale have a 400MHz ARM based CPU that is of comparable speed to a 300MHz Intel Celeron CPU, with 64M of RAM and 48M of flash storage.

An iPaQ is not designed for memory upgrades. There are some companies that perform such upgrades, but they don’t support all models, and this will void your warantee. Therefore you are stuck with a memory limit of 64M.

The flash storage in an iPaQ can only be written a limited number of times, this combined with the small amount of storage makes it impossible to use a swap space for virtual memory unless you purchase a special sleeve for using an external hard drive. Attaching an external hard drive such as the IBM/Hitachi Micro Drive is expensive and bulky. Therefore if you have a limited budget then storage expansion (for increased file storage or swap space) is not an option.

For storing files, the 32M file system can contain quite a lot. The Familiar distribution is optimised for low overheads (no documentation or man pages) and all programs are optimised for size not speed. Also the JFFS2 [jffs2] file system used by Familiar supports several compression algorithms including the Lempel-Ziv algorithm implemented in zlib, so more than 32M of files can fit in storage.

For a system such as SE Linux to be viable on an iPaQ it has to take up a small portion of the 32M of flash storage and 64M of RAM, and not require any long CPU intensive operations.

Finally the screen of an iPaQ only has a resolution of 240×320 and the default input device is a keyboard displayed on the screen. This makes an iPaQ unsuitable for interactive tasks that involve security contexts as it takes too much typing to enter them and too much screen space to display them. As a strictly end-user device this does not cause any problems.

CPU Requirements

Benchmarks that were performed on SE Linux operational overheads in the past show that trivial system calls (reading from /dev/zero and writing to /dev/null) can take up to 33% longer to complete when SE Linux is running, but that the overhead on complex operations such as compiles is so small as to be negligible [freenix]. The machines that were used for such tests had similar CPU power to a modern iPaQ.

One time consuming operation related to SE Linux installation is compiling the policy (which can take over a minute depending on the size of the policy and the speed of the CPU). This however is not an issue for an iPaQ as the policy takes over a megabyte of permanent storage and 5 megs of temporary file storage, as well as requiring many tools that are not normally installed (make, m4, the SE Linux policy compilation program checkpolicy, etc). The storage requirements make it impractical to compile policy on the iPaQ, and the typical use involves configuration being developed on other machines for deployment on iPaQ. So the time taken to compile the policy database is not relevant.

The only SE Linux operation which can take a lot of time that must be performed on an iPaQ is labeling the file system. The file system must be relabeled when SE Linux is first installed, and after an upgrade. On my iPaQ (H3900 with 400MHz X-Scale CPU) it takes 29.7 seconds of CPU time to label the root file system which contains 2421 files. For an operation that is only performed at installation or upgrade time 29.7 seconds is not going to cause any problems. Also the setfiles program that is used to label the file system could be optimised to reduce that time if it was considered to be a problem.

I conclude that for typical use of a hand-held machine SE Linux only requires the CPU power of an iPaQ. In fact the CPU use is small enough that even the older iPaQ machines (which had half the CPU power) should deliver more than adequate performance.

Kernel Resource Use

To compare the amounts of disk space and memory I compiled three kernels. One was 2.4.19-rmk6-pxa1-hh13 with the default config for the H3900 iPaQ. One was a SE Linux version of the same kernel with the options CONFIG_SECURITY, CONFIG_SECURITY_CAPABILITIES, and CONFIG_SECURITY_SELINUX. Another was the same SE Linux kernel with development mode enabled (which slightly increases the size and memory).

For this project I have no need for the multi-level-security (MLS) functionality of SE Linux or the options for labelled networking and extended socket calls. This optional functionality would increase the kernel size. I am focussing on evaluating the choice of whether or not to use SE Linux for specific applications, once you have decided to use SE Linux you would then need to decide whether the optional functionality provides useful benefits to your use to justify the extra disk space and memory use.

The kernel binaries are 658648 bytes for a non-SE kernel, 704708 bytes for the base SE Linux kernel, and 705560 bytes for the development mode kernel. The difference between the kernel with development mode enabled and the regular one is that the development kernel allows booting without policy loaded, and booting in permissive mode (with the policy decisions not being enforced). For most development work a kernel with development mode enabled will be used, also for this test it allowed me to determine the resource consumption of SE Linux without a policy loaded.

To test the memory use of the different kernels I configured an iPaQ to not load any kernel modules. My test method was to boot the machine, login at the serial console, wait 30 seconds to make sure that all daemons have started, and run free to see the amount of memory that is free. This is not entirely accurate as random factors may result in different amounts of memory usage, however this is not as significant on the Familiar distribution due to the use of devfs for device nodes and tmpfs for /var and /tmp which means that in the normal mode of operation almost nothing is written to the root file system, so two boots will be working on almost the same data.

From the results I looked at the total field in the results (which gives the amount of RAM that is available for user processes after the kernel has used memory in the early stages of the boot process), and the used field which shows how much of that has been used. The kernel message log gives a break-down of RAM that is used by the kernel for code and data in the early stages of boot, however that is not of relevance to this study only the total amount that is used matters.

The total memory available was reported as 63412k for the non-SE kernel, 63308k for the SE Linux kernel, and 63300k for the development mode kernel. So SE Linux takes 104k of kernel memory early in the boot process and 112k if you use the development mode option.

The memory reported as used varied slightly with each boot. For the vanilla kernel the value 18256k was reported in two out of four tests, with values of 18252k and 18260k also being reported. I am taking the value 18256k as the working value which I consider accurate to within 8k.

For a standard SE Linux kernel the amount reported as used was 19516k in three out of six tests with the values of 19532k, 19520k, and 19524k also being returned. So I consider 19516k as the working value and the accuracy to be within 16k.

For the SE Linux kernel with development mode enabled the memory used was 19516k in three out of four tests, and the other test was 19524k. So the difference between the development mode kernel and the regular SE Linux kernel is only 8K of kernel memory in the early stages of the boot process.

Finally I did a test of a development mode kernel with no policy loaded. The purpose of this test was to determine how much memory is used on a SE Linux kernel if the SE Linux code is not loading the policy. For this the memory reported as used was 18292k in three out of five tests, with the values of 18296k and 18300k also being returned.

Kernel	memory used
non-SE	18256k
SE no policy	18292k
SE with policy	19516k

So an SE Linux kernel without policy loaded uses approximately 36K more memory after boot than a non-SE kernel in addition to the 104k or 112k used in the early stages of boot.

With a small policy loaded (360 types and 23,386 rules for a policy file that is 583771 bytes in size) the memory used by the kernel is about 1224k for the policy and other SE Linux data structures. The policy could be reduced in size as there are many rules which would only apply to other systems (the sample policy is quite generic and was quickly ported to the iPaQ), although there may be other areas of functionality that are desired which would use any saved space.

So it seems that when using SE Linux the memory cost is 104k when the kernel is loaded, and a further 1260k for SE Linux memory structures and policy when the boot process is complete. The total is 1364k of non-swappable kernel memory out of 64M of total RAM in an iPaQ, this is about 2% of RAM.

All tests were done with GCC 3.2.3, a modified Linux 2.4.19, and an X-scale CPU. Different hardware, kernel version, and GCC version will give different results.

Porting Utilities

The main login program used on the Familiar [familiar] distribution is gpe-login, which is an xdm type program for a GUI login. This program had to be patched to check a configuration file and the security policy to determine the correct security context for the user and to launch their login shell in that context. The patch for this functionality made the binary take 4556 bytes more disk space in my build (29988 bytes for the non-SE build compared to 34544 bytes for the version with SE Linux support).

The largest porting task was to provide SE Linux support in Busybox [busybox]. Busybox provides a large number of essential utility programs that are linked into one program. Linking several programs into one reduces disk space consumption by spreading the overhead for process startup and termination code across many programs. On arm it seems that the minimum size of an executable generated by GCC 3.2.3 is 2536 bytes. In the default configuration of Familiar Busybox is used for 115 commonly used utilities, having them in one program means that the 2.5K overhead is only used once not 115 times. So approximately 285K of uncompressed disk space is saved by using busybox if the only saving is from this overhead. The amount of disk space used for initialisation and termination code would probably increase the space used by more than 80% if all the applets were compiled separately (my build of Busybox for the iPaQ is 337028 bytes).

The programs that are of most immediate note in busybox are ls, ps, id, and login. ls needs the ability to show the security contexts of the files, ps needs to show the security contexts of the running processes, and id needs to show the context of the current process. Also the /bin/login applet had to be modified in the same manner as the gpe-login program. These changes resulted in the binary being 5600 bytes larger (337028 bytes for a non-SE version and 342628 bytes for the version with SE Linux support.

Busybox Wrappers for Domain Transition

In SE Linux different programs run in different security domains. A domain change can be brought about by using the execve_secure() system call, or it can come from an automatic domain transition. An example of an automatic domain transition is when the init process (running in the init_t domain) runs /sbin/getty which has the type getty_exec_t, which causes an automatic transition to the domain getty_t. Another example is when getty runs /bin/login which has the type login_exec_t and causes an automatic transition to the domain local_login_t. This works well for a typical Linux machine where /sbin/getty and /bin/login are separate programs.

When using Busybox the getty and login programs will both be sym-links to /bin/busybox and the type of the file as used for domain transitions will be the type of /bin/busybox, which is bin_t. SE Linux does not perform domain transitions based on the type of the sym-link, and it assignes security types to the Inodes not file names (so a file with multiple hard links will only have one type). This means that we can’t have a single Busybox program automatically transitioning into the different domains.

There are several possible solutions to this problem, one possible partial solution would be to have Busybox use execve_secure() to run copies of itself in the appropriate domain. Busybox already has similar code for determining when to change UID so that some of the Busybox applets can be effectively SETUID while others aren’t. The SETUID management of Busybox requires that it be SETUID root, and involves some risk (any bug in busybox can potentially be exploited to provide root access). Providing a similar mechanism for transitioning between SE Linux security domains would have the same security problems whereby if you crack one of the Busybox applets you could then gain full access to any domain that it could transition to. This does not provide adequate security. Also it would only work for transitions between privileged domains (it would not work for transitions from unprivileged domains). I did not even bother writing a test program for this case as it is not worth considering due to a lack of security and functionality.

A better option is to split the Busybox program into smaller programs so transitions can work in the regular manner. With the current range of applets that would require one program for getty, one for login, one for klogd, one for syslogd, one for mount and umount, one for insmod, rmmod, and modprobe, one for ifconfig, one for hwclock, one for all the fsck type programs, one for su, and one for ping. Of course there would also be one final build of busybox with all the utility programs (ls, ps, etc) which run with no special privilege. To test how this would work I compiled Busybox with all the usual options apart from modutils, and I did a separate build with only support for modutils. The non-modutils build was 323236 bytes and the build with only modutils was 37764 bytes. This gave a total of 361000 bytes compared to 342628 bytes for a single image, so an extra 18372 bytes of disk space was required for doing such a split.

Splitting the binary in such a simple fashion would likely cost 18K for each of the eleven extra programs. If we changed the policy to have syslogd and klogd run in the same domain (and thus the same program) and have hwclock run with no special privs (IE the domain that runs it needs to have access to /dev/rtc) then there would only be nine extra programs for a cost of approximately 162K of disk space. This disk space use could be reduced by further optimisation of some of the applets, for example in the case of ifconfig the code to check argv[0] to determine the applet name could be removed. A simple split in this manner would also make it more difficult for an attacker to make the program perform unauthorized actions. When a single program has /bin/login functionality as well as /bin/sh then there is potential for a buffer overflow in the login code to trigger a jump to the shell code under control of the attacker! When the shell is a separate program that can only be entered through a domain transition it is much more difficult to use an attack on the login program to gain further access to the system.

Finally if we have a single Busybox program that includes applets running in different domains we need to make some significant changes to the policy. The default policy has assert rules to prevent compilation of a policy that contains mistakes which may lead to security holes. For the domains getty_t, klogd_t, and syslogd_t there are assertions to prevent them from executing other programs without a domain transition, and to prevent those domains being entered through executing files of types other than the matching executable type (this requires that each of those domains have a separate executable type, IE they are not all the same program). Adding policy which requires removing these assertions weakens the security of the base domains and also makes the policy tree different from the default tree which has been audited by many people.

Another way of doing this which uses less disk space is to have a wrapper program such as the following:

#include <unistd.h>
#include <string.h>

int main(int argc, char **argv
       , char **envp)
{
  /* ptr is the basename of the
   executable that is being run */
  char *ptr = strrchr(argv[0], '/');
  if(!ptr)
    ptr = argv[0];
  else
    ptr++;

  /* basename must match one of
     the allowed applets,
     otherwise it's a hacking
     attempt and we exit   */
  if(strcmp(ptr, "insmod")
  && strcmp(ptr, "modprobe")
  && strcmp(ptr, "rmmod"))
    return 1;
  return execve("/bin/busybox"
              , argv, envp);
}

This program takes 2912 bytes of disk space. The idea would be to have a copy of it named /sbin/insmod with type insmod_exec_t which has symlinks /sbin/rmmod and modprobe pointing to it. Then when insmod, rmmod, or modprobe is executed an automatic domain transition to the insmod_t domain will take place, and then the Busybox program will be executed in the correct context for that applet.

This option is easy to implement, one advantage is that there is no need to change the Busybox program. The fact that the entire Busybox code base is available in privileged domains is a minor weakness. Implementing this takes about 2900 bytes of disk space for each of the nine domains (or seven domains depending on whether you have separate domains for klogd and syslogd and whether you have a domain for hwclock). It will take less than 33K or 27K of disk space (depending on the number of domains). This saves about 130K over the option of having separate binaries for implementing the functionality.

A final option is to have a single program to act as a wrapper and change domains appropriately. Such a program would run in its own domain with an automatic domain transition rule to allow it to be run from all source somains. Then it would look at its parent domain and the type of the symlink to determine the domain of the child process. For example I want to have insmod run in domain insmod_t when run from sysadm_t. So I have an automatic transition rule to transition from sysadm_t to the domain for my wrapper (bbwrap_t). Then the wrapper determines that its parent domain is sysadm_t, determines that the type of the symlink for its argv[0] is insmod_exec_t and asks the kernel what domain should be entered when a process in sysadm_t executes a program of type insmod_exec_t, and the answer is insmod_t. So the wrapper then uses the execve_secure() system call to execute Busybox in the insmod_t domain and tell it to run the insmod applet.

I implemented a prototype program for this. For my prototype I used a configuration file to specify the domain transitions instead of asking the kernel. The resulting program was 6K in size (saving 27K of disk space over the multiple-wrapper method, and 156K of disk space over the separate programs method), although it did require some new SE Linux policy to be written which takes a small amount of disk space and kernel memory.

One problem with this method is that it allows security decisions to be made by an application instead of the kernel. It is preferrable that only the minimum number of applications can make such security decisions. In a typical configuration of SE Linux the only such applications will be login, an X login program (in this case gpe-login), cron (which is not installed in Familiar), and newrole (the SE Linux utility for changing the security context which operates in a similar manner to su).

The single Busybox wrapper is more of a risk than most of these other programs. The login programs are only executed by the system and can not be run by the user with any elevated privileges which makes them less vulnerable to attack. Newrole is well audited and the domains it can transition to are limited by kernel to only include domains that might be used for a login process (dangerous domains such as login_t are not permitted).

Due to the risks involved with a single busybox wrapper, and the fact that the benefits of using 6K on disk instead of 33K are very small (and are further reduced by an increase in kernel memory for the larger policy) I conclude that it is a bad idea.

I conclude that the only viable methods of using Busybox on a SE Linux system are having separate wrapper programs for each domain to be entered (taking 33K of extra disk space and requiring minor policy changes), or having entirely separate programs compiled from the Busybox source for each domain (taking approximately 162K of extra disk space with no other problems). Also with some careful optimisation the 162K of overhead could be reduced for the option of splitting the Busybox program. If 162K of disk space can be spared (which should not be a problem with a 32M file system) then splitting Busybox is the right solution.

Removed Functionality

A hand-held distribution doesn’t require all the features that are needed on bigger machines such as servers, desktop workstations, and laptops. Therefore we can reduce the size of the SE Linux policy and the number of support programs to save disk space and memory.

For a full SE Linux installation there are wrappers for the commands useradd, userdel, usermod, groupadd, groupdel, groupmod, chfn, chsh, and vipw. These can possibly be removed as there is less need for adding, deleting, or modifying users or groups on a hand-held device in the field. These programs would take 27K of disk space if they were included.

A default installation of Familiar does not include support for /etc/shadow, and therefore there is no need for the wrapper programs for the administrator to modify users’ accounts. However I think that the right solution here is to add /etc/shadow support to Familiar rather than removing functionality from SE Linux. This will slightly increase the size of the login programs.

In a full install of SE Linux there are programs chsid and chcon to allow changing the security type of files. These are of less importance for a small device. There will be fewer types available, and the effort of typing in long names of security contexts will be unbearable on a touch-screen input device. A hand-held device has to be configured to not require changing the contexts of files, and therefore these programs can be removed.

In the Debian distribution there is support for installing packages on a live server and having the security contexts automatically assigned to the files. As iPaQ’s are used in a different environment I believe that there is less need for such upgrades and such support could optionally be removed to save disk space. I have not written the code for this yet, but I estimate it to be about 100K.

The default policy for SE Linux has separate domains for loading policy and for policy compilation. On the iPaQ we can’t compile policy due to not having tools such as m4 and make, so we can skip the compilation program and its policy. Also the policy for a special domain for loading new policy is not needed as the system administration domain sysadm_t can be used for this purpose. It is possible to even save 3500 bytes of disk space by not including the program to load the policy (a reboot will cause the new policy to take affect).

A server configuration of SE Linux (or a full workstation configuration) includes the run_init program to start daemons in the correct security context. On a typical install of Familiar there are only three daemons, a program to manage X logins, a daemon to manage bluetooth connections, and the PCMCIA cardmgr daemon. For restarting these daemons it should be acceptable to reboot the iPaQ, so run_init is not needed.

Disk Space and RAM Use

In the section on kernel resource usage I determined that the kernel was using 1364K of RAM for SE Linux with a 583771 byte policy comprising 23,386 rules loaded. Since the time that I performed those tests I reduced the policy to 455,422 bytes and 18,141 rules which would reduce the kernel memory use. I did not do any further tests as it is likely that I will add new functionality which uses the memory I have freed. So I can expect that 1.3M of kernel memory is taken by SE Linux.

The SE Linux policy that is loaded by the kernel takes 67K on disk when compressed. The file_contexts file (which specifies the security contexts of files for the initial installation and for upgrades) takes 24K. The kernel binary takes 64K more disk space for the SE Linux kernel. So the kernel code and SE Linux configuration data takes 156K of disk space (most of which is compressed data).

The program setfiles is needed to apply the file_contexts data to the file system. Setfiles takes 20K of disk space. The file_contexts file could be reduced in size to 1K if necessary to save extra disk space, but in my current implementation it can not be removed entirely. In Familiar a large number of important system directories (such as /var) on Familiar are on a ramfs file system. I am using setfiles to label /mnt/ramfs. So far it has not seemed beneficial to have a small file_contexts file for booting the system and an optional larger one for use when installing new packages or upgrading, but this is an option to save 23K. Another option would be to write a separate program that hard-codes the security contexts for the ramfs. It would be smaller than setfiles and not require a file_contexts file, thus saving 30K or more of disk space. Currently this has not seemed worth implementing as I am still in a prototype phase, but it would not be a difficult task. Also if such a program was written then the next step would be to use a [jffs2] loop-back mount to label the root file system on a server before installation to the iPaQ (so that setfiles never needs to run on the iPaQ.

The patches for the gpe-login and busybox programs to provide SE Linux login support and modified ls, ps, and id programs cause the binaries to take a total of 10K extra disk space.

Splitting Busybox into separate programs for each domain will take an estimated 162K of disk space.

The total of this is approximately 348K of additional disk space for a minimal installation of SE Linux on an iPaQ. Adding support for /etc/shadow and other desirable features may increase that to as much as 450K depending on the features chosen. However if you use multiple Busybox wrappers instead of splitting Busybox then the disk space for SE Linux could be reduced to less than 213K. If you then replaced setfiles for the system boot labeling of the ramfs then it could be reduced to 190K.

Conclusion

Security Enhanced Linux on a hand-held device can consume less than 1.3M of RAM and less than 400K of disk space (or less than 200K if you really squeeze things). While the memory use is larger than I had hoped it is within a bearable range, and it could potentially be reduced by changing the kernel code to optimise for reduced memory use. The disk space usage is trivial and I don’t think it is a concern.

I believe that the benefits of reducing repair and maintenance problems with hand-held devices that are deployed in the field through better security outweigh the disadvantage of increased memory use for many applications.

All source code and security policy code releated to this article will be on my web site [my-site].

References

• [smalley] Configuring the SELinux Policy
  Stephen D. Smalley, NAI Labs
  http://www.nsa.gov/selinux/policy2-abs.html
• [play-machine] Details of SE Linux test machine
  http://www.coker.com.au/selinux/play.html
• [ols2001:loscocco-smalley] Meeting Critical Security Objectives with Security-Enhanced Linux
  Peter A.Loscocco, NSA
  Stephen D. Smalley, NAI Labs
  http://www.nsa.gov/selinux/papers/ottawa01-abs.cfm
• [uml] User-Mode Linux
  http://sourceforge.net/projects/user-mode-linux
• [ipaq] Wikipedia page about iPaQ
  http://en.wikipedia.org/wiki/Ipaq
  HP Site for iPaQ Information
  http://whp-sp-orig.extweb.hp.com/country/us/eng/prodserv/handheld.html
• [jffs2] Journalled Flash File System 2
  http://sources.redhat.com/jffs2/
• [freenix] Integrating Flexible Support for Security Policies into the Linux Operating System
  Peter A. Loscocco, NSA
  Stephen D. Smalley, NAI Labs
  http://www.nsa.gov/selinux/papers/freenix01-abs.cfm
• [familiar] Familiar Linux distribution for hand-held devices
  http://familiar.handhelds.org
• busybox Busybox – Swiss Army Knife of Embedded Linux
  http://busybox.net
• [my-site] My SE Linux Web Pages
  http://www.coker.com.au/selinux

Notes

I presented this paper at the 2006 SAGE-AU conference.

Abstract

This paper describes the problems related to shared directories such as /tmp and /var/tmp as well as problems related to having multiple SE Linux security contexts used for accessing a single home directory. It then provides detailed information on the solution to this problem that has been implemented with polyinstantiated directories by using the pam_namespace module.

Introduction

It is a long-standing Unix tradition that the directories /tmp and /var/tmp are used for temporary storage by all programs and on behalf of all users. This used to not be considered a problem, however in recent times it has been recognised that the use of such a shared directory is vulnerable to race-condition attacks with symbolic links.

Another problem is that in some situations a file name may convey secret information. If the file in question is in a public directory such as /tmp or /var/tmp (which may be an unintended result of a command by the user) then this will represent an information leak if there are any less privileged processes running on the machine.

Past attempts to deal with these problems have included restrictions on creating sym-links and hiding file names, which have both been inadequate. The solution chosen for use with SE Linux (which is also designed to work without SE Linux) is to have polyinstantiated directories based on Unix account name and/or SE Linux context. This means that every user will see a different version of the directory in question based on their context.

In the past this feature has been implemented as part of Multi-Level Security (Dr. Rick Smith [HREF2]) systems under the name multi-level directories. I believe that the multi-level directory variant of this solution was based on file system support, while the Linux support for this type of operation that I will describe is based in the VFS layer and thus does not require modification to any of the file systems that may be used.

Summary of Attacks that can be Prevented by Poly-Instantiated Directories

In this paper I am considering the following attack scenarios:

1. Attack by user on user (including the case of a non-PI user as attacker or victim)
2. Attack by user on daemon (including the case of a non-PI user as attacker)
3. Attack by non-root daemon on user
4. Attack by root daemon on user (will always succeed without SE Linux)

Each of the above four attack scenarios may occur with one of the following three attacks:

1. Race-condition attacks on the integrity of processes and data (sym-link attacks, race conditions on renaming objects, or pre-creating a file to take ownership of data)
2. Leaks of confidential data via secrets in file names
3. Denial Of Service (DOS) attacks based on race conditions and pre-allocating file/directory names

Other Solutions

One attempt at solving this problem that has been implemented in some Linux security systems is to hide file names. This can work as long as it is not possible to guess any of the file names in question. If the file name can be guessed then the hostile party can attempt to create a new file of the same name, failure to create the file in question indicates existence. But this only solves the problem of secret data in file names.

Another partial attempt at dealing with this problem is controlling the ability to create hard-links and/or sym-links to try and prevent race conditions. A well-known implementation of this is in the OpenWall kernel patch [HREF3] which prevents the user from creating hard-links to files to which they have no write access and from creating sym-links in a +t directory (a directory such as /tmp or /var/tmp) which point to a file that they don’t own. It also prevents writing to named pipes in +t directories which are owned by a different user. This deals with some of the issues related to race-condition attacks but there are potential issues that it does not address, such as a hostile user creating sym-links to their own files to divert output or creating a file with no write permissions as a denial of service against a program that uses a fixed file name.

But this only deals with the case of race conditions used to attack system integrity. It does not prevent DOS attacks or protect secret data when it is used in a file name.

SE Linux Requirements for Shared Directories

SE Linux does not attempt to hide file names in a directory, if the name of a file contains secret data then this can be a security problem on shared directories such as /tmp and /var/tmp, this is an issue that has to be solved outside of the core SE Linux code base.

The SE Linux strict and mls policies provide good protection against most race condition attacks. Most domains are not permitted to create hard links to privileged files (types such as etc_t). Daemons are all protected from sym-link attacks by each other due to being denied access to sym-links created by other daemons and by users, and users are given similar protection against attack by daemons (both root and non-root). The main benefit for PI directories in strict and mls SE Linux systems is for protection against users attacking other users, in most cases large numbers of users will have the same SE Linux domain and therefore there will not be any effective protection against such attacks in the domain-type model (the integrity protection part of SE Linux).

When a Unix account is associated with more than one SE Linux context it is necessary to have multiple instances of the home directory to match the SE Linux context. If there is only one instance of the home directory and different SE Linux contexts are used for user logins then one of the contexts may be denied access to shared files such as .bashrc and .bash_history, or they may serve as information leaks. This use creates a requirement for PI home directories in SE Linux that does not exist for non-SE systems.

The problem of multiple logins with different contexts can occur in the older version of SE Linux (known as the example policy) that was used in Red Hat Enterprise Linux 4 and Fedora Core versions 2 to 4 when running the strict policy that permits multiple roles to be allocated to a user. But this is more of an issue with the newer versions of SE Linux policy that have functional support for MLS labels and the new MCS policy that permits different sets of categories to be assigned to a user session.

Non-SE Linux Requirements for Shared Directories

Polyinstantiation of shared directories also provides benefits for non-SE Linux systems, in fact there are probably more benefits to be gained from using this on non-SE systems. The SE Linux strict policy provides protection against sym-link race condition attacks launched by users against users in different roles, attacks by users against daemons, and attacks by daemons against users. The SE Linux MLS policy provides these benefits and also protects against attack from programs running at different levels, for example a process running at sensitivity level s2 could not be tricked into leaking data to a program running at level s1, even if the two programs ran in the same domain and with the same UID. Also SE Linux prevents unprivileged processes from creating hard links to files that are important to system integrity or data confidentiality (which is almost a complete solution to hard-link based attacks).

A non-SE system has none of the above protections and only has the Unix UID to protect both system integrity and confidentiality of data.

Linux Kernel Support for Poly-Instantiated Directories

In recent versions of Linux the current list of mounted file systems is available from the /proc/mounts file which is a sym-link to /proc/self/mounts, this permits displaying the name-space which applies to the current process. If /etc/mtab is a sym-link to /proc/mounts then programs such as df will display information on the mount points that are associated with the name-space for the process.

The initial support for PI directories was via the CLONE_NEWNS flag to the clone() system call. This flag causes the child process to be allocated a separate name space. That process and each child process that it launched would have a separate name space to the process which called clone(), and to any process that resulted from another call to clone() with the CLONE_NEWNS flag. The problem with this was the requirement that applications be modified to use clone() with this flag instead of using fork().

To solve this problem a new system call sys_unshare [HREF4] was added to the Linux kernel. The unshare system call can create a separate name-space for mounted file systems among other things (the set of kernel datra structures that can be unshared has been steadily increased since the introduction of unshare).

The unshare system call requires the SYS_ADMIN capability but does not require a fork, exec, or other operation. So it can be called from a PAM module and thus work with unmodified login programs. Also it is possible for multiple PAM modules to unshare different kernel data structures.

Shared Subtrees

One obvious problem with the functionality described in the previous section is the situation where the administrator wants to mount file systems and have all users see them, or have daemons mount file systems (such as autofs).

The solution to this is a development known as Shared Subtrees [HREF5]. This gives the option of specifying that certain subtrees will not be shared. For example if the directories /tmp and /var/tmp are being instantiated
then the following commands could be run from a system boot script to cause all other mount operations to propagate to all users:

mount --make-shared /
mount --bind /tmp /tmp
mount --make-private /tmp
mount --bind /var/tmp /var/tmp
mount --make-private /var/tmp

The above commands make the root of the name-space shared and then make /tmp and /var/tmp private. Note that the --make-private option to the mount command only applies to mount points. As on my test system both /tmp and /var/tmp are on the root file system I have to bind mount them to themselves to have a mount point that can be made private. Be aware that if you don’t correctly exclude the PI directories from the shared name space then each user who logs in may get PI directories under another user’s directories, and things generally won’t work.

Design Overview of PI Directories in Linux

The initial design for PI directories was based on having them only created for user sessions at login time by PAM [HREF6] or similar mechanisms. To implement this the PAM module will create a directory under the directory that is being instantiated, create an unshared name space, and then bind mount the new directory over the PI directory. For example if /tmp is to be PI for user rjc then the directory /tmp/tmp.inst-rjc-rjc would be created as the instance of /tmp for the user rjc. After the directory is created an unshared name space would be created via the unshare system call. Finally in the new name space a bind mount would be used to replace /tmp with /tmp/tmp.inst-rjc-rjc, the bind mount operation would be equivalent to the command:

mount --bind /tmp/tmp.inst-rjc-rjc /tmp

The directory that was created was given the Unix permission mode 1777 (all users can create files and directories, but it is only permitted to remove files or directories that you own). This solved many of the problems related to users attacking users and users attacking daemons. But it does not solve the problem of a daemon attacking a user as the daemon has access to the parent of the PI directories. Also there is a configuration option to have a user excluded from the PI directory system, a user who is granted such access (either deliberately or accidentally) would also be able to attack other users. As all directories were created under /tmp with mode 1777 there was no protection of secret file names from daemons and users who were outside the PI system (for most systems I expect that there wil be some users who will be excluded from the PI configuration).

Another problem with the initial implementation was that the directories were all created at login time, therefore a hostile process could guess the names and pre-allocate directories to allow taking over ownership and potentially allowing other race condition attacks. For example any privileged process which relies on files not being unlinked or renamed for correct operation would operate incorrectly (and possibly be subject to attack) if run in a situation where the /tmp directory did not have the mode 1777 to prevent such rename and unlink operations.

Finally the initial implementation did not have a fall-back case for when the desired name for a PI directory had been taken by a file and would cause the login process to abort, this could be used as a DOS attack against user login sessions.

I have identified two possible solutions to the problem of DOS attacks against the pam_namespace module. One solution is to have it check whether the PI directory already exists, if it exists but has the wrong permissions (either Unix or SE Linux) or if there is an object other than a directory using that name then it would try creating the directory under a different name (maybe the original name with “.1” appended) and keep trying different names until it finds one that is available. This solution does not solve the problem of protecting secret file names.

The other solution I have identified solves the problems of DOS attacks and race conditions as well as the leaks of secret data in file names. This requires that a directory be pre-allocated on the system to contain all PI directories. So instead of a PI directory having the name /tmp/tmp.inst-rjc-rjc it might have the name /tmp/.inst/tmp.inst-rjc-rjc. The /tmp/.inst directory would be created and/or verified at system boot time and would have Unix permission mode 000 (the capability dac_override which every login program posesses would be requred to access it) and would also have a SE Linux context that permits only very restrictive access. Therefore non-root daemons will be denied access to /tmp/.inst and therefore would not be able to launch attacks on users via the /tmp directory. On SE Linux systems root daemons will also be denied such access. If a user session is launched with a shared system name space (through misconfiguration or unusual requirements) then they would also be denied access to the instance of /tmp used by other users.

In the initial design of PI directories the aim was to confine users to prevent them from attacking the rest of the system, and such a confined user was still vulnerable to attack from outside. The second of the two solutions that I propose above is the one that I believe to be the best, it will protect users who have a PI version of a shared directory from attack by non-root daemons on a non-SE system and from attack by root daemons as well on a SE Linux system. It would be a viable option for the sys-admin to give a single user a PI version of /tmp to protect the files for that user while allowing all other users access to the system shared name space.

At the time of writing we have agreement on the concept of using a naming system somewhat like /tmp/.inst/tmp.inst-something where the directory /tmp/.inst will have Unix mode 000 and restrictive SE Linux access controls. This will prevent daemons and users that are not included in the PI configuration from attacking daemons and users that have it enabled. This makes PI protect the user who has such a PI directory as well as protecting the rest of the system from that user. Note that at the time of writing there was no final agreement on the directory names, while the concept of a two-level directory is agreed the actual name of the directory in the default configuration is still to be resolved.

A feature that has been discussed and agreed in concept is to have the pam_namespace.so module check the permissions of the /tmp/.inst directory and abort the login process if the directory does not have Unix permission mode 000, root ownership, and a suitable SE Linux label (if SE Linux is enabled). There will be a configuration option to disable this functionality as not all systems will need this level of protection (and not all administrators will want a system to fail-closed on such a minor security issue).

Currently Released Code

As of the time of writing Fedora Rawhide has a shared object named pam_namespace.so that implements the basic functionality. To use it the PAM configuration files in the /etc/pam.d directory must be modified to have the following line at the end:

session    required     pam_namespace.so

The system will work if the pam_namespace object is not the last in the list, but the creation of the namespace may interfere with some other PAM modules (for example if a PAM module wanted to access files in the /tmp directory) and in general it is safest to have it last. The only situation in which you might not want to have pam_namespace as the last session module is if you are using pam_mkhomedir and also using pam_namespace to provide PI home directories. But currently pam_mkhomedir does not work correctly in situations where PI home directories are desired so this should not be an issue.

The most noteworthy parameter for the pam_namespace module is the optional parameter unmnt_remnt. This is used by programs that run from an unshared namespace and need to create another unshared namespace. The primary example of this is su, all other programs that perform actions which are similar in concept (IE they are run from a user session and launch a new session on behalf of another user) will have the same requirement.

The pam_namespace module uses the configuration file /etc/security/namespace.conf. This file currently has four parameters, the first gives a directory that should be instantiated (there is an option of $HOME for instantiating the user home directory). The second is the name of the real directory to be used for the instance which has variables $USER and $HOME to represent the user-name and the home directory of the user. The third parameter may have as it’s value user, context, or both to indicate whether the instantiation should be based on user-name, SE Linux context, or both. The final parameter is a list of comma-separated user-names for accounts that are exempt from poly-instantiation of the directory in question. I believe that it will be standard practice to include root in this list of accounts (usually there will be no need for other users to be excluded).

Preventing Daemons From Attacking Each Other

In the currently released code there is no protection against daemons attacking each other. I believe that to take advantage of the full benefits offered by PI directories most daemons that run as non-root need the same protection so that they can not attack each other.

In Fedora there is a new program called runuser that will start a daemon as a user other than root. It is linked against PAM and can be configured to call the pam_namespace module. When I finish the debugging then every time it launches a daemon as non-root it will be able to create a new unshared namespace. Non-root daemons that require the system shared name space will need to have their user-names specified in the namespace.conf file.

In Debian daemons are started via a program named start-stop-daemon. I plan to modify this program to have the necessary name-space functionality.

Interesting Features

There is no requirement that the PI directory be a sub-directory of the directory it replaces. In fact it can be on a different filesystem. If you have a separate filesystem for /tmp and don’t want to have a separate filesystem for /var/tmp too then you could just configure namespace.conf such that /var/tmp is instantiated under /tmp. The following is one sample configuration:

/tmp     /tmp/.inst/tmp.inst-$USER-       both      rjc,root
/var/tmp /tmp/.inst/var-tmp.inst-$USER-   both      rjc,root

Conclusion

With a two-level directory configuration (such as /tmp/.inst/whatever) we protect against all the attack scenarios that I consider (including an attack launched by a root-owned daemon on users when running SE Linux). The protection provided by the PI shared directory works in two ways, it protects the process with the unshared namespace and it also protects all other processes on the system against attack from that process.

Most operations that are described in this paper are usable in Fedora Rawhide as of the 31st of May 2006. The only operation that is not usable at this time is PI support for daemons. I hope to have PI working in Debian and have PI support for daemons in both Debian and Fedora Rawhide by the time this paper is published, I will describe my success in these efforts when I present this paper.

---

Hypertext References

HREF1 http://www.coker.com.au/selinux/
HREF2 http://www.cs.stthomas.edu/faculty/resmith/r/mls/index.html
HREF3 http://www.openwall.com/linux/README.shtml
HREF4 http://marc2.theaimsgroup.com/?l=linux-kernel&m=112350785026703&w=2
HREF5 http://lwn.net/Articles/159092/
HREF6 http://www.kernel.org/pub/linux/libs/pam/

---

Copyright

The System Administrators Guild of Australia© 2006. The authors assign to The System Administrator’s Guild of Australia and other educational and non-profit institutions a non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The authors also grant a non-exclusive licence to The System Administrators Guild of Australia to publish this document in full on the World Wide Web and on CD-ROM and in printed form with the conference papers and for the document to be published on mirrors on the World Wide Web.

Notes

I presented this paper at the OSDC conference in 2006.

The main page for my Postal benchmark is at http://doc.coker.com.au/projects/postal/. My blog posts about benchmarking can be found at http://etbe.coker.com.au/category/benchmark/.

Abstract

Postal is a mail server benchmark that I wrote. The main components of it are postal for testing the delivery of mail via SMTP, rabid for testing mail download via POP, and a new program I have just written called bhm (for Black Hole Mailer) which listens on port 25 and sends all mail to /dev/null.

The new BHM program makes it possible to test the performance of mail relay systems. This means outbound smart host gateway systems, list servers, and forwarding services. The testing method is to configure three machines, one running Postal for sending the mail, the machine to be tested running a mail server or list server configuration, and the target machine running BHM.
The initial aim of this paper was to use artificial delays in the BHM program to simulate slow network performance and also to simulate various anti-spam measures and to measure how they impact a mail relay system. However I found other issues along the way which were interesting to analyse and will be useful to other people.

Description of Postal

The first component of the benchmark suite is Postal, this program sends mail at a controlled rate. When using it you have a list of addresses for senders and a separate list of recipients that will be used for sending random messages to a mail server. It sends the mail to a specified IP address to save the effort of configuring a test DNS server because in the most common test scenario you have a single mail server that you want to test.

Postal sends at a fixed rate because in most MTAs an initial burst of mail will just go to the queue and will be actually delivered much more slowly. Often mail servers will take two minutes or more of sustained load to show the full performance impact. So I designed the programs in the Postal suite to display their results once per minute so you can watch the performance of the system over time and track the system load.

The most important thing to observe is that the load (in all areas) is below 100%. If any system resource (CPU, network IO, or disk IO) is used to 100% capacity then the queue will grow without limit. Such unlimited queue growth leads to timeouts which increases the queue and causes the system to break down. An SMTP server has a continual load from the rest of the Internet and if it goes more slowly the load will not decrease in the short-term. So a server that falls behind can simply become unusable, an unmanaged mail server can easily accumulate a queue of messages as old as a week through not having the performance required to deliver them as fast as they arrive.

The second program in the Postal suite is Rabid, a benchmark for POP servers. The experiments I document in this paper do not involve Rabid.

The most recent program is BHM which is written as an SMTP sink for testing mail relays. The idea is that a mail relay machine will have mail sent to it by Postal and then send it on to a machine running BHM. There are many ways in which machines that receive mail can delay mail and thus increase the load on the server. Unfortunately I spent all the time available for this paper debugging my code and tracking down DNS server issues so I didn’t discover much about the mail server itself.

Hardware

For running Postal I used my laptop. Postal does much less work than any other piece of software in the system so I’m sure that my laptop is not a performance bottleneck. It is however a 1700MHz Pentium-M and probably the fastest machine in my network.

For the mail relay machine (the system actually being tested) I used a Compaq Evo desktop machine with a 1.5GHz P4 CPU, 384M of RAM, and an 80G IDE disk.

For running BHM I used an identical Compaq system.

The network is 100baseT full duplex with a CableTron SmartSwitch. I don’t think it will impact the performance. During the course of testing I did not notice any reports of packet loss or collisions.

All the machines in question were running the latest Fedora rawhide as of late September 2006.

Preparation

To prepare for the testing I set up a server running BHM with 254 IP addresses to receive email (mail servers perform optimisations if they see the same IP address being used). The following script creates the interfaces:

for n in `seq 1 254`
  do ifconfig eth0:$n 10.254.0.$n netmask 255.255.255.0
done

Test 1, BIND and MTA on the Same Server

The script in appendix 1 creates the DNS configuration for the 254 zones and the file of email addresses (one per zone) to use as destinations.
I configured the server as a DNS server and a mail relay. A serious mail server will often have a DNS cache running on localhost so for my purposes
having primary zones under example.com configured on a DNS server on localhost seemed appropriate.

I initially tested with only a single thread of Postal connecting to the server. This means that there was no contention on the origin side and it was all on the MTA side. I tested Sendmail and Postfix with an /etc/aliases file expanding to 254 addresses (one per domain). All the messages had the same sender, and the message size was a random value from 0 to 10K.

The following table shows the amount of CPU time used by the server (from top output) and the load average as well as the mail server in use and the number of messages per minute sent through it.

MTA	Msgs/Minute	CPU Use	Load Average
Postfix	15	~70%	9
Postfix	18	~80%	9
Postfix	20	~90%	11
Sendmail	10	~50%	1
Sendmail	13	~70%	2
Sendmail	15	~95%	4.5
Sendmail	20	100%	*

Surprisingly the named process appeared to be using ~10% of the CPU at any given time when running Postfix and 25% of the CPU when running Sendmail (not sure why Sendmail does more DNS work – both MTAs were in fairly default Fedora configurations). As for this operation CPU was the bottleneck it appears that having the named process on the same machine might not be a good optimisation.

When testing 15 and 20 messages per minute with Sendmail the CPU use was higher than with Postfix and in my early tests with 256M of RAM in the kernel started reporting ip_conntrack: table full, dropping packet. which disrupted the tests by deferring connections.

The conntrack errors are because the TCP connection tracking code in the kernel has a fixed number of entries where the default is chosen based on the amount of RAM in the system. With 256M of RAM in the test system the number of connections that could be tracked was just under 15,000. After upgrading the system to 384M of RAM there was support for tracking 24,568 connections and the problem went away. You can change the maximum number of connections by the command echo NUMBER > /proc/sys/net/ipv4/ip_conntrack_max or for a permanent change edit /etc/sysctl.conf and add the line net.ipv4.ip_conntrack_max = NUMBER and then run sysctl -p to load the settings from /etc/sysctl.conf. Note that adding more RAM will increase many system resource limits that affect the operation of the system.

My preferred solution to this problem is to add more RAM because it keeps the machine in a default configuration which decreases the chance of finding a new bug that no-one else has found. Also an extra 128M of RAM is not particularly expensive.

After performing those tests I decided that I needed to add a minimum message size option to Postal (for results that had a lower variance).

I also decided to add an option to specify the list of sender addresses separately from the recipient addresses. When I initially wrote Postal the aim was to test a mail store system. So if you have 100,000 mail boxes then sending mail between them randomly works reasonably well. However for a mail relay system a common test scenario is having two disjoint sets of users for senders and recipients.

Test 2, Analysing DNS Performance

For the second test run I moved the DNS server to the same machine that runs the BHM process which is lightly loaded as the mail relay doesn’t send enough mail to cause BHM to take much CPU time).

I then did a test with Sendmail to see what the performance would be for messages that have a size of exactly 10K for the body which are sent from 254 random sender addresses (one per domain). I noticed that the named process rapidly approached 100% CPU use and was a bottleneck on system performance. It seems that the DNS load for Sendmail is significant!

I then analysed the tcpdump output from the DNS server and saw the following requests:

IP sendmail.34226 > DNS.domain: 61788+ A? a0.example.com. (32)
IP sendmail.34228 > DNS.domain: 22331+ MX? a0.example.com. (32)
IP sendmail.34229 > DNS.domain: 4387+ MX? a0.example.com. (32)
IP sendmail.34229 > DNS.domain: 18834+ A? mail.a0.example.com. (37)

It seems that there are four DNS requests per recipient giving a total of 1016 DNS requests per message. When 15 messages per minute are delivered to 254 recipients that means 254 DNS requests per second plus some extra requests (lookups of the sending IP address etc).

Also one thing I noticed is that Sendmail does a PTR query (reverse DNS lookup) on it’s own IP address for every delivery to a recipient. This added an extra 254 DNS queries to the total for Sendmail. I am sure that I could disable this through Sendmail configuration, but I expect that most people who use Sendmail in production would use the default settings in this regard.

Noticing that the A record is consulted first I wondered whether removing the MX record and having only an A record would change things. The following tcpdump output shows that the same number of requests are sent so it really makes no difference for Sendmail:

IP sendmail.34238 > DNS.domain: 26490+ A? a0.example.com. (32)
IP sendmail.34240 > DNS.domain: 16187+ MX? a0.example.com. (32)
IP sendmail.34240 > DNS.domain: 57339+ A? a0.example.com. (32)
IP sendmail.34240 > DNS.domain: 50474+ A? a0.example.com. (32)

Next I tested Postfix with the same DNS configuration (no MX record) and saw the following packets:

IP postfix.34245 > DNS.domain: 3448+ MX? a0.example.com. (32)
IP postfix.34261 > DNS.domain: 50123+ A? a0.example.com. (32)

The following is the result for testing Postfix with the MX based DNS configuration:

IP postfix.34675 > DNS.domain: 29942+ MX? a0.example.com. (32)
IP postfix.34675 > DNS.domain: 33294+ A? mail.a0.example.com. (37)

It seems that in all cases Postfix does less than half the DNS work that Sendmail does in this regard and as BIND is a bottleneck this means that Sendmail can’t be used. So I excluded Sendmail from all further tests.

Below is the results for the Exim queries for sending the same message, Exim didn’t check whether IPv6 was supported before doing an IPv6 DNS query. I filed a bug report about this and was informed that there is a configuration option to disable AAAA lookups, but it is agreed that looking up an IPv6 entry when there is no IPv6 support on the system (or no support other than link-local addresses) is a bad idea.

IP exim.35992 > DNS.domain: 43702+ MX? a0.example.com. (32)
IP exim.35992 > DNS.domain: 7866+ AAAA? mail.a0.example.com. (37)
IP exim.35992 > DNS.domain: 31399+ A? mail.a0.example.com. (37)

The total number of DNS packets sent and received for each mail server was 2546 for Sendmail, 1525 for Exim, and 1020 for Postfix. Postfix clearly wins in this case for being friendly to the local DNS cache and for not sending pointless IPv6 queries to external DNS servers. For further tests I will use Postfix as I don’t have time to configure a machine that is fast enough to handle the DNS needs of Sendmail.

Exim would also equal Postfix in this regard if configured correctly. However I am making a point of using configurations that are reasonably close to the Fedora defaults as that is similar to the common use on the net.

Test 3 – Postfix Performance

To test Postfix performance I used the DNS server on a separate machine which had the MX records. I decided to test a selection of message sizes to determine the correlation between message size and system load.

Msg Size	Msgs/Minute	CPU Use	Load Average
10K	20	~95%	11
0-1K	20	~85%	7
100K	10	~80%	6

In all the above tests there were a few messages not being sent due to connections timing out. This seems unreasonably poor performance. I had expected Postfix in the most simplistic mailing list configuration to be able to handle more than 5080 outbound messages per minute.

Test 4 – different Ethernet Card

If an Ethernet device driver takes an excessive amount of CPU time in interrupt context then it will be billed to the user-space process that was running at the time, this can result in an application being reported as using a lot more CPU time than it really uses. To check whether that is the case I decided to replace the Ethernet card in the test system and see if that changed the reported CPU use.

I installed a PCI Ethernet card with Intel Corporation 82557/8/9 chipset to use instead of the Ethernet port on the motherboard which had a Intel Corporation 82801BA/BAM/CA/CAM chipset and observed no performance difference. I did not have suitable supplies of spare hardware to test a non-Intel card.

Conclusion

DNS performance is more important to mail servers than I had previously thought. The choice and configuration of the mail server will affect the performance required from local DNS caches and from remote servers. Sendmail is more demanding on DNS servers and Exim needs to be carefully configured to match the requirements.

I am still considering whether it would be more appropriate for Exim to check for IPv4 addresses before checking for IPv6 addresses given that most of the Internet runs on IPv4 only. Maybe a configuration option for this would be appropriate. [After publication it occurred to me that checking for IPv4 first would be bad if you want to migrate to an IPv6 Internet.]

Also other mail servers will have the same issues to face as IPv6 increases in popularity.

The performance of 20 messages per minute doesn’t sound very good, but when you consider the outbound performance it’s more impressive. Every inbound message gets sent to 254 domains, so 20 inbound messages per minute gives 84.6 outbound messages per second on average which is a reasonable number for a low-end machine. Surprisingly there was little disk IO load.

Future Work

The next thing to implement is BHM support for tar pits, gray-listing, randomly dropping connections, temporary deferrals, and generating bounce messages. This will significantly increase the load on the mail server. Administrators of list servers often complain about the effects of being tar-pitted, I plan to do some tests to estimate the performance overhead of this and determine what it means in terms of capacity planning for list administrators.

Another thing I plan to develop is support for arbitrary delays at various points in the SMTP protocol. This will be used for similating some anti-spam measures, and also the effects of an overloaded server which will take a long time to return a SMTP 250 code in response to a complete message. It will be interesting to discover whether making your mail server faster can help the internet at large.

Appendix 1

Script to create DNS configuration

#!/usr/bin/perl
#
# use:
# mkzones.pl 100 a%s.example.com 10.254.0 10.253.0.7
# the above command creates zones a0.example.com to a99.example.com with an
# A record for the mail server having the IP address 10.254.0.X where X is
# a number from 1 to 254 and an NS record
# with the IP address 10.1.2.4
#
# then put the following in your /etc/named.conf
#include "/etc/named.conf.postal";
#
# the file "users" in the current directory will have a sample user list for
# postal
#
my $inclfile = "/etc/named.conf.postal";
open(INCLUDE, ">$inclfile") or die "Can not create $inclfile";
open(USERS, ">users") or die "Can’t create users";
my $zonedir = "/var/named/data";
for(my $i = 0; $i < $ARGV[0]; $i++)
{
my $zonename = sprintf($ARGV[1], $i);
my $filename = "$zonedir/$zonename";
open(ZONE, ">$filename") or die "Can not create $filename";
print INCLUDE "zone \"$zonename\" {\n type master;\n file \"$filename\";\n};\n\n";
print ZONE "\$ORIGIN $zonename.\n\$TTL 86400\n\@ SOA localhost. root.localhost. (\n";
# serial refresh retry expire ttl
print ZONE " 2006092501 36000 3600 604800 86400 )\n";
print ZONE " IN NS ns.$zonename.\n";
print ZONE " IN MX 10 mail.$zonename.\n";
my $final = $i % 254 + 1;
print ZONE "mail IN A $ARGV[2].$final\n";
print ZONE "ns IN A $ARGV[3]\n";
close(ZONE);
print USERS "user$final\@$zonename\n";
}
close(INCLUDE);
close(USERS);