Bonnie++

This version starts the re-write of Bonnie++! I will make it totally threaded (the new code does not use fork()). It will also support testing with a specified number of threads doing the same test, this will allow you to reallythrash those RAID arrays!

  • bonnie++-1.98.tgz – Allow specifying the number of random seeks and the number of seeker processes and store that in the CSV (for testing NVMe). Changed the text output to use KiB/MiB/GiB as units of measurement so we can fit NVMe results on screen.
  • bonnie++-1.97.3.tgz – man page fixes, fixed symlink test, and CSS fix for bon_csv2html.
  • bonnie++-1.97.2.tgz – make it build with GCC-6 and fix some Debian bugs.
  • bonnie++-1.97.tgz – fixed a bunch of bugs including bad CSV output.
  • bonnie++-1.96.tgz – fixed a bunch of bugs and got the colors working correctly in the HTML output.
  • bonnie++-1.95.tgz – support direct IO and made some changes to the build process (including dropping NT and OS/2 support).
  • bonnie++-1.94.tgz – major improvements to zcav.
  • bonnie++-1.93d.tgz – added write support to ZCAV.
  • bonnie++-1.93c.tgz – added support for GCC 3.2.
  • bonnie++-1.93b.tgz – version 1.93b. Fixed an error in calculating seek time and added support for large numbers of directories for the file creation tests.
  • bonnie++-1.93a.tgz – version 1.93a. Better support of NT, better RPM packaging code, and a minor warning fix.
  • bonnie++-1.93.tgz – version 1.93. Added a new test program for per-char IO for the people on the linux-kernel mailing list. ;)
  • bonnie++1.92b.tgz – version 1.92b. Fixed a bunch of bugs in the random seed selection code.
  • bonnie++1.92a.tgz – version 1.92a. Added support for setting the seed for random numbers and fixed a few bugs.
  • bonnie++1.92.tgz – version 1.92. Changed the code to do per-char tests with read() and write() instead. Now reports much lower results for those tests which are more useful IMHO.
  • bonnie++1.91c.tgz – version 1.91c, it now compiles with namespace support in GCC 3.0 and fixed some minor bugs.
  • bonnie++1.91b.tgz – version 1.91b
  • bonnie++1.91a.tgz – version 1.91a, fixed zcav properly and fixed a bunch of minor bugs in Bonnie++.
  • bonnie++1.91.tgz – version 1.91, fixes all known bugs in the 1.90 series.
  • bonnie++1.90g.tgz – version 1.90g, now latency works properly and always gets parsed properly. Changed the -f option to allow tests of per-char IO for small amounts of data.
  • bonnie++1.90f.tgz – version 1.90f, more work on latency in bonnie++ and some slight changes for OS/2 and NT portability.
  • bonnie++1.90e.tgz – version 1.90e, better OS/2 and NT support.
  • bonnie++1.90d.tgz – version 1.90d, contains code from the OS/2 and NT ports, may break things for some versions of UNIX.
  • bonnie++1.90c.tgz – version 1.90c, produces better web pages (full color) with bon_csv2html.
  • bonnie++1.90b.tgz – version 1.90b, adds support for measuring latency.
  • bonnie++1.90a.tgz – version 1.90a, adds basic threading, changes the format of the CSV files, and updates the programs for managing CSV files (and the man pages).

ETBE Mon

ETBE Mon is my fork of the traditional Unix system and network monitoring system “Mon” by Jim Trocki. My aim with this fork is to take the best patches available for the existing Mon codebase and the best “contrib” monitoring scripts and include them in one package. I also aim to write all scripts necessary for typical Linux systems. I don’t believe that a working system should involve “mon”, “mon-contrib”, and searching the net for scripts, it should be just the etbemon package.

Support

The Mon mailing list is at https://lists.coker.com.au/cgi-bin/mailman/listinfo/etbemon [1] . Please use the list instead of mailing me directly so other people can benefit from the answers.

I have created a Wiki for documenting etbemon [2].

If you need commercial support for Mon that is available too. As a basic offer for $100US by Paypal I’ll write a basic monitor script for any Linux service or RFC standard service. If there’s any other commercial support you want then just ask.

Downloads

References

Memlockd

Memlockd is a daemon that locks files into memory. Then if a machine starts paging heavily the chance of being able to login successfully is significantly increased. The default configuration will lock all the files needed for login to a Debian GNU/Linux system via the console or via ssh.

Postal

This is a program I wrote to benchmark SMTP servers. I started work on this because I need to know which mail server will give the best performance with more than 1,000,000 users. I have decided to release it under the GPL because there is no benefit in keeping the source secret, and the world needs to know which mail servers perform well and which don’t!

At the OSDC conference in 2006 I presented a paper on mail relay performance based on the new BHM program that is now part of Postal.

I have a Postal category on my main blog that I use for a variety of news related to Postal. This post (which will be updated periodically) will be the main reference page for the software. Please use the comments section for bug reports and feature requests.

It works by taking a list of email addresses to use as FROM and TO addresses. I originally used a template to generate the list of users because if each email address takes 30 bytes of storage then 3,000,000 accounts would take 90M of RAM which would be more than the memory in the test machine I was using at the time. Since that time the RAM size in commodity machines has increased far faster than the size of ISP mail servers so I removed the template feature (which seemed to confuse many people).

When sending the mail the subject and body will be random data. A header field X-Postal will be used so that procmail can easily filter out such email just in case you accidentally put your own email address as one of the test addresses. ;)

I have now added two new programs to the suite, postal-list, and rabid. Postal-list will list all the possible expansions for an
account name (used for creating a list of accounts to create on your test server). Rabid is the mad Biff, it is a POP benchmark.

Postal now adds a MD5 checksum in the header X-PostalHash to all messages it sends (checksum is over the Subject, Date, Message-ID, From, and To headers and the message body including the “\r\n” that ends each line of text in the SMTP protocol). Rabid now checks the MD5 checksum and displays error messages when it doesn’t match.

I have added rate limiting support in Rabid and Postal. This means that you can specify that these programs send a specific number of messages and perform a specific number of POP connections per minute respectively. This should make it easy to determine the amount of system resources that are used by a particular volume of traffic. Also if you want to run performance analysis software to determine what the bottlenecks are on your mail server then you could set Postal and Rabid to only use half the maximum speed (so the CPU and disk usage of the analysis software won’t impact on the mail server).

I will not release a 1.0 version until the following features are implemented:

  • Matching email sent by Postal and mail received by BHM and Rabid to ensure that each message is delivered correctly (no repeats and no corruption)
  • IMAP support in Rabid that works
  • Support for simulating large numbers of source addresses in Postal. This needs to support at least 2^24 addresses so it is entirely impractical to have so many IP addresses permanently assigned to the test machine.
  • Support for simulating slow servers in Postal and BHM (probably reducing TCP window size and delaying read() calls)
  • Making BHM simulate the more common anti-spam measures that are in use to determine the impact that they have on list servers
  • Determining a solution to the problem of benchmarking DNS servers. This may mean just including documentation on how to simulate the use patterns of a mail server using someone else’s DNS benchmark, but may mean writing my own DNS benchmark.

Here are links to download the source:

  • postal-0.72.tgz – made LMTP work and accept TAB as a field delimiter.
  • postal-0.71.tgz – rewrote the md5 checking code and fixed lots of little bugs.
  • postal-0.70.tgz – tidied up the man pages and made it build without SSL support.
  • postal-0.69.tgz – fixed some compile warnings, and really made it compile with GCC 4.3
  • postal-0.68.tgz – fixed some compile warnings, made it compile with GCC 4.3, and I think I made it compile correctly with OpenSolaris.
  • postal-0.67.tgz – changed the license to GPL 3
  • postal-0.66.tgz – made GNUTLS work in BHM and added MessageId to Postal.
  • postal-0.65.tgz – significant improvement, many new features and many bugs fixed!
  • postal-0.62.tgz – Slightly improved the installation documents and made it build with GCC 3.2.
  • postal-0.61.tgz – version 0.61. Fixed the bug with optind that stopped it working on BSD systems, and a few other minor bugs.
  • postal-0.60.tgz – version 0.60. Fixed the POP deletion bug, made it compile with GCC 3.0, and added logging of all network IO to disk.
  • postal-0.59.tgz – version 0.59.
  • postal-0.58.tgz – version 0.58. Added some new autoconf stuff, RPM build support, and the first steps to OS/2 and Win32 portability.
  • postal-0.57.tgz – version 0.57. Fixed lots of trivial bugs and some BSD portability issues.
  • postal-0.56.tgz – version 0.56. Added Solaris package manager support. Made it compile without SSL. Added heaps of autoconf stuff.
  • postal-0.55.tgz – version 0.55. Made Rabid work with POP servers that support the CAPA command. Fixed some compile problems on Solaris.
  • postal-0.54.tgz – version 0.54. Added a ./configure option to turn off name expansion (for systems with buggy regex). Fixed a locking bug that allowed Rabid to access the same account through two threads.
  • postal-0.53.tgz – version 0.53. Don’t use NIS domain name etc for SMTP protocol.
  • postal-0.52.tgz – version 0.52. Better portability with autoconf.
  • postal-0.51.tgz – version 0.51. Supports compiling without SSL and some hacky Solaris support.
  • postal-0.50.tgz – version 0.50. Adds SSL support to Postal (Rabid comes next).

Log Tools

The Logtools package contains a number of programs for managing log files (mainly for web servers).

  • clfmerge will merge a number of Common Logfile Format web log files into a single file while also re-ordering them in a sliding window to cope with web servers that generate log entries with the start-time of the request and write them in order of completion.
  • logprn operates like tail -f but will (after a specified period of inactivity) spawn a process and write the new data in the log file
    to it’s standard input.
  • clfsplit will split up a single CLF format web log into a number of files based on the client’s IP address.
  • funnel will write it’s standard-input to a number of files or processes.
  • clfdomainsplit split a CLF format web log containing fully qualified URLs (including the host name) into separate files, one for each host.

Download:

Maildir Bulletin

This program is designed to deliver bulletin messages to thousands of users on a system. If you want to deliver mail to a large number of people to be read through POP or a local email program (such as mutt) then the traditional approach has been to setup an alias to map to all the users. The problem with this is that mail delivery is very slow and even delivering to 1000 users on a fast machine can take a significant amount of time and system resources. Also if the message is large then you use a lot of disk space.

This program solves that problem by creating a single file with the message data and creating links to it from the ~/Maildir/new directory of every user who is in the group (it delivers mail based on Unix groups). This is fast (can deliver a bulletin to 30000 users in minutes on a slow machine), saves disk space, works with all Maildir client software, and makes it very easy to undeliver or modify a bulletin.

Download links: