This is a program I wrote to benchmark SMTP servers. I started work on this because I need to know which mail server will give the best performance with more than 1,000,000 users. I have decided to release it under the GPL because there is no benefit in keeping the source secret, and the world needs to know which mail servers perform well and which don’t!
At the OSDC conference in 2006 I presented a paper on mail relay performance based on the new BHM program that is now part of Postal.
I have a Postal category on my main blog that I use for a variety of news related to Postal. This post (which will be updated periodically) will be the main reference page for the software. Please use the comments section for bug reports and feature requests.
It works by taking a list of email addresses to use as FROM and TO addresses. I originally used a template to generate the list of users because if each email address takes 30 bytes of storage then 3,000,000 accounts would take 90M of RAM which would be more than the memory in the test machine I was using at the time. Since that time the RAM size in commodity machines has increased far faster than the size of ISP mail servers so I removed the template feature (which seemed to confuse many people).
When sending the mail the subject and body will be random data. A header field X-Postal will be used so that procmail can easily filter out such email just in case you accidentally put your own email address as one of the test addresses. ;)
I have now added two new programs to the suite, postal-list, and rabid. Postal-list will list all the possible expansions for an
account name (used for creating a list of accounts to create on your test server). Rabid is the mad Biff, it is a POP benchmark.
Postal now adds a MD5 checksum in the header X-PostalHash to all messages it sends (checksum is over the Subject, Date, Message-ID, From, and To headers and the message body including the “\r\n” that ends each line of text in the SMTP protocol). Rabid now checks the MD5 checksum and displays error messages when it doesn’t match.
I have added rate limiting support in Rabid and Postal. This means that you can specify that these programs send a specific number of messages and perform a specific number of POP connections per minute respectively. This should make it easy to determine the amount of system resources that are used by a particular volume of traffic. Also if you want to run performance analysis software to determine what the bottlenecks are on your mail server then you could set Postal and Rabid to only use half the maximum speed (so the CPU and disk usage of the analysis software won’t impact on the mail server).
I will not release a 1.0 version until the following features are implemented:
- Matching email sent by Postal and mail received by BHM and Rabid to ensure that each message is delivered correctly (no repeats and no corruption)
- IMAP support in Rabid that works
- Support for simulating large numbers of source addresses in Postal. This needs to support at least 2^24 addresses so it is entirely impractical to have so many IP addresses permanently assigned to the test machine.
- Support for simulating slow servers in Postal and BHM (probably reducing TCP window size and delaying read() calls)
- Making BHM simulate the more common anti-spam measures that are in use to determine the impact that they have on list servers
- Determining a solution to the problem of benchmarking DNS servers. This may mean just including documentation on how to simulate the use patterns of a mail server using someone else’s DNS benchmark, but may mean writing my own DNS benchmark.
Here are links to download the source:
- postal-0.76.tgz – #define PATH_MAX if it’s not already defined, for HURD.
- postal-0.75.tgz – Fix some compiler warnings. Stop autoconf controlling stripping. Add -b option to bind to an address to bhm. Do not #include
when unnecessary.
Stop linking against libgcrypt. Build against GnuTLS v3. Fix buffer underrun. Made it build with GCC-6. - postal-0.73.tgz – Make postal correctly issue the quit command after delivery failure. Fixed a bunch of valgrind errors. Fixed a nasty bug with tab separated fields.
- postal-0.72.tgz – made LMTP work and accept TAB as a field delimiter.
- postal-0.71.tgz – rewrote the md5 checking code and fixed lots of little bugs.
- postal-0.70.tgz – tidied up the man pages and made it build without SSL support.
- postal-0.69.tgz – fixed some compile warnings, and really made it compile with GCC 4.3
- postal-0.68.tgz – fixed some compile warnings, made it compile with GCC 4.3, and I think I made it compile correctly with OpenSolaris.
- postal-0.67.tgz – changed the license to GPL 3
- postal-0.66.tgz – made GNUTLS work in BHM and added MessageId to Postal.
- postal-0.65.tgz – significant improvement, many new features and many bugs fixed!
- postal-0.62.tgz – Slightly improved the installation documents and made it build with GCC 3.2.
- postal-0.61.tgz – version 0.61. Fixed the bug with optind that stopped it working on BSD systems, and a few other minor bugs.
- postal-0.60.tgz – version 0.60. Fixed the POP deletion bug, made it compile with GCC 3.0, and added logging of all network IO to disk.
- postal-0.59.tgz – version 0.59.
- postal-0.58.tgz – version 0.58. Added some new autoconf stuff, RPM build support, and the first steps to OS/2 and Win32 portability.
- postal-0.57.tgz – version 0.57. Fixed lots of trivial bugs and some BSD portability issues.
- postal-0.56.tgz – version 0.56. Added Solaris package manager support. Made it compile without SSL. Added heaps of autoconf stuff.
- postal-0.55.tgz – version 0.55. Made Rabid work with POP servers that support the CAPA command. Fixed some compile problems on Solaris.
- postal-0.54.tgz – version 0.54. Added a ./configure option to turn off name expansion (for systems with buggy regex). Fixed a locking bug that allowed Rabid to access the same account through two threads.
- postal-0.53.tgz – version 0.53. Don’t use NIS domain name etc for SMTP protocol.
- postal-0.52.tgz – version 0.52. Better portability with autoconf.
- postal-0.51.tgz – version 0.51. Supports compiling without SSL and some hacky Solaris support.
- postal-0.50.tgz – version 0.50. Adds SSL support to Postal (Rabid comes next).
Hi Russell,
Do you have any clues yet on what the best performing SMTP servers are?
Thx,
Jeroen
that would be grat to know without installing XX MTAs
hi, Russell,
i can’t compile postal0.7.0&0.6.5 on RHEL5U3, which have same error.
postal.cpp: In function ‘int gcry_pthread_mutex_init(void**)’:
postal.cpp:16: error: invalid conversion from ‘void*’ to ‘pthread_mutex_t*’
postal.cpp: In function ‘int gcry_pthread_mutex_destroy(void**)’:
postal.cpp:16: error: invalid conversion from ‘void*’ to ‘pthread_mutex_t*’
postal.cpp: In function ‘int gcry_pthread_mutex_lock(void**)’:
postal.cpp:16: error: invalid conversion from ‘void*’ to ‘pthread_mutex_t*’
postal.cpp: In function ‘int gcry_pthread_mutex_unlock(void**)’:
postal.cpp:16: error: invalid conversion from ‘void*’ to ‘pthread_mutex_t*’
postal.cpp: At global scope:
postal.cpp:16: warning: missing initializer for member ‘gcry_thread_cbs::read’
postal.cpp:16: warning: missing initializer for member ‘gcry_thread_cbs::write’
postal.cpp:16: warning: missing initializer for member ‘gcry_thread_cbs::select’
postal.cpp:16: warning: missing initializer for member ‘gcry_thread_cbs::waitpid’
postal.cpp:16: warning: missing initializer for member ‘gcry_thread_cbs::accept’
postal.cpp:16: warning: missing initializer for member ‘gcry_thread_cbs::connect’
postal.cpp:16: warning: missing initializer for member ‘gcry_thread_cbs::sendmsg’
postal.cpp:16: warning: missing initializer for member ‘gcry_thread_cbs::recvmsg’
make: *** [postal] Error 1
How to resolve it? Thanks!
I’m having the same issue on Fedora 8. Does anybody know how to resolve it?
Hi Russell,
I’m playing with Postal and Rabid, using the Debian Package on Lenny a instalation and i have a question. Can be disabled de MD5 verification without recompilation?
And i’m experiencing and issue, Rabid doesn’t print the workload values
Thx!
Congratulations for this great tool!
Here is a patch to rabid that I sent Russell a while back, not fully vetted but works for me.
====================
I ran into a problem with rabid trying to log into pop with local name logins. Instead of the second field being sent as the password, the domain of the address was being sent instead.
I changed userlist.ccp and I got it working but I having really gone thru the logic to make sure it all correct.
lars@qadev:postal-0.70 diff -c userlist.cpp.orig userlist.cpp
*** userlist.cpp.orig Tue Jun 10 11:56:28 2008
— userlist.cpp Tue Jun 10 12:10:00 2008
***************
*** 34,44 ****
printf(“Need a password for \”%s\”.”, buf);
continue;
}
if(stripDom)
strtok(buf, “@”);
m_users->push_back(buf);
– if(usePass)
– m_passwords->push_back(buf + strlen(buf) + 1);
if(strlen(buf) > m_maxNameLen)
m_maxNameLen = strlen(buf);
}
— 34,44 —-
printf(“Need a password for \”%s\”.”, buf);
continue;
}
+ if(usePass)
+ m_passwords->push_back(buf + strlen(buf) + 1);
if(stripDom)
strtok(buf, “@”);
m_users->push_back(buf);
if(strlen(buf) > m_maxNameLen)
m_maxNameLen = strlen(buf);
}
——————————————————-
Hi Russell,
I just tested out Postal 0.70 which I installed using the debian repository. It is working very well. I still have to start using all the features of Postal but thought that I should thank you before that. :). Thanks a lot for writing such a wonderful tool. I will soon try to contribute to it myself.
Thanks again.
Regards,
Vignesh
Good job my friend. Thx for this great tool!
Hi Russell,
Thanks for the smtp load tool. I can understand that this tool can be used to generate the required throughput on the mail server. But is there a way to measure the latency for mail relays (i.e the time it from when the message was sent by postal to the time when it was recieved by BHM). One way i can think of this is to stamp the epoch time onto a header in postal and them when the BHM receives the mail, parse for that header and generate the latency time, before sinking it.
do you think this can be done as an enhancement for the tool?
thanks
_vinaya
vinaya: Well the header already has a Date: field, so I guess all that’s needed is to have BHM parse it and record the data.
Also I’m thinking of putting a hash of the message in the X-Postal header so that BHM (or other tools) can parse it. What do you think?
Very interesting tool. I was looking for it and then I reach this page. Great work !
Erl.
Hi Russell,
congratulations for your work.
When it will exit the new version 1.0?
I have a great need to do a test on an imap server with rabid ….
Great work!
I tried to run rabid with some option. But, no result comes up on my screen.
How long does it take to see result? I heard the result comes up every 1 minutes.
I execute rabid like this.
# rabid -r 5 -p 1 -c 5 localhost user-list.filname
There are only 3 user’s name in user-list.filname file.
There should be a result listed every minute.
Postal on BSD seems pretty solid – esp. for hammering Exchange boxen, but rabid seems to be running in imap mode by default and won’t do pop – causing authentication errors.
I’m trying to find the flags to build it on RHEL 6.1, but I keep running into this error:
g++ -O2 -g -Wall -W -Wshadow -Wpointer-arith -Wcast-align -Wwrite-strings -Wcast-qual -pedantic -Woverloaded-virtual -ffor-scope -c tcp.cpp
tcp.cpp: In member function âint tcp::ConnectTLS()â:
tcp.cpp:181: error: invalid conversion from âconst SSL_METHOD*â to âSSL_METHOD*â
make: *** [tcp.o] Error 1
Suggestions?
I’m running .72, on Ubuntu 11.14
For me, postal works, and gives output:
henryb@thebrock:~/MailPerformance$ postal -m 1 -t 2 -c 2 -r 10 mail.optusnet.com.au user-list-filename
time,messages,data(K),errors,connections,SSL connections
16:45,5,5,0,4,0
16:46,10,9,0,7,0
rabid works (mails are getting popped), but I see not output each minute:
henryb@thebrock:~/MailPerformance$ rabid -u -r 2 -c 1 -p 2 imaptest01.syd user-list-filename
time,messages,data(K),errors,connections,SSL connections,IMAP connections
I’ll go digging in the source tomorrow, but any ideas on why that’s happening?
Like Brock, I am also not getting any stats output from rabid.
Instead I’m getting a lot of these:
=====snip=====
MD5 mis-match, calculated:cc9dea70ac3c6b528f789f85b67b462c, expected e086c473f7148f75ec1f3542923c7a81!
Account name testing-user1384@domain3.b2b.ph
MD5 mis-match, calculated:ec2dc98760735d3ff81c4c41e29ebefb, expected 629ceb71142f1729421e184bb2139b82!
Account name testing-user1384@domain3.b2b.ph
MD5 mis-match, calculated:2261d9792b78316c0b9e54f6d3a4e1f3, expected 15a6c72ed44fecd1c425bb6c5972c3b7!
Account name testing-user1384@domain3.b2b.ph
=====snip=====
Any clues how to fix this?
Downloaded, compiled, and installed version 0.72
I don’t get the MD5 errors anymore, but I still don’t see any output.
I previously reported receiving no output.
The problem has gone away now, but I believe it to be something to do with throughput. If there are not enough emails for Rabid to pop, it doesn’t report anything. Or something like that.
If I have postal running, and populating the mailboxes with enough mails for Rabid to pop, then I get the expected output from both postal and rabid.
— Brock
Hello everyone
Tool seems really nice. But I am not getting how we are comparing mail servers performance int his case since i dont see any response time metric or any toher similar thing which can illstrate thta mail server is overloaded or resources are getting exhausted. One way i think by seeing the number of errors to go. Is ther any way i can get response time or any other similar metric?
Thanks And Regards
Ankit Anand
Hello,
I’ve also no output from rabid issue. How to generate some logs?
Please,
tell me what is the clue to use -pedantic compiler flag with not allowed data type:
In file included from results.cpp:3:
results.h:36: error: ISO C++ 1998 does not support âlong longâ
TIA for Your comment. Best regards, Hubert.