Category: Internet

The Problem

DKIM is a standard for digitally signing mail to prove it’s authenticity and prove that it was not modified. In an ideal situation it can be used to reject corrupted or forged messages.

DMARC and ADSP are standards for indicating that mail from a domain should be signed. This prevents hostile parties from removing the DKIM signature and modifying the message. DKIM is only half as useful without them (it can still prove authenticity but it can’t prove that mail was forged and allow rejecting forged mail).

A mailing list is a software system that receives a message from one person and then generates messages to many people with the same content. A common setting of a mailing list is to insert “[listname]” at the start of the subject line of each message that goes through, this breaks the DKIM signature. Another common setting is to append a footer to the message giving information about the list, this breaks the DKIM signature unless the signature uses the “l=” flag (which Gmail doesn’t). When the “l=” flag is used a hostile party can append text to a message without the signature breaking which is often undesired. Mailman (one of the most common mailing list systems) parses and regenerates headers, so it can break DKIM signatures on messages with unexpected header formatting. Mailman also in some situations uses a different MIME encoding for the body which breaks DKIM signatures.

It seems almost impossible to reliably get all mail to go through a Mailman list without something happening to it that breaks DKIM signatures. The problem is that Mailman doesn’t just send the message through, it creates new messages with new headers (created from a parsed copy of the original headers not copying the original headers), and it sometimes parses and re-encodes the body. Even if you don’t choose to use the features for appending a message footer or changing the subject DKIM signatures will often be broken.

Stripping the Signatures

As there is no way to reliably (IE for every message from every sending domain that uses DKIM) pass through messages with DKIM signatures intact the only option is to strip them. To do that with Mailman edit /etc/mailman/mm_cfg.py, add the directive “REMOVE_DKIM_HEADERS = Yes“, and then restart Mailman. If none of the people who send to your list used DMARC or ADSP then that solves your problem.

However if there are senders who use DMARC or ADSP and recipients who check those features then mail will still be rejected and users will get unsubscribed. When DMARC or ADSP are in use the mailing list can’t send out list mail purporting to be from a list member, it must send out mail from it’s own domain.

A Legitimate From Field

In the web based configuration for Mailman there is a dmarc_moderation_action setting that can munge the From field on messages with a DMARC policy. One thing to note is that when one list uses the dmarc_moderation_action setting it causes DKIM users to configure DMARC which then makes more problems for the people who run lists with no settings for DKIM. Also that doesn’t solve things for ADSP messages or messages that don’t use either DMARC or ADSP. It’s not uncommon for people to have special configuration to prevent forged mail from their own domain, requiring a valid DKIM signature is one way of doing this. Finally many users of DKIM enabled mail servers don’t have the option of enabling DMARC.

If you use the from_is_list setting in the web based configuration for Mailman then all mail will have a From field which shows who the message is from as well as the fact that it came From a list server. This combined with REMOVE_DKIM_HEADERS will allow DKIM signed mail sent to the list to go through correctly in all cases.

If you run many lists then changing them all through the web interface can be tedious. Below is a sample of shell code that will use the Mailman config_list program to change the settings to use from_is_list. NB I haven’t actually run this on a Mailman server with lots of lists so check it before you use it, consider it pseudo-code.

for n in lista listb listc ; do
  config_list -o /tmp/$n $n
  sed -i -e "s/from_is_list = 0/from_is_list = 1/" /tmp/$n
  config_list -i /tmp/$n $n
done

The from_is_list setting makes a change like the following:
-From: Russell Coker <russell at coker.com.au>
+From: Russell Coker via linux-aus <linux-aus at lists.linux.org.au>

SPF

There are similar problems with SPF and other anti-forgery methods. The use of from_is_list solves them too.

Signing List Mail

An ideal list configuration has the list server checking DKIM signatures and DMARC settings before receiving mail. There is normally no reason for a mailing list to send mail to another mailing list so mail that the list server receives should pass all DKIM, DMARC, and ADSP checks. Then the list server should send mail out with it’s own DKIM signature.

When a user receives mail from the list they can verify that the DKIM signature is valid. Then if they know that the sender used DKIM (EG the mail originated from gmail.com or another domain that’s well known to use DKIM) then they know that it was verified at the list server and therefore as long as the list server was not compromised the message was not altered from what the sender wrote.

Resources

The Debian Wiki page about OpenDKIM is worth reading [1]. OpenDKIM is generally regarded as the best free software DKIM verification and signing daemon available. The Debian Wiki only documents how to install it with Postfix but the milter interface is used by other MTAs so it shouldn’t be too hard to get it working with other MTAs. Also the Debian Wiki documents the “relaxed” setting which will in some situations solve some of the problems with Mailman munging messages, but it doesn’t guarantee that they will all be solved. Also in most cases it’s not possible to get every user of your list to change the settings of their DKIM signing to “relaxed” for the convenience of the list admin.

The Mailman Wiki page about DMARC [2] and the Mailman Wiki page about DKIM [3] are both good resources. But this article summarises all you really need to know to get things working.

Here is an example of how to use SpamAssassin to score DKIM signatures and give a negative weight to mail from lists that are known to have problems [4]. Forcing list users to do this means more work overall than just having the list master configure the list server to pass DKIM checks.

• [1] https://wiki.debian.org/OpenDKIM
• [2] https://wiki.list.org/DEV/DMARC
• [3] https://wiki.list.org/DEV/DKIM
• [4] http://lists.linux.org.au/pipermail/linux-aus/2017-April/023057.html

POP (Post Office Protocol) is the most used protocol for receiving mail from a server to a MUA (Mail User Agent) for reading. It is specified in RFC1939.

But the way it works (in most cases) is quite simple and doesn’t require reading the RFC, connect to port 110 (the standard port for POP3) and a basic session transcript is as follows (data sent by the client is prefixed with C: and data sent by the server is prefixed with S:):
S:+OK POP3 Ready [HOST NAME]
C:user ABC
S:+OK USER ABC set, mate
C:pass asecret
S:+OK Mailbox locked and ready
C:list
S:+OK scan listing follows
S:1 2989
S:.
C:quit
S:+OK

When the server successfully completes an operation it will precede it’s response with “+OK“, when it fails it will precede it’s response with “-ERR“. The data after the OK or ERR statement is for humans not machines, so in most cases your MUA will discard it. Therefore connecting to the service manually is required to properly debug problems. The unfortunate thing is that often on big mail servers it takes time for the sys-admin to do such tests. If the user can do it for them and give a bug report saying “your POP server said -ERR user unknown” then things will get fixed a lot faster than if the report is “the POP server didn’t work”.

One thing that is quite important is the initial greeting string, on any system of moderate size you will have multiple back-end servers and the greeting will tell you which server you are connecting to. If POP sometimes works and sometimes fails then your ISP might have one server failing so making a note of this greeting string in a transcript of a failed session can really help in tracking down problems.

When the list of messages is displayed, the first column is message numbers (starting at one and going up sequentially) and the second column is message sizes. If you have a POP session timing out and you have an extremely large message then that might be the cause.

A commonly used program for testing POP (and other Internet services) is telnet. So start the above process you would type telnet mail.example.com 110.

There are methods of hashing POP passwords (which make things a little more complex), but they often aren’t used – and in any case don’t encrypt the data. So it’s common to run POP servers with SSL, and the standard port for this is 995. This makes testing a little more complicated (but actually no more difficult).

To make an SSL connection you can use the program stunnel, it is included in many (most?) distributions of Linux, and Windows binaries are apparently at this link (NB I’ve never tested the Windows binaries as I don’t use Windows).

The command stunnel -c -r mail.example.com:995 will connect you to your mail server via SSL and you can then type in the POP commands as normal.

If your POP server supports the STLS command (which allows negotiation of TLS/SSL on port 110) then you can use the command stunnel -n pop3 -c -r mail.example.com:110.

To use gnutls, you can use the command “gnutls-cli mail.example.com -p 995” or to work with STLS on port 110 you use the command “gnutls-cli -s mail.example.com -p 110” and press ^D after entering the STLS command.

The first thing to test is a TLS (aka SSL) connection. The stunnel program has special code for this, the command “stunnel -n smtp -c -r mail.example.com:25” will connect to the server via SMTP and negotiate SSL.

If you use gnutls then the command “gnutls-cli -s mail.example.com -p 25” will connect to the server, allow you to establish the session (by typing “ehlo hostname” and then “starttls“) after which you can press ^D to enter TLS mode. This is a little more inconvenient.

Once one of these is done and you will receive a 220 message acknowledging the connection (which is the same as if you had just connected without TLS). If you want to test the TLS certificate then use the “-v” option to stunnel. Note that if the certificate is not verified successfully then stunnel will exit and log via syslog the reason why. While stunnel seems more convenient for actually using a protocol, the openssl utility is a much better program for actually testing out the SSL functionality. The command “openssl s_client -CApath /etc/ssl/certs/ -starttls smtp -connect mail.example.com:25” will dump a lot of diagnostic information about the SSL protocol. Note that the location of the SSL certificates varies by distribution, /etc/ssl/certs is the location used on Debian.

When compared to openssl and stunnel, gnutls-cli is less convenient than stunnel, and somewhere between the other two in terms of utility for debugging. It’s good to have all three clients available for testing!

Then enter the command “ehlo hostname.example.com” (the hostname is generally not checked for the case of mail relaying so any text that vaguely resembles a real host DNS name will do).

The response to that command will be something like the following:
250-mail.example.com Hello hostname.example.com [10.10.10.10], pleased to meet you
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-8BITMIME
250-SIZE
250-DSN
250-AUTH LOGIN PLAIN
250-DELIVERBY
250 HELP

The important thing to note is the 250-AUTH message which indicates that you may authenticate, it tells us that you can use the LOGIN and PLAIN methods of authentication. All the further communication for the login will be base64 encoded, the best utilities that I know of in Debian/Etch for encoding and decoding base64 are /usr/share/fml/bin/base64encode.pl and /usr/share/fml/bin/base64decode.pl which are in the fml package. Debian/Lenny and newer have base64 as part of the coreutils package.

The command auth login will typically give the response “334 VXNlcm5hbWU6“, the command “echo VXNlcm5hbWU6|/usr/share/fml/bin/base64decode.pl” shows that it is requesting the “Username:“.

To generate a response to the Username prompt run the command “echo -n user@example.com | /usr/share/fml/bin/base64encode.pl” (or whatever your user-name is) and you will receive an encoded message such as “dXNlckBleGFtcGxlLmNvbQ==“. Enter that to the mail server and you will get a response with another 334 code similar to “334 UGFzc3dvcmQ6“, again if you decode the part after the space you will br prompted for the “Password:“. The command “echo -n mypass | /usr/share/fml/bin/base64encode.pl” will give a response that you can give to that prompt. If all goes well that will give a 235 message to tell you that you are authenticated. Then you can relay mail!

When relaying mail after authenticating using SASL, if the mail is authenticated then you can use the auth parameter. This means that instead of using the SMTP command “mail from: <user@example.com>” you use the command “mail from: <user@example.com> auth=<user@example.com>“.

Normally this will all be done by your MUA, but if something goes wrong and you don’t know why then manually running through the steps can reveal the source of the problem.

I’ve been thinking about what we need in Australia to preserve the free software community in the face of an economic collapse (let’s not pretend that the US can collapse without taking Australia down too). For current practices of using the Internet and developing free software to continue it seems that we need the following essential things:

1. Decentralised local net access.
2. Reliable electricity.
3. Viable DNS in the face of transient outages.
4. Reasonably efficient email distribution (any message should be delivered in 48 hours).
5. Distribution of bulk binary data (CD and DVD images of distributions). This includes providing access at short notice.
6. Mailing lists to provide assistance which are available with rapid turn-around (measured in minutes).
7. Good access to development resources (test machines).
8. Reliable and fast access to Wikipedia data.
9. A secure infrastructure.

To solve these I think we need the following:

1. For decentralised local net access wireless is the only option at this time. Currently in Australia only big companies such as Telstra can legally install wires that cross property boundaries. While it’s not uncommon for someone to put an Ethernet cable over their fence to play network games with their neighbors or to share net access, this can’t work across roads and is quite difficult when there are intervening houses of people who aren’t interested. When the collapse happens the laws restricting cabling will all be ignored. But it seems that a wireless backbone is necessary to then permit Ethernet in small local areas. There is a free wireless project wireless.org.au to promote and support such developments.
2. Reliable electricity is only needed for server systems and backbone routers. The current model of having huge numbers of home PCs running 24*7 is not going to last. There are currently a variety of government incentives for getting solar power at home. The cheapest way of doing this is for a grid-connected system which feeds excess capacity into the grid – the down-side of this is that when grid power goes off it shuts down entirely to avoid the risk of injuring an employee of the electricity company. But once solar panels are installed it should not be difficult to convert them to off-grid operation for server rooms (which will be based in private homes). It will be good if home-based wind power generation becomes viable before the crunch comes, server locations powered by wind and solar power will need smaller batteries to operate overnight.
   In the third-world it’s not uncommon to transport electricity by carrying around car batteries. I wonder whether something similar could be done in a post-collapse first-world country with UPS batteries.
   Buying UPSs for all machines is a good thing to do now, when the crunch comes such things will be quite expensive. Also buying energy-efficient machines is a good idea.
3. DNS is the most important service on the net. The current practice is for client machines to use a DNS cache at their ISP. For operation with severe and ongoing unreliability in the net we would need a hierarchy of caching DNS servers to increase the probability of getting a good response to a DNS request even if the client can’t ping the DNS server in question. One requirement would be the use of DNS cache programs which store their data on disk (so that a transient power outage on a machine which has no UPS won’t wipe out the cache), one such DNS cache is pdnsd [1] (I recall that there were others but I can’t find them at the moment). Even if pdnsd is not the ideal product for such tasks it’s a good proof of concept and a replacement could be written quickly enough.
4. For reasonably efficient email distribution we would need at a minimum a distribution of secondary MX records. If reliable connections end to end aren’t possible then outbound smart-hosts serving geographic regions could connect to secondary MX servers in the recipient region (similar to the way most mail servers in Melbourne used to use a university server as a secondary MX until the sys-admin of that server got fed up with it and started bouncing such mail). Of course this would break some anti-spam measures and force other anti-spam measures to be more local in scope. But as most spam is financed from the US it seems likely that a reduction in spam will be one positive result of an economic crash in the US.
   It seems likely that UUCP [2] will once again be used for a significant volume of email traffic. It is a good reliable way of tunneling email over multiple hops between hosts which have no direct connectivity.
   Another thing that needs to be done to alleviate the email problem is to have local lists which broadcast traffic from major international lists, this used to be a common practice in the early 90’s but when bandwidth increased and the level of clue on the net decreased the list managers wanted more direct control to allow them to remove bad addresses more easily.
5. Distributing bulk data requires local mirrors. Mirroring the CD and DVD images of major distributions is easy enough. Mirroring other large data (such as the talks from www.ted.com) will be more difficult and require more manual intervention. Fortunately 750G disks are quite cheap nowadays and we can only expect disks to become larger and cheaper in the near future. Using large USB storage devices to swap data at LUG meetings is a viable option (we used to swap data on floppy disks many years ago). Transferring CD images over wireless links is possible, but not desirable.
6. Local LUG mailing lists are a very important support channel, the quality and quantity of local mailing lists is not as high as it might be due to the competition with international lists. But if a post to an international list could be expected to take five days to get a response then there would be a much greater desire to use local lists. Setting up local list servers is not overly difficult.
7. Access to test machines is best provided by shared servers. Currently most people who do any serious free software development have collections of test machines. But restrictions in the electricity supply will make this difficult. Fortunately virtualisation technologies are advancing well, much of my testing could be done with a few DomU’s on someone else’s server (I already do most of my testing with local DomU’s which has allowed me to significantly decrease the amount of electricity I use).
   Another challenge with test machines is getting the data. At the moment if I want to test my software on a different distribution it’s quite easy for me to use my cable link to download a DVD image. But when using a wireless link this isn’t going to work well. Using ssh to connect to a server that someone else runs over a wireless link would be a much more viable option than trying to download the distribution DVD image over wireless!
8. Wikipedia.org is such an important resource that access to it provides a significant difference in the ability to perform many tasks. Fortunately they offer CD images of Wikipedia which can be downloaded and shared [3].
9. I believe that it will be more important to have secure computers after the crunch, because there will be less capacity for overhead. Currently when extra hardware is required due to DOS attacks and systems need to be reinstalled we have the resources to do so. Due to the impending lack of resources we need to make things as reliable as possible so that they don’t need to be fixed as often, this requires making computers more secure.

• [1] http://www.phys.uu.nl/~rombouts/pdnsd/faq.html
• [2] http://en.wikipedia.org/wiki/Uucp
• [3] http://en.wikipedia.org/wiki/Wikipedia:Wikipedia-CD/Download

One problem many people encounter is the fact that they don’t appear on Google and other search engines the way that they like. If you have an uncommon name which is not referenced in any popular web pages then a single mention in a popular site can immediately become the top hit, this may not be the post you want (EG you send a dirty joke to some friends and it ends up on the archive of a popular mailing list). If you have a more common name then you may want to compete with other people who share the same name with similar problems. The solution to these problems is known as Search Engine Optimisation (or SEO).

1. Recently to improve things in this regard I have registered the domain RussellCoker.com. Domains that end in .com are often regarded as authoritative sources of information on the topic in question, and are also often typed in to browsers. If you visit the RussellCoker.com domain then you’ll find that it’s quite bare, just links to some other things that I do. I will make it a little more user-friendly but the main aim is just having links.
2. One thing that I have been doing for almost a decade is to include URLs of some of my web pages in my signature of email that I send. When I send messages to mailing lists with public archives (or when other people forward my mail to such lists) these links will become visible to search engines. So I now have thousands of links to my web site from hundreds of sites all over the world. Some of these links are old (which may make them more highly rated by search engines). This not only gives benefits for direct hits from search engines, but when people search for terms that are relevant to my work they will often hit archives of my mailing list postings, and then they will often see a benefit in visiting my web site.

Most strategies of SEO that apply to corporate web sites will also apply to individuals (do a web search and you’ll find many pages of advice on these issues). But the above two are the only ones I have discovered which apply specifically to individuals (registering an appropriate .com domain name if possible is a standard practice for companies and something that is done long before SEO is considered).

In my work I often receive problem reports from clients regarding their email service, here is some advice to help get the problem fixed as fast as possible:

Firstly problems sending mail and problems receiving mail are often unconnected and should be reported separately. If only one aspect of a problem is reported then probably only one will be fixed… Also receiving mail via POP or IMAP is separate to the mail transfer process.

For problems on mail transfer the following points should be addressed in any report to increase the speed of problem resolution. Don’t think of this as being for the benefit of the person who fixes your mail server, think of it as being for your own benefit in getting your email working again rapidly.

1. Make sure that every complaint has a specific example. Saying “email doesn’t work for everyone” is not helpful, often when investigating some complaints I discover that email works for many people. Saying “email doesn’t work for John Smith and other people report problems too” is helpful, I can investigate John’s problem while knowing that it’s a symptom of a larger problem.
2. For every report make sure that you list a sender and recipient email address for an instance of the problem. Saying that “alpha@example.com can’t send mail to bravo@example.com” is much more useful than saying “alpha@example.com can’t send mail to some people“.
3. Whenever possible make sure that the problem can be reproduced by the person who is making the report. A report such as “John Smith can’t send email from his PC” is not nearly as helpful as “I can’t send email from my PC“, the reason is that I often require more information from the person who experienced the problem and going through a third-party slows things down.
4. Make a careful note of the time of the problem. If you report that alpha@example.com can’t send mail to bravo@example.com and the logs show that such a message was correctly delivered at 9AM I will assume that the problem was mistakenly reported and not that maybe there was a problem which first occurred at 9:30AM.
5. When noting the time make sure you note any discrepancies. If a problem occurs at 9AM and a bounce message reports a problem occurring at 10AM then please make sure to note both times when reporting the problem. Either or both times could be referenced in the log files and the more information you provide the better the chance that the problem will be fixed quickly.
6. Most serious errors in email delivery have a code numbered from 550 to 559 with an error message. Any time you see a bounce message which includes text such as “Remote host said: 554” then please report all the rest of the text on that line, it will often tell me precisely what needs to be done to fix the problem.
7. Any time a bounce or other error message includes a URL then please report it to me. Sometimes it takes me a significant amount of work to discover what I could have learned in a matter of seconds if I had the URL.

Prior to wide-spread use of the Internet it was reasonable to expect that if you had a family name that was not really common and a given name that was also not really common then you might never meet someone else with the same combination of first and last names as yourself. I have never met anyone else named Russell Coker because Coker is not a really common family name and it seems that Russell was not overly common in the last 4+ decades. If you had a family name such as Smith, Ng, or Ali then the chance of having someone else with the same given name would be significant as those family names are so incredibly popular.

Even with names that are not known to be extremely popular name clashes happen, for example my sister is named Helen Coker, it doesn’t seem likely to be an overly common combination however there was another Helen Coker at the same high-school.

On the Internet however things are different. About 10 years ago I received an email from someone named Russell Coker who just wanted to say hello after having searched for his own name and found information on me. Shortly after that I found a third Russell Coker on the net (who didn’t respond to my email). It seems likely that even in the early days of the net I didn’t find everyone named Russell Coker, and it also seems quite likely that the number of Russell Coker’s on the net has increased significantly over the last 10 years. There could be dozens of Russell Coker’s although if they are using “Russell Coker” as their name on the net then I am unlikely to find them as it’s most likely that I’m the only Russell Coker to have the same email address and web site for 10 years – so my net presence probably squashes everyone else who uses that as their entire name.

A quick google search for “Russell J Coker” (John is my middle name) found one other person using that name, I also found a Russell D Coker, a Russell M Coker, a Russell N Coker, and two Russell R Coker’s (one of whom died in 2002).

It seems that if you don’t have a family name that is rarely used (EG Torvalds which was created recently) or a made-up first-name (which seems to be a popular trend nowadays) then you are going to encounter other people with the same name as you. Using an initial will significantly reduce the incidence of this, in my case if I used my middle initial then I would remove four out of five of the other living Russell Coker’s that are found by Google.

But as is the case with me it seems that many (most?) people will have name collisions no matter what they do.

As an attempt to increase my control over my name I have registered the domain RussellCoker.com. Not sure if it will do any good but it’s cheap enough and it’s worth a try.