Defective Semantics

S/MIME is an attractive option for implementing secure messages, since it works OOTB in more MUAs than OpenPGP. However, for an 11-year-old standard, there are a lot of limitations on the lowest-common-denominator of what works.

A good (and fairly ubiquitous) implementation seemed to be the OpenSSL smime(1) command. This has a few quirks:

• It doesn’t use CRLF line endings in the outer message as required by RFC 2822. In newer versions there is an undocumented -crlfeol option, but even this doesn’t work properly. I’ve submitted a patch on their tracker.
• You probably want to use the -text option and not the -binary option when encrypting to fix the CR/LF issues for a text/plain payload.
• To generate certificates non-interactively, you are probably going to need an OpenSSL config file to go along with the req -batch option. Something like:

  [ req ]
  prompt = no
  default_bits = 2048
  extendedKeyUsage=emailProtection
  subjectAltName=email:foo@example.com
  distinguished_name = dn
  req_extensions = req_extensions
  x509_extensions = req_extensions
  
  [ req_extensions ]
  extendedKeyUsage=emailProtection
  basicConstraints=CA:FALSE
  
  [ dn ]
  O=Example Corp
  C=AU
  CN=Example Corp Secured Email <foo@example.com>
  emailAddress=foo@example.com
  

Below is the micro-HOWTO for non-interactively doing stuff (you can use pipes instead of the files in all these instances when driving OpenSSL via IPC). Note that the certificates are read before the message payload, and you want to get the I/O order right to avoid a deadlock.

openssl req -x509 -nodes -days 14 -newkey rsa:1024 \
-keyout private.pem -out public.pem
openssl pkcs12 -export -passout fd:0 -inkey private.pem -in public.pem \
-out private.pfx
openssl smime -encrypt -des3 -text -in payload.txt \
-to 'foo@example.com' -from 'Me <bar@example.net>' \
-subject "An encrypted email" \
-out message.eml public.pem # or pipe stdout to sendmail
openssl smime -decrypt -in message.eml -out payload2.txt \
-inkey private.pem public.pem
# for low-level debugging:
openssl smime -pk7out -in message.eml | openssl asn1parse

On to the MUAs. The certificate import for Apple Mail is pretty painless with the PKCS#12 key being imported by Keychain Access. Later (it didn’t work immediately for me) extra buttons for signing and encrypting automagically appear in the message editing window on the From line, but they are disabled unless your From address has a certificate. Hence you need a certificate yourself, even if you’re not signing, in order to send an encrypted email to a friend.

Apple Mail gets very confused if the CR/LF stuff isn’t correct: it will display the same message as blank, just the base64-encoded S/MIME part, and the correct decoded text on different occasions.

Thunderbird isn’t very friendly toward self-signed certificates. Choosing the .pfx from Preferences > Advanced > View Certificates > Import will get the key to show up, but when you try to encrypt an email, you get an error message like “the application failed to find an encryption certificate for <foo@example.com>”. However, Thunderbird will use the key for decrypting.

Microsoft Outlook and Outlook Express will use the Windows certificate store, which imports a .pfx with a wizard by default when you open one. You can import it from a shell (or more likely, from wine) with rundll32.exe cryptext.dll,CryptExtAddPFX private.pfx, or use the UI from Internet Explorer (Tools > Internet Options > Contents > Certificates).

Apple Mail and Thunderbird both handle multipart/mixed with text/plain and smime-envelope parts correctly (by concatenating the unencrypted and envelope contents in the message window), although they will report the message as unencrypted (which is true I guess, because only one part of the message is). However, Microsoft Outlook won’t display anything for the secure part (not even an error).