A few days ago, I posted a comment on a Yahoo delivery issue I encountered. A few persons emailed me to have more info on how we solved this issue at m--x--m. This is the purpose of this post. I will describe how a new company overcame recent delivery issues with Yahoo Mail. I hope this will be useful to others.

Context
I am an engineer for m--x--m, a new email delivery company. We use a number of new ways to deliver emails and we mostly focus on transactional and newsletter emails.

Problem
With no increase in deliveries, nor in user complaints (as inferred from other FBL since we could not get one from Yahoo), we got completely blacklisted on Yahoo with the following message.

Aug 31 14:06:36 dserv128-mtl3 m--x--mail/smtp[8516]: 4D7F314C86A3: host e.mx.mail.yahoo.com[216.39.53.1] refused to talk to me: 421 Message from (xx.xx.xx.xx)temporarily deferred - 4.16.50. Please refer to http://help.yahoo.com/help/us/mail/defer/defer-06.html

We kept having this message and sometimes some messages were delivered... directly in the junk folder.

Therefore I started investigating and understood a few tricks.

What I have understood so far
Some of this points may be out of date. Please share with us any of your insight in the comment.

1. Yahoo does not offer anymore a FBL. This is the root of most issues.


2. Yahoo's filtering system is not compatible with the way most MTA handle their retries. In practice when a minor delivering issue arises, it can sometimes blacklist completely the IP.
   
3. Yahoo uses the following graduation when dealing with a temporary blocked IP:
   
   

   [TS01] -> [TS02] -> regular deferred message. (you will find these messages in your logs)
   
   

   Your goal is to stay at TS01. I did not find a way to go back from TS02 to TS01, so be careful.



4. When encountering a serious delivery issues, Yahoo wants you stop delivering for four hours. Doing so seems to reset Yahoo's « reputation counter ».


5. It seems Yahoo signals sometimes user complaint even without FBL. This happens when you see a « deferred » error message only for a specific user (while other emails to the same MX are still delivered).


6. Yahoo ends the block progressively, you can send before the end of the four hours but you take the risk to reduce your reputation.


7. Yahoo filtering algorithms seems to be of the form number of user complaint per unit of time -- as opposed to a percentage formula (like Hotmail for instance). For instance, 100 user complaints might trigger a block even if they happened on 100 million emails.

In a nutshell, Yahoo asks for a very specific way of handling delivery. Misconfigured MTA can actually cause a permanent block; the problem is most MTA are configured not to respond well to their temporary blocks.

Solution
I wrote a MTA dedicated for Yahoo delivery (our delivery architecture is very flexible in this aspect). Its main points are:

1. When presented with a 4XX.*deferred error, it equates it to a user complaints.


2. When presented with a TS01 message, it stops delivery for four hours on this MX DNS (but it keeps going on the others).


3. It smoothen deliveries so there is no burst (which might overcome the user complaint quota).


4. It uses a cluster of « smart shared IP » to ensure delivery even when temporary blocked.


5. It uses a smart backoff algorithm to increase a lower user complaint per unit of time.

This solved of all issues and even now we keep using this solution. If you encounter further problems with Yahoo, please let us know, we might be able to assist you.

This post was written in collaboration with Krzysztof Jarecki and Nicolas Toper (http://www.m--x--m.net) based on an idea from Krzysztof.

Emails are sent from a mail server (a MTA) to another. The receiving MTA chooses to accept or not the email, hence deliverability issues. Most antispam filters are based on the idea than a given mail server has a specific probability of sending spam. For instance, a mail server sending Viagra spam has either a security breach (enabling spammer to use it) or is a spam server. In both cases, the receiving MTA should reject emails coming from this server.  (As we all know, this approach has its limitations but this is not the point of this post.) Therefore each IP sending mail is scored with respect to its history.

But using an IP address as the unit (the grain) for scoring introduces some important problems. For instance,

• IP address are usually not « owned » by email senders (whether ESP or corporate customers), hence they are unreliable over time.
• IP address changes over time: new servers are added, some are deprovisioned
• No authentication are « embedded » inside the score.
• A lot of email senders are sending their emails on a lot of different IP addresses.

For instance,

• It is easy to bypass the reputation system to send phishing emails.
• Enabling or deprovisioning a mail server is complicated.
• Shared mail servers are not handled at all in this system: what can you infer from  a shared email servers with different sending patterns?
• An ESP using three different IP addresses can have some emails accepted while others would be rejected.

A system abstracting IP address and binding an email sent to this system would solve these issues. The good news is that it is already here: it is the domain name system or DNS.

Currently SMTP (the email protocol) provides no way to bind an email to a domain name (for instance you can send email from Gmail servers with any domains you want). This is why some new protocols have been issued.  The main ones are SPF, SenderId, Domain Keys and DKIM.

Each has its own set of specificities but they all share some common traits: they bind an email sent to a DNS entry. Because of this binding, antispam filters can score now the domain names instead of the IP address (reflecting for instance a specific sender).

SPF and SenderID are using IP addresses while DK and DKIM are using a public/private key system.

It seems that antispam filters will not stop using IP addresses but the importance of the domain name will grow. In a further post, I will examine what it means for ESP, their customers.