A few days ago, I posted a comment on a Yahoo delivery issue I encountered. A few persons emailed me to have more info on how we solved this issue at m–x–m. This is the purpose of this post. I will describe how a new company overcame recent delivery issues with Yahoo Mail. I hope this will be useful to others.
I am an engineer for m–x–m, a new email delivery company. We use a number of new ways to deliver emails and we mostly focus on transactional and newsletter emails.
With no increase in deliveries, nor in user complaints (as inferred from other FBL since we could not get one from Yahoo), we got completely blacklisted on Yahoo with the following message.
Aug 31 14:06:36 dserv128-mtl3 m–x–mail/smtp: 4D7F314C86A3: host e.mx.mail.yahoo.com[126.96.36.199] refused to talk to me: 421 Message from (xx.xx.xx.xx)temporarily deferred – 4.16.50. Please refer to http://help.yahoo.com/help/us/mail/defer/defer-06.html
We kept having this message and sometimes some messages were delivered… directly in the junk folder.
Therefore I started investigating and understood a few tricks.
What I have understood so far
Some of this points may be out of date. Please share with us any of your insight in the comment.
- Yahoo does not offer anymore a FBL. This is the root of most issues.
- Yahoo’s filtering system is not compatible with the way most MTA handle their retries. In practice when a minor delivering issue arises, it can sometimes blacklist completely the IP.
- Yahoo uses the following graduation when dealing with a temporary blocked IP:
[TS01] -> [TS02] -> regular deferred message. (you will find these messages in your logs)
Your goal is to stay at TS01. I did not find a way to go back from TS02 to TS01, so be careful.
- When encountering a serious delivery issues, Yahoo wants you stop delivering for four hours. Doing so seems to reset Yahoo’s « reputation counter ».
- It seems Yahoo signals sometimes user complaint even without FBL. This happens when you see a « deferred » error message only for a specific user (while other emails to the same MX are still delivered).
- Yahoo ends the block progressively, you can send before the end of the four hours but you take the risk to reduce your reputation.
- Yahoo filtering algorithms seems to be of the form number of user complaint per unit of time — as opposed to a percentage formula (like Hotmail for instance). For instance, 100 user complaints might trigger a block even if they happened on 100 million emails.
In a nutshell, Yahoo asks for a very specific way of handling delivery. Misconfigured MTA can actually cause a permanent block; the problem is most MTA are configured not to respond well to their temporary blocks.
I wrote a MTA dedicated for Yahoo delivery (our delivery architecture is very flexible in this aspect). Its main points are:
- When presented with a 4XX.*deferred error, it equates it to a user complaints.
- When presented with a TS01 message, it stops delivery for four hours on this MX DNS (but it keeps going on the others).
- It smoothen deliveries so there is no burst (which might overcome the user complaint quota).
- It uses a cluster of « smart shared IP » to ensure delivery even when temporary blocked.
- It uses a smart backoff algorithm to increase a lower user complaint per unit of time.
This solved of all issues and even now we keep using this solution. If you encounter further problems with Yahoo, please let us know, we might be able to assist you.
This post was written in collaboration with Krzysztof Jarecki and Nicolas Toper (http://www.m–x–m.net) based on an idea from Krzysztof.