A few days ago, I posted a comment on a Yahoo delivery issue I encountered. A few persons emailed me to have more info on how we solved this issue at m--x--m. This is the purpose of this post. I will describe how a new company overcame recent delivery issues with Yahoo Mail. I hope this will be useful to others.

Context
I am an engineer for m--x--m, a new email delivery company. We use a number of new ways to deliver emails and we mostly focus on transactional and newsletter emails.

Problem
With no increase in deliveries, nor in user complaints (as inferred from other FBL since we could not get one from Yahoo), we got completely blacklisted on Yahoo with the following message.

Aug 31 14:06:36 dserv128-mtl3 m--x--mail/smtp[8516]: 4D7F314C86A3: host e.mx.mail.yahoo.com[216.39.53.1] refused to talk to me: 421 Message from (xx.xx.xx.xx)temporarily deferred - 4.16.50. Please refer to http://help.yahoo.com/help/us/mail/defer/defer-06.html

We kept having this message and sometimes some messages were delivered... directly in the junk folder.

Therefore I started investigating and understood a few tricks.

What I have understood so far
Some of this points may be out of date. Please share with us any of your insight in the comment.

1. Yahoo does not offer anymore a FBL. This is the root of most issues.


2. Yahoo's filtering system is not compatible with the way most MTA handle their retries. In practice when a minor delivering issue arises, it can sometimes blacklist completely the IP.
   
3. Yahoo uses the following graduation when dealing with a temporary blocked IP:
   
   

   [TS01] -> [TS02] -> regular deferred message. (you will find these messages in your logs)
   
   

   Your goal is to stay at TS01. I did not find a way to go back from TS02 to TS01, so be careful.



4. When encountering a serious delivery issues, Yahoo wants you stop delivering for four hours. Doing so seems to reset Yahoo's « reputation counter ».


5. It seems Yahoo signals sometimes user complaint even without FBL. This happens when you see a « deferred » error message only for a specific user (while other emails to the same MX are still delivered).


6. Yahoo ends the block progressively, you can send before the end of the four hours but you take the risk to reduce your reputation.


7. Yahoo filtering algorithms seems to be of the form number of user complaint per unit of time -- as opposed to a percentage formula (like Hotmail for instance). For instance, 100 user complaints might trigger a block even if they happened on 100 million emails.

In a nutshell, Yahoo asks for a very specific way of handling delivery. Misconfigured MTA can actually cause a permanent block; the problem is most MTA are configured not to respond well to their temporary blocks.

Solution
I wrote a MTA dedicated for Yahoo delivery (our delivery architecture is very flexible in this aspect). Its main points are:

1. When presented with a 4XX.*deferred error, it equates it to a user complaints.


2. When presented with a TS01 message, it stops delivery for four hours on this MX DNS (but it keeps going on the others).


3. It smoothen deliveries so there is no burst (which might overcome the user complaint quota).


4. It uses a cluster of « smart shared IP » to ensure delivery even when temporary blocked.


5. It uses a smart backoff algorithm to increase a lower user complaint per unit of time.

This solved of all issues and even now we keep using this solution. If you encounter further problems with Yahoo, please let us know, we might be able to assist you.

This post was written in collaboration with Krzysztof Jarecki and Nicolas Toper (http://www.m--x--m.net) based on an idea from Krzysztof.

Comments

Hey Guys,
Thanks for the post, haven't had a chance to implement any of the ideas but thanks for sharing!

Seth

Thanks for the post. I'm a CTO for an ESP and like you guys we are having similar issues. We have a sophisticated EBH (Enhanced bounce handling) system we developed that can adopt to different scenarios. Again thanks for the GREAT post.

You are welcome and thanks for the comment. If you need help further help, please contact me.

Best,
Nicolas

Hey Guys,
Useful post, I got a question for you, Have you ever dealt with this type of error message from yahoo?

20081009 154839 S-ERR [013] : eid=xxxxx, sid=xxxxx, ip=xx.xx.xx.xx, sip=xx.xx.xx.xx, sz=0, pt=1, try=1, xxxxx@yahoo.com : 4.7.1 [TS03] All messages from xx.xx.xx.xx will be permanently deferred; Retrying will NOT succeed. See http://postmaster.yahoo.com/421-ts03.html

Would be great if you can give some pointers on how to deal with this.

Hi,
Yes I document it in the post.

Is it a permanent error (I mean you have it all the time no?)?

At work i get 421, 451 errors and for few days on i get the TSO1, what should i do? I dont get it...

I keep getting a 421-ts01 error, but I don't think I've done anything to warrant this.

Emails are being blocked even when they're not being sent to a yahoo address.

This is my primary email address - how can I clear my name with Yahoo?

Hi,

If you are not a commercial emailer go on http://postmaster.yahoo.com

There is a form to fill and they will help you

and if you are commercial mailer? ;)

Hi Guys!
Im having a problem with our email, yahoo automated mail replied the ff: can you please assiste me on how to clear this issue?

Regards!
Delay reason: SMTP error from remote mail server after initial connection:
host e.mx.mail.yahoo.com [216.39.53.1]: 421 4.7.1 [TS03] All messages from 202.86.204.72 will be permanently deferred; Retrying will NOT succeed. See http://postmaster.yahoo.com/421-ts03.html

Hi John,

It is described in this article. Basically change IP, process spam complaint and unsubscription and/or stop the spam hacking.

ok what if im on TS3?

There is no solution? even if i contact yahoo?

Dear Sir,

just wondering how do i configure postfix to work in the way you have mentioned above as i am getting the permanent deferred error mentioned below. please advise.

All messages from 66.90.73.75 will be permanently deferred; Retrying will NOT succeed. See http://postmaster.yahoo.com/421-ts03.html)