A few days ago, I posted a comment on a Yahoo delivery issue I encountered. A few persons emailed me to have more info on how we solved this issue at m–x–m. This is the purpose of this post. I will describe how a new company overcame recent delivery issues with Yahoo Mail. I hope this will be useful to others.
Context
I am an engineer for m–x–m, a new email delivery company. We use a number of new ways to deliver emails and we mostly focus on transactional and newsletter emails.
Problem
With no increase in deliveries, nor in user complaints (as inferred from other FBL since we could not get one from Yahoo), we got completely blacklisted on Yahoo with the following message.
Aug 31 14:06:36 dserv128-mtl3 m–x–mail/smtp[8516]: 4D7F314C86A3: host e.mx.mail.yahoo.com[216.39.53.1] refused to talk to me: 421 Message from (xx.xx.xx.xx)temporarily deferred – 4.16.50. Please refer to http://help.yahoo.com/help/us/mail/defer/defer-06.html
We kept having this message and sometimes some messages were delivered… directly in the junk folder.
Therefore I started investigating and understood a few tricks.
What I have understood so far
Some of this points may be out of date. Please share with us any of your insight in the comment.
- Yahoo does not offer anymore a FBL. This is the root of most issues.
- Yahoo’s filtering system is not compatible with the way most MTA handle their retries. In practice when a minor delivering issue arises, it can sometimes blacklist completely the IP.
- Yahoo uses the following graduation when dealing with a temporary blocked IP:
[TS01] -> [TS02] -> regular deferred message. (you will find these messages in your logs)
Your goal is to stay at TS01. I did not find a way to go back from TS02 to TS01, so be careful.
- When encountering a serious delivery issues, Yahoo wants you stop delivering for four hours. Doing so seems to reset Yahoo’s « reputation counter ».
- It seems Yahoo signals sometimes user complaint even without FBL. This happens when you see a « deferred » error message only for a specific user (while other emails to the same MX are still delivered).
- Yahoo ends the block progressively, you can send before the end of the four hours but you take the risk to reduce your reputation.
- Yahoo filtering algorithms seems to be of the form number of user complaint per unit of time — as opposed to a percentage formula (like Hotmail for instance). For instance, 100 user complaints might trigger a block even if they happened on 100 million emails.
In a nutshell, Yahoo asks for a very specific way of handling delivery. Misconfigured MTA can actually cause a permanent block; the problem is most MTA are configured not to respond well to their temporary blocks.
Solution
I wrote a MTA dedicated for Yahoo delivery (our delivery architecture is very flexible in this aspect). Its main points are:
- When presented with a 4XX.*deferred error, it equates it to a user complaints.
- When presented with a TS01 message, it stops delivery for four hours on this MX DNS (but it keeps going on the others).
- It smoothen deliveries so there is no burst (which might overcome the user complaint quota).
- It uses a cluster of « smart shared IP » to ensure delivery even when temporary blocked.
- It uses a smart backoff algorithm to increase a lower user complaint per unit of time.
This solved of all issues and even now we keep using this solution. If you encounter further problems with Yahoo, please let us know, we might be able to assist you.
This post was written in collaboration with Krzysztof Jarecki and Nicolas Toper (http://www.m–x–m.net) based on an idea from Krzysztof.
Hey Guys,
Thanks for the post, haven't had a chance to implement any of the ideas but thanks for sharing!
Seth
Thanks for the post. I'm a CTO for an ESP and like you guys we are having similar issues. We have a sophisticated EBH (Enhanced bounce handling) system we developed that can adopt to different scenarios. Again thanks for the GREAT post.
You are welcome and thanks for the comment. If you need help further help, please contact me.
Best,
Nicolas
Hey Guys,
Useful post, I got a question for you, Have you ever dealt with this type of error message from yahoo?
20081009 154839 S-ERR [013] : eid=xxxxx, sid=xxxxx, ip=xx.xx.xx.xx, sip=xx.xx.xx.xx, sz=0, pt=1, try=1, xxxxx@yahoo.com : 4.7.1 [TS03] All messages from xx.xx.xx.xx will be permanently deferred; Retrying will NOT succeed. See http://postmaster.yahoo.com/421-ts03.html
Would be great if you can give some pointers on how to deal with this.
Hi,
Yes I document it in the post.
Is it a permanent error (I mean you have it all the time no?)?
At work i get 421, 451 errors and for few days on i get the TSO1, what should i do? I dont get it…
I keep getting a 421-ts01 error, but I don't think I've done anything to warrant this.
Emails are being blocked even when they're not being sent to a yahoo address.
This is my primary email address – how can I clear my name with Yahoo?
Hi,
If you are not a commercial emailer go on http://postmaster.yahoo.com
There is a form to fill and they will help you
and if you are commercial mailer?
Hi Guys!
Im having a problem with our email, yahoo automated mail replied the ff: can you please assiste me on how to clear this issue?
Regards!
Delay reason: SMTP error from remote mail server after initial connection:
host e.mx.mail.yahoo.com [216.39.53.1]: 421 4.7.1 [TS03] All messages from 202.86.204.72 will be permanently deferred; Retrying will NOT succeed. See http://postmaster.yahoo.com/421-ts03.html
Hi John,
It is described in this article. Basically change IP, process spam complaint and unsubscription and/or stop the spam hacking.
ok what if im on TS3?
There is no solution? even if i contact yahoo?
Dear Sir,
just wondering how do i configure postfix to work in the way you have mentioned above as i am getting the permanent deferred error mentioned below. please advise.
All messages from 66.90.73.75 will be permanently deferred; Retrying will NOT succeed. See http://postmaster.yahoo.com/421-ts03.html)
Hello,
Thank you for this great post!
As many others, I'm also on TS03…
You said you wrote a MTA dedicated to Yahoo.
Which SMTP Server could I use to implement these rules?
My SMTP Server don't let me implement this kind of rules…
Did you develop your own software to handle these rules?
Thank you for your help.
Hi Greg,
Yes I developed a SMTP to handle these rules.
If you really need some Yahoo deliveries capacities, why not use an ESP such as us?
As many we have a SMTP gateway so you simply have to relay to us (or anyone else) your SMTP server.
Best,
Nicolas
In fact, I just launched a small delivery service and got a few customers.
I'm just starting but I could have a lot more customers in the next weeks.
Would you be ok to serve as a relay for my Yahoo recipients?
If so, how many will it cost for 120,000 messages? (to begin)
Each message is customized to its recipient.
It's not spam! My lists are opt-in.
I would prefer integrate the same rules as you, but I think you won't tell me which algorithms to use precisely!
So maybe we could work together!
Regards,
Greg
Hi,
Sorry for the delay.
I would be happy to work with you but I can also explain you for free how to do it. You can contact me on nico at mxmtech dot net
Hey Krzysztof,
I have built a similar script to watch my logs and automatically manipulate the live ips. Is there a way to contact you, so that we can compare notes regarding how long (unit of time) you suppress 4XX deferrd ips for. Days, weeks ??
I appreciate the concern which is been rose. The things need to be sorted out because it is about the individual but it can be with everyone
I appreciate the concern which is been rose. The things need to be sorted out because it is about the individual but it can be with everyone.The initiative taken for the concern is very serious and need an attention of every one. This is the concern which exists in the society and needs to be eliminated from the society as soon as possible.
=============================================
Compare ISAs
I'm confronted to TS03, very bad. I'm wondering if there's any way to deal with it.
Best regards
Do you have an FBL with Yahoo?
Usually there must be something wrong
with the practices or there is a problem
that influences your deliverability that
needs to be found.
The more data you have the more likely
you are to find the cause and fix it.
I've subscribed for this service, but have not got any reports yet (5 days ago), is that normal ?
How can sometimes, I get TS-02 when FBL is showing NO complaint?
So, in other words, it is like complaint from user but FBL has *NO* complaint (and I am sure FBL is working)????