Some Yahoo findings – Let's compare approaches

A few days ago, I posted a comment on a Yahoo delivery issue I encountered. A few persons emailed me to have more info on how we solved this issue at m–x–m. This is the purpose of this post. I will describe how a new company overcame recent delivery issues with Yahoo Mail. I hope this will be useful to others.

Context
I am an engineer for m–x–m, a new email delivery company. We use a number of new ways to deliver emails and we mostly focus on transactional and newsletter emails.

Problem
With no increase in deliveries, nor in user complaints (as inferred from other FBL since we could not get one from Yahoo), we got completely blacklisted on Yahoo with the following message.

Aug 31 14:06:36 dserv128-mtl3 m–x–mail/smtp[8516]: 4D7F314C86A3: host e.mx.mail.yahoo.com[216.39.53.1] refused to talk to me: 421 Message from (xx.xx.xx.xx)temporarily deferred – 4.16.50. Please refer to http://help.yahoo.com/help/us/mail/defer/defer-06.html

We kept having this message and sometimes some messages were delivered… directly in the junk folder.

Therefore I started investigating and understood a few tricks.

What I have understood so far
Some of this points may be out of date. Please share with us any of your insight in the comment.

  1. Yahoo does not offer anymore a FBL. This is the root of most issues.
  2. Yahoo’s filtering system is not compatible with the way most MTA handle their retries. In practice when a minor delivering issue arises, it can sometimes blacklist completely the IP.
  3. Yahoo uses the following graduation when dealing with a temporary blocked IP:

    [TS01] -> [TS02] -> regular deferred message. (you will find these messages in your logs)

    Your goal is to stay at TS01. I did not find a way to go back from TS02 to TS01, so be careful.

  4. When encountering a serious delivery issues, Yahoo wants you stop delivering for four hours. Doing so seems to reset Yahoo’s « reputation counter ».
  5. It seems Yahoo signals sometimes user complaint even without FBL. This happens when you see a « deferred » error message only for a specific user (while other emails to the same MX are still delivered).
  6. Yahoo ends the block progressively, you can send before the end of the four hours but you take the risk to reduce your reputation.
  7. Yahoo filtering algorithms seems to be of the form number of user complaint per unit of time — as opposed to a percentage formula (like Hotmail for instance). For instance, 100 user complaints might trigger a block even if they happened on 100 million emails.

In a nutshell, Yahoo asks for a very specific way of handling delivery. Misconfigured MTA can actually cause a permanent block; the problem is most MTA are configured not to respond well to their temporary blocks.

Solution
I wrote a MTA dedicated for Yahoo delivery (our delivery architecture is very flexible in this aspect). Its main points are:

  1. When presented with a 4XX.*deferred error, it equates it to a user complaints.
  2. When presented with a TS01 message, it stops delivery for four hours on this MX DNS (but it keeps going on the others).
  3. It smoothen deliveries so there is no burst (which might overcome the user complaint quota).
  4. It uses a cluster of « smart shared IP » to ensure delivery even when temporary blocked.
  5. It uses a smart backoff algorithm to increase a lower user complaint per unit of time.

This solved of all issues and even now we keep using this solution. If you encounter further problems with Yahoo, please let us know, we might be able to assist you.

This post was written in collaboration with Krzysztof Jarecki and Nicolas Toper (http://www.m–x–m.net) based on an idea from Krzysztof.

Tags: ,

24 Responses to “Some Yahoo findings – Let's compare approaches”

  1. Seth Weisfeld
    September 15, 2008 at 11:35 am #

    Hey Guys,
    Thanks for the post, haven't had a chance to implement any of the ideas but thanks for sharing!

    Seth

  2. Alex
    September 16, 2008 at 12:17 pm #

    Thanks for the post. I'm a CTO for an ESP and like you guys we are having similar issues. We have a sophisticated EBH (Enhanced bounce handling) system we developed that can adopt to different scenarios. Again thanks for the GREAT post.

  3. Nicolas Toper
    September 16, 2008 at 3:18 pm #

    You are welcome and thanks for the comment. If you need help further help, please contact me.

    Best,
    Nicolas

  4. John Reyna
    October 10, 2008 at 1:39 pm #

    Hey Guys,
    Useful post, I got a question for you, Have you ever dealt with this type of error message from yahoo?

    20081009 154839 S-ERR [013] : eid=xxxxx, sid=xxxxx, ip=xx.xx.xx.xx, sip=xx.xx.xx.xx, sz=0, pt=1, try=1, xxxxx@yahoo.com : 4.7.1 [TS03] All messages from xx.xx.xx.xx will be permanently deferred; Retrying will NOT succeed. See http://postmaster.yahoo.com/421-ts03.html

    Would be great if you can give some pointers on how to deal with this.

  5. Nicolas Toper
    October 11, 2008 at 2:16 am #

    Hi,
    Yes I document it in the post.

    Is it a permanent error (I mean you have it all the time no?)?

  6. Luci
    December 19, 2008 at 2:12 am #

    At work i get 421, 451 errors and for few days on i get the TSO1, what should i do? I dont get it…

  7. Steve
    March 25, 2009 at 9:02 pm #

    I keep getting a 421-ts01 error, but I don't think I've done anything to warrant this.

    Emails are being blocked even when they're not being sent to a yahoo address.

    This is my primary email address – how can I clear my name with Yahoo?

  8. Nicolas Toper
    March 26, 2009 at 4:04 pm #

    Hi,

    If you are not a commercial emailer go on http://postmaster.yahoo.com

    There is a form to fill and they will help you

  9. Chris
    April 30, 2009 at 1:51 pm #

    and if you are commercial mailer? ;)

  10. John Y. Codorniz jr
    May 21, 2009 at 6:30 pm #

    Hi Guys!
    Im having a problem with our email, yahoo automated mail replied the ff: can you please assiste me on how to clear this issue?

    Regards!
    Delay reason: SMTP error from remote mail server after initial connection:
    host e.mx.mail.yahoo.com [216.39.53.1]: 421 4.7.1 [TS03] All messages from 202.86.204.72 will be permanently deferred; Retrying will NOT succeed. See http://postmaster.yahoo.com/421-ts03.html

  11. Nicolas Toper
    May 22, 2009 at 1:45 pm #

    Hi John,

    It is described in this article. Basically change IP, process spam complaint and unsubscription and/or stop the spam hacking.

  12. حرب الملوك
    June 23, 2009 at 5:50 am #

    ok what if im on TS3?

    There is no solution? even if i contact yahoo?

  13. Dhiraj
    July 5, 2009 at 5:56 pm #

    Dear Sir,

    just wondering how do i configure postfix to work in the way you have mentioned above as i am getting the permanent deferred error mentioned below. please advise.

    All messages from 66.90.73.75 will be permanently deferred; Retrying will NOT succeed. See http://postmaster.yahoo.com/421-ts03.html)

  14. Greg
    August 31, 2009 at 7:21 am #

    Hello,
    Thank you for this great post!

    As many others, I'm also on TS03… :(

    You said you wrote a MTA dedicated to Yahoo.
    Which SMTP Server could I use to implement these rules?
    My SMTP Server don't let me implement this kind of rules…

    Did you develop your own software to handle these rules?

    Thank you for your help.

  15. Nicolas Toper
    August 31, 2009 at 10:44 am #

    Hi Greg,

    Yes I developed a SMTP to handle these rules.

    If you really need some Yahoo deliveries capacities, why not use an ESP such as us?

    As many we have a SMTP gateway so you simply have to relay to us (or anyone else) your SMTP server.

    Best,
    Nicolas

  16. Greg
    August 31, 2009 at 5:20 pm #

    In fact, I just launched a small delivery service and got a few customers.
    I'm just starting but I could have a lot more customers in the next weeks.

    Would you be ok to serve as a relay for my Yahoo recipients?
    If so, how many will it cost for 120,000 messages? (to begin)
    Each message is customized to its recipient.
    It's not spam! My lists are opt-in.

    I would prefer integrate the same rules as you, but I think you won't tell me which algorithms to use precisely! :)
    So maybe we could work together! :)
    Regards,
    Greg

  17. Nicolas Toper
    September 4, 2009 at 6:15 am #

    Hi,

    Sorry for the delay.

    I would be happy to work with you but I can also explain you for free how to do it. You can contact me on nico at mxmtech dot net

  18. Roots
    September 13, 2009 at 11:01 pm #

    Hey Krzysztof,

    I have built a similar script to watch my logs and automatically manipulate the live ips. Is there a way to contact you, so that we can compare notes regarding how long (unit of time) you suppress 4XX deferrd ips for. Days, weeks ??

  19. Compare ISA
    June 6, 2010 at 4:48 am #

    I appreciate the concern which is been rose. The things need to be sorted out because it is about the individual but it can be with everyone

  20. shawnmichle
    June 16, 2010 at 10:39 pm #

    I appreciate the concern which is been rose. The things need to be sorted out because it is about the individual but it can be with everyone.The initiative taken for the concern is very serious and need an attention of every one. This is the concern which exists in the society and needs to be eliminated from the society as soon as possible.
    =============================================
    Compare ISAs

  21. Maroc immobilier
    July 8, 2010 at 9:38 am #

    I'm confronted to TS03, very bad. I'm wondering if there's any way to deal with it.
    Best regards

  22. Krzysztof Jarecki
    July 8, 2010 at 10:12 am #

    Do you have an FBL with Yahoo?

    Usually there must be something wrong
    with the practices or there is a problem
    that influences your deliverability that
    needs to be found.

    The more data you have the more likely
    you are to find the cause and fix it.

  23. Maroc immobilier
    July 12, 2010 at 11:24 am #

    I've subscribed for this service, but have not got any reports yet (5 days ago), is that normal ?

  24. Steve
    July 18, 2010 at 8:41 pm #

    How can sometimes, I get TS-02 when FBL is showing NO complaint?

    So, in other words, it is like complaint from user but FBL has *NO* complaint (and I am sure FBL is working)????

UA-9835597-1