Reposted from Pivotal IQ Blog

At one time or other we've all asked that age old question what's in a name? Well it's not an old age any more, it's the 21st century and the question to ask today is "what's in a path?" In a word, everything. In a few more words, the final destination of your email.

To borrow a little Jack Lalanne here, content is king and architecture is queen but together you have the entire kingdom, or more importantly the inbox. We know for a fact that content and spam filters take a close look at your p's and q's to determine if the message in your email is fit for general consumption. We know that every element in your message contributes to the overall likelihood of your email being filtered to the inbox vs. the spam folder. However I want to look at one specific component of your email's content that may not get the attention it should: the image path and URL.

An image path is simply a pointer to where the image lives out on the internet. Email marketing is sent without any attachments so the images that are rendered in the customer's inbox are by way of image paths or URLs to the images. Easy enough, this has been the way it's done since like forever, or at least 1999. But from to time I've been seeing people forget that the actual link itself is as important as the image it leads to. Let me give you an example:

<img src="http://www.foo.com/images/banners/image.jpg">  >>> What's the problem with this image link?

If you said nothing you're wrong. There is a problem, it's the word banner. ISP filters and other technology deployed across the internet such as Spamassassin or Privoxy, have complicated rule sets designed to block banners, advertising and all other manner of communication deemed to be UCE (unsolicited commercial email). By naming the folder that contains your image "banner" you are increasing the likelihood that your image, if not your entire email will be outright blocked because you've declared it as a banner in so far as a spam filter is concerned.

The idea here is to have intelligent and easy to read paths that don't attempt to obfuscate what's in them, but also take into account the imperfect nature of filtering technology. Avoid using the word banner or variations thereof, in an image path, or for that matter "ad(s)'.

The problems don't stop here, no the rabbit hole goes much deeper. We live in a fast paced world defined, at times, by 140 characters. Our lives move too quickly for war & peace, unless it can be squeezed in between 60 minutes and grey's anatomy, so we short-hand a lot of stuff including paths in our URLs, such as this:

<a href="http://www.foo.com/mar/mktgspcl/index.html">

Yes, it's March and you have a marketing special ad that you want to link to from your email. Ok, so why didn't you say so? Sometimes it pays to include the vowels if for no other reason than to avoid triggering Spamassassin's consonant filter that eagerly checks for any string of consonants with no vowels greater than 7 characters. By the way, the 7 character length is configurable to be 6 or 5 or anything the operator wants it to be. The net-net here is that although the internet is full of gibberish behind the scenes using plain English doesn't hurt once in a while.

One last thing to keep in mind, image paths, or URL paths for that matter, you should always make folder more than 2 characters. There are specific rules that look for 2 character paths, and 3 character paths at times, so an image folder called /img/ or /im/ should be named /image/ to avoid potential unpleasantries.

I know what's on your mind right now, "will any of this truly condemn me to the spam folder?" Any of this? Hm, possibly not any of this, but in conjunction with the 800 number in your email footer, the dollar signs in your subject line, well you get the picture. To quote Depeche Mode "everything counts in large amounts." Take control of your content and steer the ship to where it needs to go. These are pretty simple things to avoid and once you get the hang of it, you'll find you worry less and sleep more. Cheers!

-Len Shneyder
Director of Deliverability & Messaging
Pivotal Veracity | Unica

Good news today was posted by Return Path. For those using Return Path Certified, you now will receive automatic image and link enabling at both Hotmail and Yahoo!

What do you have to do to get this privilege at Yahoo?

1. Be a Certified level member in the program. Apply right now.
2. Have IPs that are rarely, if ever, suspended from the Certified list 
3. Authenticate your email with Domain Keys and/or DKIM and have unique domain/selector pairs dedicated to your Certified IPs
4. Submit domain [d=] and selector [s=] values associated with your Certified IPs

Pretty simple I say. This should help anyone in the program to clearly communicate their value proposition and provide your customers the best possible experience.

Congratulations Return Path and Yahoo! for making this a reality. Membership DOES have its privileges.

-Dennis
Eloqua

Don't Just Send, Deliver!

Spamhaus is announcing this week that they are launching their Domain Block List (DBL). The Spamhaus DBL is a realtime database of Uniform Resource Identifiers (URIs), typically web site domains found in spam messages. Mail server software capable of scanning email message contents can use the DBL to identify, classify or reject spam containing DBL-listed domains and other URIs.

What's this mean for you? Not only are your IP's a thing to watch over when it comes to reputation, but now your domains in your email are also.

Does this count as reputation for domains? In my eyes, YES!

They plan on launching this March 1, 2010.

For those who don't know, The Spamhaus Project is an international nonprofit organization whose mission is to track the Internet's spam operations, to provide dependable realtime anti-spam protection for Internet networks, to work with Law Enforcement Agencies to identify and pursue spammers worldwide, and to lobby governments for effective anti-spam legislation.

Spamhaus maintains a number of realtime spam-blocking databases ('DNSBLs') responsible for keeping back the vast majority of spam sent out on the Internet. These include the Spamhaus Block List (SBL), the Exploits Block List (XBL), the Policy Block List (PBL) and the Domain Block List (DBL). Spamhaus DNSBLs are today used by the majority of the Internet's Email Service Providers, Corporations, Universities, Governments and Military networks.

-Dennis
Eloqua

Don't Just Send, Deliver!

Quick note this morning. SpamAssassin has notified the industry that a bug exists in their software.

What's the deal here you ask? Well it seems that SpamAssassin had a rule in their default installations YEARS ago that would catch any spam with the date 2010+. Yes, believe it or not, but spammers forged their dates to confuse filters at one time or another. 

Well, email HAS survived up to 2010 now and the rule was never updated. Any emails sent as of January 1, 2010 to servers running SpamAssassin before the fix was made available, and to any servers running SpamAssassin that have not implemented the fix, will experience a higher than normal SpamAssassin score. This will effect delivery for messages where the score increase is significant enough to put the email's score above a filtering threshold

A fix to this issue has been made available, but it may take some time for servers running SpamAssassin to make the necessary updates since administrators are the ones who have to make the fix happen.

Make sure you poke as many people about this.

-Dennis
Eloqua

Don't Just Send, Deliver!

Not to big of a surprise in terms of a stat headline, but Return Path today released a new Email Deliverability Benchmark Report that shows marketers may not be still not getting it or that some just can't get a break no matter what they do right.

Return Path looked at more than 500,000 campaigns to determine what percentage of email is delivered to the inbox versus being diverted to the bulk folder or completely undelivered/blocked/dropped.  What's interesting in this report is that they also reveal that MSN, Hotmail, and Gmail Are The Toughest U.S. consumer inboxes to reach for marketers and Primus.ca, Shaw, Aliant, SaskTel, and Inter.net in Canada.

Commercial, permissioned emails reached only 79.3% of inboxes in the United States and Canada during the first half of 2009 (January through June), according to the report. With the undelivered email, 3.3% is routed to a "junk" or "bulk" email folder and 17.4% is not delivered at all - with no hard bounce message or other notification of non-delivery.

Hey Matt Verhout! The US deliverability rates are slightly better than Canada with an average of 82% inbox placement rate, while Canada's inbox placement rates are lower with just 75%. :P

As I said in the beginning though I wasn't surprised by some of the stats in this report i.e. Business Inboxes are even tougher to reach or Deliverability rates vary by ISP. As they said in their report and that many should already know today is each ISP has a unique recipe for determining what is appropriate for inbox placement, much of which is based on feedback they get from their customers. Understanding deliverability at this granular level is important for marketers who want to optimize their email marketing efforts.

As my good friend Sam Masiello just twittered. "Goes to show that permission is not necessarily king. Content and relevancy are still key factors to good deliverability metrics"

You can read more here on Return Path's blog as well

-Dennis

Eloqua

Don't Just Send, Deliver!

For those who monitor and suppress against the U.S. Federal Communications Commission (FCC) Wireless domain list, it seems that verizonwireless.com was removed from the list. Not sure why, but you should be aware of this in case you have customers in that domain who have never seen email sent to them because of the past suppression. This could cause complaints, unsubscriptions, hard bounces, and other issues.

Thanks to Abe Wagner Co-founder and VP of Engineering for Eloqua for noticing this.

 

-Dennis

Eloqua

Don't Just Send, Deliver!

In the past, we have discussed in much detail the ways in which email is examined for spam and phishing properties. Some of these emails and properties are contained in the things you might send out to your targets...

The AllSpammedUp blog recently did another good post on what's typically in these technologies to stop potential bad email from being received. It also explains that typically a combining effort of anti-spam technologies might be in order for some organization trying to protect it's users. It also shows how some of the older ways of stopping spam via content filters can cause issue.

To be honest, I would have loved to see this article address more detail towards the latest behavioral filtering technologies more commonly used today like Sender Score from Return Path.

----

Anti-Spam Products Are More Than the Sum of Their Parts

When you boil the spam problem down it becomes quite simple - someone is sending you emails that you don’t want to receive.  This makes the anti-spam solution a simple one too - stop unwanted emails from arriving in someone’s email account.  However, actually achieving this is a very complex task.

Any anti-spam system that is worth using will contain a range of preventative measures and features that are used to determine whether an email is likely to be spam or not.  As a complete solution they can be very effective, but taken individually and their weaknesses become more apparent.  Here are some examples...

--MORE--

http://www.allspammedup.com/2009/04/anti-spam-products-are-more-than-the-sum-of-their-parts/

-Dennis

Eloqua

Don't Just Send, Deliver!

I'm not re-posting this because I think that all these points are a need to know for the audience here. I just loved #5. If you take anything away from this today, take #5.

You should already know the other four (4)

Stop making emails so dang complicated. I'm also tired of other employee's stealing my iPhone from me to see what the email is supposed to look like since it didn't render on their Blackberry ;)

Think of the email as a teaser of what's to come when they click the link to get more information. Short, Sweet, and to the Point in your email.

Also, test your emails through services like Return Path or Pivotal Veracity before you send them out. See how they render and work in many of the email clients used today. See if your sending emails that will fail any one of the standards test there companies will run for you.

------

Top Five Mistakes from Emailvision

http://directmag.com/magill/0303-emailvision-marketing-mistakes/

UK-based e-mail marketing software-on-demand provider Emailvision has compiled a list of what it claims are the top five e-mail marketing mistakes based on its observations of its 1700 clients.

Here is an edited version of the list:

5) Trying too hard—designing an e-mail as if it was a Web site. Heavy use of flash or java script often doesn’t render properly in the finished e-mail and can be complicated to design. It’s better to keep it clean and simple, according to Emailvision, allowing the reader to click through to a Web site.

4) Adding the wrong return e-mail address—either by accident or on purpose to avoid filtering responses. Replies can provide valuable information from problems with the e-mail to inquiries about the product.

3) Missing information – having a comprehensive and segmented database is fundamental for any e-mail marketing campaign to work properly; there is nothing worse than Dear____ because the first name field is not completed, according to Emailvision. Even writing ‘Dear customer’ is preferable.

2) All image and no text—Many ISPs don’t automatically download images so recipients are left with blank boxes. This makes it hard to track response rates.

1) Stopped as spam – this is the number one fundamental flaw and probably the easiest to fix. Words in subject lines such as WIN A GREAT PRIZE will not get past spam filters.

“In today’s economy, return on investment has never been more important as marketing spend is now analysed and questioned by senior stakeholders,” said Nick Gold, UK managing director at Emailvision, in a statement. “Companies can’t afford to be making such simple mistakes and missing potential sales. The fundamental aims of any campaign should be high deliverability, targeted mailing, maximum click-through rates and basic personalisation – don’t let the e-mail be the reason customers go elsewhere.” 

-Dennis

Eloqua

Don't Just Send, Deliver!

These results should NOT be a surprise to ANYONE here, but Return Path ran a study to find out if Internet Service Providers (ISPs) treat IPs differently when it comes to filtering decisions.

You should consider that these results are also a factor of many things such as which ISP's serve which demographics, user perception in personalities for those demographics, how spammers target different ISP's because of their differences in filtering, etc.

What's more notable in this study is that in some cases you could see a "7x difference in accepted mail from the same set of IPs when comparing their behavior at two ISPs". This should point out the fact that if your not following sender common best practices that prescribe such things as having rDNS on your sending IP's could heavily effect you at some of those ISP's. In other words some ISP's have different policies, but following the strictest of rules should help you with many delivery differences at most ISP's.

ISP's will probably for the most part continue to have delivery difference because of budgets, laws they must comply with, differences in how customers are raised, competition with each other, etc.

I think its nice to finally have a third party prove that there can be differences and that NOT all ISP's will treat your mail the same. Not trying to say it's right or wrong, but that your customers should be aware of this. It's still think it's their responsibility to handle their reputation. I would be willing to stick my neck out here today and say that most ESP's from the best common technology standards standpoint are all doing the same thing to get your mail delivered. Many ISP's have the same technology required to deliver to us standards. It's now up to you to ensure your reputation is good no matter which ISP your sending to.

------

Inconsistencies with inbound traffic across ISPs

John Young, Ph.D.

Director of Product Analytics

We encourage receiving networks to share data with us at Return Path so that we can in turn provide solutions and information that will help their filtering decisions. We believe that you can learn from another company's mistakes and success. And, when working in a collaborative environment, receiving networks can learn from cases where one system accepted mail that another system was blocking erroneously or vice versa.

We decided to dig into our data to find out if Internet Service Providers (ISPs) treat IPs differently. We took a random sample of 400,000 IPs that attempted to send messages to four different receiving networks in early 2009. The ISPs used from our network consisted of two webmail providers, one cable operation, and a hosted business email provider.

By looking at IPs that mailed to all four networks, it became clear to us that receiving networks make extremely different decisions about how to treat those mailers.

From the data, we found that when ISPs make decisions on what to do with inbound mail, they cannot agree on IPs from smaller volume mailers, especially when that IP has no rDNS. Also noteworthy was a 7X difference in accepted mail from the same set of IPs when comparing their behavior at two ISPs. Now, that's not as interesting until you note the variance in complaint rates across ISPs, which vary as much as 3X between them.

One would assume that the more email sent, the higher the probability of a user complaining. However, for this set of IPs and their corresponding data, that is not always the case; especially when looking at those smaller volume IPs with no rDNS. For example, one IP sent ~100 messages to ISPs A and B. The IP had ~90% delivered rate at ISP A, 4% Delivered rates at ISP B, but less than 0.2% complaint rates at either.

Another interesting pattern indicated agreement in the treatment of IPs with a lot of trap hits. However, there was a significant percent of IPs with a large amount of traps who ended up in the disagree buckets respectively. Again, it was mostly smaller volume mailers averaging less than ~200 messages a week.

It is evident that ISPs could benefit from sharing their experiences and their data in their fight against spam. By doing so, they could minimize the amount of spam impacting their systems, and the overall costs associated with filtering.

We are currently in the process of conducting a larger study to see if the data we found in our initial analysis holds true. If you are a receiving network who would like to participate, please contact us. In exchange for participating, you will receive detailed reporting on how your filtering differs from other ISPS as well as reputation data that would greatly improve your filtering and blocking decisions.

------

-Dennis

Eloqua

Don't Just Send, Deliver!

I joined Genius.com a few months ago as a member of their Deliverability and Anti-Abuse team. My focus is on deliverability and managing the email servers. Genius has a great application that gets marketers and sales people working together. The Genius application relies heavily on emails being sent to the right people at the right time.

In addition to email deliverability, I have a background in system admin, programming (among other nerdy pursuits) and I am active in the technical (and sometimes political) Internet community which helps me in the world of deliverability . I look forward to sharing my experience with you over the coming weeks.

The world of DKIM: Domain Keys Identified Mail.

One of the challenges in identifying suspicious email is knowing reliably where it came from. With email coming from a wide variety of machines, it’s possible that a machine has been compromised and a malicious sender is spoofing mail headers, injecting false headers and essentially misrepresenting a third party. A lot of spam is generated by compromised machines which are part of botnets, networks of thousands (even millions) of remotely controlled machines. Some of these machines can be even legitimate mail servers, some can send email via ISP or ESP mail servers (ISP: Internet Service Provider, ESP: Email service provider). How do we know who is who? Who should we complain to? Some services like spamcop.net, try to analyse emails headers and figure out who was responsible for sending the email but they often also target whoever is managing the networks used to relay the emails. They aim wide, to ensure they will alert someone that will care and fix the spam source.

Would it not be easier to know who is sending emails?

There are some old techniques like Pretty Good Privacy (PGP) and S/MIME but they are heavy to implement because they require the cooperation of all of the users. Besides, we often do not need to authenticate the emails, but have a way to validate where the email is coming from. That is, to identify which system administrator configured the mail server that sent the email. This is the person we want to talk to for any problem. Hopefully he/she is a little bit more clued than the user who sent the email and can take remedial methods. Fixing problems on the Internet involves often finding the person that understands what you want and knows what action to take. End users are rarely in this category, but systems and network administrators are.

Several industry groups tried to solve this problem, the two main technologies that emerged were DomainKeys and Sender Policy Framework (SPF).

Ideally, emerging technologies will evolve to become standards. The Internet Engineering Task Force (IETF) does not generally create new Internet technologies, but they are very good at taking something that a group submits for standardization and making it rock solid because the peer review is strong. This has been the case for example for http (the web protocol), ssl/tls (securing connections between applications) and many others.

They did the same thing with DomainKeys and SPF and issued the DKIM specification. DKIM is now in the Internet Standard track of the IETF. It means if you like it you have a fixed specification you can follow that will work with various and disparate systems. It really become a de-facto standard when a large group implements it.

The main problem with SPF (and a bit with the Sender-ID upgrade) is that emails move from machines to machines before reaching the final destination. In many cases there are only two mail servers involved. The sender and the receiver. SPF inserts a header that tells if you check this domain, it will tell you that this IP address (of the sender) is authorised to send emails for this domain. But if you add a mail relay server, the whole scheme falls flat, as the receiver will have the ip of the mail relay not of the original server. On the other hand DomainKeys has another problem, it can only validate headers that contains the same domain name as the signature. So for instance you can relay a signed message but you cannot sign a message that you relay (like for a mailing list).

DKIM solves all these issues.

At the moment the DKIM Working Group at the IETF is working on specifying a standard for defining policies for the use of DKIM. But first you noticed, I did not say that DKIM authenticates an email, as it is not true. It just validates some headers. It is up to the receiver to process the email based on the successful validation of certain headers and deliver it to your inbox, junk folder, etc. DKIM allows a mail admin (or rather a mail server) to take responsibility for sending/forwarding an email and let other servers know.

This is convenient for instance, to build reputation based on who takes responsibility for sending/forwarding the email. For instance AOL has announced that they will move from IP based reputation to Domain based reputation using DKIM. Similarly, Yahoo provides a feedback loop based on the domain validating the email using DKIM. It does not matter which IP the email comes from.

But what are DKIM policies?

For instance, e-Bay and Google have announced that all the e-Bay emails will be DKIM validated and that Google will use this information to trash any email supposedly from e-Bay which is not DKIM validated. Imagine if all the banks were following this policy. It would seriously reduce the amount of phishing.

Third party validation.

I spoke earlier, that a relay mail server can validate an email. For instance in the case of mailing lists. The mailing list software receives an email, and decides to use its reputation to forward the email to all the subscribers. It can do a third party validation on an email it is forwarding. There is no requirement that any headers in the email (From, Sender, …) contain the same domain name as the domain name of the mailing list. This opens other possibilities. Let's take the bank example, instead of each bank having their own DKIM validation, they could also have a validation done by a bank confederation, authority or regulator (third party). This would add another level of reputation on the email (while they can still add their own DKIM validation to the email).

Genius is providing third party validation on all our customers’ emails. We are taking responsibility for the emails we send on behalf of our customers by ensuring they are DKIM authenticated. There is no silver bullet in ensuring a high rate of email deliverability and DKIM is just one of the layers that help deliverability from a technical perspective. The Genius Email Deliverability and Abuse team in addition to DKIM ensures high rates of deliverability through close and proactive management of its customers email campaigns, education and best practices.

PS: feel free to contact me with comments and suggestions and email delivery related topics you'd be interested in learning more about.

A few days ago, I posted a comment on a Yahoo delivery issue I encountered. A few persons emailed me to have more info on how we solved this issue at m--x--m. This is the purpose of this post. I will describe how a new company overcame recent delivery issues with Yahoo Mail. I hope this will be useful to others.

Context
I am an engineer for m--x--m, a new email delivery company. We use a number of new ways to deliver emails and we mostly focus on transactional and newsletter emails.

Problem
With no increase in deliveries, nor in user complaints (as inferred from other FBL since we could not get one from Yahoo), we got completely blacklisted on Yahoo with the following message.

Aug 31 14:06:36 dserv128-mtl3 m--x--mail/smtp[8516]: 4D7F314C86A3: host e.mx.mail.yahoo.com[216.39.53.1] refused to talk to me: 421 Message from (xx.xx.xx.xx)temporarily deferred - 4.16.50. Please refer to http://help.yahoo.com/help/us/mail/defer/defer-06.html

We kept having this message and sometimes some messages were delivered... directly in the junk folder.

Therefore I started investigating and understood a few tricks.

What I have understood so far
Some of this points may be out of date. Please share with us any of your insight in the comment.

1. Yahoo does not offer anymore a FBL. This is the root of most issues.


2. Yahoo's filtering system is not compatible with the way most MTA handle their retries. In practice when a minor delivering issue arises, it can sometimes blacklist completely the IP.
   
3. Yahoo uses the following graduation when dealing with a temporary blocked IP:
   
   

   [TS01] -> [TS02] -> regular deferred message. (you will find these messages in your logs)
   
   

   Your goal is to stay at TS01. I did not find a way to go back from TS02 to TS01, so be careful.



4. When encountering a serious delivery issues, Yahoo wants you stop delivering for four hours. Doing so seems to reset Yahoo's « reputation counter ».


5. It seems Yahoo signals sometimes user complaint even without FBL. This happens when you see a « deferred » error message only for a specific user (while other emails to the same MX are still delivered).


6. Yahoo ends the block progressively, you can send before the end of the four hours but you take the risk to reduce your reputation.


7. Yahoo filtering algorithms seems to be of the form number of user complaint per unit of time -- as opposed to a percentage formula (like Hotmail for instance). For instance, 100 user complaints might trigger a block even if they happened on 100 million emails.

In a nutshell, Yahoo asks for a very specific way of handling delivery. Misconfigured MTA can actually cause a permanent block; the problem is most MTA are configured not to respond well to their temporary blocks.

Solution
I wrote a MTA dedicated for Yahoo delivery (our delivery architecture is very flexible in this aspect). Its main points are:

1. When presented with a 4XX.*deferred error, it equates it to a user complaints.


2. When presented with a TS01 message, it stops delivery for four hours on this MX DNS (but it keeps going on the others).


3. It smoothen deliveries so there is no burst (which might overcome the user complaint quota).


4. It uses a cluster of « smart shared IP » to ensure delivery even when temporary blocked.


5. It uses a smart backoff algorithm to increase a lower user complaint per unit of time.

This solved of all issues and even now we keep using this solution. If you encounter further problems with Yahoo, please let us know, we might be able to assist you.

This post was written in collaboration with Krzysztof Jarecki and Nicolas Toper (http://www.m--x--m.net) based on an idea from Krzysztof.

It's been somewhat of a running joke in the email marketing community that the spam filtering companies don't really have any incentive to "solve" the spam problem. After all, if they were to fix it, then they would be out of business!

In many ways, the spam filter companies could be compared to weapons manufacturers. Now, McAfee certainly isn't as bad as Haliburton, in fact I think they are a good company that helps protect millions of people from spam and viruses. But like the weapons manufacturers don't really want the wars to end, I don't think the executives at McAfee are really interested in solving the spam problem.

This week BBC reported on the results of a study commissioned by McAfee to try and justify their perpetual existence. They came to the conclusion that you need a spam filter. "Surfing the web unprotected will leave the average web user with 70 spam messages each day."

Wow, that was a striking conclusion. Who doesn't have a spam filter? Every free email account, broadband email account, and most corporate email accounts have spam filters.

The BBC article ended with a quote that shows how self serving the whole thing was, "It is such an immense problem and it's never going to go away. It's no longer a question of solving it but one of managing it," said Mr Dave De Walt, chief executive of McAfee.

Well Mr De Walt, OtherInbox hasn't given up yet.