We need a real way of measuring relevance in email. e-Dialog and Responsys have both come out with their own methodologies, but they seem to be a subjective self-assessment rather than an objective measurement.

In both examples, email marketers fill out a short survey where they rank themselves on various criteria. While I'm sure this is an interesting exercise, I don't think it's going to have a huge impact on the relevance of emails that people send. This is just another way to identify best practices and new ideas, but not a measurable, actionable score. It's an interactive whitepaper.

What we really need is a metric that is objective like deliverability. I'm sure we would find that the two metrics are tightly correlated.

The challenge is that relevance is hard to measure. It would probably be a subset of traditional email reputation - less focused on bounce rate and more focused on open rate and other indicators of user activity.

I don't think this is the answer, but here is a stab at the type of thing I'm thinking of:

Relevance = (Unique Opens / Total Valid Recipients) + (Unique Clicks * 3 / Unique Opens) - (Unsubscribes / Unique Opens) - (Spam Complaints * 10 / Total Valid Recipients)

So if you sent a message to 100,000 recipients with a 5% bounce rate and got a 10% open rate with a 10% clickthrough rate, 1% unsubscribes and 0.1% complaint rate, the Relevance Score would look like:

Relevance = (10,000 / 95,000) + (1,000 * 3 / 10,000) - (950 / 10,000) - (95 * 10 / 95,000) * 100
Relevance = 0.11 + 0.3 - 0.1 - 0.01 = 0.3 * 100 = 30%

A message to 100,000 recipients with a 1% bounce rate and a 30% open rate with a 20% clickthrough rate, 0.5% unsubscribes and 0.05% complaint rate, the Relevance Score would look like:

Relevance = (30,000 / 99,000) + (6,000 * 3 / 30,000) - (975 / 10,000) - (49 * 10 / 99,000) * 100
Relevance = 0.3 + 0.6 - 0.1 - 0 = 0.8 * 100 = 80%

It's possible to get over 100%, but it would be unusual. A message to 100,000 recipients with a 1% bounce rate and a 50% open rate with a 30% clickthrough rate and no unsubscribes or complaints would have a Relevance Score of 140%.

A message with a 100% open rate and 100% clickthrough rate would have a Relevance Score of 400%. You could normalize this further by putting a max value of 1.00 for each of the 4 contributing calculations (opens, clicks, unsubs and complaints).

I don't think this is the perfect equation (yet). But I think this is the kind of thing we need to be looking at when we talk about Relevance. What other factors should we be considering besides opens, clicks, unsubs and complaints? How would you weight each item?

Comments

I am not a number! I am a free man!

Seriously...what you need here are data points that marketers will, unfortunately (for them), never get:

- how much time between logging in and reading the message?
- how much time did the recipient spend reading the message?
-- before clicking through?
-- before deleting it?
-- before marking it as spam?
- was the recipient smiling when they read it, or frowning?

There are user researchers who can get y'all closer to this kind of information. It's an entire discipline, often called human/computer interaction or human factors.

I'm glad that marketers are starting to think about this, though -- for too long, they've been forgetting that humans are even involved.