Category: Deliverability 101

Deliverability 101: The Big Three

by Brad, posted in Deliverability 101, Delivery Essentials

For over a decade now, those of us who work in email have talked about the “Big Four” email providers. This was used to refer to the four largest providers of consumer mailboxes in the industry: Microsoft, AOL, Google, and Yahoo. In online discussions and on email distribution lists, they were often known by the keyboard-friendly acronym MAGY. Early this year AOL and Yahoo merged under the Oath brand, whittling the Big Four down to three and prompting a new acronym: OMG.

For years, each of these companies has provided free email accounts to anyone with internet access and a few basic personal details to hand over. They (mostly) provide free webmail with virtually unlimited storage, while using the personal data of accountholders to sell advertising that gets displayed within their respective user interfaces. It can often be difficult for senders to identify and/or troubleshoot delivery issues, as each provider’s primary duty is to their users and advertisers and senders can get left out in the cold when searching for answers. Here’s a brief primer on each of the Big Three to help you along.

Microsoft (Outlook.com, Hotmail, MSN)

microsoft

If you’re reading this on a computer, you’re probably familiar with Microsoft. As the purveyor of Windows, their name is found on most of the computers in the US today and they’ve been providing free email since 1997.

Starting as Hotmail then eventually rebranded to Outlook.com email, Microsoft’s freemail offering remains one of the most popular.

Troubleshooting

Microsoft provides a number of tools for senders to monitor and troubleshoot delivery to their users. Signing up for their Feedback Loop (dubbed the Junk Mail Reporting Program or JMRP) is a must to ensure you receive complaint data from MS. Integrated with JMRP, Smart Network Data Services (SNDS) provides senders with insight and daily monitoring of their mail delivery performance. Instead of discrete numbers, SNDS uses a color-coded system: Green means that more than 90% of your mail went to the inbox, Red means that less than 10% made it to the inbox, and Yellow indicates something in the middle (generally closer to Red).

The provider recently started to merge their consumer offering Outlook.com with Office365, their business-class email solution, but still offers separate remediation processes for each service. While there are rumors of the Outlook.com support form working for both products, no official confirmation has been issued. Until it is, the Office365 Delist Portal should be your first stop for problems sending to corporate domains using hosted mail.

It’s also important to note that the filtering for each product remains separate – SNDS does not include data from Office365 and B2B mail may be routed differently than B2C, even from the same sender.

Oath (AOL, Yahoo, and Verizon mail)

oathWhile Oath is technically the newest company on the list, their email offerings have been around for quite some time. AOL (originally America Online) provided dial-up internet and email service to millions of users in the mid-1990s, then began to offer their webmail for free to non-AOL users in 2006. Yahoo started its email service in 1997 with the acquisition of the Rocketmail platform. In late 2017, the two services were merged and became Oath.

Troubleshooting

Prior to their merger, AOL was known for being one of the most responsive providers when seeking assistance. They offered an online reputation tool, a remediation form that generally received quick responses, and a knowledgeable, helpful Postmaster team. Yahoo was known to be a bit more of a black box when troubleshooting delivery issues, although their bounce messages and the troubleshooting pages associated with them were fairly descriptive.

Since the merger, mail to both AOL and Yahoo addresses are being directed through Yahoo’s mail servers. Many of the most common Yahoo bounce responses are now being seen with AOL addresses, and Yahoo’s Postmaster documentation and support form (login required) are the de facto method of receiving support for delivery issues.

Google (Gmail)

googleGoogle’s Gmail offering is the youngest of the Three, having been introduced via private beta in 2004. Experiencing meteoric growth, it eventually overtook Yahoo and Microsoft to become the most popular freemail provider in 2012. Today, Gmail addresses typically make up a significant portion of any consumer email list – often more than half.

Troubleshooting

Unlike the other providers on the list, Gmail offers no direct method of contact for senders. Their filtering is almost exclusively automated, so even if you know who to contact they’re generally unable to make any changes to the routing of your email. Fortunately they do offer some useful tools for monitoring your delivery performance. Google’s FBL, unlike those provided by other major players, does not pass back specific subscriber data but only aggregate complaint numbers. You can also track these FBL complaints, your IP and domain reputation, authentication status, and mail error rate via Google’s Postmaster Tools.

Because of their unconventional FBL, it’s not possible to identify and suppress those recipients who mark your messages as spam. You’ll instead have to use the data available to you (opens, clicks, site visits from email, etc) to determine if your recipients are engaging with your messages. If you experience delivery issues to Gmail, check the composition of your list. Are most of your Gmail recipients ignoring your emails (no opens recorded in 6-12 months)? If so, that’s likely the cause of the issues. One word of caution when dealing with Gmail: once you start to see problems, it can take weeks or longer to resolve them, so it is generally best to keep your lists clean proactively.
While there are hundreds of mailbox providers out there, these three often make up the vast majority of a sender’s contact database. Learning how to effectively navigate the Big Three is a major step toward mastering your email deliverability.

– BG

Deliverability 101: IP address and domain reputation

by Brad, posted in Deliverability 101, Delivery Essentials

Reputation can be very important in all walks of life, and email is no different. For senders with a good reputation, mail is more likely to be delivered to the inbox, giving your recipients more chances to read and interact. On the other hand, if your reputation is “not so hot,” you’ll be more likely to see your messages end up in the spam folder or even rejected altogether. So what exactly is this mystical “reputation,” and how is it determined?

15313888764_975c5f0c67_z

IP Address Reputation

Before we address (no pun intended) the reputation aspect, let’s have a brief refresher on the nature of IP addresses. Every device connected to the internet – computer, router, even your smart thermostat – is assigned a numeric value known as the Internet Protocol address. This IP address is how your mail server is identified on the internet. When you send mail to a recipient, their mail provider will see where the message originated, identified by that IP address.

Most inbound mail servers will also attempt to use that IP address to determine whether the sender of the mail is trustworthy. If mail from that IP address is often sent to invalid addresses, or the mail generates too many complaints, the inbound server may choose to route incoming messages from that server to spam, or block them outright. Many mailbox providers also use third-party IP address blacklists like Spamhaus or Spamcop, choosing to filter or reject mail from IPs that appear on those lists.

A few years ago, IP address was the gold standard for sender reputation. As content filters that looked for “spam words” became less effective (MILLI0NS OF DOLLAR$ IN VI4GRA, anyone?), IP reputation helped receivers identify habitual bad senders. The receivers could then act on all mail sent from those servers instead of chasing down ever-morphing content to filter inbound mail. This often caused issues for ESPs with smaller clients, because many clients were often sending from the same IP address (This is one of the reasons many ESPs tend to be more strict with shared-IP clients – because mail you send can impact delivery for many other senders on your IP).

A fact that is not widely known outside tech circles is that the internet is running out of IP addresses. As a result, a new standard for IP addresses was created: IPv6. This standard allows for a seemingly infinite number of IP addresses – theoretically 340,282,366,920,938,463,463,374,607,431,768,211,456 to be exact. With that many IP addresses, anyone whose IP is blocked can simply start using a different IP address, essentially bypassing IP address-based filtering.

Domain Reputation

There are times that an IP address alone doesn’t give the whole picture. The shared IP pools used by some ESPs, as mentioned above, are a prime example. Another example might occur when multiple divisions of a single company or organization use the same IP addresses to send mail. Because of this, and the onset of the IPv6 standard, domain reputation has become increasingly important in mail delivery.

How domain reputation is checked can vary depending on the mail server. Some providers check only the domain the mail is actually sent from, while others look at all the domains in the message headers. Some providers, including Google, check every domain in the body of the message as well. So if you send your mail from example@example.com, you can bet the reputation of example.com will be checked, but other domains in your message could be checked as well. If you are using an ESP or other hosted mail solution, there may be domains in the header that belong to your host that will also be checked (another reason authentication is so important). And don’t forget those links in the body! Many filters will block a message if it contains links to domains that have been flagged as spam or dangerous. If you’re linking to a domain you don’t own, it’s a good idea to check its reputation using one of the handful of free online tools available (we’ve compiled a few on our Resources page).

Not surprisingly, a domain’s reputation is typically impacted by mail sent from that domain or using that domain in the body. However, it can also be affected by the reputation of that domain on the web as a whole. If a specific website has gotten a bad reputation for, say, gaming Google’s search engines or potentially scamming customers, that can also impact mail delivery.

The ‘Secret Sauce’ of Sender Reputation

When it comes to actually filtering inbound email, every mailbox provider uses a unique combination of factors to determine whether mail reaches the inbox. For many providers, IP address reputation still ranks at or near the top of the list of these factors. Domain reputation is typically very close behind, if not in the top spot. After that, there are typically various ingredients that make up the proprietary methods used by each organization. Trying to figure out the secret sauce for each ISP is a fruitless effort – even if you do manage to find a trick that bypasses or overcomes a specific filter for a time, it will invariably change and you’ll be left looking for another backdoor.

While the specific components of spam filters change constantly, your sender reputation – both IP and domain – will continue to play the largest role in getting your mail to the inbox. Manage that reputation well, and you’ll save yourself a lot of delivery headaches.

For more information on IP addresses or the IPv6 standard, check out these resources:
What is an IP Address? on HowStuffWorks
Word to the Wise – IPv6 archives
IPv6.com – A Beginner’s Look

– BG

Deliverability 101: Spam traps

by Brad, posted in Deliverability 101, Delivery Essentials

It's a trap!Recently we presented a basic guide to blacklists using a rather colorful example ripped from today’s headlines. In that post we described what happens when you’re on a blacklist, but as a sender you probably want to know how to avoid getting blacklisted in the first place. And if you’re already blacklisted, you’ll certainly want to find out how you got there. Understanding spam traps can help with both.

What is a spam trap?

While the name might conjure thoughts of being lured into a sticky situation via canned meat, a spam trap is actually an email address. Sometimes referred to as honeypots, spam traps are addresses that exist for the purpose of identifying senders who are not following best practices.

A spam trap isn’t used by a real person to send or request email. Each trap is monitored by the trap operator (typically a blacklist or mailbox provider) and any mail sent to the address can cause the sender to be put on a blacklist. While the potential exists for blacklisting based on a single message sent to a trap, providers most often look for patterns of repeated hits. This could mean multiple mails to the same trap address, mail to multiple distinct trap addresses, or both.

There are two main types of spam traps in the wild: pristine and recycled. Pristine traps were created for the sole purpose of being a spam trap. These addresses have never been used by a real person and have never requested any emails. If you send mail to purchased lists or scrape addresses from the web, there’s a good chance you’ll run into this type of trap.

Recycled traps, by contrast, are the type most commonly seen by legitimate email marketers. These email addresses did, at some point in the past, belong to a real person. That person likely sent emails, signed up for mailing lists, and provided the address to others as their point of contact. Then, for whatever reason, that person abandoned the address – maybe due to an organizational change or migration to a new mail provider.

Once the recipient abandoned this address it sat dormant for some period of time (generally at least 6 months), during which time the address would have rejected all mail. After that period, the address was reactivated and became an active spam trap.

How do spam traps get on my list?

Since spam traps are designed to identify senders not using best practices, it stands to reason that failure to follow best practices typically leads to their presence in your database. Mailing to recipients who have not given opt-in permission, sending to old or outdated lists, and lack of proper bounce handling are some of the most common reasons spam traps end up within your list. In addition, typographical errors at the time of address collection can introduce traps into your list – particularly with less accurate address collection methods such as point-of-sale address transcription or collecting addresses via telephone.

What happens if I have spam traps in my list?

When you send mail to a spam trap address, the trap monitor will note the sender of the message and typically take some action against that sender. Most trap monitors also maintain their own blacklists, and in many cases these blacklists are publicly used by many ISPs and mailbox providers to filter mail. In short, sending to spam traps will probably get you on a blacklist, and that blacklisting will probably get your mail rejected by at least one major email provider.

I’m not being blocked. Why does it matter if I have spam traps in my database?

If you receive word (from your ESP, delivery monitoring service, or a trap owner) that you are sending to spam trap addresses, it’s tempting to gloss over the warning if you’re not seeing any large-scale delivery issues. A word of advice? Don’t ignore spam traps.

The presence of spam traps in your contact database is an indicator of an underlying issue with either your email acquisition practices or your list maintenance protocols. When you have spam traps in your list, you are sending mail to contacts that don’t want it or never requested it. This means that alongside the traps, you are also mailing real people who will (at best) ignore your message or (at worst) report you to their mailbox provider or a third-party spam filtering service. Even if the spam traps haven’t gotten your mail blocked (yet), you can bet the spam complaints and low engagement are keeping you out of your recipients’ inboxes.

How do I get the spam traps out of my list?

Removing spam traps from your list is, by design, a difficult process. A spam trap doesn’t (usually) bounce or reject mail. It doesn’t provide any signs or signals that it’s a trap. The trap operator doesn’t want you to be able to spot the traps in your list, because then you could simply remove the traps and not address the underlying issue.

If you have traps in your database, the best place to start is typically contacts who haven’t engaged with an email (opened or clicked) recently. Typically, we recommend targeting contacts who haven’t opened in 6-12 months. Send a confirmation request to those non-openers, asking them to confirm they are real and they still want your messages. Once that message is sent, you’ll want to suppress from your list anyone who doesn’t respond. It’s also a good idea to repeat this process at least once a year.

In conjunction with addressing spam traps already in your list, you want to make sure you cut off traps at the source. Check your list acquisition practices to be sure all of your incoming recipients have opted in for your mailings. Add CAPTCHA to any public-facing web forms to prevent automated sign-ups. Think about adding a confirmation step to your opt-in process. This could be a traditional confirmed opt-in (COI) where recipients have to click a link to be confirmed, or it could be a “soft confirmation” that considers an open to be a confirmation action.

If you can take actions that make it harder for spam traps to end up in your list, you’ll proactively decrease your risk of dealing with the difficult process of culling your list to get rid of them later.

Have a war story or questions about dealing with spam traps? Leave a comment or shoot me an email to chat!

– BG

Deliverability 101: Shared or Dedicated IP?

by Brad, posted in Deliverability 101

As children, most of us probably learned that it’s nice to share with others. It’s one of those timeless lessons that often carries throughout adulthood – but does it also apply to mail server IP addresses? It can, but there are a number of factors to consider to accurately make the determination whether a shared or dedicated IP environment is better for you.

The Basics

The question of shared vs. dedicated IP is most often asked when choosing an email service provider (ESP). While some ESPs specialize in one or the other, many offer both shared and dedicated IP options for senders depending on their sending patterns.

In a shared IP environment, one or more IP addresses are arranged into pools that are shared among multiple senders. At any given moment, mail from multiple senders is likely to be sending over one or all of the IPs in the shared pool. This also means any mail you send will be spread across some or all of the IPs in that pool.

A dedicated IP environment will provide you with an IP address that sends mail for only your organization. No other traffic from any other sender will use that mail server and it will be easily identifiable as ‘your’ IP.

Which Option is Right for You?

Hopefully we all have a clear understanding of what constitutes a shared vs. dedicated IP setup, but the burning question remains: which should you choose? There are a few factors that help make this determination:

  1. Sending volume.
    When you send from a dedicated IP, you need to be sure that you have enough sending volume to establish and maintain a reputation as a “known” sender at the ISPs you are attempting to reach. Volume recommendations will vary depending on list composition and other factors, but we typically suggest around 500,000 in average monthly volume as a good rule of thumb. This will help to ensure that getting a few extra complaints or bounces one day won’t completely derail your deliverability.
  2. Sending frequency.
    When considering a dedicated IP, the consistency of your send volume is every bit as important as the volume itself. In order to maintain that reputation with the ISPs, you need to send consistently and regularly. For example, if you send 1 million emails per month, but your sends are split 800,000 on the 3rd of the month and 200,000 on the 17th, you may not see the full benefits of a dedicated IP. We suggest at least 100,000 emails every week as a good starting point.
  3. Reputation.
    Even if you aren’t paying attention to reputation as a factor in your decision, you can bet your (potential) ESP is. If you are a sender who is following best-of-the-best practices, gets confirmed opt-in for every subscriber, and sends consistent high volume, you are not likely to see benefits from a shared IP pool. Since shared IP pools send mail for multiple marketers, the positive or negative reputation of each can impact delivery for every sender on the pool.

    While many ISPs are placing more weight on domain-based reputation, the IP address sending the mail is still a major factor at most every mailbox provider. As such, your mail delivery rate could be negatively impacted by the performance of others on the shared pool. Most ESPs have monitoring in place to ensure senders do not take actions that will seriously harm the shared IP reputation, but there will always be a higher level of shared risk than with a dedicated IP. However, by the same token, if you are not getting clear permission, sending irregular volume, or otherwise not following best practices, you are likely to see issues no matter which IP option you choose.

If you have the volume and/or consistency to support it, there are few cases where a dedicated IP setup would not be recommended. But if you are a smaller-volume sender, or only need to send sporadically, you can certainly find success on a shared IP pool with the right ESP.

– BG

Get Ready for Deliverability 101!

by Brad, posted in Deliverability 101

Question BoxOn May 27th, I presented a webinar with WhatCounts titled Deliverability 101: Back to the Basics. This webinar covered many of the core ideas and concepts of deliverability including bounces, mail filters, spam traps, and inbox delivery. The turnout for the session was excellent, and the content drew so many questions that time wasn’t available to answer them all.

Starting next week, I’ll be diving deeper into some of those same topics, both here and at the WhatCounts blog. Some of the posts will be based on questions received during the webinar, and questions or comments received on the posts as they progress. This will be an ongoing series designed to provide an easy point of reference for anyone in the email industry to get a basic education in the science (or is it art?) of deliverability.

If you’d like to download the WhatCounts webinar, you can find it here.

Also, if you’d like a bit of insight into my daily delivery activities and thoughts on the industry, check out this Q&A with Ashley Hinds of DataValidation.com.

– Brad Gurley