15 Types of Tests

Note from Jeanne: This was one of the most popular columns I wrote for ClickZ in 2013. If you’re looking to make your email marketing more effective and more profitable you have to test. As the title suggests, here are 15 types of email marketing tests you should be doing. Enjoy!

I was recently asked to develop a “testing protocol” for a client. The deliverable was a worksheet that would document tests for a particular send, but they wanted it to do more than just that. They wanted it to educate on the value of testing and spur the groups within their organization to truly leverage testing to their advantage.

Not a simple task.

One of my observations, both with this client (who I’ve been working with for more than two years) and with other clients, is the focus on subject line testing. My personal goal, with this project and in general, is to get email marketers to go beyond just testing subject lines and look at other elements that might improve response.

There are a lot of “100+ things to test in email” lists out there. But these leave me a bit flat. Many of the items they list are specific to an industry or type of email, and many aren’t things that will normally drive large increases in response.

So here’s my much-less-impressive sounding list of 15 types of email tests you can do, along with some narration on my personal favorites and least favorites. The list is roughly organized from what generally provides the most to least value in terms of increased performance for the long term.

  1. 15 Types of TestsLists
  2. Landing page elements (which could spur another list of 15 or more)
  3. Completely new creative (multiple elements)
  4. Preview pane view
  5. Customization (targeted content)
  6. Offer/offer copy/placement
  7. Call-to-action copy/placement
  8. General wireframe/layout
  9. General copy
  10. Personalization (data merge)
  11. Subject line
  12. From line
  13. Design (fonts/colors/images)
  14. Time of the send
  15. Day of the send


List testing tops my list of 15 because if you aren’t reaching the right audience it doesn’t really matter what content the email carries. This is a broad category – you can test to identify segments of your house list which are more and less responsive to certain offers, and use this knowledge to boost your revenue per email sent by targeting offers. See my recent case study on this.

You can also mail to third-party rental lists to not only boost your immediate performance but also grow your own house list with qualified names via opt-in. Infusing your house list with qualified names on an ongoing basis is a key to success in email. According to MarketingSherpa’s 2013 Email Marketing Benchmark Survey, only 17 percent of marketers have email lists that are growing rapidly. Another 50 percent say their lists are growing slowly, and a third of respondents are seeing neutral to negative growth.

Landing Page Elements

Landing page elements, which you can see could have a list of 15 or more elements of its own, is my second favorite thing to test. Email conversions tend to happen on landing pages, microsites, and web-based conversion funnels. It’s important to keep readers engaged and moving toward the end goal of your email once they click through. Even a small boost in performance here, getting 5 percent or fewer of visitors to convert, can have a large impact on your bottom line. The results of many landing page tests you do can also be applied for future campaigns, in both email and other channels, producing true long-term value for the money spent.

Now on to some of the items at the bottom of my list.

Time/Day of the Send

I’m not a big fan of day and time send tests, nor do I pay much attention to those reports released periodically that tell you the best day and time to send email. I’ve found common sense to be the best guide here. For B2B email, I shoot to have it arrive during the business day based on the time zone of the recipients. For B2C email, I set send day and time based on the audience and when I imagine they’ll have a few minutes to read it, which is typically late afternoon/evening or weekend days. I’ve done day and time testing for clients, but assuming they are starting with a “common sense” control time for their audience, I haven’t seen huge, sustainable lifts.


The same is true for design, which I consider fonts, colors, and images. This is not to be confused with wireframes/layouts, which I have much higher on the list. I love good email design; but I have also found that as long as common sense is your guide, the fonts, colors, and specific images used don’t tend to have a large impact on response. For me, common sense design is about using readable fonts and colors and images that compel readers to the action you want them to take. Testing magenta instead of maroon as the accent color on the email isn’t something that typically provides you a large lift.

A challenge to readers: when you plan your next email test, start from the top (not the bottom) of my list of 15 and choose a type of test that you’ve never done before. Then build the test, see how it does, and let me know!

ShawScott Holiday Email Checklist

It’s nearly Halloween — do you have your holiday email marketing program locked, loaded and already in process? If so, congratulations! If not, read on for a few free resources to help you get it in tip top shape before the trick-or-treaters start ringing your doorbell!

Quick note: these are a few of my favorites, but there are so many others! Please feel free to share YOUR favorites in the comments section.

Holiday Email Checklist from Shaw + Scott

ShawScott Holiday Email Marketing ChecklistI was delighted to find this guide from Shaw + Scott — it starts with quantitative data and then provides a checklist of 10 things to keep in mind for this holiday season. And there are lots of images of great emails for inspiration. It’s a quick-and-dirty way to make sure your email marketing program is hitting on all cylinders.


#1: Use What You Know — look back at last year, see what was most effective and leverage that for this holiday season.

#5: Pace Yourself — give your recipients a say in how often they will receive email from you; some will want to hear from you daily, but others may not.

#9: Give Them Something to Talk About — Look to incorporate social content, including ratings and reviews, into your email marketing program.

Key Holiday Email Campaigns for 2014 from Windsor Circle

Key Holiday Email Marketing Campaigns from Windsor Circle

No pretty pictures here — just great advice on how to market to different segments of your customer database including:

  • One Time Buyers — with people that have made only one purchase between the start of the last holiday season and now, here’s your chance to start early (like right now) and entice them to give you a larger share of their gift giving budget
  • Last Minute Big Spenders from Last Season — procrastinators who hold off until December to get gifts; it’s unlikely they’ll change so develop a series of emails just for them that will be delivered starting around December 10th.
  • Best Customers — use a scoring system to identify your VIPs, then use email marketing to reward them for their loyalty (and increase your share of wallet)

A great read for anyone looking to segment and target to improve bottom line performance this holiday season.

All the Right Stuff: Creating Campaign Content that Drives Holiday Sales from Listrak

Listrak Holiday Email MarketingThis Webinar highlights campaigns from Listrak client Naturopathica.

It’s a great case study of reaching out to your social followers (‘social proof’) and leveraging what you learn to inform your email — and other — holiday marketing programs.

Those are three of my favorite resources for this Holiday season — feel free to share some of yours in the comments!

special characters

I’m excited to be leading a session titled ‘Winning Email Marketing Strategies: Improving Opens, Clicks, Conversions and Return-on-Investment’ at ClickZ Live Chicago on Thursday, November 6, 2014. This blog post provides a sneak peak of two of the subject line tips and trends we’ll be covering in this session – visit the ClickZ Live Chicago Website to read the full session description and register.

One more note: I’ll also be leading a half–day pre-conference workshop titled  ‘Beyond Basics: Advanced Strategies and Tactics to Boost the Performance of Your Email Marketing Program’ on Monday, November 3, 2014 – I provide a taste of the advanced email marketing analytics section of that workshop in my 27-Oct-2014 ClickZ column. Hope to see you there!

Subject lines – they are one of the most critical parts of any email marketing  message you send. Subject lines can stop recipients in their tracks and get them to read your email right now — or they be yawn-inducing and drive people to scroll past your message without even thinking about opening it.

There’s actually been some interesting things going on around subject lines in the last few years; here are two trends worth following, along with tips for testing them with your own program.

Special Characters

Special characters in subject lines aren’t new; I feel like, for a while there, almost every subject line had a special character or three in it (well, maybe not every one, but a lot!).

In the most recent analysis I’ve seen, less than 6% of emails sent included special characters in the subject line (see the chart below from Email Marketing Trends for February 2014, Jim Davidson for the Bronto Blog, March 11, 2014).

special characters
As I looked through my various inboxes, in a completely unscientific study, I found only one subject line with a special character in it; it was from Foot Locker:

Footlocker Subject Line short

And while I love the little email image, I have to say that I don’t think it was closely enough tied with the subject line or the message itself — although the message did have to do with my Foot Locker email subscription. Did they test subject lines with and without the special character? And, if so, did the special character boost response rate? Would love to know.

Full disclosure: I did open, read and click on this email – I added my US Postal Service address to my profile because of it. So it worked at least at some level.

There’s anecdotal evidence showing a boost in open rates when special characters are used, but I’ve yet to prove a boost in bottom line performance (read: revenue or conversions). Even so, they are worth testing. The best ‘how-to’ resource I’ve seen is Symbols in Subject Lines, published by Experian back in 2012.


While it’s not new, personalization of subject lines is another tactic worth testing. In their 2014 Email Metrics Report, MailerMailer found that personalized subject lines depressed open rate (see below).

mailermailer Oct 2014 open rate personalization


But when they looked at click-through rate, personalizing both the subject line and the body of the message was the key to optimization (see below).

mailermailer Oct 2014 click rate personalizationI helped a client test personalization earlier this year — our key performance indicator was revenue generated. We found that personalizing both the subject line and the message with the recipient’s first name boosted revenue-per-email by 160% over the control, which had no personalization.

Give these subject lines trends a try and let me know how it goes. For more ideas to boost bottom line performance join me in Chicago next month!


ClickZ Content Marketing Jennings 15-Sept-2014

Note from Jeanne: I wanted to share this in case you missed it — it was one of the top ten most popular articles published on ClickZ in September 2014. Enjoy! 

Content marketing is all the rage these days; 90% of B2C Marketers and 93% of B2B Marketers are doing it, according to recent reports from The Content Marketing Institute (CMI). And email newsletters are a channel of choice for distributing this content, according to more than 75% of the marketers surveyed by CMI.

But like many things, successful content marketing is more difficult that it initially sounds. According to the CMI reports the #1 challenge is lack of time.

ClickZ Content Marketing Jennings 15-Sept-2014

Here are three simple ideas to help overcome it.

Have an Editorial Calendar
Successful content marketing programs are run like traditional editorial publishing groups. You need to work from an editorial calendar and, when possible, produce in bulk and schedule distribution.

A traditional editorial calendar lists the content that will be published, along with some information on sources and deadlines, for a given period going forward. How far forward depends on how frequently you publish, how timely your information needs to be and how organized you are (or want to be).

In general, you want to plan at least a week in advance (if you’re publishing daily) up to three months in advance (if you’re publishing monthly). I don’t recommend publishing less frequently than monthly; once you get to every-other-month or quarterly you won’t likely get the brand and top-of-mind impact you’re looking for.

Editorial calendars can be specific to one publishing vehicle or can cover all the content you publish via all vehicles. It’s your choice. An example of a very simple editorial calendar for a single issue of an email newsletter is below.

ClickZ Editorial Calendar 15-Sept-2014
Another way to save time is to automate the creation of each issue of your email newsletter. This is easiest when you are already publishing the content for your newsletter on your own Website or blog.

Starting from a template with distinct content blocks, a program is developed to pull content for each block from your existing Website or blog. If you’re segmenting your list, you can automate customization based on personas or other criteria.

While this type of automation requires an initial investment, the ongoing cost tends to be very low – much lower than manually creating each issue of your newsletter. The key to success here is developing the initial content block definitions and mapping each to where the content for the block will be pulled from your Website or blog.

While automation like this is, for the most part, ‘set it and forget it,’ it’s important to have someone monitoring the newsletters on a regular basis, just to be sure everything is working as it should. It’s also important to spend some of the time and resources freed up from newsletter creation to do testing to optimize performance on an ongoing basis. For this reason, the most effective automation programs include the ability to perform either A/B split or multivariate testing.

If you were going to automate the newsletter content, your mapping might look something like this:

ClickZ Automation 15-Sept-2014

Add Your Own ‘Take’ to Third Party Content
One of the easiest ways to showcase your organization’s expertise without taxing your internal resources is to start with a third-party article or report and ask a member of your team to read it and provide a value-added comment (typically no more than 100 words) related to the item.

This is a great way to put your organization’s own ‘spin’ on an issue without investing the time, money or resources to develop an original article or report. They key here is NOT to summarize the article, but to add to the points made in it.

This is something I recommended and helped implement for Alchemy Worx; here’s an example:

email worx what we're reading

Give this a try with your email newsletter and let me know how it goes!

Jeanne Signature Oct-2014 smaller still

Sherpa Blog 2010

Happy Throwback Thursday!

One of the things that I love about email and online marketing is the robust tracking and reporting available. The metrics tell you so much about how people are interacting with your email messages and Web pages — and you can use this information to identify areas with opportunity for improvement.

Back in 2010 I wrote a blog post for MarketingSherpa (at the time I was leading their full day Email Marketing Training Workshops in Canada, Germany and the United States) about metrics and numbers that every email marketer should know by heart. While some of the usual suspects (open rate, click-through rate) are on the list, many of the items are things that are often overlooked — even thought they are important for success.

Although the list will be celebrating it’s 4th birthday early next month it’s still relevant, so I decided to share it as this week’s Throwback Thursday (TBT) post.

Here are the first 5 from the list — do you know these numbers cold?

  1. Percentage of New Website Visitors that Sign-up for Email
  2. Abandon Rate on Your Online Sign-up Process
  3. List Size(s)
  4. Monthly List Growth Rate
  5. Bounce Rate

To get more detail on each of these and see the complete list of all ten numbers, read the full original blog post.

What would you add to the list today? I’ve got some ideas — let me know yours (feel free to use the comments section here) and I’ll give you full credit when I write the updated version.

Happy TBT!

Jeanne Jennings Head Shot 11 2013 smaller with spacer

Friday, October 3rd was my last day as Vice President, Global Strategic Services, for Alchemy Worx.

What’s Next?

I’m not 100% certain — but I am excited about what I’m calling ‘Jeanne 2.0.’

In the near term I’ll be doing some independent consulting. Thank you to those who have already reached out to me about projects. If you have work that I might help with, I’d love to hear from you.

In the longer term, the possibilities are endless and I am, as I said, very excited. I may:

    • Join an established agency (either multi-discipline or solely email) to help magnify the growth of its email marketing practice and provide better email marketing strategies and tactics to its clients
    • Become part of a vendor organization to help increase its industry profile and assist clients with better leveraging its technology to boost email results
    • Shift gears to the research side, to become a leading analyst on email marketing issues
    • Move back to the brand side in a Director or higher level position to significantly improve the bottom-line performance of a single organization’s email marketing efforts
    • Strike out on my own to start an email marketing agency that exceeds client expectations and raises the bar with regard to the bottom line performance of email marketing initiatives
    • Do something else that I’ve yet to envision

Once again, if you have ideas on my next career move or if you are with an organization that could benefit from my experience and expertise, I’d love to speak with you.

Some Things Change. Some Things Stay The Same.

Although my professional affiliation has changed, pretty much everything else about me remains the same, including:

  1. I am still passionate about email marketing
  2. I still love helping companies make their email marketing initiatives more effective and more profitable
  3. I still have a proven track record of email marketing success, be it measured by an increase in ROI, revenue, conversion rate or another bottom line metric
  4. I will still be writing my Email Marketing Column for ClickZ and my blog posts for Only Influencers, as well as looking for other opportunities to contribute to the industry’s body of published knowledge
  5. I will still be speaking at industry events — Catch me in November 2014 at ClickZ Live Chicago or the free All About Email Virtual Conference
  6. I will still be tweeting about email marketing (@jeajen)
  7. I will still be an active member of the Email Experience Council, helping with the Annual Conference, the Pollard and Thought-leader Awards and other initiatives that will benefit our industry
  8. I am still one of the World’s Top 50 Email Marketing Influencers

And I have some new tricks up my sleeve, including:

  1. I’ll be blogging about email marketing here at JeanneJennings.com, with a focus on practical tips to improve performance
  2. I’ll be relaunching my email newsletter in the very near future (see the sign-up at the top right of this page)
  3. I’ll be looking for ways to better leverage social media to evangelize about the benefits of strategic email marketing

Email marketing is my passion – it’s what I do, it’s what I love. I can’t imagine my life without it. Nor can I imagine my life without the friends I’ve made – and the friends I look forward to making in the future – in this industry.

Be well. Do good in all things — email and otherwise. Feel free to reach out to me – I always enjoy hearing from industry folks.


Jeanne Signature Oct-2014 smaller still