The Beginner’s Guide to Designing Marketing Experiments
November 14, 2017 | Justina Logozzo
In college, I quickly learned that science was not my forte. From memorizing the periodic table and parts of the plant cell to stoichiometry and the laws of physics, I was not an A or even a B student by any means. So, I became a marketer, because marketers don’t do science things.
Boy was I wrong.
While marketing doesn’t require hazardous chemicals and scientific equations, it does require continuous experimentation. In order to create a successful marketing strategy, marketers need to test, optimize, and analysis their content.
Let’s walk through the steps required to run experiments, get results, then implement them to improve CRO (Conversion Rate Optimization) for landing pages, email, social, or any medium that allows for experimentation.
Step 1: Define Your Purpose & Objectives
You need to know the purpose and the objectives of your experiment. Without them, you’ll be unable to know what the variations will be or how you’ll determine if the experiment was successful.
Go back to the marketing fundamentals and ask yourself WWWWW (who, what, when, where, and why).
Who: Define who you are going to target with this experiment. This will guide you in selecting the content, the channels, and the messaging you use.
What: This pertains to two things: what content you’re experimenting with and what you expect to happen. This could be a landing page CTA, a paid media ad, an ebook title, etc. You need to know the ‘who’ in order to start the experiment. When selecting your content, make sure it has ample opportunity to be successful. If you choose an ebook that is irrelevant to today’s industry or a blog that isn’t receiving much traffic, it probably isn’t going to yield good results.
Additionally, brainstorm what you think will occur during this experiment – a hypothesis. Generate a list of what you think will happen, similarly to when creating a scientific method. This could range from a higher conversion rate, increased unique visits, more social shares, etc.
When: You need to know when you’re experimenting, whether it’s throughout the next six weeks or three months. Outlining your timeframe helps with the creation of attainable goals.
Where: This is where you’d define the channels, like email or social, that you’d want to utilize within the experiment.
Why: This is very important to know before you start an experimentation. You need to know why you’re wanting to experiment with the content selected. Without this, you’re unable to align goals or create a success strategy.
To provide clarity, I’ll walk you through a recent marketing experiment we conducted at SnapApp. We have a free tool called LeadREV, which helps to transform static PDF documents into a more dynamic experience in just seconds. During a period over the summer, we saw a dip in overall traffic to the tool.
To experiment, we decided to test the LeadREV CTA text on our homepage. Since this CTA is the first place visitors see when going to our site, it was an ideal test subject.
When planning our experiment, we hypothesized that revising the language to be more directly relevant to the content on the page would (ideally) encourage more conversions. Our timelines for the experiment was set for three months, August through October, and the CTA change would only take place on our homepage.
This example walkthrough will continue in the next step.
Step 2: Define Your Goals
Before you do any true “experimentation”, you’ll need to define your goals. Your goals should align with your answers to the five Ws, especially to your “why” answer.
Your goals can range between a number of metrics, such as more clicks, improved organic traffic, increased MQLs, reduced bounce rate, and so on, but you can’t tackle all of these with one experiment, because you wouldn’t be able to identify the influencing change for each one.
A successful experiment uses numerically, measurable goals as opposed to something such as brand awareness, which is more subjective.
In addition, your goal should be reasonably attainable within the timeframe selected. If you’re experimenting for a month, you can’t expect to increase organic traffic 150% as that isn’t how search engines work.
Back to the Example:
With our LeadREV experiment, our goals were simple: improve CRO and traffic. With May-July producing an average of 21 LeadREV conversions and 571 total visits, we opted to set a 25% increase goal as it would be attainable and signify growth.
Step 3: Plan Out Your Approach
This is where you address the “how” of you experiment. How are you going to do it and how are you going to monitor its success?
First, you’ll need to outline the ideas you’re doing and the steps you’re going to take to make them happen. You’ll need to decide what you’re going to change within the content, such as language, call-to-actions, images, colors, graphics, and so on.
However, changes need to be made incrementally, i.e. only one change at a time against a control to know what is impacting the results.
Here’s a handy worksheet that will help with organization.
You’ll need to identify your experiment’s test subjects. The test subjects will consist of one control (i.e. the original unchanged version of your content) and multiple variations of the original content. By doing this, you’ll be able to effectively judge which aspect of the experiment works best.
Finally, build a timeline for when your experiment will start, when variables will be introduced, how long variables will be tested, and when the experiment will come to an end. Having a timeline will help in gauging success as well as keep your variables from cannibalizing each other and mixing results.
Here are some possible options for you to start experimenting with:
- Create four different versions of a landing page, then show site visitors the five versions (including the control) for three weeks to see which results in the highest conversion rate
- A/B test sending the same email at different times of day with a group large enough to yield significant results
- Interview 50 current users and gather demographic and qualitative data about them, then analyze the data in your contacts database to find the most prevalent persona
Back to Our Example:
The plan we concocted for the LeadREV CTA consisted of:
- Outline new messaging that better aligns with the tool’s purpose
- Make changes to the CTA banner: 1.) A button will replace the hyperlink. 2.) Colors, placement, and CTA link location will remain the same
- Implement the new banner on the homepage August 1st
Step 4: Create, Test & Monitor
Now that all of your ducks are in a row, it is time to start the actual experiment. This is when your planning comes into play as you’re able to create and activate your test content accordingly.
As your experiment is running, there is one key thing you have to keep in mind: do not make changes until it is over. Any changes made during the experiment will deliver skewed results and will not show the true impact your initial changes made to your content.
When monitoring, document everything you find to be important or interesting. If you had a spike in social share on Wednesday at 3PM, write that down.
Those notes will enable you to have a clearer idea of the circumstances that surrounded your experiment, as well as provide insights into preventing the biggest problem of all experiments: failure to replicate.
Monitoring also enables you to know when something breaks, such as a failed URL redirect or broken images. If something breaks, you’ll need to fix it as soon as possible as that can invalidate the result you gather at the end.
While you have an outlined timeline for the experiment, in some cases it may not be sufficient enough to gain enough data to declare it a success/failure. If that occurs, you can extend the timeframe and run your experiment for as long as you need to determine statistical significance.
Back to the Example:
As we outlined in our plan, we revised the CTA wording and added a button, while leaving the color scheme and location as is.
While the experiment was underway, we consistently checked our Marketo and Google analytics to ensure that (A) the experiment was working and (B) we were seeing results.
We pulled Marketo data on a weekly basis and found that LeadREV experienced the most conversions during the second and third weeks of the month.
As for traffic, we’d check it each morning by looking at overall LeadREV traffic as well as traffic generated specifically by the homepage banner.
Step 5: Analyze & Optimize
Your experiment is over and now comes the fun part, analyzing the results. This is where you’ll compile all of your data and narrow down the results into a digestible format to determine if it was a fail, success, or inconclusive.
When reviewing data, analyze the metrics of the new versions and determine if you’re looking at a legitimate statistical increase in the goals you set as well as compare them to your control subject.
When classifying the result of the experiment, keep the following in mind:
- A failure means that the results you received to do not reflect the goals you set.
- An inconclusive result means that the analysis doesn’t show enough change to draw a conclusion.
- A success mean that your experiment met or exceeded your goals.
Once you’ve analyzed and digested the results of your experiment, regardless of fail/success, start to brainstorm how you could optimize the content pieces. As I mentioned in the beginning, marketing is continuously evolving meaning we, as marketers, need to keep experimenting.
The Results of Our Example:
Once November 1st rolled around, we exported all of our data and began analyzing it. In terms is conversions, we saw an average of 34 conversions per month, which is a 60% increase from the three months prior. As for traffic, LeadREV went from generating ~571 visits per month to over 1,400 visits a month, a 150% increase. With these results, we dubbed our experiment a success as we exceeded our 25% growth goal.
After seeing the impact the new messaging made, we began to integrate it into our overall LeadREV marketing strategy while also continuing the use of the new CTA banner.
So, What Are the Key Takeaways?
Figure out your purpose and objectives
Define your goals in alignment with purpose and objective
Plan out your approach and what you’re going to do
Create, test, and monitor so you know how things are going
Analyze the data and optimize for your next experiment
Test-and-learn experiments serve an important purpose for your content as it’s a way to ensure it’s performing at a maximum level of effectiveness, and improves the experience for your user.
Your website is an incredibly powerful conversion channel – unless no one is converting, so experiment and get the most out of it.