How to set up a continuous testing culture in your marketing team

The majority of marketing teams like to think of themselves as ‘data driven’ – that is, using the data they collect to plan their roadmap, improve copy & creatives, personalise, segment and ultimately, optimise their campaigns.

One of the most important sources of marketing data is variant testing (also known as A/B testing). A/B testing is essentially setting up 2 versions of a campaign with a single difference between them, routing traffic to them randomly, and then seeing which performs best (according to a pre-selected success metric). In case you’d like a more detailed run through of variant testing, I recommend this comprehensive guide from Hubspot. 

Whilst all marketers have run A/B tests during their careers, many do so infrequently. 

As data driven marketers, we should aim to build a continuous testing culture – to provide the data we need for effective campaign optimisation.

_What is a continuous testing culture?__

Large-scale tests, say a ‘Facebook incrementality test’ or a ‘complete website redesign’ take a lot of time to plan and execute. However, small-scale tests like ‘changing the colour of a website button’ or ‘finding the best time of day to send emails’ can be set up relatively quickly and yield useful results. Most of your tech stack will include some built-in variant testing capability, making it easy for marketing campaign specialists to create, run and analyse tests without support from technical or analytical colleagues. 

In contrast to running infrequent tests, a continuous testing culture is an environment where your marketing team are encouraged to run constant tests – especially small-scale tests. A continuous testing culture in marketing might look like:

• Embedded mindset of experimentation and optimisation. Marketers should constantly look for new opportunities to test things.
• All members of the marketing team are involved in testing. Run an introductory workshop to introduce your training strategy and regular sessions for team members who haven’t done much testing in the past
• New campaigns are always launched as an A/B test, with a control group and a minimum of 2 variants 
• Control groups setup in all existing campaigns, so you can properly judge the success of your efforts
• Tests are coordinated between sub-teams to ensure that the tests don’t target the same user population and skew results
• Regular meetings, both in sub-teams and the whole marketing team, to go over test ideas, plan new tests and discuss results
• High visibility of tests. You regularly output test updates to your shared Slack channel and send monthly email newsletters to the whole company, highlighting the tests you’ve done and insights gained

_What are the benefits of building a continuous testing culture?__

Aside from providing more optimisation data, a continuous testing culture will give other benefits too.

• Help you avoid repeating past mistakes
• Upskill your team members
• Provide data to help you push Product and Engineering requests
• Help you plan new ‘large-scale’ tests
• Easily share insights with other teams in your company, and other companies at conferences (makes you look good 😉

_The importance of centralised planning and reporting__

All those well-planned and impeccably executed variant tests are not much use unless the insights are in consistent format and understandable for your entire marketing team. Although your marketing sub-teams e.g. CRM, Paid, Organic use different tools and have different objectives, A/B test insights are often shareable across functions. 

Let’s take a very simple example. CRM test a CTA with rounded vs. square corners in an email, and the rounded corner button wins. That insight is valuable for the Organic team, or whoever manages your website and landing pages. 

When setting up your variant testing procedure, I suggest you create a single template for recording AB tests that your whole marketing team can use. Key data to record includes:

• Start and end date
• Channel
• Category of the test
• Segmentation
• Success metrics
• Results and statistical significance

I have added an example template to the Martech-hacks resource library. 

How you store and access your test results is also important. 

There are commercial tools available for this, including AB Tasty, AB Smartly and Dynamic Yield. At my current company, Zenjob, we are using a self-built tool (Google sheets, Google App Scripts, with a front end hosted in Google Sites). 

Of course, you could just enter your results directly in a good old Google Sheet or Excel file. Whatever you choose, your tests should be easily accessible – otherwise your super-valuable insights may be forgotten. 

_How to sell the idea to your marketing team?__

Even the easiest to create test is still going to involve extra work for your Marketing team, so it’s important to get them on board. Here are some tips:

• Make testing as quick and easy as possible
• Provide a central place to store test results, which is also easy and quick to input data
• Ensure the test results are publicly visible. For example, output test updates to Slack.
• Celebrate successes, including those team members who are testing often. You could even introduce an element of gamification into your testing procedure! Establish a leader board for number of tests run and give some kind of reward for each month’s winner
• Make sure you actually use your test results and insights!

_What are some things you can test? __

Ok, so you’re ready to kick off your continuous testing. Here are some ideas to get you started:

Good luck with your testing!