The Art of Iterative Testing: Test Everything!

Part four in the Creating a System for Success series

The best way to move fast is to test a lot! Testing will give you an indication of the right direction you need to take and is also one of the best ways to improve performance. For example, for us to get a piece of content to work, on average, we test several dozens of variations in a time span of two months before something hits, and only then do we start to lower the pace of testing.

It took us time to master the art of testing. Here are a few tips that will help you get there:

The right split testing tool

You want to have a tool that will grant you the freedom to test without needing a team of developers and designers. A tool that will give you the ability to do A/B testing pretty easily and see results in a digestible report. There are a bunch of really nice tools out there that will give you control without breaking the bank.

Testing schedules

One of the biggest challenges with fast iteration is the time between one test and another. Waiting for results from the test and preparing a new test based on the results might take weeks and sometimes even more. That’s why it’s a good idea to work with a testing schedule. This way, the team can prepare a few tests simultaneously, allowing us to move much faster as a company.

Narrow down variables as much as possible

Not only do you want to know which variation is doing better but why it’s doing better. This is why it’s important to understand what actually made the difference so you can apply these learnings to other marketing content. That could be problematic if you are testing many different things at once. Therefore I suggest staggering the tests as much as you can and do them one by one. This might take longer, but the insights are going to be worth it.

Statistical significance as a double edge sword

The traditional way is to get to statistical significance before analyzing the data. I have a bit of a different approach. I agree that you should move with caution on the very sensitive tests and wait for statistical significance, but these will be only 5% — 10% of the tests you will conduct. For the rest, waiting for statistical significance will slow down the pace dramatically and might slow down progress.

In the online DTC space, you will need to move faster to make an impact. Therefore, if you see a significant difference before reaching statistical significance, you can move on to the next test. If they are performing the same, this means the test will most likely not have an impact, and you can test something else that might actually show results.