← Back to all articles

Experiments are for learning.

"In order to seek impact, you need to start with prioritized areas of opportunities, not just ideas or even prioritized ideas. Problems can be noisy, but they don’t necessarily correlate with being the biggest growth opportunity." - Brian Balfour

The role of a growth team, much in the same way as a data science team, is to learn as much about the customer as possible.

The added benefit of a growth team is that they can utilize experiments to learn/test hypotheses that existing datasets cannot provide the answer to.

For example, your dataset might show you how many people are visiting your site from LinkedIn, but it might not tell you which product features they're most interested in. For that, the growth team might launch an experiment to create specific use case landing pages, and using ads, see which use case has the highest click-through rate with their target audience.

Designing Hacks

When I first started out at Qatalog, I set up a simple experiment template in the format of:

This template was useful in helping us create experiments that were solely focused around "hacking" or improving metrics such as conversion rate or unique visitor numbers.

But after running a few successful experiments, we realized that although these "hacks" did produce improvements in our target metrics, we failed to derive any sort of new learnings from our experiments other than "the new redesign was an improvement over the old design".

This made us rethink how we approached experiments.

Designing Experiments

During a long weekend, I stumbled upon Hubspot's Growth Process.

Hubspot's focus was on forming quality experiments. The growth team at Hubspot spent time looking at data and formed hypotheses around what they were seeing.

They added learning outcomes to ensure that each experiment gave them new insights, and a business outcome so that stakeholders were also incentivised from the experiments.

Taking inspiration, I created a new template for experiments going forward:

Note: We share all of our experimental findings with the rest of our company using Qatalog's Post Feature

Finding Opportunities

Every company has a set of core metrics that are essential to its business. If you're Facebook, it's the average time that is spent on the site or if you're Groupon it's the number of coupons redeemed.

In order to increase those core metrics that power your startup. You have to first understand the customer. Start by looking at your power users - these are users who are the outliers in your database. For example, if people on average post on Facebook twice a day, then your power users are those who post ten to fifteen times a day.

Your aim in doing this is to increase these types of users or find out what causes them. Start by asking "Did these power users all come from one single channel? What do they have in common with each other? What led them to become power users? How can we make it easier for them to post on Facebook?"

The opposite can also be helpful. "Why are these users not engaging with the platform? What's common among non-active users? What led them to become inactive?".

These questions will lead you down a research rabbit hole that will in turn inspire experiments to be created to find out if certain hypotheses are correct.