
Often a startup will offer multiple features, like task management, video calls, messaging, etc. 

As a data scientist or growth engineer, you're always on the hunt for causal relationships as they can be optimized to drive results. In order to find casual relationships, you often have to first find correlated metrics and then run randomized controlled experiments to determine if the two are indeed casual or not.

Take this correlation matrix below:

![Correlation Matrix Example](/static/correlation-matrix.jpeg)

Where 1 is perfect correlation (i.e: impossible or it's the same metric), and where -1 is absolutely no correlation.

We can see that the final exam score and hours spent studying have a high correlation of 82%. On the flip side, it might surprise you that hours spent sleeping and exam score were not highly correlated. It's only an example but it shows the learnings you can derive from the matrix.

## How do you calculate correlation?

In statistics, the Pearson correlation coefficient is a measure of linear correlation between two sets of data. In even simpler terms it measures the closeness of movement between two linear datasets.

You can calculate it using the formula below:

![Coefficient](/static/coefficient.gif)

### Example: Tasks created versus Messages Sent

Take this dataset, where each week we calculate the total number of tasks and messages created:

| Week | Tasks created (x) | Messages sent (y) |
|------|-------------------|-------------------|
| 1    | 20                | 7                 |
| 2    | 24                | 8                 |
| 3    | 22                | 9                 |
| 4    | 27                | 12                |
| 5    | 21                | 7                 |
| 6    | 23                | 8                 |

1) First step is to make a new column called xy which is the multiplication of x and y:

| Week | Tasks created (x) | Messages sent (y) | xy  |
|------|-------------------|-------------------|-----|
| 1    | 20                | 7                 | 140 |
| 2    | 24                | 8                 | 192 |
| 3    | 22                | 9                 | 198 |
| 4    | 27                | 12                | 324 |
| 5    | 21                | 7                 | 147 |
| 6    | 23                | 8                 | 184 |

2) Next, we created an x<sup>2</sup> column, where we square the result of tasks created (x):

| Week | Tasks created (x) | Messages sent (y) | xy  | x^2 |
|------|-------------------|-------------------|-----|-----|
| 1    | 20                | 7                 | 140 | 400 |
| 2    | 24                | 8                 | 192 | 576 |
| 3    | 22                | 9                 | 198 | 484 |
| 4    | 27                | 12                | 324 | 729 |
| 5    | 21                | 7                 | 147 | 441 |
| 6    | 23                | 8                 | 184 | 529 |

3) The same for y<sup>2</sup>:

| Week | Tasks created (x) | Messages sent (y) | xy  | x^2 | y^2 |
|------|-------------------|-------------------|-----|-----|-----|
| 1    | 20                | 7                 | 140 | 400 | 49  |
| 2    | 24                | 8                 | 192 | 576 | 64  |
| 3    | 22                | 9                 | 198 | 484 | 81  |
| 4    | 27                | 12                | 324 | 729 | 144 |
| 5    | 21                | 7                 | 147 | 441 | 49  |
| 6    | 23                | 8                 | 184 | 529 | 64  |

4) Then we simply add up all of the numbers in the columns and put the result at the bottom of the column. The Greek letter sigma (Σ) is a short way of saying “sum of” or summation.

| Week | Tasks created (x) | Messages sent (y) | xy    | x^2   | y^2 |
|------|-------------------|-------------------|-------|-------|-----|
| 1    | 20                | 7                 | 140   | 400   | 49  |
| 2    | 24                | 8                 | 192   | 576   | 64  |
| 3    | 22                | 9                 | 198   | 484   | 81  |
| 4    | 27                | 12                | 324   | 729   | 144 |
| 5    | 21                | 7                 | 147   | 441   | 49  |
| 6    | 23                | 8                 | 184   | 529   | 64  |
| Σ    | 137               | 51                | 1,185 | 3,159 | 451 |

5) Then we can now calculate the correlation coefficient between x and y:

![Coefficient](/static/coefficient.gif)

```
n = sample size (which in our case is 6 rows)
Σx = 137
Σy = 51
Σxy = 1185
Σx^2 = 3159
Σy^2 = 451

R = (6 * 1185 - 137 * 51) / sqrt(
  (6 * 3159 - 137^2) * (6 * 451 - 51^2 )
) = 0.88 (88%)

```

88% signifies quite a high correlation between tasks created and messages sent. From here we can develop our own hypotheses for why this might be:

* Do users who create new tasks then message other users to talk about the task?
* Is there an issue with our data collection where a task will trigger a message?
* If we reduce the number of tasks being created will that also reduce messages sent?
* Etc etc.

Remember kids: [Correlation does not mean causation](https://www.theguardian.com/science/blog/2012/jan/06/correlation-causation). Use [randomized controlled AB experiments](https://hbr.org/2017/06/a-refresher-on-ab-testing) to determine the casual relationship of the two metrics.

Creating correlation matrixes to generate hypotheses


Sick of protein shakes and oats? Make hitting your protein target a fancy affair with this protein raspberry panna cotta

## Ingredients

* 270ml light coconut milk
* 2 scoops Impact Whey Protein (Vanilla)
* Sweetener of your choice
* 100ml water
* 1 sachet gelatine (7g)
* ½ tsp. vanilla extract
* 180g raspberries

## Method

1. Combine coconut milk, water, and the 2 scoops of whey into a blender or protein shaker and mix till it's lump-free.

2. Pour into a bowl with gelatine – leave for a few minutes before mixing, then add the vanilla extract along with a sweetener of your choice for taste.

3. Transfer to a saucepan, on a low heat gently warm up the liquid to melt the gelatine (do not let it boil).

4. Pour into 1 large ramekin or 2 ramekins, and place in the fridge to chill for a minimum of 4 hours or overnight.

5. Meanwhile, prepare the raspberry sauce by blending the raspberries in a blender and then straining through a sieve to remove the seeds. Chill in the fridge.

6. Once the panna cotta has set, serve by carefully flipping the ramekin over a plate. Cover with the strawberry sauce and a few extra strawberries for a finishing touch.

Protein Raspberry Panna Cotta


I've been asked on numerous occasions what differs a "great" growth engineer from just a "good" growth engineer. In my mind it came down to doing three things incredibly rigorously:

## 1) Enablement (60%)

> Org Growth is the summation of all it's moving parts.

If like myself you've worked in startups, you'll come to realize all the little inefficiencies and manual operations currently in practice. As a growth engineer enabling others will grow the company much faster than your [experiments](/blog/experiments-are-for-learning/) will.

For example, at [Qatalog](http://qatalog.com/) we needed a way to build landing pages at scale for our SEO, and after building 10-15 pages using our CMS, I realized that adding new component to a single page type took 2-3 minutes of manual CMS management. That's a problem, because if we scaled to 100 pages that equates to 5 hours of CMS editing time.

I resolved this issue by creating one singular template page for each page type in the CMS and used a Google Sheet to interpolate values onto that template (like how an email template renders first name and last name). Now all the editors have to do is copy and paste in the values for new pages/components. Hours saved.

## 2) Experiment (20%)

> The pretense of knowledge is our most dangerous vice, because it prevents us from getting any better - Ryan Holiday

Opinions are a growth engineer's pet peeve because they're not grounded in research. **A truly data-driven startup doesn't waste time on opinions but rather aims to answer questions using experiments or research.**

It all starts with a question that you cannot answer with the data you currently have available (often times you'll realize that you're not tracking the product correctly).

As a growth engineer, you should learn about sample sizes, AB or multivariate tests, statistical significance, and growth models. It's up to you to enforce creating experiments with good background research and clear learning outcomes.

## 3) Research (20%)

> Slow is smooth smooth is fast - US Navy

Startups move so fast that they often don't take the time to look deeply at their own data and customer research.

As a growth engineer, being proficient in SQL is a hard requirement because at some point you'll need to dig into the data beyond what Heap and other analytical tools will offer you.

Alongside data scientists, you'll be working on creating dashboards to provide analytical learnings for yourself and the rest of the company (enablement).

I can't stress how ill prepared a lot of startups are in building dashboards and tracking the correct metrics.

Enablement, Experiment, and Research


Here's my no fluff guide to making my upgraded version of a carbonara.

Carbonara originated from Rome, Italy dating all the way back to the 20th century. The dish comprises of **eggs**, **hard cheese** (usually Pecorino Romano or Parmigiano-Reggiano), **cured pork** (guanciale, pancetta or bacon lardons), and **dried pasta** (commonly fettuccine, rigatoni, linguine, or bucatini).

**Why not fresh egg pasta?**

Carbonara is already an egg based dish. Adding additional egg based pasta to the recipe would end up making the overall dish quite heavy to eat.

**Is it okay to freestyle it if I don't have all the traditional ingredients?**

That's the joy of cooking! As long as you follow the carbonara formula of _eggs + hard cheese + cured pork + dried pasta_ you'll end up with a delicious carbonara result.

_Sorry Italians._

![Confit Egg Yolk Carbonara](/static/carbonara.jpeg)

## Recipe

### Ingredients

**Carbonara Sauce**

* 1 large egg
* 4-5 large egg yolks
* 1 wedge of hard cheese (I used Parmigiano-Reggiano)
* Black Pepper
* 25-75ml of hot reserved pasta water

**Carbonara**

* Salt
* 125g of Cured Pork (I used bacon lardons)
* 2 tbsp of Refined cooking olive oil
* Handful of dried pasta (I used spaghetti)

**Confit Egg Yolk**

* 1 egg yolk
* Refined cooking olive oil (not extra virgin)
* 2 skin-on cloves of garlic (cracked using pressure from your palm)

### Confit Egg Yolk

First we'll start with the longest part of the dish. 

1. Set your fan assisted oven to 65C.

2. Grab an oven safe dish or pan and add in your **egg yolk**, **garlic cloves** and enough **olive oil** to submerge everything.

3. Bake for 55 minutes and then spoon out **egg yolks** for plating later on.

### Carbonara Sauce (Part 1)

1. In a large bowl add **1 egg**, **4-5 egg yolks** and beat till combined. 

2. Grate in enough **hard cheese** until the sauce becomes thick and no longer watery. Add **black pepper** (more than you think is necessary).

3. Leave to one side.

### Carbonara (part 1)

1. Set a pan of water over high heat, add **salt**, and once boiling add your **pasta** for the time listed on the packaging.

2. Whilst the **pasta** is cooking, set a pan over medium-high heat, add **1-2 tablespoons of refined cooking olive oil**, and **125g of cured pork**.

3. Cook until the **cured pork** has started to brown on the edges but still retain the chew/bite.

### Carbonara Sauce (part 2)

Take **25-75ml of your pasta cooking water** and slowly pour it into your **egg mixture** while whisking vigorously. This will temper the **egg mixture** and stop it from scrambling in the hot pan.

### Carbonara (part 2)

1. Using tongs, grab the al dente **pasta** from your pot of water and dump it into the pan with your **cured pork**. Mixing the **pasta** with the **olive oil** and rendered fat from the **pork**.

2. Add in your egg mixture and stir everything vigorously till combined. If the sauce is too thick then add **pasta water** to loosen it up. It should be thick enough to look creamy and still stick to the pasta. 

### Plating

1. Grab the plate of your choice (ideally something large and deep-plated).

2. Using a ladle and tongs, much like how you'd twirl pasta to eat it, twirl the pasta into the ladle, and then slide the pasta off into the dish ([video for reference](https://www.youtube.com/watch?v=3P2Bdp3eoTs)).

3. Grate **hard cheese** lightly over the top and then slide your **confit egg yolk** on top of the spaghetti. (optional: add a slight drip of **truffle oil** on top of the egg yolk to add an extra layer of depth to the dish.)

4. Serve to the delight of your friends and family.

![Confit Egg Yolk Carbonara](/static/carbonara.jpeg)

Confit Egg Yolk Carbonara


> An actionable metric is one that ties specific and repeatable actions to observed results. - [LeanStack](https://blog.leanstack.com/3-rules-to-actionable-metrics-in-a-lean-startup/#:~:text=An%20actionable%20metric%20is%20one,or%20what%20to%20do%20next.)

No matter if you're a tech SaaS company or a bakery. You should have metrics at hand which displays the performance of your business.

These metrics should be in the format of a time-series so you can compare periods and react to changes in a weekly/biweekly/monthly basis.

## Finding actionable metrics for your business

**1. What action is core to your business model?**

_Facebook relies on users spending time on their site, so improving **Avg Session Time** would impact their Ad Earnings_.

_An Instagram Page relies on followers clicking their bio link to buy their merch, so improving **Bio Link CTR** and **Website CR** would increase their e-commerce earnings._

<textarea></textarea>

**2. Now set a cadence for it (i.e: daily/weekly/biweekly/monthly)?**

_If your business has customers that only visit you once a month then set a monthly cadence._

<textarea></textarea>

Actionable Metrics


> "In order to seek impact, you need to start with prioritized areas of opportunities, not just ideas or even prioritized ideas. Problems can be noisy, but they don’t necessarily correlate with being the biggest growth opportunity." - [Brian Balfour](https://brianbalfour.com/essays/growth-insights)

The role of a growth team, much in the same way as a data science team, is to learn as much about the customer as possible.

The added benefit of a growth team is that they can utilize experiments to learn/test hypotheses that existing datasets cannot provide the answer to.

For example, your dataset might show you how many people are visiting your site from LinkedIn, but it might not tell you which product features they're most interested in. For that, the growth team might launch an experiment to create specific use case landing pages, and using ads, see which use case has the highest click-through rate with their target audience.

## Designing Hacks

When I first started out at [Qatalog](https://qatalog.com/), I set up a simple experiment template in the format of:

* **North Star Metric** - What is the metric that we're trying to improve?
* **Problem** - What is the customer problem that we're trying to fix?
* **Solution** - What is the solution that would fix the customer problem?
* **Test Method** - How are you going to test the solution to the control?
* **Success Criteria** - How much does the metric need to change in order to give us confidence in the result?
* **Result & Decision** - Did we change the metric significantly & learnings.

This template was useful in helping us create experiments that were solely focused around "hacking" or improving metrics such as conversion rate or unique visitor numbers.

But after running a few successful experiments, we realized that although these "hacks" did produce improvements in our target metrics, we failed to derive any sort of new learnings from our experiments other than _"the new redesign was an improvement over the old design"_.

This made us rethink how we approached experiments.

## Designing Experiments

During a long weekend, I stumbled upon [Hubspot's Growth Process](https://docs.google.com/presentation/d/1BOI0vR1lyP6reaMc85FcHltfGo-5nT-rVZAYfvpEQEw/edit#slide=id.g452fc8e996_1_838).

Hubspot's focus was on forming quality experiments. The growth team at Hubspot spent time looking at data and formed hypotheses around what they were seeing.

They added learning outcomes to ensure that each experiment gave them new insights, and a business outcome so that stakeholders were also incentivised from the experiments.

Taking inspiration, I created a new template for experiments going forward:

* **Background** - Provide context on the research you’ve done and the thought process behind why this experiment is worth running.
* **Hypothesis** - Something that we believe to be true based on what we know from research.
* **Learning Objective** - What do we hope to learn by running this experiment?
* **Business Objective** - What do we hope to accomplish by running the experiment?
* **Prediction** - What do we believe will happen if our hypothesis is true?
* **Experiment Design** - How are we designing this experiment to test our hypothesis?
* **Predicted Outcome** - The expected results of the experiment.
* **Experiment Post Mortem** - An open discussion at the conclusion of a project, where you and your team can identify and analyze all aspects related to the project.
* **Process Post Mortem** - Is there anything we should do better next time?
* **Observation** - If the experiment was successful and deployed, after 2 weeks what was the impact?

_**Note**: We share all of our experimental findings with the rest of our company using [Qatalog's Post Feature](https://qatalog.com/posts/)_

## Finding Opportunities

Every company has a set of core metrics that are essential to its business. If you're Facebook, it's the average time that is spent on the site or if you're Groupon it's the number of coupons redeemed.

In order to increase those core metrics that power your startup. You have to first understand the customer. Start by looking at your power users - these are users who are the outliers in your database. For example, if people on average post on Facebook twice a day, then your power users are those who post ten to fifteen times a day.

Your aim in doing this is to increase these types of users or find out what causes them. Start by asking "_Did these power users all come from one single channel? What do they have in common with each other? What led them to become power users? How can we make it easier for them to post on Facebook?_"

The opposite can also be helpful. _"Why are these users not engaging with the platform? What's common among non-active users? What led them to become inactive?"_.

These questions will lead you down a research rabbit hole that will in turn inspire experiments to be created to find out if certain hypotheses are correct.

Sun Oct 10 2021	Creating correlation matrixes to generate hypotheses	growth
Sun Oct 10 2021	Protein Raspberry Panna Cotta	cooking
Tue Aug 24 2021	Enablement, Experiment, and Research	growth
Wed Jul 28 2021	Confit Egg Yolk Carbonara	cooking
Fri Jul 02 2021	Actionable Metrics	growth
Thu Jul 01 2021	Experiments are for learning.	growth