Product Talk

Now that I’ve told you to measure what matters, let’s talk about what that means.

A good place to start is by asking, what does success look like? It varies with every product. Let’s look at a few examples:

An e-commerce site.

When I worked at Become.com, a price comparison engine, we bought traffic from Google and other search engines and then sent that traffic to one of our participating merchants. We defined success as spending less money to acquire the traffic than we received to send the traffic to a merchant. If we could do that over and over again, we were in business.

We obsessed over things like what we paid to acquire traffic, our click through rates out to merchants (where we got paid), and keeping our cost-per-click (to the merchant) competitive while maintaining our margins. This business doesn’t leave a lot of room for error.

From a product standpoint, our success funnel looked something like this:

• Click on a SEM ad.
• View a product page
• Click out to a merchant

We invested heavily in optimizing our search engine marketing strategy. This was the top of the funnel. We also optimized our product pages like nobody’s business. I used to dream about click-through-rates. Our goal was to get you to a merchant as soon as possible.

A community management platform.

At Affinity Circles, where we built white-label community software, we were focused on engagement. We helped university alumni associations  engage their members. Our funnel looked something like this:

• Opened an email.
• Visited the site
• Consumed some content
• Commented on content
• Created content.

We tracked each step of the way and worked to move people through the funnel. We also tried to increase the activity at each step along the way. How could we get you to consume more? Comment on more? Create more?

A classic job board.

At my current company, AfterCollege, we help college students find their first job or internship. We define success as simply as, did you get a job through our service? Our funnel looks something like this:

• View a job
• Apply to a job
• Get an interview
• Get hired

And just like in the other examples, we focus on moving people through the steps and sometimes increasing the activity at individual steps.

Applying it to your product. 

Okay, now you do it. What behavior, if it happened over and over again, would make your product a success?

Take a page out of Stephen Covey’s book, begin with the end in mind and work your way backwards.

Share what you come up with in the comments below.

I met with a designer yesterday who shared with me that she wants to learn  user experience design and asked for my advice. I asked her what she was reading and if she had joined any local groups. What had she found useful so far?

She was reading Alan Cooper. Good. She had joined the local product design guild. Good. So then I asked about her job. Did she have an opportunity to learn on the job?

I have a strong bias towards learning by doing. Don’t get me wrong. I love to read and I love to talk shop. But you can’t learn to build great products by reading or talking, you have to learn by doing.

Fortunately, she did have the opportunity to learn on the job. She worked on a team of engineers, where she was the sole designer, and the CEO acted as product manager. Perfect. In this type of environment, there is plenty of opportunity to learn, because it’s much easier to step in and fill a void than to convince someone else to let you contribute.

And then she asked a great question: How do I know if I’m getting better?

And that’s when I asked about analytics. How do you measure the impact of their changes you are making? Do you start with a hypothesis? Do you know why you are doing what you are doing?

Like many companies, her company wasn’t there yet. They were just starting to get their heads wrapped around analytics.

I get it. My company is going through the same thing. Getting your analytics right is one of the hardest things you can do.

But here’s the thing. If you can’t measure the impact of the changes that you make, you are just guessing. Guessing without learning is mostly pointless. You might as well buy a lottery ticket.

You don’t have to get everything right the first time. Just like we take an iterative approach to building products, you can take an iterative approach to analytics. Just start somewhere.

I’m working on several posts about analytics and how to measure what matters. Stay tuned.

Don’t miss a post. Subscribe by email: 

In the last post we ended by concluding that the results of many A/B tests lead to further questions. This is a good thing. Let’s take a look at how you can string A/B tests together to build up a knowledge base over time.

Let’s continue with our previous example where our insight is that creating a sense of urgency in our email subject line will increase open rates. We started by testing the following hypothesis:

• Including an expiration date in the subject line will increase open rates.

But this is just one way to create a sense of urgency. There are many other ways. For example, we could have just as easily tested the following hypotheses:

• Indicating a reward is limited to the first 10 people in the subject line will increase open rates.

Or we could have tested any of the following variations on these same ideas:

• Asking people to act now in the subject line will increase open rates.
• Indicating something is hot in the subject line will increase open rates.
• Highlighting the missed opportunity of not using something in the subject line will increase open rates.

And so on.

Some of these hypotheses may pass. Some may fail. Some may work for some segments of your audience, but not for others. Some may work once. Some may work over and over again.

But they all test your insight of whether or not creating a sense of urgency will increase open rates.

Each test tells us if a hypothesis is true or false for a specific context. If we test a number of related hypotheses, we start to understand the nuances of the original insight. This is where real learning happens.

It’s one thing to learn a tactic – adding an expiration date to a subject line will increase open rates. It’s another thing to learn the why behind the tactic. The more you can uncover about what works and what doesn’t, the closer you can get to the why behind your insight.

It’s one thing to conclude that a sense of urgency will increase open rates. It’s much more valuable to know that expiration dates work the first time, but tend to lose their potency with each use; strong calls-to-action create urgency over and over again; and missed opportunities are the most powerful at engaging new users.

Of course, I just made all those conclusions up. To know whether or not they work in your own context, you have to run your own tests. Every context is different. Every segment is different. The point is to keep learning. Keep coming up with related tests to run. Reach for depth of knowledge. Go beyond the tactics and seek to understand the nuances of your use-cases and your audience.

I leave you with one of my favorite quotes by Richard Feynman. He was talking about so-called experts and what it means to know something. It’s quite applicable here and captures the essence of intellectual honesty:

“See, I have the advantage of having found out how hard it is to get to really know something, how careful you have to be about checking the experiments, how easy it is to make mistakes and fool yourself. I know what it really means to know something. And therefore, I see how it is that they get their information and I can’t believe that they know it—they haven’t done the work necessary, they haven’t done the checks necessary, they haven’t done the care necessary. I have a great suspicion that they don’t know how this stuff is done and they are intimidating people by it.” -Richard Feynman

What do you really know? Have you started to build up a body of knowledge related to your context and your audience? If not, what’s stopping you? 

Now that you are measuring open, click, and conversion rates for your emails, it’s time to look at how to improve them. Let’s talk about A/B testing, sometimes called split testing.

Far too many people hear A/B testing and think button colors and small wording changes.  It is true. You can A/B test these types of changrs. But it’s not where you should start.

Do you remember fifth grade science class? Okay, neither do I. But you probably remember something called the scientific method. Maybe you even participated in a science fair. Don’t worry I won’t tell anyone. You probably followed a process that looked something like this.

• Start with an insight or question.
• Formulate a testable hypothesis.
• Design an experiment to test that hypothesis.
• Run the experiment and measure results.
• Evaluate results
• Draw conclusions.

This is the exact same process you want to follow when you run an A/B test.

Start With An Insight

Don’t start by making a list of each and every change you could make and test every variation. This method will take you years to stumble on anything of value and is akin to throwing spaghetti at the wall. You aren’t a room full of monkeys trying to recreate Hamlet.

You are a smart product manager who can start with an insight. Use your critical thinking skills. Start from the perspective of the email recipient. Why wouldn’t you open this email? Why wouldn’t you click on the content in it? Generate all the reasons. It might look something like this:

Why aren’t you opening my email:

• You get too many emails
• You’ll do it later (and of course never do)
• It looks like spam
• You have no idea who I am
• You don’t think its important

Why aren’t you clicking on the content in my email:

• It’s not relevant to you
• It looks like spam
• It will take too much time
• You’ll do it later

Now ask yourself, how might you overcome some of these problems.

• How can you create a sense of urgency?
• How can you make your email less spammy?
• How can you show credibility?
• How can you improve the relevancy of the content?

This approach helps you look for strategic changes rather than tactical ones. Want to learn more? KissMetrics has a great article about strategic optimization vs. tactics.

Next, generate a bunch of ideas. Take the best ones and write some hypotheses.

Formulate a Testable Hypothesis

A testable hypothesis is a statement that can be refuted by an experiment.

This is such an important part of A/B testing that is so often overlooked that I’m going to say it again. A testable hypothesis is a statement that can be refuted by an experiment. 

Lets look at a couple of examples . You might think that including your brand name in the from line will increase your credibility. So you might write:

• Including the brand in the from line will increase my credibility.

How are you going to test this? How do you measure credibility? If you found a way to measure credibility and credibility increased but people still didn’t open your emails, would you care? Probably not. Try this instead:

• Including my brand in the from name will increase open rates.

This can be measured. You can design an experiment to test this. This is a good hypothesis.

How about this one?

• Including an expiration date in the subject line will create a sense of urgency.

This may be true. Or it may not. How do you measure urgency? How about this instead:

• Including an expiration date in the subject line will increase open rates.

Again, this can be tested. It can be refuted. Or it can be confirmed.

Design an Experiment To Test That Hypothesis

Let’s look at how we might design an experiment to test this last hypothesis.

Many people will just want to change the subject line of their emails and see if their open, click, and conversion rates go up. But that’s not an experiment.

An experiment tests one thing at a time. It has variables and a control. The variable is the thing that you are changing, in this case, the subject line. The control is the old version of your email.

Let’s suppose you send a weekly digest to your subscribers. You might argue that you can compare this week to last week. This week is your variable and last week is your control. But that isn’t a good experiment.

What if last week, 20% of your users lost email access for a day due to an ISP outage? What if this week some of your users read an article about taking action on email as soon as they read it and are highly motivated to apply what they learned? What if last week your email was delivered on Tuesday and this week your email was deliverf on Thursday? There are any number of variables that could interfere with your comparison of last week to this week.

Instead, for any given time period, you want to split your audience into two groups. Your variable group and your control group. You want everything to be the same for these two groups, except for the variation you are testing.

To keep it simple, let’s suppose that for this week’s email, half of your audience is going to get an email with an expiration date in the subject line and the other half is going to get the same subject line you have always used. This is a good experiment. If this is the only change between the two emails then you can attribute any difference in how they perform to the change that you made.

Now you have to be careful to only make one change at a time. Suppose you also want to improve the relevancy of the content in your email with the goal of increasing click through rate. You might think well my first experiment is impacting open rates so you go ahead and make both changes.

This is a problem. Subject line changes can impact click through rate. Suppose the click through rate does go up. You’ll have no idea why. Was it because of the subject line change? Or was it because of the content change?

To avoid this, make sure for each variable and control group you are only testing one change at a time.

Now you can run multiple experiments at once by segmenting your audience. For example, if you have 100,000 subscribers, you can run one experiment on 10,000 of them and a second experiment on the next 10,000. Just be sure that when you segment your audience that your segments are large enough that you are likely to get statistically significant results. More on that in a minute. Also, do make sure each variable has a valid control group for comparison.

Run The Experiment And Measure Results

Once you’re identified your hypothesis, defined your segment, and created a variable group that you can compare to your control group, it’s time to run your experiment.

Before you start, determine for how long you are going to run your experiment and ignore your results until that period ends. Evan Miller explains why. Don’t skip that last link. It explains why your results can be statistically significant one day and not the next. It’s absolutely critical that you set a time period for your test and that you respect it.

Do make sure that you are able to collect the data that you need to evaluate the results. For emails, this includes:

• The number of recipients in the variable group
• The number of recipients in the control group
• The number of people who opened your email in the variable group
• The number of people who opened your email in the control group
• The number of people who clicked on your email in the variable group
• The number of people who clicked on your email in the control group

Note that I specified the number of people not the number of opens or clicks. Unless you get some value out of the same person opening or clicking on your email over and over again, be sure to look at the number of people taking action and not the number of actions taken. 

Once you have all this data you are ready to evaluate your results.

Evaluate Results

For evaluating email tests, I like to put together a grid like so for each test:

Then the very first thing I do is I check for statistical significance. For those of you who aren’t great with math or statistics that can be a big scary word. But it’s this simple. Statistical significance is a measure of how likely your results are repeatable. So for example, if you ran your experiment, how likely would you be to get the same result.

Typically, we say that if you have a 95% chance of repeating the results, your results are statistically significant. Some people push for a 99% chance. It’s up to you. Below 95% chance, you are probably gambling.

Let’s look for a minute at how statistical significance relates to sample size and the size of the difference in your outcomes. If you have a small sample size, let’s say a variable and control group of 1,000 people each, even big changes in the conversion rate may not be significant. You can see on the chart below that it’s not until the difference in conversion rate is 3% before we reach statistical significance.

But as your group size gets larger, the probability of your outcome being significant increase. Here, a difference of only 0.3% is significant.

In both cases, the relative percent increase stays the same. We had to see a 30% increase in our conversion rate before we reached statistical significance. But for our smaller sample, a 30% increase was 3 percentage points, whereas in our larger sample, a 30% increase was 0.3 percentage points.

So a good rule of thumb is that if you are working with small sample sizes, you need to look for big wins. You aren’t likely to get statistically significant results if a change that grows your open rates by 1 or 2% But as your sample size increases, growing your open rates by 1 or 2% percentage points not only has a much bigger impact on your outcomes, but you are also likely to get statistically significant results.

If all of this is too confusing, just remember this. You don’t want to look for a 1 or 2% improvement in your email conversion rates. You want to look for big wins. Ask yourself, how can i go from 20% open rate to 40% open rate, not how can I go from 20% open rates to 22% open rates, and you’ll be fine.

Okay, now that we’ve got a handle on statistical significance, let’s get back to our emails. The first thing you want to understand is what part of your grid is statistically significant.

Use this simple statistical significance calculator to check your results. Start by plugging in the number of recipients and the number of people who open you email for both your variable group and your control group. If the difference is significant, it tells you that the difference in your open rate is statistically significant. If it’s not, you can’t learn anything about open rates from this experiment – even if it looks like you can. It’s really important to build in the discipline from day one to only consider statistically significant results. 

To determine if your click through rate is statistically significant, plug in the number of people who opened your email and the number of people who clicked on your email for both your variable and control group and check for statistical significance. If you are dealing with small variable groups, it can be hard to get statistically significant results for click through rates because they are limited by those who open your emails. So you might have to improve your open rates before you can run statistically significant tests on your click through rates.

Now it is possible that your conversion rates can be statistically significant while your click-through rates are not. To calculate the statistical significance of your conversion rates, plug in the number of recipients and the number of people clicking for both your variable and control group. If your conversion rates are significant, but your click-through rates are not, be careful about he conclusions that you draw. You can conclude that the winning group converts better, but you can’t conclude that your change improves click-through rates. You will need to do further experimentation to determine that.

Draw Conclusions

Now sometimes your results will be significant but negative, meaning they refute your hypothesis. That’s okay. You still learned something.  Here’s a quick cheat sheet of what you can take away from your results.

If your results are not statistically significant:
If your sample size was large, it’s possible that you learned that there your change made no impact. If your sample size is not very large, it’s possible you need to run the test again with a larger sample size.

If your results are statistically significant and support your hypothesis: 
This is what you are hoping to see. You learned that the change you made had the impact you expected.

If your results are statistically significant and refute your hypothesis:
This might seem like a bad result, but it’s not. You still learned something. In fact, you probably uncovered an assumption that you thought to be true and discovered that it’s probably not. While these results can be disappointed, they can have the biggest impact on driving outcomes, as they help you uncover things that you believe that are simply not true.

Regardless of the results, you want to be careful about the conclusions that you draw. You need to be careful that you aren’t overreaching in your conclusions. For example, suppose we learn that:

• Including an expiration date in the subject line does increase open rates.

We don’t want to conclude that creating a sense of urgency increases open rates. This is overreaching. If we test several different ways of creating urgency and they all increase open rates, then we might be able to build a case for this conclusion. But right now, all we know is that including an expiration date increases open rates.

Similarly, we want to be careful about what context we are applying our results to. If we ran our experiments on one audience, the results may not carry over to other audiences. Again, if you run multiple experiments across many types of audiences and your hypothesis holds true across all audiences, then you can start to build your case for the broader generalization.

With A/B testing, it’s really important to be honest with yourself about what you actually learned and where you can apply it. The more diligent you are, the more you will learn. Often times, an A/B test will raise further questions helping you to determine your next round of tests. The goal should be to build up a knowledge base over time that allows you to start to generalize what works for your product and your company. More on that in the next post.

Was this helpful? What did I miss? What questions do you have about A/B testing? Have you encountered any problems that I didn’t cover?