A practical guide to improving product retention

In both of the startups I have been involved with, I learned the hard way that there is a massive difference between customer retention and user retention. Particularly in B2B SaaS, startups can scramble to retain customers even if their user adoption isn’t high. As a founder, you can ask customers to trust you and promise that things will get better, but if you don’t address the underlying issue of user retention, you will inevitably see customer churn that will stunt your growth.

In my last role as a product manager, one of the things we struggled with was figuring out which metrics were most important to measure. We knew that user retention was a bit of a blind spot, so we started there and set out to see the light.

We began by looking at the number of users logging in during a given month as an indication of retention. Seems logical, right?

Well, that number was skewed drastically because many customers had Single Sign-On for our app. Our system indicated they had “logged in”, even if they hadn’t actually used the product or gotten any value from it.

Next, we thought to look at the number of users that were performing the core action in a given month. Surely, that would be a good indication of retention!

Wrong. Just because someone performs an action once in a month doesn’t mean they are truly seeing value.

What I’m getting at here is that retention metrics can be misleading. If you don’t understand what the promised land looks like for your users, you will struggle to lead them there.

If you don’t understand what the promised land looks like for your users, you will struggle to lead them there.

In this post, we’re going to do the following:

1. Outline a few simple frameworks for fine-tuning how you measure retention
2. Map out the activation flow that drives users towards retention
3. Show how machine learning (ML), trust and conversational experiences can help accelerate your activation flow

Getting Started With Personas

When you commit to going deep on product retention, it is easy to get sucked in by product analytics tools right away. I’m guilty of being technology trigger happy on many occasions, but this is a classic case of slowing down to speed up. Start with pen and paper.

The first step is to build out your user personas, which provides clarity on the types of users you want to measure retention for.

Each user persona should have the following:

1. Problem the user faces
2. Specific business function / role they are in
3. Common personality traits
4. Value proposition for the product
5. Alternatives to using your product

Make sure to build these personas for all of your users, and feel free to add in any other sections that you think are helpful.

Have gaps in your understanding of some personas? Those make for good research to-do’s.

Think your personas are broad? Consider breaking them out into more specific personas, based on role, company size, location, etc. Only do this if it materially changes the details of the five items listed above.

The Retention Metric

This is where it starts to get fun – defining your retention metric. Simply put, the retention metric is your definition of what a user must do on a regular basis to be considered retained. This metric has two simple components: core action and natural frequency.

The core action is the main task a user performs within your product that leads to their value proposition. How often a user completes the core action and realizes the value proposition differs for every product – the rate at which they go through that process is called the natural frequency.

Let’s use Slack as an example here. Slack’s core action is sending a message, and it has a daily natural frequency. If Slack’s product team defined a retained user as someone who sends a message monthly, that would look very different than the # of users messaging daily. 

The last piece of the puzzle here is around volume. If Slack focuses on daily unique messagers, is a user sending only one message per day going to lead them to be retained? Not likely – maybe it’s 5, 10, or 20. I certainly don’t know the answer and will leave it up to their smart product folks, but here is a template you can use to create a hypothesis for your product.

Now, you’ll want to validate this hypothesis for each persona with both qualitative and quantitative analysis. Go out and set up interviews with a cross-section of your user base, but definitely get a solid block of time with your most passionate users, and some users who have churned. Cross-reference what you learn with usage data to make sure things line up.

After you go through all of these steps, you should have a much more clear understanding of the retention metric for your key user personas.

Visualizing and Segmenting Retention

Once you feel confident in your retention metric, that becomes your measuring stick for product analytics. But now that we know what to measure, how do we do the measurement? While every business is different and we certainly don’t have all the answers, we have found these three ways to visualize usage data to be particularly helpful and easy to get started: frequency histograms, cohort analysis, and segmented retention curves.

Let’s look at an example of each below:

The chart helps you understand how many times your users are performing your core action during a given time period. This can help you quickly visualize the biggest areas of opportunity – ie. building strategies to shift casual users to core users.

Cohort analysis is a common way to visualize your user retention funnel – it looks at a group of users acquired in a given timeframe and measures how many of them continue using the product over time. In other words, your core action completion percentage plotted against your natural frequency. Once you have your baseline done, there are infinite variations of this type of analysis… definitely do some investigation to find which methods are most helpful for your business.

Once you have your cohort analysis data, use can use it to plot retention curves like the one above. In this case, the graph portrays a massive dropoff in users completing the core action after their first week. This indicates that the product has an activation issue and that the onboarding experience likely needs a revamp.

Segmenting retention curves and evaluating how they change over time can be very eye-opening. Here are a few ways you can slice the data:

• Segmenting by weekly/monthly cohort helps you see whether product changes move the needle
• Segmenting by user persona helps you understand if some users build habits more easily than others
• Segmenting by feature usage or device type helps you understand how specific usage patterns correlate to user retention
• Segmenting by acquisition channel helps you prioritize go-to-market channels based on the likelihood of users from each channel being retained

Activation

They say you never get a second chance to make a first impression, and that is especially true in software. Every new user, every new cohort that passes through, they all have one thing in common – they will go through your activation flow. Carefully designing this flow and guiding users in the right direction is absolutely critical to retention.

There are two frameworks that we like to reference when it comes to activation. The first framework involves the ways you communicate with your user during activation.

I particularly like this graphic from Intercom because it makes you think about activation holistically. Onboarding is much more than form fills and tooltips. It is about getting the user to see the promised land for the first time – a land in which their life is markedly more efficient, more prosperous, or simply more fun because they have your magical software.

Getting there requires thoughtful product design, contextual communication both within and outside your product, and contextual content to help users overcome roadblocks or more clearly understand what they need to do to be successful.

Here’s a simple example of how these three communication mechanisms may intertwine in an activation flow:

Now let’s get a bit more tactical…

Setup > Aha > Habit

Before I start, I want to be clear that while the following section touches upon a popular product framework, I give full credit to Brian Balfour and the team at Reforge for helping me understand its nuances. If you like this blog and want to level up your thinking around product retention and engagement, I could not recommend this course more highly.

So how do you structure your activation flow to maximize your chances of retaining a user? The simplest way is to break your activation into three moments: Setup, Aha, and Habit.

For each one of these moments, you should have a qualitative and a quantitative definition. Here’s an example:

Use Case	End User
Habit Moment	I use this product for all my professional social sharing to stay top of mind with my networks.
Habit Moment Metric	Had four sessions with queued posts in first 28 days.
AHA Moment	I see that my social shares are generating engagement on my LinkedIn (and getting me points on the leaderboard).
AHA Moment Metric	Opened the social scorecard notification in their first week on the platform.
Setup Moment	I have used the queue to set up my social shares for the week.
Setup Moment Metric	Four queued posts in their first session.

In my experience, the most difficult part of defining each of these moments is understanding exactly what the Aha moment in your product is. In other words, how do your users articulate their view of the core value prop? What is the superpower that your product gives them?

When implementing this framework in my last company we discovered that the moment the user realized the core value proposition actually took place outside of the product. Our Aha moment existed but we didn’t control or reinforce it, and a lot of users didn’t fully make the connection between our product and the value it was creating. We needed to intentionally reinforce the core value proposition by creating an in-product Aha experience through a notification.

This ties in beautifully to Nir Eyal’s Hook Model. On the road to retention, it is crucial to think about how you can build triggers that drive users to complete your core action, which leads them to the reward of your core value prop.

So if you are following the Hook model and intentionally building a habit loop around your core value proposition, how do you know when you’ve succeeded? Ask the data.

You want to know how many times a user has to pass through the habit loop in a given time period in order to maximize their chances of becoming a long-term user.

One of the most impactful tools you can use here is the Habit Moment matrix. Full credit to Reforge for this one – here is their example using Zoom, which helps us answer the question: how many Zoom meetings does a user need to have, in what time period, for them to become a retained user?

---

---

Taking Activation To The Next Level with Machine Learning

As you might know, Georgian invests in high growth software companies. We look for companies with unique data sets that use their data in ML models to propel their business forward and drive additional customer value.

Cool, but what does that have to do with product activation?

Well, if you deploy trustworthy ML techniques during your activation process, you will be able to reduce the time it takes for a user to move from the setup through to habit. Shorter time to value -> more customers retained -> more revenue.

Sound good? Let’s dive a bit deeper by examining how Georgian’s three thesis areas of trust, applied AI, and conversational AI can all play a part in accelerating your product activation.

Trust

How you approach earning trust with a new user can make or break their experience. When you’re planning your activation path, trust should be front and center in your mind. Assume as a starting point that all your users begin as skeptics. You need to show why you can be trusted.

You can think about earning trust along two axes. On the first, you have value; on the second, comfort. You’ll want to show value as quickly as possible to deliver on your sales and marketing promises. You can think of comfort as anything that makes a user feel good while using your product – slick communication, easily completing tasks, getting insights they didn’t have before.

Measuring trust isn’t easy, but we’re fans of using the Psych Framework from Darius Contractor. This framework puts a rough estimate on the gives and gets that exist during an activation flow, and gives you a sense of the balance that you have left in the user’s “Trust Bank”.

Here’s what that might look like:

Source: Darius Contractor

While trust is important at all stages, it is particularly crucial to earn it during the Setup and Aha Moment experiences in your product.

Easier said than done. After all, trust is such a squishy term.

To make it more concrete, we often map out the user journey and brainstorm what we could be doing or saying to earn trust at each step. Maybe there’s an opportunity to make that privacy policy clearer? Maybe you could tweak your onboarding sequence to better explain your core action? If you have an ML product, don’t stop at providing a recommendation. Explain why it was given. Your users are people and they have questions—try to answer as many as you can to show that you’re thinking like they are.

A Case Study in Earning User Trust

The team at Turnitin created a product to detect the use of contract cheating in students’ essays. The product highlights essays automatically using machine learning. They knew that in such a sensitive scenario, they would have to earn the trust of their users. Their users—academic investigators—would need to know exactly why a certain essay was flagged.

That’s why the product highlights suspicious linguistic features. Seeing what’s suspicious helps the investigators quickly review essays to see whether they agree. Turnitin goes one step further too. They also provide questions for their users to ask during the investigation.

Applied AI and ML

Whether you’re a builder of software or a user, you have undoubtedly encountered the cold start problem.

You know that feeling… you find a cool new product, you create your new account, your psych score is sky-high.

And then you get into the product… and it’s blank. You have to start completely from scratch. You aren’t sure exactly how you should configure it. You don’t totally understand how all of the features fit together. All of a sudden, your psych plummets.

You give up, close your browser and move on to other things…

The chances of you building a long-term habit around that product are almost nil. They might try to get you to come back, but first impressions mean a lot.

If this sounds familiar, and painful, let’s talk about a better way.

Take that same product we just signed up for. Imagine they had given you a pre-built instance of the product tailored to your needs, and let you customize it from there. Imagine they have given you examples of best practices and recommendations for the next action to take.

The problem is that these recommendations are often driven by your data. And there’s the catch 22. You have no data so the product doesn’t sparkle, and you don’t really want to add the data to a product you don’t love.

This is where AI and ML techniques come in. One great technique we talk about at Georgian is transfer learning.

You have other customers that have configured your product really well. You can aggregate the frameworks that these customers have configured, see which configurations would work best for a new customer and then apply the exact ones that work for them. Cool, huh?

Rather than starting cold with an empty instance, you are giving your customers a massive head start. You are putting them on a rocket ship to the promised land.

For more in-depth reading on solving the cold start problem, check out this content package.

Another popular ML technique that helps accelerate time to value for new users is collaborative filtering. We won’t go into this too deeply, but you’ll recognize this technique as the brains behind Amazon’s “people who buy x also buy y” feature. Netflix uses collaborative filtering and a few other techniques within a hybrid recommender system. For a full 101 overview of recommender systems, this is a fun read.

Conversational AI

Here’s a question – do you prefer when people talk to you, or at you? Do you prefer talking to humans or robots?

These days, some of the most magical technology experiences come through robots that can converse like humans. We call this conversational AI. Think of the magic of Google Home or Amazon Alexa.

When done well, conversational experiences can act as an in-product concierge that nudges your users towards performing core actions and building habits. One of my favourite examples here is Expensify:

Expensify literally calls their conversational bot the “Expensify Concierge”, and it playfully guides you through what you need to do to be successful using the app.

The keys here are context and tone. Expensify knows what I need to do, guides me there and speaks to me in a tone that feels like a friend giving me a heads-up.

If you can integrate conversational experiences between your Aha Moment and your Habit Moment, you will continue to build trust with your users by bringing your product to life, and they will be sprinting through your habit loops in no time.

Conclusion

So how do you retain your users, and with them your customers? If I was to distill everything down, I would say:

1. Get to know your users. Interview them, gather qualitative and quantitative data, study it closely and take the time to understand your core action. Use the frameworks above to help visualize the data and drive a discussion around your retention metric.
2. Focus on the habit loop. There are a million and one feature ideas on your roadmap, but what if the best thing you could be adding next is a small tweak to your core action? Reinforce your habit loop before moving on to the next feature, or risk spreading yourself too thin and losing users before they build a habit.
3. Make the experience memorable. Once you have the habit loop nailed down, take a closer look at every step leading up to it and ask how it could be improved. You’ll never regret making a customer feel even more fulfilled while they’re getting started in your product.

As always, we would love to hear your feedback and questions. You can reach me at evan.lewis@georgian.io.