The problem most product managers face when they make changes in their product is measuring the impact of those changes.
Questions that come to their mind:
- How will it improve the customer experience?
- Will this change have an impact on sales?
- Will it improve the conversion rate?
- Will it solve the ultimate problem?
- Is there something else that I haven’t looked at?
Nana Chiang conducted a meetup with HelloMeets, where she explained a framework that can guide product managers in calculating a reliable score of product change. This blog is a cheat sheet for the same.
Dilemmas and Frameworks
We will talk about sizing the impact of change. Why do managers use impact sizing and how they use it.
To begin with, here are some dilemmas managers go through before implementing a change:
- Prioritization of problems
- Area of high impact in a quarter
- Rate of conversion when implementing a product change
This framework will help managers with
- Choosing the most important problem to solve.
- Evaluating the most impactful solution
- Estimate potential business value
The better we are assessing the impact, the better decision we can make.
Prioritization
A product manager’s major dilemma is what to prioritise and what not to prioritise.
This simple chart below can help rank the problems and answer the following questions:
-Will the changes made to the product bring high impact or low?
-Are these changes expected from the customers?
-Or is there an element of surprise?
-Separating the changes and problems in these sections will make prioritization easy.
A Common Framework that is used
Consider these four factors to measure the impact of change:
- Reach
- Impact
- Confidence
- Effort
Putting these in an equation form gives a RICE score.
- Reach can be found by analysing current data
- Effort can be estimated by calculating how much time and resources are spent.
But Impact and Confidence are tough to estimate.
The impact of a product change can only be guessed. Different managers use different ways of measuring this. Some use customer feedback or collect teammates' opinions. These opinions are not reliable because they cannot answer why the change will make an impact.
Every result comes with a reason. To understand the impact, we need to first understand WHY it’s making an impact.
This framework is considered to be less reliable. Hence Nana along with her data analyst came up with a more reliable framework.
Nana’s Framework
It calculates the impact and gives a much reliable change score.
A few things to keep in mind are:
- this framework is the best guess, not a definite solution
- The formulas used will give relative numbers rather than absolute numbers
- It is a product prioritisation framework, it does not give a financial estimation
- It is focused on user-facing products but can be tweaked for other products too
To explain this framework, we will start with a sample problem:
- You’re a product manager for an E-commerce platform, your ultimate goal is to improve conversion and increase the number of purchases.
- You found that a lot of customers are not able to find the right product because they don’t know what’s the best keyword to use.
- You came up with an idea to add keyword suggestion features to solve this problem.
Now our goal is to calculate the impact of the changes made on the website.
To do this we will go through the following stages:
Stage 1- Understand the ‘cause path’
In the first stage, we have to define what we are changing and what will be the consequences. The ‘cause path’ is a hypothesis, where we identify the problem and assume the product change will have an impact on user behaviour.
In this case:
- Problem= consumers can’t find the right product
- Idea= Keyword suggestion
- User behaviour 1= user clicks on the suggested keywords
- User behaviour 2= find the right product
- Outcome= Purchase
Stage 2- Translate the hypothesis into metrics
The next step, we will convert the assumptions into metrics.
Take a look at this image below.
- User click suggested keyword= % of users who have clicked suggested keyword
- Find the right product= % of users from search result to product page
- Purchase= % of users who made a purchase
These metrics are defined as:
- Actionable- it is easy to observe and calculate through the user data collected from the website
- Input Metric- observes the change in user behaviour (users searching the pro
- Output Metric- users who purchase the product
Stage 3- Calculating the problem score
A problem score helps to prioritise the user problem.
By multiplying the seriousness score and percentage of impacted users, this formula calculates the problem score.
Seriousness Score × % of impacted users= Problem Score
- Seriousness score ranges from 1-4. Where 1 is completely blocking whereas 4 is not a problem
- Percentage of impacted users is calculated by dividing the number of impacted users upon active users.
Stage 4- Make impact assumption- solution score (leading vs lagging metrics)
Making assumptions on the basis of the hypotheses is difficult. Product Managers make assumptions on the most leading metric.
What is a leading metric?
A leading metric is easily impacted, observed and involve fewer factors that can be calculated. It is used to measure the direct impact of change. Example- % of users click suggested keywords
What is a lagging metric?
A lagging metric is that that won't be directly impacted because it involves multiple factors, such as the retention of users and transaction. It is used to measure the outcomes of change.
The solution score for the sample problem is assumed to be 5
Leading metrics can give direction to predict lagging metrics. This is explained in stage 5.
Stage 5- Understand the relationship between metrics
The relationship between leading and lagging metrics can be negatively or positively correlated.
- When more users click on suggested keywords, then we can assume the checkout rate has increased. This shows a positive correlation between the two metrics.
- On the contrary, a negative correlation is seen, if the error rate in search results is high. It is likely that there will be fewer transactions going through.
Assuming the correlation between leading metrics=0.6 and lagging metrics=0.8
Stage 6- Multiply the scores
Now all that is left is to gather all our digits from every stage and multiply.
In the above equation:
Solution Score= Assumption of how impactful is the solution to the most leading metric
Problem Score= Seriousness of the problem ✖ % of impacted users
Correlation with end goal= How do the metrics link to each other
Metrics are just tools.
Understanding what is causing the impact is the key to have more accurate estimations & better product decisions.
Pro Tip- Create a Prioritisation guide by transferring each score and solution to a spreadsheet to make your work easy.
Summary
This framework gives you a clear picture of the process of product change. It helps you prioritise the most impactful item. And lastly, combining data helps in making better assumptions.
Hope this framework helps you in prioritising and calculating the impact of your product change.
Check out our upcoming online meetups on product management and related topics here.
You might also like to read "How to use Quora as a marketing channel to generate revenue"
