Measuring Innovation: A Four-Stage Framework
A simple framework can help leaders avoid the common mistakes made by corporate innovators—and accelerate the discovery of their next growth engine.
The digital era demands innovation. Every business today—indeed, every organization—must continually adapt to keep up with changing customer expectations, new digital competitors, and business models made possible by emerging technologies.
For an established business, this means innovation within your core business—via improvements to your customer experience, as well as back-of-house improvements that address operations, risk, and productivity. And it also means innovation beyond your core—testing new offerings in pursuit of new customers and new revenue models.
Yet it is not enough to blindly commit resources to innovation and the pursuit of new ideas. Leaders must ensure that every innovation adds meaningful financial value to the firm, along with value to the customer.
It is therefore essential to measure innovations at every stage to determine that they are on track to create real value.
A version of this article was first published in Rotman Management as “Measuring Digital Innovation: A Four-Stage Framework”
The Challenge: Measuring Innovation
Measuring innovation is a perennial challenge. New ventures cannot be managed by the same KPIs that you use in your core business. What’s more, the metrics needed to grow any new venture will rapidly change.
Because of your long experience with your core business, its day-to-day operations can be managed with traditional planning. But the inherent uncertainty of any new digital innovation requires that you manage it through a process of experimentation.
This process should draw on seven principles shared by agile, design thinking, lean start-up, and product management:
1. Start from the problem to be solved, not the solution.
2. Don’t write a business plan. Instead, write business hypotheses.
3. Focus on learning and testing those hypotheses.
4. Build new things quickly, cheaply, and iteratively.
5. Get feedback at each step—directly from the customer.
6. Be ready to pivot or shut down quickly if the data do not validate your hypotheses.
7. Be ready to scale up if the data indicate that you are on the right track.
I define experimentation as ‘any process of iteratively learning what does and does not work.’
Metrics are essential to this kind of iterative learning. So, what data will guide you on your innovation journey?
Based on my own years of research, teaching executives, and coaching innovation teams, I have developed a framework for measuring innovation.
I call it The Four Stages of Validation.
As indicated in Figure One, each stage is designed to answer an essential question about the feasibility of any new innovation.
All four stages are essential. Only when you can answer each of their questions will you learn whether your innovation is repeatable, scalable, and profitable in the real world.
But you cannot measure everything—or answer every question—at once. Instead, you must thoughtfully progress through the stages, adjusting your metrics as you go.
Let’s take a look at each of the four stages.
STAGE 1: Problem Validation
Are you focused on a genuine problem for an actual customer?
The aim here is to validate the problem you aim to solve, the customer who has that problem, and the urgency of the problem from their point of view.
Note that your innovation’s ‘customer’ may be a consumer, a business customer, a partner, or someone inside your company. Some will prefer the term user instead of customer.
Problem validation is done primarily through in-depth conversations with customers, conducted in the work or life context where their problem occurs. These interviews should focus solely on exploring the problem from the customer’s perspective. Avoid any mention of the solution you think you will build.
Your goal is to probe the customer’s unmet needs, the different segments of customers who share this problem, and the ‘existing alternatives’ they currently use to cope with it.
With luck, you will encounter some early adopters—customers eager to test any future solutions you come up with, even in very rough form.
Your learning at this stage is mostly qualitative. Your quantitative metrics are simple: the number and depth of interviews you conduct, the number of early adopters identified, and the projected size of the total addressable market (TAM).
The biggest red flags at this stage are if there is no agreement on the problem, if few customers share the problem, or if they express little urgency to solve it.
Many digital innovations run into these issues, discovering that they are building ‘a solution in search of a problem.’ This is a clear signal to shut down and redirect your resources elsewhere.
But if you are able to validate a real problem faced by many customers who urgently want a solution, you are ready for the next stage.
STAGE 2: Solution Validation
Does the customer see value in our proposed solution?
Here, your goal is to validate whether the customer perceives value in the solution you are planning to build, the level of customer demand, and if your solution will stand out from other competitors in the market. Solution validation is all about what entrepreneurs call ‘product-market fit.’
This stage does not make use of finished products, or even working prototypes. Instead, it uses simple artifacts that merely illustrate the expected design and features of your solution, to measure customer feedback. I call these simple artifacts ‘illustrative MVPs.’ They may be as simple as a website landing page or as elaborate as a digital twin.
At this stage, your learning is a mix of qualitative insights (e.g. what is your perceived value proposition?) and quantitative measures of customer behavior.
Metrics often include things like click-through rate or landing page conversions from messages sent to prospective customers. In other cases, metrics may include signups for product trial, or purchase orders placed before delivery of your first working product.
Your biggest red flag at this stage is if customers who see your MVPs offer polite praise, but you receive no urgent requests to use or buy what you are building.
By contrast, a clear measure of customer demand will show that you have reached product-market fit, with customers ready to adopt your solution as soon as you can deliver its promised benefits.
STAGE 3: Product Validation
Can we deliver a solution that customers use?
Here, your goal is to measure two critical things. The first is usage. When you present the customer with a working solution, do they, in fact, adopt it? If so, when, where, and how do they use it? How frequently?
The second thing to measure is delivery. Can you deliver the experience and benefits that you intend for the customer? What are the technical requirements, and can you meet them? What about legal, regulatory, and other risk or compliance requirements? Lastly, what combination of skills, assets, data, and IP are required to deliver this innovation in the real world? Does your organization have any unique advantages that allow you to deliver this innovation better (faster, cheaper, or with more user benefit) than competitors?
To measure Product Validation, you will need to build your first working solutions, which I call ‘functional MVPs.’ These solutions should start as ‘skinny’ as possible—with the absolute minimum features needed to deliver the most basic value proposition to the customer.
They should be built quickly and cheaply with off-the-shelf components or manual service that could not scale beyond a handful of customers. As you test and learn, you will iterate your solution, adding more features and shifting to more robust means of delivering them.
During this stage, many of your metrics will measure customer behavior (such as adoption of your innovation, frequency of use, customer satisfaction, and retention). Other metrics will focus on delivery, measuring your operational speed, accuracy, throughput, and downtime.
Red flags at this stage are when customers who asked for your solution fail to use it, or they try it once and never return. Similarly, your innovation is in trouble if you find you cannot deliver your solution reliably, or competitors can deliver an identical solution just as well as you.
But if the data shows you can deliver a solution that customers readily adopt, and where you have an advantage over competitors, you are ready for the next stage.
STAGE 4: Business Validation
Can we capture sufficient value from this innovation?
Here, your goal is to measure whether your innovation generates enough financial benefit to be worth scaling up beyond your initial limited tests.
Business validation starts with measuring any revenue your innovation may generate, as well as any cost savings, improved efficiency, or reduced risks. These should all be measured in financial terms.
Business validation also means measuring the costs to deliver your innovation—both fixed costs and marginal costs. And it means combining financial costs and benefits to determine your potential profit margin and your maximum possible upside.
At this stage, you will use a variety of metrics to create a financial model of your innovation. If the innovation captures value from customers, your metrics will include customer lifetime value and other metrics that depend on your revenue model (subscription, purchase, licensing, etc.) Internal innovations will use metrics to measure cost savings, efficiency gains, or reduced financial risks.
Once you have a projected return on investment (ROI) for your innovation, you must still compare that with your firm’s internal rate of return (IRR), to know if your money would be better spent elsewhere.
Red flags at this stage include customer unwillingness to pay, costs too high to reach a profit as you scale, or a maximum upside that is too small to merit further investment.
By contrast, a clear path to profitability and an attractive maximum upside will indicate it is time to commit to scaling up your innovation.
Measurement Across All Four Stages
For any innovation to succeed, it must prove itself at all four stages of validation. This requires that your innovation team maintain a clear focus on the most important metrics and qualitative learning in all four stages.
Figure Two recaps the business model hypotheses that must be validated in each of the four stages and lists examples of quantitative metrics that you may use.
Applying the Framework
Following are three principles to keep in mind when using this framework.
1. Metrics are for making decisions.
Innovation metrics are only useful if they are applied to make data-driven decisions. For an innovation team and its sponsors, each stage of validation offers critical data.
At each stage, teams and sponsors must decide: Does our latest data indicate that we are on track? Should we keep going, continuing to iterate and improve our solution based on what we’ve learned?
Or does the data tell us that something is fundamentally wrong? That we need to either ‘pivot’ (rethink the problem we are solving, the customer it is for or the solution to that problem) or quickly shut down the current effort, to focus resources on another innovation?
Figure Three summarizes the signals to look for in your metrics at each of the four stages of validation.
2. The sequence of innovation matters.
Successful innovation requires knowing both what to measure, but just as important, when to measure it.
Many companies recognize the importance of experimentation, but they stumble in getting the sequence of measurement right. Specifically, they rush to start validating the later stages.
In finance-driven firms, I often see a rush to start with Stage 4: ‘Show me the business case first, with market size and anticipated profit margin. Then we can approve a budget to spend on market testing or validation.’
In engineering-driven firms, I see the urge to start with Stage 3: ‘We know the solution. Let’s build a working prototype as a proof of concept to see if it can be done. You can show it to customers for feedback—just as soon as we build a complete product for them to try.’
And in marketing-driven firms, I see a greater focus on the customer and an inclination to start with Stage 2: ‘We have a great idea that came from our market research and brainstorming. Let’s mock up some wire frames to quickly get customer feedback.’
All of these approaches are mistaken.
Each starts the process of experimentation and measurement in the wrong place.
By contrast, every effective innovation process I have ever seen begins with Stage 1—validating the problem you hope to solve with the customer you think it will matter to—and then moves sequentially through the stages.
Of course, it is correct that your venture will need to answer financial questions such as the cost-to-serve and the profit margin. But you can’t start there.
Time after time, I see companies laboring to write business cases for a new digital innovation when they don’t yet know what problem they are solving, what solution the customer wants, or how they might deliver it legally and reliably.
These misguided business cases are just spreadsheets filled with fictional numbers that add up to a meaningless estimate of future ROI.
3. The four stages overlap.
The Four Stages Model is not a waterfall. It is not a stage-gate process where success means you conclude one stage and move on to the next.
Instead, each stage is never finished.
Even as you test the finances of your business (Stage 4) and move to a public launch of your product, you will continue to learn and revise your understanding of the problem you are solving (Stage 1), the features you need to add next to your solution (Stage 2), and how you can deliver them for different customer use cases (Stage 3).
Eventually, you will be validating all four stages simultaneously.
In Closing
The digital era demands that every business innovate to keep up with changing market needs, threats, and opportunities.
And innovation that grows your business will only happen with effective innovation measurement.
With the simple, adaptable framework described herein, any leader can avoid the most common mistakes made by corporate innovators and accelerate discovery of their next innovation—one that truly works.
How Can We Help Your Business?
Please let me know if the David Rogers Group can help with the growth strategy for your business.
We help companies around the globe to transform and unlock high-value growth. We advise senior leaders at organizations like Google, Microsoft, Citibank, Mastercard, Unilever, Procter & Gamble, Merck, Toyota, GE, Cartier, Pernod Ricard, NTT Docomo, Viettel, NC Bank Saudi, and the government of the United Arab Emirates.
To find out more, write to services@davidrogers.digital
NEW BOOK:
“THE DIGITAL TRANSFORMATION ROADMAP: Rebuild Your Organization for Continuous Change”
ORDER NOW:
Hardcover: https://amzn.to/41U85dl
Kindle: https://amzn.to/3OWD437
Audiobook: https://bit.ly/DXR-Audiobook
Bulk orders up to 60% off: https://bit.ly/DXR-bulk-orders
Great insights!