We’ve been involved in product innovation, design and development for decades. Both on the client side and as consultants. We’ve seen first hand many of the victories enjoyed and the struggles endured by companies engaged in innovation regardless of size, industry and culture. It’s clear to us that the risk inherent in launching new products hasn’t gone away. Or even been greatly diminished. What has changed is the source of that risk and the way companies have chosen to contend with it over time.

In several studies conducted by the Product Development and Management Association over the years, the failure rate of new products has remained remarkably constant from decade to decade.

1980s: 42% failure rate
1990’s: 40% failure rate
2000’s: 46% failure rate

A failed product is defined as one that doesn’t meet the objectives management established for it over a specified period and is withdrawn from the market. As you can see, not a promising trend. In fact, these are incredible statistics given all the strides we think we’ve made as an innovation community. Of course, different industries show different failure rates. Unfortunately for most readers here, consumer products and services show higher failure rates than commercial products. An interesting distinction that perhaps has something to do with the challenge of properly reading evolving consumer wants and expectations. As well as quick shifting tastes and trends that require rapid response—a difficult task for many organizations. The good news is that the often cited “80% failure rate” statistic appears to be a myth perpetuated by those who may stand to benefit from such a claim.

Why, after all our advancement in methodology and process, do so many new products continue to fail? It’s reasonable to say that the standardization we impose on other business functions just doesn’t work for innovation. That which is mission-critical for Accounting and Production may not play well in the art-and-science realm of innovation. That may be part of the answer, but far from the entire story. Let’s take a look back at innovation practices through the (recent) ages and see what we can conclude about why we seem to have hit a wall.

The 1980’s: Companies didn’t really know what innovation was, though many were doing it instinctively. 3M, GE and others were starting to measure the business impact of new products and considering “new product development” a survival tactic. It was popular to say that “X% of our revenue comes from products less than Y years old” as if that was, by definition, a good thing. Yet there was no clear, standardized approach to innovation. Companies were learning as they were doing. In this pre-methodology era, those few dedicated product development staffers were likely deported from mainstream, revenue generating business roles. And not always as a reward for strong performance.

Back then, new products were more likely to come out of R&D and then passed along to Marketing for packaging and communications. Companies would discover a new utility and then look for ways to exploit it in the marketplace. If there were people dedicated to product development, they were likely close to the technical side, challenged to find a customer for some new skunkworks invention. There wasn’t much in the way of discipline. But there was a bunch of ad-hoc creativity. And lots of dabbling.

It’s interesting that despite the risk inherent in this rather informal technology-driven innovation, the success rate was not abysmal. It could be theorized that risk in this environment almost took care of itself, as many of the skunkworks innovations didn’t prove out in the lab and never made it to market. And those that did were exposed to such a rigorous vetting process (and were so novel) that they likely stood a better than even chance of success. Perhaps the “hammer looking for a nail” approach is unfairly maligned!

Even so, it became clear that the practice of new product development needed some structure. Companies were fond of systematizing work for efficiency and to eliminate error across functions. Imposing methodology was thought to be the answer. And “innovation” as a discipline was born.

The 1990’s: Process discipline was all the rage. Standardized practices. Stage Gate methodologies and Six Sigma. Companies looked to import and install commoditized innovation methods in an attempt to minimize risk—considered the enemy of product development at the time. Organizations can’t survive without structure and process. So we asked the obvious question: Why can’t we automate innovation down to a zero error rate?

Well, we learned a few things in the 90’s: First, innovation is a blend of art, judgement and science. While we talked ourselves into believing that these ingredients are ripe for systemization, our efforts to do so underestimated the resistance natural creativity has toward any attempt to instill order. And while imposing rigorous process on innovation made everyone feel safer, it didn’t do much to ameliorate risk or improve success rates, as we know. But we did get a lot better at stopping things that appeared unpromising. But it’s not clear we were necessarily green-lighting the right initiatives.

Second, we learned that a year’s worth of standardization was in fact putting a choke hold on innovation. Production assets were designed decades earlier to do one thing really well: Fill this specific bottle with that specific goop. So now we have this early focus on innovation, people to staff it, methodology to guide it but not enough latitude to deliver on it. Frustrating.

Process, in conjunction with a lack of asset flexibility (ironic since process inherently limits flexibility as well) resulted in a desolate landscape for innovation. Great ideas were contorted to run on existing lines. And their salience suffered, victims of what we might call “Adaptation Risk”. Success rates stagnated. Methodology got the blame, but it was no one’s fault. The machines chunking out soap and cheese were paid off long ago. And no one wanted to pay to replace them. The tension was building.

The 2000’s: Design consultants save the day. Sort of. Companies decided they need outside help in cracking the innovation code. Design firms proliferated to address the mounting frustration internal methodology has left behind with copious amounts of creative energy. Some firms forwarded their own methodologies for those clients late to the discipline table and in need of that security blanket. Others promised raging creativity that internal staffers presumably couldn’t muster. And yet others dug deep into client production facilities looking for ways to squeeze a drop of flexibility out of those machines built to do one thing really well, really fast.

This was, in an odd way, the golden age for innovation. Many companies recognized that process discipline wasn’t the solution it promised to be. And we learned that risk could be a good thing. The power of creativity was winning favor on both the marketing and manufacturing sides. Those in innovation roles began to earn respect and were chosen for a set of personal and professional attributes that seemed to align with the welcomed uncertainty of innovation.

Management started—we say started—to become comfortable with risk and the notion that not all new products succeed. And that failures provided learning to feed later success. In fact, there was a certain misplaced pride in failure until management realized there was only so much failure an organization could enjoy without an occasional knock-out success to pay for it.

This, we believe, was the start of “Design Thinking” as we know it today. When companies can recognize failure as part of an iterative prototyping effort and can value the kind of user empathy that internal market research wasn’t providing at the time, we knew we were making real progress. Now, if companies could only learn to iterate and fail before launch…

The 2010’s: The smart money knows it’s not about “product” anymore. It’s about the story. The experience. The BRAND. Gone are the days of utmost concern about the new functional benefits consumers demand from products. It’s not enough to leapfrog competition with features we think we know consumers want. Consumer wants have been replaced by Need States: The emotional underpinnings that drive choice. People out there are too busy to discern why one product may be better than another without taking the time to experience it. So benefit-driven messaging no longer wins hearts and minds. And in most categories, those hearts and minds don’t care enough.

What they do care about is affiliation. Experience. Values. The absence of frustration. What does that mean for innovation? Tons. Now, innovation is about making a promise to consumers that resonates. That reflects who they think they are and what they want to say about themselves. And doing a job they care to get done with confidence and style. So, it’s the holistic offering that will win the day. Not just a new package. Not a new flavor. But the integration of everything we used to think of independently. Together. It’s about The Story.

As a consumer product, how does aesthetics, formulation and utility come together in integrated fashion to deliver a coherent brand promise? As a service, how must all the brand touch points behave to achieve performance and satisfaction? Key questions: What’s the difference between product and service in this new order? Answer: Not much.

How does a brand set parameters, provide latitude and profess values to guide new product and service development? Or better yet, become product development. It’s not about the “thing” anymore. It’s the experience. Sure, technology plays a facilitating and exciting new role. IoT for instance enables change and utility like nothing we’ve seen before, with the potential to upgrade convenience and lifestyle. But it’s a “how”, not a “what”. The “what” remains that need state fulfillment grounded in a satisfying experience, however delivered.

Why is this tectonic shift in the definition of “new product” so important to acknowledge? While this turn of events tamps down certain kinds of risk in innovation, it elevates others. And for companies, innovation has always been about risk. Now, there are more kinds. Where today’s innovation can ride on the strength of a brand and diversify across multiple touch points, its chances of success can be greater. And its costs can be lower. But—and this is a big one—in this day and age, a brand innovation story is subject to incredible scrutiny (and brand abuse). Much more so than a new feature set or package form.

Think about Dove’s recent stab at bringing the brand to life in an array of body-shaped bottles. In hindsight, it was easy to see how this would go wrong. A brand’s values are just that—values. They cannot be literally translated into a package form or any other physical manifestation. They tell a story—they don’t dictate product. Why dwell on poor Dove? Because in an attempt to define a brand message in the merging of new and old innovation paradigms (story meets design), they achieved nothing. That, coupled with a few thousand haters with the hand-held technology to make their feelings known can upend innovation like nothing has before it.

However, in an age where Tweets, stories, and pins are exchanged with lightning fast speed, companies can find a logical application for social media in the ideation and crowdsourcing phase of product innovation. There’s a definite draw for brands to use social media to replace some kinds of market research. However, brands struggle to find the right strategy.

Oftentimes, data from samples on social media can lead brands astray. There’s just too much of it and a lot of noise that comes with it. So, they’re relegated to “testing the waters” rather than to dive in completely. Why? Social media is an intrinsically consumer-focused and controlled channel. It gives consumers the freedom to express their feelings instantaneously, creating an emotionally charged environment where users are quick to react to experiences that either exceed or fall short of their expectations.

And, there lies the problem. Sharing and posting on social media is driven by emotional extremes whether negative or positive, which can make social data a bit unreliable. And it’s often the unqualified negatives that can torpedo a new product on emotional grounds before it gains momentum. One could call this “Hater Risk”. It’s important for companies to track not only the quality of the social data, but also the qualifications of the participants who provide that data. Identifying subject matter experts leads to the best expertise and increases the quality of insight for innovation. It’s all about listening to the right kind of data from the right kind of users while tuning out distractions, which is easier said than done.

While the risk to innovating has changed, it hasn’t diminished. As we pointed out, the proportion of successful new product launches has remained roughly constant since the early days. Why is that? We think it’s because:

1. We still face rigid actionability parameters in many companies. So we focus attention on the near-term little wins at the expense of category transformation. And the best ideas are often truncated to fit the parameters allowed by existing assets. One would think those little wins have a better chance to succeed. But we’d guess that incremental change does more to mess with a winning formula than it does to improve it.
2. Brand extension mania seems like a risk averse strategy for innovation. We would allege that such efforts can dilute a brand’s core attributes, perhaps weakening it’s pull for consumers. So not only might brand extensions fail of their own accord, they could bring the brand down a peg with them.
3. Cross-functional teams are great because they get things done. But they also water down great ideas. Let’s face it: If you’re representing R&D on a project team you may still be incented to push back against anything that could throw a wrench into a finely tuned production machine. And who could blame you with new product success rates as they are.
4. As a corollary to the above, organizational complexity has created project complexity. Core teams are huge and sell-in audiences are vast. We firmly believe that minds that work alone or in very small groups have the best chance of surfacing the truly transformational. That’s because mandating consensus is the surest way to dumb down a fabulous idea.
5. As suggested, consumer tastes and trends are moving so fast. Reading consumers properly and translating that insight into winning product on the store shelf has become more difficult. The science of consumer research and product development has had a difficult time keeping up.
6. Innovation as the melding of art, judgment and science continues to—and will always—defy standardization. Period.

We would like to think that process discipline, creative energy and risk is achieving a new, healthier balance. And that may finally reduce those intractable failure rate numbers. Perhaps the failure rate isn’t even that meaningful a measure. If you have one wild blow-away success in ten, you can absorb a lot of failure. Hollywood as learned that lesson, as they make bigger bets on fewer potential blockbusters. But there’s much to unpack in that change-of-paradigm for corporate America. In the meantime, here are a few tips for how to think about managing risk in your innovation projects, day-to-day:

1. Leverage consumers better. Don’t “order take”. Put them in contexts that allow them to do what they do best. And use what you see and hear as inspiration—not direction—for concept development and refinement. Talk to us about our Co-Magination approach. It’s a collaborative tool set that’s fast and flexible enough to avoid the rigid process trap and the pitfalls of taste-and-trend whimsy.
2. Don’t let process discipline choke off creativity. Reverse course. Deviate. Meander. Just don’t get lost. Remember: You manage the process—not vice-versa.
3. Use the brilliance of the single creative mind to your advantage. And then incubate it in close quarters until it has matured before letting it loose on the organization.
4. Don’t skip the design strategy phase where you assimilate defensible consumer insights (not evident observations) and chart specific innovation platforms. Use tools that focus creative energy, build team consensus when necessary and crisply communicate a path forward.
5. Be careful when concept screening. Many breakthrough ideas fall to the ground in the earliest stages because as initially presented, they don’t appear actionable or have too many moving parts and confuse consumers. In other words, rethink the “concept board”.
6. Iteration is your friend. Test and refine, test and refine, then test and refine. See #1 above.

So how can companies change the consistent flatlining of product success that’s plagued innovation for decades? There are two ways to go. Option one: Winners have to be bigger. Which means risks taken must be greater. Therefore, companies must be open to the heightened internal and external scrutiny that comes with misjudgment—fair or not. Doesn’t sound great.

Option two: Decades of circumstantial evidence points to an inability to read consumers as the largest thorn in the side of innovation. Sure, new product development has got to get faster. From ideation to production, there’s no time to waste. But that’s not enough. Tastes and trends come and go. But our new need state order remains fundamentally the same.

So focus on surfacing those underlying emotional triggers that may evolve slowly but don’t dramatically shift. Then, craft a story that’s much bigger than the singular dimension of new product success. And bring that story to life in redefined brand-relevant experiences that provide meaningful context (and content) for products and services rather than letting them hang out there all alone.

Can we break trend and revive innovation success starting in 2018? Certainly. We think the best way to do that is to rethink the role of consumers in your projects and the tools you use to engage with them. If we’ve learned anything over the past 30 years it’s that traditional research and creativity methods can reduce a project to a hit-or-miss proposition. This year, why not better your odds.