Matt Chinworth
For most leaders, the ideal high-functioning team operates with smooth collaboration guided by a clear goal that was agreed upon at the outset. Researchers studying the innovation process have also found this model to be particularly important for helping teams communicate better, coordinate tasks, and resolve conflicts as they explore diverse ideas and develop novel solutions to a problem.
Or so we thought. Our research set out to test the assumption that defining a clear problem at the beginning of an innovation project is always beneficial for innovation. We investigated when and how teams develop a clear problem over time, comparing the results of teams that started with lower versus higher levels of problem clarity at the beginning.1 Our conclusion is that there can be advantages to allowing ambiguity over team goals to linger for longer periods during the innovation process.
Defining Versus Discovering the Problem
Consider two hypothetical teams developing an innovation in their business. Team A is efficient and composed, quickly defining a clear problem and aligning around a strong shared vision. This allows team members to generate diverse ideas and converge on a joint solution with speed and efficiency. Throughout the process, their goal remains clear as they resolutely execute a plan and overcome setbacks, demonstrating persistence and resilience as they bring their vision to reality.
Meanwhile, Team B is volatile and messy, starting with only a vague sense of its goals. Team members have intense debates and disagreements as they explore various ideas and frequently pivot in new directions. But about halfway through the process, they manage to pull everything together: They evaluate diverse ideas, refine their options, and eventually converge on a shared solution that also brings greater clarity to the problem. At this point, the team can finally move forward with confidence.
Based on the long-standing assumptions described above, Team A — which defines a clear problem early in the project — should have a much higher chance of implementing its innovation in the organization. But our research revealed that the approach taken by Team B — which discovers a clear problem over time — is actually a more reliable predictor of successful idea implementation.
To study the two pathways, we collected data from 579 teams participating in an internal innovation competition at a Fortune Global 500 company. Each year, the company invited all its employees to form cross-functional teams that each develop an idea aimed at improving the customer experience. The competition, which had been held for more than 10 years, had become a reliable innovation engine for the company, according to one executive we interviewed.
The year in which we collected data, over 5,000 employees were working across more than 1,100 teams to develop an innovation. In total, 56% of teams had successfully implemented their project after seven months, leading to additional revenue and savings totaling more than $60 million.
During the competition, we measured the level of problem clarity by asking teams to complete a survey at both the beginning (at about one month or less) and midpoint (at around three to four months) of their projects, where they rated how clear and well defined their goals, outcomes, or criteria of success were at that time. We then used those measures to predict project implementation (as later verified by the company) while controlling for other important factors, such as team size, diversity, and access to resources. We chose implementation as an outcome to evaluate because it was an objective measure of success, helping us avoid biases that typically arise when a small number of managers evaluate highly diverse projects across many different lines of business at a global company. Furthermore, extensive research has shown that managers are fairly poor at predicting the impact an innovation will have, due to the inherent uncertainty surrounding such projects.2 Across many contexts, such as R&D, patent offices, scientific agencies, and even venture investing, scholars have found a consistent bias against novel and risky ideas, which also happen to be the very ideas that are more likely to achieve tail-end results. Consequently, framing innovation decisions around maximizing implementation rather than speculative impact may be a better approach to consistently finding breakthrough ideas that can substantially increase overall returns.
Our results ran contrary to the conventional wisdom. We found that when teams began with higher clarity, they had a project implementation rate of approximately 50%, or slightly below the average of all teams in the competition. However, when teams began with lower clarity and worked through a messier and more chaotic process, more than 80% of their ideas were implemented — significantly more than the overall average. (See “Teams That Discover the Problem Achieve Higher Implementation Rates.”) These results suggest that the key to increasing innovation could be supporting a process we call team problem discovery, which involves exploring and refining a problem over time through iterative learning rather than locking in assumptions too soon.
Why Messy Teams Can Succeed
Encouraging teams to embrace ambiguous problems seems counterintuitive, and it may even challenge common sense. Why would teams that started out with less-precise goals (and a consequently more chaotic process) have higher rates of idea implementation? We identified two key mechanisms to explain the results.
First, at its core, innovation comes from constructing a match between problems and solutions. When teams define a problem too early, they are essentially removing valuable degrees of freedom during the search process, limiting their options to a relatively narrow set of solutions instead of fully exploring the opportunity space. Although clear problems may help teams identify more objective performance indicators to measure the success of ideas, it does not necessarily increase their chances of fully solving the problem.
For example, one team in our study set out to fix a chronic source of customer frustration: Sixty percent of complaints involved payment disputes due to an overly complex and opaque billing process, which resulted in some customers refusing to pay altogether. The team defined this problem with high precision from day 1, placing it in the top 10% of clarity among all teams. That allowed it to assemble the right experts, map out the billing process end to end, and pilot a solution that involved manually overriding prices for major clients. Because of the project’s strong initial progress, a panel of managers selected it early on as a highly promising idea to advance to later stages of the competition. However, the project ultimately failed to get implemented due to unexpected hurdles when scaling the solution across the company, thus demonstrating the potential drawback of locking in on a problem too early.
Compare that example with the process used by another team, which aimed to address a hazardous field practice for technicians when serving customers at their homes. This project began with a more ambiguous problem — the team’s clarity rating was in the bottom 10% among teams overall — but it also allowed team members to explore a wider range of ideas without prematurely judging them as good or bad. For example, they drew inspiration from many serendipitous experiences, such as speaking with a neighbor who was a retired engineer, attending a local flea market, and watching a mountaineering documentary on TV. As they developed different prototypes and tested ideas in the field, the team eventually converged on a solution that not only improved safety, speed, and reliability but also enhanced its understanding of the problem. Unlike the other team, this one fully implemented its solution, which was eventually judged to be one of the top 10 projects in the entire competition.
The second mechanism at work emerges when an innovation team has a leader at the outset who recruits others to join a project that has the potential to make an impact. Members often look to the leader for inspiration, clarity, and vision, especially when conditions become more ambiguous. Traditional wisdom states that leaders can provide direction by defining a clear problem at the start, which translates higher-order aspirations — why a project is needed — into more tangible objectives that specifically define what the team is trying to accomplish. However, an unintended consequence is that team commitment may suffer: Even team members who are highly interested in the aspirational goal may become less engaged if they aren’t also motivated to solve the specific problem named by the leader. Team cohesion may appear to be strong when the project is performing well, but it can unravel quickly in the face of challenges and negative feedback, which are a near certainty when a team is pursuing more innovative ideas.
In contrast, consider how commitment changes for teams engaging in problem discovery over time. At first, team members may feel unsure about what they’re supposed to do. But rather than following the team leader’s clear problem definition, members can, with encouragement, share their own ideas about valuable problems to solve and interesting potential solutions. As more people share ideas, the feeling of chaos might grow, but so might the sense of opportunity and excitement. It’s important to note that this process cannot — and should not — persist for too long, since teams need to eventually stop exploration and converge onto a single idea for the project. We found that failing to do so by the halfway point can be catastrophic; the implementation rate of such teams plummeted to just 25%. This transition to idea convergence could involve intense debates, compromise, and a synthesis of perspectives, but the most important outcome is a unified idea that is more fully shared by all members because they had a hand in shaping it. If teams can succeed in discovering a clear problem over time, together, they can also benefit from a transformative experience in which working through chaos and confusion generates deeper feelings of cohesion and commitment.
How Leaders Can Cultivate Problem Discovery in Teams
The implications of our research findings are that leaders should embrace more ambiguity at the outset of innovation projects to facilitate greater problem discovery. However, applying this advice can be challenging because it requires leaders to step back from their typical mandate of providing a clear goal for the team to rally around. When following the path of team problem discovery, leaders may find it more effective to think of themselves not as a captain steering a ship toward a specific destination, but rather as the person dropping an anchor that will allow the vessel to swing with shifting currents without drifting too far in the wrong direction. And, importantly, leaders must be prepared to lift the metaphorical anchor and push the team forward when the opportunity is right. Practically speaking, teams need different kinds of leadership during different phases of the project, as we describe below.
Phase 1: Exploration. The first phase represents open-ended exploration, in which teams should share a wide range of ideas without prematurely eliminating those that don’t fit neatly into a preestablished definition of success. During this phase, the team leader’s role is to promote an open exchange of ideas, with the primary goal of learning. For example, what can the team learn about potential opportunities arising from each idea, and what can it learn about potential pitfalls or unexpected issues with different solutions? Leaders can still provide a higher-order vision at this point — setting the motivating context for why many different goals can each have value — but details on the specific outcome to accomplish them should remain open. This may make many team members — especially those with a low tolerance for ambiguity — feel uncomfortable. But leaders can reassure them that the team will eventually converge on a clear problem and solution, which will be more effective once everyone has shared and interrogated all of their ideas.
Phase 2: Convergence. Although open-ended exploration is essential to discovering a wide range of valuable opportunities to pursue, it is effective only when followed up with a deliberate effort to reflect on what was learned and to converge on a single idea that can be developed for the rest of the project. Extensive research has found that the optimal time to facilitate this transition is around the 50% completion point of a project (plus or minus 10%).3 That gives teams sufficient time to both thoroughly explore various ideas and transition to fully executing a plan. To begin the convergence process, leaders can intentionally create space for the team to pause, reflect, and discuss insights uncovered during the exploration phase. When pushing toward convergence, debates among team members can get intense, especially when they are passionate about different ideas. Therefore, one valuable skill for leaders to practice is to identify creative ways to integrate, synthesize, or combine different ideas. This not only aggregates the best elements of various ideas to improve the overall quality of a project but also allows more members to see a piece of their vision embodied in the final outcome — which can enhance overall team cohesion and commitment.
Phase 3: Execution. Once teams have identified a single shared idea to focus on, they must transition to developing and executing a plan to bring the idea to fruition and implement it in the relevant context. At this point, the initial ambiguity during exploration has been fully resolved, so leaders can transition to providing stronger direction, delegation, and coordination that is reflective of a more traditional style. Throughout the entire problem discovery process, it’s important for leaders to dynamically shift their leadership styles and behaviors to support either divergent exploration or convergent execution, as required. Research has shown that combining leadership styles or engaging in a moderate level of support for both divergent exploration and convergent execution throughout a project can be detrimental to performance because it promotes conflicting dynamics within teams.4
Our research highlights the value of messy teams for innovation but also suggests that there can be a hidden method to the madness. While some innovation contexts should follow the traditional approach, where problem clarity naturally exists and teams can easily work from a shared goal at the start, we believe that our research illuminates the merits of a different approach that can substantially increase the overall rate of innovation among teams.
If team leaders can develop new skills that smoothly guide teams toward discovering problems over time rather than just defining problems up front, we believe they can significantly increase their chances of developing and implementing successful innovations in their business. And because innovation is inherently uncertain, with the number of failures far exceeding the number of hits, even small increases in success rates over time can, in the aggregate, have a major positive impact on an organization’s long-term innovation goals.