Deep Dive: Innovation Orchestration - utilising NotebookLM
What value does an innovation orchestrator create, and what are the future aspects we need to address to increase their effectiveness?
This post aims to pick up from last week's article on innovation orchestration, and broaden to discuss:
- The challenges orchestrators may face and responses we can use to address these
- The value orchestrators create and a basic impact framework
Last week's post on (regional) innovation orchestration was trying to get to grips with the vast array of capabilities and functions needed, and articulate the different types of orchestrators I was seeing in the world. Whilst was a sensemaking piece for myself, but I've had a lot of feedback that it was useful, so I've decided to do a slightly different follow up than I would normally do utilising NotebookLM as my sidekick.
Process
To give you some sense of how I've produced this post, here's a run down:
- Identified 26 sources from across journal publications and online articles (sometimes known as grey literature), focused on a range of interconnected topics - orchestration, intermediaries, backbones organisations, and more.
- Load sources into NotebookLM to build a knowledge repo focused on innovation orchestration
- Discuss back and forth with NotebookLM to identify some themes, summarise ideas, and generate things like mind maps and an audio summary.
- Personal editing to ensure alignment and coherence.
The Innovation Orchestration - Podcast Episode
"Navigating that messy reality - those competing pressures and perspectives - can actually forge more robust, more creative, and ultimately more realistic solutions than working in some kind of neat, controlled artificial environment"
One of my favourite features of NotebookLM is the ability to generate a conversational audio summary - a bit like a science podcast - breaking the ideas down into fairly plain language. The above audio is AI generated voices discussing the takeaways from those 26 articles. Wild. Would love to know your thoughts!
(and yes, US was the only voice type available)
Challenges for Orchestration
Whilst there are traditional challenges of orchestration, it might be worth looking quickly at some of the commons ones:
- Lack of clear definitions and understanding of the role of innovation orchestrators
- Articulating and demonstrating impact of orchestrators
- Going beyond the lure of innovation silver bullets, orienting for portfolios
- Stakeholder and member engagement and participation over time
- Aligning speed and interests of different sectors
I'm trying to address the top two with the previous post and this one. So with the last post focusing on Types and Functions, the audio above giving a layperson take, let's turn to the impact and value creation piece.
NotebookLM also produced this somewhat fascinating mindmap, which demonstrates some of the elements of Challenges and Future work:
Value & Impact of Innovation Orchestrators
Innovation orchestrators, through their functions as "bridgers" and "brokers" and their ability to facilitate knowledge transfer, act as crucial mechanisms for managing the risk of innovation theatre and failure.
By reducing the cost of developing relationships and fostering crucial learning dynamics across the system, they reduce the uncertainty and unpredictability that often hinder innovation ecosystem approaches.
This means a significant, though often unquantified, aspect of their value is the reduction of systemic risk and the acceleration of learning cycles across the ecosystem. The effectiveness of an orchestrator can therefore be partly assessed by observing not only successful outcomes but also the speed at which unproductive projects are identified and discontinued, thereby optimizing resource allocation and minimizing losses.
Innovation orchestrators create multiple forms of value, working across a range of timescales, so are often best understood through a theory of change / program logic, which might look something like this:
Impact Evaluation Framework - Potential Indicators
I'm not going to go deep into Outputs, Outcomes & Impact, or contribution vs attribution right now - other people have written about this. Suffice to say I'm leaning into attributive impact indicators here, with the use of proxies where it makes sense.
Instead, let's take a really pragmatic look at a generic impact framework with associated measures.
Why? Well, we know that each innovation orchestrator will operate differently, with different goals and expectations. However, there is often a pattern of impact frameworks strating from scratch each time, and a lack of common language - so think of this as a menu for the types of measures which could be a starting point from which to build, adapt or reinvent.
The following is based on the above Theory of Change, and looks at indicators for both Outputs and Outcomes.
Output Areas & Indicators
1. Increase innovation ecosystem participation and sustainability
This output focuses on increasing participation and ensuring the sustainability of the innovation ecosystem including a focus on breadth and depth of engagement.
Output Indicators:
- Active engagement - as evidenced by new ecosystems connections made through the likes of events, communications and other engagement activities
- Connectedness and healthy dynamics - as evidenced by self reported trust, connectedness, reciprocity and collaborative intent
- Participant diversity - as evidenced by registration information and periodic surveys of sectoral, geographic and social diversity
- Active participation, co-creation and collaboration in ecosystem - as evidenced by repeat engagement in facilitated innovation events and processes, and the development of experiments, initiatives & projects connected to core portfolio and broader ecosystem
- Sustainable ecosystem resourcing - as evidenced by funding for core orchestration activities and for ecosystem experiments and initiatives. Additionally measures of in kind leverage - across 6 Capitals.
2. Improve innovation outcomes across social, environmental, and economic dimensions
This output focuses on tangible improvements and positive changes achieved across social wellbeing, environmental sustainability, and economic prosperity as a direct result of orchestrated innovation efforts.
Output Indicators:
- Innovation adoption - as evidenced by direct uptake of experiments, initiatives and projects
- Innovation capability & capacity - as evidenced by self reported innovation competence and confidence (individual) and/or collective sentiment analysis
- Ripples of change - as evidenced by reporting on signals of change (potentially indirectly) connected to experiments, initiatives & projects, or connected to innovation processes and capacity building
3. Increase knowledge translation and mobilisation quality and speed
This output focuses on the effective and efficient transfer of knowledge and lessons learned within and beyond the innovation ecosystem.
Output Indicators:
- Collaborative publication of knowledge - as evidenced by articles, podcast episodes, co-authored research publications among ecosystem actors
- Reach & perception of knowledge products - as evidenced by unique downloads / listens / reads, and reported satisfaction / use ratings
- Active interface - as evidenced by incoming enquiries about orchestration and ecosystem activity from non-geographically situated people and organisations, and proxies such as incoming funding or support from philanthropic, government and business sectors for the ecosystem.
Outcome Areas & Indicators
A. Healthy, resilient, and vibrant Innovation Ecosystem
This outcome describes a thriving and adaptable environment where diverse actors collaborate effectively, resources flow freely, and the ecosystem can withstand disruption while continuously generating innovation.
Outcome Indicators:
- Ecosystem dynamics - as evidenced by the proxy of density of trust in the ecosystem
- Adaptive capacity of the ecosystem - as evidenced by the proxy of learning capacity and speed, and shared system understanding
- Resource diversification and stability - as evidenced by the proxies of revenue generation and diversity of funding sources, and the longevity of funding types
- Narrative coherence - as evidenced by self reported feelings about coherence of purpose and activity by ecosystem actors and external stakeholders
B. Local and national quality, capacity and capability to better tackle complex challenges
This outcome focuses on the enhanced ability at both local and national levels to effectively address multifaceted and dynamic challenges, and the connections to the innovation ecosystem.
Outcome Indicators:
- Establishment of additional innovation orchestrators - as evidenced by the proxy of imitation flattery - or better put, the replication and adaptation of the innovation orchestration model to drive activity in other locations
- Growth in Practice - as evidenced by the proxies of communities of practice formed, and resources published about innovation orchestration
- Connectivity to Ecosystem - as evidenced by direct and indirect connections such as attendance at events, conference participation, personal online connections and social network analysis to summarise
Considerations, Implications & Recommendations
Evaluation overhead
To effectively measure and demonstrate the value of innovation orchestration is going to be difficult and potentially resource intensive. A pragmatic, phased approach is advisable, prioritising certain metrics that are feasible to collect and analyse without becoming an undue burden on the orchestrator or the ecosystem.
Focusing on 'impact stories' may work as a compelling approach for many stakeholders, especially in early stages whilst the ecosystem is establishing and activating.
Recommendations for Orchestrators
The following are practical suggestions for orchestrators identified by NotebookLM:
- Articulate logic and assumptions: articulate the theory of change behind orchestrator activities. This provides a documented logic and provides a starting point for an impact evaluation framework and helps to identify key assumptions that need to be tested.
- Implement a portfolio evaluation: develop a balanced portfolio of input, output, and outcome indicators for both the orchestrator activities and the innovation ecosystem activities.
- Prioritise contribution analysis: utilise qualitative and mixed-methods approaches to demonstrate how orchestration helps cause systemic improvements. This involves collecting diverse forms of evidence that collectively build a compelling case for the orchestrator's influence.
- Invest in processes and systems: establish robust systems for qualitative and quantitative data collection and ensure access to analytical capabilities.
- Focus on a culture of learning and adaptation at all levels: recognise that evaluation is an iterative process and there is a need to review and refine the measurement framework, adapting metrics as the ecosystem evolves. Ensure evaluation of the orchestration activities and the ecosystem outcomes (across the explore-implement spectrum).
- Aim for participatory approaches: involve the innovation ecosystem in defining objectives, selecting metrics, and interpreting results. This collaborative approach builds trust, fosters shared understanding, and enhances the legitimacy of the assessment.
Supporting a business case
I originally started the process of writing about innovation orchestration to create clarity about different types, functions and capabilities and value creation. This was partially aimed to be in service of developing a value case for investment and funding.
Of course it's largely impossible to do this in a generic way, but it is also very difficult to do in geographic areas where there is no real track record of concerted innovation orchestration efforts.
Looking at evidence for Australia, there is a couple of useful aspects:
- CSIRO's report on Innovation investment resulting in a 3.5X economic return on investment, which doesn't include social and environmental benefits - Report: Quantifying Australia's Returns to Innovation (CSIRO, 2021)
- One Basin CRC - a 10 year, $150m funded R&I interface and innovation orchestrator aims to generate an estimated $4.3 billion of economic impact (through innovation in water, agriculture and energy technology, enhanced forecasting and decision-making capacity, with a focus on sustainability and resilience in the face of climate change)
- Bega Valley Innovation Hub - a Uni of Wollongong x Bega x Bega Valley Shire Council collaborative initiative to spur regional innovation, which doesn't publish ROI analysis or impact reports, but received around $380k from NSW Government to spark the collaboration, with this investment leveraged to around $500k in total.
- Perhaps one undervalued opportunity for Innovation Orchestrators is less about direct innovation ecosystem stimulation, and more about acting as an Interface for larger government and industry funding and investment. For example, the Drought Resilience Adoption and Innovation Hubs saw $89 million in funding across 8 hubs for 4 years. Acting as an interface is a two way street - also acting as a cross-sectoral voice and advocate, such as BVIH's Funding Our Future report.
Conclusion
The last post and this together aim to contribute to fill the gaps in clear definitions for innovation orchestration, and identify a starting point for impact frameworks. Using NotebookLM meant I was able to work with a much larger sample of inputs, and focus on generating synthesis more quickly - but I still reviewed everything and made edits to ensure alignment and coherence.
This fieldnote built on synthesis to lay out some of the pieces of a business case for an innovation orchestrator function. I maintained half an eye on the developments in the Upper Yarra, considering both the Research & Innovation Broker and Place & Systems Orchestrator types from the original post.
Clearly it is impossible to accurately predict ROI until work is underway - it would take political will and a coalition of the willing to establish this kind of orchestration function.
The innovation orchestration I envision here aims to move innovation agendas beyond a shallow focus (such as the likes of entreprenuership for economic impact), with the intent to increase the scale and effectiveness of innovation in a region whilst avoiding the lure of silver bullets for complex challenges.
Innovation orchestrators help move from siloed innovation activity to networked, resilient innovation ecosystems - helping us tackle complex challenges, like the effects of climate change, in a coherent way which improves in effectiveness over time.
References:
To come
