Impact Assessment of Innovation Policy

Impact Assessment of Innovation Policy

This Community of Practice provides a forum for discussions regarding the impact assessment and evaluation of innovation policies. It also provides insights from different OECD projects on these questions. 
Building on the OECD's knowledge on impact assessments, it seeks to help identify best practices with regards to impact assessment of public research policies.

Impact Assessment - Current Practices

The primary purpose of impact analysis is to support policy development. The choice of technique(s) used needs therefore to be guided by policy needs, rather than the pursuit of some “methodological best practice”. Comparing across impact assessment and evaluation techniques and approaches, the following points arise:
  • Good practice in evaluation is to use multiple methods. Approaches tend to be individually unreliable so using more than one and comparing or “triangulating” among their results is a way to increase confidence in the plausibility of the results.
  • The introduction of performance-based research funding provides a laboratory for exploring the usefulness of impact analysis in steering performance.
  • Some economic techniques are best used ex ante, notably general equilibrium (GE) models and cost-benefit analysis. GE models “evaluate” only in the sense that they show expected effects of changes in input parameters, such as the amount of R&D performed, though cost-benefit analysis does provides a structured framework within which to assess the balance between benefits and costs.
  • Microeconomic analyses of impacts of individual interventions can be rich sources of understanding. The key intellectual problem is handling the counter-factual. While ingenious approaches have been developed in recent years, all involve some sort of proxy and great care is needed when interpreting results.
  • Most research and innovation impact assessments and evaluation exercises make extensive use of qualitative methods, including surveys of beneficiaries and case studies. In part this reflects a lack of alternatives but also a policy requirement to deliver answers early in the process of generating impacts, often before they become in any sense measurable.
  • These techniques remain important because they provide the detailed level of explanation of how impacts arise that is missing from economic analyses. Refined forms of tracing are also opening the door to a much more nuanced understanding of long-term impact mechanisms that ought to flow through to policymaking.
  • Bibliometrics is maturing as a discipline and becoming quite widespread as a practice while new and alternative metrics are being devised whose meaning is barely understood. Social network analysis continues to evolve but finally lacks policy impact because of a lack of theory connecting network shapes and behaviours with innovation outcomes.

Challenges for the Future

Despite the great deal of exploration of the research-innovation relationship over the last fifty years, models of “impact” are still generally linear. Coping with the non-linearity of this relationship and the fact that its form and nature vary greatly across different parts of science, technology and society remains a significant agenda item for the future development of impact analysis.

Developing more realistic and nuanced impact models remains a significant agenda item for impact analysis. In particular, making the impact-mechanism assumptions of econometric work more realistic would improve both our quantitative and our qualitative understanding, enabling us to make better policy.

Developing a better understanding of the relationship between new knowledge and policy assumes considerable urgency in the light of the “grand challenges” such as climate change. Impact analysis at the level of such challenges has barely been attempted, but the growing importance of challenge-driven policy means that impact analysis will need to tackle this task.

Content Visibility