Impact Assessment of Innovation Policy

Methodology Overview

This page provides an overview of the different approaches to impact assessment and the methodological challenges.

Causality Problems

The central problem in impact analysis is to connect an intervention or activity with its effects: that is, to establish what it caused. But interventions take place in a dynamic context. Many changes are afoot and it is not always evident what happened if the intervention had not taken place. 

More information about different causality issues including attribution effects and endogeneity are explored in the document below.

Research and Innovation Process Issues

The nature of research and innovation processes pose certain challenges to impact assessment, among which the following:

  • Both research and innovation processes are highly skewed. In research, it is axiomatic that a small minority of researchers produces the great bulk of the best research. In innovation, it is well understood that many projects will just about break even; many will fail and lose money; only a very small minority is likely to have huge success, hopefully large enough to compensate for the losses and still to generate a substantial surplus. 
  • This skew has to be considered when designing samples in impact analysis. A random sample of beneficiaries, for example, can easily miss the one or two projects that make the big money, leading the evaluator radically to misjudge the performance of the scheme.
  • Impact assessments tend to be done as individual studies using tailored methodologies. These factors alone limit their comparability. However, other issues mean that trying to standardise methods would have a limited effect in making them more comparable.
  • A well-known problem in evaluation is that, in order to make timely policy and especially to secure needed budgets, evaluation results are often needed before the relevant intervention has had any impacts. The question when to do an impact assessment is no less problematic and becomes acute if there is a desire to compare the impacts of alternative policies or instruments. In many cases, the time at which an intervention “matures” and produces impact is varies – and may even be unknown. 

Download the document below to read more about these challenges.

Different Approaches to Impact Assessment

The techniques appropriate to impact assessment necessarily depend, first, upon what the impact assessment is trying to achieve and, second, the data and other resources that can be assembled. As with most things, there is no universal and timeless “best practice”.

Since impact assessment is a policy-related activity, it operates under time and resource constraints to produce the best answer available within these constraints – not necessarily the best possible answer. As ever in policy work, if a decision has to be taken in six months, a two-year study will not answer the question. The obverse of that coin is that impact analysts could often be braver in describing – or sometimes even noticing – the limitations of their work.

The document below discusses methods in relation to four types of impact analysis:

    • Estimating or modelling impacts in money terms;
    • Studies aiming to understand how impacts happen;
    • Focusing on particular limited outcomes;
    • Impact assessment for performance measurement.