RT2. Research and Innovation Policy Design, Implementation, Evaluation
Policy evaluation becomes increasingly important as the competitive stakes for national economies and industries increase, as the sizes of R&I budgets increase, and as demands for transparency, accountability and performance across public and private entities increase. The design (strategic planning) and implementation (resource allocation) of R&I policy critically depend on what policy decision-makers learn from impact (ex post) assessment and from the blending of those results with forward looking (foresight) exercises (Link and Vonortas, 2013).
Increased emphasis on evaluation comes with many challenges. Perhaps the trickiest is related to the institutionalization of the practice of evaluation of R&I policies and programs. The challenge is how to make it systematic and effective, guaranteeing the generation and use of information in order to improve the quality, efficiency and effectiveness of policy intervention (Cunningham and Nedeva, 1999; Gertler et al., 2011). This requires the implementation of a complete evaluation cycle: complementary processes linking ex ante evaluation – also including foresight2 (Miles, 2010) – with monitoring and with impact evaluation processes (ex post) in order to achieve consistency between what is expected in terms of results and impacts and what will effectively be measured after the intervention (Campos, 2010; Cozzens, 2012;).
Related to this is the challenge of the contextualization of meta-evaluation, meta-analysis and evaluation synthesis (Scriven, 2009; Stufflebeam, 2001). Meta-evaluation is generally oriented to the measurement of the quality of a given evaluation, based on criteria of validity, credibility, clarity, suitability, utility and generalization (Hanssen et al., 2008; Jacob and Affodegon, 2015). The problem here is accounting for the different contexts of evaluation exercises across countries, industries, and scientific fields. Currently, large depositories of studies are being established for the diffusion of knowledge. One of these is being created at UNICAMP. InSysPo is already engaged in a global initiative called “Science and Innovation Policy Evaluations Repository” (SIPER) coordinated by researchers from the Manchester Institute of Innovation Research. SIPER is a repository of existing evaluation reports focused on European Union, OECD countries, and Latin American countries. InSysPo researchers are responsible for gathering, analyzing and inserting evaluation reports from Latin American Region in the repository. This project gives access to a huge database (probably the biggest worldwide) with more than 600 policy evaluations.
One of the most difficult issues in R&I investment strategy of both the public and the private sectors is the understanding of the interconnections between the diverse parts of complicated R&I portfolios in environments of intense market and technological uncertainty. In large technology intensive companies, a portfolio-wide perspective to R&I management has long been applied to better align research project investments with the firm’s overall strategic goal of economic return maximization (Schilling, 2017). With relatively few exceptions (Ruegg, 2007), however, public R&D management still tends to base selection processes on the individual excellence of projects according to peers rather than considering the merits of the whole portfolio (Linquiti, 2015; Linton and Vonortas, 2015). Besides the usual inertia and the resistance by scientists sticking to the peer review process, there are other serious reasons why this may be so. Public programs will frequently have multiple objectives requiring multiple (perhaps incompatible) performance measures for evaluation. Research projects and programs in public research portfolios can be interdependent. Their outputs are typically removed from the market, thus making monetary valuations arbitrary.
Such complications are compounded by the fact that R&I investments are subject to increased levels of risk and, often, downright uncertainty regarding both technologies and markets.3 In the case of more applied research projects, estimations of risk and return are feasible. While in such contexts the traditional discounted cash flow methods (net present value) of appraisal can be utilized, these methods have been heavily criticized (Vonortas and Desai, 2007) for being inadequate in characterizing public R&D investments. Alternative methods such as “real options” presumably allow better assessment of the value of R&D management flexibility in the presence of risk and of the differential effects on each R&D project. Presently, the use of “real option portfolios” provides the analytical state-of-the-art drawing a lot of attention in industry and, to some extent, the public sectors (Linquiti, 2015; Linton and Vonortas, 2015; Vonortas and Rafols, 2018).
There is hardly any more challenging problem nowadays than the systemic evaluation of mission-oriented R&I policies. Referring to major endeavors such as the Brain Initiative (US), the Hydrogen Society (Japan), and Clean Air London (UK), mission-oriented policies (MOP) is the closest one can find to system innovation. Going beyond fixing markets, MOP look at creating new markets, based on innovations along the entire value chain and the active entrepreneurial role of the state (Mazzucato, 2013). MOPs are characterized by directionality, organization, risk and rewards (Fisher et al., 2018). MOPs are multi-faceted, intended to create incentives to save and invest, setting institutions to promote education and skills, building infrastructures to support innovation, and shaping long-term behavior. MOPs are a major push currently in the newly designed ninth European Framework Programme for Research and a major topic of interest internationally, also including Brazil (Alves, Vonortas, Zawislak, forthcoming; Mazzucato and Penna, 2016). A recent high-level conference in Vienna, Austria, entitled “Impact of Research and Innovation Policy at the Crossroads of Policy Design, Implementation and Evaluation” (5-6 Nov. 2018) addressed the challenges involved in designing and implementing such policy initiatives.
