Dear European Research Council, evaluating grant programs is harder than you think

Today the European Research Council tweeted about a study that supposedly shows how succesful their research grants are.

ERC grants provide a lot of money to upcoming and established researchers who are based in Europe to carry out larger research projects and agendas. Of course we would like to know whether the money is well spent. That’s why the ERC commissioned this study which found out that “73% of projects evaluated have made breakthroughs or major scientific advances” (follow the link in the tweet). Great success, huh? Well, I was not so happy.

The relevant metric to judge whether ERC grants are effective is not whether they create scientific breakthroughs but whether they create additional breakthroughs that otherwise wouldn’t have occurred. That’s what I meant with counterfactual thinking in my tweet. We have to compare the status quo, i.e. number of breakthroughs that happened in projects with ERC funding (the factual situation), with a hypothetical (counterfactual) situation, in which these projects wouldn’t have received an ERC grant.

Maybe the projects would have never been carried out without the grant. But maybe they would also have gotten funding from another source—for example a national science foundation. In the latter case it could very well be that exactly the same scientific breakthroughs would have happened. Then the additionality of an ERC grant—or what statisticians call a treatment effect—would have been zero.

Teasing out the treatment effect from non-experimental data is tough, but it can be done. We need more sophisticated methods and know-how than the authors of the linked study possessed though. That’s for sure.

I admit that I felt a bit triggered when I read the tweet this morning. In my job market paper I’m evaluating the effectiveness of another European grant program for innovative young companies. And I saw this type of naive “success story evaluations” all the time in policy documents: “projects that we’ve funded created this and that”. Great, but this tells us nothing about whether ERC grants are necessary or worth the money. So we could spend millions of taxpayers’ money on programs that create little or no extra value. Or, grants are super effective and we should increase their budget immediately. We simply cannot tell from the study.

Proper evaluation of grant programs is important. We want to spend our money effectively in order to make a real contribution to society. And we want to learn which kind of programs work and which don’t. We actually have the statistical tools to do a sound evaluation. So please, ERC, next time hire people with the right skills for the job.

Advertisements

What’s Innovation Economics All About?

Preface: On Wednesday I successfully defended my dissertation and am now the proud holder of PhD in business economics from KU Leuven. In this post I would like to share the opening chapter of my thesis (title: “Three Essays on Innovation Economics”) with you. It’s a bit longer than what I usually put on this blog. But I think it’s worth a look nevertheless. I don’t only give a brief, non-technical introduction into my work but also go into what fascinates me about innovation economics—a field which still lacks the recognition it deserves in mainstream economics.


“They are more powerful, Sir, than we,” answered Imlac, “because they are wiser; knowledge will always predominate over ignorance, as man governs the other animals. But why their knowledge is more than ours I know not what reason can be given but the unsearchable will of the Supreme Being.”

Samuel Johnson, Rasselas

In his apologue The History of Rasselas, Prince of Abissinia, the famous English writer and philosopher Samuel Johnson puts these lines into the mouth of Imlac, a loyal friend of Rasselas, who is the prince of Ethiopia. One day, Rasselas grows tired of his sheltered upbringing in the “Happy Valley”. So he decides to escape with a group of trusted confidants to explore the world. As a widely-travelled man, Imlac is the ideal companion for such an expedition. He has been to Syria and lived in Palestine for several years, where he got in touch with people from various lands and cultures—including travellers from the North and West.

Rasselas is curious about Imlac’s encounter with the Europeans. He admires their power and prosperity. “By what means,” he asks his friend “are the Europeans thus powerful?” Imlac’s answer is simple. It is their superior knowledge that spurs their commerce and determines their wealth. By mastering a plethora of useful arts, these nations are able to expand their sphere of interest across the entire globe. And their ideas and innovative ways allow them to live a life of material abundance despite an unfavorable climate they have to endure. Rasselas is pleased with Imlac’s response. He cannot wait to start their journey in Palestine and see the “mighty confluence of nations”, where Occident and Orient meet, with his own eyes.

Johnson, however, keeps the source for such remarkable knowledge and dexterity secret from his character. Instead, Imlac has to refer to the inscrutable ways of God. By contrast, many people of Johnson’s time would have agreed that men are indeed capable of deciphering this secret,* even if progress were only to be made slowly. In that vein, this dissertation constitutes a modest attempt to contribute to the ongoing scholarly endeavor trying to shed light on the sources of science, technologies and innovations that eventually allow societies to prosper economically.

It took years for economists to pick up on Johnson’s idea. After decades of fixation on the accumulation of physical capital—definitely since Marx (1867), but most likely much earlier—endogenous growth theory brought back the notion that increasing living standards are driven by the introduction of innovations and new ideas (Romer, 1996, Chapter 3).

In many endogenous growth models (Romer1990, Aghion1992) the production function of knowledge takes a simple form

\frac{\dot{A}}{A} = \alpha \cdot S.

\dot{A}/{A} stands for the growth rate of total factor productivity in continuous time, but is commonly interpreted as the arrival rate of new ideas (Bloom, 2017). S is a measure of research input, for example, the number of scientists in an economy. And the parameter \alpha represents the research productivity with which inputs are translated into new knowledge.

Notwithstanding its simplicity, this generic form of the knowledge production function points towards important questions within the field of innovation economics. Do private actors sufficiently provide input S to the knowledge production? Or are there market failures that government policies need to address? At what rate are new ideas created in the economy? And how should research be organized in order to maximize its productivity \alpha?

Knowledge and new ideas, as the final product of the knowledge production function, constitute one node in a causal chain that eventually affects many other variables that economists care for (see Figure 1.1). In an endogenous growth model they result in higher exponential growth rates at the macro level. At the firm level they lead to new products and more efficient production techniques, which raise firms’ turnover and profits. Consequently, new ideas are linked to firm entry and survival and therefore determine the market structure in an industry. Furthermore, it is via all these channels that innovation affects conditions on the labor market; and thus influences the well-being of a majority of people in society.

figure11.PNG

The following chapters revolve around the various aspects of knowledge production and the effects that new ideas exert on the economy. Chapter 2 takes the output of the knowledge production function as given and explores the impact of innovation and new technologies on the evolution of industries. In particular, the chapter is concerned with shakeouts—one of the most drastic developments typically observed over the industry life-cycle. During a shakeout, an initially steady entry rate of firms into a new market suddenly comes to a stop and is followed by a short phase of unusually high exit rates. Within the time frame of ten to fifteen years, the number of producers in a market can fall by more than half (Gort, 1982). Because of this abrupt and drastic upheaval, shakeouts reallocate profits and destroy jobs. A fairly recent example for a shakeout is the global solar panel industry, which entered into a severe consolidation phase at the beginning of this decade.**

Although several explanations for the occurrence of shakeouts have been put forward in the literature, technological factors figure among the most prominent factors (Klepper, 2005). Chapter 2 develops a model of oligopolistic competition in which firms’ adoption of cost-cutting innovations increases optimal firm size and paves the way for consolidation. Strategic considerations, such as accelerated entry due to a preemption motive at the onset of an industry and increased perseverance to sustain negative profits, reinforce the shakeout even further. Previous papers were not able to model such strategic interaction in the dynamic setting of an industry evolution. In that sense, the chapter makes use of advances in economic methodology to provide a more accurate description of the phenomenon of shakeouts in industries that are characterized by rapid technological change.

Chapter 3 shifts attention to the \alpha parameter in the knowledge production function. Innovation projects are characterized by high uncertainty in terms of technical feasibility and final market acceptance. Effectively managing this uncertainty poses a considerable challenge for firms. Compared to traditional upfront investment, where the majority of costs are incurred at the beginning of a project, staging of R&D investments has been proposed as a tool for dealing with uncertainty. Because projects are split in several steps and the allocation of financial resources is organized along pre-specified milestones, staging facilitates the abandonment of underperforming innovation projects and allows firms to explore a wider range of opportunities.

Although this concept is appealing in theory, prior literature has pointed to the difficulties firms have implementing the approach in practice (Cooper, 2008). Pre-defined milestones are often not evaluated rigorously, which impedes timely discontinuation of projects and undermines the possibility to examine a larger number of R&D ventures. Given the mismatch between predictions of theoretical models of staging and observed firm behavior, Chapter 3 explores the cognitive decision processes that can lead to suboptimal behavior by R&D managers.

Eventually, Chapter 4 alludes to one of the most fundamental topics in the field of innovation economics. Incentives for private actors to invest in R&D are usually not sufficient to provide S, the input to the knowledge production function, at a socially optimal level. This is because innovation entails large positive externalities to society—something that Johnson clearly foresaw in his apologue—and financial markets for R&D often perform poorly. Governments have been called to correct these kinds of market failures by various policy measures, including the distribution of subsidies for innovation projects.

Assessing the effectiveness of direct R&D grants to firms ultimately is an empirical question. It is complicated by the fact, however, that in a majority of cases grant receivers are not selected randomly. Instead, firms need to apply for public support and, subsequently, government authorities decide upon the viability of a proposed innovation project. This two-stage process leads to a selection based on the quality of proposals, which renders the application of standard econometric techniques unsuitable. To date, a large literature has made surprisingly little progress in this regard. Chapter 4 develops an empirical strategy that is appropriate to overcome the problem of selection and thereby provides reliable estimates of the relationship between R&D subsidies and knowledge-driven firm growth.

Having laid out the roadmap for the following chapters I would like to conclude with Imlac’s words:

“Knowledge is certainly one of the means of pleasure, as is confessed by the natural desire which every mind feels of increasing its ideas. Ignorance is mere privation, by which nothing can be produced; it is a vacuity in which the soul sits motionless and torpid for want of attraction, and, without knowing why, we always rejoice when we learn, and grieve when we forget. I am therefore inclined to conclude that if nothing counteracts the natural consequence of learning, we grow more happy as our minds take a wider range.”

With this spirit in mind, I hope that reading this dissertation will stimulate the audience’s curiosity and will be perceived as an enjoyable act of knowledge production.


* In his newest book, Joel Mokyr argues that during the Age of Enlightenment in the 18th century a culture which regarded unraveling the secrets of the natural world as a reverence to God’s creation laid the ground for the subsequent Industrial Revolution.

** See https://www.technologyreview.com/s/428196/the-bright-side-of-a-solar-industry-shakeout/ (accessed on 30 March 2017)

References available upon request.

Why Tobit models are overused

In my field of research we’re often running regressions with innovation expenditures or sales with new products aon the left-hand side. Usually we observe many zeros for these variables because firms do not invest at all in R&D and therefore also do not come up with new products. Many researchers then feel inclined to use Tobit models. But frankly, I never understood why. Continue reading Why Tobit models are overused

Networking For Innovation

Olav Sorenson from Yale published a new NBER working paper called “Innovation Policy in a Networked World”. The essay is quite interesting because it reviews insights we got from social network theory (no, not Facebook, although you could analyze Facebook with the same tools) and puts them into context for designing effective policy measures to stimulate innovation. Continue reading Networking For Innovation

Innovation on (government) demand?

Next week we will organize the 7th ZEW/MaCCI Conference on the Economics of Innovation and Patenting in Mannheim and the program will be great. We will have Bronwyn Hall from Berkeley and Pierre Azoulay from MIT as keynote speakers. I’m definitely looking forward to hear them speak.

Myself, I will present a new project on the relationship between public procurement and innovation. In brief the research question is the following. Continue reading Innovation on (government) demand?