When we evaluate the past and future performance of scientists, we tend to reward their proven ability to obtain research funding. This may seem like a pretty legitimate criterion, but it is not. Read why.

Imagine you are a head of department at a university, and you are hiring a new researcher. You can choose between two candidates. Your assessment criteria include ratings of past teaching, publications, funding acquisition and management. Both candidates A and B happen to score equally on all measures, except on funding acquisition.

Criterion	Candidate A	Candidate B
teaching	E	E
publishing	45	45
management	D	D
funding	$10M	$1M

The only difference between both candidates is that candidate B was awarded $1M research funding, while candidate A obtained ten times as much, $10M. Who would you choose?

Assuming that you value funding acquisition skills, and also assuming that you believe that past performance predicts future performance, you might choose candidate A because this candidate has shown a greater ability to obtain research funding in the past. Many selection committees will do the same.

This may be a good choice for the university, but it is arguably not a good choice for society. To understand why, consider yourself a Minister who has sent out a call for research on developing something important, say a COVID test. You have to choose between the bids of two competing scientists A and B. Both proposals are exactly equal in terms of what they promise to produce: quality of the test, duration of the research and controls included. Only, proposal A costs $10M while proposal costs $1M.

Criterion	Researcher A	Researcher B
test quality	E	E
research time	45	45
controls	D	D
cost	$10M	$1M

As a Minister, which proposal do you prefer? I guess you choose B. After all, if both proposals are equally good, why should you spend $10M if the same research can be done for $1M? Why prefer a proposal that wastes $9M dollar from public resources?

How come that these two similar situations result in very different outcomes? Amongst two otherwise equal candidates, as a university head of department you select the person who spends most resources, but as a Minister you select the candidate who spends least resources. Why is this the case?

The answer is that the university and the Minister have different interests. The university is interested hiring a new researcher who is able to obtain funding (and teach, publish and manage), and all other things being equal the one with the highest earning capacity should be hired. But the Minister has the opposite frame: all other things being equal, the candidate spending least public funding is to be preferred.

From a moral point of view, we should take the Minister’s perspective, because it is the Minister who represents our collective interest rather than the interest of a single university. The collective perspective requires that whenever we judge scientists on their contribution, we should rate them positively on the output that they generate but negatively on the amount of public resources that they spend on it.

In academic assessment, we should not assess scientists positively on their ability to obtain funding, but negatively. If they used their research funding to create value in terms of publications, patents, and policy advice, this value will certainly be in the numerator of their score. The money that they spent on achieving this value, however, should not be in the numerator but in the denominator of their score. It is a cost, not an accomplishment.

Afterthoughts

When rereading the above, weeks after I had written it up, some afterthoughts popped up that made me question my own arguments.

The first afterthought was that the two situations I compared are different because the job application describes an assessment of the candidates’ past performance, whereas the COVID proposal describes a promise of future performance. The claim that “past performance is in fact a promise of future performance” might equalize the situations, but is too vague I think to settle the issue.

A better way to improve the comparison is to redefine the situation: Both researchers A and B offer exactly the same COVID test proposal:

Criterion	Proposal A	Proposal B
test quality	E	E
research time	45	45
controls	D	D
cost	$1M	$1M

However, they do differ in terms of their past research performance:     

Criterion	Researcher A	Researcher B
reports	15	15
report quality	8	8
patents	5	5
funding	$10M	$1M

Now, the Minister cannot make a decision based on the proposals because both the proposals are the same. She has to base her decision on the past performance record of researchers A and B. However, her decision should not change. Again, the Minister should choose Researcher B because this researcher used ten times less public funding than Researcher A to obtain the same output for society.

My second afterthought was this. Imagine that researchers A and B are not just anonymous competitors. More precisely, imagine it was researcher A who secured the funding that allowed researcher B to shine in research, teaching, management and funding acquisition. For example, researcher A could be the academic mentor and supervisor of researcher B. 

For the university, this certainly only strengthens their commitment to select researcher A because her funding abilities work as a multiplier for the university. But should it also make a difference for the Minister, and for society? I do not think it should. The Minister should fund researchers for their expected future output, that is for their contributions to solving scientific and societal issues, but not for securing funding. Raising research funding definitely requires a rare combination of creative, cognitive and social skills, but it should be rewarded for its contribution to research, not for its own sake.

A final caveat: In real life “all other things” are never equal, and the challenge of department chairs and Ministers is to carefully weigh the relative value of different types of inputs and outputs.  

Wim Bernasco, senior onderzoeker NSCR