The hidden costs of peer review

Sir Peter Gluckman, the Prime Minister’s Chief Science Advisor has released an interesting discussion paper today looking at peer review and its role in influencing what sort of research gets funded in New Zealand.

Those who have been through the Marsden or Health Research Council wringer will find it particularly interesting.

Many scientists have told me how much they resent the vast amount of time they are forced to invest in preparing funding applications which they have a limited chance of securing given the low success rates on some of our big contestable funds.

Sir Peter’s report tries to put a figure on the costs associated with the application process, using the Marsden Fund, which disperses around $55 million in funding annually, as an example.

…the direct administrative costs for operating the grant process are under 3% of the fund size, but this accounting ignores the vast majority of costs to applicants, referees, and panellists.

These can only be estimated, but they are substantial. The best estimate puts the total cost at 20-35% of the fund size, some NZ$10- 20 million. The majority of the cost falls onto applicants. Estimates suggest that the time spent writing proposals represents over 80% of the total fund cost, with three-quarters of that spent on first stage proposals. International reviewers and panellists make up 10% of the total cost and this is a sig- nificant burden upon the small number of people who are called upon in these roles.

Where are the costs incurred?

According to Sir Peter:

For essentially all funding schemes, the major cost is in proposal writing. For unsuccessful applicants, this may not be time that is entirely wasted – there are clear benefits from researching and clarifying ideas, building networks, and the possibility to use those applications for accessing alternative fund- ing – but nevertheless in a small science system it has a major inhibitory effect on research outputs. This is aggravated in New Zealand by the relatively short-term nature of most funding systems, the tendency to underfund requiring multiple sources of support, and the long cycle of assessment: this means that many research active staff are in a con- stant cycle of either writing grants or assisting oth- ers to write grants.

The volume of grants that senior referees are ex- pected to examine means that they increasingly avoid participating in the process9. This is a feature increasingly noted in small countries. It is unsur- prising that the system is fragile given the variable requirements of review, the sheer amount of re- viewing required, the reality that it has often been expected to be conducted over the holiday season, and the facts that institutionally it is unrecognized and, in New Zealand, unpaid. These issues are not trivial and there are increasing signs of senior scien- tists boycotting requests to participate.

Obviously, many of these costs are unavoidable if we are to have a robust peer-review process. But could we minimise the costs borne by applicants? Yes – by having less applicants.

The major system-wide cost of the funding system is the time spent by applicants in putting forward proposals. Reducing this would allow more re- searcher time to be spent in research. Equally, re- ducing the number of applications to review and the time spent per review would reduce the bur- den on the senior staff who act as reviewers.

Not that Sir Peter is necessarily recommending we tighten up the application process – that could result in significant lost opportunities.

Other areas tackled in the paper:

– Does New Zealand focus too much in the peer-review process on ideas rather than the individuals and teams presenting them?

– To what extent should national priorities feed into proposal assessment criteria?

– Is quality or relevance more important is assessing research proposals?

– How does the science system, particularly in a small country, overcome the issue of panellist bias among those chosen to assess research proposals?

– What is the best way to assess interdisciplinary research?

Well worth a read – download the paper here.








  1. kemo sabe

    The Marsden Fund takes less than 8% of the total government expenditure on R&D, yet it is getting all the flak.
    John- how would funding scientists first and projects second increase the success rate of applicants? And how would you support emerging scientists?

  2. psychokiwi

    The NZ system seems very broken. Although I’m in NZ, I’ve never contested for NZ grants. I mainly work for a soft-funded lab in the US and we contest for grants there. Our main sponsor operates a two-tier process where we put forward short white papers on areas we would like to focus on, with a basic costing, and the sponsor selects those they believe are potentially worth funding (and fits within their funding parameters). They then invite us to submit a full proposal on just those, and they are very likely to be successful because we already know there is a meeting of minds. This has significantly reduced the time spent grant chasing (a few years ago we were expected to spend 30% of our time grant writing).

  3. John Pickering

    Thanks for this Peter. It is v important & adds to my assertion that the grant system is broke. I look fwd to some constructive ideas on how to fix it starting with funding scientists first & projects second

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s