Non For Further Consideration for NHMRC project grants in Australia
The National Health and Medical Research Council (NHMRC) is the major biomedical funding agency in Australia and equivalent to the ANR in France, NIH in the USA or the CIHR in Canada. The NHMRC funds many schemes varying from development grant for academic — industry partenrships, fellowships or the projects and program grants for basic and clinical research in Australia. The project grant scheme is essential, if not crucial to many laboratories in Australia to conduct the research and represents half of the NHMRC annual budget. The selection of the best projects is a lenghty process in Australia spanning over a gestation period between the submission and the release of the outcomes. I will discuss here How a grant proposal end up as a Non for further consideration and doesn’t get discussed on the Grant Review Panel. For those who are familiar with the NHMRC project grant review system, I don’t think I will reveal much here. Once again, this is a personal opinion. This piece doesn’t represent the view of my university, the NHMRC and my colleagues reviewers for the NHMRC.
The project Grant scoring process:
After submission of the project grants, the proposals are assigned to the panel members (primary and secondary spokespersons) and to external reviewers. Only members of the grant review panel (GRP) are allowed to score the applications. The scoring system is divided between the scientific quality of the project (50%), the innovation and significance (25%) and the track record of the applicants (25%). The applications are reviewed from usually 1 or 2 external assessors and 2 members of the GRP (primary and secondary spokespersons). A first score is given to the application for each descriptor from the spokespersons (SP). In my previous piece, I detailed the scoring system and the peer review process here (https://medium.com/@GaetanBurgio/the-national-health-and-medical-research-council-nhmrc-project-grant-review-process-5ae8c0e3c23b#.huot5cbsh). I won’t go back to this again. Once the proposal is scored, the applicants have the opportunity to address the comments and the SPs to rescore the application. After the scoring process, the applications are ranked and roughly 50% of the application are considered non competitive. These non-competitive applications won’t be discussed at the GRP. The criteria for applications to be discussed on the panel are precise. Are discussed on the GRP and excluded from NFFC 1) If one or both of the SP scores are missing the application is excluded from NFFC. 2) Applications with an overall weighted mean score from the 1SP & 2SP that differs by two or more points and where one of the SP scores falls above the NFFC cut-off are excluded. 3) National Category 5 New Investigator and Cancer Australia Young Investigator applications are excluded from NFFC. 4) Applications that did not receive any External Assessment are excluded from NFFC.
The bottom 50% of applications are determined from each individual panel .A notification for Not For Further consideration (NFFC) is sent around September. The remaining proposals were discussed on the panel. Today, sadly, many researchers in Australia received a NFFC notification and I will discuss here how a proposal end up not to be discussed on the panel.
Not For Further Consideration (NFFC)
The grants with the lower scores within a panel are classified as NFFC. There are roughly 3 categories of these proposals. The first obvious one are proposals that are flawed and had bad reviews from SPs and External assessors. These proposals have usually a score of 3 to 4. More commonly proposals with respectable reviews but considered as flawed or not convincing/too risky from the SPs and/or external reviewers finish as NFFC. These proposals usually get a score of 4 to 5. Finally good to very good proposals end up as NFFC and do not get discussed on the panel. Usually their score is above 5. While a low score proposal logically got a NFFC notification, it is fairly surprising to see good to very good proposals not been discussed on the panel.
The dynamics of a grant review panel.
The dynamics of a GRP is interesting. A GRP gathers researchers from various background (basic researchers, public health, clinicians) and different level of seniority from senior postdoctoral fellow to ARC or NHMRC Laureate professors. Personalities varies too from highly confident to shy. As mentioned above the scoring system depends strongly on the SPs. Within a panel, some SPs score highly all the proposals they are reviewing (I saw in the past SPs scoring 6 or 7 to every single proposals they were reviewing) and some score badly all the applications (Tail poppy syndrome). While the large majority SPs are scoring the applications accurately, there is only the need of one or two panel members scoring highly every single application to break the dynamics of a GRP. As these applications usually finish with discordant scores, they always are discussed on the panel but rarely got funded. Some really bad proposals end up discussed at the panel meeting and their score falls from a 6 or 7 to a 3 within less than 2 minutes discussion only with unanimously bad comments from the other panel members. By contrast very good proposal with good reviews and decent score (5 to 5.2) got trapped and receive lower scores from those that were artificially over ranked or inconsistantly classified between Sp1 and SP2. As 50–60% of the proposals are discussed on a panel, these 5 to 5.2 scored proposals finish as a NFFC and are not all rescuable. From my experience, there are 10–20% of those in a GRP.
My view on NFFC and what I propose to improve this situation:
While this is understandable that SP should spend less time reviewing tones of applications, I think the NFFC gives a very bad signal to Australian biomedical researchers. Many proposals are competitive, innovative and deserves discussion on the panel. Qualifying these proposals as NFFC is simply wrong and sadly it affects primarily junior investigators as their proposals are more likely to be flawed compared to established investigators that are writing solid but non innovative proposals. The pressure on the system promotes solid proposals but certainly not innovative.
The training for SPs on how to score a proposal consists in 2 grant panel induction videos from the NHMRC, which is an absolute joke (see here https://www.nhmrc.gov.au/grants-funding/peer-review/nhmrc-grant-review-panels-induction). The incoming panel members require a minimum of proper training on how to score a proposal according to the NHMRC categories. This is clearly inexistent from the NHMRC. From my experience, I saw a lot of proposals that were wrongly scored. I don’t think it would be hard from the NHRMC to book a room at Brisbane, Melbourne or Sydney and to organise training sessions for incomming SPs. Secondly I would propose to allow 70% of the proposals to be discussed on the GRP. That would certainly put more pressure on the SPs but at least good proposals would be discussed on the GRP and this absolutely essential as 1) this would deliver some hope to primarily junior researchers and 2) would provide a fair discussion around these proposals.
In conclusion:
Finally I would like to reiterate that the grant review system is reasonable but the low funding rate and the increase in the number of application put too much pressure on the review system. Increasing the number of NFFC applications is the wrong answer to a legitimate situation. I have addressed in party this in my previous post (https://medium.com/@GaetanBurgio/the-national-health-and-medical-research-council-nhmrc-project-grant-review-process-5ae8c0e3c23b#.gowivw5x7). Sadly many very good technicians and scientist will loose their jobs and/or will quit science. These colleagues deserves much better consideration for their work and dedication.
