Research Assessment Processes
Research organisations receive many high-quality research applications and research proposals. They need to be able to identify which proposals are the best for funding, and which researchers should be appointed or promoted. They also regularly evaluate the performance of research institutes and universities. Research assessment processes are an important aspect of ensuring the quality of research, which is fundamental to the research enterprise.
What is the challenge for Research Assessment Processes?
With limited funding and research positions available, there is increasing pressure on research organisations to put processes in place that ensure assessments of research quality are effective, efficient, and fair. For this reason, research organisations dedicate significant effort and resources towards the assessments they conduct, and continually look for ways to optimise and adapt these processes.
Our Recommendations
Science Europe has created a set of policy recommendations in 2020 for its Member Organisations and other research organisations. They were developed following an extensive study performed in 2019 (see the following section), and through a comprehensive consultation process. More information about the methodology followed is also available.
The recommendations will help research organisations to review and improve the effectiveness and efficiency of their assessment processes for career progression and funding allocation. They also promote the sharing of knowledge so that organisations can learn from each other, which will enrich and strengthen national and international research systems as a whole.
“Research organisations face many common challenges. Substantial and concerted efforts are needed to ensure that the research assessment system continues to function well in the future and can keep pace with the rapidly changing research environment.”
2020 Position Statement on Research Assessment Processes
Science Europe calls on research funding and performing organisations to continuously evaluate their research assessment processes to ensure that they are effective, efficient, fair, and transparent.
Our recommendations complement other ongoing initiatives, such as the San Francisco Declaration on Research Assessment and the Leiden Manifesto for Research Metrics. This Science Europe activity also supports the central priorities of the European Research Area.
Our recommendations summarised
Each topic includes a few selected examples from what Science Europe Member Organisations do to address the issue.
TRANSPARENCY
Assessment processes must be clear and transparent at all stages.
FWO (Belgium) has implemented a two-week rebuttal phase in its project funding schemes, during which applicants can respond to comments of reviewers. This gives them the chance to clear up any misunderstandings or counter unfounded assessments.
INFN (Italy) publishes webpages and reports dedicated to the transparency of all their internal processes, including those followed for research evaluation.
DFF (Denmark) ensures the transparency of its research assessment processes by implementing strict conflict-of-interest regulations, applicant right-to-reply processes, and open (non-anonymous) reviewing.
EVALUATING ROBUSTNESS
Assessment processes should be monitored and evaluated, and best practices shared
HRB (Ireland) periodically publishes detailed evaluation reports that include both output, outcome, and impact assessments.
FNP (Poland) periodically evaluates its funding programmes in 3 ways: via self-monitoring, internal evaluation, and external evaluation. The evaluation reports are made publicly available.
FNR (Luxembourg) commissioned an external evaluation of its selection processes based on surveys of applicants, reviewers, panel members, and staff. This highlighted strong accomplishment and provided some recommendations for continued success.
BIAS, DISCRIMINATION & UNFAIR TREATMENT
Research organisations should publicly show how they address bias, discrimination and unfair treatment
FWF (Austria) regularly commissions independent, external checks of bias in its assessment and decision-making processes. It publishes the findings of these studies on ex ante peer review and the outcomes of FWF projects.
CSIC (Spain) launched the ‘Women and Science Commission’ to study the possible causes that make it difficult for women to enter and pursue a research career. It proposes actions to help CSIC achieve equality between women and men.
NCN (Poland) applicants are entitled by law to appeal in case of any perceived bias or discrimination. Scientific co-ordinators oversee the process and can request re-reviewing or expel reviewers/panel members if necessary.
Forte (Sweden) Forte incorporates ‘gender and diversity perspectives in the content of the research’ into its assessment practices as one of four sub-criteria of ‘scientific quality’.
COST, EFFICIENCY & APPLICANTS’ EFFORT
Assessment processes should be streamlined and standardised to improve efficiency for all involved
UKRI (United Kingdom) are developing and piloting a new service called the ‘UKRI Funding Service’. Based on a new digital platform, it aims to make funding applications simpler and more efficient for all involved.
FNP (Poland) have developed an interactive webpage with clear guidance and checklists on its application and evaluation processes and requirements. It also provides information on specific evaluation criteria and the type of evaluating body.
DFG (Germany) have implemented a user-friendly webpage that displays all funding schemes offered, listed by different focuses. This allows applicants to efficiently review the funding opportunities open to them.
BROADENING THE POOL OF REVIEWERS
Research organisations should consider broader selection criteria for reviewers and suitably recognise their work
SFI (Ireland) have developed a ‘Reviewer Code of Conduct’ to maintain and ensure high-quality peer review. The use of international independent reviewers is an integral part of SFI’s decision-making process.
RCN (Norway) uses a mixture of international and national experts, and bases the composition of reviewers and panels on the objectives of a scheme. Lists of reviewers for each scheme are published for a limited period of time.
AKA (Finland) provides clear information on the expertise of panel members and reviewers used, with an emphasis on recruiting international reviewers. National reviewers are used where societal relevance is an important consideration.
HRZZ (Croatia) builds international partnerships to expand its reviewer pool. This allows them to broaden the criteria used to select reviewers, alleviate the effects of reviewer fatigue, and maintain quality.
QUALITATIVE ASSESSMENTS
Assessment processes should enable evaluations to focus on content and consider a wide range of research outputs and activities
SNSF (Switzerland) are testing a new standardised structure for CVs (‘SciCV’) for applicants in biology and medicine. Among other innovations, applicants can present their contributions in a narrative style, which helps to make academic activities, beyond publications, more visible and valued.
NWO (the Netherlands) has introduced a uniform narrative CV format in its Talent Programmes Veni, Vidi, and Vici to improve quality of assessments. It rests on the premise that research needs a diversity of talents and there is not one ideal type of researcher. The use of journal based metrics is banned.
NOVEL APPROACHES
Research organisations should consider novel approaches to assessments in an evidence-based manner and share their experiences
FWF (Austria) recently piloted a hybrid double-blind assessment/random-allocation system as a new way of effectively and efficiently conducting assessments for funding allocation. This was part of their ‘1000 Ideas’ programme.
RCN (Norway) have implemented a Sandpit funding scheme called Idélab, where assessments are made during highly interactive and multidisciplinary workshops. This model promotes multidisciplinary perspectives and supports innovative ideas and projects.
EPSRC, part of UKRI (United Kingdom) are currently running a remote ‘Sandpit’ where assessments for funding allocation are made in a virtual workshop format. This approach aims to promote collaborative thinking and supports innovative solutions to research challenges.
Our Study on Research Assessment Practices
From July 2019 to January 2020, Science Europe and Technopolis Group (Vienna) studied the research assessment processes of research organisations. They intended to see how these organisations had set up their processes, and how they developed them in a time when the way research is performed and disseminated is changing quickly.
The study concluded that research assessment processes are generally well-developed, and that the implementation of new methods and technologies is usually restricted to pilot programmes and specific (rather than general) calls for funding and/or career progression schemes. It highlighted that research organisations face many common challenges and strains, and that they are open to minor and progressive adjustments.
Some key conclusions from the 2019 Study on Research Assessment Practices:
2019 Science Europe Study on Research Assessment Practices
In 2019, Science Europe conducted a flagship study on research assessment processes and practices. The study was developed and overseen by the Science Europe Task Force on Research Assessment and the Science Europe Office, and implemented by Technopolis Group Vienna.
Despite large diversity in the types of research organisation that participated, many shared practices were identified. The need for transparency was seen as a common key driver in the design of assessment processes.
Many common challenges to the effective functioning of assessment processes were identified, these included; the continual need to address bias, discrimination, and unfair treatment, and the challenge of balancing the cost and efficiency of assessment processes without compromising on the quality of assessments.
The testing of novel approaches to assessment processes was seen to take place mostly in small and restricted settings (pilot programmes, for instance), with generic assessment schemes relying on more well-established approaches. Adaptations to the criteria used to assess researchers and research proposals were also identified.
How does this relate to COVID-19?
Research organisations are vital in the response to the crisis. They can rapidly mobilise research funds to support researchers in finding solutions to the myriad of challenges posed by the crisis.
Maintaining effective and efficient research assessment processes, even in times of crisis, helps to guarantee the quality and integrity of research supported, in turn ensuring that only sound and trusted research contributes to the response to the pandemic. To fund and support researchers, Science Europe's Member Organisations have launched specific calls and initiatives for projects related to the COVID-19 pandemic, as highlighted on the Science Europe COVID-19 web page.
Given the urgency of COVID-19, assessment processes of these calls, in many cases, been adapted to ensure that research can be initiated in a timely manner. These ad-hoc adaptations will provide important insights into how assessment processes for rapid research responses can be better implemented to respond to future crises while maintaining their quality and integrity. These adaptations may also represent a point of reflection on how assessment processes are implemented more generally.
Science Europe Member Organisations play a key role in the vital research response to such global challenges.