Skip to content
© Pixabay

Research Assessment Processes

Research organisations receive many high-quality research applications and research proposals. They need to be able to identify which proposals are the best for funding, and which researchers should be appointed or promoted. They also regularly evaluate the performance of research institutes and universities. Research assessment processes are an important aspect of ensuring the quality of research, which is fundamental to the research enterprise.

What is the challenge for Research Assessment Processes?

With limited funding and research positions available, there is increasing pressure on research organisations to put processes in place that ensure assessments of research quality are effective, efficient, and fair. For this reason, research organisations dedicate significant effort and resources towards the assessments they conduct, and continually look for ways to optimise and adapt these processes.

Our Recommendations

Science Europe has created a set of policy recommendations in 2020 for its Member Organisations and other research organisations. They were developed following an extensive study performed in 2019 (see the following section), and through a comprehensive consultation process. More information about the methodology followed is also available.

The recommendations will help research organisations to review and improve the effectiveness and efficiency of their assessment processes for career progression and funding allocation. They also promote the sharing of knowledge so that organisations can learn from each other, which will enrich and strengthen national and international research systems as a whole.

Research organisations face many common challenges. Substantial and concerted efforts are needed to ensure that the research assessment system continues to function well in the future and can keep pace with the rapidly changing research environment.

Our recommendations complement other ongoing initiatives, such as the San Francisco Declaration on Research Assessment and the Leiden Manifesto for Research Metrics. This Science Europe activity also supports the central priorities of the European Research Area.

Our recommendations summarised

Each topic includes a few selected examples from what Science Europe Member Organisations do to address the issue.

TRANSPARENCY

Assessment processes must be clear and transparent at all stages

Examples from SE Member Organisations

EVALUATING ROBUSTNESS

Assessment processes should be monitored and evaluated, and best practices shared

Examples from SE Member Organisations

BIAS, DISCRIMINATION & UNFAIR TREATMENT

Research organisations should publicly show how they address bias, discrimination and unfair treatment

Examples from SE Member Organisations

COST, EFFICIENCY & APPLICANTS’ EFFORT

Assessment processes should be streamlined and standardised to improve efficiency for all involved

Examples from SE Member Organisations

  • DFG (Germany) have implemented a user-friendly web portal to collect all funding schemes offered, listed by differented focus.
  • UKRI (United Kingdom) is developing and piloting a new service called the 'UKRI Funding Service' platform that aims to make funding applications simpler and more efficient for all involved.

BROADENING THE POOL OF REVIEWERS

Research organisations should consider broader selection criteria for reviewers and suitably recognise their work

Examples from SE Member Organisations

QUALITATIVE ASSESSMENTS

Assessment processes should enable evaluations to focus on content and consider a wide range of research outputs and activities

Examples from SE Member Organisations

NOVEL APPROACHES

Research organisations should consider novel approaches to assessments in an evidence-based manner and share their experiences

Examples from SE Member Organisations

  • UKRI (United Kingdom) implements 'Sandpit' funding (ie. interactive workshops) to promote multidisciplinary perspectives and innovative solutions to research challenges.

Our Study on Research Assessment Practices

From July 2019 to January 2020, Science Europe and Technopolis Group (Vienna) studied the research assessment processes of research organisations. They intended to see how these organisations had set up their processes, and how they developed them in a time when the way research is performed and disseminated is changing quickly.

The study concluded that research assessment processes are generally well-developed, and that the implementation of new methods and technologies is usually restricted to pilot programmes and specific (rather than general) calls for funding and/or career progression schemes. It highlighted that research organisations face many common challenges and strains, and that they are open to minor and progressive adjustments.

Some key conclusions from the 2019 Study on Research Assessment Practices:

Despite large diversity in the types of research organisation that participated, many shared practices were identified. The need for transparency was seen as a common key driver in the design of assessment processes.

81% of organisations use qualitative assessments

Some forms of bias are scrutinised more often than others: Gender (82%) vs Ethnicity (31%) for example.

Many common challenges to the effective functioning of assessment processes were identified, these included; the continual need to address bias, discrimination, and unfair treatment, and the challenge of balancing the cost and efficiency of assessment processes without compromising on the quality of assessments.

The testing of novel approaches to assessment processes was seen to take place mostly in small and restricted settings (pilot programmes, for instance), with generic assessment schemes relying on more well-established approaches. Adaptations to the criteria used to assess researchers and research proposals were also identified.

Some forms of bias are scrutinised more often than others: Gender (82%) vs Ethnicity (31%) for example.

How does this relate to COVID-19?

Research organisations are vital in the response to the crisis. They can rapidly mobilise research funds to support researchers in finding solutions to the myriad of challenges posed by the crisis.

Maintaining effective and efficient research assessment processes, even in times of crisis, helps to guarantee the quality and integrity of research supported, in turn ensuring that only sound and trusted research contributes to the response to the pandemic. To fund and support researchers, Science Europe's Member Organisations have launched specific calls and initiatives for projects related to the COVID-19 pandemic, as highlighted on the Science Europe COVID-19 web page.

Given the urgency of COVID-19, assessment processes of these calls, in many cases, been adapted to ensure that research can be initiated in a timely manner. These ad-hoc adaptations will provide important insights into how assessment processes for rapid research responses can be better implemented to respond to future crises while maintaining their quality and integrity. These adaptations may also represent a point of reflection on how assessment processes are implemented more generally.

Science Europe Member Organisations play a key role in the vital research response to such global challenges.