In a 2012 article titled, “Tracking the impact of research on policy and practice: investigating the feasibility of using citations in clinical guidelines for research evaluation” the authors stated:
One of the most challenging areas of research evaluation is around impact on policy and practice.
It’s our goal to help tell the story of the impact of clinical research. We made another big step in reaching that goal by including clinical citations to the clinical guidelines published by The U.K.’s National Institute for Health and Clinical Excellence (NICE).
National Institute for Health and Clinical Excellence (NICE)
NICE was first established in 1999 to serve several core functions — reducing unwarranted variation in practice across the United Kingdom through the development and dissemination of best practice evidence-based standards; encouraging fast diffusion and uniform uptake of high-value medical innovations; and ensuring tax revenues are invested in the National Health Service (NHS) so that health benefit is maximized.
NICE publishes guidelines in several areas including clinical guidelines. The NICE clinical guidelines:
- Recommend how healthcare professionals should care for people with specific conditions.
- Cover any aspect of a condition and may include recommendations about providing information and advice, prevention, diagnosis, treatment and longer-term management.
- Are also important for health service managers and commissioners of NHS services.
To date, there are 181 NICE clinical guidelines.
PlumX & NICE Clinical Guidelines
In the above mentioned article, “Tracking the impact of research on policy and practice: investigating the feasibility of using citations in clinical guidelines for research evaluation” the authors concluded:
The study found that there is great potential for guidelines to be used as sources of information on the quality of the research used in their development and that it is possible to track the source of the funding of the research. The challenge is in harnessing the relevant information to track this in an efficient way.
It is this challenge that we are addressing. PlumX mines the NICE clinical guidelines for references to research. When research is referenced in a NICE clinical guideline, we create a Clinical Citation — as you can see in the example below. We use several identifiers to find the references, specifically, DOIs, PMIDs, PMCIDs and URLs. To date we have found over 43,000 NICE Clinical Citations. While some of these artifacts are well-cited and show usage and captures through Mendeley and EBSCO metrics, many of them show few if any other citations or altmetric measures. (Examples here, here and here.) This showcases how hard it can be to find these hidden, impactful gems as they don’t naturally rise to the top of an institution’s research set using other measures.
Increasingly, funders want more information about the impact of research than citation analysis. In the Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management produced by the Higher Education Funding Council for England (hefce) they recognize that citations in clinical guidelines have potential for quantifying societal impact:
Online clinical guidelines could be useful for medical research funders to help them to assess the societal impact of individual studies.
Demonstrating societal impact is a demand from many funding agencies and funders — and a difficult thing to demonstrate. Clinical Citations is a step toward demonstrating that societal impact.
Using PlumX Analytic Reports and filtering to look at Clinical Citations makes it easier to show this. Below is an Analytic Report page from the PlumX Dashboards product that allows you to show the Clinical Citations by researcher (NOTE: names have been hidden for privacy):
Alternatively, it is also easy to filter and download the specific research that has clinical citations.
Clinical Citations: A Type of Policy Document Citation?
Finding citations to research in policy documents is a hot altmetrics topic. We are working to make these citations more meaningful than a nebulous way of describing documents resulting from the research of non-profits, governmental organizations and think-tanks. One of the things we are doing is categorizing policy documents into meaningful topics. You can read more about our thinking behind this.
We also recognize that clinical practice guidelines and other types of clinical guidelines represent a special opportunity to demonstrate the impact of clinical research on clinical practice. Traditional research assessment methods, particularly based on academic citations, often leave clinical and translational research out in the cold. This is why we’ve chosen to give this particular type of policy document its own special type of citation, and call these out separately from other policy document citations.
Kryl, D., Allen, L., B. S., & I. V. (2012, March 30). Tracking the impact of research on policy and practice: Investigating the feasibility of using citations in clinical guidelines for research evaluation. BMJ Open. doi:10.1136/bmjopen-2012-000897
The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management(Rep.). (2015, July). Retrieved http://www.hefce.ac.uk/pubs/rereports/Year/2015/metrictide/Title,104463,en.html