PlumX and Policy Document Citations

Finding references to research in policy documents is important because there are new demands on researchers and those who support them to demonstrate public engagement with their research, its impact on government policy and cultural life, and societal impact in general.

What are Policy Documents?

The term “policy documents” can be a nebulous way of describing documents resulting from the research of non-profits, governmental organizations and think-tanks. They can take the form of white papers, monographs, pamphlets, articles, books, book chapters or reports. They can contain words in the title like “Policy”, “Guidelines”, “Recommendations” or “Guidance” or they may just be research objects that are used in the creation of policy. They can be designed to drive policy makers and legislators to a particular course of action. They can also be actual statements of policy by public policy making institutions like Congress.

Categorizing Policy Topics

To bring order to the somewhat indefinable, we looked at ways to categorize the realm of policy documents. If you want to learn more about the process we went through to develop the following 12 policy topic categories, we wrote about it in this blog post .

  1. Agriculture & Farming
  2. Arts & Culture
  3. Domestic & Global Health
  4. Economic & Fiscal
  5. Education
  6. Energy & Environment
  7. Global Affairs & Security
  8. Governance & Politics
  9. International Development
  10. Justice System & Law
  11. Science & Technology
  12. Social & Urban

Policy Citations in PlumX

Within PlumX, we will display the topic category from the list above on each policy document citation. The topics a policy document addresses aren’t always discernible in an automated way. We will try to assign a topic category for a source if it is obvious, or use the source’s topic categorizations and map these to our own schema. If we can’t determine a topic category for citation, no topic category appears in PlumX. When they are available, however, it is easy and immediate to understand what kind of policy the research has influenced.

In this example you can see that this article is influencing health policy. PlumX assigns the topic category “Domestic & Global Health” to all policy citations from the World Health Organization. Australia Policy Online provides their own categorization that helps PlumX determine the topic category for the policy document citation from APO. However, the last document from the Scottish government provides no topic category information, so no PlumX topic is assigned.

How Policy Citations will appear in PlumX

Note that we will consider these references to be a new kind of citation, called a Policy Citation, since these references are either:

  • Part of a peer-reviewed process
  • Part of a committee-review process
  • From organizations with controlled review processes

This makes policy document references rise to a higher level than something that might appear on a blog post or other website. It is important to make this distinction as citations are where you can truly understand the impact the research has on policy.

Clinical Citations: A Special Type of Policy Citation

In May 2016 we announced a new type of citation called a Clinical Citation. These are citations to research in Clinical Practice Guidelines (CPGs) or other types of clinical guidelines.

We created a special category for these types of references in clinical policy because of the history of clinical and translational science receiving less citations — and hence less prestige and funding. We wrote about this more when we announced PubMed CPGs in PlumX. This is when we noted a 2013 article titled, “Citation Analysis May Severely Underestimate the Impact of Clinical Research as Compared to Basic Research” where the authors found that different areas of medical research — even within the same medical field — have different citation practices, in fact they found “large differences in citation practices between research areas. Low-impact research areas tend to focus on clinical intervention research, while high-impact research areas are often more oriented on basic and diagnostic research.”

We recently added Clinical Citations based upon the policy documents for clinical guidelines published by the U.K.’s National Institute for Health and Care Excellence (NICE).

Here is an article in PlumX with Clinical Citations:

A research article with Clinical Citations from NICE and DynaMed Plus Topics


Jan van Eck, N., Waltman, L., Van Raan, A. F., Klautz, R. J., & Peul, W. C. (2013, April 24). Citation Analysis May Severely Underestimate the Impact of Clinical Research as Compared to Basic Research. PLOS ONE. doi:10.1371/journal.pone.0062395