Harvard Medical School Library of Evidence
Tell Me More


The information you need to work together with your provider to make the best medical decisions possible – freely available to everyone

Your health is important to both you and your provider, which is why the important medical decisions should always be shared. The “Choosing Wisely” campaign is built around this premise – the idea that we should all be working together to choose the right tests for you based on your specific clinical indications.

The Harvard Library of Evidence takes content from Choosing Wisely and a number of other trusted sources, grades it transparently, and makes it freely available to anyone to incorporate into a clinical decision support system. We are beginning by focusing on imaging, but will soon be moving on to all other diagnostic tests in order to help you and your care team decide on the best testing strategy for you.


The Future of Clinical Decision Support Content

Transparent, interoperable, customizable… so you can deliver evidence-based, patient-specific decision support at your finger tips.

What if you could look up any clinical decision, based on a particular patient scenario, and find out what type of evidence supports or challenges such decision?

The Harvard Library of Evidence provides clinicians, EMR vendors, and IT professionals with curated, interoperable content to embed into your pre-existing EMR so that you can deliver real-time, evidence-based clinical recommendations at the point of care.

Our dedicated team of librarians, clinicians, and engineers has created a library of annotated, graded recommendations regarding appropriate use of testing, using IT-industry interoperability standards. With each piece of logic having been assessed based on quality and strength of evidence, you can pick and choose the content you trust most. In order to meet PAMA requirements, we have initially focused on imaging but will soon be expanding to all diagnostic testing for which evidence-based guidance exists.

Our curation and grading process is based on a framework that has previously been shown to be effective and replicable (Lacson et al, JAMIA, 2016).

Combining our content with a computerized decision support delivery application, the result is an evidence-based, customizable recommendation engine for providers so they can take care of patients at the highest care standard.


The Harvard Library of Evidence Content can help you:

Optimize clinician’s time and workflow
Improve quality of care
Get more value from your EMR
Reduce alert fatigue
Reduce costs


Clinical Decision Support


The Harvard Library of Evidence contains graded recommendations and Appropriateness Use Criteria for Clinical Decision Support for use by EMR vendors, IT professionals, and Qualified Provider Led Entities that comply with Protecting Access to Medicare Act of 2014 (42 USC 1395m, “PAMA”), signed into law on April 1, 2014.

The curation and evidence-grading process is based on a framework that has previously been published (Lacson et al, JAMIA, 2016).

Each recommendation is represented as a unit of evidence and users can select content based on the following pre-defined categories:


  Source Type (e.g. local best practice, professional society guideline, peer-reviewed article)

  Publisher (e.g. ACR)

  Choosing Wisely

  Professional Society

  Imaging Modality

  Body Region

  Strength of Evidence

Recommendations conform to industry standards and can be downloaded, as follows:

  • Propositional logic as text files (in plain txt or xml)
  • Spreadsheet including terms mapped to standard terminologies (in Excel)
  • Clinical Decision Support (CDS) knowledge artifacts in Fast Healthcare Interoperability Resources (FHIR) specification (in xml)

Recommendations and Appropriateness Use Criteria are translated to single decision rules. Translation of single recommendations, branching recommendations (e.g. flowchart-based), and score-based recommendations to propositional logic is performed manually. These are available as plain text in IF...THEN format.

Terminology mapping to three standard terminologies, The Systematized Nomenclature of Medicine (SNOMED), Radiology Lexicon (RadLex), and the International Classification of Diseases Revision 10 (ICD10) is in publication (Yan et al, AMIA Annual Session Proceedings, 2016).


Decision rules are represented in FHIR specification using the FHIR "decisionsupportrule” resource for sharing knowledge artifacts. This is used in conjunction with the HL7 Clinical Quality Language Specification. Decision rules in FHIR will soon be available for download in xml.

QPLEs and The Library


How a Qualified Provider Led Entity will work with the HMS Library of Evidence

Literature Review and Evidence Grading Processes. In pursuit of its primary goals, the HMS Library of Evidence will initially focus on two primary functions specific to the requirements of AUC Final Rule published by CMS under PAMA

  1. A literature review process targeting a clinical decision (e.g., use of imaging in patients with right lower quadrant abdominal pain), done systematically and in a reproducible manner and
  2. An evidence grading process using widely recognized methodologies.

More specifically, the HMS Library of Evidence performs systematic literature searches starting from a list of patient conditions (organized by clinical topics) and imaging studies relevant to such clinical conditions and topics. This list addresses the Priority Clinical Areas specified by CMS as well as clinical topics requested by QPLEs who work with the HMS Library of Evidence. The list will be supplemented and updated from time to time. These literature reviews focus on publicly available guidelines and their cited references as well as evidence from peer reviewed journals, such as systematic reviews, meta-analyses, randomized controlled trials, as well as local best practices submitted by QPLEs for grading. Other clinical studies or clinical decision rules recommended by experts are also reviewed. To the extent practical, the literature reviews consider and describe the analytical validity, clinical validity, and clinical utility of the specific imaging study for the specific clinical condition or topic.

The literature review process results in lists of available evidence linking specific patient conditions and relevant imaging studies. The HMS Library of Evidence assesses and grades such evidence using a methodology based on the 2009 version of the Oxford Centre for Evidence-based Medicine (CEBM)–Levels of Evidence and the U.S. Preventive Services Task Force (USPSTF) I-Scores and NS (not scoreable). The graded evidence is then presented as clinical logic statements that can be formulated as discrete appropriate use criterion by qualified Provider-Led Entities (see http://libraryofevidence.med.harvard.edu/app/public). The key components of each criterion are identified as evidence-based or consensus-based, and are graded by three independent parties – two curators and a validating clinician. Once agreement on a final grade is reached, the full record is made available to the public. As part of its core mission, the output of the Library’s literature review and evidence grading processes is posted on its website and is available for use without charge by any provider-led entity or other third party.

How QPLEs may work with the Library. The HMS Library of Evidence is open to working with QPLEs in many ways. As one example, a QPLE may access the literature review data and the evidence grading data posted by the HMS Library of Evidence on its website as a reference or comparison when evaluating the work of its own AUC teams. This level of access would typically not require a formal relationship with the HMS Library of Evidence.

Another approach would be to work more formally with the HMS Library of Evidence in an arrangement in which the Library supplements the QPLE’s AUC team to perform the literature review or evidence grading function. Under this approach, a QPLE (likely through a domain specific AUC team) would select and review the results of the literature review for specific clinical domains and the evidence grading process conducted by the HMS Library of Evidence as part of the QPLE’s program to develop or modify AUC for specific clinical conditions as required under the Final AUC Rule. While the QPLE would use the HMS Library of Evidence’s outputs as starting points in its AUC process, the QPLE would be responsible for its own formulation of the AUC, its acceptance or rejection of the literature search, and its acceptance or rejection of the evidence grade assigned to a component of an AUC (or an entire AUC) by the HMS Library of Evidence. The QPLE would also be responsible for disclosing its use of the HMS Library of Evidence resources and outputs to CMS and on the QPLE’s website (if required under the Final AUC rule or other CMS requirements).

Conflicts of Interest Process. Recognizing that QPLE will work with the HMS Library of Evidence to comply with the Final AUC Rule, the Library will work with QPLEs to confirm that all individuals who participate in the work of the Library complete the electronic Conflicts of Interest disclosure process required by Harvard Medical School and/or Partners Healthcare. If individuals who are outside these systems should participate in the work of the Library, such individuals will be asked to complete paper versions of this disclosure. The Library will make this COI information available in timely fashion to a public request for a period of not less than 5 years after the most recent published update of its information.



Characteristics of Knowledge Content in a Curated Online Evidence Library

Sowmya Varada, Ronilda Lacson, Ali S Raja, Ivan K Ip, Louise Schneider, David Osterbur, Paul Bain, Nicole Vetrano, Jacqueline Cellini, Carol Mita, Margaret Coletti, Julia Whelan, Ramin Khorasani


Harvard Library of Evidence Newsletter

August 2017 - Volume 1, Issue 1

Evaluating Terminologies to Enable Imaging-Related Decision Rule Sharing

Yan Z1Lacson R1Ip I2Valtchinov V1Raja A3Osterbur D4Khorasani R5


Assessing Strength of Evidence of Appropriate Use Criteria for Diagnostic Imaging Examinations

J Am Med Inform Assoc. 2016 May;23(3):649-53. doi: 10.1093/jamia/ocv194. Epub 2016 Jan 17.
Lacson R1, Raja AS2, Osterbur D3, Ip I4, Schneider L5, Bain P3, Mita C3, Whelan J3, Silveira P6, Dement D6, Khorasani R7.


Assessing Strength of Evidence of Appropriate Use Criteria for Diagnostic Imaging Examinations

The Annual Meeting of the Medical Library Association

May 15-20, 2015, Austin TX

Making the Grade: Curating the Harvard Medical School Library of Evidence

P.A. Bain, R. Khorasani, D. Osterbur, A.S. Raja

The goal of the Harvard Medical School Library of Evidence is to create a sustainable public repository of clinical evidence that will enable and promote broad and consistent evidence-based medicine and improve the quality and efficiency of care. In part a response to recent federal legislation requiring evidence-based clinical decision support systems to be included as part of certified EHRs, the Library includes specific statements of clinical logic representing recommendations extracted from published guidelines or other trusted sources along with an objective quality rating of the evidence underlying each recommendation (level of evidence/grade of recommendation). This quality rating is applied and annotated by librarians with broad medical knowledge but without inherent conflicts of interest or profession-related biases and is validated by a physician with subject area expertise. The Library of Evidence will be distributed in the public domain in a structured form that can be incorporated into healthcare information systems.

Making the Grade: Curating the Harvard Medical School Library of Evidence

"iTunes" for Evidence: the HMS Library of Evidence

Ramin Khorasani, MD MPH


The Library of Evidence (LOE; http://libraryofevidence.med.harvard.edu) or ‘iTunes for Evidence’ is a project of the HMS Countway Library of Medicine, in collaboration with Brigham and Women’s Hospital, and representatives of other HMS hospitals and other Harvard institutions. The goal of the LOE is to create a sustainable, public repository of health IT consumable, objectively scored medical evidence to enable and promote the broad and consistent practice of evidence-based medicine. Launched in 2015, the LOE’s initial focus will be in medical imaging, in part to support the implementation of the Public Law 113-93 (Promoting Evidence-Based Care; Section 218b of Protecting Access to Medicare Act of 2014) in January of 2017.


Ramin Khorasani, MD, MPH is Professor of Radiology at Harvard Medical School, Distinguished Chair, Biomedical Informatics; Director, Center for Evidence-Based Imaging (CEBI) and Vice Chair of Radiology at Brigham & Women’s Hospital. Dr. Khorasani and his colleagues at CEBI are broadly focused on improving quality and reducing waste in patient care using advanced health IT tools. Promoting evidence-based imaging using clinical decision support (CDS) has been a major focus for the past decade.

iTunes for Evidence: the HMS Library of Evidence