Harvard Medical School Library of Evidence
About
Tell Me More

ABOUT THE LIBRARY OF EVIDENCE

The effective use of a clinical decision support system means patients get the right tests, the right medications, and the right treatment, particularly for chronic conditions.

Goals

The Library is a project of the HMS Countway Library of Medicine and representatives of other HMS hospitals and Harvard institutions. It was founded in collaboration with the Center for Evidence-Based Medicine at Brigham and Women’s Hospital.

The goal of the Library is to create a provider-led, sustainable, public repository of medical evidence to enable and promote the broad and consistent practice of evidence-based medicine in the United States in order to improve the quality of care while simultaneously reducing waste and cost.

"The goal of the Library is to create a provider-led, sustainable, public repository of medical evidence to enable and promote the broad and consistent practice of evidence-based medicine"

The Library has been organized to serve as a public resource for medical evidence from all sources and, therefore, will focus on the accumulation, curation, organization and functionalization of medical evidence rather than on the creation of new evidence.


Legislative Context

The Library is intended to support the national imperative to encourage the practice of evidence-based medicine, including the efforts of the U.S. government to support the adoption and ‘meaningful use’ of certified electronic health records (EHR) under the Health Information Technology for Economic and Clinical Health Act (HITECH) to achieve specified improvements in care delivery.

The HITECH Act requires, among other interventions, that healthcare providers implement clinical decision support (CDS) systems as part of certified EHRs. CDS systems are tools designed to aid directly in clinical decision making, in which characteristics of individual patients are used to generate patient specific interventions, assessments, recommendations, or other forms of guidance that are then presented to a decision making recipient or recipients that can include clinicians, patients, and others involved in care delivery. The Centers for Medicare & Medicaid Services state that CDS represents one of the most promising tools to mitigate the ever-increasing complexity of the day-to-day care practice of medicine.

The effective use of a clinical decision support system means patients get the right tests, the right medications, and the right treatment, particularly for chronic conditions.

While the HITECH Act provides the general context for the establishment of the Library, a more recent Federal statute known as the Protecting Access to Medicare Act of 2014 (42 USC 1395m, “PAMA”), signed into law on April 1, 2014, created a sense of urgency for this effort and identified medical imaging evidence as the initial domain focus for the Library. Starting on January 1, 2017, PAMA requires healthcare providers to use approved CDS systems to consult specified appropriate use criteria when ordering certain advanced imaging procedures. PAMA defines the term appropriate use criteria as “criteria, only developed or endorsed by national professional medical specialty societies or other provider-led entities, to assist ordering professionals and furnishing professionals in making the most appropriate treatment decision for a specific clinical condition for an individual. To the extent feasible, such criteria shall be evidence-based.”

"The effective use of a clinical decision support system means patients get the right tests, the right medications, and the right treatment"

PAMA instructs the U.S. Secretary of Health and Human Services (HHS) to specify the first list of approved Appropriate Use Criteria by November 15, 2015. In specifying applicable appropriate use criteria the Secretary of HHS is required to take into account whether the criteria have stakeholder consensus, are scientifically valid and evidence based and are based on studies that are published and reviewable by stakeholders.

In response to this deadline the Harvard Medical School Library of Evidence, under the leadership of healthcare providers drawn from teaching hospitals, community hospitals and private medical practices, will initially focus on assembling and curating evidence related to the appropriate use of medical imaging, and will establish a national resource of appropriate use criteria meeting the definition in PAMA for medical imaging. The Library will seek approval for this resource by the Secretary of HHS under 42 USC 1395m (q)(2)(A), with an initial list of Appropriate Use Criteria that will be supplemented annually.


Literature Search

The recommendations regarding appropriate use of imaging used to populate the evidence library were derived from a variety of sources and a number of publishers. Sources were identified through a literature search of MEDLINE (1980-2016) through the Ovid interface using the text search terms, “decision rule”, “decision instrument”, “clinical algorithm”, “prediction rule”, “low-risk criteria”, “high-risk criteria”, “practice guideline”, and “clinical policy” and the MeSH terms “radiography”, “radiology/standards”, “ultrasonography”, and “decision support techniques”. Specific techniques for identifying clinical prediction studies described by Wong et al were used. This literature search was augmented by reviewing the reference lists of identified articles. Each source was then individually examined to determine its eligibility for entry into the evidence library. Approximately 85% were excluded from the final collection of evidence sources because they: 1) contained non-imaging- related recommendations, 2) were duplicative of recommendations published in a previous source, or 3) contained rules and scoring systems that had not been validated since 2000.

Imaging-related decision rules, prediction rules, and risk-assessment scores commonly utilized in current medical practice were also identified through medical calculator websites such as MDCalc (www.mdcalc.com) and QxMD (www.qxmd.com). Imaging-related clinical practice guidelines were identified through the National Guideline Clearinghouse and by searching the websites of various professional societies for published policies and statements. All relevant documentation for local best practice guidelines were also obtained from various healthcare institutions. Imaging-related AUC were obtained primarily from two public sources, the American College of Radiology Appropriateness Criteria database and the American College of Cardiology Appropriate Use Criteria list.

From all these recommendation sources, we prioritized grading recommendations that were included in the Centers for Medicare & Medicaid Services (CMS) priority clinical conditions, and recommendations from the Choosing Wisely campaign.


Grading Methodology

The Library’s evidence grading methodology is based on the 2009 version of the Oxford Centre for Evidence-based Medicine (CEBM) – Levels of Evidence and the U.S. Preventive Services Task Force (USPSTF) I-Scores. Recommendations chosen from professional society guidelines, local best practices and peer-reviewed literature are converted into Health-IT consumable clinical logic and then graded by two independent parties – a curator and a validating clinician. Once agreement on a final grade is reached, the full record will be made freely available to the public.


Roles and Responsibilities

  • Project Manager – In consultation with the Governing Council, the project manager translates individual pieces of evidence- from guidelines, local best practices or the literature- into clinical logic to be used in the grading process.  The project manager assigns evidence to appropriate curators and validating clinicians, coordinates the grading workflow and ensures data accuracy and integrity.
  • Curator – Librarians trained in the use of the CEBM and USPSTF methodologies grade each piece of assigned evidence, record the rationale for their decision and pass the work on to a validating clinician.
  • Validating Clinician – Physicians grade each piece of evidence completed by the curators, record the rationale for their decision and, when agreement is reached, finalize the grade for public access.

Training Materials

WHO WE ARE

Governing Council


David W. Bates, M.D., M.Sc.

David W. Bates, M.D., M.Sc.

Professor
Harvard Medical School
Chief Innovation Officer
Brigham and Women’s Hospital

Dr. Bates is an internationally renowned expert in using information technology to improve clinical decision-making, patient safety, quality-of-care, cost-effectiveness, and outcomes assessment in medical practice. A practicing general internist, Dr. Bates is Chief Innovation Officer at Brigham and Women’s Hospital in Boston where he is also Chief of the Division of General Internal Medicine. He is a Professor of Medicine at Harvard Medical School, and a Professor of Health Policy and Management at the Harvard School of Public Health, where he co-directs the Program in Clinical Effectiveness. He also serves as Medical Director of Clinical and Quality Analysis for Partners HealthCare. He has been elected to the Institute of Medicine, the American Society for Clinical Investigation, the Association of American Physicians and the American College of Medical Informatics, and is past chairman of the Board of the American Medical Informatics Association. He serves as external program lead for research in the World Health Organization’s Global Alliance for Patient Safety. He is the president of the International Society for Quality in Healthcare (ISQua) and serves as the editor of the Journal of Patient Safety.

Conflicts of Interest: None.


Keith Hentel, MD MS

Keith Hentel, MD MS

Associate Professor of Clinical Radiology
Executive Vice Chairman, Department of Radiology
Chair, Quality Assurance/Quality Improvement, Weill Cornell Imaging at NewYork-Presbyterian
Chair, Patient Access/Experience, Weill Cornell Physicians Organization
Weill Cornell Medicine
NewYork-Presbyterian Hospital

Dr. Hentel is a practicing emergency and musculoskeletal radiologist who founded the Division of Emergency and Musculoskeletal Radiology at NewYork-Presbyterian Hospital/Weill Cornell Medical Center. He has used his department and institutional leadership positions to lead the implementation of programs and technologies to improve practice efficiency, access to care, patient safety, and patient experience. These efforts include leading the implementation of clinical decision support (CDS) for Weill Cornell’s participation in the Medicare Imaging Demonstration Project as part of the Brigham and Women’s convener group. Combining his advanced degrees in Medicine and Engineering with his expertise in clinical operations, Dr. Hentel continues to focus his academic interests on the effective implementation and use of CDS. Dr. Hentel currently leads the multi-departmental multi-disciplinary team responsible for imaging appropriate use criteria at Weill Cornell, a designated qualified provider led entity under the regulations set forth by the Protecting Access to Medicare Act of 2014 (PAMA2014).

Conflicts of Interest: None.


Ramin Khorasani, MD, MPH - Chair

Ramin Khorasani, MD, MPH - Chair

Professor
Harvard Medical School
Distinguished Chair, Biomedical Informatics
Brigham and Women’s Hospital
Vice Chair, Department of Radiology

A practicing radiologist and researcher, Dr. Khorasani directs the Center for Evidence-Based Imaging. His work in the arenas of quality improvement, comparative effectiveness research and health care policy emphasizes using advanced health IT tools. A focus has been to understand patterns of utilization of diagnostic imaging tests, and then develop and implement IT-enabled interventions using evidence-based guidelines and decision aids to improve quality and reduce waste. Dr. Khorasani led the BWH convenership under the CMS-funded Medicare Imaging Demonstration (MID), the largest trial of imaging clinical decision support yet undertaken. The BWH convenership inlcuded more than 10,000 physicians across four institutions and three states and comprised more that 75% of the providers participating in the MID.

Conflicts of Interest: None.


Kenneth D. Mandl, MD, MPH

Kenneth D. Mandl, MD, MPH

Professor
Harvard Medical School
Chair in Biomedical Informatics and Population Health
Children’s Hospital Informatics Program
Boston Children’s Hospital
Center for Biomedical Informatics
Harvard Medical School

Dr. Mandl (Twitter @mandl) is Professor at Harvard Medical School and the Boston Children’s Hospital Chair in Biomedical Informatics and Population Health. Through scholarship intersecting epidemiology and informatics, Mandl pioneered use of IT and big data for population health, discovery, patient engagement and care redesign. Mandl leads the transformative SMART Platforms initiative to design the “app store for health" and is principal investigator of the Scalable Collaborative Infrastructure for a Learning Health System across Boston hospitals and nationally. Recognized for research and teaching, Mandl received the Presidential Early Career Award for Scientists and Engineers and the Clifford A. Barger Award for top mentors at Harvard Medical School. He was advisor to two Directors of the CDC and chairs the Board of Scientific Counselors of the NIH’s National Library of Medicine. His clinical training and experience is in pediatrics and pediatric emergency medicine. Dr. Mandl has been elected to multiple honor societies including the American College of Medical Informatics, American Society for Clinical Investigation, Society for Pediatric Research, and American Pediatric Society.

Conflicts of Interest: None.


Andrew Menard, JD

Andrew Menard, JD

Director of Business Development
Department of Radiology
Brigham and Women’s Hospital

Mr. Menard serves as part of the management team for the BWH Department of Radiology, advising the Department Chair and Vice Chairs on strategic issues, policy issues, network and business development and other issues. Working with Dr. Khorasani, Dr. Steven Seltzer and others, he has worked on the public policy implications of imaging clinical decision support (CDS) for over 10 years. He has worked with Dr. Khorasani to organize a consortium of academic centers to conduct research on imaging CDS, to plan and execute the strategy for the BWH convenership under the MID, and to organize responses from leading national stakeholders to the draft legislation that ultimately became law as the Protecting Access to Medicare Act (PAMA). Andrew is a member of the Association of Administrators in Academic Radiology and is active in the Academy of Radiology Research and Coalition for Imaging and Biomedical Research. He is a graduate of Harvard Law School and serves on the board of directors of several private companies engaged in research and development for life sciences.

Conflicts of Interest: None.


Executive Directors


Ali S. Raja, MD, MBA, MPH

Ali S. Raja, MD, MBA, MPH

Associate Professor, Harvard Medical School
Vice Chair, Department of Emergency Medicine
Massachusetts General Hospital

A practicing emergency physician and clinical informaticist, Dr. Raja’s research focuses on the appropriate – evidence-based – use of emergency department resources. His work over the past 10 years has focused on the evaluation of testing strategies, with an emphasis on the dissemination and implementation of best practice diagnostic protocols. He has developed quality measures endorsed by the National Quality Forum and chaired an American Medical Association Optimizing Patient Exposure to Ionizing Radiation subgroup. He currently serves as the chair of the American College of Emergency Physicians Trauma and Injury Prevention Section and the Society for Academic Emergency Medicine’s Constitution and Bylaws Committee, Program Committee, and Trauma Interest Group, and is a member of the Guidelines Committee of the Eastern Association for the Surgery of Trauma.

Conflicts of Interest: None.


David Osterbur, PhD, MLS

David Osterbur, PhD, MLS

Director of Public and Access Services
Francis A. Countway Library of Medicine
Harvard Medical School

David Osterbur acquired both a bachelor's degree in Biology and a Master's degree in Genetics from the University of Illinois at Champaign. He earned a Ph.D. in Genetics from the University of California at Berkeley, did postdoctoral studies at Indiana University in Bloomington and then was a research faculty member at the University of Kansas Medical Center. He obtained a Master's degree in Library Science from Simmons College in Boston. David worked as a senior information specialist for DuPont Pharmaceuticals, then moved back to academia as the Director of the Biological Laboratories Library at Harvard University.

Conflicts of Interest: None.


Directors of Clinical Informatics


Ivan Ip, MD, MPH

Ivan Ip, MD, MPH

With backgrounds in public health and medical informatics, Dr. Ip is a practicing general internist at Brigham & Women’s Hospital and an Assistant Professor at Harvard Medical School. His interests focus on the use of information technology to improve healthcare delivery and patient outcomes. He is board certified in internal medicine and clinical informatics, and has published extensively on imaging clinical decision support, applying evidence-based guidelines, and process redesign. Ip is a graduate of Cornell University Medical College. He completed his MPH from the University of California at Berkeley and received his informatics training at Stanford University Medical School and Harvard.

Conflicts of Interest: None.


Ronilda Lacson, MD, PhD

Ronilda Lacson, MD, PhD

Dr. Ronilda Lacson completed her PhD in Computer Science and M.S. in Medical Informatics degrees at the Massachusetts Institute of Technology. She completed her research fellowhip in Biomedical Informatics at the Brigham & Women's Hospital and Harvard Medical School (HMS) Biomedical Informatics Research Training (BIRT) program, after completing her MD degree from the University of the Philippines and her Internal Medicine residency training at Harbor Hospital Center in Baltimore, MD. She has grants and publications on implementation and evaluation of clinical decision support systems and knowledge representation of clinical guidelines. Her research spans a wide range of topics in informatics, and includes natural language processing and machine learning to extract structured and meaningful data from clinical narratives and medical records. She is currently an Assistant Professor of Radiology at HMS and Faculty at the Center for Evidence-Based Imaging and the HMS BIRT program.

Conflicts of Interest: None.


Director of Knowledge Translation


Louise Schneider, MD

Louise Schneider, MD

Dr. Schneider has been a faculty member of the Center for Evidence Based Medicine (CEBI) since its inception. An Associate Professor at Harvard Medical School, she attended the Boston University School of Medicine, is board certified in Internal Medicine and has been in practice as a primary care physician for over 30 years. She has had a longstanding interest in process and quality improvement and was a pioneer in the implementation of the radiology online order entry system at Brigham & Women's Hospital. As faculty member at CEBI Dr. Schneider co-directs the clinical content committee to incorporate evidence and guidelines into the process of online ordering to improve appropriateness of imaging.

Conflicts of Interest: None.


Curators


Paul Bain, PhD, MS

Paul Bain, PhD, MS

Reference and Education Services Librarian
Francis A. Countway Library of Medicine
Harvard Medical School

Paul Bain has served since 2004 as a Reference and Education Services Librarian at Harvard Medical School's Countway Library where he provides instruction and support for both bibliographic and bioinformatics resources. An expert searcher, he has collaborated in the production of many systematic reviews and meta-analyses, and he has served as an advisor and collaborator for Web-based information resources including the Alzheimer Research Forum, the Schizophrenia Research Forum, and, most recently, the Partners ID Images infectious disease case atlas. Paul completed his Ph.D. in Molecular and Cellular Biology at the University of Michigan in 1993 and his M.S. in Library Science at Simmons College in 2003.

Conflicts of Interest: None.


Jacqueline Cellini,MLIS, MPH

Jacqueline Cellini,MLIS, MPH

Reference and Education Services Librarian
Francis A. Countway Library of Medicine
Harvard Medical School

Jacqueline Cellini is Reference and Education Services Librarian at the Countway Library of Medicine at Harvard Medical School. With over 20 years of experience in academic libraries, she has expertise in information retrieval, planning and design of clinical research, and management of data for large-scale collaborative clinical studies. Previous positions include Data Management & Sharing Manager at the Lineberger Comprehensive Cancer Center at the University of North Carolina at Chapel Hill and Reference and Education Librarian at Northwestern Memorial Hospital in Chicago, Illinois. She holds a Master's Degree in Library and Information Science from Dominican University in Illinois and a Master's of Public Health Informatics from the University of Illinois at Chicago.

Conflicts of Interest: None.


Margo Coletti AMLS, AHIP

Margo Coletti AMLS, AHIP

Director of Knowledge Services
Beth Israel Deaconess Medical Center

Margo is Director of Knowledge Services, a division of Information Systems that provides knowledge-based and evidence-based information, resources, systems and services. She has a master’s degree in library science from University of Michigan and is a distinguished member of the Academy of Health Information Professionals. Special interests include evidence-based medicine (EBM), medical ethics, health literacy, open access journals and public policy, and knowledge management (KM).

Conflicts of Interest: None.


Jessica LaBrie, MSLS

Jessica LaBrie, MSLS

Librarian
Boston Children’s Hospital

Jessica LaBrie has been a librarian at Boston Children’s Hospital since 2014. She manages the Medical Library’s electronic resources and serials collections, and she also develops and maintains library websites and subject guides. Jessica holds a Master of Science in Library Science degree from the University of North Carolina at Chapel Hill, and she completed undergraduate work in anthropology and linguistics at Dartmouth College.

Conflicts of Interest: None.


Carol Mita, MS

Carol Mita, MS

Reference & Research Services Librarian
Francis A. Countway Library of Medicine
Harvard Medical School

Carol Mita has been a reference and education librarian at the Countway Library, serving Harvard Medical School and the Harvard School of Public Health, since 1999. She provides instruction and support for literature searching using PubMed and other bibliographic databases, as well as expert searching to support systematic reviews and meta-analyses. Her areas of subject specialty include public health, preventive medicine, and mental health. She has an M.S. in Library Science from Simmons College in Boston, and completed undergraduate work in nursing and zoology at the University of Wisconsin, Madison.

Conflicts of Interest: None.


Emily Schon, MLIS

Emily Schon, MLIS

Librarian
Boston Children’s Hospital

As a medical librarian at Boston Children’s Hospital, Emily Schon provides guidance and expertise in literature searching and systematic reviews on a variety of clinical and research topics, and regularly offers instruction and support to hospital personnel. She completed her undergraduate studies at Duke University. She has a Master’s Degree in Library and Information Science from the University of Pittsburgh, with a specialization in Health Resources and Services.

Conflicts of Interest: None.


Meredith Solomon, MLS, AHIP

Meredith Solomon, MLS, AHIP

Librarian
Brigham and Women’s Hospital

Meredith Solomon is Associate Librarian at Brigham and Women’s Hospital. She has over 20 years of experience working in academic, public and medical libraries. She provides instruction and support for literature searching and other bibliographic resources as well as supporting systematic reviews. Previous affiliations include: National Network of Librarian of Medicine/New England Region, University of Massachusetts Medical School, INOVA Healthcare, University of Maryland, Baltimore and National Library of Medicine, Associate Fellow. She holds a Master’s Degree in Library and Information Management from Emporia University in Kansas and is a senior member of the Academy of Health Information Professionals.

Conflicts of Interest: None.


Julia Whelan, MS

Julia Whelan, MS

Reference & Education Services Librarian
Francis A. Countway Library of Medicine
Harvard Medical School

Julia Whelan is Reference & Education Librarian at Countway Library, Harvard Medical School. She is a graduate of Swarthmore College, has a Master’s degree from Simmons College, and is a distinguished member of the Academy of Health Information Professionals. At Harvard Medical School, she teaches in the medical school curriculum. Areas of special interest are evidence-based practice, drug information, healthcare quality improvement and patient safety, natural medicines, palliative care, and access to health information for developing countries.

Conflicts of Interest: None.


Content Managers


Nicole Vetrano, MHA, BS, RT(R)

Nicole Vetrano, MHA, BS, RT(R)

Project Manager
Brigham and Women's Hospital

Nicole Vetrano is the current Project Manager for the Harvard Library of Evidence. Nicole attended Massachusetts College of Pharmacy and Health Sciences where in 2007 she earned her Bachelor’s degree in Radiologic Sciences. She further pursued her passion for the field by acquiring a Masters in Healthcare Administration degree at Suffolk University in 2015. Her extensive experience as a Radiologic Technologist, PACS Administrator and Clinical Instructor in Radiology have supplemented her avidity for advancing patient care and quality improvement initiatives with the use of information technology, evidence-based medicine and clinical decision support.

Conflicts of Interest: None.


Fritz Dement, MS

Fritz Dement, MS

Project Manager
Brigham and Women's Hospital

As project manager for the Harvard Library of Evidence, Fritz draws on his training as a librarian and experience as a manager. Fritz earned an MS in Library & Information Science from Simmons College while performing the duties of Information Specialist at Vertex Pharmaceuticals. After leading the research/corporate library for Vertex he shifted to academic medicine, becoming a clinical librarian at the NYU School of Medicine and serving as Assistant Director for User Services & Operations for the NYU Health Sciences Library.

Conflicts of Interest: None.


Laila Cochon, MD

Laila Cochon, MD

Research Fellow
Center for Evidence-Based Imaging
Brigham and Women's Hospital

Dr. Cochon completed her MD training at Universidad Iberoamericana UNIBE in Santo Domingo, Dominican Republic. She obtained a masters degree in Emergency Medicine and Critical Care, as well as Clinical Research in the Universitat de Barcelona, Barcelona, Spain where she is also pursuing her PhD in Translational Medicine. Currently, Laila is an informatics research fellow at the Center for Evidence-Based Imaging of Brigham and Women’s Hospital/Harvard Medical School where her research mainly focuses on the use of healthcare IT (electronic health record, clinical decision support systems, machine learning, natural language processing) and its use to further enhance evidence based practice as wells as quality and patient safety.

Conflicts of Interest: None.


Patricia Silveira, MD

Patricia Silveira, MD

Research Fellow
Center for Evidence-Based Imaging
Brigham and Women's Hospital

Dr. Silveira is a post-doctoral research fellow in Biomedical Informatics at the Center for Evidence-Based Imaging of Brigham and Women’s Hospital/Harvard Medical School. Her interests are medical education, best practice dissemination and improving the quality and efficiency of care delivery using innovative information technology solutions, such as clinical decision support and speech recognition systems. After completing her MD training in Brazil, she has been carrying out investigations in evidence-based medicine including imaging appropriateness criteria, quality, safety, as well as operational projects in Biomedical Informatics.

Conflicts of Interest: None.


Sowmya Varada, MD

Sowmya Varada, MD

Research Fellow
Center for Evidence-Based Imaging
Brigham and Women's Hospital

Dr. Varada completed her MD training at Sidney Kimmel Medical College in Philadelphia, PA and an informatics research fellowship at the Center for Evidence-Based Imaging of Brigham and Women’s Hospital/Harvard Medical School, where she served as the project manager and primary knowledge translator for the library of evidence project. Dr. Varada is pursuing radiology residency training at Columbia University Medical Center, and is primarily interested in understanding how healthcare information technology solutions (electronic medical record systems, clinical decision support, and automated extraction of structured data) can promote evidence-based practices and improve the quality of patient care.

Conflicts of Interest: None.


Trainees


Howard Yan

Howard Yan

Medical Student
Harvard Medical School

Howard Yan is currently a medical student at Harvard Medical School, after graduating from the California Institute of Technology in 2014. He is interested in the cross-section of medicine and IT, such as using clinical decision support to enhance evidence-based medicine.

Conflicts of Interest: None.