Automated Clinical Notes Annotation - Improving EHR Management and Clinical Decision Making by Global Business Media

SPECIAL REPORT

Automated Clinical Notes Annotation â&#x20AC;&#x201C; Improving EHR Management and Clinical Decision Making Artificial Intelligence Advances â&#x20AC;&#x201C; Towards Evidence-Based Medicine with Automated Medical Coding The Role of Electronic Health Records in Patient Care Improving Clinical Decision Making with Artificial Intelligence Artificial Intelligence-based Approaches to Clinical Text Mining Future Outlook

Sponsored by

Published by Global Business Media

Switchboard: +44 (0)1737 850 939 Fax: +44 (0)1737 851 952 Email: info@globalbusinessmedia.org Website: www.globalbusinessmedia.org

Dr. Sophie Laurenson (BSc Hons., Ph.D. Cantab.)

Business Development Director Marie-Anne Brooks

Introduction The History and Development of EHRs Core Components and Functions of EHRs Issues with Usability and interoperability of EHRs The Importance of Structured Data in EHRs Conclusion

Editor Dr. Sophie Laurenson (BSc Hons., Ph.D. Cantab.)

Improving Clinical Decision Making 9 with Artificial Intelligence

Publisher Kevin Bell

Senior Project Manager Steve Banks Advertising Executives Michael McCarthy Abigail Coombes Production Manager Paul Davies For further information visit: www.globalbusinessmedia.org The opinions and views expressed in the editorial content in this publication are those of the authors alone and do not necessarily represent the views of any organisation with which they may be associated. Material in advertisements and promotional features may be considered to represent the views of the advertisers and promoters. The views and opinions expressed in this publication do not necessarily express the views of the Publishers or the Editor. While every care has been taken in the preparation of this publication, neither the Publishers nor the Editor are responsible for such opinions and views or for any inaccuracies in the articles.

© 2020. The entire contents of this publication are protected by copyright. Full details are available from the Publishers. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical photocopying, recording or otherwise, without the prior permission of the copyright owner.

Dr. Sophie Laurenson (BSc Hons., Ph.D. Cantab.)

Introduction AI Methodologies: Knowledge-Based versus Data-Driven Approaches Application of AI to EHR Data Conclusion

Artificial Intelligence-based 11 Approaches to Clinical Text Mining Dr. Sophie Laurenson (BSc Hons., Ph.D. Cantab.)

Introduction AI-Based Approaches to Clinical Text Mining Text Mining in EHR Analytics Application of AI-Based Approaches in CDS Systems Conclusion

Future Outlook

Dr. Sophie Laurenson (BSc Hons., Ph.D. Cantab.)

Introduction Advances in Health IT Usability and Functionality Evaluation of EHR and CDS Systems Regulatory Applications of EHR and CDS Systems Conclusion

References 15

WWW.HOSPITALREPORTS.EU | 1

AUTOMATED CLINICAL NOTES ANNOTATION — IMPROVING EHR MANAGEMENT AND CLINICAL DECISION MAKING

Foreword I

N 1995, the Computer-based Patient Record Institute Work Group defined the Electronic Health Record (EHR) as “a virtual compilation of nonredundant health data about a person across a lifetime, including facts, observations, interpretations, plans, actions, and outcomes.”1 While this definition provides a worthwhile framework, current EHR systems contain data that is not standardised, well-structured or complete, limiting their utility in clinical practise2. The Designated Record Set proposed an alternative definition, that: “there is no one-size-fits-all definition for the designated record set. The healthcare organization must explicitly define both in a multidisciplinary team approach. Medical staff, for example, should provide guidance to ensure that patient care needs will be met for immediate, long-term, and research uses.”3. While this framework is more flexible and comprehensive, its widespread adoption by the medical community is yet to be realised. As a consequence, much of data that forms the “health tapestry” of patients is absent or inadequately recorded within EHRs4. As a result, crucial information that is logged as unstructured data, often in free text format, is intractable to analysis. One solution is to impose structure on data records, through codification to extract key data elements for subsequent analysis. Clinical decision support (CDS) systems can analyse medical data to provide actionable insights and inform clinical practise. Using human-directed methods to extract key data and develop knowledge-based clinical decision tools is both time- and cost-intensive. These approaches are limited by the process of knowledge acquisition and the lack of clear decision-making guidelines in some clinical fields. Recent advances in Artificial Intelligence (AI) have the potential to overcome these

challenges. Techniques such as Machine Learning (ML) and Natural Language Processing (NLP) may be combined to mine complex, unstructured data and identify hidden patterns that can be leveraged to inform clinical decision making. This report highlights some of the recent advances in AI-based approaches to EHR data analysis and the development of advanced CDS systems. The first article, contributed by Fujitsu Laboratories of Europe, describes the development of an AI-based CDS system through collaborative co-creation with healthcare providers. Further articles describe the challenges, recent advances and evidence supporting the application of AI methods to clinical decision making and text mining applications. The report concludes with a review of the outlook for EHR and CDS systems and the impact of AI across the healthcare industry. It is expected that advances in many areas of health IT systems could improve usability and functionality, reducing the burden on healthcare providers and improving patient outcomes and experiences.

Dr Sophie Laurenson Editor

Dr. Sophie Laurenson is a scientist and social entrepreneur. She obtained a Ph.D. in Oncology (Biophysics / Biochemistry) from the University of Cambridge in 2007 and has worked in industry and academia for 17 years. Currently, she is the Founder and Managing Director of Limeburners Bay International AG, developing medical technology for resourcelimited settings.

2 | WWW.HOSPITALREPORTS.EU

AUTOMATED CLINICAL NOTES ANNOTATION — IMPROVING EHR MANAGEMENT AND CLINICAL DECISION MAKING

Artificial Intelligence (AI) Advances – Towards EvidenceBased Medicine with Automated Medical Coding Fujitsu Laboratories of Europe The development of an AI-based CDS system through collaborative co-creation with healthcare providers.

Introduction The current generation of Electronic Health Record (EHR) systems has a mixed reputation, derived from the challenges associated with balancing administrative requirements against those of improved clinical decision-making. A myriad of studies exist detailing the difficulties that medical practitioners face, attributing the burden of data input and lack of system responsiveness to clinicians’ burn-out and stress. Valuable data is deeply buried amidst volumes of useless detail, making it difficult for clinicians to realise any meaningful value in return for the work they have invested. The quest to develop new solutions that support mission-critical activities, such as better care experiences and enhanced diagnostic and prescription performance, is the Holy grail for R&D organisations such as Fujitsu Laboratories of Europe. By applying advanced new Artificial Intelligence (AI)-based technologies, tangible improvements have been demonstrated to support critical clinical decision-making. Fujitsu Laboratories of Europe has a longstanding co-creation approach, working closely with innovation partners within the healthcare sector, including Madrid’s leading San Carlos Clinical Hospital. This approach has encompassed a cross-section of successful clinical projects over the past 4 years. The rationale behind this co-creation approach is to gain important insights into the challenges faced by the healthcare sector. This has already produced a number of important innovations affecting the workflow of medical professionals, including improving the accuracy

of clinical data and automating its digitalisation for hospitals, medical insurance companies and government agencies. The result of this co-creation activity has established a promising pathway towards a future generation of systems that reflects medical practitioners’ needs for fast access to patient data and associated structured medical data. The net result is an approach that in the very near future will support decision-making, enabling medical professionals to focus on patient care.

The Medical Perspective: The Challenges and the Goals In order to understand the issues more fully, Dr Julio Mayol, Chief Medical Officer of the San Carlos Clinical Hospital presents an overview of the principal requirements, and how the hospital has been working with Fujitsu Laboratories of Europe to identify new solutions. “We are constantly looking for new ways of improving clinical decision-making, and our work with Fujitsu Laboratories of Europe is helping us to realise important advances to improve efficiency. Most of the EHR systems available today do not fulfil the requirements of the doctor/ patient relationship. In fact, the use of EHR has been directly associated to clinician burn-out, as demonstrated by a number of studies. By far the best way for doctors to describe the individual complexity of a patient is through the use of free text narrative. It is almost impossible to capture much of the essential data using fixed input fields in a fixed form structure. Doctors are already under considerable time pressure during a consultation to carry out all WWW.HOSPITALREPORTS.EU | 3

AUTOMATED CLINICAL NOTES ANNOTATION — IMPROVING EHR MANAGEMENT AND CLINICAL DECISION MAKING

“WE ARE CONSTANTLY LOOKING FOR NEW WAYS OF IMPROVING CLINICAL DECISION-MAKING, AND OUR WORK WITH FUJITSU LABORATORIES OF EUROPE IS HELPING US TO REALISE IMPORTANT ADVANCES TO IMPROVE EFFICIENCY.” DR JULIO MAYOL, CHIEF MEDICAL AND INNOVATION OFFICER, HOSPITAL CLINICO SAN CARLOS

The current generation of Electronic Health Record (EHR) systems has a mixed reputation, derived from the challenges associated with balancing administrative requirements against those of improved clinical decision-making

of the required work – from understanding the current patient problem and previous medical history, making a diagnosis and prescribing the treatment. Current EHR software makes the process even more challenging, requiring doctors to spend a considerable amount of time in front of the computer. Medical coders are highly specialised doctors and nurses, tasked with reading doctors’ reports and annotating them with the corresponding medical terms in different ontologies, such as ICD10, SNOMED-CT etc. Despite the advances in medical codification systems, in most hospital a limited number of reports (for example, mostly discharge reports) are still processed and a limited number of concepts are extracted. This is where we need to make tangible progress doctors and decision makers need structured information in order to make accurate decisions. Until now, this information has not been available due to the difficulty in transforming the free text narratives into structured reports. With new technologies such as Fujitsu’s latest AI text mining technology, we can address these challenges

directly, and achieve tangible improvements to the clinical decision-making process.”

Evolving the Technology: AI Innovations in Natural Language Processing (NPL) As an important starting point, the priority is to create a new generation of tools that can transform free text narratives into structured reports, giving doctors the data on which to make more informed clinical decisions. Fujitsu Laboratories of Europe’s research has shown how AI can be applied as a key enabling technology to achieve this transformation, using Natural Language Processing Technology (NLP) as a foundation from which to extract structured information from clinical narratives. While major progress is being made at a rapid pace, there are still many fundamental challenges – principally the lack of available data due to patient privacy concerns, as well as the need for more sophisticated algorithms capable of addressing the extreme complexities of medical language.

FUJITSU LABORATORIES OF EUROPE’S AI HEALTHCARE SOLUTION ESTABLISHES PREDICTIVE ANALYSIS MODELS AT THE SAN CARLOS CLINICAL HOSPITAL MADRID

4 | WWW.HOSPITALREPORTS.EU

AUTOMATED CLINICAL NOTES ANNOTATION — IMPROVING EHR MANAGEMENT AND CLINICAL DECISION MAKING

INTERNAL WORKFLOW DETAILING FUJITSU’S APPROACH

The result is a state-of-the-art AI solution that deploys text mining techniques to automate the medical coding of non-structured medical texts, demonstrating up to 90% accuracy regarding the extraction of terms depending on the type of documents processed and the entities extracted. This has a number of important implications, including enabling medical professionals to gain access to patient information faster, thereby freeing up their time to focus on patient care. Using its proven NLP techniques, Fujitsu Laboratories and Fujitsu Laboratories of Europe’s solution can automatically extract structured information from clinicians’ free narrative text. Using deep learning, the solution can be retrained to adapt to specific contexts or a user’s individual needs. This enables additional flexibility compared to the limitations associated with codification systems that rely solely on complex linguistic rules in order to identify the correct terms from free text. The result is a high degree of accuracy, matched by the ability to extract a wider cross-section of relevant terms.

Behind the Scenes: Detailed Insight into the Technology Unlike previous generation technologies, Fujitsu’s AI text mining technology combines semantic knowledge and Deep Learning-based Natural Language Processing (NLP) in order to analyse medical notes and extract valuable data. First, an initial Biomedical Knowledge Graph (BKG) is defined to store different medical vocabularies and ontologies (such as ICD-9 or ICD-10). The nodes in the BKG represent entities in the ontologies with basic properties such as code and description, including the hierarchical relations between them. Next, the BKG is semantically enriched in an automatic manner with external resources such as medical vocabularies and abstracts from scientific literature databases. The enriched BKG is stored in an open-source property graph database that achieves high performance in reading and writing due to its scalable architecture and native graph storage.

A web-based interface allows users to explore the BKG and visualise the relationship between the different entities. This BKG is used to disambiguate the different entities and normalise them to a standard entity. For example, the word “Asthma” should be normalised to the proper ICD10 code, which might be different depending on the surrounding context. Following our previous example, before tagging the exact ICD code, it is necessary to detect that “Asthma” is a disease. This task is called “named entity recognition” and is performed using Deep Learning models created and fine-tuned using large bodies of anonymised clinical notes that are expanded using sophisticated text augmentation techniques. By combining Deep Learning and semantic technologies in this way, we can achieve high levels of accuracy in the detection of multiple medical entities.

The Ultimate Goal: Achieving Fully Automated Data Extraction Clinical decisions taken by medical professionals have profound implications for our lives. With life expectancy now averaging 85 years in developed countries, enormous advances have been made in improving longevity and quality of life. However, there remain many important challenges, including the impact on healthcare costs associated with an expanding ageing population. Governments are struggling to keep pace with the cost of maintaining current healthcare spending levels. Despite the many technological advances in recent years, a significant percentage of patient information remains buried in databases and cannot be used. At Fujitsu, our focus is on how to unlock this potential, using healthcare informatics to transform as much as 90% of patient information into structured, readable data that is accessible to both medical professionals and machines. This will enable a new level of co-operative interaction between doctors and clinical AIs, with the potential to transform the way that healthcare is delivered. As a comparison, imaging driving your car without WWW.HOSPITALREPORTS.EU | 5

AUTOMATED CLINICAL NOTES ANNOTATION — IMPROVING EHR MANAGEMENT AND CLINICAL DECISION MAKING

MEDICAL CODING SYSTEM: (ON THE LEFT) FUJITSU’S WEB TOOL WITH A CLINICAL NOTE AND ITS ASSOCIATED CODES; (ON THE RIGHT) A PART OF THE ENRICHED KG IN neo4j

The result is a state-ofthe-art AI solution that deploys text mining techniques to automate the medical coding of non-structured medical texts, demonstrating up to 90% accuracy regarding the extraction of terms depending on the type of documents processed and the entities extracted

power steering, GPS navigation, cruise control and increasingly self-driving functionality? Fujitsu Laboratories of Europe is pioneering new approaches to make this a reality, creating technology that transforms unstructured medical data into knowledge. Rather than creating generic NLP tools, we are focusing on understanding the subtle details of the medical language. For example, a symptom is something that a patient describes, whereas a sign is something that doctors can observe. Despite major advances in deploying Deep Learning, current AI tools cannot provide this level of fine detail and a new generation of algorithms is required. Fujitsu and other leading research teams around the world are working to create the tools needed for these vital next steps. Underpinning every aspect of patient care is the essential concept of trust. Traditionally this has been based on the transparent relationship existing between doctors and patients. For patients to accept a transformation of care based

on new automated and AI-assisted decisionmaking, they must trust the entire system and we have to build trust in the new tools we are developing. At Fujitsu, this involves a complete commitment to creating ethical AI algorithms that do not violate human rights. These new tools and algorithms need to be validated in large trials and widely reported to the full spectrum of interested parties. As a key part of this, AI algorithms must be explainable and easy to understand by nonAI experts. Realising digital transformation in healthcare relies on a fundamental process of co-creation and collaboration, involving many different multidisciplinary stakeholders including the medical profession, healthcare administrators, AI and analytics experts. Success relies on working together to tackle the most time and resource-consuming healthcare processes. With a realistic set of objectives, this can provide the platform from which to advance towards more ambitious projects.

Doctors are already under considerable time pressure during a consultation to carry out all of the required work – from understanding the current patient problem and previous medical history, making a diagnosis and prescribing the treatment. Current EHR software makes the process even more challenging, requiring doctors to spend a considerable amount of time in front of the computer

6 | WWW.HOSPITALREPORTS.EU

AUTOMATED CLINICAL NOTES ANNOTATION â&#x20AC;&#x201D; IMPROVING EHR MANAGEMENT AND CLINICAL DECISION MAKING

The Role of Electronic Health Records (EHRs) in Patient Care Dr. Sophie Laurenson (BSc Hons., Ph.D. Cantab.) Electronic Health Records (EHRs) contain heterogeneous data types from multiple sources, often recorded as a narrative in unstructured free text. Extracting structured data is necessary for analysis to enable improved clinical decision-making.

Introduction Electronic Health Records (EHRs) is a collective term used to describe clinical information systems that collect, store and present longitudinal patient data in a digital format. In recent years, healthcare providers have been encouraged to adopt EHRs to improve patient outcomes and safety, boost efficiencies and aid in clinical decision making. Attention is now diverting away from EHR adoption and implementation and moving toward realising the potential benefits of digital records.

The History and Development of EHRs The first examples of clinical information systems date back to the 1960s when discrete systems were implemented by several large healthcare providers. Reports published by the National Academy of Medicine in the 1990s promoted more widespread EHR adoption in the US healthcare system based on their potential to improve patient safety1,5. These reports also defined the core functionalities and barriers to adoption. The 1990s saw a global expansion in the adoption of EHRs as they were touted as an essential tool to increasing quality of care and improving patient outcomes. Today, most healthcare providers in high-income countries have implemented EHR systems and many organisations in low and middle-income countries have followed suit.

Core Components and Functions of EHRs EHRs include a range of data types from different sources including demographics, medical history, treatment regimen, allegies, immunization status, laboratory and pathology test results, medical images, vital signs and billing information. This data may be entered and stored in a variety of formats including both structured and unstructured data types. Information is entered by

physicians during patient consultations and may be amended by annotations or codifications and assimilated with data from other clinical sources at various timepoints6. Electronic data storage eliminates the need to locate paper records and improves the accuracy and legibility of data. It may also reduce the risk of data replication, reinforces the regular updating of records and decreases the risk of lost paperwork. One of the key benefits of EHRs is the timely delivery of patient data to the medical practitioner. Combining multiple types of clinical data can aid clinicians in decision making and patient risk stratification. Appropriate access to data can support healthcare providers to achieve a variety of goals including improved care coordination, disease prevention and management and patient monitoring or support outside traditional care settings. Increased visibility across healthcare agencies can also lead to improvements in cost effectiveness through reducing unnecessary procedures and allocating resources more efficiently.

Issues with Usability and interoperability of EHRs Since the widespread implementation of EHRs, several issues have surfaced surrounding the usability and interoperability of current commercial systems. A recent review of the available literature revealed that the quality and usability of EHR systems is commonly poor7. Several issues with EHR documentation have been identified. These include structural problems manifesting in convoluted workflows and documentation quality deficiencies. In particular, the use of free text fields to record clinical narratives are common in many areas of medical practise and have demonstrated to be prone to error. Another common structural problem is the lack of standardisation in EHR systems across different aspects of healthcare delivery leading to problems with interoperability8. WWW.HOSPITALREPORTS.EU | 7

AUTOMATED CLINICAL NOTES ANNOTATION — IMPROVING EHR MANAGEMENT AND CLINICAL DECISION MAKING

FUJITSU LABORATORIES’ CO-CREATION STRATEGY WITH LEADING HEALTHCARE PARTNERS IS HELPING TO UNLOCK VALUABLE PATIENT DATA AND EVOLVE EHR PLATFORMS.

Data recorded in EHRs is highly heterogeneous, containing various and often incompatible formats

Each of these issues has received significant attention in recent years and studies are underway to investigate interventions to improve EHR design and use9. One issue highlighted by physicians is the additional workload required to complete EHR documentation and reporting. The original purpose of EHR implementation was to improve billing processes and a significant proportion of the total time and financial cost associated with EHR-based encounters is related to billing10. Despite the exponential increase in the amount of data recorded in EHRs, physicians have reported a lack of actionable data that can be applied to patient care. A recent survey reported that 65 percent of providers do not have the ability to view and utilise all the patient data they need during a consultation. Furthermore, only 36 percent were satisfied with ability to integrate data from external sources such as laboratories or referrals. Systems that can extract and analyse clinical data are of significant value in realising the full benefits of digital patient data11. However, extracting data in a structured format that can be readily analysed is both time- and cost-intensive.

The Importance of Structured Data in EHRs Data recorded in EHRs is highly heterogeneous, containing various and often incompatible formats. The most common data formats include: • structured and terminology encoded data, • structured data with limited or no encoding, • unstructured machine-readable text data, • unstructured scanned text or images12. Many EHR data entries are recorded in narrative format, enabling providers to articulate opinions and impressions via free text fields or dictation. This allows physicians and clinicians to record a nuanced version of data, without negotiating a structured entry system. However, this freedom of expression creates pragmatic issues with accurately communicating information between 8 | WWW.HOSPITALREPORTS.EU

providers. Recording patient data as a narrative history can lead to comprehension issues between providers or specialisms that arise due to differences in jargon or terminology. One approach is to prospectively encode the data in a standardised format that can be subsequently analysed. Artificial Intelligence (AI)-based methods such as Natural language processing (NLP) can be employed to extract structured data from free text narratives. For EHRs to be useful in guiding clinical decision making, it is necessary to establish standards in data structure and display. Currently, many EHR vendors utilize the ICD-9/10 and CPT code standards to apply a structure to clinical data13. Further solutions could include common data elements, such as the North American Association of Central Cancer Registries and STandards for Oncology Registry Entry standards used by national cancer registries. Terminologies for semantic tagging and annotation, such as Systematized Nomenclature of Medicine − Clinical Terms, Logical Observation Identifiers Names and Codes, and RxNorm would also be beneficial for standardising data structures14. Applying new automated artificial intelligence algorithms to the process of encoding and annotating medical data would enable more patient data to be extracted and available for analysis.

Conclusion Many chronic diseases require a multidisciplinary approach to care, involving teams of providers from disparate clinical disciplines. The coordination of care activities is facilitated through clinical documentation in EHRs, which relies on a consistent communication method between providers and between provider and patient. Automated methods of retrospectively applying standardisation and structure to information through transcoding and extracting data are required to fully realise the benefits of digital patient data.

AUTOMATED CLINICAL NOTES ANNOTATION — IMPROVING EHR MANAGEMENT AND CLINICAL DECISION MAKING

Improving Clinical Decision Making with Artificial Intelligence (AI) Dr. Sophie Laurenson (BSc Hons., Ph.D. Cantab.) AI-assisted data analysis has the potential to generate clinically relevant insights in real time.

Introduction Artificial Intelligence (AI) can be described broadly as the use of computers to imitate the processes of the human brain. The term encompasses a variety of techniques and algorithms including Machine Learning (ML) and Natural Language Processing (NPL). AI methods can be employed in isolation to specific processes or in combination by applying different methods sequentially. The application of AI in healthcare dates to the early 1990’s, when established technologies were employed to build “knowledge-based systems” or “expert systems”. AI encompasses a range of tools that can leverage immense quantities of data to improve clinical decision-making processes, patient safety and quality of care decisions15. Early applications used symbolic approaches based on rules and knowledge to perform diagnoses and treatment recommendations in healthcare settings. Modern implementations of AI combine representational approaches with advanced computational algorithms. Clinical Decision Support (CDS) systems have recently received attention as applications that could profit from AI approaches16. AI has already found applications in several CDS systems to improve medical image analysis in pathology, radiology and ophthalmology17. In the future, AI has the potential to become more autonomous, enabling the automation of routine or specific clinical tasks18.

AI Methodologies: Knowledge-Based versus Data-Driven Approaches According to the Barcelona Declaration for the Proper Development and Usage of Artificial Intelligence in Europe16, AI methodologies can be separated into two essentially different categories: knowledge-based AI and datadriven AI. The definition of knowledge-based AI embraces “an attempt to model human

knowledge in computational terms, starting in a top-down fashion from human self-reporting of what concepts and knowledge individuals use to solve problems or answer queries in a domain of expertise, including common sense knowledge”, and “formalizes and operationalizes this knowledge in terms of software. It rests primarily on highly sophisticated but now quite standard symbolic computing technologies and has already had a huge impact”16. By comparison, data-driven AI methods are characterized by “starting in a bottom-up fashion from large amounts of data of human activity, which are processed with statistical machine learning methods […] in order to abstract patterns that can then be used to make predictions, complete partial data, or emulate human behaviour in similar conditions in the past. Data-driven AI requires big data and very substantial computing power to reach adequate performance levels”16. Knowledge-based AI methodologies are well established and have been applied to various healthcare and medical challenges. However, these approaches are principally humandeveloped and are less able to leverage large volumes of data to automatically build knowledge models. The processes of human-directed knowledge acquisition and formalisation require time and effort to complete, creating a bottleneck in the development process. In contrast, datadriven methodologies combine powerful computing architectures and machine learning techniques to accurately extract data features and identify hidden patterns.

Application of AI to EHR Data Recent research has found that knowledgebased approaches to developing CDS systems are constrained by both the human knowledge authoring process and a lack of clinical evidence in some fields8. However, data driven CDS platforms, based on the analysis of large volumes WWW.HOSPITALREPORTS.EU | 9

AUTOMATED CLINICAL NOTES ANNOTATION â&#x20AC;&#x201D; IMPROVING EHR MANAGEMENT AND CLINICAL DECISION MAKING

FUJITSU LABORATORIESâ&#x20AC;&#x2122; AI-ASSISTED DECISION-MAKING SOLUTIONS ARE BUILT ON ETHICAL, EXPLAINABLE AI ALGORITHMS - USING HEALTHCARE INFORMATICS TO TRANSFORM PATIENT INFORMATION INTO STRUCTURED, READABLE DATA.

The processes of human-directed knowledge acquisition and formalisation require time and effort to complete, creating a bottleneck in the development process

10 | WWW.HOSPITALREPORTS.EU

of EHR data, could be implemented to enable real-time decision support in healthcare settings. AI has been applied to data analysis and learning tools associated with patient EHRs19. Advances in computational power and AI algorithms is enabling the merger of EHRs with other sources of health data, including life science research data, clinical and pathology laboratory data, insurance and payer information and industry surveillance systems14. Furthermore, data collected from vital signs monitoring devices can be integrated to enable patient monitoring outside of traditional care settings. AI-assisted data analysis has the potential to generate clinically relevant insights in real time. The use of ML algorithms to generate predictive models has been increasing rapidly in recent years14. These studies have applied supervised, unsupervised and semi supervised learning algorithms to clinical data. In particular, the application of neural networks or support vector machines to medical imaging data have been used to support clinical decision making based on diagnostic imaging19. Sequential strategies applying NPL to mine free text data and ML to formulate predictive models have been used to automate the extraction of detailed prostate cancer data from clinical notes20, to identify local recurrences of breast cancer21 and to extract clinical information from discharge summaries22. Information derived from data analysis or established knowledge bases can be built into CDS systems to improve healthcare decisionmaking16,23. Examples of CDS platforms exist that incorporate generalised machine learning techniques as well as those programmed with rule-based systems, fuzzy logic, Bayesian networks and artificial neural networks24. AI now

combines a diverse range of computational logic methods to assist clinicians in making decisions. Studies have been implemented in various clinical settings to examine the effects on care delivery25,26. Studies have started to investigate the methodologies used to measure the clinical impact of AI. To date research has focussed on reviewing the impact of AI on patient outcomes in medication safety, inpatient experiences and psychiatry27. As opportunities emerge to connect data from the life sciences to clinical settings, individual variations can be accounted for as opposed to using only population averages or clinical trial populations. Evaluation of AI-based applications is critical as they begin to impact on patient care 19. These systems may pose risks to patients and caregivers due to potential malfunctions or unforeseen negative impacts on particular populations or health systems.

Conclusion The volume of data available in EHR systems provides a new challenge for researchers and healthcare providers in extracting relevant data to inform clinical practice. The application of AI and ML tools and techniques has the potential to improve both data extraction and analysis of complex concepts. Traditionally, clinicians have been limited to developing clinical algorithms based on a small number of clinical factors. The application of more advanced AI methods may allow for the generation of more robust predictive models using a wider range of variables. Predictive models could be integrated into future EHR platforms for clinical applications28-30, establishing powerful tools to aid the interaction between providers and patients.

AUTOMATED CLINICAL NOTES ANNOTATION — IMPROVING EHR MANAGEMENT AND CLINICAL DECISION MAKING

Artificial Intelligence (AI)-based Approaches to Clinical Text Mining Dr. Sophie Laurenson (BSc Hons., Ph.D. Cantab.) Human-driven codification and annotation of clinical text is time- and costintensive. AI-based approaches using Natural Language Processing (NPL) and Machine Learning (ML) can aid in entity identification and codification.

Introduction The information stored in electronic health records (EHRs) is often recorded as clinical narratives in free text format. This data is essentially unstructured, making the extraction and analysis of useful data features difficult. To access the wealth of clinical data stored in EHRs, order and structure must be imposed on free text data. This can be achieved by codification, applying rules and algorithms to unstructured information. Such processes can be implemented manually using human-driven codification systems based on established standards and rules-based guidance systems. Alternatively, Artificial Intelligence (AI)based approaches leveraging Machine Learning (ML) and Natural Language Processing (NLP) techniques can convert clinical documents into data elements that can be identified and analysed31.

AI-Based Approaches to Clinical Text Mining Some aspects of digital health information are recorded in structured formats. Even records that included unstructured free text fields can retain an overall structured format which aids in data extraction and analysis. Indeed, many computerized provider order entry (CPOE) systems use controlled vocabularies to avoid unstructured narrative. However, in instances where an obvious structure cannot be superimposed by annotation, AI-based approaches can be employed to detect and extract key terms from narrative data. NLP techniques are derived from computer science and computational linguistics disciplines. They include processing tasks such as named entity recognition, tokenisation and character gazetteer. Advanced NLP systems are built on the basis of word or phrase recognition mapping

to medical terms that represent domain concepts as well as understanding the relationships between concepts. Modern NLP methods employ a combination of rule-based and supervised machine learning approaches. Once developed, NLP techniques have the advantage of scalability and may be adapted and applied to a range of datasets.

Text Mining in EHR Analytics Narrative or free text data represent a large fraction of the patient data contained within an EHR. Examples of free text data include physicians’ notes describing physical examination, symptoms and medical interventions. This unstructured data poses a challenge for extraction by automated computer processing. Several frameworks have been developed to facilitate clinical language processing and link data to scientific and medical knowledge bases. Examples include the National Library of Medicine’s Unified Medical Language System (UMLS)32, General Architecture for Text Engineering (GATE)33, Unstructured Information Management applications (UIMA)34 and provided by the Open Health Natural Language Processing (OHNLP) Consortium35. Recently, several NLP techniques have been developed to facilitate information extraction from the free text in EHRs. Applications have included diagnostic classification, identifying patient cohorts, identifying co-morbidities and postoperative complications, reporting of notifiable diseases, syndrome surveillance, medication event extraction, adverse event detection and disease management24. NPL has been used in a number of proof-ofconcept studies to extract and analyse data from EHRs. Examples have included using NLP to extract cancer staging data36, formulating oncology treatment summaries37, automation WWW.HOSPITALREPORTS.EU | 11

AUTOMATED CLINICAL NOTES ANNOTATION â&#x20AC;&#x201D; IMPROVING EHR MANAGEMENT AND CLINICAL DECISION MAKING

Artificial Intelligencebased approaches leveraging Machine Learning (ML) and Natural Language Processing (NLP) techniques can convert clinical documents into data elements that can be identified and analysed

of patient risk stratification38, predict outcomes based on radiology reports39 and EHR data for oncology patients13. Unsupervised NPL methods, such as word2vec and CUI2vec-based approaches, have gained recent popularity. These approaches use contextual language information to make determinations on specific terms and their relationships and have been applied to parsing free text in pathology reports40,41. This trend marks a significant shift from previous approaches that based classifications on predetermined labels applied by content experts.

Application of AI-Based Approaches in CDS Systems CDS systems have demonstrated improved practitioner performance in approximately 60% of cases reviewed in literature42. A CDS platform can contribute to improved patient care in several ways: by automatically and proactively providing decision support within clinician workflows; providing recommendations and providing decision support at the time and location of decision making43. The key functions of CDS systems require understanding the context within a clinical narrative from which an entity is extracted and the relationships between entities. The explicit application of NLP tools for extracting clinical data for CDS applications has been lagging, due to the high levels of accuracy in entity identification that is required6. Extracted data may suffer from quality issues including validity and accuracy. As a result, the information contained within free text data have not been widely adopted within Clinical Decision Support (CDS) systems and represents a missed opportunity for improving healthcare. Concerns

over the accuracy of CDS systems have largely been due to the lack of explicit guidelines for decision making. These concerns could be remedied through the application of data-driven decision models utilising the wealth of clinical data captured in narrative format. A recent example applied NPL to combat poor provider compliance with guidelines surrounding cervical cancer screening to create a CDSS platform combining a free-text rule base and a guideline rule base to analyse Pap smear reports24. In this instance, the explicit decisionmaking guidelines and the well-structured format of Pap smear reports enabled NPL to extract key data for use in clinical decision support. For diseases that are not supported by clear evidence-based decision-making guidelines, AIbased text mining could offer potential to unlock hidden patterns within clinical data. However, the process by which these insights become integrated into routine clinical care is slow. AI algorithms must be developed in a manner which is consistent with legal and regulatory frameworks, requiring full transparency. If CDS systems were developed based upon AI approaches, it would require reliable accurate algorithm performance integrated within flexible and fast systems43. Passive NLP CDS applications require input by the user to generate output, whereas active NLP CDS applications leverage existing data to proactively push information to users as alerts or reminders. Advanced NLP engines could also be leveraged to provide computer-assisted coding solutions to aid in human-driven annotations and codification.

Conclusion Human-driven annotation and codification of unstructured free text data is a time- and costintensive process, often limited by deficiencies in clinical knowledge and prone to error. AI-based methods can be employed to identify entities within free text data, maps entities to concepts and relations between concepts. The process of developing automated codification systems differ depending on the goals of developing a system: to process free text for any task versus solving a specific clinical task. Current research suggests that NPL for developing clinical decision support systems may be achieved using tools developed for specific tasks and rule bases. However, as AI methods advance, the potential to develop more autonomous and generalised systems will impact on clinical practise.

12 | WWW.HOSPITALREPORTS.EU

AUTOMATED CLINICAL NOTES ANNOTATION — IMPROVING EHR MANAGEMENT AND CLINICAL DECISION MAKING

Future Outlook Dr. Sophie Laurenson (BSc Hons., Ph.D. Cantab.) Advances in AI-based techniques and changes in the regulatory environment have the potential to transform decision-making across different sectors and settings of the healthcare industry.

Introduction When discussed in the late 1980s, researchers and policy makers identified the goal of clinical decision support (CDS) was to ‘‘help health professionals make clinical decisions, deal with medical data about patients or with the knowledge of medicine necessary to interpret such data”44. Clinical decision support systems were then defined as ‘‘any software designed to directly aid in clinical decision making in which characteristics of individual patients are matched to a computerised knowledge base for the purpose of generating patient-specific assessments or recommendations that are then presented to clinicians for consideration”45. Four decades later, EHRs and CDS systems built for specialised applications have gained widespread acceptance by the clinical community. However, challenges in system usability and interoperability remain. It is expected that advances in Artificial Intelligence (AI) techniques will overcome some of these issues, to make EHRs and CDS systems more userfriendly, intuitive and accurate. While the principle user base for both EHRs and CDS systems is healthcare providers, the techniques employed to enable decision-making could be leveraged across different sectors of the healthcare industry.

Advances in Health IT Usability and Functionality Health IT systems have received a mixed reception from many users including physicians and healthcare providers. Recently, the U.S. Department of Health and Human Services (HHS) published an overarching strategy to reduce clinician burden for EHR data entry, improving usability and regulatory compliance 46. It is expected that this type of strategic guidance will expand globally, to improve the user-experience for many diverse health IT systems. Some CDS platforms operate only when clinicians proactively seek support. It is now recognised that clinical decision support is more useful when CDS systems can access EHR data directly and incorporate it into proactive analyses.

These systems can automatically trigger alerts to inform decision-making in real-time. EHRs contain a mixture of data types and formats, some of which may be found in structured data elements. However, in many cases the information that stimulate a CDS system is found in unstructured free text formats. In these instances, Artificial Intelligence (AI)-based tools such as Natural Language Processing (NPL) have the potential to extract the data required to actuate clinical decisions rules. Although significant progress has been made in these fields, most CDS systems based on AI-based text mining are built for specific applications. As developers gain more experience in building advanced algorithms for clinical datasets, these tools will become more generally applicable across different settings47. The integration of medical knowledge from life sciences and clinical evidence bases into EHRs will also improve algorithm development and accuracy of CDS systems. Some have suggested that the clinical documentation process may evolve to such an extent that medical records could be “wikified” in the future48.

Evaluation of EHR and CDS Systems As the functionality of EHR and CDS technologies improves and more data is captured in digital formats, the potential for risk also increases. As with all tools destined for clinical applications, digital technologies must be rigorously evaluated to ensure adequate risk management and to optimise patient outcomes and cost-effectiveness. The Good Evaluation Practice in Health Informatics (GEP-HI) framework provides a foundation for health IT evaluation projects49. The Statement on Reporting of Evaluation Studies in Health Informatics (STARE-HI) recommends guidelines for the design, implementation and publication of evaluation studies in medical informatics fields50. The generic PIET model presents a framework for deconstructing Population, Intervention, Environment and Transfer methods for analysis51. However, these WWW.HOSPITALREPORTS.EU | 13

AUTOMATED CLINICAL NOTES ANNOTATION — IMPROVING EHR MANAGEMENT AND CLINICAL DECISION MAKING

FUJITSU LABORATORIES IS DEVELOPING ADVANCED AI SOLUTIONS TO SUPPORT CRITICAL CLINICAL DECISION-MAKING.

The integration of medical knowledge from life sciences and clinical evidence bases into EHRs will also improve algorithm development and accuracy of CDS systems

14 | WWW.HOSPITALREPORTS.EU

standards were not necessarily developed with AI tools in mind and their applicability and completeness may not be optimised for systems that continuously develop. Two publications related to health IT evaluation methodologies specifically address techniques for assessing AI-based tools52. They provide guidance for evaluating potential biases that may by introduced during the development of algorithms and compromise assessments. Biases that result from differences between training data and real-world data particularly impact machine learning (ML) algorithms. In certain applications, including clinical studies, the difficulty in selecting appropriate training datasets has led to the development of ML platforms that do not function effectively in real-world settings47. Effective evaluation methodologies would help to mitigate against these risks and optimise AIbased techniques for real-world applications. Computerised platforms that affect clinical decision-making are now regulated in most major markets. Standalone systems are often termed “software as a medical device” (SaMD) and must undergo formal clinical investigations in a manner similar to medical devices. Under current regulations, the algorithms incorporated within regulated systems are fixed and many of the advantages of continuously evolving systems are lost. However, recently the U.S. Food and Drug Administration (F.D.A.) and the European Medicines Agency (E.M.A.) have released whitepapers detailing their proposals for more flexible future regulations of AI-based systems. Changes in the regulatory environment will stimulate further investment and development in the field.

Regulatory Applications of EHR and CDS Systems The interest in expanding the use of EHR data to inform regulatory decisions is growing globally. In late 2018, the U.S. F.D.A. announced the formation of the Framework for the Real-World Evidence Program, a team dedicated to using real-world data53. It is anticipated that are large proportion of real-world data would be derived from EHRs. This data could be used to inform regulatory decisions such as evaluating label expansions and designing post marketing surveillance strategies. One interesting potential application of realworld EHR data is the generation of “synthetic controls” for use in clinical trials. Using accurately annotated EHR data, patient populations could be identified to serve as proxies to the control arm of randomized controlled trials (RCTs). Some have advocated the expansion of this concept to derive an “average patient”, representing a synthetic control that could be applied across clinical trials. Although these approaches seem radical by current standards, they are likely to inform at least some aspects of medical regulations in the near future.

Conclusion Interest in using AI techniques to leverage clinical data from EHRs and other sources has increased exponentially in recent years. While these approaches have the potential to transform almost every aspect of medical practice, it is anticipated that the greatest impact will be in transforming the manner in which clinical services are delivered. Mining data for actionable insights has the potential to inform decision making across the healthcare continuum, spanning drug and device development, regulatory and market access initiatives and clinical care delivery.

AUTOMATED CLINICAL NOTES ANNOTATION — IMPROVING EHR MANAGEMENT AND CLINICAL DECISION MAKING

References: Medicine Io. The Computer-Based Patient Record: An Essential Technology for Health Care, Revised Edition. Washington, DC: The National Academies Press; 1997. 2 Wrenn JO, Stein DM, Bakken S, Stetson PD. Quantifying clinical narrative redundancy in an electronic health record. Journal of the American Medical Informatics Association : JAMIA. 2010;17(1):49-53. 3 Haugen MB, Tegen A, Warner D. Fundamentals of the legal health record and designated record set. Journal of AHIMA. 2011;82(2):44-49. 4 Weber GM, Mandl KD, Kohane IS. Finding the missing link for big biomedical data. Jama. 2014;311(24):2479-2480. 5 Institute of Medicine Committee on Quality of Health Care in A. In: Kohn LT, Corrigan JM, Donaldson MS, eds. To Err is Human: Building a Safer Health System. Washington (DC): National Academies Press (US) Copyright 2000 by the National Academy of Sciences. All rights reserved.; 2000. 6 Demner-Fushman D, Chapman WW, McDonald CJ. What can natural language processing do for clinical decision support? Journal of biomedical informatics. 2009;42(5):760-772. 7 Botsis T, Hartvigsen G, Chen F, Weng C. Secondary Use of EHR: Data Quality Issues and Informatics Opportunities. Summit on translational bioinformatics. 2010;2010:1-5. 8 O’Leary KJ, Liebovitz DM, Feinglass J, et al. Creating a better discharge summary: improvement in quality and timeliness using an electronic discharge summary. Journal of hospital medicine. 2009;4(4):219-225. 9 Otero Varela L, Wiebe N, Niven DJ, et al. Evaluation of interventions to improve electronic health record documentation within the inpatient setting: a protocol for a systematic review. Systematic Reviews. 2019;8(1):54. 10 Tseng P, Kaplan RS, Richman BD, Shah MA, Schulman KA. Administrative Costs Associated With Physician Billing and Insurance-Related Activities at an Academic Health Care System. JAMA. 2018;319(7):691-697. 11 Khor RC, Nguyen A, O’Dwyer J, et al. Extracting tumour prognostic factors from a diverse electronic record dataset in genito-urinary oncology. International journal of medical informatics. 2019;121:53-57. 12 Kim E, Rubinstein SM, Nead KT, Wojcieszynski AP, Gabriel PE, Warner JL. The Evolving Use of Electronic Health Records (EHR) for Research. Seminars in Radiation Oncology. 2019;29(4):354-361. 13 Hernandez-Boussard T, Kourdis PD, Seto T, et al. Mining Electronic Health Records to Extract Patient-Centered Outcomes Following Prostate Cancer Treatment. AMIA Annual Symposium proceedings AMIA Symposium. 2017;2017:876-882. 14 Manion FJ, Harris MR, Buyuktur AG, Clark PM, An LC, Hanauer DA. Leveraging EHR data for outcomes and comparative effectiveness research in oncology. Current oncology reports. 2012;14(6):494-501. 15 Coiera E. The fate of medicine in the time of AI. Lancet (London, England). 2018;392(10162):2331-2332. 16 Barcelona Declaration for the Proper Development and Usage of Artificial Intelligence in Europe. March 2017. 17 Yu KH, Kohane IS. Framing the challenges of artificial intelligence in medicine. BMJ quality & safety. 2019;28(3):238-241. 18 Yu KH, Beam AL, Kohane IS. Artificial intelligence in healthcare. Nature biomedical engineering. 2018;2(10):719-731. 19 Jiang F, Jiang Y, Zhi H, et al. Artificial intelligence in healthcare: past, present and future. Stroke and vascular neurology. 2017;2(4):230-243. 20 Leyh-Bannurah SR, Tian Z, Karakiewicz PI, et al. Deep Learning for Natural Language Processing in Urology: State-of-the-Art Automated Extraction of Detailed Pathologic Prostate Cancer Data From Narratively Written Electronic Health Records. JCO clinical cancer informatics. 2018;2:1-9. 21 Zeng Z, Espino S, Roy A, et al. Using natural language processing and machine learning to identify breast cancer local recurrence. BMC bioinformatics. 2018;19(Suppl 17):498. 22 Jiang M, Chen Y, Liu M, et al. A study of machine-learning-based approaches to extract clinical entities and their assertions from discharge summaries. Journal of the American Medical Informatics Association : JAMIA. 2011;18(5):601-606. 23 Peleg M, Tu S. Decision support, knowledge representation and management in medicine. Yearbook of medical informatics. 2006:72-80. 24 Wagholikar KB, MacLaughlin KL, Henry MR, et al. Clinical decision support with automated text processing for cervical cancer screening. Journal of the American Medical Informatics Association : JAMIA. 2012;19(5):833-839. 25 Nykanen P, Chowdhury S, Wigertz O. Evaluation of decision support systems in medicine. Computer methods and programs in biomedicine. 1991;34(2-3):229-238. 26 Clarke K, O’Moore R, Smeets R, et al. A methodology for evaluation of knowledge-based systems in medicine. Artificial Intelligence in Medicine. 1994;6(2):107-121. 27 Magrabi F, Ammenwerth E, McNair JB, et al. Artificial Intelligence in Clinical Decision Support: Challenges for Evaluating AI and Practical Implications. Yearbook of medical informatics. 2019;28(1):128-134. 28 Gensheimer MF, Henry AS, Wood DJ, et al. Automated Survival Prediction in Metastatic Cancer Patients Using High-Dimensional Electronic Medical Record Data. Journal of the National Cancer Institute. 2019;111(6):568-574. 29 Lindsay WD, Ahern CA, Tobias JS, et al. Automated data extraction and ensemble methods for predictive modeling of breast cancer outcomes after radiation therapy. Medical physics. 2019;46(2):1054-1063. 30 Nakatsugawa M, Cheng Z, Kiess A, et al. The Needs and Benefits of Continuous Model Updates on the Accuracy of RT-Induced Toxicity Prediction Models Within a Learning Health System. International journal of radiation oncology, biology, physics. 2019;103(2):460-467. 31 HealthITSecurity. The difference between big data and smart data in healthcare. HealthIT Anal. 2016. 32 Humphreys BL, Lindberg DA. The UMLS project: making the conceptual connection between users and the information they need. Bull Med Libr Assoc. 1993;81(2):170-177. 33 (GATE) GAfTE. https://gate.ac.uk/. Accessed February 2020. 34 (UIMA) UIMa. https://uima.apache.org/. Accessed February 2020. 35 (OHNLP) OHNLP. http://www.ohnlp.org/. Accessed February 2020. 36 Warner JL, Levy MA, Neuss MN, Warner JL, Levy MA, Neuss MN. ReCAP: Feasibility and Accuracy of Extracting Cancer Stage Information From Narrative Electronic Health Record Data. Journal of oncology practice. 2016;12(2):157-158; e169-157. 37 Warner JL, Anick P, Hong P, Xue N. Natural language processing and the oncologic history: is there a match? Journal of oncology practice. 2011;7(4):e15-e19. 38 Gregg JR, Lang M, Wang LL, et al. Automating the Determination of Prostate Cancer Risk Strata From Electronic Medical Records. JCO clinical cancer informatics. 2017;1. 39 Chen PH, Zafar H, Galperin-Aizenberg M, Cook T. Integrating Natural Language Processing and Machine Learning Algorithms to Categorize Oncologic Response in Radiology Reports. Journal of digital imaging. 2018;31(2):178-184. 1

WWW.HOSPITALREPORTS.EU | 15

AUTOMATED CLINICAL NOTES ANNOTATION — IMPROVING EHR MANAGEMENT AND CLINICAL DECISION MAKING

Gao S, Young MT, Qiu JX, et al. Hierarchical attention networks for information extraction from cancer pathology reports. Journal of the American Medical Informatics Association : JAMIA. 2018;25(3):321-330.

Yala A, Barzilay R, Salama L, et al. Using machine learning to parse breast pathology reports. Breast cancer research and treatment. 2017;161(2):203-211.

Garg AX, Adhikari NK, McDonald H, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. Jama. 2005;293(10):1223-1238.

Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ (Clinical research ed). 2005;330(7494):765.

Shortliffe EH. Computer programs to support clinical decision making. Jama. 1987;258(1):61-66.

Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. Jama. 1998;280(15):1339-1346.

(HHS) USDoHaHS. Strategy on Reducing Burden Relating to the Use of Health IT and EHRs. 2020; https://www.healthit.gov/topic/usability-andprovider-burden/strategy-reducing-burden-relating-use-health-it-and-ehrs.

Wiens J, Saria S, Sendak M, et al. Author Correction: Do no harm: a roadmap for responsible machine learning for health care. Nature Medicine. 2019;25(10):1627-1627.

Warner JL, Smith J, Wright A. It’s Time to Wikify Clinical Documentation: How Collaborative Authorship Can Reduce the Burden and Improve the Quality of the Electronic Health Record. Academic medicine : journal of the Association of American Medical Colleges. 2019;94(5):645-650.

Nykanen P, Brender J, Talmon J, et al. Guideline for good evaluation practice in health informatics (GEP-HI). International journal of medical informatics. 2011;80(12):815-827.

Brender J, Talmon J, de Keizer N, Nykanen P, Rigby M, Ammenwerth E. STARE-HI - Statement on Reporting of Evaluation Studies in Health Informatics: explanation and elaboration. Applied clinical informatics. 2013;4(3):331-358.

Schloemer T, Schroder-Back P. Criteria for evaluating transferability of health interventions: a systematic review and thematic synthesis. Implementation science : IS. 2018;13(1):88.

Choi M. Handbook of Evaluation Methods for Health Informatics. Healthc Inform Res. 2011;17(4):276-277.

Gottlieb S. Statement from FDA Commissioner Scott Gottlieb, M.D., on FDA’s New Strategic Framework to Advance Use of Real-World Evidence to Support Development of Drugs and Biologics. In:2019.

16 | WWW.HOSPITALREPORTS.EU

Healthcare Professionals Make More Effective Evidence Based Clinical Decisions With Hospital Reports

For the past decade, Hospital Reports has been helping healthcare professionals, within both public and private hospitals, to improve patient outcomes Our Special Reports provide readers with an unparalleled depth of information on specialist subjects, which receive limited coverage in the mainstream medical media. Each report is designed to help healthcare professionals to make more effective treatment decisions, by providing a unique mix of: • Subject specific clinical information • Insight and knowledge from internationally recognised key opinion leaders • Clinical guidelines • Independent data and analysis • Unbiased editorial content

subscriptions@globalbusinessmedia.org www.globalbusinessmedia.org