Evaluation of innovative health IT applications: Importance of usability studies in hospital settings

Transcription

1 CIC-IT 807 Biocapteurs et e-santé Evaluation of innovative health IT applications: Importance of usability studies in hospital settings Marie-Catherine Beuscart-Zéphir, Régis Beuscart Inserm CIC-IT - Evalab, University Hospital Lille, France

2 Part One Why evaluate?

3 Why evaluate? Scientific Objectives Clinical Objectives Economic Objectives At the end: commercialization, payment, reimbursement

4 Innovative device CE Marking

5 Innovative device Notified Body CE Marking

6 Innovative device CE Marking Can be sold to hospitals HC organisations patients

7 For CE Marking: follow the rules The Company: Is responsible for the conception, manufacturing, and labeling of a product, for the norms and standards. Notified Body controls the company, verifies the dossier, and realizes the necessary controls Competent National Agency Controls the notified bodies, Can perform complementary controls Disseminate the results CE Mark allows for commercialization, but not for reimbursement

8 More Evaluations for payment or reimbursement Pre-clinical studies: In silico modeling, In vitro studies In vivo trials on small or large animal Usability Studies How the device (procedure) can be used? Integrated? Clinical Trials (if possible blind and comparative clinical trials)

9 Innovative device CE Marking File for device reimbursement Payment through Reimbursement

10 Innovative device CE Marking File for device reimbursement Payment through Reimbursement Certification by the National Agency Reimbursement of the device

11 Innovative device File of evaluation of the device Payment through DRG CE Marking File for device reimbursement Payment through Reimbursement Certification by the National Agency Reimbursement of the device

12 Innovative device File of evaluation of the device Payment through DRG CE Marking File for device reimbursement Payment through Reimbursement Certification by the National Agency Reimbursement of the device

13 Innovative device File of evaluation of the device Inclusion of the device In a DRG DRG Payment Payment through DRG CE Marking File for device reimbursement Payment through Reimbursement Certification by the National Agency Reimbursement of the device

14 The payment or the reimbursement of devices is a long and complicated process 2-5 years Needing different levels of evaluation Clinical validation by good clinical trials becomes mandatory Difficult for SMEs, start-ups, even international companies.

15 In the ICT domain Current examples: Telemedicine Tele-radiology Tele-cardiology Communicating pace-makers and defibrillators Computer-added surgery

16 CIC-IT: Technology Innovation Centers Improve transfer of biomedical technologies from labs to industrial companies Promote the cooperation between academic hospitals and industrial companies for innovation and evaluation. Provide evaluation methodologies and participate to the validation of innovative devices and procedures Contribute to translational research, preclinical and clinical studies. Contribute to the training of students and physicians in the domain of Healthcare Technologies

17 CIC-IT: Technology Innovation Centers Provide evaluation methodologies and participate to the validation of innovative devices and procedures Objectives: Improve the methodology and the quality of: Technical validation and pre-clinical studies -> CE Marking Clinical trials : payment / reimbursement of innovative technologies in healthcare Certification of procedures or ICT Applications

18 City specialization 8 CIC-IT In academic hospitals Besançon Micro-nanotechnologies Lille Bordeaux Grenoble Biomaterials Assisted Surgery Brest Rennes Paris - Garches Nancy Lille Biosensors and e-health Nantes Tours Dijon Besançon Nancy NMR Imaging Paris Rennes Tours Devices for handicapped Cardiologic devices Ultrasonography Bordeaux Toulouse Lyon Grenoble Montpellier Marseille

19 Innovation Usage Biosensors for monitoring, telemedicine, e-health Technological validation and clinical evaluation Usability and acceptance of ICT by Healthcare Professionals and Patients Take into account users constraints in the design of the application Valuation Scientific - Economic

20 The Evalab Human Factors for the optimization of healthcare work systems and usability of healthcare IT applications Academic hospital of Lille (3000 Beds) and University / Faculty of Medicine: research and consultancy projects Accredited by the Ministry of Health and the INSERM (national agency) for evaluation and optimization of work systems and IT applications in healthcare

21 Part two Illustration The Metrodoloris case study

22 The project Research: a new index to support the monitoring of the patient s pain during anesthesia; a new monitor to visualize the index Clinical studies: positive results European Patent, gold medal of the 2009 Innovation Awards from the French Ministry of Research Creation of SME, CE marking as next objective Evaluation questions: Demonstrate the benefits for the patients (clinical trials) Usability and acceptance? Usability file for CE marking

23 ANALGESIA MONITORING Gold medal of the Ministry of Research Innovation Awards 2009 Société en création

24 The problem General anesthesia (GA): hypnotic drugs (e.g.. Propofol) analgesic drugs (e.g. opioid) Reliable index supporting the monitoring of the depth of hypnosis (BIS index) But only few and unreliable ways to assess the analgesia/nociception balance in an unconscious patient hemodynamic reactivity (increase of heart rate or systolic blood pressure, sweat, pupils diameter change) Objective of the project = establish a reliable index supporting the analgesia / nociception monitoring

25 Analgesia / Nociception monitoring system General principle: Analgesia / Nociception monitoring system based on instantaneous heart rate variability analysis. Analgesia / Nociception Index (ANI) related to the effect of pain on the autonomous nervous system.

26 EKG acquisition Heart rate computation ANI Computation Visualization interface

27 The monitoring system 1st prototype of the monitoring system Prototype for developing and tuning the index (ANI) Visualization of the index

28 Benefits for the patients?

29 Basic principle Patients recover faster when anesthesia is appropriate Avoid excessive doses of analgesic drugs Avoid pain during unconscious period

30 Pre-clinical trials Study the evolution of the index according to: The time of occurrence of painful stimulations The level of anesthesia: Deep Analgesia (DA) = appropriate Light Analgesia (LA) = insufficient assessed through hemodynamic reactivity, defined as an increase of heart rate or systolic blood pressure of more than 30% from the baseline Anesthetists actions to correct the anesthesia level (boluses of opioid)

31 Video

32 The pre-clinical trial N = 39 patients undergoing surgery with General Anesthesia Anesthesia protocol: Propofol using a DiprifusorTM (TCI-Diprifusor, Zeneca Ltd,) Propofol dose is continuously adapted (target [40-60]) to the measured Bispectral Index (BISTM, Aspect A-2000 XPTM v3.31) Opioid (Remifentanyl) administered at induction with Propofol; additional boluses in case of hemodynamic reactivity 2 periods analyzed : DA deep analgesia = adequate LA Light Analgesia = insufficient

33 Preliminary results ANI index ** p<0.01 : effect DA vs LA (anova) ANI decreases significantly in case of painful stimulation on anesthetized patients 10mn before visible patient s reactions provoking anesthetist s actions (boluses)

34 Benefits for the patient? Promising results of pre-clinical trial Clinical trials in progress orthopedic surgery, pediatric surgery, neurosurgery, obstetrics On this basis: patent, innovation award, creation of an SME, CE marking file. Impact on outcomes such as rapidity of recovering to be evaluated in phase B trials

35 Users, usage, usability and acceptance

36 Usability evaluation Analysis of the work system: Existing knowledge (Anesthesia Clinical Information System) Completed by focused observations Usability inspection of the prototype On-site usability tests (portable labs)? Focus groups?

37 Usability inspection In-lab evaluation (system installed in lab, standalone, fake database, simulations) Ergonomics criteria (INRIA)

38 Ergonomic criteria (INRIA) 1. Guidance a. Prompting b. Grouping/Distinction of Items c. Immediate Feedback d. Legibility 2. Workload a. Brevity b. Information Density 3. Explicit Control a. Explicit User Action b. User Control 4. Adaptability a. Flexibility b. User Experience 5. Error Management a. Error Protection b. Quality of Error Messages c. Error Correction 6. Consistency. 7. Significance of Codes 8. Compatibility

39 Usability inspection 1st iteration on the computer-prototype: Intermediary tool: Objective = optimize the monitor under development Quick and dirty evaluation Two ergonomists No severity rating Frequent debriefings with developers

40 Results of first usability inspection 27 problems examples

41 Legibility

42 Workload / information density (time scale in seconds / length of anesthesia)

43 Guidance, prompting (zone for text entering) Workload (documenting free text during anesthesia) Guidance (click on ajouter to save)

44 Results of first usability inspection Mostly HCI problems Question: who is the intended user? Researcher / anesthetist in charge of documenting the events in order to validate the index Anesthetists during routine anesthesia? Problem with the model of work integrated in the system

45 Second usability inspection: prototype monitor Problems with plugging and starting the monitor, initial erroneous values, discrepancy with values on other monitors (cardiac frequency) HCI problems

46 Guidance / consistency: No unit mentioned for the scales, while the curves are dynamically generated

47 Error Management / Error Correction Once entered, an event cannot be modified When several events are entered in a row, they are not visible

48 Conclusion on usability inspections Continuous improvement of the HCI, evolution of the product

49 1st prototype of the monitoring system 2nd prototype of the monitoring system First prototype realized for index development and pre-clinical trial EKG Sensor

50 Other usage related problems Ambiguity of the model of work integrated in the product: Monitoring the ANI in routine anesthesia Continuous validation of the index, extended clinical trials The work environment of the anesthetists is already overcrowded with monitors of all kinds: Duplicate information (e.g. cardiac frequency, EKG) Implementation of an anesthesia CIS integrating the information form various monitors

51

52 Acceptance problems Not all anesthetists are convinced it is necessary to monitor the analgesia level Senior anesthetists junior anesthetists Additional workload EKG duplicate, specific EKG sensors required Suggestions: retrieve the EKG signal from existing monitor when available Technically possible Economically unfeasible Integrate the index in the CIS Limited space To be discussed with a board of anesthetists Further research required on HF questions

53 Usability file for CE marking? Collateral standard on usability for IEC for medical electrical equipment Related to: ISO 9241(10-17) (ergonomics requirements for office work with visual display terminals) ISO (Human-centered design process for interactive systems) ISO (ergonomics of human-system interactions, human- Centered lifecycle process descriptions) CIF (Common Industry Format) for reports In the present case: mostly usability evaluation of the final product (usability inspection and tests) HCI OK, but fundamental problems remain: to which extent do we reflect those problems in the report?

54 Innovative device File of evaluation of the device Inclusion of the device In a DRG DRG Payment Payment through DRG CE Marking File for device reimbursement Payment through Reimbursement Can be sold to Hospitals HC organisations Certification by the National Agency Reimbursement of the device

55 Conclusion Next challenge: integration of the usability dimension in the certification process of e- prescribing applications incorporating CDS functions in hospital setting International cooperation in defining the protocol?