Deliverable 4.4 Phase 3: evaluation

Transcription

1 FP Framework Programme (FP) 7 ICT Patient Guidance Service (PGS), safety and healthcare record information reuse Combination of CP & CSA Work Package: WP4 Evaluation Deliverable 4.4 Phase 3: evaluation Version: 1.0 Status: Draft Date of Issue: 31 March 2017 File name: DECIPHER_D4.4 Phase 3 Evaluation Report_V1 0 Page 1 of 18

2 Document Information Full title Distributed European Community Individual Patient Healthcare Electronic Record Project Number FP7 ICT Acronym DECIPHER Start date Project February 1st, 2012 January 25th, 2013 Duration 48 months Project AQuAS - Agència de Qualitat i Avaluació Sanitàries de Catalunya (SPAIN) Coordinator Project URL EU Project Officer Jaakko Aarnio Date of delivery Contractual: 31/03/17 Actual: 3/04/2017 Nature Report Dissemination Consortium (CO) Level Responsible Author Vincenzo Alberto Vella vincenzo.vella@gencat.cat Partner: AQuAS Phone: Document History Date Version Author Change Status 1/03/ Vinzenzo Vella 31/03/ Cari Almazán Authors Name Partner Main author Vinzenzo Vella AQuAS Co-authors Cari Almazán AQuAS Keywords Abstract (for dissemination) Evaluation; PCP, design; diabetes; MOR; IOR The present documents aims at describing the assessment process that was implemented during DECIPHER PCPC Phase 2. The document s constituted two main areas: one related to the monitoring activities, related to the tasks and responsibilities of the Monitoring Team; one related to the evaluation of the proposals conducted by the Expert Board, which resulted in the selection of the six proposals which accessed Phase 3. Distribution List Date Issue E- mail 03/04/2017 Report to DECIPHER Team 03/04/2017 Report to EC and Project Officer jaakko.aarnio@ec.europa.eu Page 2 of 18

3 Index 1 Introduction 4 2 Phase 3 Executive Summary 4 3 Objectives 4 4 Phase 3 evaluation and monitoring activities Monitoring activities Evaluation activities 7 5 Results Evaluation results from the monitoring activities The Monitoring Outcome Report of Nextage s proposal highlighted: The Monitoring Outcome Report of E-Results s proposal highlighted: Evaluation results from the Expert Board s activities Optional comments on Assessment of Awarding criteria for Nextage SL are: Optional comments on Assessment of Awarding criteria for E-results are: Optional comments on Assessment of Awarding criteria for Gnomon Ld. are: 18 Page 3 of 18

4 Introduction DECIPHER PCP is a project that aims at challenging companies and developers in designing mobile technologies that will provide an updated view of personal health data and a tool for long-term conditions self-management. Support for cross-border health is a specific design target for these technologies. DECIPHER PCP is making use of the Pre-Commercial procurement (PCP) approach to steer the development of novel technological solutions from early R&D stages to the performance of the Proof of Concepts of the resulting 1services. Through this process the consortium is aiming to align the supply side with the demand side and at the same time improving quality and efficiency of DECIPHER procuring authorities healthcare services through the adoption of technologies co-created with the industry. DECIPHER PCP process will take place over 3 Phases: Design (Phase 1), Prototype (Phase 2) and Proof fo Concepts (Phase 3). Prior to PCP, a Need Assessment Phase (Phase 0) was conducted to identify end-users needs (patients and healthcare professionals) and to define functionalities of innovation. The present document aims at describing the activities conducted by DECIPHER PCP Consortium for conducting evaluation activities planned for Phase 3 and for achieving the objectives set for task T4.4 Phase 3: Evaluation Report. Phase 3 Executive Summary 2 During Phase 3- Proof of concept, evaluation process will assess proposals received from selected Bidders of Phase 2.. During this Phase, those involved in the Monitoring Team and the Expert Board adopted the evaluation framework and monitoring tools defined in the project preparatory stages in order to increase the alignment between the Bidders and the project expected outcomes and to insure an objective and impartial evaluation of the proposals. The two process, the evaluation and the monitoring, have been designed with the aim of allow for an interaction between the two. In particular, the work of the Monitoring Team will produce information that will support the evaluation of the proposals. This information is generated within the context of the proposals test conducted through end users (patients and healthcare professionals).. At Phase3, the evaluation process led to the selection of 3 proposal (within the total of 6) to access Phase 3. The selection relied on the awarding criteria included in the framework presented ion the Invitation to Tender (D2.2, 3 Annex 4). The 3 selected proposals resulted as those with the best overall performance and those that were more appreciated from involved end users. Objectives DECIPHER PCP s evaluation process aims at adapting the assessment frameworks used in the innovative procurement setting to the projects specificities, both under the clinical and the operational point of view. This means that in DECIPHER PCP the typical pre-commercial procurement settings to the nature of the technology the project aims at generating. This could be described as a consumer technology, helping clinicians and patients with type II diabetes in their life and work. For this reason, DECIPHER PCP s evaluation process is characterised by a strong focus on involving potential end users in each Phase, in order to provide developers and evaluators with information on use experience that may serve for the proposals development and their assessment. Phase 3 evaluation includes activities of both the Expert Board and the Monitoring Team, which will contribute to the testing of the framework, generating indication for its development for future application and contributing to the project s Lessons Learned. Page 4 of 18

5 Before the delivery to the Experts Board, the proposals have been submitted in a preliminary version to the Monitoring Teams, which were in charge of conducting an assessment in order to review the progress of each proposal and provide further guidance for development before the final delivery. The activities of the Monitoring Teams are very well described by the below graph, which also outlines the flow of activities between the Teams and the Bidders. Figure 1 - Monitoring Team workflow Bidders Provide: Interim Outcome Report (IOR) Monitoring Team Provide: Monitoring Outcome Report (MOR) Bidders Review and implement recommendations The Interim Outcome Report (IOR) is a document submitted by the Bidders to inform the Monitoring Team members regarding to which extent the developed technology goes beyond the state of the art, how it comply with each Phase expected outcome of development and its commercial and technical feasibility. The Monitoring Outcome Report (MOR), instead, is a document produced by consensus of all members of the Monitoring Team in which it is assessment progress of the bid and any recommendation the Monitoring Team considers mandatory to ensure their good progress. Results of the assessment progress can be classified into: good, acceptable, unsuccessful and unsatisfactory. Definitions of these categories of assessment are: Good progress: Bidders proposed solution and current achievements meet the objectives of the Phase and no recommendations are needed. Acceptable progress: Bidders proposed solution and current achievements are aligned with the objectives of the Phase but recommendations are required. Unsuccessful progress: Bidders proposed solution and current achievements reveal that the proposed technologies either do not go beyond the state of the art or lack of technical and commercial feasibility. However, where the Monitoring Team considers the aforesaid solution as a reversible one, it may make the recommendations it considers appropriate. Otherwise, when the Monitoring Team understands that the solution is not reversible, it shall recommend either the exclusion of the Bidder or not having him invited to the next phase. Unsatisfactory progress: Bidders proposed solution and current achievements do not comply with the contractual commitments. Nevertheless, as above, when the Monitoring Team considers the solution as a reversible one, it may make the recommendations it considers as appropriate. Otherwise, when the Page 5 of 18

6 Monitoring Team understands the solution is not reversible, it shall recommend either the exclusion of the Bidder or not having him invited to the next phase. Phase 3 evaluation and monitoring activities 4.1 Monitoring activities 4 As described above, 3 Monitoring Teams were formed in order to conduct a pre-final delivery assessment regarding on the proposals development with respect to the expected outcomes of Phase 3. Each Monitoring Team was formed by: a representative of a Procuring Entity; 2 experts at least 6 potential end users, i.e. an healthcare professional or a diabetes II patient. 3 Monitoring Team were formed by random association in June For Phase 3, each Team was assigned to 1 of 3 proposals to assess. The 3 Teams, each one headed by a representative of a Procuring Entity, received a desktop version of each proposal. The stage of development was intermediate, in the sense that it presented the main functions and structure of the proposal. The desktop version was provided in order to allow the Monitoring Teams members to have an understating of the proposal design and functioning, in order to simulate a user experience in navigating and performing task through the proposal. Together with the Monitoring Teams, a group of end users, both clinicians and patients with type 2 diabetes, were recruited by each procuring from their geographic area of reference. They were asked to use one proposal and evaluate it according to the use experience it offers, in order to provide additional information for the Monitoring Teams assessment. Proposals prototypes assessment was thus threefold: i) each end user assessed one proposal under the perspective of use experience; ii) Monitoring Teams evaluated the proposals on a more comprehensive basis, in order to provide inputs for ameliorating proposals before the End of Phase submission, iii) experts and the Procuring Entity representative evaluated the proposal under a more comprehensive perspective. The information gathered through this process was also used by the expert board as additional insights for evaluating the proposals Commercial Feasibility. The aim of conducting this type of assessment is also to extract potential end users feedback on proposals ease of use, design, satisfaction and areas which can be associated to the concept of user experience. During the evaluation, end users went through group semi-structured interviews, which were managed by a usability expert, who acted as a supervisor, and an observer. Interviews were structured as following: first, the supervisor introduce the proposal to the end user, describing its structure, logic and main functions ; second, the supervisor asked to each potential user to perform a series of tasks, in order to simulate a real-life case of use finally, the supervisor asked the user to answer a questionnaire, aiming at evaluating the use experience. Potential user population involved in the Monitoring Team activities were selected in order to guarantee a high level of familiarity with recent technologies, especially smartphones and portable devices (tablet, portable computer, etc...). These criteria were fundamental to insure that the participants had the skills and experience for assessing the proposals. Information extracted from this process was then sent to the other members of the Monitoring Team. It must be clarified that these information generated from the process were not formally included assessment conducted by Page 6 of 18

7 the Monitoring team, i.e., end user feedback didn t constituted a formal evaluation criteria within DECIPHER PCP assessment process. The questionnaire used for end user assessment was based on the following evaluation items: The sequence of steps to perform (steps to execute in order to perform the proposal s functions, i.e., alarm function, data entry function)is clear Once fully developed, the service will be useful The proposed service will meet my needs. To express their feedback regarding each associated proposal, users were asked to assess a number of statements using a 5-point Likert scale. The Monitoring Teams activities have generated the MOR, which have been delivered to each Bidder with the assessment result and additional guidance coming from the Teams regarding the development of the proposals for the final delivery to the Expert Board. 4.2 Evaluation activities The proposals final evaluation and selection for access to Phase 3 was conducted by the Expert Board. The main responsibilities of the board were: assessing, upon the proposal of the Tendering Board, the technical offers and End of Phase s. When assessing the End of Phase s, the Expert Board was able to rely on the indications and the recommendations provided by the Monitoring Team through the MOR; producing a of each of those proposals or s, including a scoring proposal, according to the award criteria stated in ANNEX IV; supporting the Tendering Board at any stage of the DECIPHER PCP Procedure, replying timely to its requests of help, making the clarifications or analysis it can require. DECIPHER PCP Evaluation framework for the Expert Boards is based on 5 main groups of criteria (Tabla 2): Functionality; Innovation; Quality Technical feasibility Financial feasibility Price In some cases, a group is made of different criteria, which aim at exploring the different aspects included in the dimension assessed (e.g., I FUNCTIONALITY). In other cases, groups are a single dimension evaluation (II INNOVATION). There is also the case of single-dimension criteria that rely on a complex set of multi-dimension information generated by a parallel evaluation process, as for: VI COMMERCIAL FEASIBILITY. Page 7 of 18

8 Score* Max. Points Min. Points Exclusión Criteria Score* Max. Points Min. Points Exclusión Criteria Score* Max. Points Min. Points Exclusión Criteria FP Table 1 - DECIPHER PCP awarding criteria scoring table Phase 1 Phase 2 Phase 3 I FUNCTIONALITY Basic functions Secure access to PA-PHR-S Share information Care management Inform in emergency situations Design functions Provide user interface accessibility Provide data availability and redundance Satisfy technical design requirements Satisfy business model design requirements II INNOVATION III IMPACT OF INNOVATION IV QUALITY Quality in Management Quality of Risk Management V VI VII TECHNICAL FEASIBILITY COMMERCIAL FEASIBILITY FINANCIAL FEASIBILITY VIII PRICE Page 8 of 18

9 The development of this framework was based on four principles: 1. The scoring will be made according to an absolute scale, meaning that several bidders can receive the same score and that the score a specific bidder receives is not affected by the scores other Bidders have received. 2. Successful Bidders will be duly notified on the contract signature. 3. Contractors will be expected to finish their project in time. In case the projects are not finished by the deadline, the Contractors will not be entitled to submit a tender for the next Phase. 4. The same criteria and evaluation method will be subsequently used across all the phases. By pursuing these principles, DECIPHER PCP management and coordination team aimed at respecting the requirements set by the European Commission for a fair competition in public procurement and the assessors guiding principles defined in DECIPHER PCP s Guidelines for Assessors document. Phase 3: Proof of concept Phase 3 begun on November 2016 and have been finished on March During Phase 3- Proof of concept, those involved in the Monitoring Team and the Expert Board adopted the evaluation framework and monitoring tools defined in the project preparatory stages in order to increase the alignment between the Bidders and the project expected outcomes and to insure an objective and impartial evaluation of the proposals. The two process, the evaluation and the monitoring, have been designed with the aim of allow for an interaction between the two. In particular, the work of the Monitoring Team will produce information that will support the evaluation of the proposals. This information will be generate with the proposals test conducted through end users (patients and healthcare professionals). In this stage proposals will be tested by end users using a prototypes provided by the Bidders themselves. Objectives 1) Evaluation The evaluation aims to verify and compare the performance (interoperability, scalability, etc) of different solutions in simulated real-life operational conditions of the targeted public service. The main output of this phase usually includes a test product specification, a field test and an updated cost/benefit evaluation. 2) Monitoring Monitoring activities for Phase 3 are outlined in the figure below. Figure 4- Phase 3 main events related to the Evaluation process Page 9 of 18

10 PROOF OF CONCEPT PHASE 3 DECIPHER START PROO F OF CONC EPT Month Preparation PROOF OF CONCEPT Month Month Data collection Questionnaire: Us abi lity Ac ce ssi bili ty Se ma nti cs Semi-structured interview: Ba rri ers / en abl ers Pe rce pti on / ex pe rie nc e In this case, 24 patients from the three procuring entities countries with chronic conditions and ability of using mobile solution and 6 doctors were involved in the monitoring activities, although 8 of them answered the questionnaire. The end users received training about the solution and they were asked to use the solution during 15 days and then they sent the feedback by means of questionnaire. The questionnaire contained items that end users ranked according to a Likert scale. The questionnaire included the following items: Learnability: The service provide clarity of wording The service s data grouping method and information visualization is clear It is easy to learn to use the service Month 4 Month END Data analysis PROO Evaluation F OF Report CONC EPT Page 10 of 18

11 I quickly became skilful with it The service uses a simple and natural dialoguing approach Efficiency: The service presents information in logical order The sequence of screens is clear Navigation tools (menu, labels, cursors) are consistent When using the service, it is easy to return to previous tasks It is easy to remember how to perform tasks through the service The service will help me in being more effective in following my treatment The service will help me in better controlling my health status The service meets my needs The service offers the possibility to set preferences The service has a flexible data entry design Users support The service provides timely feedback about all processes The service helps the user in getting out of an undesirable state easily The service provide the user with the possibility to send feedback I can recover from mistakes quickly and easily The service diagnoses the source and cause of a problem and suggests a solution The service offers an effective package of customer support tools (e.g., websites, tutorials, Question & Answer support, offline help tools, user manual) The support package allows for an easy identification of solutions for problems that arise in the usage Satisfaction/User experience: The service offers a easy and constant access to the health records I need to consult for monitoring my treatment/patients The service is pleasant to use The service works the way I want it to work The service is satisfying The service meets my needs The service is useful I would recommend the service to a friend In order to have a deeper understating of the use experience, 6 patient semi-structured interviews will be conducted. 5 Results 5.1 Evaluation results from the monitoring activities The Monitoring Outcome Report of Nextage s proposal highlighted: 3 good, 4 acceptable and 2 unsuccessful marks for the Functionalities criteria; 2 good, 5 acceptable and 5 unsuccessful marks for the Innovations criteria; 2 good, 1 acceptable marks for Impact of Innovation; 3 acceptable marks for Quality; 1 acceptable mark for Technical Feasibility; 1 unsuccessful mark for Commercial Feasibility; 1 acceptable mark for Financial Feasibility; Here comment from the MOR Page 11 of 18

12 Basic functions: Secure Access to PA-PHR-S - acceptable Dependency of Amazon Web Services security infrastructure. Share information - unsuccessful Unclear how unregistered physicians gain access to information. External translation services (API) mentioned. Manage treatments - acceptable Not clear differences between Pharmacy and Medication Design functions: Satisfy Technical design requirements - acceptable Dependencies well detailed. Dependencies on Amazon Web services and Google services may limit future deployments Satisfy Business Model Design requirements - unsuccessful Weak strategies to maintain stakeholders engagement (patients, doctors, public and private organisms)further explanation on how to simplify business model It is necessary to introduce reading systems and synthetic and detailed data analysis Innovation: Innovative approaches - acceptable In the innovative approaches citizens and not only doctors would raise awareness about qualitative and quantitative information, about collaboration and so on Innovative methodologies unsuccessful Insufficient details on how the Prototype methodology fits a regulated environment where devices or systems are certified. Innovative processes acceptable Further details Advances in Usability and Accessibility - unsuccessful Correct use of accessibility standards, but no innovation Advances in intelligent systems unsuccessful Further information regarding the predictive analytics Quality: Adequate managing structure acceptable It is important to have marketing and sales roles Risks are indentified and monitored acceptable It is necessary to introduce market risks Commercial feasibility unsuccessful Market strategy isn t so innovative. The rule of the citizen is central to the success of solution... The preparatory phase must be strengthened Financial feasibility unsuccessful It is important to have information on the market value proposition of similar solutions and to know marketing strategies adopted by competitors. In this way we can learn and go on from what is just done Page 12 of 18

13 5.1.2 The Monitoring Outcome Report of E-Results s proposal highlighted: 3 good, 8 acceptable marks for the Functionalities criteria; 12 acceptable marks for the Innovation criteria; 2 good, 2 acceptable marks for Impact of Innovation; 3 good marks for Quality; 1 acceptable mark for Technical Feasibility; 1 acceptable mark for Commercial Feasibility; 1 acceptable mark for Financial Feasibility; Comments included in the MOR are: Basic functions Secure access to PA-PHR-S-acceptable The IOR was improved relative to phase 2 by adding figures. The organization of the text was also improved which makes it easier to followed. In registration phase (besides smart card) any authentication method supported by the PHR should be used. This should be possible in the context of OAuth. Share information acceptable It has been well understood that multiple standards need to be supported. The test could be somewhat shortened and mostly focused on those technologies which are implemented in the current version Manage treatments -acceptable Better description of the situations and authorisations Inform emergency services - good The approach to support the emergency situation. One recommendation for improvement: the emergency call and data-set should also be available when the user is signed-in. Now these are shown only before login(?).better description of the situations and authorisations. Design functions Provide user interface access good The automatic emergency call is a useful feature. Useful that the solution is suitable for PC layout and smartphone, but primarily the solution is for mobile access so the tablet / smartphone layout should be optimised for this project Provide data availability and redundancy -acceptable The semantics could be improved by using the available terminologies more extensively (was only available for medication in the prototype) Satisfy technical design requirements acceptable It is positive that a workable solution for translation and transcoding has been presented Innovation Innovative algorithms acceptable All innovative aspects listed here are highly relevant. Section 3 could be improved by more clearly defining which innovations have already been implemented in the software and which are upcoming in the future. Good implementation Innovative concepts good Good implementation of concepts Advances in usability and accessibility acceptable Demo version supplied to the assessors seems to work nicely. There is some clipping of texts, but it may be Page 13 of 18

14 caused by the fact that it is not intended to be used by a tablet, instead of normal phone. But good that it is useable on tablet as well as phone Quality Managerial quality factor 1 good It seems a good feature that the system is epsos compatible so that the final solution can be used when the epsos service is available, but as yet there are no plans to use epsos widely in the UK The Monitoring Outcome Report of Gnomon s proposal highlighted: 10 good, 6 acceptable marks for the Functionalities criteria; 9 good, 2 acceptable marks for the Innovation criteria; 1 good, 1 acceptable marks for Impact of Innovation; 10 good, 1 acceptability for marks for Quality; 1 good mark for Technical Feasibility; 1 good mark for Commercial Feasibility; 1 good mark for Financial Feasibility; Comments included in the MOR are: Basic functions Secure Access to PA-PHR-S acceptable What it is not clear is the level of implementation of the different secured mechanisms that were proposed, for example further development is proposed for embedding SIM Card; is OTP mechanism is in place? Share information good Mobile: good Integration of documents. Additional Medical information can be stored. Devices Integration configured Manage treatments acceptable Treatment plan can be created from available eprescription Inform in Emergency Situations good Panic button send information to one or several recipients Innovation Satisfy Technical design requirements acceptable Technical design is satisfied by using standards. However innovative solution need testing in large scale pilot for optimization Core service for mhealth application development acceptable Architecture to be tested by other solutions Administrative service acceptable Under development Innovative concepts acceptable How the IoT oncepts are integrated in the solution? Not described Innovative Tools or technologies -acceptable Panic Button and break of glace n the Mobile Apps Page 14 of 18

15 FP Advances in intelligent systems good Should be tested Advances in secure systems -good Should be tested Impact innovation (Impact 2)/establishment of open source community acceptable Is there specific community for open Decipher? Not clear Quality Detailed risk management mechanism good Looks like some mitigation proposal could be not strong enough The project management tools employed such as continuous guidance, periodically monitoring s etc. will continuously assess the status of the project.-good Concentrating Project coordinator and sales representative in one people could be difficult. A survey among stakeholders and domain experts is conducted in order to extract and refine user requirements and effectively designour solution.-good Good analysis. It is useful to include all stakeholders Phase 3 pre pilot testing period will minimize the probability of poor test data creation acceptable In phase 3 the project has to connect to a TEST PHR, not the actual one. Commercial feasibility-good Advertising could be a privacy concern. Crowdfunding is difficult to achieve. Clarify the volume for 100 individuals. 5.2 Evaluation results from the Expert Board s activities The Expert Board were responsible for the selection of the proposals allowed to access to Phase 2 of DECIPHER PCP. The process was guided by the Awarding criteria table presented in section 4.1 (Table 1). The results for the evaluation process are ed below in Table 2. I Nextage SL E-results srl GNOMON / iuz Technologies Lda Secure access to PA-PHR-S Share information Care management Inform in emergency situations FUNCTIONALITY Basic functions Design functions Provide user interface accessibility Page 15 of 18

16 Nextage SL E-results srl GNOMON / iuz Technologies Lda Provide data availability and redundance Satisfy technical design requirements Satisfy business model design requirements II INNOVATION III IMPACT OF INNOVATION IV QUALITY Quality in Management Quality of Risk Management V TECHNICAL FEASIBILITY VI COMMERCIAL FEASIBILITY VII FINANCIAL FEASIBILITY Optional comments on Assessment of Awarding criteria for Nextage SL are: Assessment of Functions About storing data in Amazon Cloud: this is a big issue. Specific agreements should be signed with the company to ensure the European legal aspects. It is OK to give rights to the patient, how the situation is handled with the foreign doctor is complicated. There could be a more insightful way for this. For translations, you will have to translate scales and criteria, this is very challenging. Some difficulties about sharing data with PHR. PHR often are closed systems. Inform in emergencies relies on an external device. This is not optimal since the device could be lost. Registration for patients requires lot of backoffice work for procurers or entities wanting to emprage the product. Assessment of Innovation Impact There is no information about where does the more medical and scientific information comes from - is it evidence-based we imagine. With proper testing and maturing, this application could be important. Assessment of Quality Some risks such as data protection and especially the legal part are not fully addressed. This is more important because a system like Amazon AWS is to be used Assessment of Technical Feasibility They have looked the matter from many points. So it is very feasible. They seems very competent with the technologies used. Assessment of Commercial Feasibility Business model OK, but still some doubt about the travellers/consumers point of view. Pretty good analysis, with sound figures for profit&loss for first years, but since selling to companies could be difficult, break-even point could be further in time. Page 16 of 18

17 5.2.2 Optional comments on Assessment of Awarding criteria for E-results are: Assessment of Functions With very limited exceptions (e.g. the brake-the-glass case) the phase 3 include exactly the same information provided in Phase 2; so the concerns raised for Phase 2 are still applicable and even more relevant now. More specific indications on how the requested features have been actually realized would have been expected. For instance, there are several references to existing standards and service interfaces for which it is not clear how they have been used for implementing the expected functions and how the integration with the PA PHR system could be achieved. Let s take as example the mechanism proposed in Fig : the DECIPHER service gets a document from the Country A, then ask country B to provide a not well defined transcoding / translation service that is supposed to be able to process any kind of CDA country A templates and terminology used in country A. It is not clear what this service is expecting to do, under which conditions the service should be provided and so on; that is, from a practical perspective, it is difficult to figure out how this could work. There are moreover some technical aspects, as the mentioned schema mapping, which seems to be over-simplified. Even though the technical architecture proposed, and the references to the most relevant standards suggest that scalability, modularity, independence from 3rd party of the proposed solutions could be reached; there are several not clarified technical (and architectural) challenges that reduce the awareness that this goal can be met. Regarding usability, some details have been provided about the WCAG principles used, although some evidences would have been expected, like results from user testing, questionnaires, or similar documentation. Assessment of Innovation No additional information respect to Phase 2 has been provided. The proposed solution goes beyond the current state of the art, even though there are elements that have been described as innovative that should be considered instead as state-of-the-art. User centred design, including self-management education approach, physical activity and nutrition aspects are considered, as well as the use of HL7 FHIR interoperability standards to boost the use of technology in the health sector. Socialization of the solution is proposed as part of the innovation, including different stakeholders related to the patient. Assessment of Innovation Impact There are some indications, considering the different dimensions (technical, clinical, socio-economic,...) that the proposed solution may address future and wider challenges in DECIPHER area. However, most of the phase 3 is the same of the phase 2 one. It would be expected at this stage to have more concrete information about the practical challenges and lesson learned, rather than a repetition of what was already provided in phase 2. There are moreover some not clarified technical (and architectural) aspects that reduce the awareness that this goal can be met. Assessment of Quality With few exceptions the information provided in phase 3 are the same of that included in the phase 2. Additions are related to usability and ethical constrains, although no evidences are provided about activities performed or usability test results, or of what has been delivered and what not. From a project management perspective it would be good to have at this stage an assessment of what has been realized and what was not according to the plan, and an analysis of the reasons for that; rather than a not fully clear update of the figures that consisted in the removal of a yellow rectangle in the pert diagram and the reduction of the duration of the plan from M23 (in the phase 2 ) to M17 (in the phase 3 ) of the same plan. Considering the difficulty to score this phase based on the assertions proposed by the invitation to tender (e.g. "The proposal contains <...> plan for the development of a working solution..."), the score has been therefore determined balancing between the absence of a final evaluation and the assumption that a solution has been delivered. Relevant risks have been identified and a management process seems to have been actually applied. From the technical perspective there is the impression that some of the solutions indicated for resolving the initial risks include not negligible unresolved risks. Assessment of Technical Feasibility Most parts of Phase 3 Report are the same of those provided in Phase 2, it would be expected to have at this stage more concrete information about the practical challenges and lesson learned rather than a repetition of what was provided in phase 2. There are moreover some not clarified technical (and architectural) aspects Page 17 of 18

18 that reduce the awareness that some tender s goals have been / could be met. Assessment of Commercial Feasibility Apart from update of the Annex 6 related to R+D cost structure, no new, more specific and detailed information about this subject seems to have been provided for this. The text seems to be substantially the same of that used for phase 2. It would have been expected to find some updates to the initial marketing analysis or at least the reasons for which the initial analysis is still applicable. Assessment of Financial Feasibility No new, more specific and detailed information about this subject seems to have been provided for this. The text seems to be substantially the same of that used for the phase 2 excepting for an addendum in Annex 6, that contains however few absolute costs that were not allowed by the invitation to tender. I would expect in phase 3 a final balance with the analysis of the gaps respect to the planning and a more specific analysis of the future financial sustainability Optional comments on Assessment of Awarding criteria for Gnomon Ld. are: Assessment of Quality The system will need to consider how to handle technical risks when integrating with the Patient PHR Assessment of Technical Feasibility There are a lot of nice features in the system but consideration needs to be given to how they will be implemented by the healthcare providers especially in practice Assessment of Commercial Feasibility Although freemium model is a good approach to get end user ( patient) traction, there needs to be more time and thought given to how to engage the healthcare providers Page 18 of 18