Project Plan Scientific Data Management

Transcription

1 Project Plan Scientific Data Management Figshare (where we share documents for the workshop): Team Members Software Engineers, School of Computer Science of the University of St Andrews Shyam Reyal smr20@st andrews.ac.uk Simone Conte sic2@st andrews.ac.uk Supervised by School of Computer Science of the University of St Andrews Prof Simon Dobson simon.dobson@st andrews.ac.uk Prof Alan Dearle alan.dearle@st andrews.ac.uk School of Chemistry of the University of St Andrews Prof Douglas Philp dp12@st andrews.ac.uk Dr Tomas Lebl tl12@st andrews.ac.uk 1/14

2 Table of Contents Team Members Introduction Stakeholders Existing work Goals and Objectives Deliverables General/Long Term project deliverables Deliverables for the first 3 months Indicators of Success Risk Analysis & Mitigation Deliverables for the next 6 and 12 months Indicator of Success Risk Analysis Indicator of Success Risk Analysis Schedule and Milestones Resources Financial Technical Human Communication Summary 2/14

3 Introduction Scientific machineries generate large volumes of data on a day to day basis. We call such data scientific data. We propose a software system for the management of such scientific data. The volumes of data can be quite large and it is usually shared across a large number of users of various research groups, departments. Techniques applied on data has to be efficiently managed such that they it can be easily accessed, manipulated, and stored in the most convenient manner for use by the stakeholders involved. Our proposed solution will allow end to end provenance on scientific data. Researchers will be able to back link from the transformed, analysed and interpreted data used for publication to the original experimental data. While the data goes through a specific and arbitrary scientific workflow, no storage overwrite is involved. Stakeholders We have identified the following stakeholders who would be involved with our proposed system Experimental scientists/researchers (physicists, chemists, biologists, etc) Students Publishers PI (principal investigator) within and across research departments Lab administrators Existing work We have already built a system (NOMAD) a Java based web application that aides with the administration, booking, configuration and programming of a few types of such scientific machinery (NMR spectrometers, Powder X ray diffractometers, Mass spectrometers). This is our case study system, but our proposal aims to take any scientific searchable input data. 3/14

4 Figure 1. User portal In addition to the services above, NOMAD also acquires resulting scientific data from different machines and stores them in a secure, resilient manner while providing a user friendly web interface to filter and search for specific data within the dataset. I.e. users can search for generated data (referred to as spectra) based on the submitting user s username, group name, submitted date/time, experiment, or description. Figure 2. Search portal 4/14

5 Figure 3. Spectra in browser preview accessed the search portal The diagram below summarises the proposed scientific workflow, which allows researchers to link and track back data from publications to the original experimental data. Figure 4. Proposed scientific data management workflow 5/14

6 Goals and Objectives We have defined a specific set of goals and objectives regarding the system we intend to build as part of the Research Data Spring : Independent software system that takes as input searchable scientific data and allows the identified stakeholders to manage it Provide open access required by EPSRC by May 2015 and other research bodies in the near future Browser based solution Adhere to author s privacy policies. I.e. Must be able to ensure who can access what. necessary on making only certain data public Reflect the data processing workflow. I.e. users should be able to follow through the process of modification of the data (through versioning) Enable hypersearch search data through various metadata formats In the following sections we plan out how we are going to tackle this list of goals and objectives. Longer term aims might be subject to slight changes. 6/14

7 Deliverables General/Long Term project deliverables (Considering a 3 5 years time span): Full end to end scientific data management solution Module for project based data management integration with electronic lab books Social media integration allowing improved user experience and communication for the identified stakeholders Full documentation for users and training (for administrators and researchers) User experience and user satisfaction survey/interview results User friendly data visualisation for heterogeneous displays environment (i.e. laptops, desktop machines, public displays, mobile devices) Generic integration module for any form of scientific machinery (physics, chemistry, biology machines) regardless of type or manufacturer clients will be able to provide machinery specific modules to plug in our solution Deliverables for the first 3 months (March May 2015) These deliverables will be showcased at the next Research Data Spring workshop in June 2015 Our proposed deliverables operate on the premises that we have volumes of scientific data that is searchable (e.g. data from NOMAD), but not optimally organised for keeping track of changes and does not facilitate linking data with publications (as per the EPSRC requirement in 2015). Data Tagging Scientific data mostly consists of noise. E.g. there may be 20 experiments run on a single sample (E.g. a chemical sample), 19 of which may be discarded as non satisfactory results whereas 1 result makes it into a publication. We will provide an efficient and user friendly method of tagging data as legitimate or as noise within the dataset This will be built as an extension on NOMAD search portal with tagging authorization given to the respective owner of the data or PI Linking Raw Data to Publications 1 EPSRC requires that every publication exposes their raw data as open accessible as per May This will require PI s or delegates to manually browse each data item in the set and attempt to link the data to its respective publication 1 7/14

8 We will provide a system that would semi automate this process by filtering out only the legitimate data points (as mentioned above) thus reducing the search space to about 5% of the original, and by also providing an efficient UI for the PI s to link their data with their publications This will be built into the PURE digital library system with links to raw data pointing to the NOMAD search portal Indicators of Success Working prototype by Week 4 6 Fully working production system deployed by Week % of existing publications linked with their data by (more than 3 months time dependent on PI who produced the publications) 100% of new publications linked with their data after % of existing publications linked with their data by (more than 3 months time dependent on PI who produced the publications) Risk Analysis & Mitigation hardware failures/downtimes mitigating includes cloud hosting human errors tagging incorrect data (junk and legit or otherwise) have an acknowledgement system only the linked ones forgetting skipping linking build notification alert system to notify the PI s Deliverables for the next 6 and 12 months As a team we have started to design and plan the deliverables for Phase 2 and Phase 3. Less detailed descriptions are given, since they might change slightly over time. We will make use of future workshops to get further constructive feedback. Phase 2 will be mainly focused on allowing our system to version and annotate scientific data. Versioning is a well known technology and methodology in computer science, but the same is not true in other fields. We aim to bring such practices in other scientific areas, favouring collaborations within and across groups. An important aspect of data science workflows is being able to track back the set of changes that the data has gone through. Versioning would enable this, making researchers more efficient. Indicator of Success TODO work in progress for the next workshop 8/14

9 Risk Analysis TODO work in progress for the next workshop Phase 3 will be mainly focused on building a metadata search framework within our proposed system. Currently data is text searchable only. However, most of the scientific data is more complex than just normal text. We distinguished two main factors with scientific data: structure (derivable algorithmically) and meta tagging (coming from users data processing workflow). In this six months phase the focus will be to allow users to search data using structures and the meta tags as query parameters. Indicator of Success TODO work in progress for the next workshop Risk Analysis TODO work in progress for the next workshop 9/14

10 Schedule and Milestones Objective Assignee Comments Milestone Phase 1 (3 months) Data Tagging developers data linkage framework Linking Raw Data to Publications 50% of existing publications linked 100% of new publications linked PIs, researchers, developers support 100% of existing publications linked Risk deadline met based on PIs commitment Cloud deployment developers, system administrator Phase 2 (3 months) integration with existing versioning technology (git, hg,..) developers versioning and annotation framework metadata annotation techniques developed continuous development with researchers feedback from various fields PIs Phase 3 (6 months) hypersearch exploration phase developers and PIs Risk Research question hard to estimate metadata search framework 10/14

11 effort structural search developers images search metadata search 11/14

12 Resources In this section we present a rough estimate of the costs involved for the next 3 months. Financial We estimate approximately a 19,250 budget according to the technical and human resources needed for the next three months. Expenses involve: developer costs (0.75 FTE developers) 11,250 investigator costs 5,000 hardware and cloud costs cloud deployment 1,000 travel and accommodation for June workshop (x3 attendees) 2,000 Technical Cloud deployment Amazon S3 or Windows Azure ( to be decided ) Developers hardware Already have most of the hardware costs are minimal The system will be based on free available frameworks: Java EE TomCat HTML, CSS, JS More open source and freely available technologies will be used as needed Human Our team will be composed by two part time developers (accounting for a total of approx FTE developers) and two principal investigators. The list of skills required by the developers are the following: excellent understanding of scientific research data 12/14

13 excellent storage and database knowledge good/excellent Java EE good/excellent Javascript good/excellent HTML, CSS, XML good Cloud Deployment We have identified our developers team in the following two people: Mr. Shyam Reyal, PhD student, University of St Andrews Mr. Simone Conte, PhD student, University of St Andrews 3rd developer to be hired The time capacity for each developer is 10 hours per week. Our identified Principal Investigators are: Prof Alan Dearle, Dean of Science, University of St Andrews Prof Simon Dobson, Director of Research in Computer Science, University of St Andrews Communication The mean of communications with the identified stakeholders are the following: common alias provided by ITS collaborative mail group Twitter feed 13/14

14 Summary We already have a system that provides End To End Provenance. Here we propose a Scientific Data Management independent software framework that takes not yet processed, unclean, searchable scientific data and allows optimal scientific workflow for researchers. We have identified three main key features in our proposed system that we intend to accomplish within the next 12 months: data linkage annotation/collaboration/versioning metadata search Our solution is going to be tested and run over a large quantity of scientific data that has already been collected and meets the above requirements: not yet processed, unclean, searchable. The total estimated costs for the next 3 months totals 19,500 as detailed in the Resources section. 14/14