Japanese Virtual Observatory and Workflow

Transcription

1 Japanese Virtual Observatory and Workflow Masahiro Tanaka National Astronomical Observatory of Japan Feb 26, 2008 Astro-RG, OGF22 1

2 This talk Overview of JVO JVO Workflow system Investigation of GUI Workflow builder Services called from Workflow Feb 26, 2008 Astro-RG, OGF22 2

3 JVO system : Portal and Services 解析 Grid System MDS 解析解析 Analysis Servers JVO Portal Portal Invoke User SDSS QSO SkyNode 1.0 Subaru User Storage 2MASS Data Services S. Reg. P. Reg. Other VOs SIAP/SSAP Registry P. Reg. SkyNode Feb 26, 2008 Astro-RG, OGF22 3

4 JVO Portal - top page Feb 4January 26, , 2008 ADC Astro-RG, International OGF22 Review 4

5 JVO SkyNode Architecture Uses Apache Axis and Tomcat. Accepts Four kinds of query languages. Internet JVO Portal JVO SkyNode Query is executed through JDBC-like interface. Query result is formatted as VOTable or CSV. JVOQL ADQL SIAP SSAP HTTP/SOAP AXIS HTTP/Parameter Tomcat Translator Query Executer Formatter SelectSQL JDBC for SkyNode ResultSet JVO SkyNode DBMS getmetadata() getstring() getfits() Feb 26, 2008 Astro-RG, OGF22 getjpeg() 5

6 Finding Services Registry Searchable Registry Publishing Registry Publishing Registry Query Services Data Service Analysis Service Virtual Observatory Client Feb 26, 2008 Astro-RG, OGF22 6

7 Grid service for Subaru image reduction Automated job assignment with Monitor and Discovery Service (MDS) Execute Client JVO Portal Service Request Available Server Info Confirm Job Completion Job Completion Info MDS Server Job submission Results # of Jobs CPU load Job status Analysis Server Management Data Analysis Servers Parallel Executions on Multiple Server Feb 26, 2008 Astro-RG, OGF22 7

8 JVO Workflow system Feb 26, 2008 Astro-RG, OGF22 8

9 Example of workflow JVO workflow control Query SkyNode QSO Catalog SIA Service Subaru image HyperZ WebServices Source Catalog SExtractor WebServices Analysis Services Feb 26, 2008 Astro-RG, OGF22 9

10 JVO Workflow Language XML Diagram of JVO Workflow Description Language 0..* 0..1 Variables Variable 1 Sequence 1 1 <<abstract>> SequenceActivity Flow <<rootnode>> Process 1 1..* 1..* 1 <<abstract>> Activity 1 <<abstract>> BasicActivity <<abstract>> LoopControl <<abstract>> Control 1 <<abstract>> ActivityContainer <<abstract>> ConditionControl 1 if Parfor For While awk Switch 1 Script Command invoke Input Output * 0..1 Then Else Case Otherwise Feb 26, 2008 Astro-RG, OGF22 10

11 Current workflow builder for JVO Editing XML directly Difficult to write XML Use Templates WF Editor No GUI workflow builder for JVO Execution monitor Feb 26, 2008 Astro-RG, OGF22 11

12 Investigation of GUI Workflow builder for JVO Feb 26, 2008 Astro-RG, OGF22 12

13 JFLOW Developed at CDS Good User Interface Does not support flow controls: Condition Loop Feb 26, 2008 Astro-RG, OGF22 13

14 Workflow editor for NAREGI Good Graphical WF builder Developed for Grid workflow for NAREGI Applicable to VO Services?? Feb 26, 2008 Astro-RG, OGF22 14

15 NAREGI Building Grid environment using NAREGI Middleware β2.0.1 Trying Interoperability with KEK (Institute for high energy physics) Problems: Difficult to install NAREGI middleware No support for VO Protocols Feb 26, 2008 Astro-RG, OGF22 15

16 Taverna. Developed for Biology but usable for general purposes Evaluated by AstroGrid JVO team has just started evaluation Client-side application JVO has server-side workflow Feb 26, 2008 Astro-RG, OGF22 16

17 Trying Taverna Workflow calling a single Web Service Feb 26, 2008 Astro-RG, OGF22 17

18 Executing Feb 26, 2008 Astro-RG, OGF22 18

19 Result Feb 26, 2008 Astro-RG, OGF22 19

20 Taverna good things Good User Interface Easy to call Web Services Able to save workflow and results as XML Good workflow engine Job monitoring and control Logging (intermediate results and status) Feb 26, 2008 Astro-RG, OGF22 20

21 Taverna issues Still need knowledge on Web Services Input/Output data types Programming Need Communication with : Client-side software VO Registry Feb 26, 2008 Astro-RG, OGF22 21

22 Scripting languages Define new Workflow Language easier to write than XML? Not difficult to define it from XML definition We have already workflow engine. Need language design Learning cost for users Use Existing language? Perl, Python, Ruby, No learning cost if user knows Feb 26, 2008 Astro-RG, OGF22 22

23 SOAP call example using Scripting Language Ruby script : require 'soap/wsdldriver' wsdl = ' url = ' 'db=sdss&table=spectrum&id= &format=spectrum/fits' driver = SOAP::WSDLDriverFactory.new(wsdl).create_rpc_driver result = driver.invoke(:url=>url).return Easy to call Web Services Script code can be a workflow Feb 26, 2008 Astro-RG, OGF22 23

24 We have choices We just started evaluation. Taverna is a promising tool. Need more experience. Feb 26, 2008 Astro-RG, OGF22 24

25 Services called from Workflow Feb 26, 2008 Astro-RG, OGF22 25

26 Services in VO VO framework Data archives Registry, SIAP, SSAP, ConeSearch, SkyNode, TAP, Data storage VO Space Need more Analysis services Visualization services Interoperability with client tools SAOImage, VOPlot, Aladin, Feb 26, 2008 Astro-RG, OGF22 26

27 Use Cases of Workflow Subaru data reduction Service: Parallel execution with cluster Study on AGN environment Service: SExtractor and HyperZ Search for Metal-poor stars Service: Line detection service Feb 26, 2008 Astro-RG, OGF22 27

28 Automatic Spectral Line Detection Service Detect every scale of line width Wavelet-like algorithm Feb 26, 2008 Astro-RG, OGF22 28

29 Grid Challenge Public Contest of HPC Held in mid 2008 by Grid Scientists JVO provides Subaru data and scientific scenario : Supernova search Good experience to build Grid services for Astronomy Feb 26, 2008 Astro-RG, OGF22 29

30 Useful Services Re-usable, general-purpose services Services which require computer resources CPU Storage Services with clearly-defined interface Feb 26, 2008 Astro-RG, OGF22 30

31 Issues Too few available services Publicly available services are needed. WSDL can be written freely : Variety of interface can be confusing. Granularity of Services? Need Methodology / Guideline / Standard? Feb 26, 2008 Astro-RG, OGF22 31

32 Conclusion JVO system and workflow are reviewed. GUI WF builder for JVO is evaluated : Taverna is a promising tool. Useful services called from Workflow are needed. Feb 26, 2008 Astro-RG, OGF22 32