Elements: WfCommons Tools for Workflow Research, Development, and Education on the CyberInfrastructure

Many scientific and societal breakthroughs can only be achieved by performing complex processing of vast amounts of data efficiently. In domains as crucial to our society as climate modeling or health (and in fact in most fields of physics, chemistry, and biology today), as well as more recently in the overarching field of artificial intelligence, practitioners define "scientific workflows" to describe complex computational scientific processes as data and inter-dependent computations on these data.

Scientific workflows have already supported many ground-breaking discoveries and have now become mainstays in today's science. Due to constantly evolving scientific needs and technological landscapes, it should come as no surprise that scientific workflows themselves, as well as the software and hardware systems used to support them, are the target of countless research, development, and education activities.

These activities are key to advancing the state of the art of scientific workflows, but to do so they must be driven by the use of realistic and accurate scientific workflow datasets. These datasets contain "workflow instances" that describe the structure, the compute and data demands, and the past executions of real-world scientific workflows. The goals of the WfCommons project are the generation and curation of such workflow instances, and the development of software tools by which the curated workflow instances can be explored and selected by scientific workflow researchers, developers, and educators.

The project will provide realistic, accurate, and usable data that is fundamental to advancing the state of the art of the scientific workflows that arise in virtually all fields of science.

A schema will be defined for the precise description of workflow instances, both in terms of the workflow's specification (as static or dynamic sets of tasks with control- and/or data-dependencies) and of its past execution (on Cyberinfrastructure resources using a production workflow runtime system). Standalone software tools will be developed to construct workflow instances based on real-world executions using popular workflow runtime systems.

These tools will parse and analyze the input provided to and the output and log files produced by these runtime systems, so as to extract the relevant information as defined by the above schema. A suite of integrated tools will be developed for browsing, visualizing, editing, and simulating the curated workflow instances. These tools will allow WfCommons users to easily select the workflow instances that are appropriate for their specific goals.

These users include researchers who wish to perform evaluation of their algorithms for specific kinds of workflow configurations, workflow runtime system developers who wish to augment and strengthen the testing and evaluation of their systems, or educators who wish to achieve new learning objectives by exposing students to compelling and real-world workflow scenarios. These integrated tools will be made available directly in-the-browser on the WfCommons site.

New data items (workflow instances) and software tools will be released continuously throughout the project. The impact of the project will be continuously assessed throughout the project's timeframe and beyond via several metrics, many of which will be automatically computed and reported on the project's Web site daily.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.