Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Readme cleanup #93

Merged
merged 4 commits into from
Sep 11, 2018
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 0 additions & 16 deletions Definitions.md

This file was deleted.

17 changes: 0 additions & 17 deletions Feedback.md

This file was deleted.

80 changes: 31 additions & 49 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,80 +1,62 @@
<img src="https://www.ga4gh.org/gfx/GA-logo-horizontal-tag-RGB.svg" alt="GA4GH Logo" style="width: 100px;"/>

Schemas for the Workflow Execution Service (WES) API
====================================================
Workflow Execution Service (WES) API
====================================

The [Global Alliance for Genomics and Health](http://genomicsandhealth.org/) is an international
coalition, formed to enable the sharing of genomic and clinical data.

Cloud Work Stream
-----------------

The [Cloud Work Stream](https://ga4gh/cloud) concentrates on data representation, storage,
and analysis, including working with platform development partners and
industry leaders to develop standards that will facilitate
interoperability. The Cloud Work Stream is an informal, multi-vendor working group focused on standards for exchanging Docker-based tools and CWL/WDL workflows, execution of Docker-based tools and workflows on clouds, and abstract access to cloud object stores.
The [Cloud Work Stream](https://ga4gh/cloud) helps the genomics and health communities take full advantage of modern cloud environments.
Our initial focus is on “bringing the algorithms to the data”, by creating standards for defining, sharing, and executing portable workflows.
We work with platform development partners and industry leaders to develop standards that will facilitate interoperability.

What is WES?
============

The Workflow Execution Schema is a minimal common API describing how a user can submit
workflow requests to workflow execution systems in standardized ways.
Workflow execution engines (SevenBridges, FireCloud, etc) can support this API so users can make workflow requests
programmatically, adding the ability to scale up. In addition, these workflow services could have (and probably do have)
UIs that would (possibly) use this API under the hood to facilitate workflow execution requests.
The Workflow Execution Service API describes a standard programmatic way to run and manage workflows.
Having this standard API supported by multiple execution engines will let people run
the same workflow using various execution platforms running on various clouds/environments.
Key features include:

Having this standard API supported by multiple execution engines will give people options of processing
the same workflow (CWL or WDL) across different workflow execution platforms running across various clouds/environments.
As an example use case, one can find a workflow in CWL on [Dockstore.org](http://dockstore.org), use Dockstore to
generate a JSON parameterization file, and submit this to a GA4GH-compliant
workflow execution service.
* ability to request a workflow run using CWL or WDL
* ability to parameterize that workflow using a JSON schema
* ability to get information about running workflows

Key features of the current API proposal:
API Definition
--------------

* ability to request a workflow run using CWL or WDL (and maybe future formats)
* ability to parameterize that workflow using a JSON schema (ideally a future version would be in common between CWL and WDL)
* ability to get information about running workflows, status, errors, output file locations, etc.
See the human-readable [Reference Documentation](https://ga4gh.github.io/workflow-execution-service-schemas/docs/)
and the [OpenAPI description](openapi/workflow_execution_service.swagger.yaml).

Outstanding questions:

* a common JSON parameterization format
* standardizing terms, job, workflow, steps, tools, etc
* reference implementation at https://github.com/common-workflow-language/cwltool-service/tree/ga4gh-wes
* validation service for testing WES implementations' conformance to the spec
* Including all task_logs in the workflow log request may present a scaling problem when there are 100s-1000s of tasks
* Providing a state notification callback URL (eg a webhook)
* Passing through authentication (user role)

How to view
------------

The file `openapi/workflow_execution_service.swagger.yaml` contains the API description.

Please visit http://ga4gh.github.io/workflow-execution-service-schemas to view this document in Swagger UI.
Use Cases
---------

Use cases include:

Building Documents
------------------

The OpenAPI description is in the `openapi` directory.
* "Bring your code to the data": a researcher who has built their own custom analysis can submit it to run on a dataset owned by an external organization, instead of having to make a copy of the data
* Best-practices pipelines: a researcher who maintains their own controlled data environment can find useful workflows in a shared directory (e.g. [Dockstore.org](http://dockstore.org)), and run them over their data

Possible Future Enhancements
----------------------------
* common JSON parameterization format that works with CWL and WDL
* validation service for testing WES implementations' conformance to the spec
* improved tools for troubleshooting execution failures, especially when there are 100s-1000s of tasks
* a callback mechanism for monitoring status changes in running workflows (e.g. a webhook)
* integration with GA4GH data access APIs (e.g. htsget, DOS)

How to contribute changes
-------------------------

Take cues for now from the [ga4gh/schemas](https://github.com/ga4gh/schemas/blob/master/CONTRIBUTING.rst) document.

We like [HubFlow](https://datasift.github.io/gitflow/) and using pull requests to suggest changes.

Security Considerations
-----------------------

If a security issue with any of the above specifications is realised please send an email to [email protected] detailing your concerns.
See [CONTRIBUTING.md](CONTRIBUTING.md).
If a security issue is identified with the specification, please send an email to [email protected] detailing your concerns.

License
-------

See the [LICENSE]
See the [LICENSE](LICENSE).

More Information
----------------
Expand Down
5 changes: 0 additions & 5 deletions USECASES.md

This file was deleted.