diff --git a/docs/index.md b/docs/index.md index c2d3cc2..97de1a4 100644 --- a/docs/index.md +++ b/docs/index.md @@ -8,19 +8,17 @@ nav_order: 1 ### Cloud-Optimized Data Management and Cyberinfrastructure #### A collaboration between [RPS, a Tetra Tech Company](https://www.rpsgroup.com/services/oceans-and-coastal/) and the NOAA Integrated Ocean Observing System ([IOOS](https://ioos.noaa.gov/)) -- [**Project Documentation**](https://asascience-open.github.io/nextgen-dmac/) +| | | | +|-|-|-| +| **Prototypes** | [Planning and Design](protoplan.md) | [Outcomes](outcomes.md) | +| **Presentations** | [DMAC Tech Webinar](https://www.youtube.com/watch?v=M8E2wQk3k8A) | [Pangeo Showcase](https://discourse.pangeo.io/t/pangeo-showcase-improving-access-to-noaa-national-ocean-service-model-data-with-kerchunk-and-xpublish/3725) | +| **Demos** | [XREDS Test](https://nextgen-dev.ioos.us/xreds/) | [OceansMap Test](https://dev.oceansmap.com/maracoos/?catalog=dmac_combined) | +| **Notebooks** | [NetCDF vs Zarr](https://github.com/asascience-open/nextgen-dmac/blob/main/binder/dbofs-examples/netcdf-vs-zarr.ipynb) | [Best Forecast Retrieval](https://github.com/asascience-open/nextgen-dmac/blob/main/binder/dbofs-examples/best-forecast.ipynb) | +| | [CORA 40 Year WL Reanalysis](https://github.com/asascience/ncdis) | [National Water Model](https://github.com/asascience-open/NWM) | -- **Prototypes:** [Planning and Design](protoplan.md) | [Outcomes](outcomes.md) +[**Meeting Minutes**](meetings.md) -- **Presentations:** [DMAC Tech Webinar](https://www.youtube.com/watch?v=M8E2wQk3k8A) | [Pangeo Showcase](https://discourse.pangeo.io/t/pangeo-showcase-improving-access-to-noaa-national-ocean-service-model-data-with-kerchunk-and-xpublish/3725) - -- **Demos:** [XREDS Test](https://nextgen-dev.ioos.us/xreds/) | [OceansMap Test](https://dev.oceansmap.com/maracoos/?catalog=dmac_combined) - -- **Notebooks:** [NetCDF vs Zarr](https://github.com/asascience-open/nextgen-dmac/blob/main/binder/dbofs-examples/netcdf-vs-zarr.ipynb) | [Best Forecast Retrieval](https://github.com/asascience-open/nextgen-dmac/blob/main/binder/dbofs-examples/best-forecast.ipynb) | [CORA 40 Year WL Reanalysis](https://github.com/asascience/ncdis) | [National Water Model](https://github.com/asascience-open/NWM) - -- [**Meeting Minutes**](meetings.md) - -- **[Start a Discussion](https://github.com/asascience-open/nextgen-dmac/discussions)** +**[Start a Discussion](https://github.com/asascience-open/nextgen-dmac/discussions)** ## Background diff --git a/docs/outcomes.md b/docs/outcomes.md index 3da9381..406f41e 100644 --- a/docs/outcomes.md +++ b/docs/outcomes.md @@ -6,31 +6,39 @@ nav_order: 3 # Prototype Outcomes -Prototypes explored by RPS Tetra Tech: - -- [Kubernetes infrastructure](../architecture/kubernetes.md) - - [Configuration](../../k8s/README.md) -- [RabbitMQ Messaging](https://github.com/asascience-open/nextgen-dmac/tree/main/messaging) -- [JupyterHub w/ Dask on K8s](https://github.com/asascience-open/nextgen-dmac/tree/main/jupyterhub) -- [RESTful Grids (2022 IOOS Code Sprint)](https://github.com/asascience-open/restful-grids) -- Argo Workflows - - [Test QARTOD and IOOS Compliance Checker](https://github.com/asascience-open/nextgen-dmac/tree/main/qc_and_cchecker#readme) - - [Kerchunk Ingest for NOS model data](https://github.com/asascience-open/nextgen-dmac/tree/main/kerchunk#readme) -- [Dask Gateway](https://github.com/asascience-open/dask-gateway) - - [Configuration](https://github.com/asascience-open/nextgen-dmac/tree/main/dask) -- [XPublish Web Mapping Service (WMS)](https://github.com/xpublish-community/xpublish-wms) +## Prototypes + +### Data Engineering and Services + +| Component | Documentation | Code | Description | +| --------- | ---- | ------------ | ----------- | +| Lambda-based Kerchunk Ingest | [Documentation](./ingest/ingest-prototype.md) | [Code](https://github.com/asascience-open/nextgen-dmac/tree/main/cloud_aggregator) | Executing kerchunk workflows using AWS Lambda and SNS/SQS | +| XPublish Web Mapping Service (WMS) | [Documentation](https://github.com/xpublish-community/xpublish-wms#readme) | [Code](https://github.com/xpublish-community/xpublish-wms) | Retrieve web map tiles directly from environmental data | +| RESTful Grids (2022 IOOS Code Sprint) | [Documentation](https://ioos.github.io/ioos-code-sprint/2022/topics/05-restful-services-for-gridded-data.html) | [Code](https://github.com/asascience-open/restful-grids) | Initial project investigating XPublish | +| Argo Workflows: Test QARTOD and IOOS Compliance Checker | [Documentation](https://github.com/asascience-open/nextgen-dmac/tree/main/qc_and_cchecker#readme) | [Code](https://github.com/asascience-open/nextgen-dmac/tree/main/qc_and_cchecker) | Use of standard IOOS tools within a cloud workflow environment | +| Argo Workflows: Kerchunk Ingest for NOS model data | [Documentation](https://github.com/asascience-open/nextgen-dmac/tree/main/kerchunk#readme) | [Code](https://github.com/asascience-open/nextgen-dmac/tree/main/kerchunk) | Executing kerchunk workflows within Argo Workflows platform | +| HRRR Aggregator | [Documentation](https://github.com/asascience-open/nextgen-dmac/tree/main/hrrr_aggregator#readme) | [Code](https://github.com/asascience-open/nextgen-dmac/tree/main/hrrr_aggregator) | Initial prototype to build large time-based aggregations by David Stuebe, Camus Energy | +| GRIB Index Aggregation | [Documentation](https://github.com/asascience-open/nextgen-dmac/tree/main/grib_index_aggregation#readme) | [Code](https://github.com/asascience-open/nextgen-dmac/tree/main/grib_index_aggregation) | Updated prototype to optimize large time-based aggregations for GRIB files by David Stuebe, Camus Energy | + +### Infrastructure + +| Component | Documentation | Code | Description | +| --------- | ---- | ------------ | ----------- | +| Kubernetes | [Documentation](../architecture/kubernetes.md) | [Configuration](../../k8s/README.md) | Demonstrated and learned from running K8s in AWS as a potential computing platform +| Pulumi Infrastructure-as-Code | [Documentation](https://www.pulumi.com/docs/) | [Configuration](https://github.com/asascience-open/nextgen-dmac/tree/main/cloud_aggregator#cloud-aggregator) | Using Pulumi to initialize cloud resources needed to run kerchunk ingest | +| RabbitMQ Messaging | [Documentation](./ingest/events.md#rabbitmq) | [Code](https://github.com/asascience-open/nextgen-dmac/tree/main/messaging) | Demonstrates platform-independent messaging | +| JupyterHub w/ Dask on K8s | [Documentation](https://z2jh.jupyter.org/en/stable/) | [Configuration](https://github.com/asascience-open/nextgen-dmac/tree/main/jupyterhub) | Initial experiments running Jupyterhub on K8s (later adopted Nebari) | +| Dask Gateway | [Documentation](https://gateway.dask.org/install-kube.html) | [Configuration](https://github.com/asascience-open/nextgen-dmac/tree/main/dask) | Running a Dask cluster on K8s | + + +## Demos and Notebooks + - [XREDS Data Viewer](https://nextgen-dev.ioos.us/xreds/) -- [Pulumi Infrastructure-as-Code](https://github.com/asascience-open/nextgen-dmac/tree/main/cloud_aggregator#readme) - [XPublish Notebook Examples](https://github.com/asascience-open/nextgen-ioos-2023) -Other RPS Tetra Tech projects: +## Related Projects - [NOAA Coastal Ocean Reanalysis (CORA)](../analysis/cora.md) - [National Water Model](https://github.com/asascience-open/NWM) - [HPC Cloud Subsetting](https://github.com/asascience-open/HPC-Cloud-Subsetting) - -Contributed by the Community: - -- [XPublish](https://xpublish.readthedocs.io/en/latest/) -- [HRRR Aggregator](https://github.com/asascience-open/nextgen-dmac/tree/main/hrrr_aggregator) - - by David Stuebe, Camus Energy \ No newline at end of file +- [XPublish](https://xpublish.readthedocs.io/en/latest/) \ No newline at end of file