Skip to content

Commit

Permalink
Outcomes and index updates (#60)
Browse files Browse the repository at this point in the history
* Reorganized prototype outcomes and incorporated feedback

* Reorganized outcomes
  • Loading branch information
jonmjoyce authored Apr 24, 2024
1 parent 02b9222 commit 845d12c
Show file tree
Hide file tree
Showing 2 changed files with 38 additions and 32 deletions.
20 changes: 9 additions & 11 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,19 +8,17 @@ nav_order: 1
### Cloud-Optimized Data Management and Cyberinfrastructure
#### A collaboration between [RPS, a Tetra Tech Company](https://www.rpsgroup.com/services/oceans-and-coastal/) and the NOAA Integrated Ocean Observing System ([IOOS](https://ioos.noaa.gov/))

- [**Project Documentation**](https://asascience-open.github.io/nextgen-dmac/)
| | | |
|-|-|-|
| **Prototypes** | [Planning and Design](protoplan.md) | [Outcomes](outcomes.md) |
| **Presentations** | [DMAC Tech Webinar](https://www.youtube.com/watch?v=M8E2wQk3k8A) | [Pangeo Showcase](https://discourse.pangeo.io/t/pangeo-showcase-improving-access-to-noaa-national-ocean-service-model-data-with-kerchunk-and-xpublish/3725) |
| **Demos** | [XREDS Test](https://nextgen-dev.ioos.us/xreds/) | [OceansMap Test](https://dev.oceansmap.com/maracoos/?catalog=dmac_combined) |
| **Notebooks** | [NetCDF vs Zarr](https://github.com/asascience-open/nextgen-dmac/blob/main/binder/dbofs-examples/netcdf-vs-zarr.ipynb) | [Best Forecast Retrieval](https://github.com/asascience-open/nextgen-dmac/blob/main/binder/dbofs-examples/best-forecast.ipynb) |
| | [CORA 40 Year WL Reanalysis](https://github.com/asascience/ncdis) | [National Water Model](https://github.com/asascience-open/NWM) |

- **Prototypes:** [Planning and Design](protoplan.md) | [Outcomes](outcomes.md)
[**Meeting Minutes**](meetings.md)

- **Presentations:** [DMAC Tech Webinar](https://www.youtube.com/watch?v=M8E2wQk3k8A) | [Pangeo Showcase](https://discourse.pangeo.io/t/pangeo-showcase-improving-access-to-noaa-national-ocean-service-model-data-with-kerchunk-and-xpublish/3725)

- **Demos:** [XREDS Test](https://nextgen-dev.ioos.us/xreds/) | [OceansMap Test](https://dev.oceansmap.com/maracoos/?catalog=dmac_combined)

- **Notebooks:** [NetCDF vs Zarr](https://github.com/asascience-open/nextgen-dmac/blob/main/binder/dbofs-examples/netcdf-vs-zarr.ipynb) | [Best Forecast Retrieval](https://github.com/asascience-open/nextgen-dmac/blob/main/binder/dbofs-examples/best-forecast.ipynb) | [CORA 40 Year WL Reanalysis](https://github.com/asascience/ncdis) | [National Water Model](https://github.com/asascience-open/NWM)

- [**Meeting Minutes**](meetings.md)

- **[Start a Discussion](https://github.com/asascience-open/nextgen-dmac/discussions)**
**[Start a Discussion](https://github.com/asascience-open/nextgen-dmac/discussions)**


## Background
Expand Down
50 changes: 29 additions & 21 deletions docs/outcomes.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,31 +6,39 @@ nav_order: 3

# Prototype Outcomes

Prototypes explored by RPS Tetra Tech:

- [Kubernetes infrastructure](../architecture/kubernetes.md)
- [Configuration](../../k8s/README.md)
- [RabbitMQ Messaging](https://github.com/asascience-open/nextgen-dmac/tree/main/messaging)
- [JupyterHub w/ Dask on K8s](https://github.com/asascience-open/nextgen-dmac/tree/main/jupyterhub)
- [RESTful Grids (2022 IOOS Code Sprint)](https://github.com/asascience-open/restful-grids)
- Argo Workflows
- [Test QARTOD and IOOS Compliance Checker](https://github.com/asascience-open/nextgen-dmac/tree/main/qc_and_cchecker#readme)
- [Kerchunk Ingest for NOS model data](https://github.com/asascience-open/nextgen-dmac/tree/main/kerchunk#readme)
- [Dask Gateway](https://github.com/asascience-open/dask-gateway)
- [Configuration](https://github.com/asascience-open/nextgen-dmac/tree/main/dask)
- [XPublish Web Mapping Service (WMS)](https://github.com/xpublish-community/xpublish-wms)
## Prototypes

### Data Engineering and Services

| Component | Documentation | Code | Description |
| --------- | ---- | ------------ | ----------- |
| Lambda-based Kerchunk Ingest | [Documentation](./ingest/ingest-prototype.md) | [Code](https://github.com/asascience-open/nextgen-dmac/tree/main/cloud_aggregator) | Executing kerchunk workflows using AWS Lambda and SNS/SQS |
| XPublish Web Mapping Service (WMS) | [Documentation](https://github.com/xpublish-community/xpublish-wms#readme) | [Code](https://github.com/xpublish-community/xpublish-wms) | Retrieve web map tiles directly from environmental data |
| RESTful Grids (2022 IOOS Code Sprint) | [Documentation](https://ioos.github.io/ioos-code-sprint/2022/topics/05-restful-services-for-gridded-data.html) | [Code](https://github.com/asascience-open/restful-grids) | Initial project investigating XPublish |
| Argo Workflows: Test QARTOD and IOOS Compliance Checker | [Documentation](https://github.com/asascience-open/nextgen-dmac/tree/main/qc_and_cchecker#readme) | [Code](https://github.com/asascience-open/nextgen-dmac/tree/main/qc_and_cchecker) | Use of standard IOOS tools within a cloud workflow environment |
| Argo Workflows: Kerchunk Ingest for NOS model data | [Documentation](https://github.com/asascience-open/nextgen-dmac/tree/main/kerchunk#readme) | [Code](https://github.com/asascience-open/nextgen-dmac/tree/main/kerchunk) | Executing kerchunk workflows within Argo Workflows platform |
| HRRR Aggregator | [Documentation](https://github.com/asascience-open/nextgen-dmac/tree/main/hrrr_aggregator#readme) | [Code](https://github.com/asascience-open/nextgen-dmac/tree/main/hrrr_aggregator) | Initial prototype to build large time-based aggregations by David Stuebe, Camus Energy |
| GRIB Index Aggregation | [Documentation](https://github.com/asascience-open/nextgen-dmac/tree/main/grib_index_aggregation#readme) | [Code](https://github.com/asascience-open/nextgen-dmac/tree/main/grib_index_aggregation) | Updated prototype to optimize large time-based aggregations for GRIB files by David Stuebe, Camus Energy |

### Infrastructure

| Component | Documentation | Code | Description |
| --------- | ---- | ------------ | ----------- |
| Kubernetes | [Documentation](../architecture/kubernetes.md) | [Configuration](../../k8s/README.md) | Demonstrated and learned from running K8s in AWS as a potential computing platform
| Pulumi Infrastructure-as-Code | [Documentation](https://www.pulumi.com/docs/) | [Configuration](https://github.com/asascience-open/nextgen-dmac/tree/main/cloud_aggregator#cloud-aggregator) | Using Pulumi to initialize cloud resources needed to run kerchunk ingest |
| RabbitMQ Messaging | [Documentation](./ingest/events.md#rabbitmq) | [Code](https://github.com/asascience-open/nextgen-dmac/tree/main/messaging) | Demonstrates platform-independent messaging |
| JupyterHub w/ Dask on K8s | [Documentation](https://z2jh.jupyter.org/en/stable/) | [Configuration](https://github.com/asascience-open/nextgen-dmac/tree/main/jupyterhub) | Initial experiments running Jupyterhub on K8s (later adopted Nebari) |
| Dask Gateway | [Documentation](https://gateway.dask.org/install-kube.html) | [Configuration](https://github.com/asascience-open/nextgen-dmac/tree/main/dask) | Running a Dask cluster on K8s |


## Demos and Notebooks

- [XREDS Data Viewer](https://nextgen-dev.ioos.us/xreds/)
- [Pulumi Infrastructure-as-Code](https://github.com/asascience-open/nextgen-dmac/tree/main/cloud_aggregator#readme)
- [XPublish Notebook Examples](https://github.com/asascience-open/nextgen-ioos-2023)

Other RPS Tetra Tech projects:
## Related Projects

- [NOAA Coastal Ocean Reanalysis (CORA)](../analysis/cora.md)
- [National Water Model](https://github.com/asascience-open/NWM)
- [HPC Cloud Subsetting](https://github.com/asascience-open/HPC-Cloud-Subsetting)

Contributed by the Community:

- [XPublish](https://xpublish.readthedocs.io/en/latest/)
- [HRRR Aggregator](https://github.com/asascience-open/nextgen-dmac/tree/main/hrrr_aggregator)
- by David Stuebe, Camus Energy
- [XPublish](https://xpublish.readthedocs.io/en/latest/)

0 comments on commit 845d12c

Please sign in to comment.