From 4af33125e08e12e5f15c9074d3931075a0be8747 Mon Sep 17 00:00:00 2001 From: klsheets Date: Tue, 2 Jan 2024 03:15:54 -0500 Subject: [PATCH 1/3] Update wis2node.adoc For Review: Updates to 2.2.1 Practices and Procedures, 2.2.2 Publishing data, metadata, discovery metadata and notification messages, and 2.2.3 Performance metrics. All sections likely will need formatting reviews. Some things I noted while working are when do we use WIS vs WIS2 - specifically about nodes, but I think there are few other places where they are used interchangeably. For performance metrics, the template asked specifically for WIS nodes and not global services. We focused more on the performance metrics for global services, I can add these if desired, but I stuck to nodes and drew more on what happens currently for GISCs and GISC watch for what I wrote. --- guide/sections/part2/wis2node.adoc | 56 +++++++++++++++++++++++------- 1 file changed, 44 insertions(+), 12 deletions(-) diff --git a/guide/sections/part2/wis2node.adoc b/guide/sections/part2/wis2node.adoc index 37e8265..136c374 100644 --- a/guide/sections/part2/wis2node.adoc +++ b/guide/sections/part2/wis2node.adoc @@ -4,24 +4,30 @@ ===== Registration and decommissioning of a WIS node -This section describes the process used to register or remove a WIS node within WIS2. During the initial part of the WIS2 pilot phase, a Member simply needs to notify the WMO Secretariat and primary GISC of the intent to register a new WIS node. The Secretariat and GISC will then assist in the registration. More formal procedures will be developed as the number of WIS nodes increases. +Registration and decomissionig of WIS nodes must be approved by the PR for the centre registering or decomissioning a WIS node. -TODO: To be completed +Procedure for PR Approved WIS node Registration is to create a Centre-ID based on naming convention ab-domain-nodename. Where ab is the the ISO3166 2-lettre country code, domain is the organization's domain name for the main website, and nodename is descriptive of the purpose of the WIS node (for example a node for a type of data or programme of the data e.g. climate, aviation, etc.) A sample centre-ID is ab-myorg-climate. In addition to the centre-ID the registering organization should also provide the broker endpoints. Once centre-ID and broker endpoints are provided and entered into the WIS2 node database, the WMO will contact the GISC for the organization's country and request the GISC to verify the correctnes of the provided information. The GISC will request the centre produce a test message with associated files for download. The GISC should use a test MQTT client to verify the notificaion message is correct and the download links are functional. The GISC notifies the Centre of the results of the checks and if all are good, requests the Centre to provide metadata when the dataset is ready. When metadata is proivided the GISC informs the WMO the new centre-ID is read and requests it be added to WIS2. + + +Procedure for PR approved decomissioning of a WIS node is the PR (or designate) will notify the WMO of the decomissioning of the WIS node. When possible, 30 days notice should be provided prior to the decomissioning and information on if the data from the node will also be decomissioned or if it is available via another WIS node or method. The WMO will notify the GISC for the country and global data catalogues, global brokers and global data caches of the date of the WIS node decomissioning. ===== Registration and removal of a Dataset -This section describes the process used to register a Dataset so that it may be discovered and shared within WIS2. In cases where a WIS Centre no longer wishes to share a Dataset via WIS, it must be removed as per the procedure described here. +All data shared in WIS2 must include metadata compliant with the metadata requriements described below. Data providers must be approved by their PR (or designate such as the member's WIS Focal Point) to ensure quality metadata will be provied along with the data. + +An organization ready to publish a new dataset should contact the WMO with the approval of their PR (or designate) to The WMO. The WMO will contact the GISC for the organizaiton with the new (meta)data. The GISC will verify the metadata meets the WIS2 requirements and will work with the data provider until the metadata is compliant. + -TODO: to be completed ===== Connecting with Global Services -This section describes the process by which a WIS node is registered with one or more Global Broker and Global Cache components. +Once a WIS node has been verified by a GISC and endpoints and metadata are available, the WMO provides the new centre-ID to the Global Brokers and requests they subscribe to the new broker endpoint. The Global Broker subscribes to the agreed upon topics. The Global Caches download and cache the metadata (and data where applicable for core datasets). -TODO: to be completed +Once a GISC verifies the metadata for a new dataset, the WMO will be notified. The WMO will then provide the topics to the Global Brokers (and Global Caches for core data). The Global Data Catalogues will also be notified of the new data, so they can verify the catalogue is discovering the new dataset. ==== Publishing data, discovery metadata, and notification messages + ===== Discovery metadata Discovery metadata shall be encoded according to the WMO Core Metadata Profile version 2 (WCMP2). See WCMP2 @@ -73,7 +79,7 @@ WIS-TECHSPEC-2 states: When using a Web API to publish "core" data, the URL included in the data availability notification message must be directly resolvable, i.e., the Data Consumer must not be required to complete any additional fields in the API request. This can be achieved by identifying the data object in the URL. A Data Consumer or a Global Cache instance can simply resolve the URL to download the data object regardless of the manner in which it is made available. -WIS2 isn’t yet mature enough to prescribe the use of particular Web APIs. Instead, WIS2 seeks to leverage the experience of Data Publishers who have been using Web APIs to serve their communities. +WIS2 seeks to leverage the experience of Data Publishers who have been using Web APIs to serve their communities. First, interactive Web APIs should be self-describing. A Data Consumer should not need to know, a priori, how to make requests from a Web API. They should be able to discover this information from the Web API endpoint itself – even if this is just a link to a documentation page they need to read. @@ -83,25 +89,51 @@ Third, the Open Geospatial Consortium (OGC) have developed a suite of APIs (call Finally, we’re increasingly concerned with providing access to very large Datasets. The OGC has published a series of informative blogs on the subject of cloud-native geospatial data sharing. These are listed among in section 11.4.2 Informative References. -TODO: to be completed ====== Publication and topic selection When publishing a dataset, a data publisher selects a given topic according to the WIS Topic Hierarchy. Given the multidisciplinary nature of some data, a data publisher must select a single topic for publication purposes, and always uses WCMP2 discovery metadata to provide a fulsome description of their dataset and its relevance to additional disciplines. +Metadata is the method by which datasets are ultimately made available in the WIS2 system. The goal is for data providers who have PR authorization to have a lightweight methode to provide their datasets to WIS. With this goal in mind, there are several acceptable methods to publish metadata: + + Option 1: deploy a WIS2 node + + Option 2: a MQTT broker and HTTP server + + Option 3: q metadata publication service which can be provided by GISCs, NMHS or through a WIS2 portal + + +For infrequently updated datasets the following process should be followed: + + Publish initial metadata. + + Publish update metadata. + + Data update notification: normal notification message with property.cache=false + + ==== Performance management ===== Service levels and performance indicators -This section describes the minimum performance criteria for operation of a WIS node. +A WIS2 node must be able to: -TODO: to be completed + publish datasets and compliant metadata and discovery metadata + + * Publish metadata to the Global Data Catalogue + * Publish core data to the Global Cache + * Publish data for consumer access + * data embedded in a message (CAP/warning) + * Receive metadata publication errors from the Global Data Catalogue + + Support IP filtering + Provide metadata with topics to Global Brokers ===== Provision of system performance metrics -This section describes how a WIS node should provide metrics to the Global Monitor service and its primary GISC. +WIS nodes should provide annual performance metrics to their GISC. -TODO: to be completed +If contacted by the Global Montior via GISCC for a performance issue, the WIS node should provide metrics to the GISC and Global Monitor when service is restored to indicate resolution of the issue. ==== WIS Node reference implementation: wis2box From efd5b584aab4198effbbfad92722fb8d3e0eb5fe Mon Sep 17 00:00:00 2001 From: klsheets Date: Tue, 2 Jan 2024 23:33:39 -0500 Subject: [PATCH 2/3] Update wis2node.adoc Made edits based on suggestions from Jeremy @6a6d74 on 2 January. --- guide/sections/part2/wis2node.adoc | 32 +++++++++++++++++++----------- 1 file changed, 20 insertions(+), 12 deletions(-) diff --git a/guide/sections/part2/wis2node.adoc b/guide/sections/part2/wis2node.adoc index 136c374..55d3da5 100644 --- a/guide/sections/part2/wis2node.adoc +++ b/guide/sections/part2/wis2node.adoc @@ -6,24 +6,27 @@ Registration and decomissionig of WIS nodes must be approved by the PR for the centre registering or decomissioning a WIS node. -Procedure for PR Approved WIS node Registration is to create a Centre-ID based on naming convention ab-domain-nodename. Where ab is the the ISO3166 2-lettre country code, domain is the organization's domain name for the main website, and nodename is descriptive of the purpose of the WIS node (for example a node for a type of data or programme of the data e.g. climate, aviation, etc.) A sample centre-ID is ab-myorg-climate. In addition to the centre-ID the registering organization should also provide the broker endpoints. Once centre-ID and broker endpoints are provided and entered into the WIS2 node database, the WMO will contact the GISC for the organization's country and request the GISC to verify the correctnes of the provided information. The GISC will request the centre produce a test message with associated files for download. The GISC should use a test MQTT client to verify the notificaion message is correct and the download links are functional. The GISC notifies the Centre of the results of the checks and if all are good, requests the Centre to provide metadata when the dataset is ready. When metadata is proivided the GISC informs the WMO the new centre-ID is read and requests it be added to WIS2. +Procedure for PR Approved WIS node Registration is to create a Centre-ID based on naming convention ab-domain-nodename. Where ab is the the IANA Top Level Domain code, domain is the organization's domain name for the main website, and nodename (optional) is descriptive of the purpose of the WIS node (for example a node for a type of data or programme of the data e.g. climate, aviation, etc.) when the organization is hosting multiple WIS nodes. A sample centre-ID is ab-myorg-climate. In addition to the centre-ID the registering organization should also provide the broker endpoints. Once centre-ID and broker endpoints are provided and entered into the WIS2 node database, the WMO Secretariat will contact the GISC for the organization's country and request the GISC to verify the correctness of the provided information. The GISC will request the centre produce a test message with associated files for download. The GISC should use a test MQTT client to verify the notification message is correct and the download links are functional. The GISC notifies the Centre of the results of the checks and if all are good, requests the Centre to provide metadata when the dataset is ready. When metadata is proivided the GISC informs the WMO Secretariat that the new centre-ID is ready and requests it be added to WIS2. -Procedure for PR approved decomissioning of a WIS node is the PR (or designate) will notify the WMO of the decomissioning of the WIS node. When possible, 30 days notice should be provided prior to the decomissioning and information on if the data from the node will also be decomissioned or if it is available via another WIS node or method. The WMO will notify the GISC for the country and global data catalogues, global brokers and global data caches of the date of the WIS node decomissioning. +In the case of NCs, the procedure for PR approved decomissioning of a WIS node is the PR (or designate) will notify the WMO Secretariat of the decomissioning of the WIS node. In the case of DCPCs, the sponsor (i.e., Regional Association or WMO Programme) is the Programme Chair or Regional Information Management Chair shall approve the decommissioning and notify the WMO Secretariat of the decomissioning of the WIS node. When possible, 30 days notice should be provided prior to the decomissioning and information on if the data from the node will also be decomissioned or if it is available via another WIS node or method. The WMO Secretariat will notify the GISC for the country and the Global Service Providers of the date of the WIS node decomissioning. + +NC/DCPCs operators decomissioning a WIS node shall ensure that obligations relating to data sharing within WIS continue to be met after the WIS Node is decommissioned, for example, by migrating these data sharing obligations to another WIS Node. In the case of DCPCs, this may mean the WIS node responsibilities shift from one member to another and in the case these details should be included in the decomissioning notice to the WMO Serectariat. ===== Registration and removal of a Dataset All data shared in WIS2 must include metadata compliant with the metadata requriements described below. Data providers must be approved by their PR (or designate such as the member's WIS Focal Point) to ensure quality metadata will be provied along with the data. -An organization ready to publish a new dataset should contact the WMO with the approval of their PR (or designate) to The WMO. The WMO will contact the GISC for the organizaiton with the new (meta)data. The GISC will verify the metadata meets the WIS2 requirements and will work with the data provider until the metadata is compliant. - +An organization ready to publish a new dataset should contact the WMO Secretariat with the approval of their PR (or designate) to The WMO Secretariat. The WMO Secretariat will contact the GISC for the organizaiton with the new (meta)data. The GISC will work with a Global Data Catalogue to verify the metadata. the GDC will publish a report indicating errors and/or potential improvements (based on discovery metadata KPIs). The GISC should work with the data publisher to remedy issues and incporate suggestions for improvement. +In addition, the data publisher's affiliated GISC conduct a systematic review of what's being published to make sure everything is functional. ===== Connecting with Global Services -Once a WIS node has been verified by a GISC and endpoints and metadata are available, the WMO provides the new centre-ID to the Global Brokers and requests they subscribe to the new broker endpoint. The Global Broker subscribes to the agreed upon topics. The Global Caches download and cache the metadata (and data where applicable for core datasets). +Once a WIS node has been verified by a GISC and endpoints and metadata are available, the WMO Secretariat provides the new centre-ID to the Global Brokers and requests they subscribe to the new broker endpoint. The Global Broker will recieve the data based on their topic subscriptions. The Global Caches download and cache the metadata (and data where applicable for core datasets). + +Once a GISC, in partnership with a Global Data Catalogue, verifies the metadata for a new dataset, the WMO Secretariat will be notified of the availability of the new dataset. The WMO Seretariat will then notify the Global Borkers and Global Caches of the addition of new data to the WIS system. -Once a GISC verifies the metadata for a new dataset, the WMO will be notified. The WMO will then provide the topics to the Global Brokers (and Global Caches for core data). The Global Data Catalogues will also be notified of the new data, so they can verify the catalogue is discovering the new dataset. ==== Publishing data, discovery metadata, and notification messages @@ -94,13 +97,13 @@ Finally, we’re increasingly concerned with providing access to very large Data When publishing a dataset, a data publisher selects a given topic according to the WIS Topic Hierarchy. Given the multidisciplinary nature of some data, a data publisher must select a single topic for publication purposes, and always uses WCMP2 discovery metadata to provide a fulsome description of their dataset and its relevance to additional disciplines. -Metadata is the method by which datasets are ultimately made available in the WIS2 system. The goal is for data providers who have PR authorization to have a lightweight methode to provide their datasets to WIS. With this goal in mind, there are several acceptable methods to publish metadata: +Metadata is the method by which datasets are ultimately made available in the WIS2 system. The goal is for data providers who have PR authorization to have a lightweight method to provide their datasets to WIS. With this goal in mind, there are several acceptable methods to publish metadata: Option 1: deploy a WIS2 node Option 2: a MQTT broker and HTTP server - Option 3: q metadata publication service which can be provided by GISCs, NMHS or through a WIS2 portal + Option 3: a bilateral agreemnt for another organization to publish metadata publication on behalf of the data provider (potential organizations providing this service are GISCs and NMHS or potentional through a WIS2 portal in the future). For infrequently updated datasets the following process should be followed: @@ -111,6 +114,13 @@ For infrequently updated datasets the following process should be followed: Data update notification: normal notification message with property.cache=false +Use of the "experimental" topic + + The "experimental" topic is necessary for the WIS2 pre-operational phase and future pre-operational data exchange in test mode. + + The experimental topic sits under domain (level 8), e.g. ...weather/experimental. Data publishers can can extend the experimental branch with sub-topics as they deem appropriate. + + Data consumers must not assume that experimental topics will be durable (i.e., they may change or be removed). ==== Performance management @@ -118,14 +128,12 @@ For infrequently updated datasets the following process should be followed: A WIS2 node must be able to: - publish datasets and compliant metadata and discovery metadata - + Publish datasets and compliant metadata and discovery metadata * Publish metadata to the Global Data Catalogue * Publish core data to the Global Cache * Publish data for consumer access * data embedded in a message (CAP/warning) * Receive metadata publication errors from the Global Data Catalogue - Support IP filtering Provide metadata with topics to Global Brokers @@ -133,7 +141,7 @@ A WIS2 node must be able to: WIS nodes should provide annual performance metrics to their GISC. -If contacted by the Global Montior via GISCC for a performance issue, the WIS node should provide metrics to the GISC and Global Monitor when service is restored to indicate resolution of the issue. +If contacted by the Global Montior via GISC for a performance issue, the WIS node should provide metrics to the GISC and Global Monitor when service is restored to indicate resolution of the issue. ==== WIS Node reference implementation: wis2box From ddefc8cb6e28a957b81317884d80b3eda5326772 Mon Sep 17 00:00:00 2001 From: Tom Kralidis Date: Wed, 24 Jan 2024 21:03:50 -0500 Subject: [PATCH 3/3] Update wis2node.adoc --- guide/sections/part2/wis2node.adoc | 110 +++++++++++++---------------- 1 file changed, 50 insertions(+), 60 deletions(-) diff --git a/guide/sections/part2/wis2node.adoc b/guide/sections/part2/wis2node.adoc index 55d3da5..d6c4406 100644 --- a/guide/sections/part2/wis2node.adoc +++ b/guide/sections/part2/wis2node.adoc @@ -2,28 +2,27 @@ ==== Practices and procedures -===== Registration and decommissioning of a WIS node +===== Registration and decommissioning of a WIS Node -Registration and decomissionig of WIS nodes must be approved by the PR for the centre registering or decomissioning a WIS node. +Registration and decomissionig of WIS Nodes must be approved by the PR for the centre registering or decomissioning a WIS Node. -Procedure for PR Approved WIS node Registration is to create a Centre-ID based on naming convention ab-domain-nodename. Where ab is the the IANA Top Level Domain code, domain is the organization's domain name for the main website, and nodename (optional) is descriptive of the purpose of the WIS node (for example a node for a type of data or programme of the data e.g. climate, aviation, etc.) when the organization is hosting multiple WIS nodes. A sample centre-ID is ab-myorg-climate. In addition to the centre-ID the registering organization should also provide the broker endpoints. Once centre-ID and broker endpoints are provided and entered into the WIS2 node database, the WMO Secretariat will contact the GISC for the organization's country and request the GISC to verify the correctness of the provided information. The GISC will request the centre produce a test message with associated files for download. The GISC should use a test MQTT client to verify the notification message is correct and the download links are functional. The GISC notifies the Centre of the results of the checks and if all are good, requests the Centre to provide metadata when the dataset is ready. When metadata is proivided the GISC informs the WMO Secretariat that the new centre-ID is ready and requests it be added to WIS2. +Procedure for PR Approved WIS Node Registration is to create a centre identifier based on naming convention `ab-domain-nodename`. Where `ab` is the the IANA Top Level Domain (TLD) code, domain is the organization's domain name for the main website, and nodename (optional) is descriptive of the purpose of the WIS Node (for example a node for a type of data or programme of the data e.g. climate, aviation, etc.) when the organization is hosting multiple WIS Nodes. A sample centre identifier is `ab-myorg-climate`. In addition to the centre identifier the registering organization should also provide the broker endpoints. Once centre identifier and broker endpoints are provided and entered into the WIS2 Node register, the WMO Secretariat will contact the GISC for the organization's country and request the GISC to verify the correctness of the provided information. The GISC will request the centre produce a test message with associated files for download. The GISC should use a test MQTT client to verify the notification message is correct and the download links are functional. The GISC notifies the Centre of the results of the checks and if all are good, requests the Centre to provide metadata when the dataset is ready. When metadata is proivided the GISC informs the WMO Secretariat that the new centre identifier is ready and requests it be added to WIS2. +In the case of NCs, the procedure for PR approved decomissioning of a WIS Node is the PR (or designate) will notify the WMO Secretariat of the decomissioning of the WIS Node. In the case of DCPCs, the sponsor (i.e., Regional Association or WMO Programme) is the Programme Chair or Regional Information Management Chair shall approve the decommissioning and notify the WMO Secretariat of the decomissioning of the WIS Node. Where possible, a 30 day notice period should be provided prior to the decomissioning and information on if the data from the Node will also be decomissioned or if it is available via another WIS Node or method. The WMO Secretariat will notify the GISC for the country and the Global Service Providers of the date of the WIS Node decomissioning. -In the case of NCs, the procedure for PR approved decomissioning of a WIS node is the PR (or designate) will notify the WMO Secretariat of the decomissioning of the WIS node. In the case of DCPCs, the sponsor (i.e., Regional Association or WMO Programme) is the Programme Chair or Regional Information Management Chair shall approve the decommissioning and notify the WMO Secretariat of the decomissioning of the WIS node. When possible, 30 days notice should be provided prior to the decomissioning and information on if the data from the node will also be decomissioned or if it is available via another WIS node or method. The WMO Secretariat will notify the GISC for the country and the Global Service Providers of the date of the WIS node decomissioning. +NC/DCPCs operators decomissioning a WIS Node shall ensure that obligations relating to data sharing within WIS continue to be met after the WIS Node is decommissioned, for example, by migrating these data sharing obligations to another WIS Node. In the case of DCPCs, this may mean the WIS Node responsibilities shift from one member to another and in the case these details should be included in the decomissioning notice to the WMO Serectariat. -NC/DCPCs operators decomissioning a WIS node shall ensure that obligations relating to data sharing within WIS continue to be met after the WIS Node is decommissioned, for example, by migrating these data sharing obligations to another WIS Node. In the case of DCPCs, this may mean the WIS node responsibilities shift from one member to another and in the case these details should be included in the decomissioning notice to the WMO Serectariat. - -===== Registration and removal of a Dataset +===== Registration and removal of a dataset All data shared in WIS2 must include metadata compliant with the metadata requriements described below. Data providers must be approved by their PR (or designate such as the member's WIS Focal Point) to ensure quality metadata will be provied along with the data. -An organization ready to publish a new dataset should contact the WMO Secretariat with the approval of their PR (or designate) to The WMO Secretariat. The WMO Secretariat will contact the GISC for the organizaiton with the new (meta)data. The GISC will work with a Global Data Catalogue to verify the metadata. the GDC will publish a report indicating errors and/or potential improvements (based on discovery metadata KPIs). The GISC should work with the data publisher to remedy issues and incporate suggestions for improvement. +An organization ready to publish a new dataset should contact the WMO Secretariat with the approval of their PR (or designate) to The WMO Secretariat. The WMO Secretariat will contact the GISC for the organizaiton with the new (meta)data. The GISC will work with a Global Discovery Catalogue to verify the metadata. the GDC will publish a report indicating errors and/or potential improvements (based on discovery metadata KPIs). The GISC should work with the data publisher to remedy issues and incporate suggestions for improvement. In addition, the data publisher's affiliated GISC conduct a systematic review of what's being published to make sure everything is functional. ===== Connecting with Global Services -Once a WIS node has been verified by a GISC and endpoints and metadata are available, the WMO Secretariat provides the new centre-ID to the Global Brokers and requests they subscribe to the new broker endpoint. The Global Broker will recieve the data based on their topic subscriptions. The Global Caches download and cache the metadata (and data where applicable for core datasets). +Once a WIS Node has been verified by a GISC and endpoints and metadata are available, the WMO Secretariat provides the new centre identifier to the Global Brokers and requests they subscribe to the new broker endpoint. The Global Broker will recieve the data based on their topic subscriptions. The Global Caches download and cache the metadata (and data where applicable for core datasets). Once a GISC, in partnership with a Global Data Catalogue, verifies the metadata for a new dataset, the WMO Secretariat will be notified of the availability of the new dataset. The WMO Seretariat will then notify the Global Borkers and Global Caches of the addition of new data to the WIS system. @@ -33,9 +32,9 @@ Once a GISC, in partnership with a Global Data Catalogue, verifies the metadata ===== Discovery metadata -Discovery metadata shall be encoded according to the WMO Core Metadata Profile version 2 (WCMP2). See WCMP2 +Discovery metadata shall be encoded according to the WMO Core Metadata Profile version 2 (WCMP2). -Discovery metadata may be published one of two ways. The simplest method is to encode the discovery metadata record as a file and publish it to an HTTP server. The URL of this file is included in the notification message advertising the availability of new metadata. Alternatively, a Data Publisher may choose to host a local catalogue themselves, enabling them to share discovery metadata records through an API (e.g., OGC API Records). In this case, the URL used in the notification message will refer to the API endpoint identifying the specific discovery metadata record (e.g., an item as part of their discovery metadata catalogue/collection). +Discovery metadata may be published one of two ways. The simplest method is to encode the discovery metadata record as a file and publish it to an HTTP server. The URL of this file is included in the notification message advertising the availability of new metadata. Alternatively, a data publisher may choose to host a local catalogue themselves, enabling them to share discovery metadata records through an API (e.g., OGC API - Records). In this case, the URL used in the notification message will refer to the API endpoint identifying the specific discovery metadata record (e.g., an item as part of their discovery metadata catalogue). These discovery metadata records are then propagated through the Global Service components into to the Global Discovery Catalogue where Data Consumers can search and browse. @@ -45,35 +44,35 @@ TODO: to be completed ===== Notification messages -There is no requirement for an NC/DCPC to publish "data availability" notification messages relating to infrequently changing Datasets, such as a data archive, especially where the user community have no requirement to be instantly updated about changes to a Dataset (e.g., the addition of new records into a climate observation archive). Data Publishers should note that without providing notification messages their data will not be copied into the Global Cache. However, since the Global Cache only holds real-time (or near real-time) Datasets, this is not a concern for Data Publishers with infrequently changing Datasets. +There is no requirement for an NC/DCPC to publish "data availability" notification messages relating to infrequently changing datasets, such as a data archive, especially where the user community have no requirement to be instantly updated about changes to a dataset (e.g., the addition of new records into a climate observation archive). Data publishers should note that without providing notification messages their data will not be copied into the Global Cache. However, since the Global Cache only holds real-time (or near real-time) datasets, this is not a concern for data publishers with infrequently changing datasets. TODO: to be completed ===== Data -WIS2 provides the "plumbing" for data sharing within the WMO community, but it defines neither which data to share, nor how that data should be encoded. WIS Centres need to evaluate WMO Programme requirements and the WMO Unified Data Policy to determine which Datasets should be made available through WIS. +WIS2 provides the "plumbing" for data sharing within the WMO community, but it defines neither which data to share, nor how that data should be encoded. WIS Centres need to evaluate WMO Programme requirements and the WMO Unified Data Policy to determine which datasets should be made available through WIS. WMO Technical Regulations may require that data is encoded in particular formats. For example: synoptic observations should be encoded in BUFR. The Manual on Codes (WMO No. 306) provides details of data formats formally approved for use in WMO. -However, Technical Regulations don’t cover all data sharing requirements. In such cases, Data Publishers should select data formats that are widely adopted and understood in their target user community. +However, Technical Regulations don’t cover all data sharing requirements. In such cases, data publishers should select data formats that are widely adopted and understood in their target user community. -WIS2 does not require the use of specific file-naming conventions. Where communities commonly use file-naming conventions (e.g., with embedded metadata), Data Publishers should ensure that adequate documentation is provided to users. Data Publishers cannot assume that users will understand (or respect) their file-naming rules – many Data Consumers will simply treat the filename as an opaque string. +WIS2 does not require the use of specific file-naming conventions. Where communities commonly use file-naming conventions (e.g., with embedded metadata), data publishers should ensure that adequate documentation is provided to users. data publishers cannot assume that users will understand (or respect) their file-naming rules – many Data Consumers will simply treat the filename as an opaque string. Data publishers also have choices about how they publish data. -As a minimum, Data Publishers may publish data objects (e.g., the atomic bits of data that comprise a Dataset) as files using a Web server (HTTP protocol) or FTP server (FTP protocol), using secure communications (e.g., HTTPS/SFTP). As each data object is published, a notification message should also be published to a topic in a message broker (see 4.3 Notification message format and structure, and WIS2 messages 4.4 Standard topic hierarchy). +As a minimum, data publishers may publish data objects (e.g., the atomic bits of data that comprise a dataset) as files using a Web server (HTTP protocol) or FTP server (FTP protocol), using secure communications (e.g., HTTPS/SFTP). As each data object is published, a notification message should also be published to a topic in a message broker (see 4.3 Notification message format and structure, and WIS2 messages 4.4 Standard topic hierarchy). -A Dataset (for example, a collection of climate model runs) may comprise thousands or more files. A Data Publisher may choose to provide users with a mechanism to browse through the set of files, enabling them to identify those which are relevant to them. Examples of such mechanisms include: +A dataset (for example, a collection of climate model runs) may comprise thousands or more files. A data publisher may choose to provide users with a mechanism to browse through the set of files, enabling them to identify those which are relevant to them. Examples of such mechanisms include, but are not limited to: -* Web Accessible Folders (WAF) – a Web-based folder structure listing the data object files by name . -* Spatio-Temporal Asset Catalog (STAC) – a common language based on GeoJSON to describe geospatial data files so that it can be easily indexed, discovered, and accessed. Freely available, open-source tools present STAC records (one for each data object file) through a Web-based, browse-able user interface. +* Web Accessible Folders (WAF): a Web-based folder structure listing the data object files by name +* SpatioTemporal Asset Catalog (STAC): a community standard based on GeoJSON to describe geospatial data files which can be easily indexed, browsed, and accessed. Free and open sourcr tools tools present STAC records (one for each data object file) through a Web-based, browse-able user interface -WAFs and STAC are provided to illustrate options. There is no requirement for a Data Publisher to provide any such browse-able user interface to their data. +WAFs and STAC are provided to illustrate options. There is no requirement for a data publisher to provide any such browse-able user interface to their data. Increasingly, interactive Web APIs are being used to provide access to datasets. Although requiring a little more sophistication to implement, a Web API provides significant advantages: -* Data Consumers can select and download only the parts of a dataset that they need – providing them will a smaller dataset subset to work with and reducing the burden on the Data Publisher’s network infrastructure. -* Data Consumers are insulated from the complexities of how a Data Publisher chooses to persist their data. The Web API can provide access to Datasets in a way that is easy for users to understand. +* Data Consumers can select and download only the parts of a dataset that they need – providing them will a smaller dataset subset to work with and reducing the burden on the data publisher’s network infrastructure. +* Data Consumers are insulated from the complexities of how a data publisher chooses to persist their data. The Web API can provide access to datasets in a way that is easy for users to understand. * A Web API may allow Data Consumers to download data in their preferred file format or encoding. WIS-TECHSPEC-2 states: @@ -82,16 +81,15 @@ WIS-TECHSPEC-2 states: When using a Web API to publish "core" data, the URL included in the data availability notification message must be directly resolvable, i.e., the Data Consumer must not be required to complete any additional fields in the API request. This can be achieved by identifying the data object in the URL. A Data Consumer or a Global Cache instance can simply resolve the URL to download the data object regardless of the manner in which it is made available. -WIS2 seeks to leverage the experience of Data Publishers who have been using Web APIs to serve their communities. - -First, interactive Web APIs should be self-describing. A Data Consumer should not need to know, a priori, how to make requests from a Web API. They should be able to discover this information from the Web API endpoint itself – even if this is just a link to a documentation page they need to read. +WIS2 seeks to leverage the experience of data publishers who have been using Web APIs to serve their communities. -Second, we recommend that Web APIs are compliant with OpenAPI version 3 or later. OpenAPI provides a standardised mechanism to describe the API. Effectively, OpenAPI provides metadata that describes the Web API endpoint. Tooling(free, commercial, etc.) is widely available that can read this metadata and automatically generate client applications to query the Web API. +First, interactive Web APIs should be self-describing. A Data Consumer should not need to know, apriori, how to make requests from a Web API. They should be able to discover this information from the Web API endpoint itself – even if this is just a link to a documentation page they need to read. -Third, the Open Geospatial Consortium (OGC) have developed a suite of APIs (called "OGC APIs") that are designed specifically to provide APIs for geospatial data workflows (discovery, vizualisation, access, processing/exploitation) – all of which build on OpenAPI v3. Among these, OGC API – Environmental Data Retrieval (EDR), OGC API – Features, and OGC API - Coverages are considered particularly useful. Because these are open standards, there is an ever-growing suite of software implementations (both free and commercial) that support them. We recommend that Data Publishers assess these open-standard API specifications to determine their suitability to for publishing their Datasets using APIs. +Second, we recommend that Web APIs are compliant with OpenAPI version 3 or later. OpenAPI provides a standardised mechanism to describe the API. Effectively, OpenAPI provides metadata that describes the Web API endpoint. Tooling (free and, commercial, etc.) is widely available that can read this metadata and automatically generate client applications to query the Web API. -Finally, we’re increasingly concerned with providing access to very large Datasets. The OGC has published a series of informative blogs on the subject of cloud-native geospatial data sharing. These are listed among in section 11.4.2 Informative References. +Third, the Open Geospatial Consortium (OGC) have developed a suite of APIs (called "OGC APIs") that are designed specifically to provide APIs for geospatial data workflows (discovery, vizualisation, access, processing/exploitation) – all of which build on OpenAPI v3. Among these, OGC API – Environmental Data Retrieval (EDR), OGC API – Features, and OGC API - Coverages are considered particularly useful. Because these are open standards, there is an ever-growing suite of software implementations (both free and proprietary) that support them. We recommend that data publishers assess these open-standard API specifications to determine their suitability to for publishing their datasets using APIs. +Finally, we are increasingly concerned with providing access to very large datasets. The OGC has published a series of informative blogs on the subject of cloud-native geospatial data sharing. These are listed among in section 11.4.2 Informative References TODO PROPER CROSSREF. ====== Publication and topic selection @@ -99,57 +97,51 @@ When publishing a dataset, a data publisher selects a given topic according to t Metadata is the method by which datasets are ultimately made available in the WIS2 system. The goal is for data providers who have PR authorization to have a lightweight method to provide their datasets to WIS. With this goal in mind, there are several acceptable methods to publish metadata: - Option 1: deploy a WIS2 node - - Option 2: a MQTT broker and HTTP server +- Option 1: deploy a WIS2 node +- Option 2: a MQTT broker and HTTP server +- Option 3: a bilateral agreemnt for another organization to publish metadata publication on behalf of the data provider (potential organizations providing this service are GISCs and NMHS or potentional through a WIS2 portal in the future). - Option 3: a bilateral agreemnt for another organization to publish metadata publication on behalf of the data provider (potential organizations providing this service are GISCs and NMHS or potentional through a WIS2 portal in the future). +For infrequently updated datasets the following process should be followed: +- Publish initial metadata +- Publish update metadata +- Data update notification: normal notification message with `property.cache=false` -For infrequently updated datasets the following process should be followed: +===== Use of the "experimental" topic - Publish initial metadata. - - Publish update metadata. - - Data update notification: normal notification message with property.cache=false +The "experimental" topic is necessary for the WIS2 pre-operational phase and future pre-operational data exchange in test mode. -Use of the "experimental" topic +The experimental topic sits under domain (level 8), e.g. ...weather/experimental. Data publishers can can extend the experimental branch with sub-topics as they deem appropriate. - The "experimental" topic is necessary for the WIS2 pre-operational phase and future pre-operational data exchange in test mode. - - The experimental topic sits under domain (level 8), e.g. ...weather/experimental. Data publishers can can extend the experimental branch with sub-topics as they deem appropriate. - - Data consumers must not assume that experimental topics will be durable (i.e., they may change or be removed). +Data consumers must not assume that experimental topics will be durable (i.e., they may change or be removed). ==== Performance management ===== Service levels and performance indicators -A WIS2 node must be able to: +A WIS2 Node must be able to: - Publish datasets and compliant metadata and discovery metadata - * Publish metadata to the Global Data Catalogue - * Publish core data to the Global Cache - * Publish data for consumer access - * data embedded in a message (CAP/warning) - * Receive metadata publication errors from the Global Data Catalogue - Support IP filtering - Provide metadata with topics to Global Brokers +- Publish datasets and compliant metadata and discovery metadata + * Publish metadata to the Global Data Catalogue + * Publish core data to the Global Cache + * Publish data for consumer access + * Publish data embedded in a message (i.e., CAP warnings) + * Receive metadata publication errors from the Global Data Catalogue + * Provide metadata with topics to Global Brokers ===== Provision of system performance metrics -WIS nodes should provide annual performance metrics to their GISC. +WIS Nodes should provide annual performance metrics to their GISC. -If contacted by the Global Montior via GISC for a performance issue, the WIS node should provide metrics to the GISC and Global Monitor when service is restored to indicate resolution of the issue. +If contacted by the Global Montior via GISC for a performance issue, the WIS Node should provide metrics to the GISC and Global Monitor when service is restored to indicate resolution of the issue. ==== WIS Node reference implementation: wis2box Members may use whichever software components they consider most appropriate to comply with the WIS2 Technical Regulations. -To assist Members participate in WIS2, a freely available, open-source Reference Implementation has been developed: "WIS2 in a box" (referred to as wis2box). It builds on mature and robust free and open-source software components that are widely adopted for operational use. +To assist Members participate in WIS2, a free and open-source Reference Implementation is available for use. WIS2 in a box (wis2box) implements the requirements of a WIS2 Node in as well as additional enhancements. wis2box builds on mature and robust free and open-source software components that are widely adopted for operational use. -wis2box provides functionality required for both Data Publisher and Data Consumer roles. It provides the following technical functions: +wis2box provides functionality required for both data publisher and data consumer roles. It provides the following technical functions: * Real-time or archive data and metadata publishing to WIS2 (Publish), including available data transformation and processing pipelines * MQTT Message Broker and notification message publication (Subscribe) @@ -161,6 +153,4 @@ wis2box provides functionality required for both Data Publisher and Data Consume * The modular design of wis2box makes it simple to extend to meet additional requirements or integrate with existing data management systems. * wis2box already provides a useful set of functionality and will continue to evolve and develop throughout the WIS2 pilot phase and beyond. -Documentation is published in wis2box documentation. - -The project in hosted in GitHub: https://github.com/wmo-im/wis2box +Documentation is published in wis2box documentation More information is available at https://docs.wis2box.wis.wmo.int