From 4b8bb3a8e2fa3555a7a7cf43e490075054a68a0d Mon Sep 17 00:00:00 2001 From: Jeremy Tandy Date: Tue, 6 Feb 2024 17:25:16 +0000 Subject: [PATCH] cross-references checked and fixed (#119) Co-authored-by: Tom Kralidis --- guide/sections/part1/data-consumer.adoc | 31 +++++----- guide/sections/part1/data-publisher.adoc | 62 ++++++++++--------- guide/sections/part1/introduction.adoc | 10 +-- guide/sections/part2/global-services.adoc | 17 ++--- guide/sections/part2/operations.adoc | 20 +++--- guide/sections/part2/wis2-architecture.adoc | 12 ++-- guide/sections/part2/wis2node.adoc | 26 ++++---- .../part3/information-management.adoc | 31 +++++----- 8 files changed, 107 insertions(+), 102 deletions(-) diff --git a/guide/sections/part1/data-consumer.adoc b/guide/sections/part1/data-consumer.adoc index 7db0ca5..793337e 100644 --- a/guide/sections/part1/data-consumer.adoc +++ b/guide/sections/part1/data-consumer.adoc @@ -4,11 +4,11 @@ As a Data Consumer wanting to use data published via WIS2 you should read the gu ==== How to search the Global Discovery Catalogue to find Datasets -The first step to using data published via WIS2 is to determine which Dataset or Datasets contains the data that is needed. To do this, a Data Consumer may browse discovery metadata provided by the Global Discovery Catalogue. Discovery metadata follows a standard scheme (see PART V Discovery Metadata TODO: fix ref). A Data Consumer may discover a Dataset using keywords, geographic area of interest, temporal information, or free text. Matching search results from the Global Discovery Catalogue provide high-level information (title, description, keywords, spatiotemporal extents, data policy, licensing, contact information), from which a Data Consumer can assess and evaluate their interest in accessing/downloading data associated with the Dataset record. +The first step to using data published via WIS2 is to determine which Dataset or Datasets contains the data that is needed. To do this, a Data Consumer may browse discovery metadata provided by the Global Discovery Catalogue. Discovery metadata follows a standard scheme (see _Manual on WIS_ (WMO-No. 1060), Volume II, Appendix F: WMO Core Metadata Profile 2). A Data Consumer may discover a Dataset using keywords, geographic area of interest, temporal information, or free text. Matching search results from the Global Discovery Catalogue provide high-level information (title, description, keywords, spatiotemporal extents, data policy, licensing, contact information), from which a Data Consumer can assess and evaluate their interest in accessing/downloading data associated with the Dataset record. -A key component of Dataset records in the Global Discovery Catalogue is that of "actionable" links. A Dataset record provides one to many links that clearly identify the nature and purpose of the link (informational, direct download, API, subscription) so that the Data Consumer can interact with the data accordingly. For example, a Dataset record may include a link to subscribe to notifications (see (TODO CROSSREF) <<_how_to_subscribe_to_notifications_about_availability_of_new_data>>) about the data, or a API, or an offline archive retrieval service. +A key component of Dataset records in the Global Discovery Catalogue is that of "actionable" links. A Dataset record provides one to many links that clearly identify the nature and purpose of the link (informational, direct download, API, subscription) so that the Data Consumer can interact with the data accordingly. For example, a Dataset record may include a link to subscribe to notifications (see <<_how_to_subscribe_to_notifications_about_availability_of_new_data>>) about the data, or a API, or an offline archive retrieval service. -The Global Discovery Catalogue is accessible via an API and provides a low barrier mechanism (see (TODO CROSSREF) <<_global_discovery_catalogue>>). Internet search engines are able to index the discovery metadata in the Global Discovery Catalogue, thereby providing Data Consumers with an alternative means to search for WIS2 data. +The Global Discovery Catalogue is accessible via an API and provides a low barrier mechanism (see <<_global_discovery_catalogue>>). Internet search engines are able to index the discovery metadata in the Global Discovery Catalogue, thereby providing Data Consumers with an alternative means to search for WIS2 data. ==== How to subscribe to notifications about availability of new data @@ -18,7 +18,7 @@ In WIS2, notifications are re-published by Global Brokers to ensure resilient di Exceptionally, a Data Consumer may choose to subscribe directly to the local message broker at the originating WIS2 Node. However, note that a WIS2 Node is not required to provide public access to its local message broker. -A Dataset in WIS2 is associated with a unique _topic_. Notifications about updates to a Dataset are published to the associated topic. Topics are organized according to a standard scheme (see WIS2 Topic Hierarchy TODO: crossref). +A Dataset in WIS2 is associated with a unique _topic_. Notifications about updates to a Dataset are published to the associated topic. Topics are organized according to a standard scheme (see the _Manual on WIS_ (WMO-No. 1060), Volume II, Appendix D: WIS2 Topic Hierarchy). A Data Consumer can find the appropriate topic to subscribe to either by searching the Global Discovery Catalogue, or by browsing the topic hierarchy on a message broker. @@ -29,11 +29,11 @@ Notifications from WIS2 Nodes and Global Caches are published on different topic * Environment and Climate Change Canada, Meteorological Service of Canada's WIS2 Node publishes to: ``origin/a/wis2/ca-eccc-msc/data/core/weather/surface-based-observations/synop`` * Global Caches publish to: ``cache/a/wis2/ca-eccc-msc/data/core/weather/surface-based-observations/synop`` -As per clause 3.2.13 from the Manual on WIS, Vol II, Data Consumers should access data from the Global Cache. Consequently, they need to subscribe to the ``cache/#`` topic hierachy to receive the notifications from Global Caches, each of which provides a link (i.e., URL) to download from the respective Global Cache's data server. +As per clause 3.2.13 of the _Manual on WIS_ (WMO-No. 1060), Volume II, Data Consumers should access data from the Global Cache. Consequently, they need to subscribe to the ``cache/#`` topic hierachy to receive the notifications from Global Caches, each of which provides a link (i.e., URL) to download from the respective Global Cache's data server. ==== How to use a notification message to decide whether to download data -On receipt of a notification message, a Data Consumer needs to decide whether to download the newly available data. The content of the notification message provides the information needed to make this decision. See the WIS2 Notification Message (WNM) specification (TODO CROSSREF) for details. +On receipt of a notification message, a Data Consumer needs to decide whether to download the newly available data. The content of the notification message provides the information needed to make this decision. For details of the specification, see the _Manual on WIS_ (WMO-No. 1060), Volume II, Appendix E: WIS2 Notification Message. In many cases, Data Consumers will use a software application to determine whether or not to download the data. This section provides insight about what happens. @@ -69,7 +69,7 @@ Furthermore, the WMO Unified Data Policy encourages attribution of the source of Details of the applicable WMO data policy plus any rights or licenses associated with data are provided in the discovery metadata that accompanies the data. Discovery metadata records are available from the Global Discovery Catalogue. -The WMO Core Metadata Profile 2 specification, section 7.1.17. Properties / WMO data policy (TODO CROSSREF - WCMP2/7.1.17) provides details on how data policy, rights, and/or licenses are described in the discovery metadata. +The _Manual on WIS_ (WMO-No. 1060), Volume II, Appendix F: WMO Core Metadata Profile 2, section 7.1.17. Properties / WMO data policy provides details on how data policy, rights, and/or licenses are described in the discovery metadata. When using data from WIS2, data consumers: @@ -78,16 +78,15 @@ When using data from WIS2, data consumers: ==== Further reading for data consumers - (TODO CROSSREF for the referenced sections) - As a Data Publisher planning to operate a WIS2 Node, as a minimum you should read the following sections: -* 1.1. Introduction to WIS2 -* 2.1. WIS2 Architecture -* 2.2. Roles in WIS2 -* 2.4. Components of WIS2 +* <<_introduction_to_wis2>> +* <<_wis2_architecture>> +* <<_roles_in_wis2>> +* <<_components_of_wis2>> -The following specifications are useful for further reading: +The following specifications in the _Manual on WIS_ (WMO-No. 1060), Volume II are useful for further reading: -* WMO Core Metadata Profile 2 (WCMP2) specification -* WIS2 Notification Message (WNM) specification +* Appendix D: WIS2 Topic Hierarchy +* Appendix E: WIS2 Notification Message +* Appendix F: WMO Core Metadata Profile 2 \ No newline at end of file diff --git a/guide/sections/part1/data-publisher.adoc b/guide/sections/part1/data-publisher.adoc index ffa394a..3eae7bc 100644 --- a/guide/sections/part1/data-publisher.adoc +++ b/guide/sections/part1/data-publisher.adoc @@ -2,21 +2,23 @@ As a Data Publisher with authoritative Earth system data that you want to share with the WMO community you should read the guidance presented here. In addition, a list of references to informative material in this Guide and elsewhere is provided at the end of this section. -(TODO - MAYBE; add Guidance on "coarser resource publication with interactive links" - see WNM issue #41) -(TODO - MAYBE; add Guidance on linking to external provenance objects - see WCMP2 issue #161) -(TODO - MAYBE; add Guidance on referencing WIGOS stations and observing networks - see WCMP2 issue #19) + (TODO - MAYBE; add Guidance on "coarser resource publication with interactive links" - see WNM issue #41) + + (TODO - MAYBE; add Guidance on linking to external provenance objects - see WCMP2 issue #161) + + (TODO - MAYBE; add Guidance on referencing WIGOS stations and observing networks - see WCMP2 issue #19) ==== How to get started -The first thing you need to do is consider your data, how it can be conceptually grouped into one or more datasets (see section 1.1.4. Why are datasets so important? (TODO CROSSREF)), and whether it is *Core* or *Recommended* data, as per the WMO Unified Data Policy (Resolution 1 (Cg-Ext 2021)). +The first thing you need to do is consider your data, how it can be conceptually grouped into one or more datasets (see <<_why_are_datasets_so_important?>>), and whether it is *Core* or *Recommended* data, as per the WMO Unified Data Policy (Resolution 1 (Cg-Ext 2021)). -Next, you need to consider where it is published. If your data relates to your country or territory, you need to publish it through a National Centre (NC). If your data relates to a region, programme, or other specialized function within WMO, you need to publish it through a Data Collection or Production Centre (DCPC). The functional requirements for NC and DCPC are described in the _Manual on the WMO Information System_ (WMO-No. 1060), Vol II - Part III Functions of WIS. +Next, you need to consider where it is published. If your data relates to your country or territory, you need to publish it through a National Centre (NC). If your data relates to a region, programme, or other specialized function within WMO, you need to publish it through a Data Collection or Production Centre (DCPC). The functional requirements for NC and DCPC are described in the _Manual on WIS_ (WMO-No. 1060), Volume II - Part III Functions of WIS. All NCs and DCPCs are affiliated with a Global Information System Centre (GISC) that has a responsibility to help establish efficient and effective data sharing on the WIS. Your GISC will be able to help you in getting your data onto WIS2. You may be able to identify an existing NC or DCPC that can publish your data. Alternative, you need to establish a new NC or DCPC. The main difference is that an NC is designated by a Member, whereas a DCPC is designated by a WMO or related international programme and/or a regional association. -Both NC and DCPC require operation of a WIS2 Node (see section 2.4.1. WIS2 Node (TODO CROSSREF)). The procedure for registering a new WIS2 Node is provided in Section 2.6.1.1. Registration and decommissioning of a WIS2 Node (TODO CROSSREF). +Both NC and DCPC require operation of a WIS2 Node (see <<_wis2_node>>). The procedure for registering a new WIS2 Node is provided in <<_registration_and_decommissioning_of_a_wis2_node>>. Once you have determined the scope of your datasets, the data policy which applies, and have a WIS2 Node ready for data publication, you are ready to progress to the next step: providing discovery metadata. @@ -24,7 +26,7 @@ Once you have determined the scope of your datasets, the data policy which appli Discovery metadata is the mechanism by which you tell potential consumers about your data, how it may be accessed, and any conditions you may place on the use of the data. -Each dataset you want to publish must have an associated discovery metadata record. This record is encoded as GeoJSON (RFC 7946footnote:[RFC 7946 - The GeoJSON Format https://www.rfc-editor.org/rfc/rfc7946]) must conform to the WMO Core Metadata Profile 2 (WCMP2) specification (TODO CROSSREF). +Each dataset you want to publish must have an associated discovery metadata record. This record is encoded as GeoJSON (RFC 7946footnote:[RFC 7946 - The GeoJSON Format: https://datatracker.ietf.org/doc/html/rfc7946]) must conform to the specification given in the _Manual on WIS_ (WMO-No. 1060), Volume II, Appendix F: WMO Core Metadata Profile 2. Copies of all discovery metadata records from WIS2 are held at the Global Discovery Catalogues, where data consumers can search and browse to find data that is of interest to them. @@ -33,19 +35,17 @@ Depending on local arrangements, your GISC may be able to help you transfer your * The simplest method is to encode the discovery metadata record as a file and publish it to an HTTP server where it can be accessed with a URL. * Alternatively, you may operate a local metadata catalogue through which discovery metadata records can be shared using an API (e.g., OGC API - Recordsfootnote:[OGC API - Records - Part 1: Core https://docs.ogc.org/DRAFTS/20-004.html]). Each discovery metadata record can be accessed with a unique URL via the API (e.g., as an item which is part of the discovery metadata catalogue). -In both cases, a notification message needs to be published on a message broker that tells WIS2 there is new discovery metadata to upload and that it is accessed at the specified URLfootnote:[Both data and metadata publication use the same notification message mechanism to advertise the availability of a new resource.]. The notification messages shall conform to the WIS2 Notification Message (WNM) specification (TODO CROSSREF). Furthermore, the notification message must be published on topic that conforms to the WIS2 Topic Hierarchy (WTH) specification (TODO CROSSREF). For example, metadata published by Deutscher Wetterdienst would use the following topic: - -``origin/a/wis2/de-dwd/metadata/core`` +In both cases, a notification message needs to be published on a message broker that tells WIS2 there is new discovery metadata to upload and that it is accessed at the specified URLfootnote:[Both data and metadata publication use the same notification message mechanism to advertise the availability of a new resource.]. The notification messages shall conform to the specification given in the _Manual on WIS_ (WMO-No. 1060), Volume II, Appendix E: WIS2 Notification Message. Furthermore, the notification message must be published on topic that conforms to the specification given in _Manual on WIS_ (WMO-No. 1060), Volume II, Appendix D: WIS2 Topic Hierarchy. For example, metadata published by Deutscher Wetterdienst would use the following topic: ``origin/a/wis2/de-dwd/metadata/core`` These discovery metadata records are then propagated through the Global Service components into to the Global Discovery Catalogue where Data Consumers can search and browse for datasets of interest. -Upon receipt of a new discovery metadata record, a Global Discovery Catalogue (see section 2.4.4. Global Discovery Catalogue (TODO CROSSREF)) will validate, assess, ingest, and publish the record. Validation ensures that your discovery metadata record complies with the specification. Assessment examines your discovery metadata record against good practice (TODO CROSSREF WCMP2 KPIs). The Global Discovery Catalogue will notify you if your discovery metadata record fails validation and provide recommendations for improvements for you to consider. +Upon receipt of a new discovery metadata record, a Global Discovery Catalogue (see <<_global_discovery_catalogue>>) will validate, assess, ingest, and publish the record. Validation ensures that your discovery metadata record complies with the specification. Assessment examines your discovery metadata record against good practice. The Global Discovery Catalogue will notify you if your discovery metadata record fails validation and provide recommendations for improvements for you to consider. Discovery metadata must be published in the Global Discovery Catalogues before you begin publishing data. ==== How to provide data to WIS2 -WIS2 is based on the Web architecturefootnote:[Architecture of the World Wide Web https://www.w3.org/TR/webarch/]. As such it is _resource oriented_. Datasets are resources; the "granules" of data grouped in a dataset are resources; the discovery metadata records that describe datasets are resources. In Web architecture, every resource has a unique identifier (e.g., a URIfootnote:[RFC3986 Uniform Resource Identifier (URI) - Generic Syntax https://www.rfc-editor.org/rfc/rfc3986]), and the unique identifier can be used to resolve the resource identified and interact with it (e.g., to download a representation of the resource over an open standard protocol such as HTTP). +WIS2 is based on the Web architecturefootnote:[Architecture of the World Wide Web https://www.w3.org/TR/webarch/]. As such it is _resource oriented_. Datasets are resources; the "granules" of data grouped in a dataset are resources; the discovery metadata records that describe datasets are resources. In Web architecture, every resource has a unique identifier (e.g., a URIfootnote:[RFC 3986 - Uniform Resource Identifier (URI) - Generic Syntax: https://datatracker.ietf.org/doc/html/rfc3986]), and the unique identifier can be used to resolve the resource identified and interact with it (e.g., to download a representation of the resource over an open standard protocol such as HTTP). Simply, you provide data (and metadata) to WIS2 by assigning it a unique identifier, in this case a URLfootnote:[The term "Uniform Resource Locator" (URL) refers to the subset of URIs that, in addition to identifying a resource, provide a means of locating the resource by describing its primary access mechanism (e.g., its network "location"). RFC 3986], and making it available via a data server - most typically a Web server using the HTTP protocolfootnote:[WIS2 strongly prefers secure versions of protocols (e.g., HTTPS) wherein the communication protocol is encrypted using Transport Layer Security (TLS)]. It's up to the data server to decide what to provide when resolving the identifier: the URL of a data granule may resolve as a representation encoded in a given data format, whereas the URL of a dataset may resolve as a description of the dataset (i.e., metadata) that includes links to access the data from which the set is comprised - either individual files (i.e., the data granules) or an interactive API that enables a user to request just the parts of the dataset they need by specifying query parameters. @@ -80,7 +80,7 @@ But it is more complex to implement. You need a server running software that can . Interpret a user's request; . Extract the data from wherever it is stored; -. iii. Package that data up and send it back to the user. +. Package that data up and send it back to the user. Importantly, when considering use of interactive APIs to serve your data you need to plan for costs: every request to an interactive API requires computational resources to process. @@ -98,13 +98,13 @@ More guidance on use of interactive APIs in WIS2 is anticipated in future versio WIS2 is designed to support the data sharing needs of all WMO programmes. Among these, the World Weather Watch footnote:[WMO World Weather Watch https://wmo.int/world-weather-watch] drives specific needs for the rapid exchange of data to support weather forecasting. -To enable real-time data sharingfootnote:[In the context of WIS2, real-time implies anything from a few seconds to a few minutes - not the milliseconds required by some applications.], WIS2 uses notification messages to advertise the availability of a new resource - data or discovery metadata - and how to access that resource. Notification messages are published to a queue on a message broker in your WIS2 Nodefootnote:[WIS2 ensures rapid global distribution of notification messages using a network of Global Brokers which subscribe to message brokers of WIS2 Nodes and republish notification messages (see section 2.4.2. Global Broker (TODO CROSSREF)).] using the MQTT protocol and immediately delivered to everyone subscribing to that queue. A queue is associated with a specific _topic_, such as dataset. +To enable real-time data sharingfootnote:[In the context of WIS2, real-time implies anything from a few seconds to a few minutes - not the milliseconds required by some applications.], WIS2 uses notification messages to advertise the availability of a new resource - data or discovery metadata - and how to access that resource. Notification messages are published to a queue on a message broker in your WIS2 Nodefootnote:[WIS2 ensures rapid global distribution of notification messages using a network of Global Brokers which subscribe to message brokers of WIS2 Nodes and republish notification messages (see <<_global_broker>>).] using the MQTT protocol and immediately delivered to everyone subscribing to that queue. A queue is associated with a specific _topic_, such as dataset. For example, when a new temperature profile from a radio sonde deployment is added to a dataset of upper-air data measurements, a notification message would be published that includes the URL used to access the new temperature profile data. Everyone subscribing to notification messages about the upper-air measurement dataset would receive the notification message, identify the URL and download the new temperature profile data. Optionally, data may be embedded in a notification message using a ``content`` object _in addition_ to publishing via the data server. Inline data must be encoded as ``UTF-8``, ``Base64``, or ``gzip``, and must not exceed 4096 bytes in length once encoded. -Notification messages are encoded as GeoJSON (RFC 7946) and must conform to the WIS2 Notification Message (WNM) specification (TODO CROSSREF). +Notification messages are encoded as GeoJSON (RFC 7946) and must conform to the _Manual on WIS_ (WMO-No. 1060), Volume II, Appendix E: WIS2 Notification Message. The URL used in the notification message should refer only to the newly added data object rather (e.g., the new temperature profile) than the entire dataset. However, the WIS2 Notification Message specification allows for multiple URLs to be provided. If you are providing your data through an interactive API, you might provide a "canonical" link (designated with link relation: ``"rel": "canonical"``footnote:[IANA Link Relations https://www.iana.org/assignments/link-relations/link-relations.xhtml]), and an additional link providing the URL for the root of the Web service from where one can interact with or query the entire Dataset. @@ -112,11 +112,11 @@ You should include the dataset identifier in the notification message (``metadat Furthermore, if you have implemented controlled access to your data (e.g., the use of an API key), you should include a security object in the download link that provides the pertinent information (e.g., the access control mechanism used, and where/how a Data Consumer would need to register to request access). -To ensure that data consumers can easily find the topics they want to subscribe to, data publishers must publish to an authorized topic, as specified in the WIS2 Topic Hierarchy (WTH) specification (TODO CROSSREF). +To ensure that data consumers can easily find the topics they want to subscribe to, data publishers must publish to an authorized topic, as specified in the _Manual on WIS_ (WMO-No. 1060), Volume II, Appendix D: WIS2 Topic Hierarchy. If your data seems to relate to more than one topic, select the most appropriate one. The topic-hierarchy is not a knowledge organisation system - it is only used to ensure uniqueness of topics for publishing notification messages. Discovery metadata is used to describe a dataset and its relevance to additional disciplines; each dataset is mapped to one, and only one, topic. -If the WIS2 Topic Hierarchy does not include a topic appropriate for your data, your should publish on an _experimental_ topic. This allows for data exchange to be established while the formalities are consideredfootnote:[The "experimental" topic is necessary for the WIS2 pre-operational phase and future pre-operational data exchange in test mode.]. Experimental topics are provided for each Earth-system discipline at level 8 in the topic hierarchy (e.g., ``origin/a/wis2/{centre-id}/data/{earth-system-discipline}/experimental/``). Data publishers can can extend the experimental branch with sub-topics as they deem appropriate. Experimental topics are subject to change and will be removed once they are no longer needed. For more information, see WIS2 Topic Hierarchy section 7.1.2 Publishing guidelines (TODO CROSSREF). +If the WIS2 Topic Hierarchy does not include a topic appropriate for your data, your should publish on an _experimental_ topic. This allows for data exchange to be established while the formalities are consideredfootnote:[The "experimental" topic is necessary for the WIS2 pre-operational phase and future pre-operational data exchange in test mode.]. Experimental topics are provided for each Earth-system discipline at level 8 in the topic hierarchy (e.g., ``origin/a/wis2/{centre-id}/data/{earth-system-discipline}/experimental/``). Data publishers can can extend the experimental branch with sub-topics as they deem appropriate. Experimental topics are subject to change and will be removed once they are no longer needed. For more information, see _Manual on WIS_ (WMO-No. 1060), Volume II, Appendix D: WIS2 Topic Hierarchy, section 7.1.2 Publishing guidelines. Whatever topic you choose, the discovery metadata you provided to the Global Discovery Catalogue must include subscription links using that topicfootnote:[The Global Discovery Catalogue will reject discovery metadata records containing links to topics outside the official topic-hierarchy.]. The Global Broker will only republish notification messages on topics specified in your discovery metadata records. @@ -124,7 +124,7 @@ Whatever topic you choose, the discovery metadata you provided to the Global Dis Core data, as specified in the WMO Unified Data Policy (Resolution 1 (Cg-Ext 2021)) is considered essential for provision of services for the protection of life and property and for the well-being of all nations. Core data is provided on a free and unrestricted basis, without charge and with no conditions on use. -WIS2 ensures highly available, rapid access to _most_ Core data via a collection of Global Caches (see section 2.4.3. Global Cache (TODO CROSSREF)). Global Caches subscribe to notification messages about the availability of new Core data published at WIS2 Nodes, download a copy of that data and re-publish it on a high-performance data server, discarding it after the retention period expires - normally 24-hoursfootnote:[A Global Cache provides short-term hosting of data. Consequently, it is not an appropriate mechanism to provide access to archives of Core data, such as Essential Climate Variables. Providers of such archive data must be prepared to serve such data directly from their WIS2 Node.]. Global Caches do not provide any sophisticated APIs - they publish notification messages advertising the availability of data on their cache and allow users to download data via HTTPS using the URL in the notification message. +WIS2 ensures highly available, rapid access to _most_ Core data via a collection of Global Caches (see <<_global_cache>>). Global Caches subscribe to notification messages about the availability of new Core data published at WIS2 Nodes, download a copy of that data and re-publish it on a high-performance data server, discarding it after the retention period expires - normally 24-hoursfootnote:[A Global Cache provides short-term hosting of data. Consequently, it is not an appropriate mechanism to provide access to archives of Core data, such as Essential Climate Variables. Providers of such archive data must be prepared to serve such data directly from their WIS2 Node.]. Global Caches do not provide any sophisticated APIs - they publish notification messages advertising the availability of data on their cache and allow users to download data via HTTPS using the URL in the notification message. The URL included in a notification message that is used to access Core data from a WIS2 Node, or the "canonical" URL if multiple URLs are provided, must: @@ -159,7 +159,7 @@ Recommended data is never cached by the Global Caches. Use of Core data must always be free and unrestricted. However, you may need to leverage existing systems with built-in access control when implementing the download service for your WIS2 Node. -Example 1: API key. Your data server requires a valid API key to be included in download requests. The URLs used notification messages should include a valid API key.footnote:[A specific API key should be used for data publication via WIS2 so that usage can be tracked.]footnote:[Given that users are encouraged to download Core data from the Global Cache, there will likely be only a few accesses using the WIS2 account's API key. If the usage quota for the WIS2 account is exceeded (i.e., further data access is blocked) then this should encourage users to download via the Global Cache as mandated in the _Manual on WIS_ (WMO-No. 1060).] +Example 1: API key. Your data server requires a valid API key to be included in download requests. The URLs used notification messages should include a valid API key.footnote:[A specific API key should be used for data publication via WIS2 so that usage can be tracked.]footnote:[Given that users are encouraged to download Core data from the Global Cache, there will likely be only a few accesses using the WIS2 account's API key. If the usage quota for the WIS2 account is exceeded (i.e., further data access is blocked) then this should encourage users to download via the Global Cache as mandated in the _Manual on WIS_ (WMO-No. 1060), Volume II.] Example 2: Pre-signed URLs. Your data server uses a cloud-based object store that requires credentials to be provided when downloading data. The URLs used in notification message should be _pre-signed_ with the data publisher's credentials and valid for the cache retention period (e.g., 24-hours).footnote:[Working with presigned URLs on Amazon S3 https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html] @@ -177,28 +177,30 @@ Note that some data archives are categorised as Core data; for example, Essentia ==== Further reading for data publishers -(TODO CROSSREF for the referenced sections) - As a Data Publisher planning to operate a WIS2 Node, as a minimum you should read the following sections: -* 1.1. Introduction to WIS2 -* 2.1. WIS2 Architecture -* 2.2. Roles in WIS2 -* 2.4. Components of WIS2 -* 2.6. Implementation and operation of WIS2 Node +* <<_introduction_to_wis2>> +* <<_wis2_architecture>> +* <<_roles_in_wis2>> +* <<_components_of_wis2>> +* <<_implementation_and_operation_of_a_wis2_node>> The following sections are useful for further reading: -* 3.1. Information management -* 4.1. Security -* 5.1. Competencies +* <<_information_management>> +* <<_security>> +* <<_competencies>> Note that sections _4.1. Security_ and _5.1. Competencies_ reference content originally published for WIS1. These remain largely applicable and will be updated subsequent releases of this Guide. If you are publishing aviation weather data via WIS2 for onward transmission through ICAO SWIM you should also read: -* 2.8.1.1. Publishing aviation weather data through WIS2 into ICAO SWIM +* <<_publishing_aviation_weather_data_through_wis2_into_icao_swim>> +Finally, you should also review the specifications in the _Manual on WIS_ (WMO-No. 1060), Volume II: +* Appendix D: WIS2 Topic Hierarchy +* Appendix E: WIS2 Notification Message +* Appendix F: WMO Core Metadata Profile 2 // include::sections/wis2node.adoc[] diff --git a/guide/sections/part1/introduction.adoc b/guide/sections/part1/introduction.adoc index b428098..285f178 100644 --- a/guide/sections/part1/introduction.adoc +++ b/guide/sections/part1/introduction.adoc @@ -34,7 +34,7 @@ It should be clear that hosting data and services on the cloud does not affect d WMO enables the international exchange of observations and model data for all Earth-system disciplines. -The WMO Unified Data Policy, Resolution 1 (Cg-Ext(2021)) footnote:[WMO Unified Data Policy, Resolution 1 (Cg-Ext(2021))], describes the Earth system data that are necessary for efforts to monitor, understand and predict the weather and climate - including the hydrological cycle, the atmospheric environment and space weather. +The WMO Unified Data Policy, Resolution 1 (Cg-Ext(2021)) describes the Earth system data that are necessary for efforts to monitor, understand and predict the weather and climate - including the hydrological cycle, the atmospheric environment and space weather. WIS is the mechanism by which this Earth system data is exchanged. @@ -61,9 +61,9 @@ Because we group data into a single, conceptual resource (i.e., the Dataset) we All this means that the Dataset concept is central WIS: -* We publish discovery metadata about Datasets, as specified in the WMO Core Metadata Profile footnote:[WMO Core Metadata Profile version 2]. -* We can search for Datasets that contain relevant data using the Global Discovery Catalogue. -* We can subscribe to notifications about updates about a Dataset via a Global Broker. +* We publish discovery metadata about Datasets, as specified in the _Manual on WIS_ (WMO-No. 1060), Volume II, Appendix F: WMO Core Metadata Profile 2. +* We can search for Datasets that contain relevant data using the Global Discovery Catalogue (see <<_global_discovery_catalogue>>). +* We can subscribe to notifications about updates about a Dataset via a Global Broker (see <<_global_broker>>). * We can access the data that comprises a Dataset from a single location using a well described mechanism. It is up to the Data Publisher to decide how their data is grouped into Datasets - effectively, to decide what Datasets they publish to WIS. That said, we recommend that, subject to the consistency rules above, Data Publishers should organise their data into as few Datasets as possible. @@ -77,7 +77,7 @@ There are some things that are fixed requirements for Datasets: Here are some examples of Datasets: -* The most recent 5-days of synoptic observations for an entire country or territoryfootnote:[Why 5-days in this example? Because only 5-days of observations is retained in the system used to publish the data.]. +* The most recent 5-days of synoptic observations for an entire country or territoryfootnote:[Why 5-days in this example? Because the system used to publish the data in this example only retains data for 5-days.]. * Long-term record of observed water quality for a managed set of hydrological stations. * Output from the most recent 24-hours of operational numerical weather prediction model runs. * Output from 6-months of experimental model runs. It's important to note that output from the operational and experimental should not be merged into the same dataset because they use different algorithms - it's very useful to be able to distinguish the provenance (or lineage) of data. diff --git a/guide/sections/part2/global-services.adoc b/guide/sections/part2/global-services.adoc index 48402f7..243b9a7 100644 --- a/guide/sections/part2/global-services.adoc +++ b/guide/sections/part2/global-services.adoc @@ -2,7 +2,7 @@ ==== Procedure for registration of a new Global Service -Successful operations of WIS will depend on having a set of Global Services running state of the art IT environments, with a very high level of reliability so that all WIS Users and WIS2 Nodes will be able to access and provide data they need for their duties. +Successful operations of WIS will depend on having a set of Global Services running well-managed IT environments with a very high level of reliability so that all WIS Users and WIS2 Nodes will be able to access and provide data they need for their duties. Depending on the nature of the Global Service, the following is considered to be the minimum capability of Global Service operation, so that collectively, the level of service is 100% (or very close): @@ -17,7 +17,7 @@ Running a Global Service is a significant commitment for a WIS Centre. To maint On receipt of an offer from a Member to operate a Global Service, WMO Secretariat will suggest which Global Service the Member may provide in order to improve WIS2. This suggestion will be based on the current situation of WIS2 (e.g., the number of existing Global Brokers, whether an additional Global Cache is needed, etc.). -The Manual on WIS, the Guide and other material available will help WIS Centres in deciding the best way forward. +The _Manual on WIS_ (WMO-No. 1060), Volume II, the Guide and other material available will help WIS Centres in deciding the best way forward. When decided, the WIS Focal Point will inform WMO Secretariat of its preference. Depending on the type of Global Service, WMO Secretariat will provide a checklist to the WIS Centre so that the future Global Service can be included in WIS Operations. @@ -88,10 +88,10 @@ In the following sections and for each Global Service, a set of metrics is defin ** An off the shelf broker implementing both MQTT 3.1.1 and MQTT 5.0 in a highly-available setup, typically in a cluster mode. Tools such as EMQX, HiveMQ, VerneMQ, RabbitMQ (in its latest versions) are compliant with these requirements. It must be noted that the open source version of Mosquitto cannot be clustered and therefore should not be used as part of a Global Broker. ** Additional features including anti-loop detection, notification message format compliance, validation of the published topic, and provision of metrics are required. -* When receiviong a message from a WIS Centre or Global Service broker, The metric ``wmo_wis2_gb_messages_received_total`` will be increased by 1. +* When receiving a message from a WIS Centre or Global Service broker, The metric ``wmo_wis2_gb_messages_received_total`` will be increased by 1. * A Global Broker will check if the topic on which the message is received is valid (in particular, a discovery metadata record must exist with a corresponding topic in order that data can be made available using this topic). If the topic is invalid, the Global Broker will discard non-compliant messages and will raise an alert. The metric ``wmo_wis2_gb_messages_no_metadata_total`` will be increased by 1. Global Broker should not request Global Discovery Catalogue for each notification message but should keep a cache of all valid topics for every ``centre-id``. * During the pre-operational phase (2024), Global Broker will not discard the message but will send a message on the `monitor` topic hierarchy to inform the originating centre and its GISC. -* A Global Broker will validate notification messages against the standard format (see Notification message format and structure), discarding non-compliant messages and raising an alert. The metrics ``wmo_wis2_gb_messages_invalid_total`` will be increased by 1. +* A Global Broker will validate notification messages against the standard format (see Notification message format and structure), discarding non-compliant messages and raising an alert. The metric ``wmo_wis2_gb_messages_invalid_total`` will be increased by 1. * A Global Broker instance will republish a message only once. Using the message id as defined in WIS Notification Message, the Global Broker will record id of messages already published and will discard subsequent identitical (with the same message id) messages. This is the anti-loop feature of the Global Broker. * When publishing a message to the local broker, the metric ``wmo_wis2_gb_messages_published_total`` will be increased by 1. * All aboved defined metrics will be made avalaible on HTTPS endpoints that the Global Monitor will ingest from regularly. @@ -110,8 +110,9 @@ In WIS2 Global Caches provide access to WMO Core Data for data consumers. This a ** A Cache management implementing the features needed to connect with the WIS ecosystem, receive data from WIS2 nodes and other Global Caches, store the data to the data server and manage the content of the cache (i.e. expiration of data, deduplication, etc) * The Global Cache will aim at containing copies of real-time and near real-time data designated as "core" within the WMO Unified Data Policy, Resolution 1 (Cg-Ext(2021)). * A Global Cache instance will host data objects copied from NC/DCPCs. -* A Global Cache instance will publish notification messages advertising availability of the data objects it holds. The notification messages will follow the standard structure (see TODO CROSSREF <>). -* A Global Cache instance will use the standard topic structure in their local message brokers (see TODO CROSSREF <>). +* A Global Cache instance will publish notification messages advertising availability of the data objects it holds. The notification messages will follow the standard structure (see _Manual on WIS_ (WMO-No. 1060), Volume II, Appendix E: WIS2 Notification Message). +* A Global Cache instance will use the standard topic structure in their local message brokers (see _Manual on WIS_ (WMO-No. 1060), Volume II, Appendix D: WIS2 Topic Hierarchy +). * A Global Cache instance will publish on topic ``cache/a/wis2/...``. * There will be multiple Global Cache instances to ensure highly available, low latency global provision of real-time and near real-time "core" data within WIS2. * There will be multiple Global Cache instances may attempt to download cacheable data objects from all originating centres with "cacheable" content. A Global Cache instance will also download data objects from other Global Cache instances. This ensures the instance has full global coverage, mitigating where direct download from an originating centre is not possible. @@ -174,8 +175,8 @@ In WIS2 Global Caches provide access to WMO Core Data for data consumers. This a * Global Discovery Catalogue instances operate independently of each other; each Global Discovery Catalogue instance will hold all discovery metadata records. Global Discovery Catalogues do not need to synchronise between themselves. * A Global Discovery Catalogue is populated with discovery metadata records from a Global Cache instance, receiving messages about the availability of discovery metadata records via a Global Broker. * A Global Discovery Catalogue should connect and subscribe to more than one Global Broker instance to ensure that no messages are lost in the event of a Global Broker failure. A Global Discovery Catalogue instance will discard duplicate messages as needed. -* A Global Discovery Catalogue will validate that a discovery metadata record identifier's `centre-id` token (see WCMP2 TODO CROSSREF) matches against the `centre-id` level (see WTH TODO CROSSREF) of the topic from which it was published, to ensure that discovery metadata is published by the authoritative orgnanization. -* A Global Discovery Catalogue will validate discovery metadata records against the WMO Core Metadata Profile version 2 (WCMP2). Valid WCMP2 records will be ingested into the catalogue. Invalid or malformed records will be discarded and reported to the Global Monitor against the centre identifier associated with the discovery metadata record. +* A Global Discovery Catalogue will validate that a discovery metadata record identifier's `centre-id` token (see _Manual on WIS_ (WMO-No. 1060), Volume II, Appendix F: WMO Core Metadata Profile 2) matches against the `centre-id` level of the topic from which it was published (see _Manual on WIS_ (WMO-No. 1060), Volume II, Appendix D: WIS2 Topic Hierarchy), to ensure that discovery metadata is published by the authoritative orgnanization. +* A Global Discovery Catalogue will validate discovery metadata records against the WMO Core Metadata Profile 2 (WCMP2). Valid WCMP2 records will be ingested into the catalogue. Invalid or malformed records will be discarded and reported to the Global Monitor against the centre identifier associated with the discovery metadata record. * A Global Discovery Catalogue will only update discovery metadata records to replace links for dataset subscription and notification (origin) with their equivalent links for subscription at Global Broker instances (cache). * A Global Discovery Catalogue will periodically assess discovery metadata provided by NCs and DCPCs against a set of key performance indicators (KPIs) in support of continuous improvement. Suggestions for improvement will be reported to the Global Monitor against the centre identifier associated with the discovery metadata record. * A Global Discovery Catalogue will remove discovery metadata that is marked for deletion as specified in the data notification message. diff --git a/guide/sections/part2/operations.adoc b/guide/sections/part2/operations.adoc index 59ed9ca..425e460 100644 --- a/guide/sections/part2/operations.adoc +++ b/guide/sections/part2/operations.adoc @@ -21,7 +21,7 @@ Both WIS2 and SWIM support the similar outcomes relating to data exchange. However, there are differences in both approach and implementation. -Specifications for WIS2 are defined in the Manual on WIS (WMO-No. +Specifications for WIS2 are defined in the _Manual on WIS_ (WMO-No. 1060), Volume II, and further elaborated in this Guide. Specifications for SWIM will be defined in the Procedures for Air Navigation Services – Information Management (PANS-IM) (ICAO Doc. 10199)footnote:[The PANS-IM is expected to available on ICAO NET by July 2024 and become applicable in November 2024. Information provided in herein is based on best understanding of draft proposals from ICAO.]. @@ -85,7 +85,7 @@ to SWIM. ====== Data types and format Specifications for aeronautical meteorological information are provided in ICAO -Annex 3 and other relevant guidance materials. IWXXM format (FM 205)footnote:[IWXXM (FM205) is defined in the Manual on Codes (WMO-No. 306), Volume I.3 – International Codes] +Annex 3 and other relevant guidance materials. IWXXM format (FM 205)footnote:[IWXXM (FM205) is defined in the _Manual on Codes_ (WMO-No. 306), Volume I.3 – International Codes] is to be used for encoding aeronautical meteorological information in SWIM. ====== Publishing meteorological data via WIS2 @@ -93,7 +93,7 @@ is to be used for encoding aeronautical meteorological information in SWIM. For meteorological data to be published from WIS2 to SWIM, the organisation responsible for this provision will need to operate a WIS2 Node and comply with the pertinent Technical Regulations as specified in the -Manual on WIS (WMO-No. 1060), Volume II. Onward distribution of the +_Manual on WIS_ (WMO-No. 1060), Volume II. Onward distribution of the data by the Message Broker over SWIM will be handled by the respective Information Service Provider in accordance with ICAO Standards and Recommended Practices (SARPs). @@ -105,7 +105,7 @@ provided by a separate operational unit, or there may be a requirement to easily distinguish between data for SWIM and any other meteorological data. Where a new WIS2 Node is needed, the responsible organisation must -establish a new WIS2 Node and register it with WMO Secretariat. For more information, see section 2.6.x Registration and decommissioning of a WIS2 Node (TODO CROSSREF). +establish a new WIS2 Node and register it with WMO Secretariat. For more information, see <>. Datasets are a central concept in WIS2. Where meteorological data is published via WIS2, it shall be packaged into @@ -154,7 +154,7 @@ The responsible organisation must publish discovery metadata for each dataset. N For more information on the WMO Core Metadata Profile version 2, see the -Manual on WIS (WMO-No. 1060), Volume II, Appendix F (TODO CROSSREF). +_Manual on WIS_ (WMO-No. 1060), Volume II, Appendix F. On receipt of new data, the WIS2 Node will: @@ -171,8 +171,8 @@ refers to the data resource using this URLfootnote:[Where the data resource does not exceed 4Kb, it may additionally be embedded in the notification message.]. -For more details on the WIS2 Notification Message, see the Manual on WIS -(WMO-No. 1060), Volume II, Appendix F (TODO CROSSREF). +For more details on the WIS2 Notification Message, see the _Manual on WIS_ +(WMO-No. 1060), Volume II, Appendix E: WIS2 Notification Message. The notification message must be published to the proper topic on the message broker. WIS2 defines a standard topic hierarchy to ensure @@ -187,8 +187,8 @@ aviation-specific notifications. See the example below: origin/a/wis2/{centre-id}/data/recommended/weather/aviation/qvaci ---- -For more details of the WIS Topic Hierarchy, see Manual on WIS (WMO-No. -1060), Volume II, Appendix E (TODO CROSSREF). +For more details of the WIS Topic Hierarchy, see the _Manual on WIS_ (WMO-No. +1060), Volume II, Appendix D: WIS2 Topic Hierarchy. WIS Global Brokers subscribe to the local message brokers of WIS2 Nodes and republish notification messages for global distribution. @@ -332,4 +332,4 @@ Data Policy. image::images/wis2-odis-metadata-discovery-interop.png[WIS2 and ODIS metadata and catalogue interoperability] As a result, federated discovery will be realized between both systems, allowing for -use and reuse of data in an authoritative manner, closest to the source of the data. +use and reuse of data in an authoritative manner, closest to the source of the data. \ No newline at end of file diff --git a/guide/sections/part2/wis2-architecture.adoc b/guide/sections/part2/wis2-architecture.adoc index 4815698..b97d07b 100644 --- a/guide/sections/part2/wis2-architecture.adoc +++ b/guide/sections/part2/wis2-architecture.adoc @@ -10,7 +10,7 @@ GISCs coordinate the operation of WIS within their Area of Responsibility (AoR) A WIS Centre may also operate one or more Global Services. -WIS Centres shall comply with the Technical Regulations defined in the Manual on WMO Information System (WMO-No. 1060), Volume II. +WIS Centres shall comply with the Technical Regulations defined in the _Manual on WIS_ (WMO-No. 1060), Volume II. === Roles in WIS2 @@ -72,6 +72,8 @@ Leveraging existing open standards, WIS2 defines the following specifications in |=== +Please refer to the _Manual on WIS_ (WMO-No. 1060), Volume II for details. + === Components of WIS2 // TODO: add refs to other parts of the Guide describing these components @@ -80,8 +82,8 @@ Leveraging existing open standards, WIS2 defines the following specifications in * WIS2 Nodes are central to WIS2. These are operated by National Centres (NC) and Data Collection or Production Centres (DCPC) to publish their *Core* and *Recommended* data. * WIS2 adopts Web technologies and open standards enabling WIS2 Nodes to be implemented using freely-available software components and common industry practices. * WIS2 Nodes publish data as files of a Web server or using an interactive Web service. -* WIS2 Nodes describe the data they publish using discovery metadata [TODO: ref. WIS Core Metadata Profile 2]. -* WIS2 Nodes generate notification messages [TODO: ref. WIS2 Notification Message] advertising the availability of new data. These notification messages are published to a message broker. The WIS2 Topic Hierarchy is used to ensure that all WIS2 Nodes publish to consistent topics. The information in the notification message tells the Data Consumer where to download data from. Notification messages are also used to advertise the availability of discovery metadata. +* WIS2 Nodes describe the data they publish using discovery metadata. See the _Manual on WIS_ (WMO-No. 1060), Volume II, Appendix F: WMO Core Metadata Profile 2. +* WIS2 Nodes generate notification messages advertising the availability of new data. These notification messages are published to a message broker. The WIS2 Topic Hierarchy is used to ensure that all WIS2 Nodes publish to consistent topics. The information in the notification message tells the Data Consumer where to download data from. Notification messages are also used to advertise the availability of discovery metadata. See the _Manual on WIS_ (WMO-No. 1060), Volume II, Appendix D: WIS2 Topic Hierarchy and Appendix E: WIS2 Notification Message. * WIS2 Nodes may implement controlled access for the data they publish. Global Services will operate with fixed IP addresses, enabling WIS2 Nodes to easily distinguish their requests. ==== Global Broker @@ -123,7 +125,7 @@ Leveraging existing open standards, WIS2 defines the following specifications in ==== Publish-Subscribe protocol (MQTT) -* The MQTT protocol is to be used for all WIS2 Publish-Subscribe workflow (publication and subscription). +* The MQTT protocolfootnote:[MQTT Specifications: https://mqtt.org/mqtt-specification/] is to be used for all WIS2 Publish-Subscribe workflow (publication and subscription). * MQTT v3.1.1 and v5.0 are the chosen protocols for the WIS2 Notification Messages publication and subscription. ** To connect to Global Brokers, MQTT v5.0 is preferred as it provides additional features such as the ability to used shared subscription. * The following parameters are to be used for all MQTT client/server connectivity and subscription: @@ -136,6 +138,6 @@ Leveraging existing open standards, WIS2 defines the following specifications in ==== Download protocol (HTTP) -* The HTTP protocol is to be used for all WIS2 download workflow. +* The HTTP protocol (RFC 7231footnote:[TFC 7231 - Hypertext Transfer Protocol (HTTP/1.1): https://datatracker.ietf.org/doc/html/rfc7231]) is to be used for all WIS2 download workflow. * To improve the overall level of security of WIS2, the secure version of the HTTP protocol is preferred. If used, the certificate must be valid. * The standard TCP port to be used is 443 for Secure HTTP (HTTPS). diff --git a/guide/sections/part2/wis2node.adoc b/guide/sections/part2/wis2node.adoc index a41cf3b..181456f 100644 --- a/guide/sections/part2/wis2node.adoc +++ b/guide/sections/part2/wis2node.adoc @@ -6,12 +6,12 @@ Registration and decomissioning of WIS2 Nodes must be approved by the Permanent Representative to WMO (PR) for the country or territory in which the WIS Centre resides. Where the WIS2 Node is part of a Data Collection or Production Centre (DCPC), the sponsoring WMO Programme or Regional Association shall be consulted. -WMO Secretariat will operate a WIS2 register. +WMO Secretariat will operate a WIS2 register as an authoritative list of WIS2 Nodes and Global Services. The registration of a WIS2 Node involves the following steps: * Request hosting a WIS2 Node: A request for hosting a WIS2 Node shall be put forward by the Permanent Representative of the country of the WIS2 Node host centre, or, in the case of international organizations, by either the Permanent Representative (PR) of the country or territory where the WIS2 Node host centre is located or the president of the relevant organization in case of WMO partner or programme designated as DCPC. -* Assign a centre-id: The centre identifier (``centre-id``) is an acronym as specified by the member and endorsed by the PR of the country and WMO. It is a single identifier comprised of a top level domain (TLD) and centre name, and represents the data publisher, distributor or issuing centre of a given dataset or data product/granule (see the WIS2 Topic Hierarchy (WTH) specification (TODO CROSSREF)). See below for guidance on assigning a centre identifier. +* Assign a centre-id: The centre identifier (``centre-id``) is an acronym as specified by the member and endorsed by the PR of the country and WMO. It is a single identifier comprised of a top level domain (TLD) and centre name, and represents the data publisher, distributor or issuing centre of a given dataset or data product/granule (see the _Manual on WIS_ (WMO-No. 1060), Volume II, Appendix D: WIS2 Topic Hierarchy). See below for guidance on assigning a centre identifier (<<_guidance_on_assigning_a_centre_identifier_for_a_wis2_n ode>>). * Complete the WIS2 Register: The WIS National Focal Point shall complete the WIS2 Register operated by the WMO Secretariat. * Provide Global Service details: The WMO Secretariat provides connection details for the Global Services (e.g., IP addresses) so that the WIS2 Node can be configured to provide the access. * WIS2 Node assessment: The principal GISC verifies that the WIS2 Node is compliant with WIS2 requirements. The assessment includes: @@ -26,45 +26,45 @@ A diagram of the process of registering a WIS2 Node is presented below. image::images/add-wis2node.png[Adding a WIS2 Node,link=images/add-wis2node.png] -Once a WIS2 Node has been registered and connected with the Global Services, it can procede to register the datasets it will publish via WIS2. To register a dataset, the authorized WIS2 Node publishes discovery metadata about the new dataset. Validation of the discovery metadata is completed by the Global Discovery Catalogues and Global Brokers automatically subscribes to the topics provided in the discovery metadata record. For more information, see section 1.3.x How to provide discovery metadata to WIS2 (TODO CROSSREF). +Once a WIS2 Node has been registered and connected with the Global Services, it can procede to register the datasets it will publish via WIS2. To register a dataset, the authorized WIS2 Node publishes discovery metadata about the new dataset. Validation of the discovery metadata is completed by the Global Discovery Catalogues and Global Brokers automatically subscribes to the topics provided in the discovery metadata record. For more information, see <<_how_to_provide_discovery_metadata_to_wis2>>. -Once the dataset has successfully been registered, the WIS2 Node can procede to exchange data - see section 1.3.x How to provide data in WIS2 (TODO CROSSREF). +Once the dataset has successfully been registered, the WIS2 Node can procede to exchange data - see <<_how_to_provide_data_in_wis2>>. When decommissioning a WIS2 Node operators must ensure that obligations relating to data sharing within WIS continue to be met after the WIS2 Node is decommissioned, for example, by migrating these data sharing obligations to another WIS2 Node. In the case of a DCPC, this may mean the responsibilities are transferred to another Member. ===== Guidance on assigning a Centre Identifier for a WIS2 Node -The Centre Identifier (``centre-id``) is used in WIS2 to uniquely identify a participating WIS2 Node. The Centre Identifier must conform to the specification given in the WIS2 Topic Hierarchy (WTH), section 7.1.6 Centre identification (TODO CROSSREF). +The Centre Identifier (``centre-id``) is used in WIS2 to uniquely identify a participating WIS2 Node. The Centre Identifier must conform to the specification given in the _Manual on WIS_ (WMO-No. 1060), Volume II, Appendix D: WIS2 Topic Hierarchy, section 7.1.6 Centre identification. The Centre Identifier comprises two dash-separated tokens. *Token 1* is a _Top Level Domain_ (TLD) defined by IANAfootnote:[IANA Top Level Domains https://data.iana.org/TLD]. -This is usually a simple choice for a Member. However, overseas territories require some thought. The recommended approach depends on the governance of the overseas territory. Take some French examples. La Reunion is a proper French “department” – it’s considered part of France, it uses the Euro. Here, we would use the “fr” TLD. New Caledonia is a French overseas territory with top-level-domain of “nc”. It has separate, devolved governance. The recommendation is to use “nc”. All that said, it’s a national decision which TLD to use. +This is usually a simple choice for a Member. However, overseas territories require some thought. The recommended approach depends on the governance of the overseas territory. Take some French examples. Réunion is a French Department – it’s considered part of France, it uses the Euro. Here, we would use the “fr” TLD. New Caledonia is a French overseas territory with top-level-domain of “nc”. It has separate, devolved governance. The recommendation is to use “nc”. All that said, it’s a national decision which TLD to use. *Token 2* is a descriptive name for the centre and this may contain dashes (but not other special characters). -The descriptive name should be something recognisable - not only by our community, but by other users too. Basing things on the Web domain name is likely to ensure that centre identifiers remain unique within a particular country/territory. A UK example this time: the UK's NMS is the Met Office (http://www.metoffice.gov.uk), so “metoffice” is better than “ukmo”footnote:[The “.gov” part of the domain name is superfluous for the purposes of WIS2. There is nothing preventing its use, but it doesn’t add any value.]. Using the 4-letter GTS centre identifiers (*CCCC*) is not recommended because people unfamiliar with the GTS do not understand them. +The descriptive name should be something recognisable - not only by our community, but by other users too. Basing things on the Web domain name is likely to ensure that centre identifiers remain unique within a particular country/territory. A UK example this time: the UK's National Meteorological Service is the Met Office (http://www.metoffice.gov.uk), so “metoffice” is better than “ukmo”footnote:[The “.gov” part of the domain name is superfluous for the purposes of WIS2. There is nothing preventing its use, but it doesn’t add any value.]. Using the 4-letter GTS centre identifiers (*CCCC*) is not recommended because people unfamiliar with the GTS do not understand them. -The Centre Identifier specification says that larger organisations operating multiple centres may wish to register separate centre-ids for each centre. This is good practice. Keeping with the UK example, Met Office operates a NMC, 9 DCPCs and some WIS2 functions, so it’s important to split them out. For example: +The Centre Identifier specification says that larger organisations operating multiple centres may wish to register separate centre-ids for each centre. This is good practice. Keeping with the UK example, Met Office operates a National Meteorological Centre (NMC), 9 DCPCs (e.g., a Volcanic Ash Advisory Centre) and a WIS2 Global Service, so it’s important to split them out. For example: * ``uk-metoffice-nmc`` * ``uk-metoffice-vaac`` -* ``uk-metoffice-globalcache`` +* ``uk-metoffice-global-cache`` Using a system name in the centre-id is not a good idea because these may change over time. Functional designations are long-term durable. Appending ```-test`` may be used to designate test WIS Nodes. -Appending “-test” may be used to designate test WIS Nodes. I don’t think we need that right now, because the whole of WIS2 is in test mode. +Appending “-test” may be used to designate test WIS Nodes. ===== Authentication, authorization, and access control for a WIS2 Node -When configuring your WIS2 Node you need to consider how it will accessed by other components in WIS2 - Global Services and Data Consumers. +When configuring your WIS2 Node you need to consider how it will accessed by Global Services and Data Consumers. -Global Brokers must authenticate when they connect to the MQTT message broker in your WIS2 Node. Simple username and password credentials are usedfootnote:[The default connection credentials for a WIS2 Node message broker are username ``everyone`` and password ``everyone``. WIS2 Node operators should choose credentials that meet their local policies (e.g., password complexity).]. When registering your WIS2 Node with the WMO Secretariat, you will need to provide these credentials. The WMO Secretariat will share these credentials with the Global Service operators and store them in the WIS register. You should not consider these credentials as confidential or secret. +Global Brokers must authenticate when they connect to the MQTT message broker in your WIS2 Node. Username and password credentials are usedfootnote:[The default connection credentials for a WIS2 Node message broker are username ``everyone`` and password ``everyone``. WIS2 Node operators should choose credentials that meet their local policies (e.g., password complexity).]. When registering your WIS2 Node with the WMO Secretariat, you will need to provide these credentials. The WMO Secretariat will share these credentials with the Global Service operators and store them in the WIS register. You should not consider these credentials as confidential or secret. Given that Global Brokers will re-publish notification messages provided by your WIS2 Node, you may decide to restrict access to the MQTT message broker. Global Brokers operate using a fixed IP address which allows you to permit them access using IP filteringfootnote:[In WIS2 we use IP addresses to determine the origin of connections and therefore confer trust to remote systems. It is well documented that IP addresses can be hi-jacked and that there are alternative, more sophisticated, mechanisms available for reliably determining the origin of connections requests, such as Public Key Infrastructure (PKI). However, the complexities of such implementation would introduce a barrier to Member's participation in WIS2. IP addresses are considered to provide an adequate level of trust for the purposes of WIS2: distributing publicly accessible data and messages.]. You must ensure that your MQTT message broker is accessible for more than one Global Broker to provide resilient transmission of notification messages to WIS2. -If your WIS2 Node is only publishing Core datafootnote:[In some cases, WIS2 Nodes will need to serve Core data directly (see section 1.3.x Considerations when providing Core data in WIS2 (TODO CROSSREF)). In these situations, the WIS2 Node data server must remain publicly accessible.], you may also restrict access to your data server - instead, relying on the Global Caches to distribute your data. Similarly, Global Caches also operate on fixed IP addresses allowing connections from them to be easily identified. Again, you must ensure that access is given to more than one Global Broker to ensure resilience. +If your WIS2 Node is only publishing Core datafootnote:[In some cases, WIS2 Nodes will need to serve Core data directly (see <<_considerations_when_providing_core_data_in_wis2>>). In these situations, the WIS2 Node data server must remain publicly accessible.], you may also restrict access to your data server - instead, relying on the Global Caches to distribute your data. Similarly, Global Caches also operate on fixed IP addresses allowing connections from them to be easily identified. Again, you must ensure that access is given to more than one Global Broker to ensure resilience. During registration, the WMO Secretariat will provide host names and IP addresses of the Global Services to enable configuration of access control. diff --git a/guide/sections/part3/information-management.adoc b/guide/sections/part3/information-management.adoc index 8f72d2a..3740e4f 100644 --- a/guide/sections/part3/information-management.adoc +++ b/guide/sections/part3/information-management.adoc @@ -12,7 +12,7 @@ Note: The term "information" is used in a general sense and includes data and pr High-level guidance on information management practices that apply in the context of information related to the Earth system is provided in this part of the Guide. Detailed technical information, such as specification of data formats or quality control and assurance methods, is provided in other parts of the Guide and in other WMO publications. These are referenced where applicable. -The principles of information management are described in Section 6.2. Section 6.3 describes the information management lifecycle through the identification of five focus areas. These are: +The <<_principles_of_information_management>> are described below. Five focus areas are described in <<_the_information_management_lifecycle>>. These are: . Planning, information creation and acquisition. Creation of information using internal and external data sources and the acquisition of information from various sources. . Representation and metadata. Standards to represent metadata, data and information are of primary importance to enable interoperability and long-term usability of the information. @@ -38,7 +38,7 @@ Effective management of information is essential for WMO Centres to deliver oper ===== Principle 1: Information is a valued asset * An information asset is information that has value. This value may be related to the cost of generating and collecting the information, a value associated with the immediate use or a value associated with the longer term preservation and subsequent reuse of the information. -* 10.2.1.2 This value should be recognizable and quantifiable and the asset should have an identifiable lifecycle. Risks associated with, and to, an information asset should also be identified. As such, information management must be considered an integral part of a WMO centre’s responsibilities and needs to be adequately resourced over the full lifecycle of the information. +* This value should be recognizable and quantifiable and the asset should have an identifiable lifecycle. Risks associated with, and to, an information asset should also be identified. As such, information management must be considered an integral part of a WMO centre’s responsibilities and needs to be adequately resourced over the full lifecycle of the information. ===== Principle 2: Information must be managed @@ -50,7 +50,7 @@ Effective management of information is essential for WMO Centres to deliver oper * Information should be developed and managed in accordance with its function and use for internal and external users. * WMO Centres should regularly assess information to ensure that it is fit for its purpose and that processes, procedures, and documentation are adequate. -* Processes should be consistent with the general provisions and principles of quality management as described in the WMO Technical Regulations (WMO-No. 49). +* Processes should be consistent with the general provisions and principles of quality management as described in the _WMO Technical Regulations_ (WMO-No. 49). ===== Principle 4: Information must be standardized and interoperable @@ -74,7 +74,8 @@ Effective management of information is essential for WMO Centres to deliver oper ===== Principle 7: Information should be reusable * In order to maximize the economic benefits of an information asset it should be made as widely available and as accessible as possible. -* The WMO Unified Data Policy encourages the reuse of data and information through the open and unrestricted exchange of core WMO data. The WMO encourages the free and unrestricted exchange of information in all circumstances.10.2.7.3 The publisher should provide an explicit and well-defined license for each information item or dataset as part of the associated metadata. +* The WMO Unified Data Policy encourages the reuse of data and information through the open and unrestricted exchange of core WMO data. The WMO encourages the free and unrestricted exchange of information in all circumstances. +* The publisher should provide an explicit and well-defined license for each information item or dataset as part of the associated metadata. * The Findable, Accessible, Interoperable and Reusable (FAIR) data principles promote open data with the ultimate goal of optimizing reuse of data. These principles should be followed where possible. Note: Information on the FAIR data principles can be found at: FAIR Principles - GO FAIR footnote:[https://go-fair.org] @@ -141,7 +142,7 @@ To ensure traceability and reproducibility the information and documents at this The formats used to store and exchange information should be standardized to ensure its usability, both in the short and long term. It is essential that the information can be accessed many years after archival if required. To ensure this usability, the format and version information should be recorded in the metadata record for the information and should be included in the information where the format allows. -Information exchanged on the WMO Information System and between WMO centres is standardized through the use the formats specified in the WMO Manual on Codes (WMO-No. 306, Volume I.2) and the Manual on the WMO Information System (WMO-No. 1060). This includes the GRIB and BUFR formats for numerical weather prediction products and observational data and the WIS Core Metadata Profile for discovery, access and retrieval metadata. The format for the exchange of station and instrumental metadata, the WIGOS Metadata Data Representation, is also defined in the WMO Manual on Codes (WMO-No. 306, Volume I.3). +Information exchanged on the WMO Information System and between WMO centres is standardized through the use the formats specified in the _Manual on Codes_ (WMO-No. 306), Volume I.2 and the _Manual on WIS_ (WMO-No. 1060), Volume II. This includes the GRIB and BUFR formats for numerical weather prediction products and observational data and the WIS Core Metadata Profile for discovery, access and retrieval metadata. The format for the exchange of station and instrumental metadata, the WIGOS Metadata Data Representation, is also defined in the _Manual on Codes_ (WMO-No. 306), Volume I.3. These formats have been developed within the WMO community to enable the efficient exchange of information between WMO centres and for the information to be interoperable between centres and systems. The formats, including detailed technical information, have also published openly through the WMO manuals, enabling use of the formats and information by other communities, promoting reuse of the information. @@ -151,9 +152,9 @@ The WMO formats specified in the manuals are subject to strong governance proces To maximize the benefits and return on investment in the acquisition and generation of information there needs to be a clear method as to how the information will be published, exchanged and accessed by users. -Information is published on the WMO Information System through the creation of discovery metadata records. These records are publicly searchable and retrievable via WMO cataloguing services, providing access to the records via the Web and via a Web Application Programming Interface (API). The metadata records should include information on how to access the described datasets and services (see also 10.3.3.3.4) and how to subscribe to receive updates and new data. +Information is published on the WMO Information System through the creation of discovery metadata records. These records are publicly searchable and retrievable via WMO cataloguing services, providing access to the records via the Web and via a Web Application Programming Interface (API). The metadata records should include information on how to access the described datasets and services (see also <<_representation_and_metadata>>) and how to subscribe to receive updates and new data. -Guidance on the creation of these discovery metadata records is included in Part V of this Guide. Technical regulations are provided in the Manual on the WMO Information System (WMO-No. 1060). Before exchange and publication the metadata should be assessed using the WMO Core Metadata Profile Key Performance Indicators to ensure usable and high quality metadata in addition to metadata that conforms with the technical standard. +Guidance on the creation of these discovery metadata records is included in Part V of this Guide. Technical regulations are provided in the _Manual on WIS_ (WMO-No. 1060), Volume II. Before exchange and publication the metadata should be assessed using the WMO Core Metadata Profile Key Performance Indicators to ensure usable and high quality metadata in addition to metadata that conforms with the technical standard. Note: Further information on the Key Performance indicators can be found on the WMO Community Website at https://community.wmo.int/activity-areas/wis/wis-metadata-kpis @@ -167,14 +168,14 @@ Note: The newsletter is available from: https://community.wmo.int/news/operation For information to have value it must inform users, aid knowledge discovery and have impact through informed decision making. Ensuring that the user can make effective use of the information is an important step in the information management lifecycle. This takes two forms: -. Provision of suitable information within the discovery metadata (See 10.3.3.4), enabling users to discover and access the information and to assess whether it meets their requirements. This should include licensing information. +. Provision of suitable information within the discovery metadata, enabling users to discover and access the information and to assess whether it meets their requirements. This should include licensing information. . Provision of user guides and documentation on the suitability of the information for different uses, including any technical caveats or restrictions on the use of the information. -For common types of information the guides may be generic or link to standard documentation. Information on the observations available from the WMO Integrated Global Observing System is provided within the Manual and Guide to the WMO Integrated Global Observing system, WMO-No. 1160 and WMO-No. 1165 respectively. This includes information on the expected uses and quality of the data, either directly or through links within. Similarly, information on the data and products available through the Global Data Processing and Forecasting System is provided in the Manual on the Global Data Processing and Forecasting System (WMO-No. 485). +For common types of information the guides may be generic or link to standard documentation. Information on the observations available from the WMO Integrated Global Observing System is provided within the _Manual on the WMO Integrated Global Observing System_ (WMO-No. 1160) and the _Guide to the WMO Integrated Global Observing system_ (WMO-No. 1165) respectively. This includes information on the expected uses and quality of the data, either directly or through links within. Similarly, information on the data and products available through the Global Data Processing and Forecasting System is provided in the _Manual on the Global Data Processing and Forecasting System_ (WMO-No. 485), Volume II. -For non-standard and specialist products targeted user guides may be more appropriate. These should include a plain text summary for the non-technical user and should also be accessible and retrievable via a link within the discovery metadata. Any user guide should be in addition to the technical documentation described under Planning, information creation and acquisition (see 10.3.3.1). +For non-standard and specialist products targeted user guides may be more appropriate. These should include a plain text summary for the non-technical user and should also be accessible and retrievable via a link within the discovery metadata. Any user guide should be in addition to the technical documentation described under <<_planning,_information_creation_and_acquisition>>. -Updates and the availability of new information should be announced and published via the WMO Operational Newsletter (see 10.3.3.4.6). Other communication methods may also be used but these should not be in place of the operational newsletter. It is also recommended to allow users to subscribe to receive updates directly. +Updates and the availability of new information should be announced and published via the WMO Operational Newsletter (see <<_publication_and_exchange_of_information>>). Other communication methods may also be used but these should not be in place of the operational newsletter. It is also recommended to allow users to subscribe to receive updates directly. The discovery metadata should include a valid point of contact, enabling users to provide feedback and ask questions about the information provided. @@ -182,13 +183,13 @@ The discovery metadata should include a valid point of contact, enabling users t The type of storage used should be appropriate to the type of information stored. Core information exchanged operationally should be stored and made available via high-availability and low latency media and services. For some operation critical information, such as hazard warnings, there is a requirement for the end-to-end global distribution of the information to be completed in two minutes. For other operational data there is a requirement for the global exchange to be completed in 15 minutes. -The storage requirements for non-operational services and information may be different but the guidance provided in this section applies equally. Further information on the performance requirements is provided within the WIS Technical Specifications listed in the Manual on the WMO Information System (WMO-No. 1060). +The storage requirements for non-operational services and information may be different but the guidance provided in this section applies equally. Further information on the performance requirements is provided within the WI2S Technical Specifications listed in the _Manual on WIS_ (WMO-No. 1060), Volume II. Backup policies and data recovery plans should be documented as part of the information management plan. These should be implemented either before or when the information is created or acquired and should include both the information and the associated metadata. The backup and recovery process should be routinely tested. Specific guidance on the expectations and requirements for WMO centres is provided under the operational guidance in Part VII of this Guide. Business rules governing the access to and modification of the information should be clearly documented in the information management plan. This must include the clear specification of roles and responsibilities of those managing the information. Information on who can authorize the archival and disposal of the information and the processes for doing so should be included. The roles associated with an information resource are standardized as part of the WIS Core Metadata Profile, see Part V of this Guide for further information. -The archival and long-term preservation of an information resource should be identified and included in the information management plan. This may be at a national data centre and/or a WMO centre. The WMO centres are recommended for globally exchanged core data and include those centres contributing to the Global Atmosphere Watch, the Global Climate Observing System and the Marine Climate Data System (see Manual on Marine Meteorological Services, WMO-No. 558), as well as the WMO World Data Centres and those defined in the Manual on the WMO Information System (WMO-No. 1060) and those defined in the Manual on the Global Data Processing and Forecasting System (WMO-No. 485). +The archival and long-term preservation of an information resource should be identified and included in the information management plan. This may be at a national data centre and/or a WMO centre. The WMO centres are recommended for globally exchanged core data and include those centres contributing to the Global Atmosphere Watch, the Global Climate Observing System and the Marine Climate Data System (see _Manual on Marine Meteorological Services_ (WMO-No. 558), Volume II, as well as the WMO World Data Centres and those defined in the _Manual on WIS_ (WMO-No. 1060), Volume II and those defined in the _Manual on the Global Data Processing and Forecasting System_ (WMO-No. 485), Volume II. Earth system information, especially observational data, are often irreplaceable. Other information, whilst technically replaceable, is often costly to produce and therefore not easily replaceable. This includes output from numerical models and simulations. Before an information resource is marked for disposal careful consideration must be given to whether long term archival or disposal is more appropriate. This consideration must follow a clearly defined process documented in the information management plan. @@ -198,8 +199,8 @@ When an information resource is marked for disposal the reasons for disposal, in ===== Technology and technology migration -Information managers must be aware of the need to ensure that the technologies, hardware and software used do not become obsolete and must be aware of emerging data issues. This topic is discussed further in the WMO Guide to Emerging Data Issues (WMO-No. 1239). +Information managers must be aware of the need to ensure that the technologies, hardware and software used do not become obsolete and must be aware of emerging data issues. This topic is discussed further in the _WMO Guidelines on Emerging Data Issues_ (WMO-No. 1239). ===== Information security -Further information on information security and best practices can be found in the WMO Guide to Information Technology Security (WMO-No. 1115). +Further information on information security and best practices can be found in the _Guide to Information Technology Security_ (WMO-No. 1115).