From 9a13296bd826c2ac97b403cea9b822b378e58383 Mon Sep 17 00:00:00 2001 From: Jack Berg Date: Thu, 26 Oct 2023 18:20:39 -0500 Subject: [PATCH 1/5] Define file configuration file format and env var substitution --- .../configuration/file-configuration.md | 101 ++++++++++++++++-- 1 file changed, 95 insertions(+), 6 deletions(-) diff --git a/specification/configuration/file-configuration.md b/specification/configuration/file-configuration.md index e9a40ba4d26..72ec3b41645 100644 --- a/specification/configuration/file-configuration.md +++ b/specification/configuration/file-configuration.md @@ -12,6 +12,8 @@ linkTitle: File - [Configuration Model](#configuration-model) * [Stability Definition](#stability-definition) - [Configuration file](#configuration-file) + * [YAML file format](#yaml-file-format) + * [Environment variable substitution](#environment-variable-substitution) - [SDK Configuration](#sdk-configuration) * [In-Memory Configuration Model](#in-memory-configuration-model) * [Operations](#operations) @@ -46,12 +48,80 @@ TODO: define stability guarantees and backwards compatibility ## Configuration file -A configuration file is a file representation of +A configuration file is a serialized file-based representation of the [Configuration Model](#configuration-model). -TODO: define acceptable file formats +Configuration files SHOULD use one the following serialization formats: -TODO: define environment variable substitution +### YAML file format + +[YAML](https://yaml.org/spec/1.2.2/) configuration files SHOULD follow YAML spec +revision >= 1.2. + +YAML configuration files MUST use file extensions `.yaml` or `.yml`. + +TODO: define JSON file format once prototypes are available + +### Environment variable substitution + +Configuration files support environment variables substitution for references +which match the following regular expression: + +```regexp +\$\{env:(?[a-zA-Z_]+[a-zA-Z0-9_]*)} +``` + +The `ENV_NAME` MUST start with an alphabetic or `_` character, and is followed +by 0 or more alphanumeric or `_` characters. + +For example, `${env:API_KEY}` is valid, while `${env:1API_KEY}` +and `${env:API_$KEY}` are invalid. + +Environment variable substitution MUST only apply to scalar values. NOTE, +mapping keys are not candidates for substitution. + +If a referenced environment variable is not defined, it MUST be replaced with an +empty value. + +Node types MUST be interpreted after environment variable substitution takes +place. This ensures the environment string representation of boolean, integer, +or floating point fields can be properly converted to expected types. + +For example, consider the following environment variables, +and [YAML](#yaml-file-format) configuration file: + +```shell +export STRING_VALUE="value" +export BOOl_VALUE="true" +export INT_VALUE="1" +export FLOAT_VALUE="1.1" +``` + +```yaml +string_key: ${env:STRING_VALUE} # Valid reference to STRING_VALUE +other_string_key: "${env:STRING_VALUE}" # Valid reference to STRING_VALUE inside double quotes +another_string_key: "${env:BOOl_VALUE}" # Valid reference to BOOl_VALUE inside double quotes +bool_key: ${env:BOOl_VALUE} # Valid reference to BOOl_VALUE +int_key: ${env:INT_VALUE} # Valid reference to INT_VALUE +float_key: ${env:FLOAT_VALUE} # Valid reference to FLOAT_VALUE +combo_string_key: foo ${env:STRING_VALUE} ${env:FLOAT_VALUE} # Valid reference to STRING_VALUE and FLOAT_VALUE +undefined_key: ${env:UNDEFINED_KEY} # Invalid reference, UNDEFINED_KEY is not defined and is replaced with "" +${env:STRING_VALUE}: value # Invalid reference, substitution is not valid in mapping keys and reference is ignored +``` + +Environment variable substitution results in the following YAML: + +```yaml +string_key: value # Interpreted as type string, tag URI tag:yaml.org,2002:str +other_string_key: "value" # Interpreted as type string, tag URI tag:yaml.org,2002:str +another_string_key: "true" # Interpreted as type string, tag URI tag:yaml.org,2002:str +bool_key: true # Interpreted as type bool, tag URI tag:yaml.org,2002:bool +int_key: 1 # Interpreted as type int, tag URI tag:yaml.org,2002:int +float_key: 1.1 # Interpreted as type float, tag URI tag:yaml.org,2002:float +combo_string_key: foo value 1.1 # Interpreted as type string, tag URI tag:yaml.org,2002:str +undefined_key: # Interpreted as type null, tag URI tag:yaml.org,2002:null +${env:STRING_VALUE}: value # Interpreted as type string, tag URI tag:yaml.org,2002:str +``` ## SDK Configuration @@ -81,10 +151,19 @@ with OpAmp Parse and validate a [configuration file](#configuration-file). +Parse MUST perform [environment variable substitution](#environment-variable-substitution). + +Parse MUST interpret null as equivalent to unset. + **Parameters:** * `file`: The [configuration file](#configuration-file) to parse. This MAY be a file path, or language specific file data structure, or a stream of a file's content. +* `file_format`: The file format of the `file` (e.g. [yaml](#yaml-file-format)). + Implementations MAY accept a `file_format` parameter, or infer it from + the `file` extension, or include file format specific overloads of `parse`, + e.g. `parseYaml(file)`. If `parse` accepts `file_format`, the API SHOULD be + structured so a user is obligated to provide it. **Returns:** [configuration model](#in-memory-configuration-model) @@ -94,12 +173,19 @@ This SHOULD return an error if: * The parsed `file` content does not conform to the [configuration model](#configuration-model) schema. -TODO: define behavior if some portion of configuration model is not supported - #### Create Interpret [configuration model](#in-memory-configuration-model) and return SDK components. +If a field is null or unset and a default value is defined, Create MUST ensure +the SDK component is configured with the default value. If a field is null or +unset and no default value is defined, Create SHOULD return an error. For +example, if configuring +the [span batching processor](../trace/sdk.md#batching-processor) and +the `scheduleDelayMillis` field is null or unset, the component is configured +with the default value of `5000`. However, if the `exporter` field is null or +unset, Create fails fast since there is no default value for `exporter`. + **Parameters:** * `configuration` - The configuration model. @@ -115,7 +201,10 @@ The multiple responses MAY be returned using a tuple, or some other data structure encapsulating the components. This SHOULD return an error if it encounters an error in `configuration` (i.e. -fail fast). +fail fast) in accordance with +initialization [error handling principles](../error-handling.md#basic-error-handling-principles). + +TODO: define behavior if some portion of configuration model is not supported ## References From e8c2ca5fb55fbefe4266828311d82f3c24b94441 Mon Sep 17 00:00:00 2001 From: Jack Berg Date: Tue, 21 Nov 2023 13:13:17 -0600 Subject: [PATCH 2/5] Remove env: prefix in env var substitution syntax --- .../configuration/file-configuration.md | 26 +++++++++---------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/specification/configuration/file-configuration.md b/specification/configuration/file-configuration.md index 72ec3b41645..695a6629052 100644 --- a/specification/configuration/file-configuration.md +++ b/specification/configuration/file-configuration.md @@ -68,14 +68,14 @@ Configuration files support environment variables substitution for references which match the following regular expression: ```regexp -\$\{env:(?[a-zA-Z_]+[a-zA-Z0-9_]*)} +\$\{(?[a-zA-Z_]+[a-zA-Z0-9_]*)} ``` The `ENV_NAME` MUST start with an alphabetic or `_` character, and is followed by 0 or more alphanumeric or `_` characters. -For example, `${env:API_KEY}` is valid, while `${env:1API_KEY}` -and `${env:API_$KEY}` are invalid. +For example, `${API_KEY}` is valid, while `${1API_KEY}` and `${API_$KEY}` are +invalid. Environment variable substitution MUST only apply to scalar values. NOTE, mapping keys are not candidates for substitution. @@ -98,15 +98,15 @@ export FLOAT_VALUE="1.1" ``` ```yaml -string_key: ${env:STRING_VALUE} # Valid reference to STRING_VALUE -other_string_key: "${env:STRING_VALUE}" # Valid reference to STRING_VALUE inside double quotes -another_string_key: "${env:BOOl_VALUE}" # Valid reference to BOOl_VALUE inside double quotes -bool_key: ${env:BOOl_VALUE} # Valid reference to BOOl_VALUE -int_key: ${env:INT_VALUE} # Valid reference to INT_VALUE -float_key: ${env:FLOAT_VALUE} # Valid reference to FLOAT_VALUE -combo_string_key: foo ${env:STRING_VALUE} ${env:FLOAT_VALUE} # Valid reference to STRING_VALUE and FLOAT_VALUE -undefined_key: ${env:UNDEFINED_KEY} # Invalid reference, UNDEFINED_KEY is not defined and is replaced with "" -${env:STRING_VALUE}: value # Invalid reference, substitution is not valid in mapping keys and reference is ignored +string_key: ${STRING_VALUE} # Valid reference to STRING_VALUE +other_string_key: "${STRING_VALUE}" # Valid reference to STRING_VALUE inside double quotes +another_string_key: "${BOOl_VALUE}" # Valid reference to BOOl_VALUE inside double quotes +bool_key: ${BOOl_VALUE} # Valid reference to BOOl_VALUE +int_key: ${INT_VALUE} # Valid reference to INT_VALUE +float_key: ${FLOAT_VALUE} # Valid reference to FLOAT_VALUE +combo_string_key: foo ${STRING_VALUE} ${FLOAT_VALUE} # Valid reference to STRING_VALUE and FLOAT_VALUE +undefined_key: ${UNDEFINED_KEY} # Invalid reference, UNDEFINED_KEY is not defined and is replaced with "" +${STRING_VALUE}: value # Invalid reference, substitution is not valid in mapping keys and reference is ignored ``` Environment variable substitution results in the following YAML: @@ -120,7 +120,7 @@ int_key: 1 # Interpreted as type int, tag URI tag:yaml.org float_key: 1.1 # Interpreted as type float, tag URI tag:yaml.org,2002:float combo_string_key: foo value 1.1 # Interpreted as type string, tag URI tag:yaml.org,2002:str undefined_key: # Interpreted as type null, tag URI tag:yaml.org,2002:null -${env:STRING_VALUE}: value # Interpreted as type string, tag URI tag:yaml.org,2002:str +${STRING_VALUE}: value # Interpreted as type string, tag URI tag:yaml.org,2002:str ``` ## SDK Configuration From be346236df0d2087dc1ccde9e602a37ed040d038 Mon Sep 17 00:00:00 2001 From: Jack Berg Date: Tue, 12 Dec 2023 09:57:11 -0600 Subject: [PATCH 3/5] PR feedback --- specification/configuration/file-configuration.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/specification/configuration/file-configuration.md b/specification/configuration/file-configuration.md index 695a6629052..addd2b6d906 100644 --- a/specification/configuration/file-configuration.md +++ b/specification/configuration/file-configuration.md @@ -68,7 +68,7 @@ Configuration files support environment variables substitution for references which match the following regular expression: ```regexp -\$\{(?[a-zA-Z_]+[a-zA-Z0-9_]*)} +\$\{(?[a-zA-Z_][a-zA-Z0-9_]*)} ``` The `ENV_NAME` MUST start with an alphabetic or `_` character, and is followed @@ -77,8 +77,8 @@ by 0 or more alphanumeric or `_` characters. For example, `${API_KEY}` is valid, while `${1API_KEY}` and `${API_$KEY}` are invalid. -Environment variable substitution MUST only apply to scalar values. NOTE, -mapping keys are not candidates for substitution. +Environment variable substitution MUST only apply to scalar values. Mapping keys +are not candidates for substitution. If a referenced environment variable is not defined, it MUST be replaced with an empty value. From 38eb5f145a082de44fd660db2c1a46254e63a7ef Mon Sep 17 00:00:00 2001 From: Jack Berg Date: Tue, 12 Dec 2023 10:55:38 -0600 Subject: [PATCH 4/5] YAML injection is not allowed --- .../configuration/file-configuration.md | 28 +++++++++++-------- 1 file changed, 17 insertions(+), 11 deletions(-) diff --git a/specification/configuration/file-configuration.md b/specification/configuration/file-configuration.md index addd2b6d906..eb909a9f383 100644 --- a/specification/configuration/file-configuration.md +++ b/specification/configuration/file-configuration.md @@ -60,7 +60,7 @@ revision >= 1.2. YAML configuration files MUST use file extensions `.yaml` or `.yml`. -TODO: define JSON file format once prototypes are available +TODO: decide if JSON file format is required ### Environment variable substitution @@ -85,7 +85,10 @@ empty value. Node types MUST be interpreted after environment variable substitution takes place. This ensures the environment string representation of boolean, integer, -or floating point fields can be properly converted to expected types. +or floating point fields can be properly converted to expected types. + +It MUST NOT be possible to inject YAML structures by environment variables. For +example, references to `INVALID_MAP_VALUE` environment variable below. For example, consider the following environment variables, and [YAML](#yaml-file-format) configuration file: @@ -95,12 +98,14 @@ export STRING_VALUE="value" export BOOl_VALUE="true" export INT_VALUE="1" export FLOAT_VALUE="1.1" +export INVALID_MAP_VALUE"value\nkey:value" # An invalid attempt to inject a map key into the YAML ``` ```yaml string_key: ${STRING_VALUE} # Valid reference to STRING_VALUE other_string_key: "${STRING_VALUE}" # Valid reference to STRING_VALUE inside double quotes another_string_key: "${BOOl_VALUE}" # Valid reference to BOOl_VALUE inside double quotes +yet_another_string_key: ${INVALID_MAP_VALUE} # Valid reference to INVALID_MAP_VALUE, but YAML structure from INVALID_MAP_VALUE MUST NOT be injected bool_key: ${BOOl_VALUE} # Valid reference to BOOl_VALUE int_key: ${INT_VALUE} # Valid reference to INT_VALUE float_key: ${FLOAT_VALUE} # Valid reference to FLOAT_VALUE @@ -112,15 +117,16 @@ ${STRING_VALUE}: value # Invalid reference, subst Environment variable substitution results in the following YAML: ```yaml -string_key: value # Interpreted as type string, tag URI tag:yaml.org,2002:str -other_string_key: "value" # Interpreted as type string, tag URI tag:yaml.org,2002:str -another_string_key: "true" # Interpreted as type string, tag URI tag:yaml.org,2002:str -bool_key: true # Interpreted as type bool, tag URI tag:yaml.org,2002:bool -int_key: 1 # Interpreted as type int, tag URI tag:yaml.org,2002:int -float_key: 1.1 # Interpreted as type float, tag URI tag:yaml.org,2002:float -combo_string_key: foo value 1.1 # Interpreted as type string, tag URI tag:yaml.org,2002:str -undefined_key: # Interpreted as type null, tag URI tag:yaml.org,2002:null -${STRING_VALUE}: value # Interpreted as type string, tag URI tag:yaml.org,2002:str +string_key: value # Interpreted as type string, tag URI tag:yaml.org,2002:str +other_string_key: "value" # Interpreted as type string, tag URI tag:yaml.org,2002:str +another_string_key: "true" # Interpreted as type string, tag URI tag:yaml.org,2002:str +yet_another_string_key: "value\nkey:value" # Interpreted as type string, tag URI tag:yaml.org,2002:str +bool_key: true # Interpreted as type bool, tag URI tag:yaml.org,2002:bool +int_key: 1 # Interpreted as type int, tag URI tag:yaml.org,2002:int +float_key: 1.1 # Interpreted as type float, tag URI tag:yaml.org,2002:float +combo_string_key: foo value 1.1 # Interpreted as type string, tag URI tag:yaml.org,2002:str +undefined_key: # Interpreted as type null, tag URI tag:yaml.org,2002:null +${STRING_VALUE}: value # Interpreted as type string, tag URI tag:yaml.org,2002:str ``` ## SDK Configuration From 60756c6da977b15e7d99a61c95051e741a923461 Mon Sep 17 00:00:00 2001 From: Jack Berg Date: Tue, 12 Dec 2023 14:00:26 -0600 Subject: [PATCH 5/5] lint --- specification/configuration/file-configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/configuration/file-configuration.md b/specification/configuration/file-configuration.md index eb909a9f383..8b2bcec2564 100644 --- a/specification/configuration/file-configuration.md +++ b/specification/configuration/file-configuration.md @@ -85,7 +85,7 @@ empty value. Node types MUST be interpreted after environment variable substitution takes place. This ensures the environment string representation of boolean, integer, -or floating point fields can be properly converted to expected types. +or floating point fields can be properly converted to expected types. It MUST NOT be possible to inject YAML structures by environment variables. For example, references to `INVALID_MAP_VALUE` environment variable below.