OmegaConfigLoader does not ignores keys starting with '_'

[zerodarkzone]

Description

OmegaConfigLoader does not ignores keys starting with '_'.
Because of this, there are some problems using yaml templates in the catalog.
Kedro tries to instantiate catalog entries starting with '_' so it is difficult to define partial entries because of missing parameters.

Context

I'm trying to use the OmegaConfigLoader with yaml templates in the catalog

Steps to Reproduce

1. Define OmegaConfigLoader as the config loader in the project settings
2. Create a partial yaml template in the catalog definition. ej:

_spark_dataset: &spark_dataset
  type: spark.SparkDataSet
  file_format: delta

3. Use the template in a catalog entry.

test_table:
  <<: *spark_dataset
  filepath: <path>

Expected Result

The catalog should have one entry called 'test_table'.

{
    'test_table': {
        'type': 'spark.SparkDataSet',
        'filepath': 'path', 
        'file_format': 'delta'
    }
}

Actual Result

The catalog tries to instantiate a dataset called '_spark_dataset' and fails because of a missing parameter.

kedro.io.core.DataSetError:
__init__() missing 1 required positional argument: 'filepath'.
DataSet '_spark_dataset' must only contain arguments valid for the constructor of 'kedro.extras.datasets.spark.spark_dataset.SparkDataSet'.

Your Environment

• Kedro version: 0.18.6
• Python version: 3.9.16

[merelcht]

Hi @zerodarkzone, thanks for trying out our new OmegaConfigLoader! The behaviour you describe here is expected for the time being. You shouldn't use yaml anchors for this kind of thing but omegaconf interpolation instead. See omry/omegaconf#93 and https://omegaconf.readthedocs.io/en/2.3_branch/usage.html#variable-interpolation

However, the OmegaConfigLoader only support variable interpolation for parameters at the moment. We have an open issue to explore how to add templating to the catalog as well: #2175 but that isn't implemented for now.

[merelcht]

I'm closing this issue for now, because this is expected behaviour and we don't plan on adding yaml anchor support for the OmegaConfigLoader.

[zerodarkzone]

I'm closing this issue for now, because this is expected behaviour and we don't plan on adding yaml anchor support for the OmegaConfigLoader.

Thank you for the response.
Sadly for me, this makes the OmegaConfigLoader kind of useless. I'll keep using the TemplatedConfigLoader for now.