feat: Support validating configuration for any tap with a dynamic catalog

[edgarrmondragon]

Closes #294

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (596323b) 88.02% compared to head (8ba50d3) 88.03%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #937      +/-   ##
==========================================
+ Coverage   88.02%   88.03%   +0.01%     
==========================================
  Files          60       60              
  Lines        5001     5007       +6     
  Branches     1014     1014              
==========================================
+ Hits         4402     4408       +6     
  Misses        423      423              
  Partials      176      176              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[edgarrmondragon]

Before:

Long, unreadable output

$ poetry run python samples/sample_tap_sqlite --discover                                            
2024-01-08 20:40:09,544 | WARNING  | tap-sqlite-sample    | Config validation failed: 'path_to_db' is a required property
JSONSchema was: {'type': 'object', 'properties': {'path_to_db': {'type': ['string'], 'description': 'The path to your SQLite database file(s).', 'examples': ['./path/to/my.db', '/absolute/path/to/my.db']}, 'stream_maps': {'type': ['object', 'null'], 'properties': {}, 'description': 'Config object for stream maps capability. For more information check out [Stream Maps](https://sdk.meltano.com/en/latest/stream_maps.html).'}, 'stream_map_config': {'type': ['object', 'null'], 'properties': {}, 'description': 'User-defined config values to be used within map expressions.'}, 'flattening_enabled': {'type': ['boolean', 'null'], 'description': "'True' to enable schema flattening and automatically expand nested properties."}, 'flattening_max_depth': {'type': ['integer', 'null'], 'description': 'The max depth to flatten schemas.'}, 'batch_config': {'type': ['object', 'null'], 'properties': {'encoding': {'type': ['object', 'null'], 'properties': {'format': {'type': ['string', 'null'], 'description': 'Format to use for batch files.', 'enum': ['jsonl', 'parquet']}, 'compression': {'type': ['string', 'null'], 'description': 'Compression format to use for batch files.', 'enum': ['gzip', 'none']}}, 'description': 'Specifies the format and compression of the batch files.'}, 'storage': {'type': ['object', 'null'], 'properties': {'root': {'type': ['string', 'null'], 'description': 'Root path to use when writing batch files.'}, 'prefix': {'type': ['string', 'null'], 'description': 'Prefix to use when writing batch files.'}}, 'description': 'Defines the storage layer to use when writing batch files'}}}}, 'required': ['path_to_db']}
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/edgarramirez/Code/meltano/sdk/samples/sample_tap_sqlite/__main__.py", line 5, in <module>
    SQLiteTap.cli()
  File "/Users/edgarramirez/Code/meltano/sdk/.venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/edgarramirez/Code/meltano/sdk/.venv/lib/python3.11/site-packages/click/core.py", line 1077, in main
    with self.make_context(prog_name, args, **extra) as ctx:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/edgarramirez/Code/meltano/sdk/.venv/lib/python3.11/site-packages/click/core.py", line 943, in make_context
    self.parse_args(ctx, args)
  File "/Users/edgarramirez/Code/meltano/sdk/.venv/lib/python3.11/site-packages/click/core.py", line 1408, in parse_args
    value, args = param.handle_parse_result(ctx, opts, args)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/edgarramirez/Code/meltano/sdk/.venv/lib/python3.11/site-packages/click/core.py", line 2400, in handle_parse_result
    value = self.process_value(ctx, value)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/edgarramirez/Code/meltano/sdk/.venv/lib/python3.11/site-packages/click/core.py", line 2362, in process_value
    value = self.callback(ctx, self, value)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/edgarramirez/Code/meltano/sdk/singer_sdk/tap_base.py", line 529, in cb_discover
    tap.run_discovery()
  File "/Users/edgarramirez/Code/meltano/sdk/singer_sdk/tap_base.py", line 289, in run_discovery
    catalog_text = self.catalog_json_text
                   ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/edgarramirez/Code/meltano/sdk/singer_sdk/tap_base.py", line 309, in catalog_json_text
    return json.dumps(self.catalog_dict, indent=2)
                      ^^^^^^^^^^^^^^^^^
  File "/Users/edgarramirez/Code/meltano/sdk/singer_sdk/tap_base.py", line 660, in catalog_dict
    result["streams"].extend(connector.discover_catalog_entries())
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/edgarramirez/Code/meltano/sdk/singer_sdk/connectors/sql.py", line 510, in discover_catalog_entries
    engine = self._engine
             ^^^^^^^^^^^^
  File "/Users/edgarramirez/Code/meltano/sdk/singer_sdk/connectors/sql.py", line 309, in _engine
    self._cached_engine = self.create_engine()
                          ^^^^^^^^^^^^^^^^^^^^
  File "/Users/edgarramirez/Code/meltano/sdk/singer_sdk/connectors/sql.py", line 328, in create_engine
    self.sqlalchemy_url,
    ^^^^^^^^^^^^^^^^^^^
  File "/Users/edgarramirez/Code/meltano/sdk/singer_sdk/connectors/sql.py", line 161, in sqlalchemy_url
    self._sqlalchemy_url = self.get_sqlalchemy_url(self.config)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/edgarramirez/Code/meltano/sdk/samples/sample_tap_sqlite/__init__.py", line 21, in get_sqlalchemy_url
    return f"sqlite:///{config[DB_PATH_CONFIG]}"
                        ~~~~~~^^^^^^^^^^^^^^^^
KeyError: 'path_to_db'

After:

Slightly less ugly output (helped a ton by)

$ poetry run python samples/sample_tap_sqlite --discover
2024-01-08 20:40:54,326 | ERROR    | tap-sqlite-sample    | Config validation error: 'path_to_db' is a required property

[codspeed-hq]

CodSpeed Performance Report

Merging #937 will not alter performance

_{Comparing fix/configurable-dynamic-catalog (8ba50d3) with main (596323b)}

Summary

✅ 6 untouched benchmarks