Failed validating 'multipleOf'

[jpuris]

The problem

I'm not sure what this is and how it works, but seems I'm hitting the upstream issue documented in the Meltano SDK issues - meltano/sdk#344

Also discussed in python-jsonschema/jsonschema#247

The error

jsonschema.exceptions.ValidationError: -4.8 is not a multiple of 0.0001 cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
                               cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
Failed validating 'multipleOf' in schema['properties']['AMT_EXCL_TAX']: cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
    {'multipleOf': 0.0001, 'type': ['null', 'number']} cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
                               cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
On instance['AMT_EXCL_TAX']:   cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
    -4.8                       cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake

This particular column in Oracle is defined as

AMT_EXCL_TAX NUMBER(28, 4) DEFAULT 0 NOT NULL

with 20 example values

AMT_EXCL_TAX
-2332.4000
-1212.9600
-1146.0000
-929.0000
-619.0000
-500.0000
-448.0000
-346.0000
-304.0000
-203.5200
-103.2000
-94.9000
-50.4000
-4.8000
21.6000
31.2000
325.2000
1565.6000
3605.0300
3939.7500

MRE

I do not have a valid MRE at this time.. Please let me know, if I need to make one.

Runtime details

❯ python --version                                    
Python 3.10.11
❯ meltano --version                                        
meltano, version 2.19.1
❯ pip freeze | grep json     
check-jsonschema==0.22.0
jsonschema==4.17.3

I'm running a job with

meltano --environment=dev run tap-oracle target-snowflake

meltano.yaml

version: 1
default_environment: dev
project_id: b1d562d0-d114-42a7-bad1-957891a7285b

environments:
  - name: dev
    config:
      plugins:
        loaders:
          - name: target-snowflake
            config:
              database: redacted
              warehouse: redacted
              role: redacted
              user: redacted
              schema: redacted
              default_target_schema: redacted
  - name: uat
  - name: prod

plugins:
  extractors:
    - name: tap-oracle
      pip_url: git+https://github.com/christianelliott/pipelinewise-tap-oracle.git
      config:
        filter_schemas: MY_SCHEMA
        default_replication_method: FULL_TABLE
        port: 1521
        host: redacted
        user: redacted
        service_name: redacted
        use_ora_rowscn: true
        add_metadata_columns: true
      select:
        - MY_SCHEMA-FINANCIAL_TRANSACTION.*
      metadata:
        MY_SCHEMA-FINANCIAL_TRANSACTION:
          replication-method: INCREMENTAL
          replication-key: ID
  loaders:
    - name: target-snowflake
      variant: meltanolabs
      pip_url: meltanolabs-target-snowflake
      config:
        account: redacted.eu-west-1
        add_record_metadata: true
        clean_up_batch_files: true
        flattening_enabled: false
jobs:
  - name: icx--snowflake
    tasks:
      - tap-oracle target-snowflake

2023-07-03T19:56:58.891691Z [info     ] 2023-07-03 21:56:58,891 | INFO     | snowflake.connector.cursor | Number of results in first chunk: 1 cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-03T19:59:07.192610Z [info     ] time=2023-07-03 21:59:07 name=singer level=INFO message=METRIC: b'{"type":"counter","metric":"record_count","value":1,"tags":{}}' cmd_type=elb consumer=False name=tap-oracle producer=True stdio=stderr string_id=tap-oracle
2023-07-03T19:59:07.254068Z [info     ] Traceback (most recent call last): cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-03T19:59:07.254354Z [info     ]   File "/meltano_poc/project/.meltano/loaders/target-snowflake/venv/bin/target-snowflake", line 8, in <module> cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-03T19:59:07.255159Z [info     ]     sys.exit(TargetSnowflake.cli()) cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-03T19:59:07.255444Z [info     ]   File "/meltano_poc/project/.meltano/loaders/target-snowflake/venv/lib/python3.10/site-packages/click/core.py", line 1130, in __call__ cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-03T19:59:07.256212Z [info     ]     return self.main(*args, **kwargs) cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-03T19:59:07.256460Z [info     ]   File "/meltano_poc/project/.meltano/loaders/target-snowflake/venv/lib/python3.10/site-packages/click/core.py", line 1055, in main cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-03T19:59:07.256815Z [info     ]     rv = self.invoke(ctx)      cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-03T19:59:07.257037Z [info     ]   File "/meltano_poc/project/.meltano/loaders/target-snowflake/venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-03T19:59:07.257552Z [info     ]     return ctx.invoke(self.callback, **ctx.params) cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-03T19:59:07.257721Z [info     ]   File "/meltano_poc/project/.meltano/loaders/target-snowflake/venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-03T19:59:07.257945Z [info     ]     return __callback(*args, **kwargs) cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-03T19:59:07.258128Z [info     ]   File "/meltano_poc/project/.meltano/loaders/target-snowflake/venv/lib/python3.10/site-packages/singer_sdk/target_base.py", line 546, in invoke cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-03T19:59:07.258422Z [info     ]     target.listen(file_input)  cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-03T19:59:07.258502Z [info     ]   File "/meltano_poc/project/.meltano/loaders/target-snowflake/venv/lib/python3.10/site-packages/singer_sdk/io_base.py", line 33, in listen cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-03T19:59:07.258562Z [info     ]     self._process_lines(file_input) cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-03T19:59:07.258615Z [info     ]   File "/meltano_poc/project/.meltano/loaders/target-snowflake/venv/lib/python3.10/site-packages/singer_sdk/target_base.py", line 291, in _process_lines cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-03T19:59:07.258667Z [info     ]     counter = super()._process_lines(file_input) cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-03T19:59:07.258717Z [info     ]   File "/meltano_poc/project/.meltano/loaders/target-snowflake/venv/lib/python3.10/site-packages/singer_sdk/io_base.py", line 79, in _process_lines cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-03T19:59:07.258855Z [info     ]     self._process_record_message(line_dict) cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-03T19:59:07.259054Z [info     ]   File "/meltano_poc/project/.meltano/loaders/target-snowflake/venv/lib/python3.10/site-packages/singer_sdk/target_base.py", line 338, in _process_record_message cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-03T19:59:07.259223Z [info     ]     sink._validate_and_parse(transformed_record) cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-03T19:59:07.259390Z [info     ]   File "/meltano_poc/project/.meltano/loaders/target-snowflake/venv/lib/python3.10/site-packages/singer_sdk/sinks/core.py", line 314, in _validate_and_parse cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-03T19:59:07.259445Z [info     ]     self._validator.validate(record) cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-03T19:59:07.259495Z [info     ]   File "/meltano_poc/project/.meltano/loaders/target-snowflake/venv/lib/python3.10/site-packages/jsonschema/validators.py", line 314, in validate cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-03T19:59:07.259973Z [info     ]     raise error                cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-03T19:59:07.260326Z [info     ] jsonschema.exceptions.ValidationError: -4.8 is not a multiple of 0.0001 cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-03T19:59:07.260406Z [info     ]                                cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-03T19:59:07.260493Z [info     ] Failed validating 'multipleOf' in schema['properties']['AMT_EXCL_TAX']: cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-03T19:59:07.260721Z [info     ]     {'multipleOf': 0.0001, 'type': ['null', 'number']} cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-03T19:59:07.260791Z [info     ]                                cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-03T19:59:07.260846Z [info     ] On instance['AMT_EXCL_TAX']:   cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-03T19:59:07.260946Z [info     ]     -4.8                       cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-03T19:59:07.298085Z [info     ] 2023-07-03 21:59:07,297 | INFO     | snowflake.connector.connection | closed cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-03T19:59:07.298331Z [info     ] 2023-07-03 21:59:07,298 | INFO     | snowflake.connector.connection | No async queries seem to be running, deleting session cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-03T19:59:07.368337Z [error    ] [Errno 32] Broken pipe
...

[jpuris]

Now that a fix has been merged for meltano/sdk#344, we would need to test the target with newer SDK.

How can we go about doing that? 🤔

[edgarrmondragon]

@jpuris that will be addressed by #73

[jpuris]

Can confirm, the issue has been fixed. Thank you!