Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix facebook marketing source SAT fail on invalid config file #5621

Merged
merged 4 commits into from
Aug 31, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,13 @@ repos:
args: ["--tmpl=LICENSE", "--ext=py", "-f"]

- repo: https://github.com/ambv/black
rev: stable
rev: 20.8b1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did you decide to specify the correct version and not leave the stable?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This version are used in our gradle config.

hooks:
- id: black
args: ["--line-length=140"]

- repo: https://github.com/timothycrosley/isort
rev: 4.3.21
rev: 5.6.4
hooks:
- id: isort
args: ["--settings-path=tools/python/.isort.cfg"]
Expand All @@ -24,8 +24,8 @@ repos:
types_or: [yaml, json]
exclude: ^.github/

- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v1.2.3
- repo: https://gitlab.com/pycqa/flake8
rev: 3.8.4
hooks:
- id: flake8
args: ["--config=tools/python/.flake8"]
Expand Down
3 changes: 3 additions & 0 deletions airbyte-cdk/python/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
# Changelog

## 0.1.14
If the input config file does not comply with spec schema, raise an exception instead of `system.exit`.

## 0.1.13
Fix defect with user defined backoff time retry attempts, number of retries logic fixed

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,6 @@
import json
import os
import pkgutil
import sys
from typing import Any, Dict, Mapping

import pkg_resources
Expand Down Expand Up @@ -133,8 +132,7 @@ def get_schema(self, name: str) -> dict:
def check_config_against_spec_or_exit(config: Mapping[str, Any], spec: ConnectorSpecification, logger: AirbyteLogger):
"""
Check config object against spec. In case of spec is invalid, throws
SystemExit exception causing application to make system exit call with
errorcode 1
an exception with validation error description.
:param config - config loaded from file specified over command line
:param spec - spec object generated by connector
:param logger - Airbyte logger for reporting validation error
Expand All @@ -143,5 +141,4 @@ def check_config_against_spec_or_exit(config: Mapping[str, Any], spec: Connector
try:
validate(instance=config, schema=spec_schema)
except ValidationError as validation_error:
logger.error("Config validation error: " + validation_error.message)
sys.exit(1)
raise Exception("Config validation error: " + validation_error.message)
2 changes: 1 addition & 1 deletion airbyte-cdk/python/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@

setup(
name="airbyte-cdk",
version="0.1.13",
version="0.1.14",
description="A framework for writing Airbyte Connectors.",
long_description=README,
long_description_content_type="text/markdown",
Expand Down
4 changes: 2 additions & 2 deletions airbyte-cdk/python/unit_tests/test_entrypoint.py
Original file line number Diff line number Diff line change
Expand Up @@ -156,9 +156,9 @@ def test_config_validate(entrypoint: AirbyteEntrypoint, mocker, config_mock, sch
messages = list(entrypoint.run(parsed_args))
assert [_wrap_message(check_value)] == messages
else:
with pytest.raises(SystemExit) as ex_info:
with pytest.raises(Exception) as ex_info:
list(entrypoint.run(parsed_args))
assert ex_info.value.code == 1
assert "Config validation error:" in str(ex_info.value)


def test_run_check(entrypoint: AirbyteEntrypoint, mocker, spec_mock, config_mock):
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ def test_check(self, connector_config, inputs: ConnectionTestConfig, docker_runn
docker_runner.call_check(config=connector_config)

assert err.value.exit_status != 0, "Connector should exit with error code"
assert "Traceback" in err.value.stderr.decode("utf-8"), "Connector should print exception"
assert "Traceback" in err.value.stderr, "Connector should print exception"


@pytest.mark.default_timeout(30)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ tests:
- config_path: "secrets/config.json"
status: "succeed"
- config_path: "integration_tests/invalid_config.json"
status: "failed"
status: "exception"
discovery:
- config_path: "secrets/config.json"
basic_read:
Expand Down