Source Okta: add resource-sets (incremental supported) #14700

YiyangLi · 2022-07-14T00:16:40Z

What

stop using the deprecated method get_updated_state, use state property and state setter instead
the payload resource-sets is enveloped, _links.next.href contains the cursor
add smallest_cursor property to handle various types of cursor in steams

How

Describe the solution

🚨 User Impact 🚨

No impact, it's a new API stream

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

collecting ... 
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_config_match_spec[inputs0] ✓                                                                                                                         4% ▌         
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_match_expected[inputs0] ✓                                                                                                                            8% ▉         
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_docker_env[inputs0] ✓                                                                                                                               12% █▍        
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_oneof_usage[inputs0] ✓                                                                                                                              17% █▋        
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_required[inputs0] ✓                                                                                                                                 21% ██▏       
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_optional[inputs0] ✓                                                                                                                                 25% ██▌       
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_has_secret[inputs0] ✓                                                                                                                               29% ██▉       
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_secret_never_in_the_output[inputs0] ✓                                                                                                               33% ███▍      
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_defined_refs_exist_in_json_spec_file[inputs0] ✓                                                                                                     38% ███▊      
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_oauth_flow_parameters[inputs0] ✓                                                                                                                    42% ████▎     
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestConnection.test_check[inputs0] ✓                                                                                                                              46% ████▋     
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestConnection.test_check[inputs1] ✓                                                                                                                              50% █████     
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_discover[inputs0] ✓                                                                                                                            54% █████▌    
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_cursors_exist_in_schema[inputs0] ✓                                                                                                     58% █████▉    
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_refs_exist_in_schema[inputs0] ✓                                                                                                        62% ██████▍   
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_keyword_exist_in_schema[inputs0-allOf] ✓                                                                                               67% ██████▋   
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_keyword_exist_in_schema[inputs0-not] ✓                                                                                                 71% ███████▏  
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_primary_keys_exist_in_schema[inputs0] ✓                                                                                                        75% ███████▌  
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestBasicRead.test_read[inputs0] ✓                                                                                                                                79% ███████▉  {"type": "LOG", "log": {"level": "ERROR", "message": "Docker container was failed, code 1, error:\n{\"type\": \"TRACE\", \"trace\": {\"type\": \"ERROR\", \"emitted_at\": 1657757361857.2422, \"error\": {\"message\": \"Something went wrong in the connector. See the logs for more details.\", \"internal_message\": \"2 validation errors for ConfiguredAirbyteCatalog\\nstreams -> 0 -> sync_mode\\n  value is not a valid enumeration member; permitted: 'full_refresh', 'incremental' (type=type_error.enum; enum_values=[<SyncMode.full_refresh: 'full_refresh'>, <SyncMode.incremental: 'incremental'>])\\nstreams -> 0 -> destination_sync_mode\\n  value is not a valid enumeration member; permitted: 'append', 'overwrite', 'append_dedup' (type=type_error.enum; enum_values=[<DestinationSyncMode.append: 'append'>, <DestinationSyncMode.overwrite: 'overwrite'>, <DestinationSyncMode.append_dedup: 'append_dedup'>])\", \"stack_trace\": \"Traceback (most recent call last):\\n  File \\\"/airbyte/integration_code/main.py\\\", line 13, in <module>\\n    launch(source, sys.argv[1:])\\n  File \\\"/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py\\\", line 116, in launch\\n    for message in source_entrypoint.run(parsed_args):\\n  File \\\"/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py\\\", line 104, in run\\n    config_catalog = self.source.read_catalog(parsed_args.catalog)\\n  File \\\"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/source.py\\\", line 54, in read_catalog\\n    return ConfiguredAirbyteCatalog.parse_obj(self.read_config(catalog_path))\\n  File \\\"/usr/local/lib/python3.9/site-packages/pydantic/main.py\\\", line 521, in parse_obj\\n    return cls(**obj)\\n  File \\\"/usr/local/lib/python3.9/site-packages/pydantic/main.py\\\", line 341, in __init__\\n    raise validation_error\\npydantic.error_wrappers.ValidationError: 2 validation errors for ConfiguredAirbyteCatalog\\nstreams -> 0 -> sync_mode\\n  value is not a valid enumeration member; permitted: 'full_refresh', 'incremental' (type=type_error.enum; enum_values=[<SyncMode.full_refresh: 'full_refresh'>, <SyncMode.incremental: 'incremental'>])\\nstreams -> 0 -> destination_sync_mode\\n  value is not a valid enumeration member; permitted: 'append', 'overwrite', 'append_dedup' (type=type_error.enum; enum_values=[<DestinationSyncMode.append: 'append'>, <DestinationSyncMode.overwrite: 'overwrite'>, <DestinationSyncMode.append_dedup: 'append_dedup'>])\\n\", \"failure_type\": \"system_error\"}}}\n"}}

 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestBasicRead.test_airbyte_trace_message_on_failure[inputs0] ✓                                                                                                    83% ████████▍ 
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_full_refresh.py::TestFullRefresh.test_sequential_reads[inputs0] ✓                                                                                                          88% ████████▊ 
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py::TestIncremental.test_two_sequential_reads[inputs0] ✓                                                                                                       92% █████████▎
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py::TestIncremental.test_read_sequential_slices[inputs0] ✓                                                                                                     96% █████████▋
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py::TestIncremental.test_state_with_abnormally_large_values[inputs0] ✓                                                                                        100% ██████████
{"type": "LOG", "log": {"level": "INFO", "message": "/Users/bartdev/Playground/airbyte/airbyte-integrations/connectors/source-okta - SAT run - 6ad1dbc6c4a896c6fcb6ee69dabc3577e1c095a4 - PASSED"}}

============================================================================================================================== warnings summary ==============================================================================================================================
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py: 17 warnings
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_full_refresh.py: 1 warning
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py: 3 warnings
  /Users/bartdev/Playground/airbyte/airbyte-integrations/connectors/source-okta/.venv/lib/python3.9/site-packages/docker/utils/utils.py:52: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    s1 = StrictVersion(v1)

airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py: 17 warnings
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_full_refresh.py: 1 warning
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py: 3 warnings
  /Users/bartdev/Playground/airbyte/airbyte-integrations/connectors/source-okta/.venv/lib/python3.9/site-packages/docker/utils/utils.py:53: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    s2 = StrictVersion(v2)

-- Docs: https://docs.pytest.org/en/stable/warnings.html

Results (57.35s):
      24 passed

YiyangLi · 2022-07-15T03:08:24Z

@marcosmarxm, can you help review the PR? Thanks.

marcosmarxm · 2022-07-15T17:41:51Z

/test connector=connectors/source-okta

🕑 connectors/source-okta https://github.com/airbytehq/airbyte/actions/runs/2678415263
❌ connectors/source-okta https://github.com/airbytehq/airbyte/actions/runs/2678415263
🐛 https://gradle.com/s/glp72c53rjwiu

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - AssertionError: All ...
======================== 1 failed, 23 passed in 21.56s =========================

marcosmarxm · 2022-07-15T17:42:09Z

I trggered the test but @sajarin will review it.

YiyangLi · 2022-07-19T23:41:32Z

FAILED test_core.py::TestBasicRead::test_read[inputs0] fails because there's no resource-set created in the test account. Can you follow the steps below to create one?

Under the admin portal, go to Security -> Administrators -> Resources tab, to create a new resource-set

In the following screenshot, I create a resource set for all slack app instances. You can pick users or groups. Creating a new resource set but not assigning it to a role doesn't play no effect on your account.

marcosmarxm · 2022-07-25T11:00:31Z

/test connector=connectors/source-okta

🕑 connectors/source-okta https://github.com/airbytehq/airbyte/actions/runs/2731796003
❌ connectors/source-okta https://github.com/airbytehq/airbyte/actions/runs/2731796003
🐛 https://gradle.com/s/6z23r3c22cllm

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - AssertionError: All ...
FAILED test_core.py::TestBasicRead::test_read[inputs1] - AssertionError: All ...
======================== 2 failed, 27 passed in 39.58s =========================

YiyangLi · 2022-07-26T03:30:27Z

/test connector=connectors/source-okta

🕑 connectors/source-okta https://github.com/airbytehq/airbyte/actions/runs/2731796003
❌ connectors/source-okta https://github.com/airbytehq/airbyte/actions/runs/2731796003
🐛 https://gradle.com/s/6z23r3c22cllm

Build Failed

Test summary info:
=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - AssertionError: All ...
FAILED test_core.py::TestBasicRead::test_read[inputs1] - AssertionError: All ...
======================== 2 failed, 27 passed in 39.58s =========================

@marcosmarxm , the tests failed because of missing resource-set in the test account, can you help add one? I included some screenshots in the PR, let me know if you have a question, thanks.

YiyangLi · 2022-07-28T02:51:47Z

/test connector=connectors/source-okta

sajarin · 2022-08-01T14:16:10Z

/test connector=connectors/source-okta

🕑 connectors/source-okta https://github.com/airbytehq/airbyte/actions/runs/2775487630
❌ connectors/source-okta https://github.com/airbytehq/airbyte/actions/runs/2775487630
🐛

YiyangLi · 2022-08-03T00:00:28Z

/test connector=connectors/source-okta

sajarin · 2022-08-03T17:13:34Z

/test connector=connectors/source-okta

🕑 connectors/source-okta https://github.com/airbytehq/airbyte/actions/runs/2791197868
❌ connectors/source-okta https://github.com/airbytehq/airbyte/actions/runs/2791197868
🐛 https://gradle.com/s/a6jl6hf2oh7vq

Build Failed

Test summary info:

Could not find result summary

YiyangLi · 2022-08-03T18:06:03Z

rebased again, not sure why "Could not find result summary"

sajarin · 2022-08-09T17:39:20Z

/test connector=connectors/source-okta

🕑 connectors/source-okta https://github.com/airbytehq/airbyte/actions/runs/2827193707
❌ connectors/source-okta https://github.com/airbytehq/airbyte/actions/runs/2827193707
🐛 https://gradle.com/s/hykjt3hnrgtvk

Build Failed

Test summary info:

	 =========================== short test summary info ============================
	 FAILED unit_tests/test_streams.py::TestStreamGroupMembers::test_group_members
	 FAILED unit_tests/test_streams.py::TestStreamResourceSets::test_resource_sets
	 FAILED unit_tests/test_streams.py::TestStreamResourceSets::test_resource_sets_parse_response
	 FAILED unit_tests/test_streams.py::TestStreamResourceSets::test_resource_sets_next_page_token
	 FAILED unit_tests/test_streams.py::TestStreamResourceSets::test_resource_sets_request_params
	 �[31m================== �[31m�[1m5 failed�[0m, �[32m55 passed�[0m, �[33m134 warnings�[0m�[31m in 0.90s�[0m�[31m ==================�[0m

sajarin · 2022-08-09T17:54:26Z

@YiyangLi I think one of the latest PRs for this connector added a start_date param which is causing some of your tests to no longer pass. If you have time, could you push a fix?

YiyangLi · 2022-08-10T04:17:29Z

/test connector=connectors/source-okta

YiyangLi · 2022-08-10T04:19:41Z

@YiyangLi I think one of the latest PRs for this connector added a start_date param which is causing some of your tests to no longer pass. If you have time, could you push a fix?

@sajarin I fixed the issue, and it passes both unit tests and acceptance tests in my local environment, can you help kick off the tests again? thanks.

marcosmarxm · 2022-08-10T12:16:48Z

/test connector=connectors/source-okta

🕑 connectors/source-okta https://github.com/airbytehq/airbyte/actions/runs/2832534188
❌ connectors/source-okta https://github.com/airbytehq/airbyte/actions/runs/2832534188
🐛 https://gradle.com/s/kwwvprub2mn5a

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestConnection::test_check[inputs3] - docker.errors.Cont...
FAILED test_core.py::TestBasicRead::test_read[inputs0] - AssertionError: All ...
FAILED test_core.py::TestBasicRead::test_read[inputs1] - AssertionError: All ...
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
=================== 4 failed, 32 passed in 220.52s (0:03:40) ===================

YiyangLi · 2022-08-10T17:28:06Z

@sajarin

FAILED test_core.py::TestBasicRead::test_read[inputs0] - AssertionError: All ...
FAILED test_core.py::TestBasicRead::test_read[inputs1] - AssertionError: All ...

I checked the logs, TestBasicRead is failed because there are no resource-sets entities in the test account. Can you help add one? I attached the screenshots above to illustrate the problem. Ping me in Slack if you have a question.

Meanwhile, I will work on the other 2 failures, and will get back to you in a few hours. Thanks.

YiyangLi · 2022-08-10T17:58:31Z

@sajarin Are you able to access the secrets/config.json used to run the acceptance test?

  full_refresh:
    - config_path: "secrets/config.json"
      configured_catalog_path: "integration_tests/configured_catalog.json"

I checked the logs regarding the FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]. The test runs 2 full imports and compares the results. It's a very good test. I attached the difference below. The test runner gets a rate limit warning from the first run, which appends a log when the 2nd full import is run.

I wondered if you can help to adjust the config.json, so that it fetches less when full imports are run. It seems that the startDate is around 2021-03-21. The feature startDate is added in the PR #15050

{
  "domain": "myOrg",
  "start_date": "2022-08-09T00:00:00Z", // <-- can you add a date or modify here? 
  "credentials": {
    "auth_type": "api_token",
    "api_token": "yourToken"
  }
}

The log difference between 2 full imports.

2022-08-10T12:28:24.1519329Z  {
2022-08-10T12:28:24.1519462Z   "actor": {
2022-08-10T12:28:24.1519607Z    "id": "00umj7d1bEH95JjIE5d6",
2022-08-10T12:28:24.1519726Z    "type": "User",
2022-08-10T12:28:24.1519974Z    "alternateId": "[email protected]",
2022-08-10T12:28:24.1520121Z    "displayName": "Shrif Nada",
2022-08-10T12:28:24.1520256Z    "detailEntry": null
2022-08-10T12:28:24.1520514Z   },
2022-08-10T12:28:24.1520640Z   "client": {
2022-08-10T12:28:24.1520774Z    "userAgent": {
2022-08-10T12:28:24.1521014Z     "rawUserAgent": "python-requests/2.28.1",
2022-08-10T12:28:24.1521133Z     "os": "Unknown",
2022-08-10T12:28:24.1521267Z     "browser": "UNKNOWN"
2022-08-10T12:28:24.1521384Z    },
2022-08-10T12:28:24.1521507Z    "zone": "null",
2022-08-10T12:28:24.1521636Z    "device": "Unknown",
2022-08-10T12:28:24.1521752Z    "id": null,
2022-08-10T12:28:24.1521883Z    "ipAddress": "3.143.7.192",
2022-08-10T12:28:24.1522036Z    "geographicalContext": null
2022-08-10T12:28:24.1522158Z   },
2022-08-10T12:28:24.1522284Z   "device": null,
2022-08-10T12:28:24.1522440Z   "authenticationContext": {
2022-08-10T12:28:24.1522600Z    "authenticationProvider": null,
2022-08-10T12:28:24.1522748Z    "credentialProvider": null,
2022-08-10T12:28:24.1522879Z    "credentialType": null,
2022-08-10T12:28:24.1523004Z    "issuer": null,
2022-08-10T12:28:24.1523134Z    "interface": null,
2022-08-10T12:28:24.1523279Z    "authenticationStep": 0,
2022-08-10T12:28:24.1523468Z    "externalSessionId": "trsX0Y095K_SkabzZ3hx_h_Eg"
2022-08-10T12:28:24.1523585Z   },
2022-08-10T12:28:24.1523743Z   "displayMessage": "Rate limit warning",
2022-08-10T12:28:24.1523927Z   "eventType": "system.org.rate_limit.warning",
2022-08-10T12:28:24.1524054Z   "outcome": {
2022-08-10T12:28:24.1524185Z    "result": "SUCCESS",
2022-08-10T12:28:24.1524309Z    "reason": null
2022-08-10T12:28:24.1524424Z   },
2022-08-10T12:28:24.1524632Z   "published": "2022-08-10T12:25:09Z",
2022-08-10T12:28:24.1524888Z   "securityContext": {
2022-08-10T12:28:24.1525014Z    "asNumber": null,
2022-08-10T12:28:24.1525139Z    "asOrg": null,
2022-08-10T12:28:24.1525264Z    "isp": null,
2022-08-10T12:28:24.1525388Z    "domain": null,
2022-08-10T12:28:24.1525514Z    "isProxy": null
2022-08-10T12:28:24.1525627Z   },
2022-08-10T12:28:24.1525751Z   "severity": "WARN",
2022-08-10T12:28:24.1525888Z   "debugContext": {
2022-08-10T12:28:24.1526018Z    "debugData": {
2022-08-10T12:28:24.1526208Z     "requestId": "YvOjpRl9PpbIUf9vvql73AAADE8",
2022-08-10T12:28:24.1526353Z     "warningPercent": "60",
2022-08-10T12:28:24.1526618Z     "dtHash": "ee75b9f0b0b90bc30b3a57aac75ce62134196d23b9a85aaecc14a077387c0cfd",
2022-08-10T12:28:24.1526783Z     "rateLimitSecondsToReset": "45",
2022-08-10T12:28:24.1526912Z     "threshold": "20",
2022-08-10T12:28:24.1527034Z     "timeSpan": "1",
2022-08-10T12:28:24.1527190Z     "rateLimitScopeType": "ORG",
2022-08-10T12:28:24.1527337Z     "requestUri": "/api/v1/logs",
2022-08-10T12:28:24.1527497Z     "userId": "00umj7d1bEH95JjIE5d6",
2022-08-10T12:28:24.1527848Z     "url": "/api/v1/logs?limit=200&since=2021-03-21T20%3A49%3A13%2B00%3A00&after=1658513870546_1",
2022-08-10T12:28:24.1527983Z     "timeUnit": "MINUTES"
2022-08-10T12:28:24.1528091Z    {
2022-08-10T12:28:24.1528207Z   },
2022-08-10T12:28:24.1528429Z   "legacyEventType": "core.framework.ratelimit.warning",
2022-08-10T12:28:24.1528565Z   "transaction": {
2022-08-10T12:28:24.1528692Z    "type": "WEB",
2022-08-10T12:28:24.1528862Z    "id": "YvOjpRl9PpbIUf9vvql73AAADE8",
2022-08-10T12:28:24.1528989Z    "detail": {
2022-08-10T12:28:24.1529171Z     "requestApiTokenId": "00T1f11psiA9Zb3hT5d6"
2022-08-10T12:28:24.1529278Z    {
2022-08-10T12:28:24.1529393Z   },
2022-08-10T12:28:24.1529642Z   "uuid": "79364c9f-18a7-11ed-824c-f39c5aba5bdb",
2022-08-10T12:28:24.1529766Z   "version": "0",
2022-08-10T12:28:24.1529892Z   "request": {
2022-08-10T12:28:24.1530015Z    "ipChain": [
2022-08-10T12:28:24.1530130Z     {
2022-08-10T12:28:24.1530258Z      "ip": "3.143.7.192",
2022-08-10T12:28:24.1530408Z      "geographicalContext": null,
2022-08-10T12:28:24.1530539Z      "version": "V4",
2022-08-10T12:28:24.1530661Z      "source": null
2022-08-10T12:28:24.1530776Z     {
2022-08-10T12:28:24.1530885Z    ]
2022-08-10T12:28:24.1530996Z   },
2022-08-10T12:28:24.1531111Z   "target": [
2022-08-10T12:28:24.1531227Z    {
2022-08-10T12:28:24.1531359Z     "id": "/api/v1/logs",
2022-08-10T12:28:24.1531503Z     "type": "URL Pattern",
2022-08-10T12:28:24.1531639Z     "alternateId": null,
2022-08-10T12:28:24.1531774Z     "displayName": null,
2022-08-10T12:28:24.1531909Z     "detailEntry": null
2022-08-10T12:28:24.1532017Z    },
2022-08-10T12:28:24.1532134Z    {
2022-08-10T12:28:24.1532380Z     "id": "b192d91c-b242-36da-9332-d97a5579f865",
2022-08-10T12:28:24.1532517Z     "type": "Bucket Uuid",
2022-08-10T12:28:24.1532653Z     "alternateId": null,
2022-08-10T12:28:24.1532789Z     "displayName": null,
2022-08-10T12:28:24.1532912Z     "detailEntry": null
2022-08-10T12:28:24.1533030Z    }
2022-08-10T12:28:24.1533141Z   ]
2022-08-10T12:28:24.1533255Z  }
2022-08-10T12:28:24.1533368Z ]

YiyangLi · 2022-08-11T03:25:54Z

/test connector=connectors/source-okta

sajarin · 2022-08-11T16:19:33Z

/test connector=connectors/source-okta

🕑 connectors/source-okta https://github.com/airbytehq/airbyte/actions/runs/2841266067
❌ connectors/source-okta https://github.com/airbytehq/airbyte/actions/runs/2841266067
🐛 https://gradle.com/s/t7nybhvib5tpm

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - AssertionError: All ...
=================== 1 failed, 35 passed in 95.99s (0:01:35) ====================

sajarin · 2022-08-11T16:56:16Z

/test connector=connectors/source-okta

🕑 connectors/source-okta https://github.com/airbytehq/airbyte/actions/runs/2841463823
❌ connectors/source-okta https://github.com/airbytehq/airbyte/actions/runs/2841463823
🐛 https://gradle.com/s/6wx4b76tyzbu2

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - AssertionError: All ...
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
=================== 2 failed, 34 passed in 212.77s (0:03:32) ===================

sajarin · 2022-08-11T20:14:45Z

/test connector=connectors/source-okta

marcosmarxm · 2022-08-11T21:18:49Z

/test connector=connectors/source-okta

🕑 connectors/source-okta https://github.com/airbytehq/airbyte/actions/runs/2842884349
✅ connectors/source-okta https://github.com/airbytehq/airbyte/actions/runs/2842884349
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        82      6    93%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/tests/test_incremental.py       121     25    79%
source_acceptance_test/utils/common.py                  77     17    78%
source_acceptance_test/tests/test_core.py              355    107    70%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
------------------------------------------------------------------------
TOTAL                                                 1023    247    76%
Name                           Stmts   Miss  Cover
--------------------------------------------------
source_okta/utils.py              44      0   100%
source_okta/authenticator.py      14      0   100%
source_okta/__init__.py            2      0   100%
source_okta/source.py            194      7    96%
--------------------------------------------------
TOTAL                            254      7    97%

Build Passed

Test summary info:

All Passed

YiyangLi · 2022-08-12T15:29:15Z

is it ready to merge?

sajarin · 2022-08-12T16:26:10Z

/publish connector=connectors/source-okta

🕑 Publishing the following connectors:
connectors/source-okta
https://github.com/airbytehq/airbyte/actions/runs/2848036528

Connector	Did it publish?	Were definitions generated?
connectors/source-okta	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

YiyangLi · 2022-08-12T20:22:24Z

take a bow! thank you @sajarin and @marcosmarxm

github-actions bot added the area/connectors Connector related issues label Jul 14, 2022

octavia-squidington-iii added community connectors/source/okta labels Jul 14, 2022

YiyangLi force-pushed the YL_add-resource-sets branch from 5745609 to d5b8c59 Compare July 14, 2022 00:19

YiyangLi marked this pull request as ready for review July 14, 2022 00:19

github-actions bot added the area/documentation Improvements or additions to documentation label Jul 14, 2022

octavia-squidington-iii added the bounty label Jul 14, 2022

YiyangLi force-pushed the YL_add-resource-sets branch from d5b8c59 to 0743078 Compare July 15, 2022 03:08

alafanechere assigned sajarin Jul 15, 2022

alafanechere changed the title ~~Source Okta: add resource-sets (inremental supported)~~ Source Okta: add resource-sets (incremental supported) Jul 15, 2022

YiyangLi force-pushed the YL_add-resource-sets branch from 0743078 to 7e2fa64 Compare July 19, 2022 23:36

YiyangLi force-pushed the YL_add-resource-sets branch from 7e2fa64 to fff86c9 Compare July 28, 2022 02:51

YiyangLi force-pushed the YL_add-resource-sets branch 3 times, most recently from ba9f0d2 to a6f8a14 Compare July 29, 2022 00:43

sajarin self-requested a review August 1, 2022 19:33

YiyangLi force-pushed the YL_add-resource-sets branch 2 times, most recently from b229479 to 51ca4b7 Compare August 2, 2022 23:57

YiyangLi force-pushed the YL_add-resource-sets branch from 51ca4b7 to e096980 Compare August 3, 2022 18:05

fix: change assert statement with flake formatting

f636cb8

YiyangLi added 4 commits August 9, 2022 15:32

fix unit tests

3008092

clean up get_updated_state overriding

3349ad9

rename and correct sample config

2853521

correct sample valid and invalid config

bd502d9

fake the token more

c4c9f8d

YiyangLi added 2 commits August 10, 2022 11:01

fix the invalid_config.json

98b6274

change the order of stream, hopefully logs is run firstly

bcb6c8d

fix: remove log stream from configured catalog

13cf1da

sajarin approved these changes Aug 12, 2022

View reviewed changes

fix: bump connector version on okta.md and Dockerfile

a88c4cb

auto-bump connector version [ci skip]

4fb92cf

sajarin merged commit d82632f into airbytehq:master Aug 12, 2022

octavia-squidington-iii mentioned this pull request Aug 19, 2022

Bump Airbyte version from 0.40.0-alpha to 0.40.1-alpha #15775

Closed

evantahler mentioned this pull request Aug 22, 2022

Bump Airbyte version from 0.40.0-alpha to 0.40.1 #15857

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source Okta: add resource-sets (incremental supported) #14700

Source Okta: add resource-sets (incremental supported) #14700

YiyangLi commented Jul 14, 2022 •

edited

Loading

YiyangLi commented Jul 15, 2022

marcosmarxm commented Jul 15, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Jul 15, 2022

YiyangLi commented Jul 19, 2022

marcosmarxm commented Jul 25, 2022 •

edited by github-actions bot

Loading

YiyangLi commented Jul 26, 2022

Build Failed

YiyangLi commented Jul 28, 2022

sajarin commented Aug 1, 2022 •

edited by github-actions bot

Loading

YiyangLi commented Aug 3, 2022

sajarin commented Aug 3, 2022 •

edited by github-actions bot

Loading

YiyangLi commented Aug 3, 2022

sajarin commented Aug 9, 2022 •

edited by github-actions bot

Loading

sajarin commented Aug 9, 2022

YiyangLi commented Aug 10, 2022

YiyangLi commented Aug 10, 2022

marcosmarxm commented Aug 10, 2022 •

edited by github-actions bot

Loading

YiyangLi commented Aug 10, 2022

YiyangLi commented Aug 10, 2022 •

edited

Loading

YiyangLi commented Aug 11, 2022

sajarin commented Aug 11, 2022 •

edited by github-actions bot

Loading

sajarin commented Aug 11, 2022 •

edited by github-actions bot

Loading

sajarin commented Aug 11, 2022

marcosmarxm commented Aug 11, 2022 •

edited by github-actions bot

Loading

YiyangLi commented Aug 12, 2022

sajarin commented Aug 12, 2022 •

edited by github-actions bot

Loading

YiyangLi commented Aug 12, 2022

Source Okta: add resource-sets (incremental supported) #14700

Source Okta: add resource-sets (incremental supported) #14700

Conversation

YiyangLi commented Jul 14, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Tests

YiyangLi commented Jul 15, 2022

marcosmarxm commented Jul 15, 2022 • edited by github-actions bot Loading

Build Failed

marcosmarxm commented Jul 15, 2022

YiyangLi commented Jul 19, 2022

marcosmarxm commented Jul 25, 2022 • edited by github-actions bot Loading

Build Failed

YiyangLi commented Jul 26, 2022

Build Failed

YiyangLi commented Jul 28, 2022

sajarin commented Aug 1, 2022 • edited by github-actions bot Loading

YiyangLi commented Aug 3, 2022

sajarin commented Aug 3, 2022 • edited by github-actions bot Loading

Build Failed

YiyangLi commented Aug 3, 2022

sajarin commented Aug 9, 2022 • edited by github-actions bot Loading

Build Failed

sajarin commented Aug 9, 2022

YiyangLi commented Aug 10, 2022

YiyangLi commented Aug 10, 2022

marcosmarxm commented Aug 10, 2022 • edited by github-actions bot Loading

Build Failed

YiyangLi commented Aug 10, 2022

YiyangLi commented Aug 10, 2022 • edited Loading

YiyangLi commented Aug 11, 2022

sajarin commented Aug 11, 2022 • edited by github-actions bot Loading

Build Failed

sajarin commented Aug 11, 2022 • edited by github-actions bot Loading

Build Failed

sajarin commented Aug 11, 2022

marcosmarxm commented Aug 11, 2022 • edited by github-actions bot Loading

Build Passed

YiyangLi commented Aug 12, 2022

sajarin commented Aug 12, 2022 • edited by github-actions bot Loading

YiyangLi commented Aug 12, 2022

YiyangLi commented Jul 14, 2022 •

edited

Loading

marcosmarxm commented Jul 15, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Jul 25, 2022 •

edited by github-actions bot

Loading

sajarin commented Aug 1, 2022 •

edited by github-actions bot

Loading

sajarin commented Aug 3, 2022 •

edited by github-actions bot

Loading

sajarin commented Aug 9, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Aug 10, 2022 •

edited by github-actions bot

Loading

YiyangLi commented Aug 10, 2022 •

edited

Loading

sajarin commented Aug 11, 2022 •

edited by github-actions bot

Loading

sajarin commented Aug 11, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Aug 11, 2022 •

edited by github-actions bot

Loading

sajarin commented Aug 12, 2022 •

edited by github-actions bot

Loading