source-okta: add custom roles stream #14610

YiyangLi · 2022-07-12T04:19:02Z

What

add custom roles, which supports full refresh only

How

In okta, each org is allowed to create at most 100 custom roles. The custom roles API returns at most 100 roles. Though the doc says the response is paginated, but the pagination strategy is not specified. Created 100 roles in my test org, and all roles are returned in one request.

Also, unlike users, groups and logs, the payload of roles is enveloped. The response is an object with a key of roles, and the value is an array of custom role.

🚨 User Impact 🚨

no impact, it's a new API stream

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

Tests

Unit

** airbyte-integrations/connectors/source-okta/unit_tests/unit_test.py::TestPagination.test_pagination[None-XYZ-expected_npt0-Expected to receive a new page token if next header is set and self is not set] ✓                                                   25% ██▌       
 airbyte-integrations/connectors/source-okta/unit_tests/unit_test.py::TestPagination.test_pagination[ABC-XYZ-expected_npt1-Expected to receive a new page token if next and self headers have different values] ✓                                               50% █████     
 airbyte-integrations/connectors/source-okta/unit_tests/unit_test.py::TestPagination.test_pagination[ABC-ABC-None-Expected no new page token if next and self headers are the same] ✓                                                                           75% ███████▌  
 airbyte-integrations/connectors/source-okta/unit_tests/unit_test.py::TestPagination.test_pagination[ABC-None-None-Expected no new page token if next header is not set] ✓                                                                                     100% ██████████
============================================================================================================================== warnings summary ==============================================================================================================================
airbyte-integrations/connectors/source-okta/unit_tests/unit_test.py::TestPagination::test_pagination[None-XYZ-expected_npt0-Expected to receive a new page token if next header is set and self is not set]
airbyte-integrations/connectors/source-okta/unit_tests/unit_test.py::TestPagination::test_pagination[ABC-XYZ-expected_npt1-Expected to receive a new page token if next and self headers have different values]
airbyte-integrations/connectors/source-okta/unit_tests/unit_test.py::TestPagination::test_pagination[ABC-ABC-None-Expected no new page token if next and self headers are the same]
airbyte-integrations/connectors/source-okta/unit_tests/unit_test.py::TestPagination::test_pagination[ABC-None-None-Expected no new page token if next header is not set]
  /Users/bartdev/Playground/airbyte/airbyte-integrations/connectors/source-okta/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py:41: DeprecationWarning: Call to deprecated class NoAuth. (Set `authenticator=None` instead) -- Deprecated since version 0.1.20.
    self._authenticator: HttpAuthenticator = NoAuth()

airbyte-integrations/connectors/source-okta/unit_tests/unit_test.py::TestPagination::test_pagination[None-XYZ-expected_npt0-Expected to receive a new page token if next header is set and self is not set]
airbyte-integrations/connectors/source-okta/unit_tests/unit_test.py::TestPagination::test_pagination[ABC-XYZ-expected_npt1-Expected to receive a new page token if next and self headers have different values]
airbyte-integrations/connectors/source-okta/unit_tests/unit_test.py::TestPagination::test_pagination[ABC-ABC-None-Expected no new page token if next and self headers are the same]
airbyte-integrations/connectors/source-okta/unit_tests/unit_test.py::TestPagination::test_pagination[ABC-None-None-Expected no new page token if next header is not set]
  /Users/bartdev/Playground/airbyte/airbyte-integrations/connectors/source-okta/.venv/lib/python3.9/site-packages/deprecated/classic.py:173: DeprecationWarning: Call to deprecated class HttpAuthenticator. (Use requests.auth.AuthBase instead) -- Deprecated since version 0.1.20.
    return old_new1(cls, *args, **kwargs)

-- Docs: https://docs.pytest.org/en/stable/warnings.html

Results (0.21s):
       4 passed

Integration

Test session starts (platform: darwin, Python 3.9.13, pytest 6.2.5, pytest-sugar 0.9.4)
cachedir: .pytest_cache
rootdir: /Users/bartdev/Playground/airbyte, configfile: pytest.ini
plugins: sugar-0.9.4, timeout-1.4.2
collecting ... 

Results (0.01s):

Acceptance

============================================================================================================================== warnings summary ==============================================================================================================================
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py: 17 warnings
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_full_refresh.py: 1 warning
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py: 3 warnings
  /Users/bartdev/Playground/airbyte/airbyte-integrations/connectors/source-okta/.venv/lib/python3.9/site-packages/docker/utils/utils.py:52: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    s1 = StrictVersion(v1)

airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py: 17 warnings
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_full_refresh.py: 1 warning
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py: 3 warnings
  /Users/bartdev/Playground/airbyte/airbyte-integrations/connectors/source-okta/.venv/lib/python3.9/site-packages/docker/utils/utils.py:53: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    s2 = StrictVersion(v2)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
========================================================================================================================== short test summary info ===========================================================================================================================
FAILED ../../bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestBasicRead::test_read[inputs0] - Failed: Please check your json_schema in selected streams ('logs',).
FAILED ../../bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py::TestIncremental::test_state_with_abnormally_large_values[inputs0] - AssertionError: The sync should produce no records when run with the state with abnormally large values

Results (85.81s):
      22 passed
       2 failed
         - airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py:381 TestBasicRead.test_read[inputs0]
         - airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py:207 TestIncremental.test_state_with_abnormally_large_values[inputs0]

alafanechere · 2022-07-12T15:48:53Z

Thank for your contribution @YiyangLi , I formatted the code and will now run the acceptance tests.

alafanechere · 2022-07-12T15:49:09Z

/test connector=connectors/source-okta

🕑 connectors/source-okta https://github.com/airbytehq/airbyte/actions/runs/2657748710
❌ connectors/source-okta https://github.com/airbytehq/airbyte/actions/runs/2657748710
🐛 https://gradle.com/s/cxmjr2mevv5ja

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - Failed: Please check...
FAILED test_incremental.py::TestIncremental::test_two_sequential_reads[inputs0]
FAILED test_incremental.py::TestIncremental::test_read_sequential_slices[inputs0]
FAILED test_incremental.py::TestIncremental::test_state_with_abnormally_large_values[inputs0]
======================== 4 failed, 20 passed in 24.01s =========================

🕑 connectors/source-okta https://github.com/airbytehq/airbyte/actions/runs/2657748710
❌ connectors/source-okta https://github.com/airbytehq/airbyte/actions/runs/2657748710
🐛 https://gradle.com/s/7gbganm3h5px6

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - Failed: Please check...
FAILED test_incremental.py::TestIncremental::test_state_with_abnormally_large_values[inputs0]
======================== 2 failed, 22 passed in 28.00s =========================

- add custom roles, which supports full refresh only

YiyangLi · 2022-07-13T02:54:16Z

@alafanechere can you try to run the test again?

If you don't have a custom role in your test okta org, the acceptance test will throw an error on the basic test that expects at least one record on Custom Role, the new stream I create in this PR.

The following are the steps to create one.

Go to Admin after login
choose Navigation drawer on the left --> Security --> Administrators --> Roles tab
Create a new Role

it's safe to create a role, as long as it's not assigned to a user or a group, there is no effect.

alafanechere · 2022-07-13T07:54:45Z

/test connector=connectors/source-okta

🕑 connectors/source-okta https://github.com/airbytehq/airbyte/actions/runs/2661882540
✅ connectors/source-okta https://github.com/airbytehq/airbyte/actions/runs/2661882540
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        77      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/tests/test_incremental.py       121     25    79%
source_acceptance_test/utils/common.py                  80     17    79%
source_acceptance_test/tests/test_core.py              294    106    64%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
------------------------------------------------------------------------
TOTAL                                                  960    246    74%
Name                      Stmts   Miss  Cover
---------------------------------------------
source_okta/__init__.py       2      0   100%
source_okta/source.py       127     59    54%
---------------------------------------------
TOTAL                       129     59    54%

Build Passed

Test summary info:

All Passed

alafanechere · 2022-07-13T07:55:35Z

The following are the steps to create one.
Thank you! I created a custom role on our test account, running the test again now.

alafanechere · 2022-07-13T08:24:30Z

/publish connector=connectors/source-okta

🕑 Publishing the following connectors:
connectors/source-okta
https://github.com/airbytehq/airbyte/actions/runs/2662049745

Connector	Did it publish?	Were definitions generated?
connectors/source-okta	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

🕑 Publishing the following connectors:
connectors/source-okta
https://github.com/airbytehq/airbyte/actions/runs/2662049745

Connector	Did it publish?	Were definitions generated?
connectors/source-okta	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

github-actions bot added the area/connectors Connector related issues label Jul 12, 2022

octavia-squidington-iii added community connectors/source/okta labels Jul 12, 2022

YiyangLi force-pushed the YL_add-custom-roles branch from db83187 to 06ba96c Compare July 12, 2022 04:22

YiyangLi marked this pull request as ready for review July 12, 2022 04:22

github-actions bot added the area/documentation Improvements or additions to documentation label Jul 12, 2022

octavia-squidington-iii added the bounty label Jul 12, 2022

alafanechere added the bounty-S Maintainer program: claimable small bounty PR label Jul 12, 2022

alafanechere self-assigned this Jul 12, 2022

alafanechere changed the title ~~[source okta] add custom roles stream~~ source-okta: add custom roles stream Jul 12, 2022

YiyangLi added 2 commits July 12, 2022 19:40

[source okta] add custom roles stream

f7cccba

- add custom roles, which supports full refresh only

rebased, formatted and remove empty_streams

b0241ed

YiyangLi force-pushed the YL_add-custom-roles branch from 14f2e47 to b0241ed Compare July 13, 2022 02:48

format

8f6eb48

alafanechere approved these changes Jul 13, 2022 •

edited

Loading

View reviewed changes

alafanechere and others added 2 commits July 13, 2022 10:34

bump version

499d6bb

auto-bump connector version

d16b9dc

alafanechere merged commit 48f8f2f into airbytehq:master Jul 13, 2022

octavia-squidington-iii mentioned this pull request Jul 13, 2022

Bump Airbyte version from 0.39.35-alpha to 0.39.36-alpha #14690

Merged

YiyangLi mentioned this pull request Jul 15, 2022

Source Okta: add permission stream under a custom role #14739

Merged

37 tasks

bleonard added the reward-50 label Oct 31, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

source-okta: add custom roles stream #14610

source-okta: add custom roles stream #14610

YiyangLi commented Jul 12, 2022 •

edited

Loading

alafanechere commented Jul 12, 2022

alafanechere commented Jul 12, 2022 •

edited by github-actions bot

Loading

YiyangLi commented Jul 13, 2022

alafanechere commented Jul 13, 2022 •

edited by github-actions bot

Loading

alafanechere commented Jul 13, 2022

alafanechere commented Jul 13, 2022 •

edited by github-actions bot

Loading

source-okta: add custom roles stream #14610

source-okta: add custom roles stream #14610

Conversation

YiyangLi commented Jul 12, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Tests

alafanechere commented Jul 12, 2022

alafanechere commented Jul 12, 2022 • edited by github-actions bot Loading

Build Failed

Build Failed

YiyangLi commented Jul 13, 2022

alafanechere commented Jul 13, 2022 • edited by github-actions bot Loading

Build Passed

alafanechere commented Jul 13, 2022

alafanechere commented Jul 13, 2022 • edited by github-actions bot Loading

YiyangLi commented Jul 12, 2022 •

edited

Loading

alafanechere commented Jul 12, 2022 •

edited by github-actions bot

Loading

alafanechere commented Jul 13, 2022 •

edited by github-actions bot

Loading

alafanechere commented Jul 13, 2022 •

edited by github-actions bot

Loading