-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
🎉 New source: Pagar.me [python cdk] #18622
Conversation
Hello @sarafonseca-123 Saj from Airbyte here 👋 . We received more than 25 new contributions along the weekend. One of which is yours 🎉 thank you so much! Our team is limited and the review process may take longer than expected. As described in the Airbyte's Hacktoberfest your contribution was submitted before November 2nd and it is eligible to win the prize. The review process will validate other requirements. I ask to you patience until someone from the team review it. Because we reviewed some contributions for Hacktoberfest so far we've seen some common errors/patterns you can double check in advance:
If possible send to me a DM in Slack with the tests credentials, this process will make easier to us run integration tests and publish your connector. If you only has production keys, make sure to create a bootstrap.md explaining how to get the keys. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@sarafonseca-123 some comments, do you have a test account? Also check Saj's comment.
airbyte-integrations/connectors/source-pagarme/source_pagarme/streams.py
Outdated
Show resolved
Hide resolved
Hey @sajarin |
My company is still analyzing if it's ok to share the test key, so I won't be able to provide it for now. But I addressed all comments, and here are the test results after the code change 😄 : Unit: Acceptance: IntegrationBuilding all of Airbyte.
source_pagarme/init.py 2 0 100%
|
### Prerequisites | ||
**To iterate on this connector, make sure to complete this prerequisites section.** | ||
|
||
#### Minimum Python version required `= 3.9.0` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should this version required be >=
?
@@ -0,0 +1,25 @@ | |||
{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could this be changed to a spec.yaml
file instead of a JSON file?
|
||
class BankAccounts(PagarmeStream): | ||
def path(self, **kwargs) -> str: | ||
return "bank_Accounts" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should this be bank_accounts
instead?
}, | ||
"id": { | ||
"type": [ | ||
"null", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Primary key fields shouldn't be null
(this applies for all entities)
|
||
class PagarmeStream(HttpStream, ABC): | ||
page = 1 | ||
page_size_limit = 1000 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What do you think about adding a new config property to configure this page_size_limit
?
page = 1 | ||
page_size_limit = 1000 | ||
primary_key = "id" | ||
url_base = "https://api.pagar.me/1/" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Out of curiosity, why using version 1
which looks a bit old?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Left some comments.
Btw, did you run ./gradlew format
?
|
||
## Authentication | ||
|
||
Authentication is done through an `api_key`, that can be found in the Dashboard. For more informations, consult the [documentation](https://docs.pagar.me/v1/reference/principios-basicos#autentica%C3%A7%C3%A3o). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
typo in informations
@sarafonseca-123 are you still working on this> |
@marcosmarxm I got an access key for the latest Pagar.me API version, now I will update this code to use the V5 instead of V1 |
Because this contribution didn't receive recent updates it won't be consider valid to hacktoberfest event. You can continue the work to get this merge in your own pace. Thank so much for your contribution! |
Hello 👋, first thank you for this amazing contribution. We really appreciate the effort you've made to improve the project. If you have any questions feel free to send me a message in Slack! |
@sarafonseca-123 are you going to continue the work here? |
Close due inactivity. |
I'm also interested in being able to sync Pagar data with airbyte . Is this connector not being developed as we speak ? |
@guillaume-chech, the connector was made using an old version of the API. If you have a key for the newest version I advise you to begin from scratch, since the entities change a lot from v5 to v1. I can gladly help you to conclude this connector if you want to! |
What
Add a new source, Pagar.me.
How
Using Airbyte's Python CDK.
Pre-merge Checklist
Expand the relevant checklist and delete the others.
Community member or Airbyter
airbyte_secret
./gradlew :airbyte-integrations:connectors:<name>:integrationTest
.README.md
bootstrap.md
. See description and examplesdocs/integrations/<source or destination>/<name>.md
including changelog. See changelog exampledocs/integrations/README.md
airbyte-integrations/builds.md
Airbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>
command is passing/publish
command described hereTests
Unit
Integration
Building all of Airbyte.
/Users/sarasantos/git-jusbrasil/airbyte/airbyte-integrations/connectors
Type-safe dependency accessors is an incubating feature.
source_pagarme/init.py 2 0 100%
source_pagarme/source.py 18 2 89%
source_pagarme/streams.py 113 23 80%
TOTAL 133 25 81%
checkin catalogs #1 [internal] load build definition from Dockerfile
checkin catalogs #1 sha256:a6649410914dc037b645dee3c03ab5cdd87cc3a725127a494d7af5e98442d51b
checkin catalogs #1 transferring dockerfile: 37B done
checkin catalogs #1 DONE 0.0s
#2 [internal] load .dockerignore
#2 sha256:cf5b3ab111e1f8b3146b5e71a9f3c7dd29722534f6df0d5692e0926bc2534468
#2 transferring context: 34B done
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/library/python:3.9.11-alpine3.15
#3 sha256:615609793cad3a170d91fb8a37323caa348ca31e0a749e824da4c8c1d8da2aa4
#3 DONE 2.1s
#4 [base 1/1] FROM docker.io/library/python:3.9.11-alpine3.15@sha256:45ddd216e6b4efee0617e15d541e9148ffd6898203fcbe86a9f5bf906ce7837f
#4 sha256:4c436ac5d39299d948ef79080102d272b18d64395fbed9cc821104f1e2bd7d93
#4 DONE 0.0s
#7 [internal] load build context
#7 sha256:ac53c14be86f38833ba2a7a1e8c30f2669275b4ab658461771964321fed8107f
#7 transferring context: 18.07kB done
#7 DONE 0.0s
#5 [builder 1/4] WORKDIR /airbyte/integration_code
#5 sha256:736c564b6c0d634b064a8145fffebbf56c57a10eea338e59977586dee56a7251
#5 CACHED
#6 [builder 2/4] RUN apk --no-cache upgrade && pip install --upgrade pip && apk --no-cache add tzdata build-base
#6 sha256:fddd1028d4c855a19a9d12c308b03a65954965cf7ce3303f4d640a7d59aba08d
#6 CACHED
#8 [builder 3/4] COPY setup.py ./
#8 sha256:b2a0e9b65e4c3f1109e2ff0c059a779415eb7662d8c8c1fb10137bb04b6a4db1
#8 DONE 0.0s
#9 [builder 4/4] RUN pip install --prefix=/install .
#9 sha256:11966c612c8376ff87c7cb008be6edbf8b0f034e761eb4853219805e0457519a
#9 1.026 Processing /airbyte/integration_code
#9 1.028 Preparing metadata (setup.py): started
#9 1.471 Preparing metadata (setup.py): finished with status 'done'
#9 1.820 Collecting airbyte-cdk~=0.2
#9 2.136 Downloading airbyte_cdk-0.4.2-py3-none-any.whl (246 kB)
#9 2.293 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 246.6/246.6 kB 1.6 MB/s eta 0:00:00
#9 2.405 Collecting requests
#9 2.440 Downloading requests-2.28.1-py3-none-any.whl (62 kB)
#9 2.462 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.8/62.8 kB 2.9 MB/s eta 0:00:00
#9 2.547 Collecting PyYAML~=5.4
#9 2.601 Downloading PyYAML-5.4.1.tar.gz (175 kB)
#9 2.646 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 175.1/175.1 kB 4.0 MB/s eta 0:00:00
#9 2.873 Installing build dependencies: started
#9 6.855 Installing build dependencies: finished with status 'done'
#9 6.857 Getting requirements to build wheel: started
#9 8.087 Getting requirements to build wheel: finished with status 'done'
#9 8.092 Preparing metadata (pyproject.toml): started
#9 8.525 Preparing metadata (pyproject.toml): finished with status 'done'
#9 8.599 Collecting jsonschema~=3.2.0
#9 8.637 Downloading jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)
#9 8.650 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56.3/56.3 kB 5.4 MB/s eta 0:00:00
#9 8.710 Collecting Jinja2~=3.1.2
#9 8.745 Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
#9 8.805 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 133.1/133.1 kB 2.2 MB/s eta 0:00:00
#9 8.843 Collecting jsonref~=0.2
#9 8.865 Downloading jsonref-0.3.0-py3-none-any.whl (8.8 kB)
#9 9.065 Collecting dataclasses-jsonschema==2.15.1
#9 9.088 Downloading dataclasses_jsonschema-2.15.1-py3-none-any.whl (17 kB)
#9 9.148 Collecting Deprecated~=1.2
#9 9.172 Downloading Deprecated-1.2.13-py2.py3-none-any.whl (9.6 kB)
#9 9.238 Collecting backoff
#9 9.268 Downloading backoff-2.2.1-py3-none-any.whl (15 kB)
#9 9.364 Collecting pendulum
#9 9.389 Downloading pendulum-2.1.2.tar.gz (81 kB)
#9 9.415 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 81.2/81.2 kB 3.1 MB/s eta 0:00:00
#9 9.463 Installing build dependencies: started
#9 10.97 Installing build dependencies: finished with status 'done'
#9 10.98 Getting requirements to build wheel: started
#9 11.10 Getting requirements to build wheel: finished with status 'done'
#9 11.10 Preparing metadata (pyproject.toml): started
#9 11.44 Preparing metadata (pyproject.toml): finished with status 'done'
#9 11.62 Collecting pydantic~=1.9.2
#9 11.66 Downloading pydantic-1.9.2-cp39-cp39-musllinux_1_1_x86_64.whl (12.5 MB)
#9 14.59 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.5/12.5 MB 4.5 MB/s eta 0:00:00
#9 14.85 Collecting dpath~=2.0.1
#9 14.90 Downloading dpath-2.0.6-py3-none-any.whl (15 kB)
#9 15.18 Collecting requests-cache
#9 15.21 Downloading requests_cache-0.9.7-py3-none-any.whl (48 kB)
#9 15.23 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.5/48.5 kB 7.5 MB/s eta 0:00:00
#9 15.32 Collecting python-dateutil
#9 15.35 Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
#9 15.42 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 247.7/247.7 kB 3.7 MB/s eta 0:00:00
#9 15.61 Collecting wrapt<2,>=1.10
#9 15.63 Downloading wrapt-1.14.1-cp39-cp39-musllinux_1_1_x86_64.whl (82 kB)
#9 15.67 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 82.3/82.3 kB 2.0 MB/s eta 0:00:00
#9 15.77 Collecting MarkupSafe>=2.0
#9 15.81 Downloading MarkupSafe-2.1.1-cp39-cp39-musllinux_1_1_x86_64.whl (29 kB)
#9 15.90 Collecting attrs>=17.4.0
#9 15.93 Downloading attrs-22.1.0-py2.py3-none-any.whl (58 kB)
#9 15.95 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.8/58.8 kB 4.3 MB/s eta 0:00:00
#9 16.00 Collecting six>=1.11.0
#9 16.03 Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
#9 16.03 Requirement already satisfied: setuptools in /usr/local/lib/python3.9/site-packages (from jsonschema~=3.2.0->airbyte-cdk~=0.2->source-pagarme==0.0.0) (58.1.0)
#9 16.09 Collecting pyrsistent>=0.14.0
#9 16.12 Downloading pyrsistent-0.18.1.tar.gz (100 kB)
#9 16.14 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.5/100.5 kB 6.4 MB/s eta 0:00:00
#9 16.18 Installing build dependencies: started
#9 18.16 Installing build dependencies: finished with status 'done'
#9 18.16 Getting requirements to build wheel: started
#9 18.47 Getting requirements to build wheel: finished with status 'done'
#9 18.48 Preparing metadata (pyproject.toml): started
#9 18.82 Preparing metadata (pyproject.toml): finished with status 'done'
#9 18.89 Collecting typing-extensions>=3.7.4.3
#9 18.92 Downloading typing_extensions-4.4.0-py3-none-any.whl (26 kB)
#9 19.02 Collecting pytzdata>=2020.1
#9 19.04 Downloading pytzdata-2020.1-py2.py3-none-any.whl (489 kB)
#9 19.15 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 490.0/490.0 kB 4.6 MB/s eta 0:00:00
#9 19.27 Collecting urllib3<1.27,>=1.21.1
#9 19.30 Downloading urllib3-1.26.12-py2.py3-none-any.whl (140 kB)
#9 19.33 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 140.4/140.4 kB 5.3 MB/s eta 0:00:00
#9 19.38 Collecting idna<4,>=2.5
#9 19.41 Downloading idna-3.4-py3-none-any.whl (61 kB)
#9 19.42 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.5/61.5 kB 11.8 MB/s eta 0:00:00
#9 19.47 Collecting certifi>=2017.4.17
#9 19.51 Downloading certifi-2022.9.24-py3-none-any.whl (161 kB)
#9 19.54 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 161.1/161.1 kB 6.5 MB/s eta 0:00:00
#9 19.65 Collecting charset-normalizer<3,>=2
#9 19.68 Downloading charset_normalizer-2.1.1-py3-none-any.whl (39 kB)
#9 19.90 Collecting url-normalize>=1.4
#9 19.93 Downloading url_normalize-1.4.3-py2.py3-none-any.whl (6.8 kB)
#9 19.98 Collecting cattrs>=22.2
#9 20.01 Downloading cattrs-22.2.0-py3-none-any.whl (35 kB)
#9 20.07 Collecting appdirs>=1.4.4
#9 20.10 Downloading appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
#9 20.26 Collecting exceptiongroup
#9 20.29 Downloading exceptiongroup-1.0.0-py3-none-any.whl (12 kB)
#9 20.50 Building wheels for collected packages: source-pagarme, PyYAML, pendulum, pyrsistent
#9 20.50 Building wheel for source-pagarme (setup.py): started
#9 20.92 Building wheel for source-pagarme (setup.py): finished with status 'done'
#9 20.92 Created wheel for source-pagarme: filename=source_pagarme-0.0.0-py3-none-any.whl size=1212 sha256=950b3008e570c25d1ab82ed2347999a97161096672701ab8fcb36bfdd8713130
#9 20.92 Stored in directory: /tmp/pip-ephem-wheel-cache-0xnfpaou/wheels/ba/6e/40/876b2449257c8324ccb203b3be14dad99463cc90e93c37cb2b
#9 20.93 Building wheel for PyYAML (pyproject.toml): started
#9 21.68 Building wheel for PyYAML (pyproject.toml): finished with status 'done'
#9 21.69 Created wheel for PyYAML: filename=PyYAML-5.4.1-cp39-cp39-linux_x86_64.whl size=45657 sha256=a13c6161b837ff6c0e5ae78391f895dd77fb9479b73b572630727b6bd6175e39
#9 21.69 Stored in directory: /root/.cache/pip/wheels/b7/a5/c4/504d913c2a55bb09c607541578ec5f844d1ff33467abe93ba5
#9 21.69 Building wheel for pendulum (pyproject.toml): started
#9 23.05 Building wheel for pendulum (pyproject.toml): finished with status 'done'
#9 23.05 Created wheel for pendulum: filename=pendulum-2.1.2-cp39-cp39-musllinux_1_2_x86_64.whl size=126863 sha256=dd74b5d62e3d521629e7edfc7b475e69fb5b44bdff52e80ef264c52d4e73bd1d
#9 23.05 Stored in directory: /root/.cache/pip/wheels/c9/c7/29/3cf0033406b18958d1a7b107c38be2405ef9c081b64a264fd5
#9 23.06 Building wheel for pyrsistent (pyproject.toml): started
#9 24.46 Building wheel for pyrsistent (pyproject.toml): finished with status 'done'
#9 24.46 Created wheel for pyrsistent: filename=pyrsistent-0.18.1-cp39-cp39-linux_x86_64.whl size=72096 sha256=7d4719e35f7d0ebbc53180f80caa450c74e14a2b12e07c492f116438af28410d
#9 24.46 Stored in directory: /root/.cache/pip/wheels/87/fe/e6/fc8deeb581a41e462eafaf19fee96f51cdc8391e0be1c8088a
#9 24.47 Successfully built source-pagarme PyYAML pendulum pyrsistent
#9 24.81 Installing collected packages: appdirs, wrapt, urllib3, typing-extensions, six, PyYAML, pytzdata, pyrsistent, MarkupSafe, jsonref, idna, exceptiongroup, dpath, charset-normalizer, certifi, backoff, attrs, url-normalize, requests, python-dateutil, pydantic, jsonschema, Jinja2, Deprecated, cattrs, requests-cache, pendulum, dataclasses-jsonschema, airbyte-cdk, source-pagarme
#9 26.88 Successfully installed Deprecated-1.2.13 Jinja2-3.1.2 MarkupSafe-2.1.1 PyYAML-5.4.1 airbyte-cdk-0.4.2 appdirs-1.4.4 attrs-22.1.0 backoff-2.2.1 cattrs-22.2.0 certifi-2022.9.24 charset-normalizer-2.1.1 dataclasses-jsonschema-2.15.1 dpath-2.0.6 exceptiongroup-1.0.0 idna-3.4 jsonref-0.3.0 jsonschema-3.2.0 pendulum-2.1.2 pydantic-1.9.2 pyrsistent-0.18.1 python-dateutil-2.8.2 pytzdata-2020.1 requests-2.28.1 requests-cache-0.9.7 six-1.16.0 source-pagarme-0.0.0 typing-extensions-4.4.0 url-normalize-1.4.3 urllib3-1.26.12 wrapt-1.14.1
#9 26.88 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
#9 26.88
#9 26.88 [notice] A new release of pip available: 22.2.2 -> 22.3
#9 26.88 [notice] To update, run: pip install --upgrade pip
#9 DONE 27.3s
#5 [builder 1/4] WORKDIR /airbyte/integration_code
#5 sha256:736c564b6c0d634b064a8145fffebbf56c57a10eea338e59977586dee56a7251
#5 CACHED
#10 [stage-2 2/7] COPY --from=builder /install /usr/local
#10 sha256:70d8ffdb84a1df4593afccd86ee91533dc29dd0171c464024db338d24efa3c5d
#10 DONE 0.4s
#11 [stage-2 3/7] COPY --from=builder /usr/share/zoneinfo/Etc/UTC /etc/localtime
#11 sha256:c6ba323e0b03710bfe3ec8a9b8b1b3abe1e3003298b45453c72f18cd13949012
#11 DONE 0.0s
#12 [stage-2 4/7] RUN echo "Etc/UTC" > /etc/timezone
#12 sha256:ea9de1658a07d13a818258af8b7f7f8f6e3821fb97d81d9b9c62cedafcb3506e
#12 DONE 0.3s
#13 [stage-2 5/7] RUN apk --no-cache add bash
#13 sha256:1bd0e5c8308f0c7376fe9a2bba2ab34da7989bdb0fab753b1c8f905a4b50dc47
#13 0.249 fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/main/x86_64/APKINDEX.tar.gz
#13 0.655 fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/community/x86_64/APKINDEX.tar.gz
#13 1.163 (1/1) Installing bash (5.1.16-r0)
#13 1.261 Executing bash-5.1.16-r0.post-install
#13 1.265 Executing busybox-1.34.1-r4.trigger
#13 1.273 OK: 16 MiB in 37 packages
#13 DONE 1.4s
#14 [stage-2 6/7] COPY main.py ./
#14 sha256:ab73d8aa7549cbee79d49396771ca52d7d6b669716e989b2795dce732132fa4f
#14 DONE 0.1s
#15 [stage-2 7/7] COPY source_pagarme ./source_pagarme
#15 sha256:247d96ba0dd885a1dafa4e19e446a94796197d56fd270dde7104471aba0c12f3
#15 DONE 0.1s
#16 exporting to image
#16 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00
#16 exporting layers
#16 exporting layers 1.0s done
#16 writing image sha256:3a85ccee0ce7afcae71344c0d4920797e6d4ed39436525776a54b15c73b90ee4 done
#16 naming to docker.io/airbyte/source-pagarme:dev done
#16 DONE 1.0s
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings
Execution optimizations have been disabled for 1 invalid unit(s) of work during this build to ensure correctness.
Please consult deprecation warnings for more details.
BUILD SUCCESSFUL in 1m 13s
42 actionable tasks: 21 executed, 21 up-to-date
Acceptance