-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Native Github Connector failed to sync #4612
Comments
Need to implement rate limit handling. An example may be found here. |
Any update on this? Short term work-around has been to disable the If there is also a problem with rate-limiting, it would be good for that to be confirmed. |
@Zirochkaa can you update the status? The PR is almost finished, as far I saw. |
Hello :)
All of the above issues are fixed now. Right now I'm working on optimizing |
Hi @garden-of-delete , |
@keu Thanks so much! Will get into testing and report back tomorrow. |
Enviroment
Current Behavior
Description: I’ve been experimenting with the Github Native connector, v0.1.0 and v0.1.1. I'm getting empty tables for some of the streams, but the results are inconsistent. One possibility is that when the API rate limit gets hit, the connector just moves on and concludes syncing that stream. I base this hypothesis on these observations:
Variable outcome when syncing even a single large stream (apache/superset issues). Sometimes you get a populated table, sometimes nothing, sometimes a populated table, but with a different (but similar) number of records.
Empty tables seem more likely right after a previous sync fails. I can demonstrate that syncing just the issues stream produces an empty table 100% of the time after running a full sync (which also produces some empty tables). Wait an hour, sync just issues again without changing any configuration -> populated issues table.
Syncing all streams for even a small repo fails right after syncing a large repo.
I have attached the log from an example run. Connector v0.1.1, all streams, apache/superset repo. This run prouced a mix of populated and empty tables. For example, events had records, but issues is empty. (edited)
Expected Behavior
Tell us what should happen.
Logs
If applicable, please upload the logs from the failing operation.
For sync jobs, you can download the full logs from the UI by going to the sync attempt page and
clicking the download logs button at the top right of the logs display window.
LOG
Steps to Reproduce
Are you willing to submit a PR?
Not at the moment
The text was updated successfully, but these errors were encountered: