Public repos in a github org dont return all streams

[Moofasax]

Expected Behavior

Disable the collaborators, projects, events, and teams streams for a public repo i dont have access to should still pull data from public repos that i dont have access to. I expect the streams to output data into postgres tables.

Current Behavior

Only the issue_events, and assignee tables are populated, every other table is blank. Not sure if this is because the repo i am pulling from is large and airbyte cant handle it, or if its because of permissions issues.

Logs

If applicable, please upload the logs from the failing operation. For sync jobs, you can download the full logs from the UI by going to the sync attempt page and clicking the download logs button at the top right of the logs display window.

Steps to Reproduce

1. create a github source with this public repo: https://github.com/department-of-veterans-affairs/vets-website
2. disable the collaborators, projects, events, and teams streams
3. create a postgres destination
4. run the sync job, it will succeed, but no data is returned.

Severity of the bug for you

Critical

Airbyte Version

0.24.7-alpha

Connector Version (if applicable)

Github singer: 0.2.7

Additional context

Environment, version, integration...

[Moofasax]

logs-2-0 (1).txt
Attached logs

[Moofasax]

just attempted this with a smaller repo: https://github.com/ablevets/test-public and i was able to pull in data in all expected tables, is this a performance issue with airbyte? The repo i mentioned in the initial issue: https://github.com/department-of-veterans-affairs/vets-website is really large.

[marcosmarxm]

@Moofasax from your logs

io.airbyte.config.ReplicationAttemptSummary@29f1d914[status=completed,recordsSynced=40444,bytesSynced=227838114,startTime=1623163689523,endTime=1623164818856]

First, could you check the *_stargazers table?

[Moofasax]

all stargazers tables are empty, the only two tables that have any data in them are issue_events, and the assignees tables, all other tables are empty.

[Moofasax]

are you seeing this if you try what was stated in the original issue? exporting data out of this public repo: https://github.com/department-of-veterans-affairs/vets-website with these streams disabled the collaborators, projects, events?

[garden-of-delete]

@Moofasax I may be experiencing the same thing. Try syncing the stargazers stream on its own

(connection settings -> update latest source schema buttton -> deselect all except stargazers -> save changes and reset data button -> update latest source schema button in the pop up -> save changes button -> reset button -> let the reset finish and start a new sync)

Also how many records does Airbyte say it synced when the task returns successful?

[Moofasax]

It returns 40,444 and i will try to reset and sync only the stargazers stream now:

I noticed theres a new github version 0.28 i will try that one after?

[Moofasax]

With only stargazers checked, it does return 100 records for the _stargazers tables!!

[garden-of-delete]

With only stargazers checked, it does return 100 records for the _stargazers tables!!

@Moofasax @marcosmarxm this lines up with my experience of this issue

[Moofasax]

thanks for reporting, i am trying this with the newer github singer 0.2.8 and will report. Also nice username @garden-of-delete

[Moofasax]

Same issue on 0.2.8

[marcosmarxm]

@Moofasax and @garden-of-delete I had run an integration sync yesterday. Is it possible to try again?

[Moofasax]

What versions of everything? I can try!!

[marcosmarxm]

What versions of everything? I can try!!

Airbyte 0.24.7-alpha
Github 0.2.7
Postgres 0.3.5

[Moofasax]

@marcosmarxm I am still seeing the failed behavior, no other tables get created. Did you test with this repo? https://github.com/department-of-veterans-affairs/vets-website

[Moofasax]

I tested with the latest version too, same result:
Airbyte: 0.25.0-alpha
Github: 0.2.8
postgres: 0.3.4

@marcosmarxm i dont seem to have the option for postgres 0.3.5

[garden-of-delete]

Worth adding to the conversation. I'm getting the same result as @Moofasax on github 0.2.8. I have tried with bigquery and postgres destinations and have the exact same result in both cases. Just FYI.

[Moofasax]

based off @garden-of-delete comment, does that pin point its the github singer issue, and not a destination issue?

[marcosmarxm]

@Moofasax probably. I'm running using apache/superset. The sync didn't finish yet but got records in _raw tables. After finished I'll update here. This is the current state of my sync connection btw Github => Postgres.

[Moofasax]

Im not seeing the connection to what apache/superset has to do with this, but that data does not look right, there should not be that many stargazers...

[marcosmarxm]

@Moofasax superset is the project @garden-of-delete works and it has 39k stars 😄
When I used my credentials with a repository I had full access like Airbyte to get all data. When I pull data from another repository that I don't have full access I faced the same issue as you. Looking at the logs there are a lot of errors because Airbyte is trying to pull data from API endpoints that are not allowed for that credential. Should be a problem, but Github closed the connection and Airbyte doesn't handle that error and finish the sync.

[Moofasax]

Oh I see I'm sorry.

So any idea on why I can pull data from this public repo, https://github.com/ablevets/test-public and not this public repo: https://github.com/department-of-veterans-affairs/vets-website my access to both are the exact same.

[marcosmarxm]

So any idea on why I can pull data from this public repo, https://github.com/ablevets/test-public and not this public repo: https://github.com/department-of-veterans-affairs/vets-website my access to both are the exact same.
Probably because one is small and the number of errors didn't exceed the limit to close the connection, the second achieve that.

@Moofasax #3695 is a WIP and probably will solve the issue. Is possible to wait until this release? I'll keep you update on any news.

[Moofasax]

Understood, keep me posted and I'll test!! Appreciate the support!

[garden-of-delete]

@Moofasax @marcosmarxm

@Moofasax superset is the project @garden-of-delete works and it has 39k stars 😄
When I used my credentials with a repository I had full access like Airbyte to get all data. When I pull data from another repository that I don't have full access I faced the same issue as you. Looking at the logs there are a lot of errors because Airbyte is trying to pull data from API endpoints that are not allowed for that credential. Should be a problem, but Github closed the connection and Airbyte doesn't handle that error and finish the sync.

What permissions the token being used has on the repo could be a factor, but it does not explain the full issue. Specifically, it doesn't explain why I pull empty tables when syncing all streams, but can pull a populated table with the same token when syncing one stream at a time.

I pull data regularly from apache/superset, but i don't have write access to that repo, so my results should be the same as any of yours.

[keu]

blocked by #3695

[Moofasax]

Just tested this with the new native github source, still having this issue.

Streams that did not pull data: issues, pull_requests, reviews

Going to test with events streams enabled and see if that helps

[marcosmarxm]

@Moofasax #4612 was open to solve the problem. You can remove both events streams and the sync should work.

[Moofasax]

@marcosmarxm with the evetns streams disabled i still do not get data for issues, pull_requests, or reviews with this repo:
https://github.com/department-of-veterans-affairs/vets-api

[yevhenii-ldv]

Hi @Moofasax
Six days ago we released new version of Github Native connector - 0.1.2, where we fixed the error described in the ticket #4612 .
That fix should also correct this issue.
You can upgrade your connector from the Admin page.
Please let us if the issue was fixed. Thanks

[sherifnada]

@Moofasax could you please reopn this issue if the issue persists? as far as we can tell we can not reproduce this on the new connector version

[garden-of-delete]

I've tested 0.1.2 and am getting data for all streams now. thanks for all the hard work!