-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(airbyte-cdk): CSV decoder #47131
base: master
Are you sure you want to change the base?
Conversation
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Skipped Deployment
|
/format-fix
|
I'll do mypy in a bit, too. |
mypy checks are on legacy to per partition state, not related to the CSVDecoder per se. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
APPROVED
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Approved. See one comment/question.
|
||
parameters: InitVar[Mapping[str, Any]] | ||
|
||
def __post_init__(self, parameters: Mapping[str, Any]): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a benefit to injecting the delimiter as a parameter property vs directly injecting as an argument?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't know! Give me a suggestion here, I have no idea how this is done or if it will actually work when instantiated from an actual manifest. It could easily be that this is not the right way.
def is_stream_response(self) -> bool: | ||
return False | ||
|
||
def decode(self, response: requests.Response) -> Generator[MutableMapping[str, Any], None, None]: | ||
try: | ||
csv_content = io.StringIO(response.text) | ||
reader = csv.reader(csv_content, delimiter=self.delimiter) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🚨This is a potential OOM🚨
is_stream_response
should beTrue
- I'd suggest to use
pandas.read_csv
and read response in chunks (see Salesforce as an example)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
!!111 you're right! Thank you kindly.
What
This PR adds CSV decoder to the declarative schema, and implements the decoder itself.
TODO