feat(airbyte-cdk): CSV decoder #47131
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Skipped Deployment
|
|
/format-fix
|
|
I'll do mypy in a bit, too. |
|
mypy checks are on legacy to per partition state, not related to the CSVDecoder per se. |
|
|
||
| parameters: InitVar[Mapping[str, Any]] | ||
|
|
||
| def __post_init__(self, parameters: Mapping[str, Any]): |
There was a problem hiding this comment.
Is there a benefit to injecting the delimiter as a parameter property vs directly injecting as an argument?
There was a problem hiding this comment.
I don't know! Give me a suggestion here, I have no idea how this is done or if it will actually work when instantiated from an actual manifest. It could easily be that this is not the right way.
| def is_stream_response(self) -> bool: | ||
| return False | ||
|
|
||
| def decode(self, response: requests.Response) -> Generator[MutableMapping[str, Any], None, None]: | ||
| try: | ||
| csv_content = io.StringIO(response.text) | ||
| reader = csv.reader(csv_content, delimiter=self.delimiter) |
There was a problem hiding this comment.
🚨This is a potential OOM🚨
is_stream_responseshould beTrue- I'd suggest to use
pandas.read_csvand read response in chunks (see Salesforce as an example)
There was a problem hiding this comment.
!!111 you're right! Thank you kindly.
What
This PR adds CSV decoder to the declarative schema, and implements the decoder itself.
TODO