Clear field meta data during reading so that it can be processed #2949
Conversation
|
ACTION NEEDED The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification. For details on the error please inspect the "PR Title Check" action. |
| let mut batch_stream = result.map(|b| b); | ||
| let mut batches = Vec::new(); | ||
| while let Some(batch) = batch_stream.next().await { | ||
| assert!(batch.is_ok()); |
There was a problem hiding this comment.
The error occurs at this point, but for some reason, the exact error message is not reported by cargo test (it crashes without providing useful information). However, if the test case is placed in lance/examples/write_read_ds.rs, the error message described in issue #2947 will be displayed.
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2949 +/- ##
==========================================
+ Coverage 78.99% 79.01% +0.02%
==========================================
Files 234 234
Lines 72822 72890 +68
Branches 72822 72890 +68
==========================================
+ Hits 57525 57596 +71
+ Misses 12332 12325 -7
- Partials 2965 2969 +4
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
|
Close this PR in favor of #2950 |
This PR aims to demonstrate the issue described in LanceDB/lance#2947, where a dataset containing a nested field with associated metadata can be written successfully but cannot be read correctly by Lance.
In this PR, I propose a quick fix that clears the field metadata when reading from the manifest. However, this approach doesn't seem to be a perfect solution to the issue. I also attempted to remove the field metadata right before the
decodemethod’sStructArray::try_newcall, but there appear to be multiple places with similar issues, so I opted to clear the metadata once when reading the field from the manifest instead.It would be great if we have better solution to this issue. Thanks.