-
Notifications
You must be signed in to change notification settings - Fork 159
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[CHORE] Enable read_sql for swordfish #2918
Conversation
CodSpeed Performance ReportMerging #2918 will not alter performanceComparing Summary
|
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2918 +/- ##
==========================================
+ Coverage 78.43% 78.47% +0.03%
==========================================
Files 603 610 +7
Lines 71504 71742 +238
==========================================
+ Hits 56086 56299 +213
- Misses 15418 15443 +25
|
return Err(common_error::DaftError::TypeError( | ||
"Database file format not implemented".to_string(), | ||
)); | ||
FileFormatConfig::Database(common_file_formats::DatabaseSourceConfig { sql, conn }) => { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The existing read_sql code for the current executor is here:
Daft/src/daft-micropartition/src/micropartition.rs
Lines 356 to 380 in 3f37a69
FileFormatConfig::Database(DatabaseSourceConfig { sql, conn }) => { | |
let predicate = scan_task | |
.pushdowns | |
.filters | |
.as_ref() | |
.map(|p| (*p.as_ref()).clone().into()); | |
Python::with_gil(|py| { | |
let table = crate::python::read_sql_into_py_table( | |
py, | |
sql, | |
conn, | |
predicate.clone(), | |
scan_task.schema.clone().into(), | |
scan_task | |
.pushdowns | |
.columns | |
.as_ref() | |
.map(|cols| cols.as_ref().clone()), | |
scan_task.pushdowns.limit, | |
) | |
.map(std::convert::Into::into) | |
.context(PyIOSnafu)?; | |
Ok(vec![table]) | |
})? | |
} |
futures::stream::once
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Might be nice to leave a comment about this
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Enables read_sql tests for swordfish.
Additionally adds an
EmptyScan
operator. This is needed because some sql tests produce 0 scan tasks (some other IO tests that are skipped also need empty scan, so this PR will enable those to pass as well, I will unskip them in future PRs).Technically, we don't need a dedicated empty scan operator for this, but it might be useful to be explicit. (For what it's worth the current runner does have an empty scan operator as well)