-
Notifications
You must be signed in to change notification settings - Fork 59
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
834 support string value type in frame #860
base: main
Are you sure you want to change the base?
834 support string value type in frame #860
Conversation
…> struct for std::string and FixedStr16
…ng and FixedStr16
- Renamed test CSV and meta data files. - Removed unnecessary casts of ValueTypeUtils::default_value to the type it already has. - Renamed ValueTypeUtils::default_value to defaultValue. - oneHot-kernel - Removed superfluous additional convenience functions. - Release recoded intermediate. - Made index calculation view-aware. - Reduced the code duplication of the specializations for std::string and FixedStr16 by factoring out the code into a separate function template. - Tidied up BinaryOpCode.h. - Little formatting corrections. - And some more minor things.
- The FixedStr16 buffer no longer requires a null-terminator. - This change optimizes memory usage for FixedStr16 value type.
…es of strings in DaphneDSL.
-Add EW_UNARY operation upper and lower on c_strings -Add support for EW_UNARY operation upper and lower on dense matrix of strings in DSL. -Add support for EW_UNARY operation upper and lower on strings in DSL.
- Add String values in `mlirTypeForCode` in `TypeOpInterface.cpp` for proper type mapping - Fix memory deallocation for String columns in the Frame constructor using a custom deleter - Add String `ValueTypeCode` and `cppNameFor` for recognizing and naming the String type - Add a test in `api/cli/io` with String values for reading a CSV file into a frame
Hi @saminbassiri , thanks for providing this PR, reading frames with strings now works 😃 ! However, I was following up on a small example use-case using SQL on dataframes, and this crashes. I'm not sure, but I guess it's still related to string processing. data.csv and the corresponding .meta file can be taken from #834
attached is the complete output of above script (I compiled your PR on my own using the daphne build container) - it starts with printing the first 5 sliced rows…:
Also note that the last 2 lines do not compile, as daphne is not able to properly infer the string type:
when the dataframe had only numbers (instead of strings in the 1st column), the SQL function works. KR, Benjamin |
This PR closes issue #834 by adding support for reading CSV files containing String values into Frames.
Changes:
General Support for String Values in Frames at Kernel Level:
Support for Reading String Values in Frames at Kernel Level:
Support for Reading String Values from CSV Files into Frames in DSL:
str
in the.meta
file to ensure proper type recognition.Testing: