[GLUTEN-1434] Serialize and deserialize RowVector #250
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jinchengchenghh
force-pushed
the
serialize
branch
from
4a38461
to
2696a36
Compare
jinchengchenghh
force-pushed
the
serialize
branch
from
9e57e79
to
ee02bea
Compare
|
Can you help review this one, it will replace current netty serialize Arrow ColumnarBatch to bytes. https://github.com/oap-project/gluten/blob/main/gluten-data/src/main/scala/io/glutenproject/backendsapi/glutendata/GlutenSparkPlanExecApi.scala#L227 |
rui-mo
reviewed
May 15, 2023
| @@ -0,0 +1,761 @@ | |||
| /* | |||
There was a problem hiding this comment.
Is there any other proper name instead of SparkXXX?
There was a problem hiding this comment.
How about SingleVectorSerializer?
There was a problem hiding this comment.
Would RowVectorSerializer make sense?
There was a problem hiding this comment.
The super class is following, it is really RowVectorSerializer, other serializer can append several RowVectorPtr, but this serializer can only append one non-empty vector, so I named it SingleVectorSerializer
class VectorSerializer {
public:
virtual ~VectorSerializer() = default;
virtual void append(
const RowVectorPtr& vector,
const folly::Range<const IndexRange*>& ranges) = 0;
// Writes the contents to 'stream' in wire format
virtual void flush(OutputStream* stream) = 0;
};
| #include "velox/common/memory/ByteStream.h" | ||
| // #include "velox/functions/prestosql/types/TimestampWithTimeZoneType.h" | ||
| // #include "velox/type/Date.h" | ||
| // #include "velox/vector/BiasVector.h" |
rui-mo
reviewed
May 16, 2023
|
|
||
| template <TypeKind kind> | ||
| void serializeFlatVector(const BaseVector* vector, SparkVectorStream* stream) { | ||
| // using T = typename TypeTraits<kind>::NativeType; |
| // time, call appendNull or appendNonNull first. Then call | ||
| // appendLength if the type has a length. A null value has a length of | ||
| // 0. Then call appendValue if the value was not null. | ||
| class SparkVectorStream { |
jinchengchenghh
force-pushed
the
serialize
branch
from
ee02bea
to
2ede5ad
Compare
|
Fixed all the comments. Can you help review again? @rui-mo |
@jinchengchenghh LGTM. Is this PR verified on CI and Jenkins? |
|
The API does not used in Gluten now, it is only tested by velox unit tests |
zhejiangxiaomai
pushed a commit
to zhejiangxiaomai/velox
that referenced
this pull request
May 18, 2023
Support serialize and deserialize RowVector.
zhejiangxiaomai
pushed a commit
to zhejiangxiaomai/velox
that referenced
this pull request
May 31, 2023
Support serialize and deserialize RowVector.
zhejiangxiaomai
added a commit
to zhejiangxiaomai/velox
that referenced
this pull request
May 31, 2023
relative pr: Serialize and deserialize RowVector oap-project#250
zhejiangxiaomai
added a commit
to zhejiangxiaomai/velox
that referenced
this pull request
May 31, 2023
relative pr: Support more data types for read filter oap-project#139 Fix cast double to decimal oap-project#179 Fix casting from string to decimal oap-project#281 Support cast decimal to int oap-project#177 Fix null on overflow and multiply as spark precision and support cast varchar to decimal oap-project#169 Disable tokenizing the path by dot oap-project#109 Serialize and deserialize RowVector oap-project#250
zhejiangxiaomai
added a commit
to zhejiangxiaomai/velox
that referenced
this pull request
May 31, 2023
relative pr: Support more data types for read filter oap-project#139 Fix cast double to decimal oap-project#179 Fix casting from string to decimal oap-project#281 Support cast decimal to int oap-project#177 Fix null on overflow and multiply as spark precision and support cast varchar to decimal oap-project#169 Disable tokenizing the path by dot oap-project#109 Serialize and deserialize RowVector oap-project#250 Support datetime pattern in spark oap-project#94
zhejiangxiaomai
added a commit
to zhejiangxiaomai/velox
that referenced
this pull request
May 31, 2023
relative pr: Support more data types for read filter oap-project#139 Fix cast double to decimal oap-project#179 Fix casting from string to decimal oap-project#281 Support cast decimal to int oap-project#177 Fix null on overflow and multiply as spark precision and support cast varchar to decimal oap-project#169 Disable tokenizing the path by dot oap-project#109 Serialize and deserialize RowVector oap-project#250 Support datetime pattern in spark oap-project#94
zhejiangxiaomai
pushed a commit
to zhejiangxiaomai/velox
that referenced
this pull request
Jun 25, 2023
Support serialize and deserialize RowVector.
zhejiangxiaomai
pushed a commit
to zhejiangxiaomai/velox
that referenced
this pull request
Jun 25, 2023
Support serialize and deserialize RowVector.
zhejiangxiaomai
pushed a commit
to zhejiangxiaomai/velox
that referenced
this pull request
Jun 26, 2023
Support serialize and deserialize RowVector.
zhejiangxiaomai
pushed a commit
to zhejiangxiaomai/velox
that referenced
this pull request
Jun 27, 2023
Support serialize and deserialize RowVector.
zhejiangxiaomai
added a commit
to zhejiangxiaomai/velox
that referenced
this pull request
Jul 3, 2023
relative pr: Serialize and deserialize RowVector oap-project#250
zhejiangxiaomai
added a commit
to zhejiangxiaomai/velox
that referenced
this pull request
Jul 3, 2023
relative pr: Support more data types for read filter oap-project#139 Fix cast double to decimal oap-project#179 Fix casting from string to decimal oap-project#281 Support cast decimal to int oap-project#177 Fix null on overflow and multiply as spark precision and support cast varchar to decimal oap-project#169 Disable tokenizing the path by dot oap-project#109 Serialize and deserialize RowVector oap-project#250 Support datetime pattern in spark oap-project#94
zhejiangxiaomai
added a commit
to zhejiangxiaomai/velox
that referenced
this pull request
Jul 4, 2023
relative pr: Serialize and deserialize RowVector oap-project#250
zhejiangxiaomai
added a commit
to zhejiangxiaomai/velox
that referenced
this pull request
Jul 4, 2023
relative pr: Support more data types for read filter oap-project#139 Fix cast double to decimal oap-project#179 Fix casting from string to decimal oap-project#281 Support cast decimal to int oap-project#177 Fix null on overflow and multiply as spark precision and support cast varchar to decimal oap-project#169 Disable tokenizing the path by dot oap-project#109 Serialize and deserialize RowVector oap-project#250 Support datetime pattern in spark oap-project#94
zhejiangxiaomai
added a commit
to zhejiangxiaomai/velox
that referenced
this pull request
Jul 11, 2023
relative pr: Serialize and deserialize RowVector oap-project#250
zhejiangxiaomai
added a commit
to zhejiangxiaomai/velox
that referenced
this pull request
Jul 11, 2023
relative pr: Support more data types for read filter oap-project#139 Fix cast double to decimal oap-project#179 Fix casting from string to decimal oap-project#281 Support cast decimal to int oap-project#177 Fix null on overflow and multiply as spark precision and support cast varchar to decimal oap-project#169 Disable tokenizing the path by dot oap-project#109 Serialize and deserialize RowVector oap-project#250 Support datetime pattern in spark oap-project#94
zhejiangxiaomai
added a commit
to zhejiangxiaomai/velox
that referenced
this pull request
Jul 12, 2023
relative pr: Serialize and deserialize RowVector oap-project#250
zhejiangxiaomai
added a commit
to zhejiangxiaomai/velox
that referenced
this pull request
Jul 12, 2023
relative pr: Support more data types for read filter oap-project#139 Fix cast double to decimal oap-project#179 Fix casting from string to decimal oap-project#281 Support cast decimal to int oap-project#177 Fix null on overflow and multiply as spark precision and support cast varchar to decimal oap-project#169 Disable tokenizing the path by dot oap-project#109 Serialize and deserialize RowVector oap-project#250 Support datetime pattern in spark oap-project#94
zhejiangxiaomai
added a commit
to zhejiangxiaomai/velox
that referenced
this pull request
Jul 12, 2023
relative pr: Serialize and deserialize RowVector oap-project#250
zhejiangxiaomai
added a commit
to zhejiangxiaomai/velox
that referenced
this pull request
Jul 12, 2023
relative pr: Support more data types for read filter oap-project#139 Fix cast double to decimal oap-project#179 Fix casting from string to decimal oap-project#281 Support cast decimal to int oap-project#177 Fix null on overflow and multiply as spark precision and support cast varchar to decimal oap-project#169 Disable tokenizing the path by dot oap-project#109 Serialize and deserialize RowVector oap-project#250 Support datetime pattern in spark oap-project#94
zhejiangxiaomai
added a commit
to zhejiangxiaomai/velox
that referenced
this pull request
Jul 17, 2023
relative pr: Serialize and deserialize RowVector oap-project#250
zhejiangxiaomai
added a commit
to zhejiangxiaomai/velox
that referenced
this pull request
Jul 17, 2023
relative pr: Support more data types for read filter oap-project#139 Fix cast double to decimal oap-project#179 Fix casting from string to decimal oap-project#281 Support cast decimal to int oap-project#177 Fix null on overflow and multiply as spark precision and support cast varchar to decimal oap-project#169 Disable tokenizing the path by dot oap-project#109 Serialize and deserialize RowVector oap-project#250 Support datetime pattern in spark oap-project#94
marin-ma
pushed a commit
to marin-ma/velox-oap
that referenced
this pull request
Dec 15, 2023
…ct#250) * fix backend velox codestyle * change remark codestyle to compact mode and add intellij codestyle xml
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.