release 0.188-tw-0.43 #123
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
processBatch() is responsible for telling the caller if it has produced a page, yields to time, or needs to shrink the batch size. Currently there is a local variable and a global variable to pass the information back to the caller, which is not elegant. Unify the interface to return a class to informm the caller the state returned by processBatch().
Add a map in SliceBigArray to track the underlying data of the slice it sets. The map aims to avoid under or over counting the memory usage. In production, we found the original approximation can be off by 2X.
`fieldMappings` is a `ImmutableList`, it cannot contain nulls.
We have observed repeated compilation of MethodHandle that leads to full GCs. We notice that flushing the specialized function cache mitigate the problem. We suspect that it is a JVM bug that is related to stale/corrupted profiling data associated with generated classes and/or dynamically-created MethodHandles. This might also mitigate problems like deoptimization storm or unintended interpreted execution.
Reverse the sign of offset when TZ is defined in Etc/GMT=/-<offset> format
When iterating the drivers, a snapshot will be taken to guarantee a view consistency. To iterate directly on drivers should be safe.
When `SpillAwareLookupSourceProvider.close` runs concurrently with `PartitionedLookupSourceFactory.closeCachedLookupSources`, `LookupSource.close` can be invoked twice concurrenctly on single `LookupSource` instance. This is a bug, since `LookupSource` doesn't need to be thread-safe.
With the current db-backed resource manager to support multiple clusters we need to have multiple databases (one database per cluster). With this change the schema of the db-backed resource manager will support multiple clusters with the new environment column. The db-backed resource manager will load the right configuration based on the configured environment.
Flattened plans contain textual representations of every fragment, the plan tree, and a flattened list of nodes. The flattened list is constructed using a visitor with a custom serializer which produces only ids and types for children. This format was selected to simplify post-hoc plan analysis.
Since `PartitionedLookupSource` implements `Closeable`, the `close()` method should be idempotent.
Since `PartitionedLookupSource` encapsulates an array of `LookupSource` instances, the `PartitionedLookupSource.close` should close all the partitions too. Currently the `LookupSource` implementations that have no-op `close` are `OuterLookupSource` (used for RIGHT OUTER JOIN when there would be only a single partition) and `IndexLookupSource` (used instead of `PartitionedLookupSource` when JOIN is optimized as index-join), so this commit effectively does not change anything, but makes code more future-proof.
We found in production users may flush a single cell with a size of 1GB. This can cause Hive writers to a huge amount of memory. Add stats to monitor the distribution of pages flushed.
We have observed deoptimization storm that leads to slowness. We suspect that it is a JVM bug that is related to stale/corrupted profiling data associated with generated classes.
This particular test takes 2 seconds on an idle mac. The original 4 second is too close, especially on a shared server.
The transaction was marked as cleared on the statement immediately following the START TRANSACTION statement.
This change fixes a corner case where we can pass -1 as offset to Block::getRegion(). This happens when fromIndex = -N, length = N, and array size is (N - 1).
When block.copyRegion() is used to copy the whole block and the block is already in a compact representation, the data in original block can be used and no memory allocation is needed.
If a slice is already in compact representation, the original slice can be used instead of copying it.
The following heuristic is used in PagesIndex.compact() and
Page.compact() to decide whether a block should be compacted
to save memory:
if (block.getSizeInBytes() < block.getRetainedSizeInBytes()) {
......
}
The purpose is to avoid unnecessary compact for a block already in
compact representation. However, retained size is always larger than
the logical size, as retained size includes the block instance size.
We have optimized Block.copyRegion() so the original block will
be returned when copying a whole compact block. Thus, this heuristic
is no longer necessary.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.