HIVE-28575: Reduce hdfs filesystem rpc call #5504
Conversation
zhangbutao
force-pushed
the
reduce_hdfs_fs_call
branch
from
f4a9faa
to
7c0e5e8
Compare
| * Hadoop File System reverse lookups paths with raw ip addresses The File | ||
| * System URI always contains the canonical DNS name of the Namenode. | ||
| * Subsequently, operations on paths with raw ip addresses cause an exception | ||
| * since they don't match the file system URI. |
There was a problem hiding this comment.
I realy don't understand what's meaning of this comment.
https://github.com/apache/hive/blob/master/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestWarehouseDnsPath.java The tests show that getDnsPath has nothing to do with Hadoop File System reverse lookups.
IMO, the hdfs filesystem should be same as object stores, s3a/s3n, we can also optimizie hdfs fs like HIVE-24838. #2085 (comment) also said getDnsPath : The only useful thing this function does is transforming a relative path to an absolute path.
input = /warehouse/tablespace/managed/hive
output = hdfs://namenode-address.site:8020/warehouse/tablespace/managed/hive
I hope someone can help me to understand this comment about the Hadoop File System reverse lookups paths.
@ayushtkn Do you have any thought?
| @@ -1796,8 +1796,6 @@ public enum ConfVars { | |||
| "hive.metastore.custom.database.product.classname", "none", | |||
| "Hook for external RDBMS. This class will be instantiated only when " + | |||
| "metastore.use.custom.database.product is set to true."), | |||
| HIVE_BLOBSTORE_SUPPORTED_SCHEMES("hive.blobstore.supported.schemes", "hive.blobstore.supported.schemes", "s3,s3a,s3n", | |||
There was a problem hiding this comment.
I think we can delete this propertity, we can do optimizie the fs call for all schemas including hdfs.
zhangbutao
changed the title
zhangbutao
force-pushed
the
reduce_hdfs_fs_call
branch
from
7c0e5e8
to
e21ceef
Compare
zhangbutao
changed the title
asf-ci-hive
added
tests pending
tests unstable
and removed
tests passed
tests pending
labels
zhangbutao
force-pushed
the
reduce_hdfs_fs_call
branch
from
e21ceef
to
64134c7
Compare
asf-ci-hive
added
tests pending
tests unstable
and removed
tests unstable
tests pending
labels
zhangbutao
force-pushed
the
reduce_hdfs_fs_call
branch
from
64134c7
to
b40524a
Compare
|
What changes were proposed in this pull request?
Why are the changes needed?
HIVE-24838 #2085 added the optimization to reduce BlobStorages fs call. I think it is also can be used to reduce HDFS filesystem call.
This can also optimize some issues like HIVE-28523(if partition location is outside from table location)
Does this PR introduce any user-facing change?
Is the change a dependency upgrade?
No
How was this patch tested?
The existing tests