feat: Add webhdfs support

[Xuanwo]

Signed-off-by: Xuanwo [email protected]

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

This PR add webdhfs support.

Also, this PR bump opendal to 0.27 to address the duplicated deps backon.

Checklist

• I have written necessary rustdoc comments
• I have added necessary unit tests and integration tests
• All checks passed in ./risedev check (or alias, ./risedev c)

Documentation

Click here for Documentation

Types of user-facing changes

• Installation and deployment

Release note

• Added webhdfs support

[wcy-fdu]

Thanks for your contribution! 🥰

[wcy-fdu]

LGTM for your changes!
I'll run some tests for a while and merge this PR later.

[Xuanwo]

I'll run some tests for a while and merge this PR later.

To run this test on existing hdfs deployment, please make sure dfs.webhdfs.enabled in hdfs-site.xml has been set to true.

[wcy-fdu]

I can not start RisingWave via webhdfs backend, is there any configuration problem?

Error: while parsing a block mapping, did not find expected key at line 145 column 17

[Xuanwo]

I can not start RisingWave via webhdfs backend, is there any configuration problem?

Error: while parsing a block mapping, did not find expected key at line 145 column 17

Doesn't seem to be an error from OpenDAL or hdfs.

[wcy-fdu]

Can I use webhdfs just by addingdfs.webhdfs.enabled == trueto the original hdfs configuration?

[wcy-fdu]

Can I use webhdfs just by addingdfs.webhdfs.enabled == trueto the original hdfs configuration?

[Xuanwo]

Can I use webhdfs just by addingdfs.webhdfs.enabled == trueto the original hdfs configuration?

It should work. OpenDAL test webhdfs in the same way: https://github.com/beyondstorage/setup-hdfs/blob/master/src/setup-hdfs.ts#L31-L48

[wcy-fdu]

Good news: successfully start RisingWave via webhdfs
Bad news: Got an error in compute node

2023-02-17T06:04:57.351493Z  WARN risingwave_storage::hummock::compactor::shared_buffer_compact: Shared Buffer Compaction failed with error: ObjectStore failed with IO error Unexpected (permanent) at write, context: { service: webhdfs, path: hummock_001/2.data } => building request, source: invalid format.
  backtrace of `HummockError`:
   0: <risingwave_storage::hummock::error::HummockError as core::convert::From<risingwave_storage::hummock::error::HummockErrorInner>>::from
             at ./src/storage/src/hummock/error.rs:68:10
   1: <T as core::convert::Into<U>>::into
             at /rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/core/src/convert/mod.rs:726:9
   2: risingwave_storage::hummock::error::HummockError::object_io_error
             at ./src/storage/src/hummock/error.rs:78:9
   3: core::ops::function::FnOnce::call_once
             at /rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/core/src/ops/function.rs:250:5
   4: core::result::Result<T,E>::map_err
             at /rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/core/src/result.rs:860:27
   5: risingwave_storage::hummock::sstable_store::SstableStore::put_sst_data::{{closure}}
             at ./src/storage/src/hummock/sstable_store.rs:205:9
   6: <risingwave_storage::hummock::sstable_store::BatchUploadWriter as risingwave_storage::hummock::sstable::writer::SstableWriter>::finish::{{closure}}::{{closure}}
             at ./src/storage/src/hummock/sstable_store.rs:573:17

[Xuanwo]

Oh, it's an error happened while buidling http request. What's the config are you using?

[Xuanwo]

cc @wcy-fdu, Hi opendal v0.27.2 has been released, please try again~

[wcy-fdu]

cc @wcy-fdu, Hi opendal v0.27.2 has been released, please try again~

Thanks, let me try it.

[wcy-fdu]

cc @wcy-fdu, Hi opendal v0.27.2 has been released, please try again~

Sorry for the wait, just found that the seem error occurs again when after updating to v0.27.2, and I guess it's because my env is incorrect. Let me rebuild a machine for verification.

[wcy-fdu]

I still can't write successfully with webhdfs, the hdfs-site.xml:

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <property>
        <name>dfs.permissions</name>
        <value>false</value>   
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:/home/ubuntu/hadoop/tmp/dfs/name</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:/home/ubuntu/hadoop/tmp/dfs/data</value>
    </property>
    <property>
        <name>dfs.webhdfs.enabled</name>
        <value>true</value>
    </property>
    <property>
        <name>dfs.namenode.http-address</name>
        <value>localhost:9870</value>
    </property>
    <property>
        <name>dfs.secondary.http.address</name>
        <value>localhost:9100</value>
    </property>
</configuration>

error log:

Caused by:
    statement failed: db error: ERROR: QueryError: internal error: Rpc error: gRPC error (Internal error): Storage error: Hummock error: Other error sync task failed for ObjectStore failed with IO error Unexpected (temporary) at write, context: { called: http_util::Client::send_async, service: webhdfs, path: hummock_001/2.data } => send async request, source: error sending request for url (http://127.0.0.1:9870/webhdfs/v1/risingwave/webhdfs/hummock_001/2.data?op=CREATE&overwrite=true): error trying to connect: tcp connect error: Connection refused (os error 111).
      backtrace of `HummockError`:
       0: <risingwave_storage::hummock::error::HummockError as core::convert::From<risingwave_storage::hummock::error::HummockErrorInner>>::from
                 at ./src/storage/src/hummock/error.rs:68:10
       1: <T as core::convert::Into<U>>::into
                 at /rustc/3984bc5833db8bfb0acc522c9775383e4171f3de/library/core/src/convert/mod.rs:726:9
       2: risingwave_storage::hummock::error::HummockError::object_io_error
                 at ./src/storage/src/hummock/error.rs:78:9

can you please help me to have a look? cc @Xuanwo

[ClSlaid]

Looks like the request from OpenDAL is refused. Will 'localhost' open to HTTP requests from foreign machines?

[Xuanwo]

"Connection refused" indicates that OpenDAL is unable to establish a connection with this endpoint. If you are running HDFS in Docker or Docker Compose, please ensure that the "dfs.namenode.http-address" is set correctly to Docker's network bridge.

[wcy-fdu]

"Connection refused" indicates that OpenDAL is unable to establish a connection with this endpoint. If you are running HDFS in Docker or Docker Compose, please ensure that the "dfs.namenode.http-address" is set correctly to Docker's network bridge.

I'm running webhdfs in aws ec2, so dfs.namenode.http-address should be set to the public ip of this machine?

[Xuanwo]

I'm running webhdfs in aws ec2, so dfs.namenode.http-address should be set to the public ip of this machine?

The core question is are you running hdfs and risingwave on the same machine? If not, you need to change it to ec2's public ip.

[wcy-fdu]

I'm running webhdfs in aws ec2, so dfs.namenode.http-address should be set to the public ip of this machine?

The core question is are you running hdfs and risingwave on the same machine? If not, you need to change it to ec2's public ip.

It is the same machine.

[Xuanwo]

For current information, I think it's not related to OpenDAL, please make sure curl http://127.0.0.1:9870 return valid WebHDFS response.

[wcy-fdu]

Finally I successfully ran RisingWave on webhdfs, thanks for your contribution!