bug: webhdfs doesn't handle redirection correctly

[Xuanwo]

WARN risingwave_storage::hummock::compactor::shared_buffer_compact: Shared Buffer Compaction failed with error: ObjectStore failed with IO error Unexpected (permanent) at write, context: { service: webhdfs, service: webhdfs, path: hummock_001/1.data } => redirection fail, source: missing field `Location` at line 1 column 155.

[ClSlaid]

Seems like the HTTP response lacks a Location field, what's the version of Hadoop?

OpenDAL will always PUT with noredirect enabled. For Hadoop 3, an HTTP 200 OK response carrying a Location JSON should be given.

[ClSlaid]

Considering users may put their self defined HTTP clients to OpenDAL, whose redirect policies may vary and causing different runtime behavior, OpenDAL's WebHDFS support relies on a noredirect argument, which was only offered for HDFS3.

I'll support low versioned WebHDFS services as soon as possible, sorry again for the bad coding experience brought to you. :(

BTW, we need to clarify that self defined HTTP clients for OpenDAL should never follow redirects automatically, since we will handle them manually.

[Xuanwo]

BTW, we need to clarify that self defined HTTP clients for OpenDAL should never follow redirects automatically, since we will handle them manually.

LGTM, can you help add this statement in HttpClient's doc?