Tracking issues of user metadata support

[Xuanwo]

This issue is used to track the implementation of user metadata support.

Context

There are two types of metadata across various storage services.

• System Defined: metadata like last-modified, content-type and so on. Some of them are editable, like content-type, but some are non-editable like last-modified.
• User Defined: extra metadata that provided by user, can be any valid string.

OpenDAL now only supports System Defined metadata, we should add support for User Defined metadata so that users can write most useful information along with the file data. For example, write the location for this file.

Tasks

• Implement user_metadata for metadata
  • Add user_metadata: HashMap<String, String> in Metadata
  • Add user_metadata in metakey
  • Add write_with_user_metadata in capability
  • Add user_metadata integration tests
• Impelment user_metadata for services
  • s3: feat(core/services-s3): support user defined metadata #5030
  • azblob: https://learn.microsoft.com/en-us/rest/api/storageservices/set-blob-metadata
  • oss: feat(core): support user defined metadata for oss #4881
  • gcs: feat(core/services-gcs): support user defined metadata #5276
  • cos
  • obs
  • swift
  • azdls
  • azfile
  • b2
  • fs
  • upyun
  • webdav

[meteorgan]

Hi, I'm interested in this issue.
but I have one problem to confirm：how do clients intend to use this feature, like

let mut w = op
      .writer_with("path/to/file")
      .content_type("application/octet-stream")
      .user_metadata(map)
      .await?;

?

[Xuanwo]

but I have one problem to confirm：what client wants to use this feature, like

Hi, there are many use cases of this feature. For example, a db can write their own metadata like tenant, catalog, database, table in this map. So they can query those metadata without extra meta services.

[meteorgan]

emmm, I should make it clear: I've reviewed the use cases, and would like to confirm the API for users.
should I add an function to FutureWrite like system defined metadata ?

[Xuanwo]

should I add an function to FutureWrite like system defined metadata ?

Yes, we need a new API called user_metadata for both write and writer

[meteorgan]

should I add an function to FutureWrite like system defined metadata ?

Yes, we need a new API called user_metadata for both write and writer

Ok, i'll begin addressing this issue now

[jorgehermo9]

Hi @Xuanwo, I see that there are some of the tasks that are already done (the ones from the "Implement user_metadata for metadata" section and the s3 service). Should this tracking issue checklist be updated with that?

I can help implementing user_metadata for other services, I'd appreciate if the checklist can be extended to at least the ones with the most priority?

[Xuanwo]

Hi, thank you @jorgehermo9 for your work on this!

I will take some time to extend the list and sort it by priority. Before that, I believe the next good option could be gcs support.