Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

doc: improve FileIO doc #642

Merged
merged 1 commit into from
Sep 23, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 13 additions & 3 deletions crates/iceberg/src/io/file_io.rs
Original file line number Diff line number Diff line change
Expand Up @@ -32,16 +32,25 @@ use crate::{Error, ErrorKind, Result};
///
/// All path passed to `FileIO` must be absolute path starting with scheme string used to construct `FileIO`.
/// For example, if you construct `FileIO` with `s3a` scheme, then all path passed to `FileIO` must start with `s3a://`.
///
/// Supported storages:
///
/// | Storage | Feature Flag | Schemes |
/// |--------------------|-------------------|------------|
/// | Local file system | `storage-fs` | `file` |
/// | Memory | `storage-memory` | `memory` |
/// | S3 | `storage-s3` | `s3`, `s3a`|
/// | GCS | `storage-gcs` | `gs` |
#[derive(Clone, Debug)]
pub struct FileIO {
inner: Arc<Storage>,
}

impl FileIO {
/// Try to infer file io scheme from path.
/// Try to infer file io scheme from path. See [`FileIO`] for supported schemes.
///
/// If it's a valid url, for example http://example.org, url scheme will be used.
/// If it's not a valid url, will try to detect if it's a file path.
/// - If it's a valid url, for example `s3://bucket/a`, url scheme will be used, and the rest of the url will be ignored.
/// - If it's not a valid url, will try to detect if it's a file path.
///
/// Otherwise will return parsing error.
pub fn from_path(path: impl AsRef<str>) -> crate::Result<FileIOBuilder> {
Expand Down Expand Up @@ -111,6 +120,7 @@ pub struct FileIOBuilder {

impl FileIOBuilder {
/// Creates a new builder with scheme.
/// See [`FileIO`] for supported schemes.
pub fn new(scheme_str: impl ToString) -> Self {
Self {
scheme_str: Some(scheme_str.to_string()),
Expand Down
1 change: 1 addition & 0 deletions crates/iceberg/src/io/storage.rs
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@ impl Storage {
Scheme::Gcs => Ok(Self::Gcs {
config: super::gcs_config_parse(props)?.into(),
}),
// Update doc on [`FileIO`] when adding new schemes.
_ => Err(Error::new(
ErrorKind::FeatureUnsupported,
format!("Constructing file io from scheme: {scheme} not supported now",),
Expand Down
Loading