Releases: apache/opendal
v0.29.1
What's Changed
- docs: Add convert from m*n to m+n by @Xuanwo in #1454
- docs: Polish comments for public types by @Xuanwo in #1455
- chore: fix typo by @jackwener in #1456
- chore: fix typo by @suyanhanx in #1459
- benches: Generate into Bytes instead by @Xuanwo in #1463
- feat(bindings/python): Add basic IO support by @Xuanwo in #1464
- feat(binding/node.js): basic IO by @suyanhanx in #1416
- chore(bindings/nodjes): Don't check-in binaries by @Xuanwo in #1469
- chore(binding/nodejs): specific package manager version with hash by @suyanhanx in #1470
- docs: Add discord chat link by @Xuanwo in #1474
- feat(bindings/nodejs): Align to OpenDAL exports by @suyanhanx in #1466
- chore(bindings/nodejs): remove duplicate attribute & unused comment by @suyanhanx in #1478
- refactor: Promote operator as a mod for futher refactor by @Xuanwo in #1479
- Bump to version 0.29.1 by @Xuanwo in #1481
New Contributors
- @suyanhanx made their first contribution in #1459
Full Changelog: v0.29.0...v0.29.1
v0.29.0
Upgrade to v0.29
In v0.29, we introduced [Object Writer][crate::docs::rfcs::rfc_1420_object_writer] to replace existing Multipart related APIs.
Users can now append multiparts bytes into object via:
let mut w = o.writer().await?;
w.write(bs1).await?;
w.write(bs2).await?;
w.close()
Along with this change, we cleaned up a lot of internal structs and traits. Users who used to depend on opendal::raw::io::{input,output}
should use opendal::raw::oio
instead.
Also, decompress related feature also removed. Users can use async-compression
with ObjectReader
directly.
What's Changed
- docs: Add services-dashmap feature by @PsiACE in #1404
- docs: Fix incorrect indent for title by @Xuanwo in #1405
- refactor: Decouple decompress read feature from opendal by @Xuanwo in #1406
- docs: Add internal sections of Accessor and Layer by @Xuanwo in #1408
- docs: Add more guide for Accessor by @Xuanwo in #1409
- docs: Add tutorial of building a duck storage service by @Xuanwo in #1410
- ci: Consistently apply license header by @tisonkun in #1411
- chore: typo fix by @jackwener in #1418
- RFC-1420: Object Writer by @Xuanwo in #1420
- docs: Add a basic object example by @Xuanwo in #1422
- chore: Make license check happy by @Xuanwo in #1423
- ci: add typos check by @PsiACE in #1425
- feat: oss backend support http protocol by @jackwener in #1432
- chore: typo-fix by @jackwener in #1434
- feat: Implement ObjectWriter Support by @Xuanwo in #1431
- refactor: Cleanup pager related implementation by @Xuanwo in #1439
- refactor: Polish the implement details for Writer by @Xuanwo in #1445
- refactor: Remove
io::input
and Renameio::output
tooio
by @Xuanwo in #1446 - feat/layers/retry: Add Write Retry support by @Xuanwo in #1447
- feat: Add Write append tests by @Xuanwo in #1448
- fix(services/s3): Fix part number for AWS S3 by @Xuanwo in #1450
- Bump to version 0.29 by @Xuanwo in #1451
New Contributors
- @tisonkun made their first contribution in #1411
- @jackwener made their first contribution in #1418
Full Changelog: v0.28.0...v0.29.0
v0.28.0
Upgrade to v0.28
In v0.28, we introduced Query Based Metadata. Users can query cached metadata with ObjectMetakey
to make sure that OpenDAL always makes the best decision.
- pub async fn metadata(&self) -> Result<ObjectMetadata>;
+ pub async fn metadata(
+ &self,
+ flags: impl Into<FlagSet<ObjectMetakey>>,
+ ) -> Result<Arc<ObjectMetadata>>;
Please visit Object::metadata()
's example for more details.
What's Changed
- feat: add dashmap support by @PsiACE in #1390
- RFC-1391: Object Metadataer by @Xuanwo in #1391
- refactor: Implement query based object metadata cache by @Xuanwo in #1395
- refactor: Store complete inside bits and add more examples by @Xuanwo in #1397
- RFC-1398: Query Based Metadata by @Xuanwo in #1398
- refactor: Trigger panic if users try to visit not fetched metadata by @Xuanwo in #1399
- refactor: Polish the implement of Query Based Metadata Cache by @Xuanwo in #1400
- Bump version to 0.28 by @Xuanwo in #1401
Full Changelog: v0.27.2...v0.28.0
v0.27.2
What's Changed
- feat: Add batch API for Accessor by @Xuanwo in #1339
- docs: Fix broken links by @Xuanwo in #1344
- feat: add Content-Disposition for inner API by @ClSlaid in #1347
- feat: add content-disposition support for services by @ClSlaid in #1350
- feat: webdav service support bearer token by @fatelei in #1349
- refactor: Authorization logic for WebdavBackend by @Young-Flash in #1348
- feat: support auth for HttpBackend by @Young-Flash in #1359
- feat: Add batch delete support by @Xuanwo in #1357
- docs: clarify about opendal user defined client by @ClSlaid in #1356
- fix(webhdfs): should prepend http:// scheme by @ClSlaid in #1354
- ci: Pin time <= 0.3.17 until we decide to bump MSRV by @Xuanwo in #1361
- ci: Only run service test on changing by @Xuanwo in #1363
- refactor(webhdfs): handle 307 redirection instead of noredirect by @ClSlaid in #1358
- refactor: Polish http authorization related logic by @Xuanwo in #1367
- ci: run tests with nextest by @PsiACE in #1370
- refactor: Cleanup duplicated code by @Xuanwo in #1373
- feat(webdav): add list and improve create by @imWildCat in #1330
- refactor: Cleanup some not needed error context by @Xuanwo in #1374
- feat: Integrate batch with existing ecosystem better by @Xuanwo in #1378
- feat: Add batch delete support for oss by @Xuanwo in #1385
- Bump to version 0.27.2 by @Xuanwo in #1386
New Contributors
- @fatelei made their first contribution in #1349
- @imWildCat made their first contribution in #1330
Full Changelog: v0.27.1...v0.27.2
v0.27.1
What's Changed
- feat: Add username and password support for WebDAV by @Young-Flash in #1323
- ci: Add test case for webdav with basic auth by @Xuanwo in #1327
- feat(oli): support s3 uri without profile by @xxchan in #1328
- chore: Make clippy happy by @Xuanwo in #1329
- chore(deps): update moka requirement from 0.9 to 0.10 by @dependabot in #1331
- chore(deps): update rocksdb requirement from 0.19 to 0.20 by @dependabot in #1332
- feat: Add scan support for kv adapter by @Xuanwo in #1333
- feat: Add scan support for sled by @Xuanwo in #1334
- fix(services/oss,s3): Metadata should be marked as complete by @Xuanwo in #1335
- Bump to version 0.27.1 by @Xuanwo in #1336
New Contributors
- @Young-Flash made their first contribution in #1323
Full Changelog: v0.27.0...v0.27.1
v0.27.0
Upgrade to v0.27
In v0.27, we refactored our list
related logic and added scan
support. So make Pager
and BlockingPager
associated types in Accessor
too!
pub trait Accessor: Send + Sync + Debug + Unpin + 'static {
type Reader: output::Read;
type BlockingReader: output::BlockingRead;
+ type Pager: output::Page;
+ type BlockingPager: output::BlockingPage;
}
User defined layers
Due to this change, all layers implementation should be changed. If there is not changed over pager, they can by changed like the following:
impl<A: Accessor> LayeredAccessor for MyAccessor<A> {
type Inner = A;
type Reader = MyReader<A::Reader>;
type BlockingReader = MyReader<A::BlockingReader>;
+ type Pager = A::Pager;
+ type BlockingPager = A::BlockingPager;
+ async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Pager)> {
+ self.inner.list(path, args).await
+ }
+ async fn scan(&self, path: &str, args: OpScan) -> Result<(RpScan, Self::Pager)> {
+ self.inner.scan(path, args).await
+ }
+ fn blocking_list(&self, path: &str, args: OpList) -> Result<(RpList, Self::BlockingPager)> {
+ self.inner.blocking_list(path, args)
+ }
+ fn blocking_scan(&self, path: &str, args: OpScan) -> Result<(RpScan, Self::BlockingPager)> {
+ self.inner.blocking_scan(path, args)
+ }
}
Usage of ops
To reduce the understanding overhead, we move all OpXxx
into opendal::ops
now. User may need to change:
- use opendal::OpWrite;
+ use opendal::ops::OpWrite;
Usage of RetryLayer
backon
is the implementation detail of our RetryLayer
, so we hide it from our public API. Users of RetryLayer
need to change the code like:
- RetryLayer::new(backon::ExponentialBackoff::default())
+ RetryLayer::new()
What's Changed
- ci: Fix dev container Dockerfile by @PsiACE in #1298
- fix: Rocksdb's scheme not output correctly by @Xuanwo in #1300
- refactor: Hide backon from our public API by @Xuanwo in #1302
- refactor: Don't expose ops structs to users directly by @Xuanwo in #1303
- feat: Add Retryable Pager Support by @ClSlaid in #1304
- refactor: Move and rename ObjectPager and ObjectEntry for more clear semantics by @Xuanwo in #1308
- refactor: Implement strong typed pager by @Xuanwo in #1311
- feat: Add Sled support by @jkleinknox in #1305
- chore: fix name typo in oss backend by @wcy-fdu in #1316
- feat: Add Object::scan() support by @Xuanwo in #1314
- feat: Add object page size support by @Xuanwo in #1318
- chore: Add typos-cli and fix typos by @Xuanwo in #1320
- deps: remove unused deps by @xxchan in #1321
- docs: Add risingwave in projects by @Xuanwo in #1322
- refactor: Extract scan as a new API and remove ListStyle by @Xuanwo in #1324
- Bump to version 0.27 by @Xuanwo in #1325
New Contributors
- @jkleinknox made their first contribution in #1305
- @wcy-fdu made their first contribution in #1316
- @xxchan made their first contribution in #1321
Full Changelog: v0.26.2...v0.27.0
v0.26.2
What's Changed
- docs: fix typo by @hezhizhen in #1285
- feat: Add ChaosLayer to inject errors into underlying services by @Xuanwo in #1287
- docs: Polish docs for better reading by @Xuanwo in #1288
- refactor: Split CompleteReaderLayer from TypeEraserLayer by @Xuanwo in #1290
- feat: Implement retry reader by @Xuanwo in #1291
- feat: use std::path::Path for fs backend by @baszalmstra in #1100
- refactor(services/fs): Remove not needed generic by @Xuanwo in #1292
- fix: FsBuilder can't be used with empty root anymore by @Xuanwo in #1293
- fix: Fix retry happened in seek's read ahead logic by @Xuanwo in #1294
- feat: Implement services webhdfs by @ClSlaid in #1263
- Bump to version 0.26.2 by @Xuanwo in #1296
New Contributors
- @hezhizhen made their first contribution in #1285
- @baszalmstra made their first contribution in #1100
Full Changelog: v0.26.1...v0.26.2
v0.26.1
v0.26.0
Upgrade to v0.26
In v0.26 we have replaced all internal dynamic dispatch usage with static dispatch. With this change, we can ensure that all operations performed inside OpenDAL are zero cost.
Due to this change, we have to refactor the logic of Operator
's init logic. In v0.26, we added opendal::Builder
trait and opendal::OperatorBuilder
. For the first glance, the only change to existing code will be like:
- let op = Operator::new(builder.build()?);
+ let op = Operator::new(builder.build()?).finish();
By adding a finish()
call, we will erase all generic types so that Operator
can still be easily to used everywhere as before.
Accessor
In v0.26, Accessor
has been changed into trait with associated types.
All services need to decalare the types returned as Reader
or BlockingReader
:
pub trait Accessor: Send + Sync + Debug + Unpin + 'static {
type Reader: output::Read;
type BlockingReader: output::BlockingRead;
}
If your service doesn't support read
or blocking_read
, we can use ()
to represent an dummy reader:
impl Accessor for MyDummyAccessor {
type Reader = ();
type BlockingReader = ();
}
Layer
As described before, OpenDAL prefer to use static dispatch. Layers are required to implement the new Layer
and LayeredAccessor
trait:
pub trait Layer<A: Accessor> {
type LayeredAccessor: Accessor;
fn layer(&self, inner: A) -> Self::LayeredAccessor;
}
#[async_trait]
pub trait LayeredAccessor: Send + Sync + Debug + Unpin + 'static {
type Inner: Accessor;
type Reader: output::Read;
type BlockingReader: output::BlockingRead;
}
LayeredAccessor
is a wrapper of Accessor
with the typed Innder
. All methods that not implemented will be forward to inner instead.
Builder
Since v0.26, we implement opendal::Builder
for all services, and services' mod will not be exported.
- use opendal::services::s3::Builder;
+ use opendal::services::S3;
Conclusion
Sorry again for the big changes in this release. It's a big step for OpenDAL to work in more critical systems.
What's Changed
- refactor: remove the duplicated dependency in dev-dependencies by @Kilerd in #1257
- feat: Add benchmarks for blocking_seek operations by @Xuanwo in #1258
- feat: add dev container by @PsiACE in #1261
- chore: refine devcontainer by @PsiACE in #1262
- feat: Zero Cost OpenDAL by @Xuanwo in #1260
- refactor: some code in GitHub Actions by @yihong0618 in #1269
- refactor: Don't expose services mod directly by @Xuanwo in #1271
- refactor: Polish Builder API by @Xuanwo in #1272
- feat: Allow dynamic dispatch layer by @Xuanwo in #1273
- Bump to version 0.26.0 by @Xuanwo in #1274
New Contributors
Full Changelog: v0.25.2...v0.26.0
v0.25.2
What's Changed
- fix(services/ghac): Fix log message for
ghac_upload
inwrite
by @rajivshah3 in #1239 - docs(http): remove out-dated comments by @PsiACE in #1240
- chore: Make sure oli/oay's cargo.lock is updated by @Xuanwo in #1241
- refacor(services/fs): Make normalized path check optional by @Xuanwo in #1242
- feat: Add basic object_store support by @Xuanwo in #1243
- docs: Add bindings in README by @Xuanwo in #1244
- chore: Fix binding of object_store by @Xuanwo in #1245
- feat: Implement webdav support by @Xuanwo in #1246
- docs: Add docs for webdav and http services by @Xuanwo in #1248
- docs: Add webdav in lib docs by @Xuanwo in #1249
- feat: Allow passing content_type to OSS presign by @flaneur2020 in #1252
- feat: Make sure short functions have been inlined by @Xuanwo in #1253
- chore(deps): bump amondnet/vercel-action from 25.1.0 to 25.1.1 by @dependabot in #1254
- Bump to version 0.25.2 by @Xuanwo in #1255
New Contributors
- @rajivshah3 made their first contribution in #1239
Full Changelog: v0.25.1...v0.25.2