Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(services/webdav): Add possibility to answer without response if file isn't exist #4170

Merged
merged 1 commit into from
Feb 7, 2024
Merged

fix(services/webdav): Add possibility to answer without response if file isn't exist #4170

merged 1 commit into from
Feb 7, 2024

Conversation

AJIOB
Copy link
Contributor

@AJIOB AJIOB commented Feb 7, 2024

While testing sccache with JFrog Artifactory as a WebDAV storage, I found the problem: sccache cannot store marker file while the curl can do it.

After deeper investigation & debugging I found that Artifactory answers to the PROPFIND request with this XML when file is not presented on the remote server:

<?xml version="1.0" encoding="UTF-8"?>
<D:multistatus xmlns:D="DAV:" xmlns:ns0="DAV:" />

And on the next step Multistatus cannot be deserialized => sccache was panicked with this logs:

[2024-02-07T14:24:29Z DEBUG opendal::services::webdav::backend] backend stat response bytes (string): "<?xml version=\"1.0\" encoding=\"utf-8\" ?>\n<D:multistatus xmlns:D=\"DAV:\" xmlns:ns0=\"DAV:\"></D:multistatus>\n"
[2024-02-07T14:24:29Z ERROR opendal::services] service=webdav operation=stat path=.sccache_check -> Unexpected (permanent) at stat, context: { service: webdav, path: .sccache_check } => deserialize xml, source: missing field `response`
[2024-02-07T14:24:30Z ERROR sccache::server] storage check failed for: cache storage failed to read: Unexpected (permanent) at stat => deserialize xml

    Context:
       service: webdav
       path: .sccache_check

    Source:
       missing field `response`

    Backtrace:
       0: opendal::types::error::Error::new
                 at /home/alex/Projects/waste/opendal/core/src/types/error.rs:338:24
       1: opendal::raw::serde_util::new_xml_deserialize_error
                 at /home/alex/Projects/waste/opendal/core/src/raw/serde_util.rs:33:5
       2: core::ops::function::FnOnce::call_once
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
       3: core::result::Result<T,E>::map_err
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/result.rs:829:27
       4: <opendal::services::webdav::backend::WebdavBackend as opendal::raw::accessor::Accessor>::stat::{{closure}}
                 at /home/alex/Projects/waste/opendal/core/src/services/webdav/backend.rs:315:13
       5: <core::pin::Pin<P> as core::future::future::Future>::poll
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/future/future.rs:125:9
       6: <F as futures_core::future::TryFuture>::try_poll
                 at /home/alex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.30/src/future.rs:82:9
       7: <futures_util::future::try_future::into_future::IntoFuture<Fut> as core::future::future::Future>::poll
                 at /home/alex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/try_future/into_future.rs:34:9
       8: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
                 at /home/alex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/future/map.rs:55:37
       9: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
                 at /home/alex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/lib.rs:91:13
      10: <futures_util::future::try_future::MapErr<Fut,F> as core::future::future::Future>::poll
                 at /home/alex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/lib.rs:91:13
      11: <opendal::layers::error_context::ErrorContextAccessor<A> as opendal::raw::layer::LayeredAccessor>::stat::{{closure}}
                 at /home/alex/Projects/waste/opendal/core/src/layers/error_context.rs:173:14
      12: <core::pin::Pin<P> as core::future::future::Future>::poll
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/future/future.rs:125:9
      13: opendal::raw::layer::<impl opendal::raw::accessor::Accessor for L>::stat::{{closure}}
                 at /home/alex/Projects/waste/opendal/core/src/raw/layer.rs:246:39
      14: <core::pin::Pin<P> as core::future::future::Future>::poll
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/future/future.rs:125:9
      15: <alloc::sync::Arc<T> as opendal::raw::accessor::Accessor>::stat::{{closure}}
                 at /home/alex/Projects/waste/opendal/core/src/raw/accessor.rs:408:40
      16: <core::pin::Pin<P> as core::future::future::Future>::poll
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/future/future.rs:125:9
      17: opendal::layers::complete::CompleteAccessor<A>::complete_stat::{{closure}}
                 at /home/alex/Projects/waste/opendal/core/src/layers/complete.rs:226:37
      18: <opendal::layers::complete::CompleteAccessor<A> as opendal::raw::layer::LayeredAccessor>::stat::{{closure}}
                 at /home/alex/Projects/waste/opendal/core/src/layers/complete.rs:566:40
      19: <core::pin::Pin<P> as core::future::future::Future>::poll
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/future/future.rs:125:9
      20: opendal::raw::layer::<impl opendal::raw::accessor::Accessor for L>::stat::{{closure}}
                 at /home/alex/Projects/waste/opendal/core/src/raw/layer.rs:246:39
      21: <core::pin::Pin<P> as core::future::future::Future>::poll
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/future/future.rs:125:9
      22: <opendal::layers::logging::LoggingAccessor<A> as opendal::raw::layer::LayeredAccessor>::stat::{{closure}}
                 at /home/alex/Projects/waste/opendal/core/src/layers/logging.rs:464:14
      23: <core::pin::Pin<P> as core::future::future::Future>::poll
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/future/future.rs:125:9
      24: opendal::raw::layer::<impl opendal::raw::accessor::Accessor for L>::stat::{{closure}}
                 at /home/alex/Projects/waste/opendal/core/src/raw/layer.rs:246:39
      25: <core::pin::Pin<P> as core::future::future::Future>::poll
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/future/future.rs:125:9
      26: opendal::raw::layer::LayeredAccessor::stat::{{closure}}
                 at /home/alex/Projects/waste/opendal/core/src/raw/layer.rs:167:39
      27: <core::pin::Pin<P> as core::future::future::Future>::poll
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/future/future.rs:125:9
      28: opendal::raw::layer::<impl opendal::raw::accessor::Accessor for L>::stat::{{closure}}
                 at /home/alex/Projects/waste/opendal/core/src/raw/layer.rs:246:39
      29: <core::pin::Pin<P> as core::future::future::Future>::poll
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/future/future.rs:125:9
      30: <alloc::sync::Arc<T> as opendal::raw::accessor::Accessor>::stat::{{closure}}
                 at /home/alex/Projects/waste/opendal/core/src/raw/accessor.rs:408:40
      31: <core::pin::Pin<P> as core::future::future::Future>::poll
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/future/future.rs:125:9
      32: opendal::types::operator::operator::Operator::read_with::{{closure}}::{{closure}}
                 at /home/alex/Projects/waste/opendal/core/src/types/operator/operator.rs:536:26
      33: opendal::types::operator::operator::Operator::read::{{closure}}
                 at /home/alex/Projects/waste/opendal/core/src/types/operator/operator.rs:421:30
      34: <opendal::types::operator::operator::Operator as sccache::cache::cache::Storage>::check::{{closure}}
                 at /home/alex/Projects/waste/sccache/src/cache/cache.rs:487:31
      35: <core::pin::Pin<P> as core::future::future::Future>::poll
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/future/future.rs:125:9
      36: sccache::server::start_server::{{closure}}
                 at /home/alex/Projects/waste/sccache/src/server.rs:446:31
      37: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
                 at /home/alex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:63
      38: tokio::runtime::coop::with_budget
                 at /home/alex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:107:5
      39: tokio::runtime::coop::budget
                 at /home/alex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:73:5
      40: tokio::runtime::park::CachedParkThread::block_on
                 at /home/alex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:31
      41: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
                 at /home/alex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/blocking.rs:66:9
      42: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
                 at /home/alex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:87:13
      43: tokio::runtime::context::runtime::enter_runtime
                 at /home/alex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/runtime.rs:65:16
      44: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
                 at /home/alex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:86:9
      45: tokio::runtime::runtime::Runtime::block_on
                 at /home/alex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/runtime.rs:350:45
      46: sccache::server::start_server
                 at /home/alex/Projects/waste/sccache/src/server.rs:445:22
      47: sccache::commands::run_command
                 at /home/alex/Projects/waste/sccache/src/commands.rs:663:13
      48: sccache::main
                 at /home/alex/Projects/waste/sccache/src/lib.rs:88:30
      49: sccache::main
                 at /home/alex/Projects/waste/sccache/src/main.rs:18:5
      50: core::ops::function::FnOnce::call_once
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
      51: std::sys_common::backtrace::__rust_begin_short_backtrace
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:154:18
      52: std::rt::lang_start::{{closure}}
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:167:18
      53: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:284:13
      54: std::panicking::try::do_call
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
      55: std::panicking::try
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
      56: std::panic::catch_unwind
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
      57: std::rt::lang_start_internal::{{closure}}
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:48
      58: std::panicking::try::do_call
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
      59: std::panicking::try
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
      60: std::panic::catch_unwind
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
      61: std::rt::lang_start_internal
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
      62: std::rt::lang_start
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
      63: main
      64: __libc_start_call_main
                 at ./csu/../sysdeps/nptl/libc_start_call_main.h:58:16
      65: __libc_start_main_impl
                 at ./csu/../csu/libc-start.c:392:3
      66: _start

With my patch all works on the JFrog Artifactory WebDAV and on this WebDAV servers.
Without them, only the second server really works.

P.S. I've rebuilt the latest SCCache 0.7.7 with the latest OpenDAL 0.45.0 for collecting this backtrace & investigation.
P.P.S. After patches both SCCache cache hits & writes are works nice, tested locally.

@AJIOB AJIOB requested a review from Xuanwo as a code owner February 7, 2024 16:29
@Xuanwo Xuanwo changed the title Add possibility to answer without response if file isn't exist fix(services/webdav): Add possibility to answer without response if file isn't exist Feb 7, 2024
Xuanwo
Xuanwo approved these changes Feb 7, 2024
View reviewed changes
Copy link
Member

@Xuanwo Xuanwo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perfect, thanks a lot!

@Xuanwo
Copy link
Member

Xuanwo commented Feb 7, 2024

Does jfrog-artifactory-oss also provides webdav features? Maybe we can setup a new service to test webdav.

@AJIOB
Copy link
Contributor Author

AJIOB commented Feb 7, 2024

@Xuanwo, there is no info about this here.

As I can see, they wrote if feature is available in the JFrog Artifactory Pro and higher, as it was written for S3 backend here.

@Xuanwo
Copy link
Member

Xuanwo commented Feb 7, 2024

@Xuanwo, there is no info about this here.

As I can see, they wrote if feature is available in the JFrog Artifactory Pro and higher, as it was written for S3 backend here.

Thanks for the info! I'll give it a shot. By adding a test for WebDAV, we can ensure that OpenDAL doesn't accidentally break it.

@Xuanwo Xuanwo merged commit e628198 into apache:main Feb 7, 2024
60 checks passed
@Xuanwo Xuanwo mentioned this pull request Feb 7, 2024
Closed
@AJIOB AJIOB deleted the AJIOB/webdav-artifactory-sccache-workaround branch February 8, 2024 13:10
@Xuanwo Xuanwo mentioned this pull request Feb 22, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Xuanwo Xuanwo Xuanwo approved these changes

@G-XD G-XD Awaiting requested review from G-XD

@Young-Flash Young-Flash Awaiting requested review from Young-Flash

Assignees
No one assigned
Labels
releases-note/fix
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@AJIOB @Xuanwo