Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add rclone tests to testsuite #90

Closed
wthorp opened this issue Nov 1, 2021 · 5 comments
Closed

add rclone tests to testsuite #90

wthorp opened this issue Nov 1, 2021 · 5 comments
Assignees
@halkyon

Comments

@wthorp
Copy link
Contributor

wthorp commented Nov 1, 2021

It seems a number of clients are actively using rclone, so I think it makes sense to write a few tests to make sure it's working correctly as part of our existing test suite scripts.

It's quite straightforward to get something going without a config file. Assuming same env vars we usually use:

rclone config create s3 env_auth true
rclone --s3-endpoint "$AWS_ENDPOINT" mkdir s3:test-files
rclone --s3-endpoint "$AWS_ENDPOINT" --s3-chunk-size 64M --s3-upload-cutoff 64M --s3-upload-concurrency 30 -vv copy /tmp/testfile.dat s3:test-files
rclone --s3-endpoint "$AWS_ENDPOINT" ls s3:test-files
@mniewrzal
Copy link
Contributor

We are testing rclone + current uplink against Stroj/Tardigrade backend so maybe gateway can also reuse existing rclone tests. One issue might be current problems with setting metadata as I think rclone tests are failing on that.

https://github.com/storj/uplink/blob/main/testsuite/scripts/rclone.sh

@halkyon halkyon self-assigned this Dec 2, 2021
@halkyon
Copy link
Contributor

halkyon commented Dec 6, 2021 •
edited
Loading

Current progress here: https://review.dev.storj.io/c/storj/gateway-mt/+/6430, and failures to be investigated: https://build.dev.storj.io/blue/organizations/jenkins/gateway-mt-gerrit/detail/gateway-mt-gerrit/3112/pipeline/

@storjBuildBot
Copy link
Collaborator

Change https://review.dev.storj.io/c/storj/gateway/+/6460 mentions this issue.

@halkyon
Copy link
Contributor

halkyon commented Dec 7, 2021 •
edited
Loading

Fixing storj/gateway-st#44 (https://review.dev.storj.io/c/storj/gateway/+/6431/15) was enough to fix some of the rclone tests, so here's a change to add rclone tests to ST: https://review.dev.storj.io/c/storj/gateway/+/6460. We can add these to MT later as well once we update the miniogw component.

storjBuildBot pushed a commit to storj/gateway-st that referenced this issue Dec 13, 2021
@halkyon
Jenkinsfile,testsuite: add rclone tests
adb9dbe 
This is the same set up as uplink rclone tests except we are testing
the s3 backend of rclone instead of the native Storj DC integration.

Additionally, this disables the metainfo rate limiter for testing
Gateway-ST, just as Gateway-MT currently does. rclone tests throw
a lot of requests at the test Satellite, so we need to disable this.

Updates storj/edge#90

Change-Id: I54462f14a251db562375025cf5faabf22ba86673
@storjBuildBot
Copy link
Collaborator

Change https://review.dev.storj.io/c/storj/gateway-mt/+/6529 mentions this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@halkyon halkyon

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@halkyon @wthorp @mniewrzal @storjBuildBot