Datatx pull model #2052

redblom · 2021-09-08T08:27:05Z

This PR implements CS3 tx (pull model) (cs3org/cs3apis#143) using rclone as data transfer tool.
The following commands are available: transfer-create, transfer-list, transfer-get-status, transfer-retry, transfer-cancel

Create a transfer (sender):

#  ./cmd/reva/reva transfer-create -grantee 3000989c-6gh5-4cec-a9ce-e876tf7de452 -granteeType user -idp cesnet.cz /home/data/big-data
+--------------------------------------+-------------+--------------------------------------+-----------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------+-------------+--------------------------------------+---------------------+-------------------------------+-------------------------------+
| #                                    | OWNER.IDP   | OWNER.OPAQUEID                       | RESOURCEID                                                                                          | PERMISSIONS                                                                                                                                                                                                     | TYPE              | GRANTEE.IDP | GRANTEE.OPAQUEID                     | SHARETYPE           | CREATED                       | UPDATED                       |
+--------------------------------------+-------------+--------------------------------------+-----------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------+-------------+--------------------------------------+---------------------+-------------------------------+-------------------------------+
| 775b47bd-4cca-4470-86f7-0424b7a998ab | surfsara.nl | 4029579c-6ad5-4cec-a9ce-e843f77de452 | storage_id:"123e4567-e89b-12d3-a456-426655440000" opaque_id:"fileid-jimmie%2Fdata%2Fdata/big-data"  | permissions:<create_container:true delete:true get_path:true initiate_file_download:true initiate_file_upload:true list_container:true list_file_versions:true move:true restore_file_version:true stat:true >  | GRANTEE_TYPE_USER | cesnet.cz   | 3000989c-6gh5-4cec-a9ce-e876tf7de452 | SHARE_TYPE_TRANSFER | 2021-09-06 10:52:47 +0000 UTC | 2021-09-06 10:52:47 +0000 UTC |
+--------------------------------------+-------------+--------------------------------------+-----------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------+-------------+--------------------------------------+---------------------+-------------------------------+-------------------------------+

Discover and accept the transfer (receiver):

# ./cmd/reva/reva ocm-share-list-received
+--------------------------------------+-------------+--------------------------------------+----------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------+-------------+--------------------------------------+-------------------------------+-------------------------------+----------------------+---------------------+
| #                                    | OWNER.IDP   | OWNER.OPAQUEID                       | RESOURCEID                                                                                               | PERMISSIONS                                                                                                                                                       | TYPE              | GRANTEE.IDP | GRANTEE.OPAQUEID                     | CREATED                       | UPDATED                       | STATE                | SHARETYPE           |
+--------------------------------------+-------------+--------------------------------------+----------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------+-------------+--------------------------------------+-------------------------------+-------------------------------+----------------------+---------------------+
| 40ff72c6-c30c-4380-827d-5e2d3970b57c | surfsara.nl | 4029579c-6ad5-4cec-a9ce-e843f77de452 | storage_id:"123e4567-e89b-12d3-a456-426655440000" opaque_id:"fileid-jimmie%2Fdata%2Fbig-data"       | permissions:<get_path:true get_quota:true initiate_file_download:true list_grants:true list_container:true list_file_versions:true list_recycle:true stat:true >  | GRANTEE_TYPE_USER | cesnet.cz   | 3000989c-6gh5-4cec-a9ce-e876tf7de452 | 2021-09-06 10:52:47 +0000 UTC | 2021-09-06 10:52:47 +0000 UTC | SHARE_STATE_PENDING  | SHARE_TYPE_TRANSFER |
+--------------------------------------+-------------+--------------------------------------+----------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------+-------------+--------------------------------------+-------------------------------+-------------------------------+----------------------+---------------------+

# ./cmd/reva/reva ocm-share-update-received -state accepted 40ff72c6-c30c-4380-827d-5e2d3970b57c
OK

Receiver discovers the transfer ID needed to get the status of, retry, or cancel the transfer:

# ./cmd/reva/reva transfer-list -shareId 40ff72c6-c30c-4380-827d-5e2d3970b57c
+--------------------------------------+--------------------------------------+
| SHAREID.OPAQUEID                     | ID.OPAQUEID                          |
+--------------------------------------+--------------------------------------+
| 40ff72c6-c30c-4380-827d-5e2d3970b57c | 76508e0e-eda1-43fc-9b4d-c6b526b5a75c |
+--------------------------------------+--------------------------------------+

Get the transfer status:

# ./cmd/reva/reva transfer-get-status -txId 76508e0e-eda1-43fc-9b4d-c6b526b5a75c
+--------------------------------------+--------------------------------------+----------------+-----------------------------------+
| SHAREID.OPAQUEID                     | ID.OPAQUEID                          |         STATUS | CTIME                             |
+--------------------------------------+--------------------------------------+----------------+-----------------------------------+
| 40ff72c6-c30c-4380-827d-5e2d3970b57c | 76508e0e-eda1-43fc-9b4d-c6b526b5a75c | STATUS_TRANSFER_COMPLETE | Mon Sep 6 10:53:08 +0000 UTC 2021 |
+--------------------------------------+--------------------------------------+----------------+-----------------------------------+

Retry a failed transfer:

# ./cmd/reva/reva transfer-retry -txId 76508e0e-eda1-43fc-9b4d-c6b526b5a75c
+--------------------------------------+--------------------------------------+---------------------+-------------------------------+
| SHAREID.OPAQUEID                     | ID.OPAQUEID                          |              STATUS | CTIME                         |
+--------------------------------------+--------------------------------------+---------------------+-------------------------------+
| 40ff72c6-c30c-4380-827d-5e2d3970b57c | 76508e0e-eda1-43fc-9b4d-c6b526b5a75c | STATUS_TRANSFER_NEW | 2021-09-06 10:53:08 +0000 UTC |
+--------------------------------------+--------------------------------------+---------------------+-------------------------------+

Cancel a running transfer:

# ./cmd/reva/reva transfer-cancel -txId f82396c4-386b-484f-856c-b75936980f1e
+--------------------------------------+--------------------------------------+---------------------------+-----------------------------------+
| SHAREID.OPAQUEID                     | ID.OPAQUEID                          |                    STATUS | CTIME                             |
+--------------------------------------+--------------------------------------+---------------------------+-----------------------------------+
| a940df85-9023-4cf8-809a-4c33748f2c66 | f82396c4-386b-484f-856c-b75936980f1e | STATUS_TRANSFER_CANCELLED | Wed Sep 1 11:56:25 +0000 UTC 2021 |
+--------------------------------------+--------------------------------------+---------------------------+-----------------------------------+

redblom · 2021-09-27T13:12:47Z

@labkode @ishank011 please review

labkode · 2021-12-13T09:09:22Z

@redblom the PR is ready, can you just rebase on top of master and then we merge it with the cs3apis changes?

redblom · 2021-12-13T11:15:38Z

@redblom the PR is ready, can you just rebase on top of master and then we merge it with the cs3apis changes?

@labkode : Done!

proper logging properly decode rclone http error response

glpatcern

As this work was already reviewed and it was simply pending the CI, I'm approving it.

ishank011

One comment and then we can merge

go.mod

lgtm-com · 2022-03-16T10:12:04Z

This pull request introduces 1 alert when merging 02754ff into dcb9b01 - view on LGTM.com

new alerts:

1 for Wrapped error is always nil

redblom requested a review from labkode as a code owner September 8, 2021 08:27

redblom mentioned this pull request Sep 8, 2021

CS3 pulltransfer #1824

Closed

redblom force-pushed the datatx-pull-model branch from 0ee5178 to 200db48 Compare September 8, 2021 12:15

redblom changed the title ~~[WIP] Datatx pull model~~ Datatx pull model Sep 27, 2021

redblom mentioned this pull request Oct 13, 2021

[WIP] Datatx initiate transfer #1759

Closed

redblom force-pushed the datatx-pull-model branch from 20d05eb to 9afc29c Compare November 4, 2021 13:49

redblom requested a review from ishank011 as a code owner November 4, 2021 13:49

redblom force-pushed the datatx-pull-model branch from 9afc29c to e15d83d Compare November 4, 2021 14:10

labkode mentioned this pull request Dec 13, 2021

Data transfer pull model cs3org/cs3apis#143

Merged

redblom mentioned this pull request Dec 13, 2021

[WIP] Datatx driver #1242

Closed

redblom force-pushed the datatx-pull-model branch from e15d83d to 1ad992b Compare December 13, 2021 10:56

redblom requested review from glpatcern and a team as code owners December 13, 2021 10:56

Antoon Prins added 14 commits February 14, 2022 10:54

New methods PullTransfer,RetryTransfer,ListTransfers.

1e6c8e2

Add changelog

6cf498a

PR link added

a91aadf

new commands transfer-list, transfer-retry

2fae89e

Transfer id is parameter. Pretty output.

e80ca96

Transfer id is parameter. Pretty output.

ff69a40

Add pkg datatx loader.

50e31f9

Data transfers folder renamed.

f7a0ae4

Implement datatx pull model and methods.

67a1d20

pkg datatx (rclone) implemented.

eb62539

stat to analyse path

27b77a5

proper logging properly decode rclone http error response

implement handling of both src and dest targetURI of pull model

1b43865

command comments fixed

e84d0b6

datatx service example configuration

bdaf0fc

Antoon Prins and others added 6 commits February 14, 2022 10:54

Always save the transfer regardless of start transfer outcome

1d15446

New PR.

8c00dab

Lint fixes.

84a1c5f

Make sure to be in end state on any transfer failure.

e926829

Merge branch 'main' into datatx-pull-model

c73e8d8

New cs3apis version: data transfers

f412f5d

redblom force-pushed the datatx-pull-model branch from 1ad992b to f412f5d Compare February 14, 2022 10:52

glpatcern previously approved these changes Feb 14, 2022

View reviewed changes

ishank011 added the under review: don't merge label Feb 14, 2022

glpatcern mentioned this pull request Mar 2, 2022

update cs3apis #2594

Closed

ishank011 previously approved these changes Mar 16, 2022

View reviewed changes

go.mod Outdated Show resolved Hide resolved

ishank011 removed the under review: don't merge label Mar 16, 2022

Remove obsolete line.

02754ff

redblom dismissed stale reviews from ishank011 and glpatcern via 02754ff March 16, 2022 09:49

ishank011 approved these changes Mar 16, 2022

View reviewed changes

ishank011 merged commit 559fccb into cs3org:master Mar 16, 2022

Daniel-WWU-IT pushed a commit to Daniel-WWU-IT/reva that referenced this pull request Mar 16, 2022

Datatx pull model (cs3org#2052)

94418be

micbar mentioned this pull request May 5, 2022

Release 2.0.0-beta1 owncloud/ocis#3627

Closed

45 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Datatx pull model #2052

Datatx pull model #2052

redblom commented Sep 8, 2021 •

edited

Loading

redblom commented Sep 27, 2021

labkode commented Dec 13, 2021

redblom commented Dec 13, 2021

glpatcern left a comment

ishank011 left a comment

lgtm-com bot commented Mar 16, 2022

Datatx pull model #2052

Datatx pull model #2052

Conversation

redblom commented Sep 8, 2021 • edited Loading

redblom commented Sep 27, 2021

labkode commented Dec 13, 2021

redblom commented Dec 13, 2021

glpatcern left a comment

Choose a reason for hiding this comment

ishank011 left a comment

Choose a reason for hiding this comment

lgtm-com bot commented Mar 16, 2022

redblom commented Sep 8, 2021 •

edited

Loading