[QOL-8518] [QOL-8612] Multiple fixes (Redis ignore exceptions, speed improvements, cleanup s3 acl flags) #80

duttonw · 2022-02-09T04:13:32Z

Add plugin features to manage S3 visibility
- Ensure old resources can be hidden via property flag
- Allow non-referenced resources be removed just like an upload with same name.
Don't crash on full redis server when link download is being requested (ignore error on put/get/delete)
Skip initialising s3 sub-system when url already exists in redis when download link is requested
Cache is_private flag for dataset to skip after_package updates when no change occurred. (Since Validation and XLoader Plugins causes dataset/package to be updated from enqueued jobs)

- link resources may create a FileStorage object representing a zero-length binary stream, but they should not be treated as uploads

- attributes cannot be asserted directly, use 'hasattr' with assert_false - combine near-identical tests that only use different resource inputs

[QOL-8518] handle link resources using placeholder empty FileStorage

- this makes it easier to wipe the 'all objects' cache when uploading a new object

…urces

…store into develop

… ckan this is fine but with Archiver, Validator and QA the meta data is Huge

…date-all-resource-visibility Feature/qol 8612 add cli to update all resource visibility

…d ACL if any

- ACLs can have a very long expiry, since they should only change when a resource or dataset is changed, will will override the cache settings anyway.

…ensure still works on python2

…ier to reuse

QOL-8612 make non-current S3 objects private

- otherwise the Redis helper will ignore it

- don't provide a 'queue' arg unless we have a value - 'on_failure' and 'failure_ttl' require newer rq, so they only work on CKAN 2.9+ - 'title' should be passed at the top level, not injected into 'kwargs' or 'rq_kwargs'

- looks like the default is set before our tests can intercept the config

- CKAN 2.7 can't accept 'rq_kwargs' at all - even CKAN 2.9 doesn't have new enough RQ to accept 'on_failure'

QOL-8612 fix enqueuing of visibility updates

… so not inside S3FilestoreCommands class

…ence

QOL-8612 tweak class structure

ThrawnCA

Looks good to me, but then again I wrote most of it, so we should get someone else's opinion for it to be a proper peer review.

ThrawnCA and others added 30 commits January 31, 2022 13:20

[QOL-8518] handle link resources using placeholder empty FileStorage

ec8ef0d

- link resources may create a FileStorage object representing a zero-length binary stream, but they should not be treated as uploads

[QOL-8518] add tests for link resources being ignored

8acfaf9

[QOL-8518] fix tests

0ea9fd1

- attributes cannot be asserted directly, use 'hasattr' with assert_false - combine near-identical tests that only use different resource inputs

[QOL-8518] improve assertion for better debugging

132b7c2

[QOL-8518] oops fix assertion name

939918f

Merge pull request #74 from qld-gov-au/QOL-8518-ckan-2.9

5b55d90

[QOL-8518] handle link resources using placeholder empty FileStorage

[QOL-8612] make non-current S3 objects private

3ba849d

[QOL-8612] increase test debugging

1d851c4

[QOL-8612] increase test debugging

4d4e398

[QOL-8612] increase test debugging

67f3bec

[QOL-8612] increase debugging

0a01e52

[QOL-8612] increase debugging

bb77dff

[QOL-8612] make test filenames match URLs

ea1f474

[QOL-8612] clear cache of object visibility when updating resource

86e1fbd

[QOL-8612] fix visibility cache key names

fa684a0

[QOL-8612] base visibility cache on path, not resource ID

fa24291

- this makes it easier to wipe the 'all objects' cache when uploading a new object

[QOL-8612] revert logging levels

8f169fa

QOL-8612 Added CLI command to update visibility for all existing reso…

39bf674

…urces

QOL-8612 Corrected exception handling and some formatting

273e45b

Merge branch 'master' of https://github.com/qld-gov-au/ckanext-s3file…

11ecd80

…store into develop

Improve redis memory usage by not storing the full dict, on a vanilla…

c92db5f

… ckan this is fine but with Archiver, Validator and QA the meta data is Huge

Merge pull request #76 from qld-gov-au/feature/QOL-8612-add-cli-to-up…

097286e

…date-all-resource-visibility Feature/qol 8612 add cli to update all resource visibility

[QOL-8612] add test to check that non-current objects match configure…

382cd3f

…d ACL if any

[QOL-8612] use config flag to control visibility of non-current objects

5982862

[QOL-8612] set non-current visibility to match current if flag is 'auto'

8f7edac

remove ckan ini, add on_failure method for cleanup

f7236e4

ensure etag is properly encoded so new relic does not complain

7e72b2e

[QOL-8612] split cache expiry settings vs URLs vs ACLs

04393ce

- ACLs can have a very long expiry, since they should only change when a resource or dataset is changed, will will override the cache settings anyway.

ensure urlencoded params is not byte string in python3

7d2dee4

ensure urlencoded params is not byte string in python3 and checks to …

31bfcdc

…ensure still works on python2

ThrawnCA and others added 20 commits February 7, 2022 15:44

[QOL-8612] drop duplicate resource loading

6bb0655

[QOL-8612] extract Redis functions to a separate class so they're eas…

fa019d9

…ier to reuse

[QOL-8612] only update object visibility if package visibility changes

65e15c5

[QOL-8612] oops check new privacy status even if old doesn't exist

3b87820

[QOL-8612] fix logic

9d541c6

[QOL-8612] use fully path to redis helper class

c50c8df

Merge pull request #75 from qld-gov-au/QOL-8612-manage-s3-visibility

70dbb38

QOL-8612 make non-current S3 objects private

[QOL-8612] oops specify package visibility cache expiry

e9151db

- otherwise the Redis helper will ignore it

[QOL-8612] fix enqueue args

f5767c2

- don't provide a 'queue' arg unless we have a value - 'on_failure' and 'failure_ttl' require newer rq, so they only work on CKAN 2.9+ - 'title' should be passed at the top level, not injected into 'kwargs' or 'rq_kwargs'

[QOL-8612] add unit test for enqueue syntax

6068b01

[QOL-8612] add 'timeout' to expected enqueue args

e196741

[QOL-8612] use default timeout instead of trying to change it

2fd0688

- looks like the default is set before our tests can intercept the config

[QOL-8612] split enqueue testing according to CKAN version

5d930f1

[QOL-8612] fix versioning of enqueue parameters

3f79c43

- CKAN 2.7 can't accept 'rq_kwargs' at all - even CKAN 2.9 doesn't have new enough RQ to accept 'on_failure'

[QOL-8612] drop on_failure function as we don't have new enough RQ

bad14ee

[QOL-8612] update enqueue failure message to put exception at end

185fbe5

Merge pull request #78 from qld-gov-au/QOL-8612-manage-s3-visibility

d274b37

QOL-8612 fix enqueuing of visibility updates

QOL-8612 spacing matters in python, new function not on correct level…

9c7fb26

… so not inside S3FilestoreCommands class

QOL-8612 pathlib2 requires from six.moves.collections_abc import Sequ…

e653a09

…ence

Merge pull request #79 from qld-gov-au/qol-8612_fixups

30a4984

QOL-8612 tweak class structure

duttonw requested review from ThrawnCA, devonpis, kundupz, ganapavz, chris-randall-qol, tonymcneil and RossWebsterWork February 9, 2022 04:13

ThrawnCA approved these changes Feb 9, 2022

View reviewed changes

chris-randall-qol approved these changes Feb 11, 2022

View reviewed changes

ThrawnCA merged commit 661f720 into master Feb 11, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[QOL-8518] [QOL-8612] Multiple fixes (Redis ignore exceptions, speed improvements, cleanup s3 acl flags) #80

[QOL-8518] [QOL-8612] Multiple fixes (Redis ignore exceptions, speed improvements, cleanup s3 acl flags) #80

duttonw commented Feb 9, 2022

ThrawnCA left a comment

[QOL-8518] [QOL-8612] Multiple fixes (Redis ignore exceptions, speed improvements, cleanup s3 acl flags) #80

[QOL-8518] [QOL-8612] Multiple fixes (Redis ignore exceptions, speed improvements, cleanup s3 acl flags) #80

Conversation

duttonw commented Feb 9, 2022

ThrawnCA left a comment

Choose a reason for hiding this comment