Delete also blobs ending in slash when deleting directory contents

[findinpath]

Description

AWS S3 allows creating "folder" blobs with the media type "application/x-directory". These blobs should be deleted as well along with the normal blobs when deleting the contents of a directory in order to ensure that the directory corresponding to a table is actually fully deleted.

Additional context and related issues

Use solution inspired from #13974

Fixes #21111

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(x) Release notes are required, with the following suggested text:

# S3 File System
* Delete also blobs ending in slash when deleting directory contents. ({issue}`issuenumber`)

[ebyhr]

/test-with-secrets sha=723929b0c851fc76607639506c5b5309a32c5f95

[github-actions]

The CI workflow run with tests that require additional secrets finished as failure: https://github.com/trinodb/trino/actions/runs/8334770270

[findinpath]

CI flake #21153 on https://github.com/trinodb/trino/actions/runs/8334770270

[anusudarsan]

@findinpath can you also confirm if #21135 (comment) is green with this fix?

[findinpath]

tests               | 2024-03-19 20:49:19 INFO: [8 of 1] io.trino.tests.product.deltalake.TestDeltaLakeDropTableCompatibility.testDropTable [DELTA, DELTA, false] (Groups: profile_specific_tests, delta-lake-databricks, delta-lake-oss)
presto-master       | 2024-03-19T20:49:21.192+0545	INFO	dispatcher-query-5	io.trino.event.QueryMonitor	TIMELINE: Query 20240319_150420_00019_3xrdv :: FINISHED :: elapsed 433ms :: planning 0ms :: waiting 0ms :: scheduling 433ms :: running 0ms :: finishing 433ms :: begin 2024-03-19T20:49:20.759+05:45 :: end 2024-03-19T20:49:21.192+05:45
tests               | 2024-03-19 20:49:28 INFO: SUCCESS     /    io.trino.tests.product.deltalake.TestDeltaLakeDropTableCompatibility.testDropTable [DELTA, DELTA, false] (Groups: profile_specific_tests, delta-lake-databricks, delta-lake-oss) took 8.8 seconds
tests               | 2024-03-19 20:49:28 INFO: 
tests               | 2024-03-19 20:49:28 INFO: Completed 8 tests
tests               | 2024-03-19 20:49:28 INFO: 8 SUCCEEDED      /      0 FAILED      /      0 SKIPPED
tests               | 2024-03-19 20:49:28 INFO: Tests execution took 1 minutes and 18 seconds
tests               | 2024-03-19 20:49:28 INFO: ManageTestResources.onFinish: running checks
tests               | 
tests               | ===============================================
tests               | tempto-tests
tests               | Total tests run: 8, Failures: 0, Skips: 0
tests               | ===============================================
tests               | 
....
➜  trino git:(findinpath/s3-delete-directory-objects) git log --oneline 

bf9b8a0023 (HEAD -> findinpath/s3-delete-directory-objects) Use native S3 filesystem for Databricks tests
cca6b2d138 (findinpath/findinpath/s3-delete-directory-objects) Delete also blobs ending in slash when deleting directory contents
e23c4165f8 (trino/master, trino/HEAD, master) Remove unused methods

can you also confirm if #21135 (comment) is green with this fix?

Yes, I can. The current solution works as expected also for Databricks tests.
Do note that there was no need to do apply any .acl(...) on PutObjectRequest

[findinpath]

Unrelated CI failurept (default, suite-tpch, )
https://github.com/trinodb/trino/actions/runs/8349481254/job/22854092598#step:4:32
Unable to download artifact(s): Unable to download and extract artifact: Artifact download failed after 5 retries.

[findepi]

/test-with-secrets sha=5fda039a57f5e106d3c0990f93da389a1f4c30cd

[github-actions]

The CI workflow run with tests that require additional secrets has been started: https://github.com/trinodb/trino/actions/runs/8357686550