Issue with large files using Swift storage

[FUSAKLA]

While migrating from MinIO to Swift I found out that the Gophercloud library does not support large file support out of the box. Swift has by default limit of 5GB for file (our setup has 500MB).
More info here https://docs.openstack.org/swift/latest/overview_large_objects.html

I'd say this is pretty big limitation since the indexes can grow to enormous file size.

We could implement the Large File support with gophercloud ourselves but it's definitely not trivial.
There is also gophercloud/utils where is helper function for upload which implements the support but it requires the file to be on the disk (regarding out bucket interface it would require changes or writing to tmp file) but this could be possibly changed upstream. Also it has no releases and does not look much maintained. https://github.com/gophercloud/utils/blob/master/openstack/objectstorage/v1/objects/upload.go#L119

Other possibility is to use different library for the Swift.
Probably only one which we could consider to use is https://github.com/ncw/swift
It supports those Large Files so should allow Thanos to upload gig indexes.

@sudhi-vm @bwplotka WDYT?

[FUSAKLA]

I needed to get this working so i I took the stab and tried to implement the migration on the ncw/swift library.

I'm in phase of testing and verification right now but the PR should be good to go I hope.
#2732

PTAL

[stale]

Hello 👋 Looks like there was no activity on this issue for last 30 days.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity for next week, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[stale]

Closing for now as promised, let us know if you need this to be reopened! 🤗