[self hosting] Support B2 download url like Cloudflare

[ahxxm]

Background

Backblaze B2 offers free egress via Cloudflare, rclone, for example, supports listing object keys by API, then download via a --b2-download-url to avoid fees.

This would be helpful for general browsing, sharing, generating embeddings and exporting. On a new model release and on-device rolling-in, this might save server owner a few bucks.

Least-effort Hacks

There are 2 parts: setup Cloudflare worker, pass the domain into server.

Cloudflare worker

Following the repo:

• fill B2_APPLICATION_KEY_ID, B2_ENDPOINT, BUCKET_NAME
• patch for CORS, deploy
• set application key

To patch for CORS OPTIONS requests, add the following to beginning of fetch(request, env):

if (request.method === 'OPTIONS') {
    return new Response(null, {
        status: 200,
        headers: {
            "Access-Control-Allow-Origin": "*",
            "Access-Control-Allow-Headers": "*",
            "Access-Control-Allow-Methods": "GET, HEAD, OPTIONS",
        }
    });
}

You'll get a domain, let's assume it's https://example-b2.workers.dev.

To verify it works:

• open up dev console in Desktop app, click a image and find object key in Location, it's userid+uuid+signatures
• access https://example-b2.workers.dev/{userid}/{uuid}, without signatures because the reverse proxy signs again

Museum

Now that we offload presign to Cloudflare workers, the museum API needs to 307 clients to {domain}/{objectKey}, one-line patch

diff --git a/server/pkg/controller/file.go b/server/pkg/controller/file.go
index 1774e4cc1..023a55edc 100644
--- a/server/pkg/controller/file.go
+++ b/server/pkg/controller/file.go
@@ -379,7 +379,7 @@ func (c *FileController) getSignedURLForType(ctx *gin.Context, fileID int64, obj
        if err != nil {
                return "", stacktrace.Propagate(err, "")
        }
-       return c.getHotDcSignedUrl(s3Object.ObjectKey)
+       return "https://example-b2.workers.dev/" + s3Object.ObjectKey, nil
 }

This only affects download.

PoC Result

I'm generating embeddings for recently imported takeouts, the first request to get URL

Location is simply domain+objectkey as expected, then download request

I can upload images as usual, the presigned url points to B2 bucket directly.

Implementation Considerations

I'll leave this as an exercise for the overwhelmed developers with limited bandwidth.

(I don't want to break the presign design, it's just that I'd rather hack around with Golang than Javascript)

Since the existing presign locks with s3client, yet the download URL highly dependents on how reverse proxy is setup, it might be easier to only add explicit support via variable name, e.g.

diff --git a/server/pkg/controller/file.go b/server/pkg/controller/file.go
index 1774e4cc1..c0aee1e39 100644
--- a/server/pkg/controller/file.go
+++ b/server/pkg/controller/file.go
@@ -755,6 +756,9 @@ func (c *FileController) cleanupDeletedFile(qItem repo.QueueItem) {
 }

 func (c *FileController) getHotDcSignedUrl(objectKey string) (string, error) {
+       if c.S3Config.hotDCPublicReverseProxyURL != nil {
+               return url.JoinPath(c.S3Config.hotDCPublicReverseProxyURL, objectKey)
+       }
        s3Client := c.S3Config.GetHotS3Client()
        r, _ := s3Client.GetObjectRequest(&s3.GetObjectInput{
                Bucket: c.S3Config.GetHotBucket(),

This is easy to revert as well, user can comment out the settings line and restart server.

[mnvr]

Related:

• https://ente.io/blog/tech/making-uploads-faster/
• https://github.com/ente-io/ente/blob/main/infra/workers/uploader/src/index.ts
• https://github.com/ente-io/ente/blob/main/infra/workers/files/src/index.ts

• ente/web/packages/new/photos/services/download.ts

  Line 673 in 9515cf7

  `https://files.ente.io/?fileID=${file.id}`,

  (similar switches exist for the uploader, and in mobile app code too)

[ahxxm]

ah I see, the files worker is deployed with files.ente.io and it simply proxies download request with token such that traffic is between bucket and Cloudflare, clients are already aware of the switch, clients make a decision based on customAPIOrigin or endpoint

Might server-side URL decision simplify if-else branches? and hopefully to prevent issues like this, in this case it would be old versions making retire the domain impossible

[mnvr]

Right, it can be done server side too. It is client side because (a) it historically evolved that way, and (b) there is one case where we still need to make this decision client side - Sometimes we get support requests from users who are unable to upload files because of issues between their ISP and Cloudflare, and for such cases, we have provided a flag in advanced settings to bypass the worker for uploads (their uploads start working if they bypass the worker using this setting, or they switch to a different network)

Case (b) can be handled by a query parameter indicating a client preference, so yes, it would be nice to move this server side in the future.

There is a related long-term code restructuring we have planned, where the configuration of the S3 buckets in the museum configuration is much more flexible than what it is right now (per file type replication preferences, separate primary/secondary buckets per file type etc). As part of this, this choice of whether or not to use an associated worker for downloads / uploads, and the URL of the worker, these all can be part of the museum config itself.