FR: Alternative to getDownloadURL that doesn't expose a public download URL

[jhuleatt]

[REQUIRED] Describe your environment

• Operating System version: [all]
• Firebase SDK version: 4.1.3
• Firebase Product: storage

[REQUIRED] Describe the problem

Currently, the only way to get an image is to call getDownloadUrl. The download URL is unguessable but not affected by security rules, which means that once someone has the download URL, they could theoretically use a bot to create a huge number of downloads and run up a big Cloud Storage bill or hit the project quota.

The Android SDK provides getFile, which downloads the file directly to the device. Each download involves a security rules check (source).

Request

Provide a function like getFile that checks security rules every time.

Alternatively, perhaps a function that returns a signed URL (for example, getSignedUrl(expireTime)) could accomplish a similar goal.

[sphippen]

We've been considering something like this for a while. The reason we decided to only expose a downloadURL on web is because of its versatility.

If we wanted to give you a way to download the file checked by security rules, the best we could do would probably be to give you an XMLHttpRequest you could use to download the object yourself. The problem with this is that if you're using an XHR the downloaded data ends up in memory, which means if it's a large file (say, more than a few gigabytes) it's difficult to do anything reasonable with the XHR.

On the other hand, with a download URL, many things are simpler (you can embed it directly in an img tag) and you can give the user the option to download the file (an <a> tag with the "download" attribute, tell them to right click > "Save file as...", etc.).

Of course, it's also easier for anybody to automate abusive behavior if they have a download URL (as you pointed out). At the same time, for a sufficiently dedicated malicious user there isn't a big difference between exposing the download URL vs not exposing it (any user capable of downloading the file via security rules can get a download URL and do anything they want with it). As a result, we don't consider this a high-priority change.

[dinvlad]

Could you clarify (apologies if these questions are better addressed elsewhere):

1. Does getDownloadUrl from FB JS SDK check security rules before generating a link? In other words, is there a reasonable guarantee that a download URL, unless shared, can only be obtained by FB users who have access to the object via security rules? And what's the lifetime of this link?

2. Is it possible to use accessToken generated by FB (with Google auth provider) to access Cloud Storage directly (bypassing FB mechanism for getDownloadUrl), e.g. by using Google Cloud SDK for JS? In which case, do the FB rules still apply? Or do we then need to manage access manually through IAM rules on the affected buckets? In which case, does the accessToken represent their "normal" Google user identity (so we can just add their GSuite email to IAM rules)?

[sphippen]

1. Access to the download URL through the SDK is indeed controlled by security rules. If you don't have read access to the object, getDownloadURL will fail. The link lasts until it is revoked manually in the console.
2. The Firebase accessToken cannot be used to access GCS except through Firebase Storage (which we only officially support through the SDKs). Even if the Firebase accessToken was generated with a Google user login, you'll still have to use the more typical credentials: a service account key, etc., if you want to talk to GCS' APIs directly.

[dinvlad]

Thanks @sphippen, regarding (2) I don't quite understand: if I authenticate with FB and ask for https://www.googleapis.com/auth/devstorage.read_only scope (privacy/app verification issues aside), then I sure am able to access GCS APIs (https://www.googleapis.com/storage/v1/b/...), as I have just verified through Postman. What do you mean then by

The Firebase accessToken cannot be used to access GCS except through Firebase Storage

[sphippen]

Sorry, I misunderstood what you meant by accessToken. I thought you meant the Firebase Auth JWT, but I see you were referring to the accessToken associated with the OAuth credential.

So, if you log a Google user into your app with Firebase Auth and request the https://www.googleapis.com/auth/devstorage.read_only permission, you get the OAuth accessToken back, which you can use to read GCS resources that that user has access to. This access is controlled by the IAM policy (or GCS' legacy ACLs) on the bucket/object you attempt to access.

In which case, does the accessToken represent their "normal" Google user identity (so we can just add their GSuite email to IAM rules)?

Exactly correct.

To summarize, the Firebase Auth JWT lets you access buckets associated with the project in which they were created through Firebase Storage. The Storage security rules are applied to these requests. (third-party auth)

If you log a Google user into your app, they'll click through a page granting you the permissions you request, and the OAuth accessToken will give you those permissions. You can use that token to make requests to GCS' normal API, but only to access resources the user already has access to through Cloud IAM or GCS' legacy ACL system. (first-party auth)

[dinvlad]

Thanks @sphippen! I've tested this approach and it works as you described. Hopefully it will help others as an alternative to getDownloadUrl().

That being said, requesting a devstorage scope is a more blanket permission and since last July Google requires verification of apps that use it. We may end up just using getDownloadUrl(), or (a 3rd alternative suggested by @jhuleatt) getSignedUrl() on the backend. The latter option is better for us because it enables limited-lifetime links. I just wish getDownloadUrl() contained an option to specify the lifetime.

[tigerpup]

I have a really basic question.
does getDownloadUrl give the same url on every call till the file has been reuploaded ?

[schmidt-sebastian]

In general, we return the same download URL. It is however possible to revoke an old URL in the Console, in which the next invocation will return a different URL.

[tborja]

Hello,
We use getDownloadUrl to give the firebase storage file link access to an authenticated user. The problem is the Url returned by getDownloadUrl has an access token attached to the link that grants access to the file even to unauthenticated users. How do we prevent this from happening?

[burtonator]

@tborja is right.. this is a major problem and I'm very very concerned that it hasn't been addressed yet.

Once this is handed out, there are NO security rules applied for reading that document.

This isn't really highlighted anywhere in the SDK docs and it's really really easy for a developer to shoot themselves in the foot here.

Specifically, I just did. I had assumed that getDownloadUrl would apply the storage security rules but that's not the case. I'm going to have to spend 1-2 weeks building an alternative.

Why can't there just be a proxy in between that evaluates the rules on every HTTP request and passes the rest on to cloud storage?

That's literally what I'm going to be doing next but Firebase should be doing this for me...

Maybe change this API call to getInsecureDownloadURL or getPublicDownloadURL or at least highlight in the API docs that there are NO security rules applied to this URL once created.

[burtonator]

Access to the download URL through the SDK is indeed controlled by security rules. If you don't have read access to the object, getDownloadURL will fail. The link lasts until it is revoked manually in the console.

Is there a way to revoke them via API and not manually?

I'm working on a feature for public sharing of documents and this way I could at least revoke the URL and generate a new one.

[dinvlad]

As an alternative, you can generate a signed URL with an expiration time using admin SDK inside a cloud function.

[burtonator]

I saw that but this doesn't really rectify the problem. Once the timed URL is issued there's no way to revoke it... This is back to the same issue. I need to hand out the URLs up to the moment the user revokes access.

This is why the Firebase rules would be ideal for this situation.

[dinvlad]

As a way around this problem, one could request a signed URL with a very short expiration time, upon user request. This way, the first time they request the URL through a cloud function, they obtain a link, but it will expire within seconds (or minutes maybe, I cannot recall what the lower limit is). Would that work for you?

[toddpi314]

@dinvlad What we could use is a method on the client SDK that has a fixed expiry on the download URL where the actual expiry term isn't defined in the user-agent. That would prevent people from just flat-out hacking the expiry value seeded into an API like the getSignedUrl and would allow us to create flash tokens within the AuthN/Firestore rule scope.

Forgive the inelegance, but something like getShortTermDownloadUrl().

The alternative currently is just to create an HTTPs Function that generates a short-term URL in the AuthN context for the Firebase user. This is a bummer, since calling getSignedUrl from the function against the Storage bucket relies on convention to enforce the security rules. That pretty much duplicates the storage AuthZ rules between the rule JSON in Storage and the functions that reference it.

[dinvlad]

IMO even having it as an option on the client would be good. For now, we just have to resort to getSignedUrl inside a CF.

[garyo]

I'm facing the same problem. My web app's users upload their private data to /users/UID/data/FILE, and I absolutely don't want there to be a public URL to that FILE -- I need to ensure that only that authenticated user can download it. A short-term public URL (seconds or a minute) would be OK.
I see @sphippen 's workaround above; I'm not sure how that would work since as far as I understand it, you can't set an IAM rule to only allow access to a subset of a bucket (users/UID in my case).

[zaquas77]

Hi @garyo,
you are not alone (like you could read in post googleapis/nodejs-storage#697 as franklyn suggest above)

The only things is that this Firebase Storage Gotchas 😅 documents helped me. I hope it can help you too!

[frankyn]

Hi @schmidt-sebastian question that's related to this. We've received requests to add getDownloadURL() to the nodejs-storage library in (googleapis/nodejs-storage#697) but this support is very specific to Storage for Firebase and not Cloud Storage API.

Has adding getDownloadURL() method been considered within the firebase-admin package? IIUC it's an alias to the nodejs-storage package so it might not be straight forward.

[nicoqh]

Adding getDownloadURL() to firebase-admin would definitely help a lot. Then we no longer need to store the download URLs in, say, Firestore in order to save the client from having to call getDownloadURL() all over the place and wait for those calls to resolve. Storing the URLs makes it a pain to revoke/regenerate tokens.

It would also allow us to use this Firebase functionality in Cloud Functions without storing tokens manually as metadata and bypassing getDownloadURL().

It doesn't solve the issue of download URLs being persistent and having no inherent security checks. But, creating a persistent, obscure URL that has no inherent security check (getDownloadURL()), is a valid use case, so I hope this functionality isn't removed.

I agree that having a "short-lived" option would be ideal, either as a new method or as an argument to getDownloadURL().

[schmidt-sebastian]

@frankyn We currently bundle the @google-cloud/storage SDK without modification. If we were to add our own getDownloadURL() method, we would have to monkey-patch your library. This poses a couple of challenges:

• We now have a "firebase-admin" version of GCS, but we don't control the actual library. Not only will we have to take on all aspects that come with maintaining a full library (documentation, support, releases), but this also creates a very strange discrepancy between these two libraries that our users will likely not understand.

• We don't have an endpoint to talk to to generate public storage URLs with Service Accounts. We have to build a completely separate request flow for this (the implications of which I am not at all familiar with).

[nVitius]

@schmidt-sebastian @frankyn
This issue hasn't seen activity in a while. Commenting to try and start conversation around this again.

Downloading files inside an application

I think the request in the original post was overlooked a bit in the discussion here.

Provide a function like [Android SDK's] getFile that checks security rules every time.

I understand the reasoning for exposing getDownloadURL as the main way to download firebase storage files from web. Downloading files fetched with XHR is definitely cumbersome and doesn't work well for large files. That said, it shouldn't it be the responsibility of the developer to decide that?

Consider this implementation:
I have an app that allows users to create PDF versions of content that they create. These PDFs are stored in Firebase storage.
The users might want to share a link to their PDF to another user of the app. I can generate a Firebase Dynamic Link that resolves back to the app with metadata for finding that file in Storage. The user logs in, then a request to download the file is made and it is previewed/downloaded. Firebase Storage could use the authenticated user's JWT claims to determine if they have access to the file.

This could definitely be implemented with getDownloadURL. The Dynamic Link would pass along the path to the file in Storage and the app would call getDownloadURL based on the path (generating the download URL does trigger Storage security rules). This does however create a problem where, in the case that I might open a PDF in another tab for previewing, the file URL along with the token is visible. A user is likely to mistakenly share that URL. Potentially granting access to someone that shouldn't have it.

Of course you could argue that they could also just download the file and share it. I think the point here though is to limit any accidental exposure that could occur.

For what it's worth, creating an authenticated XMLHttpRequest to firebasestorage.googleapis is really simple (requires a JWT token passed through an Authorization header). Developers can already create these requests; I just think it would be valuable for the JS SDK to provide endpoints to generate them.

getDownloadURL Authentication

Would it be possible to have multiple tokens associated with a single file? (The metadata field is called firebaseStorageDownloadTokens. Maybe this was originally the plan?) We already assume security for these via their being unguessable. Why not allow for many tokens to be created, each one associated with a user that has access to the file.
This could allow us to generate a specific download token for a single user, display it on a webpage with an <a> or <img src=> or whatever, and delete the token after we are done using it. (This last piece requires that we can manage the access token from the client SDK. Not currently possible).

Token Security

This is probably a separate issue, but I'll touch on it a little bit as it feels relevant.
It was mentioned earlier, but getDownloadURL could potentially enable a bad-actor to create a large amount of requests to storage and incur costs for the project owner. This doesn't even have to be someone that has access to the app, just someone that happens across a link that a user could have shared.

any user capable of downloading the file via security rules can get a download URL and do anything they want with it

Does that sound reasonable? If a user wants to download a file and upload it somewhere else, that's their business. If it's going to affect my projects billing though, I should have some level of control over who can create these and who can't.

At the very least, it would be nice if we could set either time limits or request limits (or both/none) for the tokens generated from getDownloadURL.

I suppose that it's possible to create a Cloud Function that checks files in storage and replaces the existing access-token with a new one. But that would imply that I would have to keep track of which files I want to expire somewhere. It would be great if it was an option of the Firebase Storage API.

Which sort of leads me to the last thing I wanted to talk about:

getDownloadURL in Cloud Functions

It was demonstrated in this SO post that it's possible to set the download token using the @google-cloud/storage SDK by setting the file's metadata.metadata.firebaseStorageDownloadTokens to a UUID.
I understand the difficulties in trying to patch in dedicated functions for creating these from the admin-sdk. Given how many questions there have been about doing this, I think it makes sense at least to make a note in the documentation about this method.
@frankyn I also feel that it's not so bad to have a single method for creating these in the @google-cloud/storage SDK. As long as its usage is properly documented, I don't see how it could hurt. From reading the other thread, it seemed like Storage doesn't have a 1:1 answer for getDownloadURL (though you did list several alternatives). Since firebasestorage.googleapis doesn't require any further configuration to operate on existing Storage buckets (I think), I'm sure that non-firebase customers would be happy to use this if it fit their use-case.

[nVitius]

@schmidt-sebastian @avolkovi
Just pinging you guys as you're assigned to this issue. Can we get some feedback on this please?

[avolkovi]

@tonyjhuang can you take a look?

[nVitius]

@tonyjhuang @schmidt-sebastian
Can one of you guys take a look at this issue this week? Even if it's just to say that it's on your radar.
If there's anyone else that should be looking at it instead, please let me know. Also, if there's a better place to discuss this kind of issue, we can move the conversation there.

Thanks for your help.

[tonyjhuang]

Hi @nVitius we have an idea on how to support temporary download URLs but we probably won't have resources to pursue it this quarter. I'll keep this thread updated as our timeline changes.

Related to this feature would be to change permanent download token generation to be an opt-in process so users that fetch the object with the temporary URL won't be able to scrape the permanent token from the metadata but that will have to be a separate effort as it would require changes across our SDKs.

[kevinmahrous]

Hi Everyone !
I have basic project that uploads image (as Profile Picture) in Firebase Storage under User UID (e.g users => UID => Image)
So Every thing is working fine but I need to Preview the image inside the <img src="URL of User Image" />
But it is not working with downloadURL() .. So I was thinking about Firebase Auth offer photoURL() .. So I upload image then get downloadURL() then Update User Profile for then New Link (As Firebase Storage downloadURL()) But it is not working with this way .. I searched for a lot of Answers on (StackOverFlow .. Youtube .. Github) but no success .. I will put my code below to help me find answer .. I will be apprecipated and thankful

firebase.storage().ref("users").child(name + "/UserPic/").put(file, metadata).then(function(snapshot) {
    var url = snapshot.firebase.storage().ref("users").child(name + "/UserPic/").downloadURL;
    user.updateProfile({
        photoURL: url
    });
}).catch((error) => {
    var errorMessage = error.message;
    swal.fire({
        icon: 'error',
        title: 'Error',
        text: errorMessage,
    });
});

Thanks .. Sorry for a lot of content and text :)

[nVitius]

@kevinmmansour The way I've done this before is to set the access rules to the avatars as public. Then you don't need to use the downloadURL. If you must maintain the permissions, then you can download the image using XHR and dynamically set the src of your <img> to a data-url.

I agree though, that it would be great to see the addition of downloadURLs that don't use query parameters for the token.

For what it's worth, you can also generate your own UUID and set it as the filename and use public permissions. This provides essentially the same security that the getDownloadURL method does. If you do this, be sure to allow only get operations and not list operations on the directory that you store the images in. In Firebase Storage security rules, you can set permissions for read, which is both get and list, or you can set them individually.

[nVitius]

@tonyjhuang I appreciate your reply. I know this was a while ago, but I wanted to add a couple comments:

I know that the Firebase team supports a lot of projects and has their own internal roadmap. I understand that some things take priority over others. That said, the developers on here asking for information about feature development and offering ideas on how they would like things to be solved are not just your users but also your paying customers.

I personally love working with Firebase and I recommend it as a solution to most of my clients and colleagues. But It can be frustrating not having much (often none at all) insight into what the team is working on and if certain issues/features are at least being considered.

I imagine that this is most likely not something that is up to you. I hope at least that someone sees this and can raise the concerns with the internal team and there can be more discussion around this.

[kevinmahrous]

@nVitius Really appreciate your reply. I just solved it by Uploading the photo to Firebase Storage then get download url then store it inside Firebase Auth (photoURL) to solve that but it really worked after finding answer on google. Thanks again for your reply as you give some time for my comment to read it . I hope Firebase Team create more simplest method to do that with more features which will help a lot of developers to finish their project.

[LeadDreamer]

Client-Side, Security-Rule-Compliant Firebase Storage Fetch of IMages

It is possible, and relatively straight-forward, to download a Firebase Storage object to client while fully respecting Security Rules - but, as noticed above, it is NOT possible directly from an tag. After sleuthing through network logs, I saw the metadata fetch (well, storage object fetch), and poked through this repository to discover that .getDownloadURL() first calls a storage URL to get the "file object" - and uses a JWT token in an "authorization" header to do it - and this header is why tags can't play this game (they don't allow extra headers). They do allow for CORS "use-credentials" - but Firebase doesn't allow for these, yet (they should, see issue #5373 ). In the meantime, you can create a short-lived (about an hour) usable URL with the following code:

/**
 * ----------------------------------------------------------------------
 * @function
 * @static
 * This Utility checks if URL is Secured by Storage Security Rules
 * If the URL carries an access token, it is simply returned.
 * If the URL does NOT have an access token, it is fetched to local stoage
 * with authentication, and a new URL created to point to the local
 * copy.
 * NOTE: URL MUST have an ?alt=media parameter; if this is omitted the
 * "FileObject" metadata is returned instead
 * @param {string} URL
 * @returns {Promise<URL>}
 * @fulfil {string}  a local URL to access the downloaded data
 * @reject {string}
 */
export const accessibleStorage = async (thisURL) => {
  //detect if already a public-access token present
  if (thisURL.includes("token=") || !thisURL.startsWith("https://firebase"))
    return thisURL;

  //actually a call to auth.getIdToken(), see below
  const authorization = await fetchJWT();

  //add the formatted JWT token as an "authorization" header
  let headers = {};
  if (authorization !== null && authorization.length > 0) {
    headers["Authorization"] = "Firebase " + authorization;
  }
  const options = {
    headers: headers
  };
  //fetch the object
  const res = await fetch(thisURL, options);
  // if the fetch has any difficulty just return null
  if (res.status !== 200) return null;

  //fetch the contents as a blob
  const blob = await res.blob();

  //if it fails also return null
  if (!blob) return null;

  //create a URL that points to the downloaded blob
  //return this URL for use in <img>, background-image and other image applications.
  return URL.createObjectURL(blob);
};

/**
 * @async
 * @static
 * Fetch a JWT token for authenticated signed requests
 * @param {FirebaseAuthUser} user
 * @returns {Promise<JWT>}
 * @fulfil Returnsa JWT token
 * @reject returns an err
 */

 export const fetchJWT = async (user) => {
  const thisUser = user || FirebaseAuth.currentUser;
  //the "true" below forces a reset
  const JWT = await thisUser.getIdToken(true);
  return JWT;

 }

You can use the following to generate the URL to a known storage object:

const bucket_domain = "https://firebasestorage.googleapis.com/v0/b/";
const bucket_head = "/o/";
let bucket_name; // this can be retrieved from your Firebase Firestore and Storage config object, config.storageBucket

export const getPrivateURL = async (ref) => {
  return makePrivateURLFromPath(ref.fullPath)
}

/**
 * ----------------------------------------------------------------------
 * @function
 * @static
 * This function is part of a storage scheme that uses parallel structures
 * between Firestore collection/documents and Storage paths.  The concept
 * here is all Storage items are part of/belong to Firestore documents.
 * This function takes a full path to a Storage object and turns it into
 * URL.  If "type"is not included, the URL will return the metadata, not
 * the contents.
 * Note this simply makes the URL - it does not carry out *any* operations
 * @param {!string} fullPath required path to the stored item.
 * @returns {string} constructed Security-Rule-compliant URL
 */
export const makePrivateURLFromPath = (fullPath) => {
  // note: ?alt-media parameter is just to return the FILE, not
  // the meta-data object
  //the "/" in the path have to be escaped to "%2F"
  const localFfullPath = fullPath.replace(/\//g, "%2F");

  const privateURL = `${bucket_domain}${bucket_name}${bucket_head}${localFullPath}?alt=media`;
  return privateURL;
};