Add quota to cache API

https://bugs.webkit.org/show_bug.cgi?id=177552

Patch by Youenn Fablet <[email protected]> on 2017-10-09
Reviewed by Alex Christensen.

Source/WebCore:

Tests: http/wpt/cache-storage/cache-quota.any.html

Storing padded opaque response body sizes within FetchResponse and CacheStorageConnection.
See whatwg/storage#31 for the rationale about this padding.
Storing in CacheStorageConnection is needed for handling cloned network fetched created responses.
Storing in FetchResponse is needed for handling cloned cache-storage created opaque responses.

Adding internals to query and set the fuzzed size of a response.

* Modules/cache/CacheStorageConnection.cpp:
(WebCore::computeRealBodySize):
(WebCore::CacheStorageConnection::computeRecordBodySize):
(WebCore::CacheStorageConnection::setResponseBodySizeWithPadding):
(WebCore::CacheStorageConnection::responseBodySizeWithPadding const):
* Modules/cache/CacheStorageConnection.h:
* Modules/cache/DOMCache.cpp:
(WebCore::DOMCache::toConnectionRecord):
(WebCore::DOMCache::updateRecords):
* Modules/cache/DOMCache.h:
* Modules/cache/DOMCacheEngine.cpp:
(WebCore::DOMCacheEngine::errorToException):
(WebCore::DOMCacheEngine::Record::copy const):
* Modules/cache/DOMCacheEngine.h:
* Modules/cache/WorkerCacheStorageConnection.cpp:
(WebCore::toCrossThreadRecordData):
(WebCore::fromCrossThreadRecordData):
* Modules/fetch/FetchResponse.cpp:
(WebCore::FetchResponse::clone):
(WebCore::FetchResponse::BodyLoader::didReceiveResponse):
* Modules/fetch/FetchResponse.h:
* Modules/fetch/FetchResponse.idl:
* testing/Internals.cpp:
(WebCore::Internals::setResponseSizeWithPadding):
(WebCore::Internals::responseSizeWithPadding const):
* testing/Internals.h:
* testing/Internals.idl:

Source/WebKit:

Adding support for quota checking in CacheStorage::Caches.
It is passed to NetworkProcess at creation time.
Default quota size is configured to 400Ko by origin per default.
This value is suitable for testing.
Future patch should raise this default value and allows configuring it.

Quota is computed based on the response body size.
This size is padded at WebCore for opaque responses.
Size is stored persistently as opaque response padded size should remain stable.
See whatwg/storage#31 for the rationale about this padding.

In case of putting several records at the same time, the size of all records
is computed so that all records will be written or rejected together.

Sending QuotaExceeded error when quota is exceeded.
Future effort should allow asking UIProcess for quota extension.

* NetworkProcess/NetworkProcess.cpp:
(WebKit::NetworkProcess::cacheStoragePerOriginQuota const):
* NetworkProcess/NetworkProcess.h:
* NetworkProcess/NetworkProcessCreationParameters.cpp:
(WebKit::NetworkProcessCreationParameters::encode const):
(WebKit::NetworkProcessCreationParameters::decode):
* NetworkProcess/NetworkProcessCreationParameters.h:
* NetworkProcess/cache/CacheStorageEngine.cpp:
(WebKit::CacheStorage::Engine::readCachesFromDisk):
* NetworkProcess/cache/CacheStorageEngineCache.cpp:
(WebKit::CacheStorage::Cache::toRecordInformation):
(WebKit::CacheStorage::isolatedCopy):
(WebKit::CacheStorage::Cache::open):
(WebKit::CacheStorage::Cache::storeRecords):
(WebKit::CacheStorage::Cache::put):
(WebKit::CacheStorage::Cache::writeRecordToDisk):
(WebKit::CacheStorage::Cache::updateRecordToDisk):
(WebKit::CacheStorage::Cache::removeRecordFromDisk):
(WebKit::CacheStorage::Cache::encode):
(WebKit::CacheStorage::Cache::decodeRecordHeader):
(WebKit::CacheStorage::Cache::decode):
* NetworkProcess/cache/CacheStorageEngineCache.h:
* NetworkProcess/cache/CacheStorageEngineCaches.cpp:
(WebKit::CacheStorage::Caches::Caches):
(WebKit::CacheStorage::Caches::initialize):
(WebKit::CacheStorage::Caches::initializeSize):
(WebKit::CacheStorage::Caches::requestSpace):
(WebKit::CacheStorage::Caches::writeRecord):
(WebKit::CacheStorage::Caches::removeRecord):
(WebKit::CacheStorage::Caches::removeCacheEntry):
* NetworkProcess/cache/CacheStorageEngineCaches.h:
(WebKit::CacheStorage::Caches::create):
(WebKit::CacheStorage::Caches::hasEnoughSpace const):
* NetworkProcess/cache/NetworkCacheStorage.cpp:
(WebKit::NetworkCache::Storage::traverse):
* NetworkProcess/cocoa/NetworkProcessCocoa.mm:
(WebKit::NetworkProcess::platformInitializeNetworkProcessCocoa):
* Shared/WebCoreArgumentCoders.cpp:
(IPC::ArgumentCoder<DOMCacheEngine::Record>::encode):
(IPC::ArgumentCoder<DOMCacheEngine::Record>::decode):
* UIProcess/API/APIProcessPoolConfiguration.cpp:
(API::ProcessPoolConfiguration::copy):
* UIProcess/API/APIProcessPoolConfiguration.h:
* UIProcess/WebProcessPool.cpp:
(WebKit::WebProcessPool::ensureNetworkProcess):

LayoutTests:

* http/wpt/cache-storage/cache-quota.https.any-expected.txt: Added.
* http/wpt/cache-storage/cache-quota.https.any.html: Added.
* http/wpt/cache-storage/cache-quota.https.any.js: Added.

git-svn-id: http://svn.webkit.org/repository/webkit/trunk@223073 268f45cc-cd09-0410-ab3c-d52691b4dbfc

LayoutTests/ChangeLog

@@ -1,3 +1,14 @@
+2017-10-09  Youenn Fablet  <[email protected]>
+
+        Add quota to cache API
+        https://bugs.webkit.org/show_bug.cgi?id=177552
+
+        Reviewed by Alex Christensen.
+
+        * http/wpt/cache-storage/cache-quota.https.any-expected.txt: Added.
+        * http/wpt/cache-storage/cache-quota.https.any.html: Added.
+        * http/wpt/cache-storage/cache-quota.https.any.js: Added.
+
 2017-10-09  Matt Lewis  <[email protected]>
 
         Unskipped http/tests/cache/disk-cache/disk-cache-validation-no-body.html

LayoutTests/http/wpt/cache-storage/cache-quota.any-expected.txt

@@ -0,0 +1,7 @@
+
+PASS Testing synthetic response body size padding 
+PASS Testing non opaque response body size padding 
+PASS Testing opaque response body size padding 
+PASS Hitting cache quota for non opaque responses 
+PASS Hitting cache quota for padded responses 
+

LayoutTests/http/wpt/cache-storage/cache-quota.any.html

@@ -0,0 +1 @@
+<!-- This file is required for WebKit test infrastructure to run the templated test -->

LayoutTests/http/wpt/cache-storage/cache-quota.any.js

@@ -0,0 +1,169 @@
+// META: script=/service-workers/cache-storage/resources/test-helpers.js
+// META: script=/common/get-host-info.sub.js
+
+var test_url = 'https://example.com/foo';
+var test_body = 'Hello world!';
+
+function getResponseBodySizeWithPadding(response)
+{
+    var cache;
+    var request = new Request("temp");
+    var clone = response.clone();
+    return self.caches.open("test").then((c) => {
+        cache = c;
+        return cache.put(request, clone);
+    }).then(() => {
+        return self.caches.delete("temp");
+    }).then(() => {
+        return window.internals.responseSizeWithPadding(clone);
+    });
+}
+
+promise_test(() => {
+    if (!window.internals)
+        return Promise.reject("Test requires internals");
+
+    var response = new Response("");
+    return getResponseBodySizeWithPadding(response).then(size => {
+        assert_equals(size, 0, "zero size synthetic response");
+        return getResponseBodySizeWithPadding(response.clone());
+    }).then((size) => {
+        assert_equals(size, 0, "zero size synthetic cloned response");
+
+        response = new Response("a");
+        return getResponseBodySizeWithPadding(response);
+    }).then((size) => {
+        assert_equals(size, 1, "non zero size synthetic response");
+        return getResponseBodySizeWithPadding(response.clone());
+    }).then((size) => {
+        assert_equals(size, 1, "non zero size synthetic cloned response");
+    })
+}, "Testing synthetic response body size padding");
+
+promise_test(() => {
+    if (!window.internals)
+        return Promise.reject("Test requires internals");
+
+    var paddedSize;
+    var response, responseClone;
+    return fetch("").then(r => {
+        response = r;
+        responseClone = response.clone();
+        return getResponseBodySizeWithPadding(response);
+    }).then((size) => {
+        paddedSize = size;
+        return response.arrayBuffer();
+    }).then((buffer) => {
+        assert_equals(buffer.byteLength, paddedSize, "non opaque network response");
+        return getResponseBodySizeWithPadding(responseClone);
+    }).then((size) => {
+        assert_equals(size, paddedSize, "non opaque network cloned response");
+    });
+}, "Testing non opaque response body size padding");
+
+promise_test(() => {
+    if (!window.internals)
+        return Promise.reject("Test requires internals");
+
+    var actualSize, paddedSize;
+    var response, responseClone1;
+    return fetch(get_host_info().HTTP_REMOTE_ORIGIN, {mode: "no-cors"}).then(r => {
+        response = r;
+    }).then(() => {
+        return fetch(get_host_info().HTTP_ORIGIN);
+    }).then(r => {
+        return r.arrayBuffer();
+    }).then((buffer) => {
+        actualSize = buffer.byteLength;
+    }).then(() => {
+        responseClone1 = response.clone();
+        return getResponseBodySizeWithPadding(response);
+    }).then((size) => {
+        paddedSize = size;
+        return getResponseBodySizeWithPadding(responseClone1);
+    }).then((size) => {
+        assert_not_equals(size, actualSize, "padded size should be different from actual size");
+        assert_equals(size, paddedSize, "opaque network cloned response");
+    });
+}, "Testing opaque response body size padding");
+
+async function doCleanup()
+{
+    var cachesKeys = await self.caches.keys();
+    for (let name of cachesKeys) {
+        let cache = await self.caches.open(name);
+        let keys = await cache.keys();
+        for (let key of keys)
+            await cache.delete(key);
+    }
+}
+
+promise_test((test) => {
+    var cache;
+    var response1ko = new Response(new ArrayBuffer(1 * 1024));
+    var response399ko = new Response(new ArrayBuffer(399 * 1024));
+
+    return doCleanup().then(() => {
+        return self.caches.open("temp1");
+    }).then((c) => {
+        cache = c;
+        return cache.put("399ko", response399ko.clone());
+    }).then(() => {
+        return cache.put("1ko-v1", response1ko.clone());
+    }).then(() => {
+        return cache.put("1ko-v2", response1ko.clone()).then(assert_unreached, (e) => {
+            assert_equals(e.name, "QuotaExceededError");
+        });
+   }).then(() => {
+        return cache.delete("1ko-v1");
+    }).then(() => {
+        return cache.put("1ko-v2", response1ko.clone());
+    }).then(() => {
+        return cache.delete("399ko");
+    }).then(() => {
+        return cache.delete("1ko-v1");
+    }).then(() => {
+        return cache.delete("1ko-v2");
+    });
+}, 'Hitting cache quota for non opaque responses');
+
+promise_test((test) => {
+    if (!window.internals)
+        return Promise.reject("Test requires internals");
+
+    var cache;
+    var response1ko = new Response(new ArrayBuffer(1 * 1024));
+    var responsePadded = new Response(new ArrayBuffer(1 * 1024));
+    var response200ko = new Response(new ArrayBuffer(200 * 1024));
+
+    return doCleanup().then(() => {
+        return self.caches.open("temp2");
+    }).then((c) => {
+        cache = c;
+        return fetch(get_host_info().HTTP_REMOTE_ORIGIN, {mode: "no-cors"});
+    }).then((r) => {
+        responsePadded = r;
+        internals.setResponseSizeWithPadding(responsePadded, 200 * 1024);
+        return cache.put("200ko", response200ko.clone());
+    }).then(() => {
+        return cache.put("1ko-padded-to-200ko", responsePadded.clone());
+    }).then(() => {
+        return cache.put("1ko", response1ko.clone()).then(assert_unreached, (e) => {
+            assert_equals(e.name, "QuotaExceededError");
+        });
+   }).then(() => {
+        return cache.delete("1ko-padded-to-200ko");
+    }).then(() => {
+        return cache.put("1ko-v2", response1ko.clone());
+    }).then(() => {
+        return cache.put("1ko-v3", response1ko.clone());
+    }).then(() => {
+        return cache.delete("200ko");
+    }).then(() => {
+        return cache.delete("1ko-v2");
+    }).then(() => {
+        return cache.delete("1ko-v3");
+    });
+}, 'Hitting cache quota for padded responses');
+
+done();

Source/WebCore/ChangeLog

@@ -1,3 +1,47 @@
+2017-10-09  Youenn Fablet  <[email protected]>
+
+        Add quota to cache API
+        https://bugs.webkit.org/show_bug.cgi?id=177552
+
+        Reviewed by Alex Christensen.
+
+        Tests: http/wpt/cache-storage/cache-quota.any.html
+
+        Storing padded opaque response body sizes within FetchResponse and CacheStorageConnection.
+        See https://github.com/whatwg/storage/issues/31 for the rationale about this padding.
+        Storing in CacheStorageConnection is needed for handling cloned network fetched created responses.
+        Storing in FetchResponse is needed for handling cloned cache-storage created opaque responses.
+
+        Adding internals to query and set the fuzzed size of a response.
+
+        * Modules/cache/CacheStorageConnection.cpp:
+        (WebCore::computeRealBodySize):
+        (WebCore::CacheStorageConnection::computeRecordBodySize):
+        (WebCore::CacheStorageConnection::setResponseBodySizeWithPadding):
+        (WebCore::CacheStorageConnection::responseBodySizeWithPadding const):
+        * Modules/cache/CacheStorageConnection.h:
+        * Modules/cache/DOMCache.cpp:
+        (WebCore::DOMCache::toConnectionRecord):
+        (WebCore::DOMCache::updateRecords):
+        * Modules/cache/DOMCache.h:
+        * Modules/cache/DOMCacheEngine.cpp:
+        (WebCore::DOMCacheEngine::errorToException):
+        (WebCore::DOMCacheEngine::Record::copy const):
+        * Modules/cache/DOMCacheEngine.h:
+        * Modules/cache/WorkerCacheStorageConnection.cpp:
+        (WebCore::toCrossThreadRecordData):
+        (WebCore::fromCrossThreadRecordData):
+        * Modules/fetch/FetchResponse.cpp:
+        (WebCore::FetchResponse::clone):
+        (WebCore::FetchResponse::BodyLoader::didReceiveResponse):
+        * Modules/fetch/FetchResponse.h:
+        * Modules/fetch/FetchResponse.idl:
+        * testing/Internals.cpp:
+        (WebCore::Internals::setResponseSizeWithPadding):
+        (WebCore::Internals::responseSizeWithPadding const):
+        * testing/Internals.h:
+        * testing/Internals.idl:
+
 2017-10-09  Zalan Bujtas  <[email protected]>
 
         Remove redundant RenderObject::virtualContinuation

Source/WebCore/Modules/cache/CacheStorageConnection.cpp

@@ -27,6 +27,9 @@
 #include "config.h"
 #include "CacheStorageConnection.h"
 
+#include "FetchResponse.h"
+#include <wtf/RandomNumber.h>
+
 using namespace WebCore::DOMCacheEngine;
 
 namespace WebCore {
@@ -71,6 +74,37 @@ void CacheStorageConnection::batchDeleteOperation(uint64_t cacheIdentifier, cons
     doBatchDeleteOperation(requestIdentifier, cacheIdentifier, request, WTFMove(options));
 }
 
+static inline uint64_t computeRealBodySize(const DOMCacheEngine::ResponseBody& body)
+{
+    uint64_t result = 0;
+    WTF::switchOn(body, [&] (const Ref<WebCore::FormData>& formData) {
+        result = formData->lengthInBytes();
+    }, [&] (const Ref<WebCore::SharedBuffer>& buffer) {
+        result = buffer->size();
+    }, [] (const std::nullptr_t&) {
+    });
+    return result;
+}
+
+uint64_t CacheStorageConnection::computeRecordBodySize(const FetchResponse& response, const DOMCacheEngine::ResponseBody& body, ResourceResponse::Tainting tainting)
+{
+    if (!response.opaqueLoadIdentifier()) {
+        ASSERT_UNUSED(tainting, tainting != ResourceResponse::Tainting::Opaque);
+        return computeRealBodySize(body);
+    }
+
+    return m_opaqueResponseToSizeWithPaddingMap.ensure(response.opaqueLoadIdentifier(), [&] () {
+        uint64_t realSize = computeRealBodySize(body);
+
+        // Padding the size as per https://github.com/whatwg/storage/issues/31.
+        uint64_t sizeWithPadding = realSize + static_cast<uint64_t>(randomNumber() * 128000);
+        sizeWithPadding = ((sizeWithPadding / 32000) + 1) * 32000;
+
+        m_opaqueResponseToSizeWithPaddingMap.set(response.opaqueLoadIdentifier(), sizeWithPadding);
+        return sizeWithPadding;
+    }).iterator->value;
+}
+
 void CacheStorageConnection::batchPutOperation(uint64_t cacheIdentifier, Vector<Record>&& records, RecordIdentifiersCallback&& callback)
 {
     uint64_t requestIdentifier = ++m_lastRequestIdentifier;

Source/WebCore/Modules/cache/CacheStorageConnection.h

@@ -32,6 +32,8 @@
 
 namespace WebCore {
 
+class FetchResponse;
+
 class CacheStorageConnection : public ThreadSafeRefCounted<CacheStorageConnection> {
 public:
     static Ref<CacheStorageConnection> create() { return adoptRef(*new CacheStorageConnection()); }
@@ -44,6 +46,7 @@ class CacheStorageConnection : public ThreadSafeRefCounted<CacheStorageConnectio
     void retrieveRecords(uint64_t cacheIdentifier, const URL&, DOMCacheEngine::RecordsCallback&&);
     void batchDeleteOperation(uint64_t cacheIdentifier, const ResourceRequest&, CacheQueryOptions&&, DOMCacheEngine::RecordIdentifiersCallback&&);
     void batchPutOperation(uint64_t cacheIdentifier, Vector<DOMCacheEngine::Record>&&, DOMCacheEngine::RecordIdentifiersCallback&&);
+    uint64_t computeRecordBodySize(const FetchResponse&, const DOMCacheEngine::ResponseBody&, ResourceResponse::Tainting);
 
     virtual void reference(uint64_t /* cacheIdentifier */) { }
     virtual void dereference(uint64_t /* cacheIdentifier */) { }
@@ -78,6 +81,7 @@ class CacheStorageConnection : public ThreadSafeRefCounted<CacheStorageConnectio
     HashMap<uint64_t, DOMCacheEngine::CacheInfosCallback> m_retrieveCachesPendingRequests;
     HashMap<uint64_t, DOMCacheEngine::RecordsCallback> m_retrieveRecordsPendingRequests;
     HashMap<uint64_t, DOMCacheEngine::RecordIdentifiersCallback> m_batchDeleteAndPutPendingRequests;
+    HashMap<uint64_t, uint64_t> m_opaqueResponseToSizeWithPaddingMap;
 
     uint64_t m_lastRequestIdentifier { 0 };
 };

Source/WebCore/Modules/cache/DOMCache.cpp

@@ -38,8 +38,6 @@ using namespace WebCore::DOMCacheEngine;
 
 namespace WebCore {
 
-static Record toConnectionRecord(const FetchRequest&, FetchResponse&, ResponseBody&&);
-
 DOMCache::DOMCache(ScriptExecutionContext& context, String&& name, uint64_t identifier, Ref<CacheStorageConnection>&& connection)
     : ActiveDOMObject(&context)
     , m_name(WTFMove(name))
@@ -480,7 +478,7 @@ void DOMCache::batchDeleteOperation(const FetchRequest& request, CacheQueryOptio
     });
 }
 
-Record toConnectionRecord(const FetchRequest& request, FetchResponse& response, DOMCacheEngine::ResponseBody&& responseBody)
+Record DOMCache::toConnectionRecord(const FetchRequest& request, FetchResponse& response, DOMCacheEngine::ResponseBody&& responseBody)
 {
     // FIXME: Add a setHTTPHeaderFields on ResourceResponseBase.
     ResourceResponse cachedResponse = response.resourceResponse();
@@ -493,9 +491,15 @@ Record toConnectionRecord(const FetchRequest& request, FetchResponse& response,
     ASSERT(!cachedRequest.isNull());
     ASSERT(!cachedResponse.isNull());
 
+    auto sizeWithPadding = response.bodySizeWithPadding();
+    if (!sizeWithPadding) {
+        sizeWithPadding = m_connection->computeRecordBodySize(response, responseBody, cachedResponse.tainting());
+        response.setBodySizeWithPadding(sizeWithPadding);
+    }
+
     return { 0, 0,
         request.headers().guard(), WTFMove(cachedRequest), request.fetchOptions(), request.internalRequestReferrer(),
-        response.headers().guard(), WTFMove(cachedResponse), WTFMove(responseBody)
+        response.headers().guard(), WTFMove(cachedResponse), WTFMove(responseBody), sizeWithPadding
     };
 }
 
@@ -533,7 +537,7 @@ void DOMCache::updateRecords(Vector<Record>&& records)
             if (current.updateResponseCounter != record.updateResponseCounter) {
                 auto responseHeaders = FetchHeaders::create(record.responseHeadersGuard, HTTPHeaderMap { record.response.httpHeaderFields() });
                 auto response = FetchResponse::create(*scriptExecutionContext(), std::nullopt, WTFMove(responseHeaders), WTFMove(record.response));
-                response->setBodyData(WTFMove(record.responseBody));
+                response->setBodyData(WTFMove(record.responseBody), record.responseBodySize);
 
                 current.response = WTFMove(response);
                 current.updateResponseCounter = record.updateResponseCounter;
@@ -545,7 +549,7 @@ void DOMCache::updateRecords(Vector<Record>&& records)
 
             auto responseHeaders = FetchHeaders::create(record.responseHeadersGuard, HTTPHeaderMap { record.response.httpHeaderFields() });
             auto response = FetchResponse::create(*scriptExecutionContext(), std::nullopt, WTFMove(responseHeaders), WTFMove(record.response));
-            response->setBodyData(WTFMove(record.responseBody));
+            response->setBodyData(WTFMove(record.responseBody), record.responseBodySize);
 
             newRecords.append(CacheStorageRecord { record.identifier, record.updateResponseCounter, WTFMove(request), WTFMove(response) });
         }

Source/WebCore/Modules/cache/DOMCache.h

@@ -80,6 +80,7 @@ class DOMCache final : public RefCounted<DOMCache>, public ActiveDOMObject {
 
     void updateRecords(Vector<DOMCacheEngine::Record>&&);
     Vector<Ref<FetchResponse>> cloneResponses(const Vector<CacheStorageRecord>&);
+    DOMCacheEngine::Record toConnectionRecord(const FetchRequest&, FetchResponse&, DOMCacheEngine::ResponseBody&&);
 
     String m_name;
     uint64_t m_identifier;

Source/WebCore/Modules/cache/DOMCacheEngine.cpp

@@ -44,6 +44,8 @@ Exception errorToException(Error error)
         return Exception { TypeError, ASCIILiteral("Failed reading data from the file system") };
     case Error::WriteDisk:
         return Exception { TypeError, ASCIILiteral("Failed writing data to the file system") };
+    case Error::QuotaExceeded:
+        return Exception { QuotaExceededError, ASCIILiteral("Quota exceeded") };
     default:
         return Exception { TypeError, ASCIILiteral("Internal error") };
     }
@@ -135,7 +137,7 @@ ResponseBody copyResponseBody(const ResponseBody& body)
 
 Record Record::copy() const
 {
-    return Record { identifier, updateResponseCounter, requestHeadersGuard, request, options, referrer, responseHeadersGuard, response, copyResponseBody(responseBody) };
+    return Record { identifier, updateResponseCounter, requestHeadersGuard, request, options, referrer, responseHeadersGuard, response, copyResponseBody(responseBody), responseBodySize };
 }
 
 static inline CacheInfo isolateCacheInfo(const CacheInfo& info)