Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ambient cache initialization speed improvement #1959

Open
wants to merge 27 commits into
base: main
Choose a base branch
from
Open

Ambient cache initialization speed improvement #1959

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
07f7fa4
Update MBXViewController.m
Oct 31, 2023
a878d90
Revert "Update MBXViewController.m"
Nov 13, 2023
b7a4bcf
New tables created for ambient cache + first version of putTile / put…
Nov 20, 2023
35ba68d
Moving data from ambient_resources/ambient_tiles table to resources/t…
Nov 23, 2023
864837f
Indexes added on accessed field in ambient_tiles and ambient_resource…
Nov 23, 2023
e0bc77f
Indexes added on accessed field in ambient_tiles and ambient_resource…
Nov 23, 2023
5c75a7b
Fix for EXC_BAD_ACCESS crash when downloading maps - code is not clea…
Dec 6, 2023
2085fcb
Code refactoring for hasTile and hasResource methods to make the code…
Dec 6, 2023
cb8cf11
Refactoring of getTile and getResouce methods to avoid using lambdas …
Dec 7, 2023
7758093
Update offline_database.test.cpp
Dec 12, 2023
67767ac
Create v6.db
Dec 12, 2023
4307dc9
Resources and tiles which are not used in any offline region are now …
Dec 13, 2023
9d0f1d9
Merge branch 'maplibre:main' into fix/ambient-cache-slowness
Dec 13, 2023
d0e6330
Re-enabled tests which were temporarily disabled
Dec 14, 2023
2857360
Databases merging (sideloading) feature now supports databases with n…
Dec 14, 2023
14716aa
Test databases used for database merging (sideloading) upgraded to ne…
Dec 14, 2023
b3aaeae
Merge branch 'maplibre:main' into fix/ambient-cache-slowness
Dec 14, 2023
67faa0b
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 14, 2023
a21ee01
Removed unused variable
Dec 14, 2023
d948995
Removed debug log
Dec 19, 2023
f4838dd
Update offline_database.test.cpp
Feb 16, 2024
16f3390
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Feb 16, 2024
4a1a8fd
Merge branch 'maplibre:main' into fix/ambient-cache-slowness
Feb 16, 2024
dcd4d00
Merge branch 'maplibre:main' into fix/ambient-cache-slowness
Feb 29, 2024
760cb5e
Update offline_database.cpp
Feb 29, 2024
ba243f5
Merge branch 'fix/ambient-cache-slowness' of https://github.com/geoli…
Feb 29, 2024
a9e4fd3
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Feb 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 12 additions & 3 deletions platform/default/include/mbgl/storage/offline_database.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,7 @@ class OfflineDatabase {
void migrateToVersion5();
void migrateToVersion3();
void migrateToVersion6();
void migrateToVersion7();
void cleanup();
bool disabled();
void vacuum();
Expand All @@ -118,18 +119,24 @@ class OfflineDatabase {
mapbox::sqlite::Statement& getStatement(const char*);

std::optional<std::pair<Response, uint64_t>> getTile(const Resource::TileData&);
void updateTileTimestamp(const Resource::TileData&, const char*);
std::optional<std::pair<Response, uint64_t>> extractTileData(const Resource::TileData&, const char*);
std::optional<int64_t> hasTile(const Resource::TileData&);
bool putTile(const Resource::TileData&, const Response&, const std::string&, bool compressed);
std::optional<int64_t> extractTileDataSize(const Resource::TileData&, const char*);
bool putTile(const Resource::TileData&, const Response&, const std::string&, bool compressed, bool ambient);

std::optional<std::pair<Response, uint64_t>> getResource(const Resource&);
void updateResourceTimestamp(const Resource&, const char*);
std::optional<std::pair<Response, uint64_t>> extractResourceData(const Resource&, const char*);
std::optional<int64_t> hasResource(const Resource&);
bool putResource(const Resource&, const Response&, const std::string&, bool compressed);
std::optional<int64_t> extractResourceDataSize(const Resource&, const char*);
bool putResource(const Resource&, const Response&, const std::string&, bool compressed, bool ambient);

uint64_t putRegionResourceInternal(int64_t regionID, const Resource&, const Response&);

std::optional<std::pair<Response, uint64_t>> getInternal(const Resource&);
std::optional<int64_t> hasInternal(const Resource&);
std::pair<bool, uint64_t> putInternal(const Resource&, const Response&, bool evict);
std::pair<bool, uint64_t> putInternal(const Resource&, const Response&, bool evict, bool ambient);

// Return value is true iff the resource was previously unused by any other regions.
bool markUsed(int64_t regionID, const Resource&);
Expand All @@ -151,6 +158,8 @@ class OfflineDatabase {

bool evict(uint64_t neededFreeSize, DatabaseSizeChangeStats& stats);

std::exception_ptr clearUnusedResourcesAndTiles();

TileServerOptions tileServerOptions;

class DatabaseSizeChangeStats {
Expand Down
33 changes: 33 additions & 0 deletions platform/default/include/mbgl/storage/offline_schema.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,35 @@ static constexpr const char* offlineDatabaseSchema =
" must_revalidate INTEGER NOT NULL DEFAULT 0,\n"
" UNIQUE (url_template, pixel_ratio, z, x, y)\n"
");\n"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file is generated and should not be modified manually.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you try updating offline_schema.js and running that?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I didn't know that the file was autogenerated, sorry, I will try to do that :-)

"CREATE TABLE ambient_resources (\n"
" id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,\n"
" url TEXT NOT NULL,\n"
" kind INTEGER NOT NULL,\n"
" expires INTEGER,\n"
" modified INTEGER,\n"
" etag TEXT,\n"
" data BLOB,\n"
" compressed INTEGER NOT NULL DEFAULT 0,\n"
" accessed INTEGER NOT NULL,\n"
" must_revalidate INTEGER NOT NULL DEFAULT 0,\n"
" UNIQUE (url)\n"
");\n"
"CREATE TABLE ambient_tiles (\n"
" id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,\n"
" url_template TEXT NOT NULL,\n"
" pixel_ratio INTEGER NOT NULL,\n"
" z INTEGER NOT NULL,\n"
" x INTEGER NOT NULL,\n"
" y INTEGER NOT NULL,\n"
" expires INTEGER,\n"
" modified INTEGER,\n"
" etag TEXT,\n"
" data BLOB,\n"
" compressed INTEGER NOT NULL DEFAULT 0,\n"
" accessed INTEGER NOT NULL,\n"
" must_revalidate INTEGER NOT NULL DEFAULT 0,\n"
" UNIQUE (url_template, pixel_ratio, z, x, y)\n"
");\n"
"CREATE TABLE regions (\n"
" id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,\n"
" definition TEXT NOT NULL,\n"
Expand All @@ -54,6 +83,10 @@ static constexpr const char* offlineDatabaseSchema =
"ON resources (accessed);\n"
"CREATE INDEX tiles_accessed\n"
"ON tiles (accessed);\n"
"CREATE INDEX ambient_resources_accessed\n"
"ON ambient_resources (accessed);\n"
"CREATE INDEX ambient_tiles_accessed\n"
"ON ambient_tiles (accessed);\n"
"CREATE INDEX region_resources_resource_id\n"
"ON region_resources (resource_id);\n"
"CREATE INDEX region_tiles_tile_id\n"
Expand Down
Loading
Loading