Raster Tools > Generate XYZ tiles (MBTiles) - problems in 3.34.1 (new problems since 3.28.12)

[caver456]

What is the bug or the crash?

(none of these issues existed in LTR 3.28.12 - I believe they were introduced during fixes for #50170 - however, thanks very much for fixing that core issue!)

1. this warning shows up (in orange) on every run, even when PNG is the selected file type: "Background color setting ignored, the JPG format only supports fully opaque colors"
2. aborted run due to failure to overwrite an existing mbtiles file (and no prompt asking whether to overwrite) on subsequent runs writing to the same file, even though the first run does prompt asking to overwrite the existing file. In 3.28, all subsequent runs to the same file either prompted to overwrite at the start of each run, or, remembered the answer to the prompt from the first run and applied it each time. So - not sure if this is just a failure to prompt to overwrite on each run, or, something else. The error shows in red in the log file, and in the Log Messages -> Processing pane:
   "2023-12-02T05:50:01 CRITICAL Failed to create MBTiles file C:/......."
   (choosing a new filename allows the run to proceed)
3. runtime is massively longer than 3.28 - apparently because it is trying to write the mbtiles file after each individual tile, or something to that effect. For a job that completed at 2.41GB in 3.28 and took about 4 hours, it's now been 5.5 hours on the same computer and the job is at 40%. When running in 3.28, a windows file navigator left open to the output directory only refreshed with a new size of the mbtiles file being generated once every few minutes, but in 3.34.1 it updates once a second or maybe a bit more. So probably a LOT more disc access is happening in 3.34.1 vs 3.28. QGIS is not taking up much resource at all - 5%CPU and a couple hundred MB memory.

Here's the generate tiles log output from the failed run in #2, which also shows #1:

QGIS version: 3.34.1-Prizren
QGIS code revision: 133927424d
Qt version: 5.15.3
Python version: 3.9.5
GDAL version: 3.8.0
GEOS version: 3.12.1-CAPI-1.18.1
PROJ version: Rel. 9.3.0, September 1st, 2023
PDAL version: 2.6.0 (git-version: e4d82c)
Algorithm started at: 2023-12-02T05:50:01
Algorithm 'Generate XYZ tiles (MBTiles)' starting…
Input parameters:
{ 'ANTIALIAS' : False, 'BACKGROUND_COLOR' : QColor(0, 0, 0, 0), 'DPI' : 96, 'EXTENT' : '6765527.653800000,7126073.346100000,2129220.777300000,2320679.835700000 [EPSG:2226]', 'METATILESIZE' : 4, 'OUTPUT_FILE' : 'C:/Users/caver/CalTopo/data/tiles/NevCoParcels20231103h.mbtiles', 'QUALITY' : 75, 'TILE_FORMAT' : 0, 'ZOOM_MAX' : 19, 'ZOOM_MIN' : 14 }

Background color setting ignored, the JPG format only supports fully opaque colors
Failed to create MBTiles file C:/Users/caver/CalTopo/data/tiles/NevCoParcels20231103h.mbtiles
Execution failed after 0.08 seconds

Loading resulting layers

Steps to reproduce the issue

Any dataset: Processing Toolbox --> Raster Tools --> Generate XYZ tiles (MBTiles)

Versions

QGIS version
3.34.1-Prizren
QGIS code revision
1339274
Qt version
5.15.3
Python version
3.9.5
GDAL/OGR version
3.8.0
PROJ version
9.3.0
EPSG Registry database version
v10.094 (2023-08-08)
GEOS version
3.12.1-CAPI-1.18.1
SQLite version
3.41.1
PDAL version
2.6.0
PostgreSQL client version
15.2
SpatiaLite version
5.1.0
QWT version
6.1.6
QScintilla2 version
2.13.4
OS version
Windows 10 Version 2009

Active Python plugins
qtiles
1.7.2
QuickOSM
2.2.3
db_manager
0.1.20
grassprovider
2.12.99
MetaSearch
0.3.6
processing
2.12.99

(none of these issues existed in LTR 3.28.12)

Supported QGIS version

• I'm running a supported QGIS version according to the roadmap.

New profile

• I tried with a new QGIS profile

Additional context

[agiudiceandrea]

@caver456, thanks for reporting. Since the issue report's title mentions QGIS 3.28.12 while in the issue report description is reported that the affected version is QGIS 3.34, could you please better specify which version exactly is affected by the issue and, if the case, change the issue report title accordingly?
Could you please provide the full version info of the affected QGIS version ("In the QGIS Help menu -> About, click in the table, Ctrl+A and then Ctrl+C. Finally paste here. Do not make a screenshot") and try if the issue also occurs using a new QGIS user profile as requested in the issue report form?

[caver456]

Updated the title, the 'versions' field, the 'new profile' field, and added a note at the top of the report referencing #50170. Installing the fix of #50170 was the motivation to upgrade from 3.28.12 LTR to 3.34.1.

[caver456]

All of the reported issues still happen in the new profile, including the long runtime (based on 6 minutes of runtime so far, with the progress bar still at 0% and file size at 120MB out of an expected 6GB) and rapid-directory-listing-updates in the Windows file navigator.

[caver456]

Also of note, the new version no longer generates a line in the log when it starts processing each zoom level - even if the log level is set to Verbose in the MBTiles dialog, in Advanced --> Alogorithm Settings. If it's possible to bring those log lines back, I felt they were helpful as a confirmation of progress with a bit more detail than the progress bar.

[caver456]

The resulting mbtiles file generated with 3.34.1 is about 2.5x as big as the one generated with 3.28.12. (5.72GB vs 2.41GB) Since some of the tiles were generated with opaque black background in 3.28.12 (#50170) the file size difference might not be relevant until really digging in to see exactly what tiles exist in each file.

[tobinbradley]

I think what's happening is It creates JPG tiles even if you pick PNG, leading to larger mbtiles files and no background transparency.

[caver456]

The 3.34.1-rendered 5.7GB mbtiles metadata says it's png, as viewed in 'DB Browser for SQLite' for Windows, and the tiles do have the correct transparency (I view them in CalTopo Desktop). [image: image.png] The smaller-but-buggy 3.28.12-rendered 2.4GB mbtiles metadata also shows png. The bounds are the same between the two versions, but are rounded to fewer significant figures in 3.34.1 above. 3.28.12: [image: image.png] Spot-checking a couple of the same tiles between the two versions, yes they are bigger in 3.34.1 (14.37kB vs 12.04kB for example). Curious... antialiasing was off and the quality was the same in both renderings (75%).

…

On Wed, Dec 6, 2023 at 2:03 PM Tobin Bradley ***@***.***> wrote: I think what's happening is It creates JPG tiles even if you pick PNG, leading to larger mbtiles files and no background transparency. — Reply to this email directly, view it on GitHub <#55489 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEPCEZVDHUHPEHPAG25OHOTYIDTSTAVCNFSM6AAAAABAEG77N6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNBTG43DCOBSG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[agiudiceandrea]

1. this warning shows up (in orange) on every run, even when PNG is the selected file type: "Background color setting ignored, the JPG format only supports fully opaque colors"

I think what's happening is It creates JPG tiles even if you pick PNG

The warning is displayed by mistake. I've fixed it with PR #55573 (3.34 #55575).

I suppose the issue 3. is due to the fact that in the old Python version of the algorithm the "journal_mode=WAL" is used which drastically reduces the disk access and enables "faster sqlite processing for parallel access"

QGIS/python/plugins/processing/algs/qgis/TilesXYZ.py

Lines 363 to 364 in 58aa7b8

	# faster sqlite processing for parallel access https://stackoverflow.com/questions/15143871/simplest-way-to-retry-sqlite-query-if-db-is-locked
	self._execute_sqlite("PRAGMA journal_mode=WAL")

[caver456]

Thank you! Interesting that WAL seems to give such a speedup. Any chance that could be restored?