Migration to vector tiles #3201
Comments
|
Would be really awesome to support Vector Tiles ! 🎉 |
|
I would agree, but maintaining two different styles is too complicated, specially because they don't seem to be compatible, or at least nobody has written a converter from one to the other. Think about all the contributions, having to be implemented twice... |
|
This problem is much wider than style, it's also related to deployment strategy (especially on OSMF servers, but most probably also for the forks following us, like German or Belgian) and tools to be used. @jbelien The whole point with server side raster tiles rendering is about planning deployment so that vector osm-carto tiles are then transformed into raster tiles. In other words instead of: osm-carto -> raster tiles it would be: osm-carto -> vector tiles -> raster tiles so the final output should be very similar. Please note however that from the osm-carto perspective it's just a migration. Raster tiles rendering will be done with some other tools, so it's a deployment problem. Yet my intention is to plan the whole process with our partners (including you 😄), not just abrupt change of the styling output. @StyXman Rory's vector fork is designed to be easily synchronized with standard version, but that might not be the proper solution. Andy suggested making a development branch form the current release (like v4.10.0-vector) and not try to synchronize them. After some time, when we will be ready to really deploy vector tiles on production, we should only update the vector fork to the current version of plain osm-carto (like v4.44.4) or just port it directly (into v4.44.4-vector), whatever will be easier. This would be the switch point. From now on the vector version (which will become v5.0.0 most probably - or we could rename the whole project) will take over and the raster version will become the legacy. So we plan to make only some plain development fork just to learn things, but after that there will be no more two versions, the old one won't be supported probably. There's always possibility that somebody will like to support it, but given that the migration should not affect end users (and possibly be friendly for our forks) and there are only few people involved, I don't quite believe it would be the case. |
|
I would like to emphasize that the ideas @kocio-pl is presenting here are not a consensus position of the maintainers of this style. We have briefly touched this topic in non-public discussion but as most can probably imagine the opinions were more differentiated and no common conclusions were drawn so far. So if @kocio-pl speaks of a 'we' here take that with a grain of salt. If this project makes major changes in the toolchain it is based on this does not automatically mean deployments of this style (including the one on OSMF servers) or derivative styles follow suit. Should such a major change become concrete in the future - no matter if this happens within this project or separately (as it was done when making the change from Mapnik XML to CartoCSS) it will be up to each of the deployments and forks to decide if to follow this change or not. Personally and from a design perspective i think a fresh start as with the change from Mapnik XML to CartoCSS would - even if it maintains some level of design continuity - be a very healthy step. |
|
This ticket is exactly to make the discussion more open. You're right that this is not anything set in stone, it's just a draft made by Andy (based on his real life experience) and approved by me, so we could start with some clear vision. Thanks for a disclaimer. So yes - take it with a big grain of salt... and join the discussion!
It was hard enough in 2012 for one determined man to make it. Now it's much more complex, so I don't believe that somebody else will try to do that again. But if you - or anybody else for that matter - really want to make such rewrite, it can be just as good solution. It might be even better in the long run, given your cartographic experience together with designing and coding skills. I just think it will take years to come with a first demo after the "simple" port will be ready, so I want to start with more limited approach, which I find to be more realistic. I entirely share your position that any kind of competition could only make the state of OSM rendering more healthy. |
|
For clarification - with:
I meant creating a new project that does not try to imply being the mandatory/automatic successor of current osm-carto. I did not want to speculate how this project would start practically (in form of a new implementation from scratch or as a manual or automatic conversion of existing code) - that is a completely different question. This would encourage everyone - both style developers and style users - to evaluate their choices and make a conscious decision about these choices and take responsibility for them - something that at the moment rarely happens which is rather problematic. Project continuity can appear convenient but the truth is this project is not the same as it was five years ago in almost any aspect even as it is right now - i for example just illustrated in #1630 (comment) how much the development focus has shifted. If there is a major change in the underlying toolchain in addition to that this would be an excellent opportunity to give everyone a kick to make themselves aware of how much things have changed overall and to check if this still matches their individual goals as developers or style users. |
|
A consequence of doing vector tiles is, I think, the ability to turn on/off on the client the rendering certain features, typically POIs. This would probably help with things like rendering offices and, maybe, with PIOs that fall in multiple categories (for instance, here in .fr, some bakeries also sell take away food and coffee). Just an idea; I really don't know if this is possible. |
|
If the same stylesheet code can support both deployment targets then I think we could (and should) reuse/extend the current project. Otherwise, I'm with @imagico on that - it is better to fork the project under a new name, so that we don't force existing users to switch, and let people use and maintain the existing implementation if they wish so. Otherwise we end up with Gnome3 scenario where Gnome2 users have to fork and copy the existing implementation. And, to make it clear, I am in favour of using vector tiles on the main osm map. Especially if, as planned, it can be rendered on the server side to avoid Web-GL. Many of the current usability issues (lack of language versions, lack of overlays for transport, POI categories etc) could be easily solved. |
|
Sure, we could make a fork(s) under some other name. Instead of long and muddy "OSM Carto Classic/Legacy" I would rather go with something like "Vektor Flagship" and call the other styles/deployments from the same family like "Vektor Drakkar" or some other fancy vessel type that makes sense ("Vektor Flagship One" comes to mind for OSMF deployment). Right name can be quite important to avoid confusion (see #2927) and with many possible styles basing on the same vector engine such a double part name could help identify them (hence "Vektor" instead of "Vector" - not too common name on the net). It's an interesting idea that new style could be based on automatic conversion of existing code, but I don't believe it. It would mean writing code parser and this is not trivial programming task, as seen lately on a very simple example (see #3151). That means that we have not too much options left. One is kind of Rory's fork, which tries to add some vector-related things on top of existing CartoCSS+SQL syntax and I believe it's the way to go. Another is to write something from scratch and this would be quite another fork, which basically would be very light at the beginning. It might be nice competitor, but it would not allow smooth upgrade. That's why I want to keep the Andy's path. It's easier to make a lighter skin for such full-featured fork than make a different fork and try to recreate osm-carto features. |
|
@gravitystorm Could you tell what software stack are you using for Thunderforest styles? |
|
@kocio-pl Custom software, based on the tilelive modules. |
|
Rory told me that he was trying tilelive-copy (so probably https://github.com/mapbox/tilelive), but it was slow. His new tool is much faster, but not production ready due to bugs: https://github.com/rory/tileigi. Would somebody like to test them a bit? |
|
tileigi is not suitable for the osm.org because it only does pre-generation of vector tiles, which won't work with regular updates. It's also not really stable enough that I'd trust it in production yet. |
|
I've been talking with @rory today. More on that later. If we want to have a port which is a Mapnik style based on server-side vector tiles, I recommend starting by eliminating all layers which label polygons. Instead, use ST_PointOnSurface, and when applicable, combine them with the point version of the layer. This will eliminate one major difference between the port and what we currently have, without breaking what we currently have. |
|
@pnorman I'm not sure if I follow you entirely. Why would polygon labels cause a difference between the current style and the vector port? When we use ST_PointOnSurface, we have no guarantee about the location that is chosen (except that is lies in the polygon), right? |
|
Very good news! 👍 I wait for the summary of your talk with Rory. In case you wonder how it looks like, the new style is named Bolder and you can find it here: https://github.com/pnorman/bolder |
You can't label polygons with vector tiles because they're cut at tile boundaries. You have to turn the polygons to a point that is labeled. https://youtu.be/wQc492mtK-c?t=9m30s has an overview of the causes.
The only guarantee the function specification gives is that it lies on the surface. In practice, the algorithm does the centroid if possible. #349 (comment) has some examples of how Mapnik and ST_PointOnSurface work. Neither is uniformly better. There are methods believed to generate better results than ST_PointOnSurface like PIA, but those wouldn't be what we'd use initially. In general, moving to vector tiles makes labels harder, and even if you do your best, you end up with worse labels. |
|
From the conversation with @rory, we discussed a few things Geofabrik's interest is multiple languages with reasonable performance, retina tiles, and slightly changed styles for clients (colour changes, dropped features, etc). Vector tiles are good at this. Rory found tilelive had bad performance, so he started tileigi, which can be 100x at generating tiles. Tileigi currently only pregenerates tiles, it doesn't serve them live. Most of the performance gains he got are from using vector metatiles: the practice of requesting the area of 8x8 tiles, and slicing the result up into 64 individual tiles, and storing individual tiles on disk. osm-carto is a very complex style, and it can take hundreds of milliseconds to go from vector to raster for a single tile. This is exceptionally slow, and I'd consider times under 10ms more "normal". We discussed a number of ways to improve performance to make up for the osm-carto complexity and number of layers. The most likely route is to carry the metatiling farther. Instead of querying the area of 8x8 z14 tiles and slicing them before putting them on disk, put the unsliced tile (a z11 tile with a high resolution) on disk. Then, when a request for a z14 raster tile comes in, fetch the combined tile, generate a raster for the 8x8 area, and store the sliced tiles in memory. The in-memory tiles can be used for the subsequent requests for nearby tiles. This is technically overzoom for the vectors, and metatiles for the rasters. What you can see from the above is that there is no software serving stack for Mapnik vector tiles that is 100% there for our needs. Some more changes that would help maintaining a vector fork are
|
|
From my point of view:
But also:
|
|
2018-04-30 18:43 GMT+02:00 kocio-pl <[email protected]>:
From my point of view:
- rendering based on the size is essential, for example I was not able
to find anything better for natural reserves to avoid mess; but since the
area size data are in the database and the software is somehow aware what
is the current zoom, so it should be possible to calculate current size of
the area, or am i missing something?
with vector tiles you already have this precomputed and filtered for area
labels. You only have a point and a size / importance class for it
(referring to the curent zoomlevel or range of levels), and you won't have
label nodes for vector tiles at zoom levels where you don't want to show
them. At least this is how mapbox does it.
|
|
This is how Bolder looks like when rendered with repo instructions and Liechtenstein data:
I was wondering about possible geometry distortion, which is a big issue for @imagico. Here it is on z19 compared to osm-carto on z19 - I don't understand why the size is different with the same zoom level, but at least geometry seems to be unchanged (even without "Turning Off Simplification of Geometries" option):
|
|
Bolder client-side style with Mazovian voivodship/Warsaw data:
|
|
Client side rendering is quite fast only on highest zoom levels, but we have just 7 layers here instead of 82. It's very CPU intensive process. |
|
Client-side rendering is incompatible with the approach that Andy has suggested of incremental work, using small steps.
afaik, Mapnik doesn't let you use the zoom in calculations. If it did, we could stop converting area to pixels, and do the math in the MSS. But without that ability, we need to use different thresholds for different levels of overzoom.
It's one of the big reasons to use vector tiles. I don't know osm.org's usage numbers, but in general >50% of users can be expected to have a device with a high DPI screen.
There's no direct relation with vector patterns. It's already an issue. It might become more common to do retina tiles with vector tiles, so more people might hit the blurry issue.
We'd still be using Mapnik, Kosmtik, and CartoCSS. We'd need some Mapnik-based vector tile server for production, probably something based on tilelive. There are no clear candidates here. Tessera, Kartotherian, and others all would sort of work, but I'm not convinced that vector tiles have the performance without tricks like mentioned above, and no one has coded a stack with those kinds of tricks yet.
There's no rewriting involved. See @rory's talk for what's needed to port the style. |
I see. @talaj Could you comment on this? Is this something doable in a near future or maybe something that could be ported from your fork, for example? |
|
@kocio-pl This is possible already by render-time variables . |
|
See also #3912. |
|
Most of the layers have been switched to us ST_PointOnSurface, or have a PR pending for the switch. Still remaining are |
|
is there a solution to unfortunate icon placement like in this case:
https://www.openstreetmap.org/way/22883270#map=13/41.8133/12.2508
The label and icon are placed in a corner rather than close to the access.
|
|
There is a general and indirect one - to deploy Mapnik current If someone is willing to do that, it would be really great. Do you have an idea who could try to do this? EDIT - for some reason your image is not included properly, so I just put it here:
|
|
There are now PRs open which will move the remaining layers to use ST_PointOnSurface. There is also a PR to remove use of classes - #3912 The next steps for vector files would be:
"This would get us a working style and allow us to start exploring what works and doesn't" - per @pnorman |
|
Re: step 1. Use ST_Border to filter admin geoms We could use Could we instead switch to |
|
Oh, planet_osm_roads doesn't include names so it's not an option, and planet_osm_line does not appear to include administrative boundaries or protected areas, so that's why we need to use |
|
"Right now the most serious limitations we have in terms of style design are the limitations imposed by the software we use. It seems natural to me that based on that experience we take a critical look at what kind of limitations might be imposed by software we might move to in the future." "so far there has been no discussion that led to any kind of (even preliminary) decision on concrete software use yet." (#3948 (comment)) So we should discuss if you are going to move toward server-side vector tiles and how to implement it, and what costs this would impose. @pnorman previously suggested: "1. Vector tile generator: "The options can be divided into ST_AsMVT based or not. I prefer ST_AsMVT, because it avoids a single-company maintainer of the complicated part of vector tile generation. "PosTile is the leading candidate here, as it supports .tm2source files "2. Identify a vector to raster tile server "Tessera should work. All options are painful because they involve node-mapnik." |
|
Is anyone interested in discussing this issue further? I opened #3948 to use ST_Boundary, but this is not useful until there is an actual implementation for server-side vector tiles. Do we want to work on this here, or does it need to be a separate repository at this point? I would also consider redesigning this raster style for high DPI tiles, which would provide improved style options for thin features like barriers and borders, but I understand than many people are enthusiastic about vector tiles due to the potential to support multiple languages and resolutions. |
|
There is a chance to have better rendering in the "middle" of shapes than by http://postgis.net/2020/07/18/postgis-3.1.0alpha2/ |
|
Could a rendering use a base bitmap layer (without labels/icons, and probably without the linear features with thickness is uncontrolled) so that it can considerably reduce the the complexity for rendering low zoom levels ? This would not require extensive caching as those bitmaps would cover only these low levels and they could be refreshed if needed more often and more precisely (with halftoning and translucent pixels for smoothing curves). |
there's also the common transform of 2D shapes into their linear skeleton with nodes connected by straight segments: the node with the longest segment defines not just a maximum radius, and its shortest segment its minimum radius, but it also defines a direction that an be used to properly orient the labels: if you join the segments joining the longest run, you get a line that can also be used to create a spline over which you would orient the label, and the average of the smallest orthogonal segments can be used to scale the fonts... |
|
That would be great to have. Do you know if this feature is implemented in PostGIS? |
|
@kocio-pl as said in PostGIS 3.1.0alpha2. |
|
A migration to vector tiles is a complete do-over of the technology stack, even if all the cartographic decisions remain the same. IMHO that's too big of a change to be executed within this repository (not to mention the name "openstreetmap-carto" no longer makes sense as a repo name -- it's really "openstreetmap-vector" or something along those lines). A vector version of this style should really start from a new repo and start with some fundamental decisions about what technology stack to build and then iteratively build up the features until they match. I note the following points which may be of interest to folks looking for an update on vector tile work, and not otherwise mentioned on this thread:
|
That is a good point. Should we close this issue, or is there anything still to be done in this style which would be useful for a theoretical vector tile alternative? |
|
Hello, I have been playing around with @pnorman's osm-carto style vector reimplementation, and the whole rendering stack (I recently wrote https://github.com/feludwig/pgsql-omt-schema, and a little demo https://feludwig.github.io/pgsql-omt-schema)
The sql functions would be easy as a prototype: just take the table from mapnik xml.
The bigger challenge is to transpile all the xml s to mapbox-json style layers (there would be a lot of them) and invert the <Min/MaxScaleDenominator> s to "expressions", eg I mean : .
I looked for the "migration to vector tiles" discussion, is this the correct issue to post into ? |
This issue was created to discuss ideas and potential concepts to transfer/re-implement this CartoCSS map style for a different tool-chain as an initiative from within the OSM-Carto developer community. This discussion never went beyond the brainstorming stage, largely because none of the currently available tool-chains for client side rendering is really compatible with this project (both in terms of the cartography and in terms of the openly cooperative development model). Several independent projects have meanwhile been started to create styles with client side rendering that are designed for some (usually superficial) similarity with OSM-Carto. The one that got closest so far is actually the earliest (the ESRI demo that was mentioned in the discussion above).
I am not familiar with projects in that direction. I don't think this would be practically useful for a map style like OSM-Carto. Keep in mind this is nothing like the conversion of code between two low level turing-complete languages. This is about converting map design between two frameworks that are based on completely different overall paradigms. And this is definitively the wrong forum for asking this kind of question. |
Follow up to #1975 (comment)
As discussed privately among team members, it's good to migrate osm-carto to be rendered with vector tiles. To allow as smooth and painless migration path as possible, it should be ready for server side raster tiles rendering. It should be non-disruptive change for clients while allowing to start developing client side rendering and testing vector tiles in general.
This is not a fresh and untested approach, @gravitystorm has been using it for his own styles for a few years already - you can watch his presentation about it from SotM EU 2014:
https://www.youtube.com/watch?v=NoLJHgqDCzc
There's also a vector fork of osm-carto by @rory (also his presentation from SotM 2016):
https://github.com/geofabrik/openstreetmap-carto-vector-tiles
https://2016.stateofthemap.org/2016/converting-a-regular-carto-project-to-vector-tiles-osmcarto-case-study/
However this fork is quite old and happened before our v4.x database migration. It's not clear now how useful it could be, other than being proof of concept.
This ticket is about coordination of the effort, more specific tickets might be needed for subtasks and technical background (like choosing the proper software stack) is yet to be discussed and tested.
The text was updated successfully, but these errors were encountered: