Convert springs to use ST_PointOnSurface and reformat SQL #3233

pnorman · 2018-05-13T02:39:01Z

This partly replaces #3228 and is an example of converting a layer to use ST_PointOnSurface

kocio-pl · 2018-05-13T02:42:57Z

project.mml

+              "natural"
+            FROM planet_osm_point
+            WHERE way && !bbox!
+            ) _


Looks like unwanted space + underscore.

What would you suggest as an alternative for what to call the inner subselect? _ is a reasonably standard name when you just need a name. s, p, q are also common throw-away names.

I don't know, however I don't like such shortcuts in the big SQL section of a public project. Typical i for loop index might be OK in typical code, but this one is already hard to maintain, underscore is worse than any letter and really looks like a typo, and you can't expect that other people will be real programmers to catch it. Something self-commenting like inner_subselect would be probably good.

BTW: there are more undocumented conventions and sections that I don't understand, it might make sense to add more comments one day.

inner_subselect as a name would do for me.

sommerluk · 2018-05-13T08:09:48Z

Something self-commenting like inner_subselect would be probably good.

I support this. Longer identifiers do no harm during processing, but increase legibility of the code.

matthijsmelissen · 2018-05-13T23:31:54Z

project.mml

+            "natural"
+          FROM
+          (SELECT
+              ST_PointOnSurface(way) AS way,


Question: is psql smart enough to only do the ST_PointOnSurface calculation for objects satisfying "natural" IN ('spring'), in this query?

Ok, I see now it avoids the duplication of WHERE "natural" IN ('spring').

In this case the conditions on the tag columns are simple enough a subselect isn't necessary, but on longer queries, these can be over a dozen lines.

Thanks, clear. And what about the question above?

Postgres can do the query either way, the results are the same. It will pick whichever is cheapest, which will normally be using the index, filtering by geometry, then calculating st_pointonsurface.

So if I understand you correctly, in this query it will also calculate ST_PointOnSurface for polygons that are not natural=spring, right?

There is no guarantee one way or the other. It's unlikely to, but with a small database and the right indexes it might.

matthijsmelissen · 2018-05-13T23:42:26Z

project.mml

@@ -285,14 +285,22 @@ Layer:
      <<: *osm2pgsql
      table: |-
        (SELECT


Does wrapping this in a superquery accomplish anything at all? It seems to me that leaving lines 287-290 and 302 out would yield exactly the same result?

No, since then the query would return all polygons regardless of their natural column.

matthijsmelissen · 2018-05-14T00:15:19Z

This is essentially the same method as #3228 (after my last changes), which gives performance problems. We therefore should check if this PR causes performance problems too.

pnorman · 2018-05-15T21:23:11Z

This is essentially the same method as #3228 (after my last changes)

This isn't the same method, since #3228 doesn't filter the points table by bounding box.

matthijsmelissen · 2018-05-15T22:36:20Z

This isn't the same method, since #3228 doesn't filter the points table by bounding box.

I fixed the filter on the points table now. Unfortunately, this still gives me a performance hit of about 50% of overall rendering time.

matthijsmelissen · 2018-05-16T19:38:36Z

project.mml

+            FROM planet_osm_point
+            WHERE way && !bbox!
+            ) _
+          WHERE "natural" IN ('spring')


Should we not add another WHERE way && !bbox! in the outer subquery? Because now it will also forward points to Mapnik that are outside of the bounding box that is currently being rendered (as long as they are the label position of a polygon that falls within that bounding box).

We could, but the PR is no different than current behavior. I suppose the difference is that filtering on contains is now cheap to do in-db

matthijsmelissen · 2019-01-03T11:56:40Z

Marking this as WIP as I suspect this will cause performance issues.

pnorman · 2019-01-04T22:23:14Z

Marking this as WIP as I suspect this will cause performance issues.

What are those? I don't see any.

matthijsmelissen · 2019-01-04T22:31:01Z

Is anybody able to check the performance of this PR?

I think it would be good if we could give this attention, as it is a necessary step for the move to vector tikes.

pnorman · 2019-01-04T22:34:13Z

Is anybody able to check the performance of this PR?

I think it would be good if we could give this attention, as it is a necessary step for the move to vector tikes.

I found no performance issues when testing this and see nothing that could cause performance issues.

matthijsmelissen · 2019-01-10T23:31:25Z

@pnorman I still don't fully understand why we need the subselect. Why can't we simply do this?

(SELECT
    ST_PointOnSurface(way) AS way,
    "natural"
  FROM planet_osm_polygon
  WHERE way && !bbox!
    AND "natural" IN ('spring')
UNION ALL
SELECT
    way,
    "natural"
  FROM planet_osm_point
  WHERE way && !bbox!
    AND "natural" IN ('spring')

Is the only reason to avoid duplication of "natural" IN ('spring'), or is there a performance reason?

matthijsmelissen · 2019-01-11T00:38:01Z

Regarding performance: I made a quick and dirty test to extend the method proposed here to all layers. This shows no performance degradation. So I'm happy to merge this - I'm sorry that I didn't review it sooner.

matthijsmelissen

Looks good to me.

Awaiting your question about the reason for the subselect, and then this can be merged.

pnorman · 2019-01-11T05:42:51Z

Is the only reason to avoid duplication of "natural" IN ('spring'), or is there a performance reason?

Performance is the same. It avoids duplication and is a different way to think about it. The duplication isn't a big issue here, but when we get to a query like the POI layers, there's a huge WHERE clause, and deduplicating that is a big gain.

matthijsmelissen · 2019-01-11T22:47:09Z

Clear. You can merge it if you like (or I can merge it, as you prefer).

@pnorman

Resolves gravitystorm#1644. This is a necessary step towards vector rendering (see gravitystorm#1644). This is a second trial of gravitystorm#3228, based on the word by @pnorman in gravitystorm#3233. This is a first step, once we have this than we can probably merge the point and poly layers (for example amenity-points and amenity-points-poly).

Convert springs to use ST_PointOnSurface and reformat SQL

09e30b0

pnorman requested a review from matthijsmelissen May 13, 2018 02:39

kocio-pl reviewed May 13, 2018

View reviewed changes

matthijsmelissen reviewed May 13, 2018

View reviewed changes

pnorman mentioned this pull request May 15, 2018

[WIP] Switch to ST_PointOnSurface #3228

Closed

matthijsmelissen reviewed May 16, 2018

View reviewed changes

matthijsmelissen changed the title ~~Convert springs to use ST_PointOnSurface and reformat SQL~~ [WIP] Convert springs to use ST_PointOnSurface and reformat SQL Jan 3, 2019

matthijsmelissen changed the title ~~[WIP] Convert springs to use ST_PointOnSurface and reformat SQL~~ Convert springs to use ST_PointOnSurface and reformat SQL Jan 4, 2019

matthijsmelissen approved these changes Jan 11, 2019

View reviewed changes

pnorman merged commit 16db0b8 into gravitystorm:master Jan 11, 2019

matthijsmelissen mentioned this pull request Jan 12, 2019

[WIP] Switch to ST_PointOnSurface #3636

Closed

pnorman deleted the springs_pointonsurface branch April 1, 2019 03:20

This was referenced Sep 22, 2019

Names of POIs instead of their housenumbers #962

Closed

Switch to ST_PointOnSurface? #1644

Closed

jeisenbe added the code label Oct 15, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Convert springs to use ST_PointOnSurface and reformat SQL #3233

Convert springs to use ST_PointOnSurface and reformat SQL #3233

pnorman commented May 13, 2018

kocio-pl May 13, 2018

pnorman May 13, 2018

kocio-pl May 13, 2018

kocio-pl Jan 11, 2019

sommerluk commented May 13, 2018

matthijsmelissen May 13, 2018

matthijsmelissen May 14, 2018

pnorman May 15, 2018

matthijsmelissen May 15, 2018

pnorman May 16, 2018

matthijsmelissen May 16, 2018

pnorman May 26, 2018

matthijsmelissen May 13, 2018

pnorman May 15, 2018

matthijsmelissen commented May 14, 2018

pnorman commented May 15, 2018

matthijsmelissen commented May 15, 2018

matthijsmelissen May 16, 2018

pnorman May 17, 2018

matthijsmelissen commented Jan 3, 2019

pnorman commented Jan 4, 2019

matthijsmelissen commented Jan 4, 2019

pnorman commented Jan 4, 2019

matthijsmelissen commented Jan 10, 2019

matthijsmelissen commented Jan 11, 2019

matthijsmelissen left a comment

pnorman commented Jan 11, 2019

matthijsmelissen commented Jan 11, 2019

Convert springs to use ST_PointOnSurface and reformat SQL #3233

Convert springs to use ST_PointOnSurface and reformat SQL #3233

Conversation

pnorman commented May 13, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sommerluk commented May 13, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matthijsmelissen commented May 14, 2018

pnorman commented May 15, 2018

matthijsmelissen commented May 15, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matthijsmelissen commented Jan 3, 2019

pnorman commented Jan 4, 2019

matthijsmelissen commented Jan 4, 2019

pnorman commented Jan 4, 2019

matthijsmelissen commented Jan 10, 2019

matthijsmelissen commented Jan 11, 2019

matthijsmelissen left a comment

Choose a reason for hiding this comment

pnorman commented Jan 11, 2019

matthijsmelissen commented Jan 11, 2019