Update failed with: `psycopg2.errors.InternalError_: Splitting of Point geometries is unsupported` #3022

amandasaurus · 2023-04-03T09:24:51Z

Describe the bug

nominatim-updates started failing at 2023-04-01T19 with a PostGIS error: psycopg2.errors.InternalError_: Splitting of Point geometries is unsupported. It has continued to fail since. The database was about 14hrs behind at the time. The command ran is python3 /srv/nominatim/build/nominatim replication --once. Worldwide nominatim data.

Full output from nominatim-updates.service:

Apr 01 19:27:24 geocoding6 nominatim[1488975]: 2023-04-01 19:27:24: Done 2067 in 22 @ 91.494 per second - rank 0 ETA (seconds): 0.80
Apr 01 19:27:25 geocoding6 nominatim[1488975]: 2023-04-01 19:27:25: Done 2140/2140 in 23 @ 91.395 per second - FINISHED rank 0
Apr 01 19:27:25 geocoding6 nominatim[1488975]: 2023-04-01 19:27:25: Starting interpolation lines (location_property_osmline) (using batch size 20)
Apr 01 19:27:26 geocoding6 nominatim[1488975]: Traceback (most recent call last):
Apr 01 19:27:26 geocoding6 nominatim[1488975]:   File "/srv/nominatim/build/nominatim", line 14, in <module>
Apr 01 19:27:26 geocoding6 nominatim[1488975]:     exit(cli.nominatim(module_dir='/srv/nominatim/build/module',
Apr 01 19:27:26 geocoding6 nominatim[1488975]:   File "/srv/nominatim/Nominatim/nominatim/cli.py", line 264, in nominatim
Apr 01 19:27:26 geocoding6 nominatim[1488975]:     return parser.run(**kwargs)
Apr 01 19:27:26 geocoding6 nominatim[1488975]:   File "/srv/nominatim/Nominatim/nominatim/cli.py", line 126, in run
Apr 01 19:27:26 geocoding6 nominatim[1488975]:     return args.command.run(args)
Apr 01 19:27:26 geocoding6 nominatim[1488975]:   File "/srv/nominatim/Nominatim/nominatim/clicmd/replication.py", line 196, in run
Apr 01 19:27:26 geocoding6 nominatim[1488975]:     self._update(args)
Apr 01 19:27:26 geocoding6 nominatim[1488975]:   File "/srv/nominatim/Nominatim/nominatim/clicmd/replication.py", line 161, in _update
Apr 01 19:27:26 geocoding6 nominatim[1488975]:     indexer.index_full(analyse=False)
Apr 01 19:27:26 geocoding6 nominatim[1488975]:   File "/srv/nominatim/Nominatim/nominatim/indexer/indexer.py", line 140, in index_full
Apr 01 19:27:26 geocoding6 nominatim[1488975]:     if self.index_by_rank(26, 30) > 1000:
Apr 01 19:27:26 geocoding6 nominatim[1488975]:   File "/srv/nominatim/Nominatim/nominatim/indexer/indexer.py", line 178, in index_by_rank
Apr 01 19:27:26 geocoding6 nominatim[1488975]:     total += self._index(runners.InterpolationRunner(analyzer), 20)
Apr 01 19:27:26 geocoding6 nominatim[1488975]:   File "/srv/nominatim/Nominatim/nominatim/indexer/indexer.py", line 234, in _index
Apr 01 19:27:26 geocoding6 nominatim[1488975]:     runner.index_places(pool.next_free_worker(), part)
Apr 01 19:27:26 geocoding6 nominatim[1488975]:   File "/srv/nominatim/Nominatim/nominatim/db/async_connection.py", line 201, in next_free_worker
Apr 01 19:27:26 geocoding6 nominatim[1488975]:     return next(self.free_workers)
Apr 01 19:27:26 geocoding6 nominatim[1488975]:   File "/srv/nominatim/Nominatim/nominatim/db/async_connection.py", line 209, in _yield_free_worker
Apr 01 19:27:26 geocoding6 nominatim[1488975]:     if thread.is_done():
Apr 01 19:27:26 geocoding6 nominatim[1488975]:   File "/srv/nominatim/Nominatim/nominatim/db/async_connection.py", line 159, in is_done
Apr 01 19:27:26 geocoding6 nominatim[1488975]:     if self.conn.poll() == psycopg2.extensions.POLL_OK:
Apr 01 19:27:26 geocoding6 nominatim[1488975]: psycopg2.errors.InternalError_: Splitting of Point geometries is unsupported
Apr 01 19:27:26 geocoding6 nominatim[1488975]: CONTEXT:  PL/pgSQL function osmline_update() line 77 at assignment
Apr 01 19:27:27 geocoding6 systemd[1]: nominatim-updates.service: Main process exited, code=exited, status=1/FAILURE
Apr 01 19:27:27 geocoding6 systemd[1]: nominatim-updates.service: Failed with result 'exit-code'.

To Reproduce
Run an update. The error occurs always constantly now, on many servers.

Software Environment (please complete the following information):

Nominatim version: 4.2.0
Postgresql version: 14.7
Postgis version: 3.2.0
OS: Ubuntu Linux 22.04.01

Hardware Configuration (please complete the following information):

RAM: 128GiB
number of CPUs: 12
type and size of disks: 2 × 1TiB NVMe disks
bare metal/AWS/other cloud service: bare metal server from Hetzner

The text was updated successfully, but these errors were encountered:

lonvia · 2023-04-03T09:30:29Z

I wonder if this is another incarnation of #2989. Can you check if the Postgres log has the list of place_id in the interpolation table that failed and then follow the instructions for debugging in #2989 (comment)

amandasaurus · 2023-04-04T09:11:28Z

Can you check if the Postgres log has the list of place_id in the interpolation table that failed and then follow the instructions for debugging in #2989 (comment)

Running with -vvv gave the following:

…
2023-04-04 10:47:44: SELECT word_id FROM word WHERE word_token = 'sheringham avenue' and type = 'W'
2023-04-04 10:47:44: SELECT create_analyzed_hnr_id('77', ARRAY['77'])
2023-04-04 10:47:44: SELECT create_analyzed_hnr_id('142', ARRAY['142'])
2023-04-04 10:47:44: Processing places: [[323189049, {'street': 'Dersingham Avenue', '_inherited': '', 'housenumber': '2', 'interpolation': 'even'}], [323189050, {'street': 'Dersingham Avenue', '_inherited': '', 'housenumber': '28', 'interpolation': 'even'}], [323189051, {'street': 'Dersingham Avenue', '_inherited': '', 'housenumber': '92', 'interpolation': 'even'}], [323189052, {'street': 'Dersingham Avenue', '_inherited': '', 'housenumber': '148', 'interpolation': 'even'}], [323189053, {'street': 'Dersingham Avenue', '_inherited': '', 'housenumber': '158', 'interpolation': 'even'}], [323189054, {'street': 'Dersingham Avenue', '_inherited': '', 'housenumber': '206', 'interpolation': 'even'}], [323189055, {'street': 'Dersingham Avenue', '_inherited': '', 'housenumber': '169', 'interpolation': 'odd'}], [323189056, {'street': 'Dersingham Avenue', '_inherited': '', 'housenumber': '157', 'interpolation': 'odd'}], [323189057, {'street': 'Dersingham Avenue', '_inherited': '', 'housenumber': '228', 'interpolation': 'even'}], [323189058, {'street': 'Dersingham Avenue', '_inherited': '', 'housenumber': '282', 'interpolation': 'even'}], [323189059, {'street': 'Dersingham Avenue', '_inherited': '', 'housenumber': '251', 'interpolation': 'odd'}], [323189060, {'street': 'Brancaster Road', '_inherited': '', 'housenumber': '25', 'interpolation': 'odd'}], [323189061, {'street': 'Brancaster Road', '_inherited': '', 'housenumber': '26', 'interpolation': 'even'}], [323192728, {'street': 'San Juan', 'inclusion': 'actual', '_inherited': '', 'housenumber': '1001', 'interpolation': 'odd'}], [323192848, {'street': 'San Juan', 'inclusion': 'actual', '_inherited': '', 'housenumber': '1002', 'interpolation': 'even'}], [323192890, {'street': 'La Rioja', 'inclusion': 'actual', '_inherited': '', 'housenumber': '1001', 'interpolation': 'odd'}], [323193012, {'street': 'La Rioja', 'inclusion': 'actual', '_inherited': '', 'housenumber': '1002', 'interpolation': 'even'}], [323193034, {'street': 'Catamarca', 'inclusion': 'actual', '_inherited': '', 'housenumber': '1001', 'interpolation': 'even'}], [323193155, {'street': 'Catamarca', 'inclusion': 'actual', '_inherited': '', 'housenumber': '1002', 'interpolation': 'even'}], [323193238, {'street': 'Mitre', 'inclusion': 'actual', '_inherited': '', 'housenumber': '1001', 'interpolation': 'odd'}]]
Traceback (most recent call last):
  File "/srv/nominatim/build/nominatim", line 14, in <module>
    exit(cli.nominatim(module_dir='/srv/nominatim/build/module',
  File "/srv/nominatim/Nominatim/nominatim/cli.py", line 264, in nominatim
    return parser.run(**kwargs)
  File "/srv/nominatim/Nominatim/nominatim/cli.py", line 126, in run
    return args.command.run(args)
  File "/srv/nominatim/Nominatim/nominatim/clicmd/replication.py", line 196, in run
    self._update(args)
  File "/srv/nominatim/Nominatim/nominatim/clicmd/replication.py", line 161, in _update
    indexer.index_full(analyse=False)
  File "/srv/nominatim/Nominatim/nominatim/indexer/indexer.py", line 140, in index_full
    if self.index_by_rank(26, 30) > 1000:
  File "/srv/nominatim/Nominatim/nominatim/indexer/indexer.py", line 178, in index_by_rank
    total += self._index(runners.InterpolationRunner(analyzer), 20)
  File "/srv/nominatim/Nominatim/nominatim/indexer/indexer.py", line 234, in _index
    runner.index_places(pool.next_free_worker(), part)
  File "/srv/nominatim/Nominatim/nominatim/db/async_connection.py", line 201, in next_free_worker
    return next(self.free_workers)
  File "/srv/nominatim/Nominatim/nominatim/db/async_connection.py", line 209, in _yield_free_worker
    if thread.is_done():
  File "/srv/nominatim/Nominatim/nominatim/db/async_connection.py", line 159, in is_done
    if self.conn.poll() == psycopg2.extensions.POLL_OK:
psycopg2.errors.InternalError_: Splitting of Point geometries is unsupported
CONTEXT:  PL/pgSQL function osmline_update() line 77 at assignment

However for all of the place ids listed, I always got the same message of success:

nominatim=# BEGIN; UPDATE location_property_osmline SET indexed_status = 0 WHERE place_id = 323193238; ROLLBACK;
BEGIN
UPDATE 1
ROLLBACK

L3o-pold · 2023-04-05T08:26:14Z

Same issues occurred 2 days ago using Postgresql 12.3.0 and Nominatim 4.2.2

mtmail · 2023-04-05T12:10:42Z

Similar here on multiple separate servers. Postgresql 14.7, 4.1.99-0 (9c152a03)

nominatim index --threads 1 -v -v -v
[...]
2023-04-05 11:15:23: SELECT word_id FROM word WHERE word_token = 'avenida doutor plinio de castro prado' and type = 'W'
2023-04-05 11:15:23: SELECT create_analyzed_hnr_id('400', ARRAY['400'])
2023-04-05 11:15:23: Processing places: [[306317797, {'street': 'Avenida Doutor Plínio de Castro Prado', '_inherited': '', 'housenumber': '981', 'interpolation': 'odd'}], [306317971, {'city': 'Chapecó', 'street': 'Rua Cristian Pies Gionbelli', 'suburb': 'Autódromo', 'inclusion': 'estimate', '_inherited': '', 'housenumber': '1030', 'subdistrict': 'Vederti', 'interpolation': 'even'}], [306317980, {'street': 'Rua Cristian Pies Gionbelli', 'suburb': 'Autódromo', 'inclusion': 'estimate', '_inherited': '', 'housenumber': '2', 'subdistrict': 'Vederti', 'interpolation': 'even'}], [344487755, {'city': 'Bernardo de Irigoyen', 'street': 'Cristóbal Colón', 'inclusion': 'actual', '_inherited': '', 'housenumber': '199', 'interpolation': 'odd'}], [344487756, {'city': 'Bernardo de Irigoyen', 'street': 'Cristóbal Colón', 'inclusion': 'actual', '_inherited': '', 'housenumber': '200', 'interpolation': 'even'}], [344488708, {'city': 'Bernardo de Irigoyen', 'street': 'Ex Combatiente Ramón Acosta', 'inclusion': 'actual', '_inherited': '', 'housenumber': '99', 'interpolation': 'odd'}], [344489713, {'city': 'Bernardo de Irigoyen', 'street': 'Malvinas Argentinas', 'inclusion': 'actual', '_inherited': '', 'housenumber': '201', 'interpolation': 'odd'}], [344489715, {'city': 'Bernardo de Irigoyen', 'street': 'Malvinas Argentinas', 'inclusion': 'actual', '_inherited': '', 'housenumber': '702', 'interpolation': 'odd'}], [383842117, {'street': 'Avenida Morumbi', 'inclusion': 'estimate', 'interpolation': 'all'}], [389701232, {'street': 'Rua Salomé Queiroga', 'postcode': '03434-000', 'inclusion': 'estimate', 'interpolation': 'all'}], [390857384, {'block': 'K', 'place': 'Jahangir Puri', 'postcode': '110033', '_inherited': '', 'housenumber': '1081', 'interpolation': 'all'}], [390857385, {'block': 'K', 'place': 'Jahangir Puri', 'postcode': '110033', '_inherited': '', 'housenumber': '1251', 'interpolation': 'all'}], [390857389, {'block': 'K', 'place': 'Jahangir Puri', 'postcode': '110033', '_inherited': '', 'housenumber': '1101', 'interpolation': 'all'}], [392695338, {'street': 'Avenida Bernardino de Campos', '_inherited': '', 'housenumber': '12', 'interpolation': 'even'}], [392818527, {'state': 'NSW', 'street': 'Dalmeny Avenue', 'suburb': 'Rosebery', 'postcode': '2018', '_inherited': '', 'housenumber': '49', 'interpolation': 'odd'}], [392818528, {'state': 'NSW', 'street': 'Dalmeny Avenue', 'suburb': 'Rosebery', 'postcode': '2018', '_inherited': '', 'housenumber': '61', 'interpolation': 'odd'}], [392818529, {'state': 'NSW', 'street': 'Dalmeny Avenue', 'suburb': 'Rosebery', 'postcode': '2018', '_inherited': '', 'housenumber': '35', 'interpolation': 'odd'}], [392818530, {'state': 'NSW', 'street': 'Dalmeny Avenue', 'suburb': 'Rosebery', 'postcode': '2018', '_inherited': '', 'housenumber': '19', 'interpolation': 'odd'}], [392818531, {'state': 'NSW', 'street': 'Dalmeny Avenue', 'suburb': 'Rosebery', 'postcode': '2018', '_inherited': '', 'housenumber': '5', 'interpolation': 'odd'}], [392818532, {'state': 'NSW', 'street': 'Dalmeny Avenue', 'suburb': 'Rosebery', 'postcode': '2018', '_inherited': '', 'housenumber': '79', 'interpolation': 'odd'}]]
Traceback (most recent call last):
  File "/usr/local/bin/nominatim", line 14, in <module>
    exit(cli.nominatim(module_dir='/usr/local/lib/nominatim/module',
  File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 264, in nominatim
    return parser.run(**kwargs)
  File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 126, in run
    return args.command.run(args)
  File "/usr/local/lib/nominatim/lib-python/nominatim/clicmd/index.py", line 58, in run
    indexer.index_by_rank(args.minrank, args.maxrank)
  File "/usr/local/lib/nominatim/lib-python/nominatim/indexer/indexer.py", line 178, in index_by_rank
    total += self._index(runners.InterpolationRunner(analyzer), 20)
  File "/usr/local/lib/nominatim/lib-python/nominatim/indexer/indexer.py", line 234, in _index
    runner.index_places(pool.next_free_worker(), part)
  File "/usr/local/lib/nominatim/lib-python/nominatim/db/async_connection.py", line 201, in next_free_worker
    return next(self.free_workers)
  File "/usr/local/lib/nominatim/lib-python/nominatim/db/async_connection.py", line 209, in _yield_free_worker
    if thread.is_done():
  File "/usr/local/lib/nominatim/lib-python/nominatim/db/async_connection.py", line 159, in is_done
    if self.conn.poll() == psycopg2.extensions.POLL_OK:
psycopg2.errors.InternalError_: Splitting of Point geometries is unsupported
CONTEXT:  PL/pgSQL function osmline_update() line 77 at assignment

nominatim=# SELECT place_id, osm_id, startnumber, endnumber, step, address FROM location_property_osmline WHERE place_id in (306317797, 306317971, 306317980, 344487755, 344487756, 344488708, 344489713, 344489715, 383842117, 389701232, 390857384, 390857385, 390857389, 392695338, 392818527, 392818528, 392818529, 392818530, 392818531, 392818532);
 place_id  |   osm_id   | startnumber | endnumber | step |                                                  address
-----------+------------+-------------+-----------+------+-----------------------------------------------------------------------------------------------------------
 306317797 |  554008278 |         217 |       979 |    2 | "interpolation"=>"odd"
 306317971 |  376912728 |         902 |      1028 |    2 | "interpolation"=>"even"
 306317980 |  376912734 |           4 |       462 |    2 | "interpolation"=>"even"
 344487755 | 1105537353 |             |       197 |    2 | "inclusion"=>"actual", "interpolation"=>"odd"
 344487756 | 1105537354 |             |       198 |    2 | "inclusion"=>"actual", "interpolation"=>"even"
 344488708 | 1105542496 |             |        97 |    2 | "inclusion"=>"actual", "interpolation"=>"odd"
 344489713 | 1105548644 |             |       296 |    2 | "inclusion"=>"actual", "interpolation"=>"odd"
 344489715 | 1105548646 |             |       796 |    2 | "inclusion"=>"actual", "interpolation"=>"odd"
 383842117 | 1124877480 |         489 |      1461 |    1 | "street"=>"Avenida Morumbi", "inclusion"=>"estimate", "interpolation"=>"all"
 389701232 | 1149390868 |             |       700 |    1 | "street"=>"Rua Salomé Queiroga", "postcode"=>"03434-000", "inclusion"=>"estimate", "interpolation"=>"all"
 390857384 | 1152877144 |        1052 |      1080 |    1 | "interpolation"=>"all"
 390857385 | 1152877145 |        1252 |      1284 |    1 | "interpolation"=>"all"
 390857389 | 1152877152 |        1102 |      1139 |    1 | "interpolation"=>"all"
 392695338 |  321394425 |             |       402 |    2 | "interpolation"=>"even"
 392818527 | 1159049370 |             |           |      | "interpolation"=>"odd"
 392818528 | 1159049371 |             |           |      | "interpolation"=>"odd"
 392818529 | 1159049372 |             |           |      | "interpolation"=>"odd"
 392818530 | 1159049373 |             |           |      | "interpolation"=>"odd"
 392818531 | 1159049374 |             |           |      | "interpolation"=>"odd"
 392818532 | 1159049375 |             |           |      | "interpolation"=>"odd"
(20 rows)

SELECT count(*) FROM location_property_osmline WHERE startnumber IS NULL and endnumber IS NOT NULL;
   217

On other servers it's 212.

SELECT count(*) FROM location_property_osmline where startnumber is NOT NULL and endnumber is NULL;
     0

Concentrating on one street https://www.openstreetmap.org/way/321394425

SELECT nodes FROM planet_osm_ways WHERE id = 321394425;
                                                                                                      nodes
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 {2537172992,10778202927,3281370112,3281370103,3281370139,10778943965,10778943964,3281370106,10776573766,10778932134,10763637214,10763637218,10778932132,10763637213,3281374561,3281370111,3281374574,3281370115}
(that's 18 ids)

SELECT osm_id, address, indexed_date, ST_AsText(geometry)
FROM placex, planet_osm_ways
WHERE osm_type = 'N' and osm_id = any(nodes) and id = 321394425;
   osm_id   |                            address                             |        indexed_date        |           st_astext
------------+----------------------------------------------------------------+----------------------------+--------------------------------
 2537172992 | "street"=>"Avenida Bernardino de Campos", "housenumber"=>"12"  | 2023-04-03 05:12:23.216256 | POINT(-46.7693884 -22.7055954)
 3281370103 | "street"=>"Avenida Bernardino de Campos", "housenumber"=>"404" | 2023-04-03 05:12:23.216256 | POINT(-46.77254 -22.7078928)
 3281370106 | "street"=>"Avenida Bernardino de Campos", "housenumber"=>"512" | 2023-04-03 05:12:20.206303 | POINT(-46.7734441 -22.7083393)
 3281370111 | "street"=>"Avenida Bernardino de Campos", "housenumber"=>"840" | 2023-04-03 05:12:23.216256 | POINT(-46.7757961 -22.7100661)
 3281370112 | "street"=>"Avenida Bernardino de Campos", "housenumber"=>"86"  | 2023-04-03 05:12:23.216256 | POINT(-46.7699327 -22.7059685)
 3281370115 | "street"=>"Avenida Bernardino de Campos", "housenumber"=>"950" | 2023-04-03 05:12:23.216256 | POINT(-46.7767706 -22.710945)
(6 rows)

lonvia · 2023-04-05T14:04:52Z

Horribly broken interpolation lines in Taiwan: https://www.openstreetmap.org/way/780098768 Oddly enough they have been horribly broken for a while. I'm not sure why they would fail now.

It also looks like this is a problem with Postgis <= 3.2 only.

lonvia · 2023-04-06T10:06:47Z

Possible workaround for people using Nominatim 4.2x (including 4.1.99) is to enable forward dependencies. This is only for people who know what they are doing:

stop updates
create necessary indexes (this will take quite a while)

CREATE OR REPLACE FUNCTION public.planet_osm_index_bucket(bigint[]) RETURNS bigint[]
  LANGUAGE sql IMMUTABLE AS $function$ SELECT ARRAY(SELECT DISTINCT unnest($1) >> 5) $function$;
CREATE INDEX planet_osm_ways_nodes_bucket_idx ON planet_osm_ways USING gin (planet_osm_index_bucket(nodes))                                WITH (fastupdate=off);
CREATE INDEX planet_osm_rels_parts_idx ON planet_osm_rels USING gin (parts) WITH (fastupdate=off);
ANALYZE planet_osm_ways;

enable forward dependecies by adding NOMINATIM_UPDATE_FORWARD_DEPENDENCIES=yes to your settings in .env.

amandasaurus · 2023-04-24T07:09:37Z

Thanks, this seems to have fixed my problem. 🙂

lonvia mentioned this issue Apr 7, 2023

Fix a number of corner cases with interpolation splitting #3030

Merged

lonvia closed this as completed in #3030 Apr 7, 2023

MichaelKorn mentioned this issue Apr 12, 2023

Supervisor integration mediagis/nominatim-docker#318

Open

MrFiestaxxl mentioned this issue Apr 13, 2023

Issue with importing a item mediagis/nominatim-docker#443

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update failed with: `psycopg2.errors.InternalError_: Splitting of Point geometries is unsupported` #3022

Update failed with: `psycopg2.errors.InternalError_: Splitting of Point geometries is unsupported` #3022

amandasaurus commented Apr 3, 2023

lonvia commented Apr 3, 2023

amandasaurus commented Apr 4, 2023

L3o-pold commented Apr 5, 2023

mtmail commented Apr 5, 2023

lonvia commented Apr 5, 2023

lonvia commented Apr 6, 2023

amandasaurus commented Apr 24, 2023

Update failed with: psycopg2.errors.InternalError_: Splitting of Point geometries is unsupported #3022

Update failed with: psycopg2.errors.InternalError_: Splitting of Point geometries is unsupported #3022

Comments

amandasaurus commented Apr 3, 2023

lonvia commented Apr 3, 2023

amandasaurus commented Apr 4, 2023

L3o-pold commented Apr 5, 2023

mtmail commented Apr 5, 2023

lonvia commented Apr 5, 2023

lonvia commented Apr 6, 2023

amandasaurus commented Apr 24, 2023

Update failed with: `psycopg2.errors.InternalError_: Splitting of Point geometries is unsupported` #3022

Update failed with: `psycopg2.errors.InternalError_: Splitting of Point geometries is unsupported` #3022