Error During Import Process with Pelias for OSM and Other Data Sources

[Team-Cpptr]

Getting this error at import process when execute pelias import osm or other data sources as well

VirtualBox:~/Desktop/docker/projects/maha$ pelias import osm
info: [openstreetmap] Creating read stream for: /data/openstreetmap/india-latest.osm.pbf
/code/pelias/openstreetmap/node_modules/pelias-blacklist-stream/parser.js:11
    throw new Error( 'file not found' );
    ^

Error: file not found
    at load (/code/pelias/openstreetmap/node_modules/pelias-blacklist-stream/parser.js:11:11)
    at /code/pelias/openstreetmap/node_modules/pelias-blacklist-stream/loader.js:26:48
    at Array.map (<anonymous>)
    at loader (/code/pelias/openstreetmap/node_modules/pelias-blacklist-stream/loader.js:26:31)
    at Object.blacklistStream (/code/pelias/openstreetmap/node_modules/pelias-blacklist-stream/index.js:22:15)
    at Object.streams.import (/code/pelias/openstreetmap/stream/importPipeline.js:29:20)
    at Object.<anonymous> (/code/pelias/openstreetmap/index.js:12:22)
    at Module._compile (node:internal/modules/cjs/loader:1198:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1252:10)
    at Module.load (node:internal/modules/cjs/loader:1076:32)

following is the modified pelias.json file for India region

{
  "logger": {
    "level": "info",
    "timestamp": false
  },
  "esclient": {
    "apiVersion": "7.5",
    "hosts": [
      { "host": "elasticsearch" }
    ]
  },
  "elasticsearch": {
    "settings": {
      "index": {
        "refresh_interval": "10s",
        "number_of_replicas": "0",
        "number_of_shards": "1"
      }
    }
  },
  "acceptance-tests": {
    "endpoints": {
      "docker": "http://api:4000/v1/"
    }
  },
  "api": {
    "services": {
      "placeholder": { "url": "http://placeholder:4100" },
      "pip": { "url": "http://pip:4200" },
      "interpolation": { "" },
      "libpostal": { "url": "http://libpostal:4400" }
    }
  },
  "imports": {
    "adminLookup": {
      "enabled": true
    },
    "geonames": {
      "datapath": "/data/geonames",
      "countryCode": "IN"
    },
    "openstreetmap": {
      "download": [
        { "sourceURL": "https://download.geofabrik.de/asia/india-latest.osm.pbf" }
      ],
      "leveldbpath": "/tmp",
      "datapath": "/data/openstreetmap",
      "import": [{
        "filename": "india-latest.osm.pbf"
      }]
    },
    "openaddresses": {
      "datapath": "/data/openaddresses",
      "files": []
    },
    "polyline": {
      "datapath": "/data/polylines",
      "files": [ "extract.0sv" ]
    },
    "whosonfirst": {
      "datapath": "/data/whosonfirst",
      "countryCode": "IN",
      "importPlace": [
        "85688753"
      ]
    },
    "transit": {
      "datapath": "/data/transit",
      "feeds": []
    }
  }
}

Also, do I need to make any configuration changes for it?

Is there any existing template configuration of Pelias for the India region?

[Team-Cpptr]

Additional at pelias prepare polylines phase it's failed to deal with larger pbf file of India(~1.3GB).

In order to deal with larger files, do we need to change anywhere configuration settings?

[arnesetzer]

1. Is there a valid .osm.pbf file in `data/openstreetmap/india-latest.osm.pbf``?
2. Did you change anything on the mounting settings in the docker-compose.yml?
3. Regarding the screenshot for interpolation: See https://github.com/pelias/polylines#download-data and use valhalla or missinglinks polyline generator to generate the polylines on your own. Technically you could bash into the polyline container and modify https://github.com/pelias/polylines/blob/a2f59be9f4bdfc4fd0f47308ce01b62f99130b07/docker_extract.sh#L25 to let it run larger files but then you are on your own if this crashes.

[billypap1]

@arnesetzer I have the same issue, how can I bash into the polyline container?

[arnesetzer]

Ah, I forgot that the polyline image is not runnning as a daemon in the background all the time. In this case you could build a modified image yourself or mount a modified docker_extract.sh

But I still don't recommend this. Usually warnings like this have a valid reason.

[Team-Cpptr]

Okay, I'll try that way.

@arnesetzer Additionally, are there any predefined Pelias configuration files (pelias.json and docker-compose) available for the India region? I haven't been able to find them.

[arnesetzer]

Go to the project folder. Copy any europe country project. Replace every URL/ISO 3166 shortcode with the correct url/shortcode for india. Remove openaddress, they have no data for india. Run the steps according to the readme. Please ask follow up questions in the pelias gitter room since they are not related to this issue.

@missinglink @orangejulius Since the mentioned problem already has good documentation and the code returns a solution I propose to close this issue.