Import UK postcodes from ONSPD instead of Code-Point Open

[h-lame]

Code-Point Open only contains live Postcodes for GB and we need ONSPD to get the postcodes for Northern Ireland and Crown Dependencies. As well as the NI and CD postcodes the ONSPD also contains all the postcodes that Code-Point Open has and all the terminated postcodes. This PR introduces a new mapit_UK_import_onspd command that can be used to import all of these postcodes instead of using a mixture of Code-Point Open and ONSPD.

Because the ONSPD now contains only GSS codes we also introduce a new mapit_UK_import_onspd_ni_areas command to replace the existing mapit_UK_import_nspd_ni_areas command. The new command a new set of fixtures that generates the most recent (April 2015) Northern Ireland areas for: Councils (LGD), Electoral Areas (LGE), Wards (LGW), Westminister Constituencies (WMC), and Northern Irish Assembly Constituencies (NIE). These areas, like those created by the old command, do not have boundary data.

The new mapit_UK_import_onspd importer has lots of options to configure which postcodes to import from the dataset and to allow it to be used as a drop in replacement for the various old commands if required. The default behaviour with no options is to import only what the mapit_UK_import_codepoint command would: live GB postcodes with locations. The configuration options are:

• --allow-terminated-postcodes - supplying this option imports terminated postcodes as well as live ones
• --allow-no-location-postcodes - supplying this option imports postcodes with no location information
• --crown-dependencies - this option controls how to import postcodes from Crown Dependencies
• --northern-ireland - this option controls how to import postcodes from Northern Ireland

The last two accept "include" (to mean import these postcodes), "exclude" (to mean don't import these postcodes), or "only" (to mean only import these postcodes, ignoring all others) as values, with "exclude" being the default.

We don't remove the mapit_UK_import_codepoint importer, people may still want to use it afterall - it's a smaller dataset and if you don't want terminated, NI, or CD postcodes, it may still be useful. We do however make the scilly command work with either Code-Point Open or ONSPD files to work its magic.

Documentation for importing the UK includes these lines currently (see: http://mapit.poplus.org/docs/self-hosted/import/uk/):

./manage.py mapit_UK_import_codepoint ../data/Code-Point-Open/*.csv
./manage.py mapit_UK_scilly ../data/Code-Point-Open/tr.csv
./manage.py mapit_UK_import_nspd_ni_areas
./manage.py mapit_UK_import_nspd_ni ../data/ONSPD.csv
./manage.py mapit_UK_import_nspd_crown_dependencies ../data/ONSPD.csv

With this new importer we can replace it with:

./manage.py mapit_UK_import_onspd_ni_areas
./manage.py mapit_UK_import_onspd --northern-ireland=include --crown-dependencies=include ../data/ONSPD.csv
./manage.py mapit_UK_scilly ../data/ONSPD.csv

Or, if we wanted to import everything from ONSPD:

./manage.py mapit_UK_import_onspd_ni_areas
./manage.py mapit_UK_import_onspd --allow-terminated-postcodes --allow-no-location-postcodes --northern-ireland=include --crown-dependencies=include ../data/ONSPD.csv
./manage.py mapit_UK_scilly ../data/ONSPD.csv

Some more detail (for example where the data comes from for the new NI fixtures) can be found in the commit messages.

[h-lame]

The commits here tell the whole story, in particular that I started with many separate commands and then merged them after feedback in irc. Let me know if you'd like things squashed to hide the "miss-step".

[dracos]

I think the indent is wrong here, though it has no material effect :)

[h-lame]

Oops! I reckon I must have rebase-awayed the else: part.

[dracos]

Thanks for this, looks really good :-) As you have now spotted OSNI have released ward etc boundary data as OpenData, so the NI aspects of this can probably be much simplified! I guess it might be worth doing something with that first rather than later? I think almost all of this can remain the same, it just wouldn't need to add anything to areas in the post_row any longer (as the boundaries would do it for you), and we'd need to adjust the area script to actually load in the boundaries instead of working things out.

On your last point, yes please do squash it however you want :) Perhaps we could for now ignore the boundary things and start with 3a4d450 and then all the ONSPD postcode import script stuff (which works assuming boundaries are okay, as I understand it), and then have new NI stuff on top.

[h-lame]

Apologies for the delay. I've updated the PR to squash the work somewhat and incorporate importing boundaries for NI areas from the OSNI data available under open government licence from http://osni.spatial-ni.opendata.arcgis.com/. This data is (currently) missing GSS codes for NI Electoral Areas (LGE) so we still use the ni-electoral-areas-2015.csv fixture data to do name->GSS matching.

Would there be interest in me providing the specific datafiles I used to write the osni importer so they can be mirrored on http://parlvid.mysociety.org/os/ ? The software powering http://osni.spatial-ni.opendata.arcgis.com/ means you can't just link directly to a download because they're generated on the fly (and then cached for a bit I guess). A given download URL may return a JSON payload telling you how the download generation is going instead of the requested download itself. The URLS also aren't persistent - as the datasets are re-released with fixes the uuid/sha in the urls changes.

[h-lame]

FWIW - this PR no longer includes the ability to import boundary-less NI areas from the new fixtures, and no longer supports directly assigning these shapeless NI Areas to NI postcodes in the ONSPD importer (I do still have those versions in another branch if we wanted to keep them though). I haven't removed the old NI importers though, you may still want them for importing NI postcodes/ areas for historical datasets.

[dracos]

Wow, thanks :) No need to apologise! Good work on the random SRID stuff…

If someone (cough, us) has already got Northern Ireland areas in our database, from LGW up to NIE, is there a way we can add the boundaries from this data without creating new areas (given they aren’t new areas, they’re the same)? Could there be a control file provided that would do name matching, that people (cough, us) could then use? :)

"Would there be interest in me providing the specific datafiles I used to write the osni importer so they can be mirrored on http://parlvid.mysociety.org/os/ ?” - Yes please, the OS site behaves the same and we grab copies from there and mirror them as you suggest.

[h-lame]

I think this latest version covers everything.

1. Instead of a control file for the OSNI import I added a script to go through the ni-electoral-areas-2015.csv and do name lookups on any existing NI areas (EUR, LGD, LGW, LGE only) to add GSS codes (mapit_UK_add_gss_codes_to_ni_areas in 822f443). All we can do is a name lookup and this might not be particularly useful depending on the existing data because...
2. The OSNI data sources have names that don't include punctuation and in some cases are completely different (e.g. "East Coast" vs. "North Down and Ards"). To help with this I added a script to go through the ni-electoral-areas-2015.csv and do GSS code lookups on the same areas and add any missing names as overrides (mapit_UK_add_names_to_ni_areas in 2617420). Of course this can only do GSS lookup so there's something of a chicken+egg problem for those who already have areas, but no boundaries and no GSS codes.
3. On IRC we talked about the datasets question and there's now a mirror of the data I used at http://parlvid.mysociety.org/os/osni-dec-2015.tar.gz
4. Had to allow for specifying the SRID of each OSNI shapefile as it can change depending on the release / download. Gory details in 645a722.

Is the documentation available as a collaborative site? There's enough change here that http://mapit.poplus.org/docs/self-hosted/import/uk/ would become out of date and it would be remiss of me if I didn't offer to help update that too.

[dracos]

It’s actually Ards and North Down, they didn’t revert :-) http://www.belfasttelegraph.co.uk/news/politics/inside-the-truly-bonkers-world-of-our-new-super-councils-30968587.html

Your script in 1 works fine on our MapIt, great, apart from a couple of tweaks, given below (I guess the Ards one should be changed in the CSV). I’ve gone for the direct name entry as for some reason our current NI areas don’t have anything in names; name should always be set, so that should be fine. I’ve got it do a startswith match if the exact fails, which catches the council names ending in District/Borough/City Council.

The docs site is at https://github.com/mysociety/mapit.poplus.org :)

$ diff mapit_gb/management/commands/mapit_UK_add_gss_codes_to_ni_areas.py{O,}
83a84,89
>         area_name = area_name.replace('St. ', 'St ')
>         if area_type == 'LGD':
>             area_name = area_name.replace('Derry and', 'Derry City and')
>             area_name = area_name.replace('Armagh, ', 'Armagh City, ')
>             area_name = area_name.replace('North Down and Ards', 'Ards and North Down')
> 
87c93
<                 names__name__iexact=area_name,
---
>                 name__iexact=area_name,
90a97,107
>         except Area.DoesNotExist:
>             try:
>                 area = area_source.get(
>                     country=self.country, type__code=area_type,
>                     name__istartswith=area_name,
>                     generation_low__lte=self.current_generation,
>                     generation_high__gte=self.current_generation
>                 )
>             except Area.DoesNotExist:
>                 return None, False
>         else:
95,96d111
<         except Area.DoesNotExist:
<             return None, False

[h-lame]

Thanks for trying that out. I thought it might need tweaking to work correctly on your data, so I'll fold in your change - it's ultimately for your install so it's best if it reflects your data.

I'll update the CSV too in a new commit for "Ards and North Down" and I'll check the names of all of them just in case (hope that wikipedia is up to date ;).

[h-lame]

Updated the csv with names that include the council type for LGDs. Also added "City" where appropriate and changed "North Down and Ards" to "Ards and North Down" so I didn't fold in these lines:

>         if area_type == 'LGD':
>             area_name = area_name.replace('Derry and', 'Derry City and')
>             area_name = area_name.replace('Armagh, ', 'Armagh City, ')
>             area_name = area_name.replace('North Down and Ards', 'Ards and North Down')

from your diff as I reckon they'd be redundant now.

[h-lame]

Added one (hopefully) final commit to allow running some of the new scripts on first import. Checking for objects in the right generation was always returning None because we look for something no higher than the 0th generation (which doesn't exist).

[h-lame]

Disappointingly I expect another commit (or rebase) tomorrow - some checks locally show me that the NI shape data has imported into the wrong place. It's dropped it all on top of Liverpool / North Wales. I suspect I need to do more SRID/projection manipulation to get it into the correct place.

[h-lame]

Ok. This last commit 0730197 should finally close this off as ready to review/merge/etc...

Details in the commit but in summary because the rest of the shape data for UK is in 27700, and the default SRID for the instance is 27700 it makes sense to transform the NI shape areas to 27700 for storage. If we don't do this then when the geometry is retrieved from the DB it reports as being in 27700 without having actually being transformed from whatever SRID is is in, so ultimately is wrong (e.g. the NI shapes end up being over Liverpool/North Wales).

Apologies for the churn here. I've done more local testing and visualising and am way more confident that I'm actually done now.

[dracos]

I have cherry-picked b2d1d6d 74aa673 7f0f41e 8dafb12 32d772e and 99847c4 (the first five commits and the ni-electoral-areas-2015.csv fixup) into master (I word wrapped and tweaked a few of the commit messages (e.g. removed an unused footnote from one) but otherwise made no changes). If you want to rebase the rest of this PR on top of that, feel free, otherwise I'll continue merging as time allows, thanks :-)

[dracos]

Sigh, Safari ate my comment :-/ This is all now rebased/squashed/merged, thanks very much for your help and patience. Using diff <(git diff master...alphagov/upstream-onspd-importers mapit_gb) <(git diff bb2bc4a..upstream-onspd-importers mapit_gb) you can see I've made the following changes which you might want to look at:

• Changed some print to self.stdout.writes (helped with testing with piping without encoding issues)
• Stopped a "St." creeping in via add_names_to_ni_areas ;)
• Allowed the two add_*_to_ni_areas scripts to be run without needing a new generation (as I've run them first on our MapIt), just using the latest generation, active or not
• Added a name match of NIE areas in import_osni so that our existing NIE areas can be used, without having to create new ones (this last one as a new commit, the rest squashed in).

This is all now present on mapit.mysociety.org too, e.g. http://mapit.mysociety.org/area/16985.html has a boundary for the first time ever, hooray! I'm just going through and checking the manually created postcode–area lookups match before dropping them (it doesn't affect the API call but they do duplicate on the HTML page). Thanks again, ask if you have any questions.

[h-lame]

Brilliant, thanks @dracos! If we have more changes in the future we'll do our best to make them smaller and so they're easier to incorporate, rather than an single massive "change everything plz" PR like this one.

[timwis]

Forgive me if I'm misunderstanding, but doesn't this suggest MapIt will return results for terminated postcodes? I'm getting no results for G41 5PF, which is in the ONSPD dataset and existed from 1990-1992 (the good ol' days, of Bryan Adams, Mariah Carey, and Boyz II Men yore).

[h-lame]

Forgive me if I'm misunderstanding, but doesn't this suggest MapIt will return results for terminated postcodes? I'm getting no results for G41 5PF, which is in the ONSPD dataset and existed from 1990-1992 (the good ol' days, of Bryan Adams, Mariah Carey, and Boyz II Men yore).

Hi @timwis - it might depend on the import arguments fo the mapit instance you're using as allowing terminated postcodes is an optional feature, specified with --allow-terminated-postcodes when importing. If it's the public mapit instance maybe other MySociety folks can confirm what options were used, and if you should be seeing terminated postcodes.

[timwis]

@h-lame thanks for the quick reponse! We are indeed using the public mapit instance, so it would be great to know whether terminated postcodes are included in that one.

[dracos]

Umm, just checking - it doesn't look like we import terminated postcodes on mapit.mysociety.org at present, but also have not e.g. deleted postcodes that became terminated during the lifespan of the site, so it's not exactly consistent. I can't see a reason not to include older ones in that case, unless there's some overwhelming number of them, will look to include them in the next import.

[timwis]

Thanks @dracos. According to the ONSPD documentation (Table 4 / Page 29), there are 872,377 terminated postcodes in the UK (including Norther Ireland, Channel Islands, etc.). Does that sound reasonable to include? Not sure the exact number of records overall in ONSPD, but the full csv file has 2,632,805 lines, so assuming that's the number of records, terminated postcodes represent 33% of it.

[chris48s]

Keeping terminated postcodes around might not be as useful as you think it is. I don't really know the full mechanics of it, but a large number of them end up being assigned to Royal Mail sorting offices so your centroid point ends up not describing the centroid of the group of properties the postcode used to describe before it was terminated, but the location of the nearest Royal Mail sorting office. It might depend on the postcode type. To give a couple of examples:

• CB21 5RB used to be somewhere around here https://www.google.co.uk/maps/place/Balsham,+Cambridge+CB21+5RB/ but now its assigned to a RM sorting office https://mapit.mysociety.org/postcode/CB215RB.html
• CB5 8ZJ used to be somewhere around here https://www.google.co.uk/maps/place/Cambridge+CB5+8ZJ/ but now its assigned to the same RM sorting office https://mapit.mysociety.org/postcode/CB58ZJ.html

(just for the lolz, have a look how many different postcodes in the mapit DB have a point somewhere in that one sorting office).

Broadly this situation already exists in mapit for the reason you've described (tbh I've always assumed you just import terminated postcodes because of the number that exist). I guess it depends on your viewpoint whether you see this as a problem or a desirable quality, but its worth thinking about if you're making a decision on what to do about terminated postcodes..