Skip to content

Latest commit

 

History

History
203 lines (172 loc) · 13 KB

README.rst

File metadata and controls

203 lines (172 loc) · 13 KB
Raw

Bike Collision Data Analytics

This repository contains a set of Python modules that process and analyze public accident report data from the Lincoln (Nebraska) Police Department in order to gather statistics about collisions that involve a bicycle.

This README documents the code. The results of the data can be found at http://stpierre.github.io/crashes/.

The CLI is divided into several subcommands that should be run in order:

  • fetch downloads the raw PDF accident reports from LPD;
  • jsonify extracts key data from the PDFs and generates a single (large) JSON document containing those data;
  • curate assists in manually categorizing all of the potentially-relevant accident reports;
  • geocode assists in manually geocoding (i.e., determining the exact location) of curated accident reports;
  • graph generates useful graphs of the curated data;
  • results generates the input data for the explanation of results.

Running any command before its "parent" commands have been run will automatically invoke the prerequisite commands.

Each command is described in more detail below.

fetch

fetch downloads the raw PDF accident reports from LPD. It does so by searching for reports by date, then screen-scraping the search results and downloading each report. It sleeps randomly between requests to avoid a DoS, or even the appearance of a DoS.

PDFs are saved to datadir.

jsonify

jsonify extracts data that we care about from the PDFs so that the data are more usable. It is very computationally expensive, and so runs multiple processes. (By default, it spawns one process per CPU.) jsonify extracts four data:

  • location: The location of the accident. Often this is just a street name, and the accident report must be read to find a specific location.
  • date: The date of the accident.
  • time: The time of the accident
  • report: The full text of the accident description.
  • injury_severity: The severity of the injury to the cyclist using LPD's bespoke 1-5 scale, where 1 is "Killed" and 5 is "No injury." 5's are rarely (if ever) reported.
  • injury_region: Body region of primary injury to the cyclist.
  • cyclist_dob: Date of birth of the cyclist.

Additional data can be added and (fairly) easily added to the data set with the --reparse-curated flag.

As it turns out, PDFs in general and the accident report PDFs specifically are an appalling disaster, so this parsing is decidedly crufty. RTFS at your peril.

jsonify takes a few(optional) arguments:

  • --processes can be used to specify the number of processes to spawn, in case you don't want to melt your CPU.
  • --reparse-curated tells jsonify to only parse those accident reports that have already been curated and identified as bike-car collisions.
  • Any additional arguments are filenames to parse, which will be used instead of trying to parse all of the PDFs in the datadir.

If filenames are supplied to jsonify, the results are printed to stdout instead of added to reports.json. This is mostly useful for testing changes to the jsonify code.

curate

Once the data have been extracted, we must find collisions that involved a bicycle. This is, unfortunately, a manual process. curate iterates over every accident report that includes the word bicycle, bike, cyclist, or bicyclist. Each report is then manually assigned one of five statuses:

  • crosswalk (C): Collision happened while a person on a bicycle was using a crosswalk.
  • sidewalk (S): Collision happened while a person on a bicycle was riding on a sidewalk. For instance, a car entering or leaving a private driveway or, in extreme situations, a car that jumps the curb.
  • road (R): Collision happened while a person on a bicycle was riding on the road, excluding intersections.
  • intersection (I): Collision happened while a person on a bicycle was riding through an intersection on the road, not using a crosswalk.
  • elsewhere (E): Collision happened elsewhere. This also includes collisions that happened on the road, but where the cyclist was not riding on the road as such. (E.g., the cyclist was crossing the street away from a crosswalk.)
  • not_involved (N): Bicycle was not involved in the collision. A cyclist may have been a witness, or a bike rack damaged, etc.

geocode

After the data has been curated, we want to geocode the bike-related collisions in order to map them; this command assists with that semi-manual process. The "Location" field on accident reports is frequently ambiguous or incomplete, so geocode iterates over each bike-related accident and attempts to use the "Location" field as provided on the report, plus any user input necessary, to look up the exact location of the accident (using the Google Geocoding API) and output GeoJSON to be used in mapping.

locate

After the data has been geocoded, this command can help find particularly interesting collisions by location. Currently it's hardcoded to find sidewalk and crosswalk collisions near bike paths. Bikeway data is taken from https://github.com/stpierre/lincoln-bike-routes

xform

Transform data so that we can produce pretty graphs of the data.

results

Render a template that includes an explanation of the results in long form. Currently that template is a Jinja2 template, so Jinja2 must be run to generate the final site.

Configuration

The following configuration options (in crashes.conf) are recognized:

Section Name Description Default
form url The POST URL of LPD's accident report search form. HTTP://CJIS.LINCOLN.NE.GOV/HTBIN/CGI.COM
form token The POST token to include in accident report search POSTs. DISK0:[020020.WWW]ACCDESK.COM
form sleep_min Minimum time, in seconds, to sleep between requests to LPD's website. 5
form sleep_max Maximum time, in seconds, to sleep between requests to LPD's website. 30
fetch days Days of accident report data to download. 365
fetch start Date (in YYYY-MM-DD format) from which to download collision data. If start is given, it takes precedence over days. None
fetch retries Number of times to retry an HTTP request to LPD's website, either for submitting the search form or for downloading a report. 3
files datadir Base directory to use for persistent data storage. ./data
files pdfdir Directory, relative to datadir, where accident report PDFs will be stored. pdfs
files all_reports File, relative to datadir, where the results of the jsonify command will be stored. reports.json
files curation_results File, relative to datadir, where the results of the curate command will be stored. curation.json
files geocoding Directory, relative to datadir, where output from the geocode command will be stored. geojson
files imagedir Directory, relative to datadir, where graph images will be stored. images
files template Jinja2 template for results. ./results.html
files results_output Filename to write results output to. ./index.html
files bike_route_geojson Path to a GeoJSON file containing all known bikeways. None
files lb716_results File, relative to datadir, Where the results of the locate command will be stored. lb716.json