Core repo for election results data acquisition, transformation and output.
OpenElections core is developed and tested using Python 2.7.*. The package might not work with older or newer Python distributions.
You'll:
- set up a virtual environment
- fork/clone this repository, install dependencies
- add any optional configuration details you need (e.g. Mongo or AWS)
You should use virtualenv and virtualenvwrapper to work on Open Elections inside a virtualized development environment.
The easiest way is to install these tools system-wide with pip
(you may need to use sudo
):
$ pip install virtualenv
$ pip install virtualenvwrapper
Then, to make a virtual environment called openelex
for open elections work:
$ mkvirtualenv openelex
If your terminal is unable to find mkvirtualenv
, you may need to add the virtualenvwrapper.sh
script to your .bashrc
(probably in /usr/bin/
or /usr/local/bin
):
$ echo 'source "/usr/bin/virtualenvwrapper.sh"' >> ~/.bashrc
This will automatically activate the openelex
environment. To turn it on for future sessions:
$ workon openelex
Fork this repo by hitting the "Fork" button above, and clone your fork to your computer:
$ git clone https://github.com/[my_github_user]/openelections-core.git
$ cd openelections-core
Turn on your virtual environment from the previous step, if you haven't already:
$ workon openelex
Then install the openelex package in editable mode
$ pip install -e .
Create a settings.py
file.
$ cp settings.py.tmplt settings.py
At the very least, you'll want to make sure the values in the MONGO
variable work for the way you've installed and configured MongoDB on your system.
You can put this settings file anywhere on your filesystem. You'll need to set the OPENELEX_SETTINGS
environment variable to the absolute path to the settings.py
file that you created.
You'll probably want to add a line to the postactivate
script ($VIRTUAL_ENV/bin/postactivate
) in your virtualenv to set the OPENELEX_SETTINGS
environment variable.
Test it out by running openelex --help
, you should see something like:
$ openelex --help
Usage: openelex [OPTIONS] COMMAND [ARGS]...
Options:
--help Show this message and exit.
Commands:
bake.election_file Write election and candidate data with on...
bake.results_status_json Output a JSON file describing available...
bake.state_file Write election and candidate data along
with...
cache.clear Delete files in state cache diretory
cache.files List files in state cache diretory
datasource.elections List elections for a state.
datasource.filename_url_pairs List mapping of standard filenames to
source...
datasource.mappings List metadata mappings for a state
datasource.target_urls List source data urls for a state
fetch Scrape data files and store in local file...
load.run Load cached data files into the database
load_metadata.run Populate metadata in database from fixture...
publish Publish baked result files
shell Open a Python shell, bootstrapping the...
transform.list Show available data transformations
transform.reverse Reverse a previously run transformation
transform.run Run data transformations
validate.list Show available validations for state
validate.run Run data validations for state
Running commands looks something like this:
$ openelex cache.clear --state=NY
0 files deleted
0 files still in cache
You can also get help on particular commands, e.g. openelex --help cache.clear
.
If you are going to load results, you need to configure MongoDB connections in
openelex/settings.py
.
If you are a core contributor who needs to publish baked results to one of the GitHub repositories, you will need to define further settings.
To set GitHub credentials, you must first create a personal access token and then uncomment and set these values in openelex/settings.py
:
GITHUB_USERNAME = ''
GITHUB_ACCESS_TOKEN = ''
You only need to install MongoDB if you plan to write data loaders.
To store your data in MongoDB, you need only install Mongo. The default configuration should auto-create the databases and tables you need, as you need them.
You only need to do this if you plan to write data loaders or transforms.
$ cd openelex
$ openelex load_metadata.run --collection=office
$ openelex load_metadata.run --collection=party