8/23/24
Detailed documentation is provided in the following sections:
- Overview- this page
- Database Architecture- technical overview of data structures within the database's three layers (raw, processed, app)
- Web App and Data Visualization Guide- specific details about data prepared for web applications and ad-hoc data visualizations (work in progress)
- ETL Script- overview of the ETL script and how to run it
The Jacksonville Tributary's Legislative Compass analyzes and visualizes roll call voting patterns of Florida state legislators. Key metrics include legislators' party loyalty and congressional district electorates' partisan lean and demographics. This dashboard is intended to support development of a voter guide, reporting on party polarization, and disparities between legislators and the districts they represent.
This repo contains a data pipeline which:
- extracts legislative voting data from LegiScan, and census and demographics data from Daves Redistricting Maps.
- transforms data by organizing and integrating data sources
- reviews data quality to identify and explain anomalies
- loads data into a Postgres database while exporting key tables as .csv files
- prepares data for use by the legislator dashboard web app (see web app repo):
- production web app
- staging web app- currently hosted on Rob's ShinyApps.io account, but may need a new home
This project is a work in progress, with the following work underway as of August 2024:
- superuser journalists at partner outlets review and beta test
- publish the web app for public use
- document data definitions for web app data sources
See changelog.md for info on most recent updates.
Following are definitions of key terminology used throughout this project:
- Party loyalty is a legislator’s tendency to vote with or against their party. 1 = most loyal, 0 = least loyal.
- Partisan lean is a legislative electorate’s partisanship as measured by percentage point difference between voting for Democrats vs. Republicans.
- Party unity is a roll-call-level measure, by party, of how unified roll calls are within parties
(# votes aligned with legislator's own party)
-----------------------------------------
(# votes aligned with either one party or the other)
Calculations for party loyalty are based on classifying and weighing each present (yea or nay) vote. Each vote is filed into one of six categories.
| vote type | descriptino | loyalty weight |
|---|---|---|
| party line bipartisan | voting with the same party's majority in roll calls supported by both parties | N/A |
| party line partisan | voting with the same party's majority in roll calls split on party lines | 1 |
| cross party | voting against the same party's majority in roll calls split on party lines | 0 |
| against both parties | voting against both party majorities | N/A |
| absent nv | absent or no vote | N/A |
| other | legislator's party is split evenly, etc. | N/A |
Following is an overview of files in this repository:
- data-app- data supporting web applications, in csv format
- data-raw- raw data in JSON format, as bulk downloaded from LegiScan's API
- docs- data dictionaries, diagrams, and documentation
- docs- data explorations in Python, informing development of this pipeline
- qa- includes quality assurance log and tables of data anomalies. See QA script at scripts/qa_checks.R.
- scripts- ETL scripts
Figure 1: Overview of ETL pipeline, as described in Database Architecture
For more info about all apps and visualizations, see Web App and Data Visualization Guide.
