Add QAQC tables for Condo Checks #428

mbh329 · 2023-02-09T20:16:35Z

Addresses issue #414.

Changes

Add dcp_housing as in input data source to compare the Certificate of Occupancy # of units residential (most accurate units residential number) in the Housing DB to PLUTO data
Creates new QAQC logic in two separate scripts to generate two separate QAQC tables to be displayed in the app to catch condos with extremely high unit counts
output. NOTE: the qaqc_housing_units table failed to generate because of a datatype error but latest commit implements a fix, a new output will take a few hours to produce. I suggest looking at the qaqc_pts_condo output first until both tables are generated. This action was built off of a minor release of PLUTO.
Add new dataset to source_data_version.csv
AD developed the logic for both QAQC scripts. Implementation is based off other QAQC processes that have already been established in PLUTO

damonmcc · 2023-02-10T15:53:18Z

thoughts so far from looking at qaqc_pts_condo.csv:

since there are only 24 rows with an old_value, does that mean only 24 were de
12,312 total rows, so appears to have a row for all condos?
a description of what each final column represents would be helpful. maybe at the top of the sql file? or maybe more descriptive column names?

AmandaDoyle

See comments for detailed feedback.

AmandaDoyle · 2023-02-12T14:58:19Z

pluto_build/sql/qaqc_housing_units.sql

@@ -0,0 +1,45 @@
+-- Create qaqc table All PLUTO records where there is a match in Housing Database 


Missing "of"
I think this need to be " Create qaqc table of all PLUTO records where there is a match in Housing Database"

AmandaDoyle · 2023-02-12T15:00:02Z

pluto_build/sql/qaqc_housing_units.sql

+-- Create qaqc table All PLUTO records where there is a match in Housing Database 
+-- and the PLUTO residential units value does not match the housing database certificates 
+-- of occupancy value. Have flag indicating if bbl has residential unit correction in manual corrections table.
+-- select PLUTO records that have a match in the HousingDB subset where unitsres does not equal units co


Is this last sentence repetitive of the first 3 lines?

AmandaDoyle · 2023-02-12T15:01:08Z

pluto_build/sql/qaqc_housing_units.sql

+-- Data Dictionary 
+-- bbl - billing BBL
+-- job_number - The DOB job application number assigned when the applicant begins the application. This is the unique identifier for the application submitted to the Department of Buildings (DOB). 
+-- units_res - The number of residential units as reported by dcp_housing database


unitsres is coming from PLUTO

AmandaDoyle · 2023-02-12T15:01:59Z

pluto_build/sql/qaqc_housing_units.sql

+-- units_res - The number of residential units as reported by dcp_housing database
+-- units_co - The number of units listed on the DOB issued Certificate of Occupancy 
+-- new_value 
+-- old_value 


Would adding definitions of what these fields are be useful?

AmandaDoyle · 2023-02-12T15:47:18Z

pluto_build/sql/qaqc_housing_units.sql

+  old_value text
+);
+
+INSERT INTO qaqc_housing_units


There are duplicate records in the output table in DO. This may have to do with the CREATE TABLE IF NOT EXISTS and INSERT INTO statements. I confirmed that the duplicates do not appear when querying the database directly.

AmandaDoyle · 2023-02-12T15:52:15Z

pluto_build/sql/qaqc_pts_condo.sql

+-- primebbl - Billing BBL  
+-- bbl - Unit level BBl
+-- units - The number of units listed by Department of Finance for the property (compare to units_co in dcp_housing)
+-- coop_apts - The number of coop apartments listed for the property 


This is the number of residential units

AmandaDoyle · 2023-02-12T15:57:59Z

pluto_build/sql/qaqc_pts_condo.sql

+-- old_value - The previous number of units as reported by the pluto_correction file
+
+
+CREATE TABLE IF NOT EXISTS qaqc_pts_condo(


There are duplicate records in the output table in DO. This may have to do with the CREATE TABLE IF NOT EXISTS and INSERT INTO statements. I confirmed that the duplicates do not appear when querying the database directly.

AmandaDoyle · 2023-02-12T16:00:49Z

pluto_build/sql/qaqc_housing_units.sql

+-- select only corrections to unitsres field
+corrections_subset as (
+SELECT * 
+FROM pluto_corrections 


We're no longer pulling in pluto_corrections per PR #425 so this needs to be changed.

AmandaDoyle · 2023-02-12T16:01:10Z

pluto_build/sql/qaqc_pts_condo.sql

+-- select only corrections to unitsres field
+corrections_subset as (
+SELECT * 
+FROM pluto_corrections 


We're no longer pulling in pluto_corrections per PR #425 so this needs to be changed.

fvankrieken · 2023-06-05T13:23:00Z

@AmandaDoyle @damonmcc let's check in on this pr this afternoon?

mbh329 added 6 commits February 9, 2023 14:43

add dcp_housing to dataloading

4bbcbc1

add updated source data versions

a827d79

add logic and qaqc table for pts condo

7c71fe8

add qaqc_pts_condo to qaqc and export .sh

e7b0949

add sql script for qaqc_housing_units

ec9b541

add qaqc housing units to qaqc and export shell script

814b2b2

mbh329 added the enhancement New feature or request label Feb 9, 2023

mbh329 requested review from AmandaDoyle and damonmcc February 9, 2023 20:16

concert column to text

51f87f4

mbh329 changed the title ~~WIP: Add QAQC tables for Condo Checks~~ Add QAQC tables for Condo Checks Feb 10, 2023

damonmcc linked an issue Feb 10, 2023 that may be closed by this pull request

Catch incorrect unit totals for condos #414

Open

mbh329 added 3 commits February 10, 2023 11:33

add description of cols

7ca3a4a

add data dict

5d57405

add data dict

d0f4c12

damonmcc approved these changes Feb 10, 2023

View reviewed changes

AmandaDoyle reviewed Feb 12, 2023

View reviewed changes

clean up comments

8b97245

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add QAQC tables for Condo Checks #428

Add QAQC tables for Condo Checks #428

mbh329 commented Feb 9, 2023 •

edited

Loading

damonmcc commented Feb 10, 2023

AmandaDoyle left a comment

AmandaDoyle Feb 12, 2023

AmandaDoyle Feb 12, 2023

AmandaDoyle Feb 12, 2023

AmandaDoyle Feb 12, 2023

AmandaDoyle Feb 12, 2023

AmandaDoyle Feb 12, 2023

AmandaDoyle Feb 12, 2023

AmandaDoyle Feb 12, 2023

AmandaDoyle Feb 12, 2023

fvankrieken commented Jun 5, 2023

		@@ -0,0 +1,45 @@
		-- Create qaqc table All PLUTO records where there is a match in Housing Database

		-- old_value - The previous number of units as reported by the pluto_correction file


		CREATE TABLE IF NOT EXISTS qaqc_pts_condo(

Add QAQC tables for Condo Checks #428

Are you sure you want to change the base?

Add QAQC tables for Condo Checks #428

Conversation

mbh329 commented Feb 9, 2023 • edited Loading

damonmcc commented Feb 10, 2023

AmandaDoyle left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fvankrieken commented Jun 5, 2023

mbh329 commented Feb 9, 2023 •

edited

Loading