Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add base sql to schema repo #133

Merged
merged 1 commit into from
Mar 6, 2024
Merged

Add base sql to schema repo #133

merged 1 commit into from
Mar 6, 2024

Conversation

jenningsanderson
Copy link
Collaborator

@jenningsanderson jenningsanderson commented Feb 26, 2024

Description

Adds the SQL files that contain the logic to map features between OSM tags and Overture values. These are added to the task-force-docs.

Currently, the logic behind the base theme is a series of SQL files here that are assembled by a (perhaps hacky) render script and then included in the Airflow DAG to create the base data from the daylight map distribution. Limitations include:

  1. Must run render_queries.py script before committing / pushing code.
  2. Logic is hidden in private data-platform repo.

This PR moves these SQL files to the public schema repo. Advantages include:

  1. The logic is publicly available and can be version-controlled by release.
  2. Matches other themes (such as buildings) by publishing the OSM tag mappings in schema/task-force-docs

The biggest advantage here is that the actual SQL logic that creates a given release can be tied to a schema release version.

On the data-platform side, https://github.com/OvertureMaps/tf-data-platform/pull/158 alters the Airflow DAG to pull these SQL files from the schema repo before each run.

Reference

List of relevant links to GitHub issues, PRs, and other documentation.

  1. https://github.com/OvertureMaps/tf-data-platform/pull/158

Testing

Brief description of the testing done for this change showing why you are confident it works as expected and does not introduce regressions. Provide sample output data where appropriate.

The Airflow DAG for base_theme_staging reads the files from this PR.

Checklist

Checklist of tasks commonly-associated with schema pull requests. Please review the relevant checklists and ensure you do all the tasks that are required for the change you made.

  1. [N/A] Add relevant examples.
  2. [N/A] Add relevant counterexamples.
  3. [N/A] Update in-schema documentation using plain English written in complete sentences, if an update is required.
  4. [N/A] Update Docusaurus documentation, if an update is required.
  5. [N/A] Review change with Overture technical writer to ensure any advanced documentation needs will be taken care of, unless the change is trivial and would not affect the documentation.

Documentation Website

(Doesn't affect the docs)
Docs preview for this PR.

@jenningsanderson jenningsanderson marked this pull request as draft February 26, 2024 22:31
@jenningsanderson jenningsanderson changed the base branch from main to dev February 26, 2024 22:32
@jenningsanderson jenningsanderson marked this pull request as ready for review February 27, 2024 18:59
@jenningsanderson jenningsanderson changed the title [WIP] Add base sql to schema repo Add base sql to schema repo Feb 28, 2024
Copy link
Collaborator

@vcschapp vcschapp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

─────────────────────░██░▇▆▅▄▃▂
────────────────────░█▓▓█░▇▆▅▄▃▂
───────────────────░█▓▓▓█░▇▆▅▄▃▂
──────────────────░█▓▓▓▓▓█░▇▆▅▄▃▂
─────────────────░█▓▓▓▓▓█░▇▆▅▄▃▂
──────────░░░───░█▓▓▓▓▓▓█░▇▆▅▄▃▂
─────────░███░──░█▓▓▓▓▓█░▇▆▅▄▃▂
───────░██░░░██░█▓▓▓▓▓█░▇▆▅▄▃▂
──────░█░░█░░░░██▓▓▓▓▓█░▇▆▅▄▃▂
────░██░░█░░░░░░█▓▓▓▓█░▇▆▅▄▃▂
───░█░░░█░░░░░░░██▓▓▓█░▇▆▅▄▃▂
──░█░░░░█░░░░░░░░█▓▓▓█░▇▆▅▄▃▂
──░█░░░░░█░░░░░░░░█▓▓▓█░▇▆▅▄▃▂
──░█░░█░░░█░░░░░░░░█▓▓█░▇▆▅▄▃▂
─░█░░░█░░░░██░░░░░░█▓▓█░▇▆▅▄▃▂
─░█░░░░█░░░░░██░░░█▓▓▓█░▇▆▅▄▃▂
─░█░█░░░█░░░░░░███▓▓▓▓█░▇▆▅▄▃▂
░█░░░█░░░██░░░░░█▓▓▓▓▓█░▇▆▅▄▃▂
░█░░░░█░░░░█████▓▓▓▓▓█░▇▆▅▄▃▂
░█░░░░░█░░░░░░░█▓▓▓▓▓█░▇▆▅▄▃▂
░█░█░░░░██░░░░█▓▓▓▓▓█░▇▆▅▄▃▂
─░█░█░░░░░████▓▓▓▓██░▇▆▅▄▃▂
─░█░░█░░░░░░░█▓▓██▓█░▇▆▅▄▃▂
──░█░░██░░░██▓▓█▓▓▓█░▇▆▅▄▃▂
───░██░░███▓▓██▓█▓▓█░▇▆▅▄▃▂
────░██▓▓▓███▓▓▓█▓▓▓█░▇▆▅▄▃▂
──────░█▓▓▓▓▓▓▓▓█▓▓▓█░▇▆▅▄▃▂
──────░█▓▓▓▓▓▓▓▓▓▓▓▓▓█░▇▆▅▄▃▂

@vcschapp vcschapp merged commit 9bb20af into dev Mar 6, 2024
2 checks passed
@vcschapp vcschapp deleted the add-base-sql-to-schema-repo branch March 6, 2024 16:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants