support static_index.html docs #999

dwreeves · 2024-05-27T19:42:54Z

Description

Support static_index.html for dbt docs.
Refactor remote filesystem access functions in anticipation of moving them out of cosmos/plugins/__init__.py. Refactoring is designed to make them behave a little more predictably, and to make them look a little more like Airflow 2.8+'s ObjectStoragePath class. Of course, this is far, far from complete.

Related Issue(s)

Main: Support docs generated with dbt docs generate --static #986
Related: Support caching remotely #927

Breaking Change?

No breaking changes as far as the high level API is concerned.

Checklist

I have made corresponding changes to the documentation (if required)
I have added tests that prove my fix is effective or that my feature works

netlify · 2024-05-27T19:43:07Z

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name	Link
🔨 Latest commit	`9eb3b82`
🔍 Latest deploy log	https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/6665c7e53358e1000826b11b

tatiana · 2024-05-29T08:58:13Z

@dwreeves, thanks for these improvements!
I just noticed some tests are failing. Would you like us to review the code before the tests pass, or is this PR still in draft?

dwreeves · 2024-05-29T12:30:28Z

Sorry, for a long time I had issues with pre-commit due to a strange error. So for a while I'd never had pre-commit turned on for the repo.

I only recently fixed the issue and just forgot to pre-commit install. 😅 That is what is failing.

…os into support-static-index

dwreeves · 2024-06-03T00:03:38Z

I added packaging to dependencies because it's explicitly used in one of the Cosmos modules.

Even though apache-airflow also depends on it, it's best practice to explicitly define what is depended on.

…os into support-static-index

dwreeves · 2024-06-03T00:51:45Z

@tatiana Tests are stuck for some odd reason, but otherwise passing, so this is ready for review.

codecov · 2024-06-04T10:13:26Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.81%. Comparing base (bbe4e86) to head (9eb3b82).
Report is 6 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #999      +/-   ##
==========================================
+ Coverage   95.75%   95.81%   +0.06%     
==========================================
  Files          62       62              
  Lines        2965     3009      +44     
==========================================
+ Hits         2839     2883      +44     
  Misses        126      126

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

tatiana

@dwreeves I think we're super close to get this change merged, do you by any chance have bandwith to improve a bit the test coverage? Codecov gave some feedback.
We'll release this in 1.5

dwreeves · 2024-06-06T14:33:10Z

@tatiana When are you intending on releasing 1.5? I can try to get this done over the weekend but not until then.

tatiana · 2024-06-06T15:16:44Z

@tatiana When are you intending on releasing 1.5? I can try to get this done over the weekend but not until then.

The current plan is the 24 June 2024, and we can have alphas/release candidates with the change beforehand:
https://github.com/astronomer/astronomer-cosmos/milestone/19

…os into support-static-index

…stronomer-cosmos into support-static-index

tatiana

Thanks, @dwreeves , it looks great!

@tatiana

New Features * Speed up ``LoadMode.DBT_LS`` by caching dbt ls output in Airflow Variable by @tatiana in #1014 * Support to cache profiles created via ``ProfileMapping`` by @pankajastro in #1046 * Support for running dbt tasks in AWS EKS in #944 by @VolkerSchiewe * Add Clickhouse profile mapping by @roadan and @pankajastro in #353 and #1016 * Add node config to TaskInstance Context by @linchun3 in #1044 Bug fixes * Support partial parsing when cache is disabled by @tatiana in #1070 * Fix disk permission error in restricted env by @pankajastro in #1051 * Add CSP header to iframe contents by @dwreeves in #1055 * Stop attaching log adaptors to root logger to reduce logging costs by @glebkrapivin in #1047 Enhancements * Support ``static_index.html`` docs by @dwreeves in #999 * Support deep linking dbt docs via Airflow UI by @dwreeves in #1038 * Add ability to specify host/port for Snowflake connection by @whummer in #1063 Docs * Fix rendering for env ``enable_cache_dbt_ls`` by @pankajastro in #1069 Others * Update documentation for DbtDocs generator by @arjunanan6 in #1043 * Use uv in CI by @dwreeves in #1013 * Cache hatch folder in the CI by @tatiana in #1056 * Change example DAGs to use ``example_conn`` as opposed to ``airflow_db`` by @tatiana in #1054 * Mark plugin integration tests as integration by @tatiana in #1057 * Ensure compliance with linting rule D300 by using triple quotes for docstrings by @pankajastro in #1049 * Pre-commit hook updates in #1039, #1050, #1064 * Remove duplicates in changelog by @jedcunningham in #1068

- Support `static_index.html` for dbt docs. - Refactor remote filesystem access functions in anticipation of moving them out of `cosmos/plugins/__init__.py`. Refactoring is designed to make them behave a little more predictably and to make them look a little more like Airflow 2.8+'s `ObjectStoragePath` class. Of course, this is far, far from complete. # Related Issue(s) - Main: astronomer#986 - Related: astronomer#927

@tatiana

New Features * Speed up ``LoadMode.DBT_LS`` by caching dbt ls output in Airflow Variable by @tatiana in astronomer#1014 * Support to cache profiles created via ``ProfileMapping`` by @pankajastro in astronomer#1046 * Support for running dbt tasks in AWS EKS in astronomer#944 by @VolkerSchiewe * Add Clickhouse profile mapping by @roadan and @pankajastro in astronomer#353 and astronomer#1016 * Add node config to TaskInstance Context by @linchun3 in astronomer#1044 Bug fixes * Support partial parsing when cache is disabled by @tatiana in astronomer#1070 * Fix disk permission error in restricted env by @pankajastro in astronomer#1051 * Add CSP header to iframe contents by @dwreeves in astronomer#1055 * Stop attaching log adaptors to root logger to reduce logging costs by @glebkrapivin in astronomer#1047 Enhancements * Support ``static_index.html`` docs by @dwreeves in astronomer#999 * Support deep linking dbt docs via Airflow UI by @dwreeves in astronomer#1038 * Add ability to specify host/port for Snowflake connection by @whummer in astronomer#1063 Docs * Fix rendering for env ``enable_cache_dbt_ls`` by @pankajastro in astronomer#1069 Others * Update documentation for DbtDocs generator by @arjunanan6 in astronomer#1043 * Use uv in CI by @dwreeves in astronomer#1013 * Cache hatch folder in the CI by @tatiana in astronomer#1056 * Change example DAGs to use ``example_conn`` as opposed to ``airflow_db`` by @tatiana in astronomer#1054 * Mark plugin integration tests as integration by @tatiana in astronomer#1057 * Ensure compliance with linting rule D300 by using triple quotes for docstrings by @pankajastro in astronomer#1049 * Pre-commit hook updates in astronomer#1039, astronomer#1050, astronomer#1064 * Remove duplicates in changelog by @jedcunningham in astronomer#1068

@tatiana

New Features * Speed up ``LoadMode.DBT_LS`` by caching dbt ls output in Airflow Variable by @tatiana in #1014 * Support to cache profiles created via ``ProfileMapping`` by @pankajastro in #1046 * Support for running dbt tasks in AWS EKS in #944 by @VolkerSchiewe * Add Clickhouse profile mapping by @roadan and @pankajastro in #353 and #1016 * Add node config to TaskInstance Context by @linchun3 in #1044 Bug fixes * Support partial parsing when cache is disabled by @tatiana in #1070 * Fix disk permission error in restricted env by @pankajastro in #1051 * Add CSP header to iframe contents by @dwreeves in #1055 * Stop attaching log adaptors to root logger to reduce logging costs by @glebkrapivin in #1047 Enhancements * Support ``static_index.html`` docs by @dwreeves in #999 * Support deep linking dbt docs via Airflow UI by @dwreeves in #1038 * Add ability to specify host/port for Snowflake connection by @whummer in #1063 Docs * Fix rendering for env ``enable_cache_dbt_ls`` by @pankajastro in #1069 Others * Update documentation for DbtDocs generator by @arjunanan6 in #1043 * Use uv in CI by @dwreeves in #1013 * Cache hatch folder in the CI by @tatiana in #1056 * Change example DAGs to use ``example_conn`` as opposed to ``airflow_db`` by @tatiana in #1054 * Mark plugin integration tests as integration by @tatiana in #1057 * Ensure compliance with linting rule D300 by using triple quotes for docstrings by @pankajastro in #1049 * Pre-commit hook updates in #1039, #1050, #1064 * Remove duplicates in changelog by @jedcunningham in #1068 (cherry picked from commit 18d2c90)

dwreeves added 2 commits May 27, 2024 15:01

support static_index.html docs

9673ef0

support static_index.html docs

79d0626

dosubot bot added the size:M This PR changes 30-99 lines, ignoring generated files. label May 27, 2024

dwreeves temporarily deployed to external May 27, 2024 19:42 — with GitHub Actions Inactive

dosubot bot added area:docs Relating to documentation, changes, fixes, improvement dbt:docs Primarily related to dbt docs command or functionality labels May 27, 2024

dwreeves added 3 commits May 29, 2024 08:30

fix

8f53801

Merge branch 'main' of github.com-dwreeves:astronomer/astronomer-cosm…

d910cbd

…os into support-static-index

update

c4c2ed1

dwreeves had a problem deploying to external May 29, 2024 12:32 — with GitHub Actions Error

dwreeves had a problem deploying to external June 2, 2024 23:26 — with GitHub Actions Error

dwreeves force-pushed the support-static-index branch from 86f017d to bb35621 Compare June 2, 2024 23:46

dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. and removed size:M This PR changes 30-99 lines, ignoring generated files. labels Jun 2, 2024

dwreeves had a problem deploying to external June 2, 2024 23:46 — with GitHub Actions Error

dwreeves force-pushed the support-static-index branch from bb35621 to ce93392 Compare June 2, 2024 23:58

dwreeves had a problem deploying to external June 2, 2024 23:59 — with GitHub Actions Error

dwreeves added 2 commits June 2, 2024 20:06

add google provider to tests

6ce3bb4

Merge branch 'main' of github.com-dwreeves:astronomer/astronomer-cosm…

f40edd6

…os into support-static-index

dwreeves force-pushed the support-static-index branch from ce93392 to f40edd6 Compare June 3, 2024 00:06

dwreeves temporarily deployed to external June 3, 2024 00:06 — with GitHub Actions Inactive

Merge branch 'main' into support-static-index

781c6eb

tatiana temporarily deployed to external June 6, 2024 09:10 — with GitHub Actions Inactive

tatiana reviewed Jun 6, 2024

View reviewed changes

tatiana added this to the Cosmos 1.5.0 milestone Jun 6, 2024

dwreeves added 2 commits June 8, 2024 21:48

Merge branch 'main' of github.com-dwreeves:astronomer/astronomer-cosm…

c294223

…os into support-static-index

update

2e3acda

dwreeves had a problem deploying to external June 9, 2024 02:52 — with GitHub Actions Error

dwreeves force-pushed the support-static-index branch from 1866af8 to 0f2ca86 Compare June 9, 2024 03:02

dwreeves had a problem deploying to external June 9, 2024 03:02 — with GitHub Actions Error

dwreeves force-pushed the support-static-index branch from 0f2ca86 to d717e5e Compare June 9, 2024 03:13

dwreeves had a problem deploying to external June 9, 2024 03:14 — with GitHub Actions Error

dwreeves force-pushed the support-static-index branch from d717e5e to aa013ff Compare June 9, 2024 03:19

dwreeves had a problem deploying to external June 9, 2024 03:19 — with GitHub Actions Error

dwreeves force-pushed the support-static-index branch from aa013ff to 0c46d00 Compare June 9, 2024 03:23

dwreeves had a problem deploying to external June 9, 2024 03:24 — with GitHub Actions Error

Merge branch 'support-static-index' of github.com-dwreeves:dwreeves/a…

1baa00f

…stronomer-cosmos into support-static-index

dwreeves force-pushed the support-static-index branch from 0c46d00 to 1baa00f Compare June 9, 2024 03:31

dwreeves temporarily deployed to external June 9, 2024 03:31 — with GitHub Actions Inactive

dwreeves had a problem deploying to external June 9, 2024 14:53 — with GitHub Actions Error

update

9eb3b82

dwreeves force-pushed the support-static-index branch from 883b67f to 9eb3b82 Compare June 9, 2024 15:18

dwreeves temporarily deployed to external June 9, 2024 15:19 — with GitHub Actions Inactive

tatiana approved these changes Jun 10, 2024

View reviewed changes

dosubot bot added the lgtm This PR has been approved by a maintainer label Jun 10, 2024

tatiana merged commit 74c28e2 into astronomer:main Jun 10, 2024
64 checks passed

pankajkoti mentioned this pull request Jun 27, 2024

Release 1.5.0 #1071

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

support static_index.html docs #999

support static_index.html docs #999

dwreeves commented May 27, 2024

netlify bot commented May 27, 2024 •

edited

Loading

tatiana commented May 29, 2024

dwreeves commented May 29, 2024

dwreeves commented Jun 3, 2024

dwreeves commented Jun 3, 2024

codecov bot commented Jun 4, 2024 •

edited

Loading

tatiana left a comment

dwreeves commented Jun 6, 2024 •

edited

Loading

tatiana commented Jun 6, 2024

tatiana left a comment

support static_index.html docs #999

support static_index.html docs #999

Conversation

dwreeves commented May 27, 2024

Description

Related Issue(s)

Breaking Change?

Checklist

netlify bot commented May 27, 2024 • edited Loading

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

tatiana commented May 29, 2024

dwreeves commented May 29, 2024

dwreeves commented Jun 3, 2024

dwreeves commented Jun 3, 2024

codecov bot commented Jun 4, 2024 • edited Loading

Codecov Report

tatiana left a comment

Choose a reason for hiding this comment

dwreeves commented Jun 6, 2024 • edited Loading

tatiana commented Jun 6, 2024

tatiana left a comment

Choose a reason for hiding this comment

netlify bot commented May 27, 2024 •

edited

Loading

codecov bot commented Jun 4, 2024 •

edited

Loading

dwreeves commented Jun 6, 2024 •

edited

Loading