Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[assets] Precalculate dependedBy for better perf, fix foreign asset deps #5917

Merged
merged 2 commits into from
Dec 9, 2021
Merged

[assets] Precalculate dependedBy for better perf, fix foreign asset deps #5917

merged 2 commits into from
Dec 9, 2021

Conversation

bengotow
Copy link
Collaborator

@bengotow bengotow commented Dec 9, 2021

Summary

I originally hacked a dependedBy resolver into AssetNode because Dagit needed to be able to fetch bidirectionally from a single asset node for the new asset details view. It turns out scanning dependencies to build the reverse relations was pretty slow and pre-computing on ExternalAssetNode is a 50%+ performance win on the mega asset DAG.

The only reason we were fetching dependedBy when rendering the Asset DAG explorer (eg: requesting it for EVERY asset node) was because it was the only way to show arrows going in to downstream foreign assets and assets without definitions. It turns out we were just leaving the dependencies field blank when constructing the ExternalAssetNodes, so I fixed that in this PR and removed the overfetching in Dagit in #5904. (I think this makes sense?)

This makes the performance improvements less necessary so I'd be happy to revert to the scan approach if that's easier to maintain.

Test Plan

Edited the tests to check the depended_by relations

Checklist

  • My change requires a change to the documentation and I have updated the documentation accordingly.
  • I have added tests to cover my changes.

@bengotow bengotow requested review from sryza and prha December 9, 2021 04:49
@vercel
Copy link

vercel bot commented Dec 9, 2021

This pull request is being automatically deployed with Vercel (learn more).
To see the status of your deployment, click below or on the icon next to each commit.

🔍 Inspect: https://vercel.com/elementl/dagster/B8izutSu1CSFUZBhkJ5nzvwAtMVe
✅ Preview: Canceled

@prha prha merged commit 75f0503 into master Dec 9, 2021
@prha prha deleted the bbash/large-asset-dag-perf branch December 9, 2021 06:04
prha pushed a commit that referenced this pull request Dec 9, 2021
@bengotow @prha
[assets] Precalculate dependedBy for better perf, fix foreign asset d…
64d2fa0 
…eps (#5917)

* [assets] Precalculate dependedBy for better load performance on large asset dags

* Allow foreign nodes to have deps/dep-by links within the graph
yuhan pushed a commit that referenced this pull request Nov 10, 2023
@dependabot
Bump axios from 0.26.0 to 1.6.0 in /docs/next (#17898)
6317ca3 
Bumps [axios](https://github.com/axios/axios) from 0.26.0 to 1.6.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/axios/axios/releases">axios's
releases</a>.</em></p>
<blockquote>
<h2>Release v1.6.0</h2>
<h2>Release notes:</h2>
<h3>Bug Fixes</h3>
<ul>
<li><strong>CSRF:</strong> fixed CSRF vulnerability CVE-2023-45857 (<a
href="https://redirect.github.com/axios/axios/issues/6028">#6028</a>)
(<a
href="https://github.com/axios/axios/commit/96ee232bd3ee4de2e657333d4d2191cd389e14d0">96ee232</a>)</li>
<li><strong>dns:</strong> fixed lookup function decorator to work
properly in node v20; (<a
href="https://redirect.github.com/axios/axios/issues/6011">#6011</a>)
(<a
href="https://github.com/axios/axios/commit/5aaff532a6b820bb9ab6a8cd0f77131b47e2adb8">5aaff53</a>)</li>
<li><strong>types:</strong> fix AxiosHeaders types; (<a
href="https://redirect.github.com/axios/axios/issues/5931">#5931</a>)
(<a
href="https://github.com/axios/axios/commit/a1c8ad008b3c13d53e135bbd0862587fb9d3fc09">a1c8ad0</a>)</li>
</ul>
<h3>PRs</h3>
<ul>
<li>CVE 2023 45857 ( <a
href="https://api.github.com/repos/axios/axios/pulls/6028">#6028</a>
)</li>
</ul>
<pre><code>
⚠️ Critical vulnerability fix. See
https://security.snyk.io/vuln/SNYK-JS-AXIOS-6032459
</code></pre>
<h3>Contributors to this release</h3>
<ul>
<li><!-- raw HTML omitted --> <a
href="https://github.com/DigitalBrainJS" title="+449/-114
([#6032](axios/axios#6032)
[#6021](axios/axios#6021)
[#6011](axios/axios#6011)
[#5932](axios/axios#5932)
[#5931](axios/axios#5931) )">Dmitriy
Mozgovoy</a></li>
<li><!-- raw HTML omitted --> <a
href="https://github.com/valentin-panov" title="+4/-4
([#6028](axios/axios#6028) )">Valentin
Panov</a></li>
<li><!-- raw HTML omitted --> <a href="https://github.com/therealrinku"
title="+1/-1 ([#5889](axios/axios#5889)
)">Rinku Chaudhari</a></li>
</ul>
<h2>Release v1.5.1</h2>
<h2>Release notes:</h2>
<h3>Bug Fixes</h3>
<ul>
<li><strong>adapters:</strong> improved adapters loading logic to have
clear error messages; (<a
href="https://redirect.github.com/axios/axios/issues/5919">#5919</a>)
(<a
href="https://github.com/axios/axios/commit/e4107797a7a1376f6209fbecfbbce73d3faa7859">e410779</a>)</li>
<li><strong>formdata:</strong> fixed automatic addition of the
<code>Content-Type</code> header for FormData in non-browser
environments; (<a
href="https://redirect.github.com/axios/axios/issues/5917">#5917</a>)
(<a
href="https://github.com/axios/axios/commit/bc9af51b1886d1b3529617702f2a21a6c0ed5d92">bc9af51</a>)</li>
<li><strong>headers:</strong> allow <code>content-encoding</code> header
to handle case-insensitive values (<a
href="https://redirect.github.com/axios/axios/issues/5890">#5890</a>)
(<a
href="https://redirect.github.com/axios/axios/issues/5892">#5892</a>)
(<a
href="https://github.com/axios/axios/commit/4c89f25196525e90a6e75eda9cb31ae0a2e18acd">4c89f25</a>)</li>
<li><strong>types:</strong> removed duplicated code (<a
href="https://github.com/axios/axios/commit/9e6205630e1c9cf863adf141c0edb9e6d8d4b149">9e62056</a>)</li>
</ul>
<h3>Contributors to this release</h3>
<ul>
<li><!-- raw HTML omitted --> <a
href="https://github.com/DigitalBrainJS" title="+89/-18
([#5919](axios/axios#5919)
[#5917](axios/axios#5917) )">Dmitriy
Mozgovoy</a></li>
<li><!-- raw HTML omitted --> <a href="https://github.com/DavidJDallas"
title="+11/-5 ()">David Dallas</a></li>
<li><!-- raw HTML omitted --> <a href="https://github.com/fb-sean"
title="+2/-8 ()">Sean Sattler</a></li>
<li><!-- raw HTML omitted --> <a href="https://github.com/0o001"
title="+4/-4 ()">Mustafa Ateş Uzun</a></li>
<li><!-- raw HTML omitted --> <a
href="https://github.com/sfc-gh-pmotacki" title="+2/-1
([#5892](axios/axios#5892) )">Przemyslaw
Motacki</a></li>
<li><!-- raw HTML omitted --> <a href="https://github.com/Cadienvan"
title="+1/-1 ()">Michael Di Prisco</a></li>
</ul>
<h2>Release v1.5.0</h2>
<h2>Release notes:</h2>
<h3>Bug Fixes</h3>
<ul>
<li><strong>adapter:</strong> make adapter loading error more clear by
using platform-specific adapters explicitly (<a
href="https://redirect.github.com/axios/axios/issues/5837">#5837</a>)
(<a
href="https://github.com/axios/axios/commit/9a414bb6c81796a95c6c7fe668637825458e8b6d">9a414bb</a>)</li>
<li><strong>dns:</strong> fixed <code>cacheable-lookup</code>
integration; (<a
href="https://redirect.github.com/axios/axios/issues/5836">#5836</a>)
(<a
href="https://github.com/axios/axios/commit/b3e327dcc9277bdce34c7ef57beedf644b00d628">b3e327d</a>)</li>
<li><strong>headers:</strong> added support for setting header names
that overlap with class methods; (<a
href="https://redirect.github.com/axios/axios/issues/5831">#5831</a>)
(<a
href="https://github.com/axios/axios/commit/d8b4ca0ea5f2f05efa4edfe1e7684593f9f68273">d8b4ca0</a>)</li>
<li><strong>headers:</strong> fixed common Content-Type header merging;
(<a
href="https://redirect.github.com/axios/axios/issues/5832">#5832</a>)
(<a
href="https://github.com/axios/axios/commit/8fda2766b1e6bcb72c3fabc146223083ef13ce17">8fda276</a>)</li>
</ul>
<h3>Features</h3>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/axios/axios/blob/v1.x/CHANGELOG.md">axios's
changelog</a>.</em></p>
<blockquote>
<h1><a
href="https://github.com/axios/axios/compare/v1.5.1...v1.6.0">1.6.0</a>
(2023-10-26)</h1>
<h3>Bug Fixes</h3>
<ul>
<li><strong>CSRF:</strong> fixed CSRF vulnerability CVE-2023-45857 (<a
href="https://redirect.github.com/axios/axios/issues/6028">#6028</a>)
(<a
href="https://github.com/axios/axios/commit/96ee232bd3ee4de2e657333d4d2191cd389e14d0">96ee232</a>)</li>
<li><strong>dns:</strong> fixed lookup function decorator to work
properly in node v20; (<a
href="https://redirect.github.com/axios/axios/issues/6011">#6011</a>)
(<a
href="https://github.com/axios/axios/commit/5aaff532a6b820bb9ab6a8cd0f77131b47e2adb8">5aaff53</a>)</li>
<li><strong>types:</strong> fix AxiosHeaders types; (<a
href="https://redirect.github.com/axios/axios/issues/5931">#5931</a>)
(<a
href="https://github.com/axios/axios/commit/a1c8ad008b3c13d53e135bbd0862587fb9d3fc09">a1c8ad0</a>)</li>
</ul>
<h3>PRs</h3>
<ul>
<li>CVE 2023 45857 ( <a
href="https://api.github.com/repos/axios/axios/pulls/6028">#6028</a>
)</li>
</ul>
<pre><code>
⚠️ Critical vulnerability fix. See
https://security.snyk.io/vuln/SNYK-JS-AXIOS-6032459
</code></pre>
<h3>Contributors to this release</h3>
<ul>
<li><!-- raw HTML omitted --> <a
href="https://github.com/DigitalBrainJS" title="+449/-114
([#6032](axios/axios#6032)
[#6021](axios/axios#6021)
[#6011](axios/axios#6011)
[#5932](axios/axios#5932)
[#5931](axios/axios#5931) )">Dmitriy
Mozgovoy</a></li>
<li><!-- raw HTML omitted --> <a
href="https://github.com/valentin-panov" title="+4/-4
([#6028](axios/axios#6028) )">Valentin
Panov</a></li>
<li><!-- raw HTML omitted --> <a href="https://github.com/therealrinku"
title="+1/-1 ([#5889](axios/axios#5889)
)">Rinku Chaudhari</a></li>
</ul>
<h2><a
href="https://github.com/axios/axios/compare/v1.5.0...v1.5.1">1.5.1</a>
(2023-09-26)</h2>
<h3>Bug Fixes</h3>
<ul>
<li><strong>adapters:</strong> improved adapters loading logic to have
clear error messages; (<a
href="https://redirect.github.com/axios/axios/issues/5919">#5919</a>)
(<a
href="https://github.com/axios/axios/commit/e4107797a7a1376f6209fbecfbbce73d3faa7859">e410779</a>)</li>
<li><strong>formdata:</strong> fixed automatic addition of the
<code>Content-Type</code> header for FormData in non-browser
environments; (<a
href="https://redirect.github.com/axios/axios/issues/5917">#5917</a>)
(<a
href="https://github.com/axios/axios/commit/bc9af51b1886d1b3529617702f2a21a6c0ed5d92">bc9af51</a>)</li>
<li><strong>headers:</strong> allow <code>content-encoding</code> header
to handle case-insensitive values (<a
href="https://redirect.github.com/axios/axios/issues/5890">#5890</a>)
(<a
href="https://redirect.github.com/axios/axios/issues/5892">#5892</a>)
(<a
href="https://github.com/axios/axios/commit/4c89f25196525e90a6e75eda9cb31ae0a2e18acd">4c89f25</a>)</li>
<li><strong>types:</strong> removed duplicated code (<a
href="https://github.com/axios/axios/commit/9e6205630e1c9cf863adf141c0edb9e6d8d4b149">9e62056</a>)</li>
</ul>
<h3>Contributors to this release</h3>
<ul>
<li><!-- raw HTML omitted --> <a
href="https://github.com/DigitalBrainJS" title="+89/-18
([#5919](axios/axios#5919)
[#5917](axios/axios#5917) )">Dmitriy
Mozgovoy</a></li>
<li><!-- raw HTML omitted --> <a href="https://github.com/DavidJDallas"
title="+11/-5 ()">David Dallas</a></li>
<li><!-- raw HTML omitted --> <a href="https://github.com/fb-sean"
title="+2/-8 ()">Sean Sattler</a></li>
<li><!-- raw HTML omitted --> <a href="https://github.com/0o001"
title="+4/-4 ()">Mustafa Ateş Uzun</a></li>
<li><!-- raw HTML omitted --> <a
href="https://github.com/sfc-gh-pmotacki" title="+2/-1
([#5892](axios/axios#5892) )">Przemyslaw
Motacki</a></li>
<li><!-- raw HTML omitted --> <a href="https://github.com/Cadienvan"
title="+1/-1 ()">Michael Di Prisco</a></li>
</ul>
<h3>PRs</h3>
<ul>
<li>CVE 2023 45857 ( <a
href="https://api.github.com/repos/axios/axios/pulls/6028">#6028</a>
)</li>
</ul>
<pre><code>
⚠️ Critical vulnerability fix. See
https://security.snyk.io/vuln/SNYK-JS-AXIOS-6032459
</code></pre>
<h1><a
href="https://github.com/axios/axios/compare/v1.4.0...v1.5.0">1.5.0</a>
(2023-08-26)</h1>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/axios/axios/commit/f7adacdbaa569281253c8cfc623ad3f4dc909c60"><code>f7adacd</code></a>
chore(release): v1.6.0 (<a
href="https://redirect.github.com/axios/axios/issues/6031">#6031</a>)</li>
<li><a
href="https://github.com/axios/axios/commit/9917e67cbb6c157382863bad8c741de58e3f3c2b"><code>9917e67</code></a>
chore(ci): fix release-it arg; (<a
href="https://redirect.github.com/axios/axios/issues/6032">#6032</a>)</li>
<li><a
href="https://github.com/axios/axios/commit/96ee232bd3ee4de2e657333d4d2191cd389e14d0"><code>96ee232</code></a>
fix(CSRF): fixed CSRF vulnerability CVE-2023-45857 (<a
href="https://redirect.github.com/axios/axios/issues/6028">#6028</a>)</li>
<li><a
href="https://github.com/axios/axios/commit/7d45ab2e2ad6e59f5475e39afd4b286b1f393fc0"><code>7d45ab2</code></a>
chore(tests): fixed tests to pass in node v19 and v20 with
<code>keep-alive</code> enabl...</li>
<li><a
href="https://github.com/axios/axios/commit/5aaff532a6b820bb9ab6a8cd0f77131b47e2adb8"><code>5aaff53</code></a>
fix(dns): fixed lookup function decorator to work properly in node v20;
(<a
href="https://redirect.github.com/axios/axios/issues/6011">#6011</a>)</li>
<li><a
href="https://github.com/axios/axios/commit/a48a63ad823fc20e5a6a705f05f09842ca49f48c"><code>a48a63a</code></a>
chore(docs): added AxiosHeaders docs; (<a
href="https://redirect.github.com/axios/axios/issues/5932">#5932</a>)</li>
<li><a
href="https://github.com/axios/axios/commit/a1c8ad008b3c13d53e135bbd0862587fb9d3fc09"><code>a1c8ad0</code></a>
fix(types): fix AxiosHeaders types; (<a
href="https://redirect.github.com/axios/axios/issues/5931">#5931</a>)</li>
<li><a
href="https://github.com/axios/axios/commit/2ac731d60545ba5c4202c25fd2e732ddd8297d82"><code>2ac731d</code></a>
chore(docs): update readme.md (<a
href="https://redirect.github.com/axios/axios/issues/5889">#5889</a>)</li>
<li><a
href="https://github.com/axios/axios/commit/88fb52b5fad7aabab0532e7ad086c5f1b0178905"><code>88fb52b</code></a>
chore(release): v1.5.1 (<a
href="https://redirect.github.com/axios/axios/issues/5920">#5920</a>)</li>
<li><a
href="https://github.com/axios/axios/commit/e4107797a7a1376f6209fbecfbbce73d3faa7859"><code>e410779</code></a>
fix(adapters): improved adapters loading logic to have clear error
messages; ...</li>
<li>Additional commits viewable in <a
href="https://github.com/axios/axios/compare/v0.26.0...v1.6.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=axios&package-manager=npm_and_yarn&previous-version=0.26.0&new-version=1.6.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/dagster-io/dagster/network/alerts).

</details>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@prha prha prha approved these changes

@sryza sryza Awaiting requested review from sryza

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@bengotow @prha