query to include data plane attributes #18531

xiaohansong · 2022-10-27T05:56:40Z

What

#15898

Include processing_task_queue or connection geography into metric report query.

pmossman

Looks good to me! I just added a few method rename suggestions to include the new groupBy info.

Test coverage around this looks solid, thanks for updating that as well!

pmossman · 2022-10-31T17:46:27Z

airbyte-metrics/reporter/src/main/java/io/airbyte/metrics/reporter/MetricRepository.java

@@ -25,22 +30,32 @@ class MetricRepository {
    this.ctx = ctx;
  }

-  int numberOfPendingJobs() {
-    return ctx.selectCount()
+  Map<String, Integer> numberOfPendingJobs() {


I would rename this to numberOfPendingJobsByGeography

pmossman · 2022-10-31T17:46:44Z

airbyte-metrics/reporter/src/main/java/io/airbyte/metrics/reporter/MetricRepository.java

  }

-  int numberOfRunningJobs() {
-    return ctx.selectCount()
+  Map<String, Integer> numberOfRunningJobs() {


I would rename this to numberOfRunningJobsByTaskQueue

pmossman · 2022-10-31T17:48:24Z

airbyte-metrics/reporter/src/main/java/io/airbyte/metrics/reporter/MetricRepository.java

@@ -50,12 +65,32 @@ int numberOfOrphanRunningJobs() {
        .fetchOne(0, int.class);
  }

-  long oldestPendingJobAgeSecs() {
-    return oldestJobAgeSecs(JobStatus.pending);
+  Map<GeographyType, Double> oldestPendingJobAgeSecs() {


I would rename this to oldestPendingJobAgeSecsByGeography

pmossman · 2022-10-31T17:48:54Z

airbyte-metrics/reporter/src/main/java/io/airbyte/metrics/reporter/MetricRepository.java

  }

-  long oldestRunningJobAgeSecs() {
-    return oldestJobAgeSecs(JobStatus.running);
+  Map<String, Double> oldestRunningJobAgeSecs() {


I would rename this to oldestRunningJobAgeSecsByGeography

* master: (59 commits) 🪟🔧 Remove styled components (round 1) (#18766) Bump Airbyte version from 0.40.17 to 0.40.18 (#18827) ci: add job and run id to test reports (#18832) hide ConfigPersistence inside ConfigRepository to discourage use (#18803) Remove the bulk actions from ConfigPersistence (#18800) remove config persistence from seeding logic (#18749) Remove ConfigPersistence usage from SecretsMigrator (#18747) Add normalization changelog and bump normalization version in platform (#18813) 🐛Source Exchange Rates: Fix handling error during check connection (#18726) 🐛Destination Google Sheets: Fix empty headers list (#18729) Bump helm chart version reference to 0.40.40 (#18815) Use equalsIgnoreCase (#18810) [charts/airbyte-cron] Cleanup env vars (#18787) 🐛 Source Facebook Marketing: reduce request limit after specific error (#18734) Parameterize test_empty_streams and test_stream_with_1_airbyte_column by destination (#18197) Correct coinmarket spec (#18790) ci: use custom test-reporter action to upload job results (#18004) Fix unit tests in source relational db (#18789) query to include data plane attributes (#18531) Mark/update notification settings design (#18159) ...

* query to include data plane attributes * rename functions * fix java build * more renaming fix

* solve conflicts * solve conflict in json schema * bump to version 0.1.8 for the changes of this pr * change ad account id in the schemas * query to include data plane attributes (#18531) * query to include data plane attributes * rename functions * fix java build * more renaming fix * Fix unit tests in source relational db (#18789) * Fix unit tests * Add extra test case for record count > 1 * Store record count in variable * ci: use custom test-reporter action to upload job results (#18004) * ci: use custom action to upload job results * Correct coinmarket spec (#18790) * correct coinmarket spec * remove duplicate support normalization from source spec * rollback coinmarketcap version in source def seed * update connector version * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Parameterize test_empty_streams and test_stream_with_1_airbyte_column by destination (#18197) * Remove lines that always add Postgres to list of destinations * Parameterize all tests in test_ephemeral by destination * 🐛 Source Facebook Marketing: reduce request limit after specific error (#18734) * 🐛 Source Facebook Marketing: reduce request limit after specific error * 🐛 Source Facebook Marketing: bump version; update docs * 🐛 Source Facebook Marketing: add test * 🐛 Source Facebook Marketing: increase timeout * [charts/airbyte-cron] Cleanup env vars (#18787) * [charts/airbyte-cron] Cleanup env vars * Remove unused env var * Use equalsIgnoreCase (#18810) * Bump helm chart version reference to 0.40.40 (#18815) Co-authored-by: perangel <[email protected]> Co-authored-by: Kyryl Skobylko <[email protected]> * 🐛Destination Google Sheets: Fix empty headers list (#18729) * Fix empty headers list * Updated PR number * Bumped version * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * 🐛Source Exchange Rates: Fix handling error during check connection (#18726) * Fix handling error during check connection * Updated PR number * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Add normalization changelog and bump normalization version in platform (#18813) * Remove ConfigPersistence usage from SecretsMigrator (#18747) * remove config persistence from seeding logic (#18749) * Remove the bulk actions from ConfigPersistence (#18800) * hide ConfigPersistence inside ConfigRepository to discourage use (#18803) * ci: add job and run id to test reports (#18832) * Bump Airbyte version from 0.40.17 to 0.40.18 (#18827) Co-authored-by: grishick <[email protected]> * 🪟🔧 Remove styled components (round 1) (#18766) * refactor EditorHeader (untested) * refactor BaseClearView * delete unused Subtitle * refactor ConfirmationModal * refactor Arrow * refactor BulkHeader * refactor CatalogTreeSearch * refactor StreamFieldTable * refactor StreamHeader * refactor ConnectorIcon * refactor TreeRowWrapper * refactor DeleteBlock * refactor EmptyResourceBlock * revert unintended element change * fixed acceptance tests (#18699) * 🪟🔧 Reactor Breadcrumbs component to use anchors (#18764) * refactor breadcrumbs to use actual links * PR comments on styles * increase test timeout for some webapp tests to prevent flakes (#18807) * Remove "Filters and Segments" from Google Analytics v4 (#18508) Filters and Segments info was incorrectly added to the Google Analytics v4 connector instead of the Google Analytics (Universal Analytics) Connector. * Add notes about EU OAUth (#18835) EU OAuth is not fully tested so adding a note to account for that. * 🪟🐛 Fix: visual regression in ConnectorIcon (#18849) * fix visual regression * remove unused prop * Add links to demo page (#18828) * mysql-source:fix tinyint unsigned handling (#18619) * mysql-source:fix tinyint unsigned handling * update doc * format * upgrade version * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * 🪟 🎉 Allow environment specific sections in docs (#18829) * Allow environment specific sections in docs * Change syntax to lower case * ci: replace GITHUB_OUTPUT with GITHUB_ENV on multiline variables (#18809) * ci: replace GITHUB_OUTPUT with GITHUB_ENV on multiline variables * ci: replace github set-ouput with new syntax in ./tools/bin/ * Add connection ID to span (#18852) * edited connector docs (#18855) * 🪟 🔧 Upgrade husky to 8.0.1 (#18719) * Upgrade Husky * Upgrade Husky * Upgrade Husky * Upgrade Husky * Upgrade Husky * ci: replace GITHUB_OUTPUT with GITHUB_ENV for multiline variables (#18857) * Avoid NPE when adding connection ID to trace (#18856) * Filter exit errors by operation name (#18850) * add label * auto-bump connector version Co-authored-by: marcosmarxm <[email protected]> Co-authored-by: Xiaohan Song <[email protected]> Co-authored-by: Liren Tu <[email protected]> Co-authored-by: Conor <[email protected]> Co-authored-by: Marcos Marx <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> Co-authored-by: Greg Solovyev <[email protected]> Co-authored-by: Artem Inzhyyants <[email protected]> Co-authored-by: perangel <[email protected]> Co-authored-by: Jonathan Pearlin <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: perangel <[email protected]> Co-authored-by: Kyryl Skobylko <[email protected]> Co-authored-by: Serhii Lazebnyi <[email protected]> Co-authored-by: Charles <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> Co-authored-by: Joey Marshment-Howell <[email protected]> Co-authored-by: darynaishchenko <[email protected]> Co-authored-by: Michael Siega <[email protected]> Co-authored-by: Tyler B <[email protected]> Co-authored-by: Yowan Ramchoreeter <[email protected]> Co-authored-by: Tim Roes <[email protected]> Co-authored-by: Subodh Kant Chaturvedi <[email protected]> Co-authored-by: Volodymyr Pochtar <[email protected]> Co-authored-by: Amruta Ranade <[email protected]>

xiaohansong added 2 commits October 26, 2022 22:51

query to include data plane attributes

d2578f9

Merge remote-tracking branch 'origin/master' into xiaohan/q

11dd701

xiaohansong temporarily deployed to more-secrets October 27, 2022 05:58 Inactive

xiaohansong changed the title ~~Xiaohan/q~~ query to include data plane attributes Oct 27, 2022

xiaohansong requested a review from pmossman October 27, 2022 16:36

pmossman approved these changes Oct 31, 2022

View reviewed changes

rename functions

b96b6b2

xiaohansong temporarily deployed to more-secrets October 31, 2022 23:17 Inactive

fix java build

0c6a8ae

xiaohansong temporarily deployed to more-secrets November 1, 2022 05:40 Inactive

more renaming fix

1ee78be

xiaohansong temporarily deployed to more-secrets November 1, 2022 16:52 Inactive

xiaohansong merged commit 7d558c9 into master Nov 1, 2022

xiaohansong deleted the xiaohan/q branch November 1, 2022 17:56

octavia-squidington-iii mentioned this pull request Nov 1, 2022

Bump Airbyte version from 0.40.17 to 0.40.18 #18827

Merged

drewrasm pushed a commit to drewrasm/airbyte that referenced this pull request Nov 2, 2022

query to include data plane attributes (airbytehq#18531)

3bd1e98

* query to include data plane attributes * rename functions * fix java build * more renaming fix

natalyjazzviolin pushed a commit that referenced this pull request Nov 3, 2022

query to include data plane attributes (#18531)

95eabae

* query to include data plane attributes * rename functions * fix java build * more renaming fix

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

query to include data plane attributes #18531

query to include data plane attributes #18531

xiaohansong commented Oct 27, 2022 •

edited

Loading

pmossman left a comment

pmossman Oct 31, 2022

pmossman Oct 31, 2022

pmossman Oct 31, 2022

pmossman Oct 31, 2022

query to include data plane attributes #18531

query to include data plane attributes #18531

Conversation

xiaohansong commented Oct 27, 2022 • edited Loading

What

pmossman left a comment

Choose a reason for hiding this comment

pmossman Oct 31, 2022

Choose a reason for hiding this comment

pmossman Oct 31, 2022

Choose a reason for hiding this comment

pmossman Oct 31, 2022

Choose a reason for hiding this comment

pmossman Oct 31, 2022

Choose a reason for hiding this comment

xiaohansong commented Oct 27, 2022 •

edited

Loading