Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add repo sync GitHub Action #2

Merged
merged 1 commit into from
Aug 8, 2022
Merged

Add repo sync GitHub Action #2

merged 1 commit into from
Aug 8, 2022

Conversation

timvaillancourt
Copy link
Member

Description

This PR adds automation for syncing the following branches from vitessio/vitess:

  • main
  • release-12.0
  • release-13.0
  • release-14.0

This GitHub Action will run once daily or when triggered manually at https://github.com/slackhq/vitess/actions. Luckily, this file does not get paved-over by itself

Related Issue(s)

Checklist

  • "Backport me!" label has been added if this change should be backported
  • Tests were added or are not required
  • Documentation was added or is not required

Deployment Notes

@CLAassistant
Copy link

CLAassistant commented Aug 3, 2022

CLA assistant check
All committers have signed the CLA.

Copy link

@tanjinx tanjinx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cool, thanks Tim @timvaillancourt, lets merge it and see if it works.

@tanjinx tanjinx merged commit 0495649 into main Aug 8, 2022
@timvaillancourt timvaillancourt deleted the add-repo-sync-job branch August 8, 2022 21:12
brirams pushed a commit that referenced this pull request Sep 22, 2022
* decouple olap tx timeout from oltp tx timeout

Since workload=olap bypasses the query timeouts
(--queryserver-config-query-timeout) and also row limits, the natural
assumption is that it also bypasses the transaction timeout.

This is not the case, e.g. for a tablet where the
--queryserver-config-transaction-timeout is 10.

This commit:

 * Adds new CLI flag and YAML field to independently configure TX
   timeouts for OLAP workloads (--queryserver-config-olap-transaction-timeout).
 * Decouples TX kill interval from OLTP TX timeout via new CLI flag and
   YAML field (--queryserver-config-transaction-killer-interval).

Signed-off-by: Max Englander <[email protected]>

* decouple ol{a,t}p tx timeouts: pr comments #1

Signed-off-by: Max Englander <[email protected]>

* decouple ol{a,t}p tx timeouts: pr comments #2 consolidate timeout logic in sc

Signed-off-by: Max Englander <[email protected]>

* decouple ol{a,t}p tx timeouts: remove unused tx killer flag

Signed-off-by: Max Englander <[email protected]>

* decouple ol{a,t}p tx timeouts: update 15_0_0_summary.md

Signed-off-by: Max Englander <[email protected]>

* decouple ol{a,t}p tx timeouts: fix race cond

Signed-off-by: Max Englander <[email protected]>

* decouple ol{a,t}p tx timeouts: pr comments #3 -txProps.timeout, +sc.expiryTime

Signed-off-by: Max Englander <[email protected]>

* decouple ol{a,t}p tx timeouts: pr comments #4 -atomic.Value for expiryTime

Signed-off-by: Max Englander <[email protected]>

* decouple ol{a,t}p tx timeouts: fix race cond (without atomic.Value)

Signed-off-by: Max Englander <[email protected]>

* decouple ol{a,t}p tx timeouts: pr comments #5 -unused funcs, fix comments, set ticks interval once

Signed-off-by: Max Englander <[email protected]>

* decouple ol{a,t}p tx timeouts: pr comments #5 +txkill tests

Signed-off-by: Max Englander <[email protected]>

* revert fmt changes

Signed-off-by: Max Englander <[email protected]>

* implement pr review suggestion

Signed-off-by: Max Englander <[email protected]>

Signed-off-by: Max Englander <[email protected]>
@tanjinx tanjinx added the v12 label Dec 9, 2022
@tanjinx tanjinx removed the v12 label Feb 17, 2023
timvaillancourt added a commit that referenced this pull request May 2, 2023
)

* Cleanup tx_throttler.go

Signed-off-by: Tim Vaillancourt <[email protected]>

* Cleanup tx_throttler.go #2

Signed-off-by: Tim Vaillancourt <[email protected]>

* Fix throttlerFactoryFunc

Signed-off-by: Tim Vaillancourt <[email protected]>

* Undo if-cond consolidation

Signed-off-by: Tim Vaillancourt <[email protected]>

* Undo struct shuffling

Signed-off-by: Tim Vaillancourt <[email protected]>

* prove that disabled config returns nil error

Signed-off-by: Tim Vaillancourt <[email protected]>

* Improve test

Signed-off-by: Tim Vaillancourt <[email protected]>

---------

Signed-off-by: Tim Vaillancourt <[email protected]>
timvaillancourt added a commit that referenced this pull request May 12, 2023
)

* Cleanup tx_throttler.go

Signed-off-by: Tim Vaillancourt <[email protected]>

* Cleanup tx_throttler.go #2

Signed-off-by: Tim Vaillancourt <[email protected]>

* Fix throttlerFactoryFunc

Signed-off-by: Tim Vaillancourt <[email protected]>

* Undo if-cond consolidation

Signed-off-by: Tim Vaillancourt <[email protected]>

* Undo struct shuffling

Signed-off-by: Tim Vaillancourt <[email protected]>

* prove that disabled config returns nil error

Signed-off-by: Tim Vaillancourt <[email protected]>

* Improve test

Signed-off-by: Tim Vaillancourt <[email protected]>

---------

Signed-off-by: Tim Vaillancourt <[email protected]>
timvaillancourt added a commit that referenced this pull request May 12, 2023
…itessio#12901 (#83)

* Cleanup panics in `txthrottler`, reorder for readability (vitessio#12901)

* Cleanup tx_throttler.go

Signed-off-by: Tim Vaillancourt <[email protected]>

* Cleanup tx_throttler.go #2

Signed-off-by: Tim Vaillancourt <[email protected]>

* Fix throttlerFactoryFunc

Signed-off-by: Tim Vaillancourt <[email protected]>

* Undo if-cond consolidation

Signed-off-by: Tim Vaillancourt <[email protected]>

* Undo struct shuffling

Signed-off-by: Tim Vaillancourt <[email protected]>

* prove that disabled config returns nil error

Signed-off-by: Tim Vaillancourt <[email protected]>

* Improve test

Signed-off-by: Tim Vaillancourt <[email protected]>

---------

Signed-off-by: Tim Vaillancourt <[email protected]>

* remove unused cell string

---------

Signed-off-by: Tim Vaillancourt <[email protected]>
timvaillancourt added a commit that referenced this pull request Oct 16, 2023
Signed-off-by: Tim Vaillancourt <[email protected]>
timvaillancourt added a commit that referenced this pull request Apr 16, 2024
)

* Cleanup tx_throttler.go

Signed-off-by: Tim Vaillancourt <[email protected]>

* Cleanup tx_throttler.go #2

Signed-off-by: Tim Vaillancourt <[email protected]>

* Fix throttlerFactoryFunc

Signed-off-by: Tim Vaillancourt <[email protected]>

* Undo if-cond consolidation

Signed-off-by: Tim Vaillancourt <[email protected]>

* Undo struct shuffling

Signed-off-by: Tim Vaillancourt <[email protected]>

* prove that disabled config returns nil error

Signed-off-by: Tim Vaillancourt <[email protected]>

* Improve test

Signed-off-by: Tim Vaillancourt <[email protected]>

---------

Signed-off-by: Tim Vaillancourt <[email protected]>
timvaillancourt added a commit that referenced this pull request May 7, 2024
)

* Cleanup tx_throttler.go

Signed-off-by: Tim Vaillancourt <[email protected]>

* Cleanup tx_throttler.go #2

Signed-off-by: Tim Vaillancourt <[email protected]>

* Fix throttlerFactoryFunc

Signed-off-by: Tim Vaillancourt <[email protected]>

* Undo if-cond consolidation

Signed-off-by: Tim Vaillancourt <[email protected]>

* Undo struct shuffling

Signed-off-by: Tim Vaillancourt <[email protected]>

* prove that disabled config returns nil error

Signed-off-by: Tim Vaillancourt <[email protected]>

* Improve test

Signed-off-by: Tim Vaillancourt <[email protected]>

---------

Signed-off-by: Tim Vaillancourt <[email protected]>
timvaillancourt added a commit that referenced this pull request May 10, 2024
* Add basic metrics to `vttablet` transaction throttler (vitessio#12418)

* Add basic stats to vttablet tx throttler

Signed-off-by: Tim Vaillancourt <[email protected]>

* test new metrics

Signed-off-by: Tim Vaillancourt <[email protected]>

* reorder

Signed-off-by: Tim Vaillancourt <[email protected]>

* short names

Signed-off-by: Tim Vaillancourt <[email protected]>

* Add max rate

Signed-off-by: Tim Vaillancourt <[email protected]>

* Move NewGaugeFunc to under conditional

Signed-off-by: Tim Vaillancourt <[email protected]>

* Use env

Signed-off-by: Tim Vaillancourt <[email protected]>

* Remove env from TxThrottler struct

Signed-off-by: Tim Vaillancourt <[email protected]>

* Fix tests

Signed-off-by: Tim Vaillancourt <[email protected]>

* PR suggestion

Signed-off-by: Tim Vaillancourt <[email protected]>

* Fix unit test

Signed-off-by: Tim Vaillancourt <[email protected]>

* reorder test vars

Signed-off-by: Tim Vaillancourt <[email protected]>

---------

Signed-off-by: Tim Vaillancourt <[email protected]>

* Fix transaction throttler ignoring the initial rate (vitessio#12618)

* Fix transaction throttler ignoring the initial rate

This addresses the issue reported in vitessio#12549

Signed-off-by: Eduardo J. Ortega U <[email protected]>

* Add missing override of max replication lag in `throttler.newThrottler()`

Signed-off-by: Eduardo J. Ortega U <[email protected]>

* Reorder functions to make diff easier to read

Signed-off-by: Eduardo J. Ortega U <[email protected]>

* Fix check for maxRate in `newThrottlerFromConfig()`

Signed-off-by: Eduardo J. Ortega U <[email protected]>

* Fix some CI pipeline issues

Signed-off-by: Eduardo J. Ortega U <[email protected]>

* Address PR comment.

Signed-off-by: Eduardo J. Ortega U <[email protected]>

* Fix typo

Signed-off-by: Eduardo J. Ortega U <[email protected]>

---------

Signed-off-by: Eduardo J. Ortega U <[email protected]>
Signed-off-by: Eduardo J. Ortega U. <[email protected]>

* Cleanup panics in `txthrottler`, reorder for readability (vitessio#12901)

* Cleanup tx_throttler.go

Signed-off-by: Tim Vaillancourt <[email protected]>

* Cleanup tx_throttler.go #2

Signed-off-by: Tim Vaillancourt <[email protected]>

* Fix throttlerFactoryFunc

Signed-off-by: Tim Vaillancourt <[email protected]>

* Undo if-cond consolidation

Signed-off-by: Tim Vaillancourt <[email protected]>

* Undo struct shuffling

Signed-off-by: Tim Vaillancourt <[email protected]>

* prove that disabled config returns nil error

Signed-off-by: Tim Vaillancourt <[email protected]>

* Improve test

Signed-off-by: Tim Vaillancourt <[email protected]>

---------

Signed-off-by: Tim Vaillancourt <[email protected]>

* Emit per workload labels for existing per table vttablet metrics (vitessio#12394)

* Emit per workload labels for existing per table vttablet metrics

This adds the possibility to configure vttablet (via CLI flag) to also have a
workload label for existing per table metrics (query counts, query times, query
errors, query rows affected, query rows returned, query error counts). Workload
can be any string that makes sense for the client application. For example, API
endpoint name, controller, batch job name, application name or something else.

This is usefult to be able to gain observability about how the query load is
distributed across different workloads.

This is achieved with two new CLI flags, namely:

* `enable-per-workload-table-metrics`: whether to enable or disable per
  workload metric collection - disabled by default to preserve the current
  behavior, thus making the new feature opt-in only.
* `workload-label`: a string to look for in query comments to identify the
  workload running the current query.

The workload is obtained by parsing query comments of the form:

/* ... <workload_label>=<workload_name>; ... */

For example, if vttablet is started with

`--enable-per-workload-table-metrics --workload-label app_name`

anda query is issued with a comment like

/* ... app_name=shop; ... */

then metrics will look like

```
vttablet_query_counts{plan="Select",table="dual", workload="shop"} 15479
```

instead of

```
vttablet_query_counts{plan="Select",table="dual"} 15479
```

Query comment parsing only takes place if `--enable-per-workload-table-metrics`
is used, as to not incur parsing performance impact if the user does not want
per workload metrics.

Signed-off-by: Eduardo J. Ortega U <[email protected]>

* make linter happy

Signed-off-by: Eduardo J. Ortega U <[email protected]>

* fix flags e2e test

Signed-off-by: Eduardo J. Ortega U <[email protected]>

* Address PR comments:

* Obtain workload information on the vtgate instead of the vttablet, avoiding
  double parsing.
* Treat workload name as a query directive.
* Send workload name from vtgate to vttablet as ExecuteOptions.

Additionally, annotate tabletserver's execution span with the workload name
to also enrich traces with workload name data, in addition to metrics.

Signed-off-by: Eduardo J. Ortega U <[email protected]>

* A few fixes:

1. Rebuild some files with `make proto`.
2. Protect against nil ExecuteOptions on the tabletserver.

Signed-off-by: Eduardo J. Ortega U <[email protected]>

* Fix flags e2e test

Signed-off-by: Eduardo J. Ortega U <[email protected]>

* Address PR comments

Signed-off-by: Eduardo J. Ortega U <[email protected]>

* Fixes

Signed-off-by: Eduardo J. Ortega U <[email protected]>

* Fix a comment

Signed-off-by: Eduardo J. Ortega U <[email protected]>

* Fix e2e flag test

Signed-off-by: Eduardo J. Ortega U <[email protected]>

* Update JS code for protobuf changes.

Signed-off-by: Eduardo J. Ortega U <[email protected]>

* Fix QueryEngine unit test

Signed-off-by: Eduardo J. Ortega U <[email protected]>

* Fix e2e flag test

Signed-off-by: Eduardo J. Ortega U <[email protected]>

* Fix spurious tab in comment

Signed-off-by: Eduardo J. Ortega U <[email protected]>

* Address PR comment

Don't use dual format flag for new flags - stick with - separated ones.

Signed-off-by: Eduardo J. Ortega U <[email protected]>

---------

Signed-off-by: Eduardo J. Ortega U <[email protected]>

* remove mistaken git add

Signed-off-by: Tim Vaillancourt <[email protected]>

* make vtadmin_web_proto_types

Signed-off-by: Tim Vaillancourt <[email protected]>

* test unit_race test on go-version: 1.18.9

Signed-off-by: Tim Vaillancourt <[email protected]>

* Revert "test unit_race test on go-version: 1.18.9"

This reverts commit 922e897.

* CI: Misc test improvements to limit failures with various runners (vitessio#13825)

Signed-off-by: Matt Lord <[email protected]>

* Fix setup order to avoid races (vitessio#13871)

Signed-off-by: Dirkjan Bussink <[email protected]>

---------

Signed-off-by: Tim Vaillancourt <[email protected]>
Signed-off-by: Eduardo J. Ortega U <[email protected]>
Signed-off-by: Eduardo J. Ortega U. <[email protected]>
Signed-off-by: Matt Lord <[email protected]>
Signed-off-by: Dirkjan Bussink <[email protected]>
Co-authored-by: Eduardo J. Ortega U <[email protected]>
Co-authored-by: Matt Lord <[email protected]>
Co-authored-by: Dirkjan Bussink <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants