Skip to content

Commit

Permalink
br: merge into feature branch br_ddl for binlog test (#31044)
Browse files Browse the repository at this point in the history
* dumpling: fix default collation with upstream when dump database and table (#30292)

* ddl: fix the enum default value by triming trailing space (#30356)

* expression: migrate test-infra to testify for flag_simplify_test.go (#30407)

* server: refine code logic in handleDownloadFile (#30422)

* refine logic

Signed-off-by: yisaer <[email protected]>

* fix

Signed-off-by: yisaer <[email protected]>

* ddl: migrate test-infra to testify for ddl/table_test.go (#30267)

* ddl: handle the error from `addBatchDDLJobs()` correctly (#30401)

* br: fix the integration tests (#30423)

* util, cmd: remove unused filesort (#30438)

* *: update client-go for small backoff time (#30436)

* server: Fix unstable tests with FakeAuthSwitch (#30287)

* dumpling: fix dump failed when sequence exists (#30164)

* *: replace compareDatum by compare (#30421)

* lightning: fix gcs max key limit (#30393)

* expression, parser: add built-in func is_uuid (#30318)

* expression: migrate test-infra to testify for constant_fold_test.go (#30424)

* executor: fix pipelined window invalid memory address (#30418)

* makefile: add gotestsum for verify ci (#29848)

* server: close sql rows to fix unstable test (#30306)

* Makefile: add coverage record for BR and Dumpling (#30457)

* executor: track the mem usage of IndexMergeReader (#30210)

* infosync: close body when ReadAll encounters error (#30462)

* planner: show accessed partition when explain mpp query over partition table (#30367)

* *: Fix use of user identity in SHOW GRANTS + error messages (#30294)

* ddl: add not null flag for auto_increment column  (#30477)

* expression: make some unstable test serial (#30323)

* expression: migrate test-infra to testify for constant_propagation_test.go (#30430)

* executor: stable test TestSetDDLReorgBatchSize and TestSetDDLReorgWorkerCnt (#30480)

* statistics, util/ranger: add cardinality estimation trace for `GetRowCountBy...` (#30321)

* *: skip mysql client goroutine leak detection in integration ddl (#30467)

* executor,util: write slow query to slow log no matter what log level (#30461)

* executor: enable index_merge used in transaction. (#29875)

* logutil: add testcase for SlowQueryLogger.MaxDays/MaxSize/MaxBackups (#30316)

* expression: fix data race in builtin_other_vec_generated_test.go (#30503)

* expression: fix data race in the collationInfo (#30490)

* planner/core, session: fix error message of wrong variable scope (#30510)

* lightning: support Re/ReregisterMySQL by different tls name (#30463)

* executor: TestBatchGetandPointGetwithHashPartition test typo (#29669) (#29671)

* mockstore: improve log to avoid panic for nil pointer (#30513)

* *: replace compareDatum by compare, PR 10 (#30456)

* planner: Disable dynamic partition prune mode for all non-autocommit (#27532) (#30505)

* expression: change the log level of an confusing log from warn to debug (#30484)

* br: Check crypter.key valid before backup (#29991)

* *: replace compareDatum by compare, PR 11 (#30465)

* dumpling: fix default column collation with upstream when dump table (#30531)

* server: fix prepared cursor select (#30285)

* executor: HashJoinExec checks the buildError even if the probeSide is empty (#30471)

* parser, expression: follow mysql, increase interval precedence (#30528)

* makefile: set timeout 25m for make race (#30555)

* planner: fix the unstable test TestAnalyzeGlobalStatsWithOpts/2 (#30576)

* expression,types: Adjusts UNIX_TIMESTAMP() for non-existing DST values (#28739) (#30405)

* br: add res.Body.close to avoid leak (#30545)

* lightning: add back integration test lightning_error_summary (#30547)

* sessionctx/variable: small refactor (split large file) (#30511)

* ddl: let `admin cancel ddl jobs` run in a new transaction (#30549)

* *: Retry when placement PutBundles failed (#30590)

* dumpling: delete unit test in github actions (#30562)

* *: support trace plan target='estimation' statement (#30491)

* expression: migrate test-infra to testify for integration_test.go (#30548)

* planner: support trace for min/max eliminate (#30441)

* support min/max trace

Signed-off-by: yisaer <[email protected]>

* address the comment

Signed-off-by: yisaer <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

* br: remove cdclog in br (#30573)

* *: show cmd to check if all needed histograms are loaded (#29672)

* expression: clone repertoire when clone the scalar function (#30602)

* *: use the real StateRemote interface implementation for cached table (#30066)

* *: query failed after add index / timestamp out-of-range (#28424) (#29323)

* planner: implement collecting predicate columns from logical plan (#29878)

* *: show PK name when decoding the clustered index row key (#30623)

* ddl/callback_test.go: migrate test-infra to testify (#30317)

* *: Rename some names of placement ddl operation (#30622)

* executor: fix data race in the index_lookup_hash_join (#30619)

* ddl: remove unnecessary locking when adding an index (#29772)

* server: try to make `TidbTestSuite` more stable (#30643)

* *: Add some PD tests for placement and fix some bug found (#30621)

* *: migrate sync.WaitGroup to util.WaitGroupWrapper (#30644)

* planner: add trace for join eliminate rule (#30343)

* executor: migrate test-infra to testify for executor/shuffle_test.go (#30514)

* planner: make (*AccessPath).OnlyPointRange more succinct (#30520)

* planner: add trace for join reorder (#30394)

* executor: migrate test-infra to testify for executor/union_scan_test.go (#30525)

* expression: make cast return error if cast binary literal to another character set (#30537)

* *: update tikv client (#30670)

* *: update sysutil in go.mod to fix panic when search log (#30523)

* topsql: shouldn't evict the SQL meta, since the evicted SQL can be appear on Other components (TiKV) TopN records (#27050)

* testify: migrate test-infra to testify for analyze_test.go (#30640)

* util: replace compareDatum by compare,  point part (#30575)

* test: make all the tests run in serial (#30692)

* statistics: add mutex for Handle.globalMap and Handle.feedback (#30550)

* executor: fix regular expression in json so that it could match identifer start with '$' (#29750)

* util/testkit/testkit.go: fix typo (#30638)

* planner: Introduce a new global variable to control the historical statistics feature (#30646)

* topsql: introduce datasink interface (#30662)

* planner: unify the argument of stats functions to use SessionCtx instead of StatementContext (#30668)

* metrics: fix the Max SafeTS Gap metrics (#30689)

* lightning: Add source dir existence check for s3 (#30674)

* golangci-lint: support durationcheck (#30027)

* executor: fix data race on IndexHashJoin.cancelFunc (#30701)

* sessionctx/variable: change tidb_store_limit to global only (#30522)

* statistics: remove reassignment of Handle.pool in NewHandle (#30675)

* br: fix some unstable unit test cases. (#30716)

* bindinfo: fix the comment typo (#30616)

* server: support decoding prepared string args to character_set_client (#30723)

* expression: fix enum type join binary get wrong result (#30445)

* cmd/explaintest: fix wrong result comparison for explain test (#30717)

* parallel create tables in br

* metrics: fix copr-cache metrics (#30712)

* test: merge executor's serial tests to other tests (#30711)

* statistics: avoid deadlock when create/drop extended stats and analyze at the same time (#30566)

* ddl: add batch create table api

Signed-off-by: xhe <[email protected]>

* ddl: add unit tests

Signed-off-by: xhe <[email protected]>

* ddl: fix fmt

Signed-off-by: xhe <[email protected]>

* ddl: typo

Co-authored-by: Arenatlx <[email protected]>

* ddl: fix tests

Signed-off-by: xhe <[email protected]>

* ddl: rename to BatchCreateTableWithInfo

Signed-off-by: xhe <[email protected]>

* ddl: trace the error

Signed-off-by: xhe <[email protected]>

* ddl: comments

Signed-off-by: xhe <[email protected]>

* ddl: cancle the job right

Signed-off-by: xhe <[email protected]>

* ddl: cancel the job right 2

Signed-off-by: xhe <[email protected]>

* ddl: report error if entry too large

Signed-off-by: xhe <[email protected]>

* ddl: report error when table is duplicated

Signed-off-by: xhe <[email protected]>

* ddl: go fmt

Signed-off-by: xhe <[email protected]>

* infoschema: improve batch memory perf

Signed-off-by: xhe <[email protected]>

* ddl: retain ID

Signed-off-by: xhe <[email protected]>

* sessionctx: fix the value of analyze_version when upgrading 4.x to 5.… (#30743)

* ddl: reduce log frequency

Signed-off-by: xhe <[email protected]>

* ddl: fix tests

Signed-off-by: xhe <[email protected]>

* server: disable socket listener for `basicHTTPHandlerTestSuite` (#30680)

* planner: support the plan cache aware of bindings (#30169)

* planner: fix early set of plan's statisticsTable (#30754)

* *: implement renew write lock lease for cached table (#30206)

* *: Modify placement rule index to reserve some indexes for future work (#30737)

* executor: add an unit test case for unreasonable invoking Close (#30696)

* planner: fix wrong subquery's coercibility  (#30750)

* executor: add more testcases for index merge (#30497)

* server: add grpc server config for a suitable behavior (#30774)

* config, charset: make charset config not affected by collation config (#30572)

* lightning: emit tidb log by change FilterCore to only allow matched packages (#30700)

* topsql: a centralized place to generate tipb report data (#30781)

* planner: add trace for partition pruning (#30546)

* planner: refine collation handling for between (#30793)

* test: merge serial tests in bindinfo, expression, parser and statistics (#30749)

* br: update log description for split check (#30763)

* *: replace compareDatum by compare, range part (#30710)

* *: placement policy ref will be converted to direct options when recover or flashback table (#30705)

* ddl: handle the incorrect number of placement followers (#30715)

* ddl: revert "ddl: remove unnecessary locking when adding an index" (#30667)

* br/pkg/task: migrate test-infra to testify (#30605)

* *: fix the flen type datetime for union/case-when/control-funcs (#30588)

* types, util: clean up compareDatum (#30815)

* ddl: add helper function to set and query TiFlash's sync status (#30473)

* dumpling: fix more dumpling log level query template (#30176)

* parser: support `admin flush plan_cache` (#30747)

* topsql: support multiple datasinks (#30808)

* br: update permission, so tikv can write to folder when backup to local (#30396)

* session: fix bootstrap to only persist global variables (#30593)

close #28667

* docs/design: update collation compatibility issues in charsets doc (#30806)

* executor: improve SET sysvar=DEFAULT handling (#29680)

close #29670

* br: add error handling for group context cancel when restore file is corrupted (#30190)

close #30135

* executor: buildWindow cannot call typeInfer twice (#30773)

close #30402

* *: refactor encoding and uniform usages (#30288)

* lightning: optimize region split check logic (#30428)

close #30018

* br: ignore mock directory when gcov in br (#30586)

* *: forbid set tiflash replica count for a placement table (#30844)

close #30741

* execute: don't transform charset in internal sql (#30843)

close #30789

* planner: update PlanBuilder.windowSpecs when building subquery (#30878)

close #30804

* br: fix S3 backup endpoint suffix (#30530)

close #30104

* lightning: make pre-check output message clearer (#30439)

close #30395

* expression: wrap to_binary and from_binary for cast function's argument (#30706)

* executor: fix bug when using IndexMerge in transaction (#30719)

close #30685

* ddl: migrate test-infra to testify for ddl/foreign_key_test.go (#30853)

close #29103

* expression: fix wrong retType for reverse function (#30829)

close #30809

* planner: support trace topn push down (#30800)

ref #29661

* github: add issue requirement to pull request template (#30817)

close #30814

* fix merge issue

* topsql: introduce stmtstats and sql execution count (#30277)

* topsql: add pubsub datasink (#30860)

* executor: fix the incorrect untouch used in optimistic transactions (#30447)

close #30410

* expression, cmd: let crc32() support gbk (#30900)

close #30898

* server: Add uptime status var and statistics (#29790)

close #8842

* br: error log optimization (#29640)

close #27015

* planner: fix wrong collation when rewrite in condition (#30492)

close #30486

* planner: add extractor for tikv_region_peers (#30656)

* fix issue that loss table restore

* lightning: add back table empty check and add a switch config (#30887)

close #27919

* br: improve backoff unit test (#30892)

* *: add TxnManager to manage txn in session (#30574)

* *: add TxnManager to manage txn in session

* modify

* add tests

* move failpoint content to a single file

* Makefile: add `t.Parallel` check to ensure tests are run in serial (#30869)

* refactoring code

* refactoring code

* placement: remove isolationlevel (#30859)

close #30858

* planner: revise the optimize trace output (#30882)

* table: set the datum collation correctly in CastValue() (#30931)

close #30930

* *: Use TxnManager.GetTxnInfoSchema() to get the txn infoschema (#30934)

close #30933

* parser: add IsValid() to Encoding to speed up string validation for UTF-8 (#30937)

close #30936

* planner: rename pstmtPlanCacheXX to PlanCacheXX (#30909)

* table/tables: make CI TestCacheTableBasicReadAndWrite more stable (#30924)

close #30922

* restore: use new ScatterRegions API (#30899)

close #30425

* *: when placement conflicts with tiflash, cancel the job (#30945)

* Makefile,tools: make CI great again! (#30828)

close #30822

* br/pkg/membuf: remove global buffer pool (#29934)

* ddl: add format error for incorrect dict syntax in the placement rule (#30919)

close #30454

* planner: fix index merge plan when expr cannot be pushed to tikv (#30341)

close #30200

* executor: display 'show create table' and INFOSCHEMA for cached table correctly (#30951)

close #30950

* br: extend the timeout for scan region since 3 seconds is not enough (#30889)

close #30720

* planner: remove bindSQL from planCacheKey to planCacheValue (#30916)

* execution: refine precision of cast as decimal in agg func (#30805)

* *: fix data race in the tikv_client (#30964)

close #30658

* ddl: migrate test-infra to testify for ddl/db_partition_test.go (#30952)

close #28635

* planner: fix `AccessPath.TableFilters` got modified unexpectedly (#30966)

close #30965

* test: merge serial tests in ddl, infoschema, session, store, table, telemetry and types (#30874)

* executor: fix the returned field count of the prepare statement (#30981)

close #30971

* binlog: allow multiple ddl targets (#30904)

* planner: trace predicate push down  (#30902)

ref #29661

* placement: give default 2 followers for non-sugar syntax (#31000)

* flatten the json output (#30905)

Signed-off-by: yisaer <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

* test: control log level with environment variables (#30871)

* planner: add usage of paging copr in optimizer (#30536)

close #30578

* test: merge serial tests in cmd, planner, server, util (#31003)

* planner: change predicateColumnCollector to columnStatsUsageCollector and collect histogram-needed columns (#30671)

* executor: migrate test-infra to testify for distsql_test.go (#31023)

close #28574

* remote uncessary package errors

* reused the retry code from lightning

* refactoring retryable

* ddl: add batch create table api

Signed-off-by: xhe <[email protected]>

* ddl: add unit tests

Signed-off-by: xhe <[email protected]>

* ddl: fix fmt

Signed-off-by: xhe <[email protected]>

* br ddl code

* parallel create tables in br

* ddl: add batch create table api

Signed-off-by: xhe <[email protected]>

* ddl: add unit tests

Signed-off-by: xhe <[email protected]>

* ddl: fix fmt

Signed-off-by: xhe <[email protected]>

* ddl: typo

Co-authored-by: Arenatlx <[email protected]>

* ddl: fix tests

Signed-off-by: xhe <[email protected]>

* ddl: rename to BatchCreateTableWithInfo

Signed-off-by: xhe <[email protected]>

* ddl: trace the error

Signed-off-by: xhe <[email protected]>

* ddl: comments

Signed-off-by: xhe <[email protected]>

* ddl: cancle the job right

Signed-off-by: xhe <[email protected]>

* ddl: cancel the job right 2

Signed-off-by: xhe <[email protected]>

* ddl: report error if entry too large

Signed-off-by: xhe <[email protected]>

* ddl: report error when table is duplicated

Signed-off-by: xhe <[email protected]>

* ddl: go fmt

Signed-off-by: xhe <[email protected]>

* infoschema: improve batch memory perf

Signed-off-by: xhe <[email protected]>

* ddl: retain ID

Signed-off-by: xhe <[email protected]>

* ddl: reduce log frequency

Signed-off-by: xhe <[email protected]>

* ddl: fix tests

Signed-off-by: xhe <[email protected]>

* ddl: remove retainID from the interface

Signed-off-by: xhe <[email protected]>

* ddl: fix tests

Signed-off-by: xhe <[email protected]>

* executor: fix rebasing problem

Signed-off-by: xhe <[email protected]>

Co-authored-by: WizardXiao <[email protected]>
Co-authored-by: sylzd <[email protected]>
Co-authored-by: tison <[email protected]>
Co-authored-by: Song Gao <[email protected]>
Co-authored-by: Weizhen Wang <[email protected]>
Co-authored-by: tangenta <[email protected]>
Co-authored-by: 3pointer <[email protected]>
Co-authored-by: wjHuang <[email protected]>
Co-authored-by: Lei Zhao <[email protected]>
Co-authored-by: Daniël van Eeden <[email protected]>
Co-authored-by: Jianjun Liao <[email protected]>
Co-authored-by: unconsolable <[email protected]>
Co-authored-by: Shenghui Wu <[email protected]>
Co-authored-by: guo-shaoge <[email protected]>
Co-authored-by: Ryan Leung <[email protected]>
Co-authored-by: xufei <[email protected]>
Co-authored-by: Morgan Tocker <[email protected]>
Co-authored-by: Zhou Kunqin <[email protected]>
Co-authored-by: 王超 <[email protected]>
Co-authored-by: TonsnakeLin <[email protected]>
Co-authored-by: Ehco <[email protected]>
Co-authored-by: Mattias Jonsson <[email protected]>
Co-authored-by: HuaiyuXu <[email protected]>
Co-authored-by: Zak Zhao <[email protected]>
Co-authored-by: xhe <[email protected]>
Co-authored-by: Hangjie Mo <[email protected]>
Co-authored-by: Yuanjia Zhang <[email protected]>
Co-authored-by: glorv <[email protected]>
Co-authored-by: djshow832 <[email protected]>
Co-authored-by: Chunzhu Li <[email protected]>
Co-authored-by: Ti Chi Robot <[email protected]>
Co-authored-by: Xiaoju Wu <[email protected]>
Co-authored-by: xiongjiwei <[email protected]>
Co-authored-by: tiancaiamao <[email protected]>
Co-authored-by: Yifan Xu <[email protected]>
Co-authored-by: JmPotato <[email protected]>
Co-authored-by: Zach <[email protected]>
Co-authored-by: bb7133 <[email protected]>
Co-authored-by: lvtu <[email protected]>
Co-authored-by: crazycs <[email protected]>
Co-authored-by: znhh6018 <[email protected]>
Co-authored-by: eddie lin <[email protected]>
Co-authored-by: dongjunduo <[email protected]>
Co-authored-by: Zhenchi <[email protected]>
Co-authored-by: wangggong <[email protected]>
Co-authored-by: zhangjinpeng1987 <[email protected]>
Co-authored-by: Jack Yu <[email protected]>
Co-authored-by: Arenatlx <[email protected]>
Co-authored-by: Yiding Cui <[email protected]>
Co-authored-by: Chengpeng Yan <[email protected]>
Co-authored-by: bestwoody <[email protected]>
Co-authored-by: Calvin Neo <[email protected]>
Co-authored-by: Lynn <[email protected]>
Co-authored-by: Zhuhe Fang <[email protected]>
Co-authored-by: Mini256 <[email protected]>
Co-authored-by: Xiang Zhang <[email protected]>
Co-authored-by: Yexiang Zhang <[email protected]>
Co-authored-by: cfzjywxk <[email protected]>
Co-authored-by: db <[email protected]>
Co-authored-by: 山岚 <[email protected]>
Co-authored-by: Yujie Xia <[email protected]>
Co-authored-by: Yilong Li <[email protected]>
Co-authored-by: tuuuuuu <[email protected]>
Co-authored-by: qupeng <[email protected]>
Co-authored-by: you06 <[email protected]>
  • Loading branch information
Show file tree
Hide file tree
Showing 506 changed files with 23,647 additions and 15,609 deletions.
1 change: 1 addition & 0 deletions .codecov.yml
Original file line number Diff line number Diff line change
Expand Up @@ -43,4 +43,5 @@ ignore:
- "executor/seqtest/.*"
- "metrics/.*"
- "expression/generator/.*"
- "br/pkg/mock/.*"

10 changes: 10 additions & 0 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,16 @@ PR Title Format:
-->

### What problem does this PR solve?
<!--
Please create an issue first to describe the problem.
There MUST be one line starting with "Issue Number: " and
linking the relevant issues via the "close" or "ref".
For more info, check https://pingcap.github.io/tidb-dev-guide/contribute-to-tidb/contribute-code.html#referring-to-an-issue.
-->

Issue Number: close #xxx

Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ coverage.out
*.iml
*.swp
*.log
*.test.bin
tags
profile.coverprofile
explain_test
Expand Down
50 changes: 23 additions & 27 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ dev: checklist check explaintest devgotest gogenerate br_unit_test test_part_par
# Install the check tools.
check-setup:tools/bin/revive tools/bin/goword

check: fmt unconvert lint tidy testSuite check-static vet errdoc
check: fmt check-parallel unconvert lint tidy testSuite check-static vet errdoc

fmt:
@echo "gofmt (simplify)"
Expand Down Expand Up @@ -75,6 +75,13 @@ testSuite:
@echo "testSuite"
./tools/check/check_testSuite.sh

check-parallel:
# Make sure no tests are run in parallel to prevent possible unstable tests.
# See https://github.com/pingcap/tidb/pull/30692.
@! find . -name "*_test.go" -not -path "./vendor/*" -print0 | \
xargs -0 grep -F -n "t.Parallel()" || \
! echo "Error: all the go tests should be run in serial."

clean: failpoint-disable
$(GO) clean -i ./...

Expand Down Expand Up @@ -119,30 +126,23 @@ devgotest: failpoint-enable
$(GOTEST) -ldflags '$(TEST_LDFLAGS)' $(EXTRA_TEST_ARGS) -cover $(PACKAGES_TIDB_TESTS) -check.p true > gotest.log || { $(FAILPOINT_DISABLE); grep -v '^\([[]20\|PASS:\|ok \)' 'gotest.log'; exit 1; }
@$(FAILPOINT_DISABLE)

ut: failpoint-enable tools/bin/ut
tools/bin/ut $(X);
@$(FAILPOINT_DISABLE)

gotest: failpoint-enable
@echo "Running in native mode."
@export log_level=info; export TZ='Asia/Shanghai'; \
$(GOTEST) -ldflags '$(TEST_LDFLAGS)' $(EXTRA_TEST_ARGS) -timeout 20m -cover $(PACKAGES_TIDB_TESTS) -coverprofile=coverage.txt -check.p true > gotest.log || { $(FAILPOINT_DISABLE); cat 'gotest.log'; exit 1; }
@$(FAILPOINT_DISABLE)

gotest_in_verify_ci_part_1: failpoint-enable tools/bin/gotestsum tools/bin/gocov tools/bin/gocov-xml
@echo "Running gotest_in_verify_ci_part_1."
gotest_in_verify_ci: failpoint-enable tools/bin/gotestsum
@echo "Running gotest_in_verify_ci"
@mkdir -p $(TEST_COVERAGE_DIR)
@export log_level=info; export TZ='Asia/Shanghai'; \
CGO_ENABLED=1 tools/bin/gotestsum --junitfile "$(TEST_COVERAGE_DIR)/tidb-junit-report.xml" -- -v -p $(P) --race \
@export TZ='Asia/Shanghai'; \
CGO_ENABLED=1 tools/bin/gotestsum --junitfile "$(TEST_COVERAGE_DIR)/tidb-junit-report.xml" -- -v -p $(P) \
-ldflags '$(TEST_LDFLAGS)' $(EXTRA_TEST_ARGS) -coverprofile="$(TEST_COVERAGE_DIR)/tidb_cov.unit_test.out" \
$(PACKAGES_TIDB_TESTS_EXPENSIVE) -check.p true || { $(FAILPOINT_DISABLE); exit 1; }
tools/bin/gocov convert "$(TEST_COVERAGE_DIR)/tidb_cov.unit_test.out" | tools/bin/gocov-xml > "$(TEST_COVERAGE_DIR)/tidb-coverage.xml"
@$(FAILPOINT_DISABLE)

gotest_in_verify_ci_part_2: failpoint-enable tools/bin/gotestsum tools/bin/gocov tools/bin/gocov-xml
@echo "Running gotest_in_verify_ci_part_2."
@mkdir -p $(TEST_COVERAGE_DIR)
@export log_level=info; export TZ='Asia/Shanghai'; \
CGO_ENABLED=1 tools/bin/gotestsum --junitfile "$(TEST_COVERAGE_DIR)/tidb-junit-report.xml" -- -v -p $(P) --race \
-ldflags '$(TEST_LDFLAGS)' $(EXTRA_TEST_ARGS) -coverprofile="$(TEST_COVERAGE_DIR)/tidb_cov.unit_test.out" \
$(PACKAGES_TIDB_TESTS_OTHERS) -check.p true || { $(FAILPOINT_DISABLE); exit 1; }
tools/bin/gocov convert "$(TEST_COVERAGE_DIR)/tidb_cov.unit_test.out" | tools/bin/gocov-xml > "$(TEST_COVERAGE_DIR)/tidb-coverage.xml"
$(PACKAGES_TIDB_TESTS) -check.p true || { $(FAILPOINT_DISABLE); exit 1; }
@$(FAILPOINT_DISABLE)

race: failpoint-enable
Expand Down Expand Up @@ -213,6 +213,10 @@ failpoint-disable: tools/bin/failpoint-ctl
# Restoring gofail failpoints...
@$(FAILPOINT_DISABLE)

tools/bin/ut: tools/check/ut.go
cd tools/check; \
$(GO) build -o ../bin/ut ut.go

tools/bin/megacheck: tools/check/go.mod
cd tools/check; \
$(GO) build -o ../bin/megacheck honnef.co/go/tools/cmd/megacheck
Expand Down Expand Up @@ -326,13 +330,12 @@ br_unit_test:
$(GOTEST) $(RACE_FLAG) -ldflags '$(LDFLAGS)' -tags leak $(ARGS) -coverprofile=coverage.txt || ( make failpoint-disable && exit 1 )
@make failpoint-disable
br_unit_test_in_verify_ci: export ARGS=$$($(BR_PACKAGES))
br_unit_test_in_verify_ci: tools/bin/gotestsum tools/bin/gocov tools/bin/gocov-xml
br_unit_test_in_verify_ci: tools/bin/gotestsum
@make failpoint-enable
@export TZ='Asia/Shanghai';
@mkdir -p $(TEST_COVERAGE_DIR)
CGO_ENABLED=1 tools/bin/gotestsum --junitfile "$(TEST_COVERAGE_DIR)/br-junit-report.xml" -- $(RACE_FLAG) -ldflags '$(LDFLAGS)' \
-tags leak $(ARGS) -coverprofile="$(TEST_COVERAGE_DIR)/br_cov.unit_test.out" || ( make failpoint-disable && exit 1 )
tools/bin/gocov convert "$(TEST_COVERAGE_DIR)/br_cov.unit_test.out" | tools/bin/gocov-xml > "$(TEST_COVERAGE_DIR)/br-coverage.xml"
@make failpoint-disable

br_integration_test: br_bins build_br build_for_br_integration_test
Expand Down Expand Up @@ -392,11 +395,10 @@ dumpling_unit_test: failpoint-enable
$(DUMPLING_GOTEST) $(RACE_FLAG) -coverprofile=coverage.txt -covermode=atomic -tags leak $(DUMPLING_ARGS) || ( make failpoint-disable && exit 1 )
@make failpoint-disable
dumpling_unit_test_in_verify_ci: export DUMPLING_ARGS=$$($(DUMPLING_PACKAGES))
dumpling_unit_test_in_verify_ci: failpoint-enable tools/bin/gotestsum tools/bin/gocov tools/bin/gocov-xml
dumpling_unit_test_in_verify_ci: failpoint-enable tools/bin/gotestsum
@mkdir -p $(TEST_COVERAGE_DIR)
CGO_ENABLED=1 tools/bin/gotestsum --junitfile "$(TEST_COVERAGE_DIR)/dumpling-junit-report.xml" -- -tags leak $(DUMPLING_ARGS) \
$(RACE_FLAG) -coverprofile="$(TEST_COVERAGE_DIR)/dumpling_cov.unit_test.out" || ( make failpoint-disable && exit 1 )
tools/bin/gocov convert "$(TEST_COVERAGE_DIR)/dumpling_cov.unit_test.out" | tools/bin/gocov-xml > "$(TEST_COVERAGE_DIR)/dumpling-coverage.xml"
@make failpoint-disable

dumpling_integration_test: dumpling_bins failpoint-enable build_dumpling
Expand All @@ -420,9 +422,3 @@ dumpling_bins:

tools/bin/gotestsum: tools/check/go.mod
cd tools/check && $(GO) build -o ../bin/gotestsum gotest.tools/gotestsum

tools/bin/gocov: tools/check/go.mod
cd tools/check && $(GO) build -o ../bin/gocov github.com/axw/gocov/gocov

tools/bin/gocov-xml: tools/check/go.mod
cd tools/check && $(GO) build -o ../bin/gocov-xml github.com/AlekSi/gocov-xml
3 changes: 0 additions & 3 deletions Makefile.common
Original file line number Diff line number Diff line change
Expand Up @@ -47,11 +47,8 @@ MAC := "Darwin"

PACKAGE_LIST := go list ./...
PACKAGE_LIST_TIDB_TESTS := go list ./... | grep -vE "github.com\/pingcap\/tidb\/br|github.com\/pingcap\/tidb\/cmd|github.com\/pingcap\/tidb\/dumpling"
PACKAGE_LIST_TEST_OTHERS := go list ./... | grep -vE "github.com\/pingcap\/tidb\/br|github.com\/pingcap\/tidb\/cmd|github.com\/pingcap\/tidb\/dumpling|github.com\/pingcap\/tidb\/executor|github.com\/pingcap\/tidb\/cmd|github.com\/pingcap\/tidb\/ddl"
PACKAGES ?= $$($(PACKAGE_LIST))
PACKAGES_TIDB_TESTS ?= $$($(PACKAGE_LIST_TIDB_TESTS))
PACKAGES_TIDB_TESTS_EXPENSIVE ?= "github.com/pingcap/tidb/executor" "github.com/pingcap/tidb/ddl"
PACKAGES_TIDB_TESTS_OTHERS ?= $$($(PACKAGE_LIST_TEST_OTHERS))
PACKAGE_DIRECTORIES := $(PACKAGE_LIST) | sed 's|github.com/pingcap/$(PROJECT)/||'
PACKAGE_DIRECTORIES_TIDB_TESTS := $(PACKAGE_LIST_TIDB_TESTS) | sed 's|github.com/pingcap/$(PROJECT)/||'
FILES := $$(find $$($(PACKAGE_DIRECTORIES)) -name "*.go")
Expand Down
File renamed without changes.
File renamed without changes.
6 changes: 3 additions & 3 deletions bindinfo/handle.go
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ func (h *BindHandle) Update(fullLoad bool) (err error) {
}

exec := h.sctx.Context.(sqlexec.RestrictedSQLExecutor)
stmt, err := exec.ParseWithParams(context.TODO(), `SELECT original_sql, bind_sql, default_db, status, create_time, update_time, charset, collation, source
stmt, err := exec.ParseWithParamsInternal(context.TODO(), `SELECT original_sql, bind_sql, default_db, status, create_time, update_time, charset, collation, source
FROM mysql.bind_info WHERE update_time > %? ORDER BY update_time, create_time`, updateTime)
if err != nil {
return err
Expand Down Expand Up @@ -697,7 +697,7 @@ func (h *BindHandle) extractCaptureFilterFromStorage() (filter *captureFilter) {
tables: make(map[stmtctx.TableEntry]struct{}),
}
exec := h.sctx.Context.(sqlexec.RestrictedSQLExecutor)
stmt, err := exec.ParseWithParams(context.TODO(), `SELECT filter_type, filter_value FROM mysql.capture_plan_baselines_blacklist order by filter_type`)
stmt, err := exec.ParseWithParamsInternal(context.TODO(), `SELECT filter_type, filter_value FROM mysql.capture_plan_baselines_blacklist order by filter_type`)
if err != nil {
logutil.BgLogger().Warn("[sql-bind] failed to parse query for mysql.capture_plan_baselines_blacklist load", zap.Error(err))
return
Expand Down Expand Up @@ -923,7 +923,7 @@ func (h *BindHandle) SaveEvolveTasksToStore() {
}

func getEvolveParameters(ctx sessionctx.Context) (time.Duration, time.Time, time.Time, error) {
stmt, err := ctx.(sqlexec.RestrictedSQLExecutor).ParseWithParams(
stmt, err := ctx.(sqlexec.RestrictedSQLExecutor).ParseWithParamsInternal(
context.TODO(),
"SELECT variable_name, variable_value FROM mysql.global_variables WHERE variable_name IN (%?, %?, %?)",
variable.TiDBEvolvePlanTaskMaxTime,
Expand Down
File renamed without changes.
2 changes: 1 addition & 1 deletion bindinfo/main_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ import (
)

func TestMain(m *testing.M) {
testbridge.WorkaroundGoCheckFlags()
testbridge.SetupForCommonTest()
opts := []goleak.Option{
goleak.IgnoreTopFunction("go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop"),
goleak.IgnoreTopFunction("go.opencensus.io/stats/view.(*worker).start"),
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
8 changes: 7 additions & 1 deletion br/pkg/backup/client.go
Original file line number Diff line number Diff line change
Expand Up @@ -470,7 +470,13 @@ func (bc *Client) BackupRanges(
elctx := logutil.ContextWithField(ectx, logutil.RedactAny("range-sn", id))
err := bc.BackupRange(elctx, sk, ek, req, metaWriter, progressCallBack)
if err != nil {
return errors.Trace(err)
// The error due to context cancel, stack trace is meaningless, the stack shall be suspended (also clear)
if errors.Cause(err) == context.Canceled {
return errors.SuspendStack(err)
} else {
return errors.Trace(err)
}

}
return nil
})
Expand Down
2 changes: 1 addition & 1 deletion br/pkg/conn/main_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,6 @@ func TestMain(m *testing.M) {
goleak.IgnoreTopFunction("go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop"),
goleak.IgnoreTopFunction("go.opencensus.io/stats/view.(*worker).start"),
}
testbridge.WorkaroundGoCheckFlags()
testbridge.SetupForCommonTest()
goleak.VerifyTestMain(m, opts...)
}
4 changes: 2 additions & 2 deletions br/pkg/gluetidb/glue.go
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ func (gs *tidbSession) CreateDatabase(ctx context.Context, schema *model.DBInfo)
if len(schema.Charset) == 0 {
schema.Charset = mysql.DefaultCharset
}
return d.CreateSchemaWithInfo(gs.se, schema, ddl.OnExistIgnore, true)
return d.CreateSchemaWithInfo(gs.se, schema, ddl.OnExistIgnore)
}

// CreateTable implements glue.Session.
Expand Down Expand Up @@ -188,7 +188,7 @@ func (gs *tidbSession) CreateTable(ctx context.Context, dbName model.CIStr, tabl
newPartition.Definitions = append([]model.PartitionDefinition{}, table.Partition.Definitions...)
table.Partition = &newPartition
}
return d.CreateTableWithInfo(gs.se, dbName, table, ddl.OnExistIgnore, true)
return d.CreateTableWithInfo(gs.se, dbName, table, ddl.OnExistIgnore)
}

// Close implements glue.Session.
Expand Down
2 changes: 1 addition & 1 deletion br/pkg/kv/main_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,6 @@ import (
)

func TestMain(m *testing.M) {
testbridge.WorkaroundGoCheckFlags()
testbridge.SetupForCommonTest()
goleak.VerifyTestMain(m)
}
2 changes: 0 additions & 2 deletions br/pkg/lightning/backend/backend.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,12 +25,10 @@ import (
"github.com/pingcap/failpoint"
"github.com/pingcap/tidb/br/pkg/lightning/backend/kv"
"github.com/pingcap/tidb/br/pkg/lightning/checkpoints"
"github.com/pingcap/tidb/br/pkg/lightning/config"
"github.com/pingcap/tidb/br/pkg/lightning/log"
"github.com/pingcap/tidb/br/pkg/lightning/metric"
"github.com/pingcap/tidb/br/pkg/lightning/mydump"
"github.com/pingcap/tidb/br/pkg/utils"
"github.com/pingcap/tidb/parser/model"
"github.com/pingcap/tidb/table"
"go.uber.org/zap"
)
Expand Down
2 changes: 1 addition & 1 deletion br/pkg/lightning/backend/local/engine.go
Original file line number Diff line number Diff line change
Expand Up @@ -883,7 +883,7 @@ func (e *Engine) loadEngineMeta() error {
jsonBytes, closer, err := e.db.Get(engineMetaKey)
if err != nil {
if err == pebble.ErrNotFound {
log.L().Debug("local db missing engine meta", zap.Stringer("uuid", e.UUID), zap.Error(err))
log.L().Debug("local db missing engine meta", zap.Stringer("uuid", e.UUID), log.ShortError(err))
return nil
}
return err
Expand Down
25 changes: 17 additions & 8 deletions br/pkg/lightning/backend/local/local.go
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,8 @@ const (
gRPCBackOffMaxDelay = 10 * time.Minute

// See: https://github.com/tikv/tikv/blob/e030a0aae9622f3774df89c62f21b2171a72a69e/etc/config-template.toml#L360
regionMaxKeyCount = 1_440_000
// lower the max-key-count to avoid tikv trigger region auto split
regionMaxKeyCount = 1_280_000
defaultRegionSplitSize = 96 * units.MiB

propRangeIndex = "tikv.range_index"
Expand Down Expand Up @@ -149,9 +150,9 @@ type local struct {
duplicateDetection bool
duplicateDB *pebble.DB
errorMgr *errormanager.ErrorManager
}

var bufferPool = membuf.NewPool(1024, manual.Allocator{})
bufferPool *membuf.Pool
}

func openDuplicateDB(storeDir string) (*pebble.DB, error) {
dbPath := filepath.Join(storeDir, duplicateDBName)
Expand Down Expand Up @@ -243,6 +244,8 @@ func NewLocalBackend(
checkTiKVAvaliable: cfg.App.CheckRequirements,
duplicateDB: duplicateDB,
errorMgr: errorMgr,

bufferPool: membuf.NewPool(membuf.WithAllocator(manual.Allocator{})),
}
local.conns = common.NewGRPCConns()
if err = local.checkMultiIngestSupport(ctx); err != nil {
Expand Down Expand Up @@ -422,6 +425,7 @@ func (local *local) Close() {
engine.unlock()
}
local.conns.Close()
local.bufferPool.Destroy()

if local.duplicateDB != nil {
// Check whether there are duplicates.
Expand Down Expand Up @@ -775,14 +779,19 @@ func (local *local) WriteToTiKV(
requests = append(requests, req)
}

bytesBuf := bufferPool.NewBuffer()
bytesBuf := local.bufferPool.NewBuffer()
defer bytesBuf.Destroy()
pairs := make([]*sst.Pair, 0, local.batchWriteKVPairs)
count := 0
size := int64(0)
totalCount := int64(0)
firstLoop := true
regionMaxSize := regionSplitSize * 4 / 3
// if region-split-size <= 96MiB, we bump the threshold a bit to avoid too many retry split
// because the range-properties is not 100% accurate
regionMaxSize := regionSplitSize
if regionSplitSize <= defaultRegionSplitSize {
regionMaxSize = regionSplitSize * 4 / 3
}

for iter.First(); iter.Valid(); iter.Next() {
size += int64(len(iter.Key()) + len(iter.Value()))
Expand Down Expand Up @@ -1658,14 +1667,14 @@ func (local *local) LocalWriter(ctx context.Context, cfg *backend.LocalWriterCon
return nil, errors.Errorf("could not find engine for %s", engineUUID.String())
}
engine := e.(*Engine)
return openLocalWriter(cfg, engine, local.localWriterMemCacheSize)
return openLocalWriter(cfg, engine, local.localWriterMemCacheSize, local.bufferPool.NewBuffer())
}

func openLocalWriter(cfg *backend.LocalWriterConfig, engine *Engine, cacheSize int64) (*Writer, error) {
func openLocalWriter(cfg *backend.LocalWriterConfig, engine *Engine, cacheSize int64, kvBuffer *membuf.Buffer) (*Writer, error) {
w := &Writer{
engine: engine,
memtableSizeLimit: cacheSize,
kvBuffer: bufferPool.NewBuffer(),
kvBuffer: kvBuffer,
isKVSorted: cfg.IsKVSorted,
isWriteBatchSorted: true,
}
Expand Down
6 changes: 5 additions & 1 deletion br/pkg/lightning/backend/local/local_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ import (
"github.com/pingcap/tidb/br/pkg/lightning/backend/kv"
"github.com/pingcap/tidb/br/pkg/lightning/common"
"github.com/pingcap/tidb/br/pkg/lightning/mydump"
"github.com/pingcap/tidb/br/pkg/membuf"
"github.com/pingcap/tidb/br/pkg/mock"
"github.com/pingcap/tidb/br/pkg/pdutil"
"github.com/pingcap/tidb/br/pkg/restore"
Expand Down Expand Up @@ -357,7 +358,10 @@ func testLocalWriter(c *C, needSort bool, partitialSort bool) {
f.wg.Add(1)
go f.ingestSSTLoop()
sorted := needSort && !partitialSort
w, err := openLocalWriter(&backend.LocalWriterConfig{IsKVSorted: sorted}, f, 1024)
pool := membuf.NewPool()
defer pool.Destroy()
kvBuffer := pool.NewBuffer()
w, err := openLocalWriter(&backend.LocalWriterConfig{IsKVSorted: sorted}, f, 1024, kvBuffer)
c.Assert(err, IsNil)

ctx := context.Background()
Expand Down
5 changes: 5 additions & 0 deletions br/pkg/lightning/backend/local/localhelper_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,11 @@ func newTestClient(
}
}

// ScatterRegions scatters regions in a batch.
func (c *testClient) ScatterRegions(ctx context.Context, regionInfo []*restore.RegionInfo) error {
return nil
}

func (c *testClient) GetAllRegions() map[uint64]*restore.RegionInfo {
c.mu.RLock()
defer c.mu.RUnlock()
Expand Down
7 changes: 6 additions & 1 deletion br/pkg/lightning/common/storage_unix.go
Original file line number Diff line number Diff line change
Expand Up @@ -24,13 +24,18 @@ import (
"syscall"

"github.com/pingcap/errors"
"github.com/pingcap/failpoint"
"golang.org/x/sys/unix"
)

// GetStorageSize gets storage's capacity and available size
func GetStorageSize(dir string) (size StorageSize, err error) {
var stat unix.Statfs_t
failpoint.Inject("GetStorageSize", func(val failpoint.Value) {
injectedSize := val.(int)
failpoint.Return(StorageSize{Capacity: uint64(injectedSize), Available: uint64(injectedSize)}, nil)
})

var stat unix.Statfs_t
err = unix.Statfs(dir, &stat)
if err != nil {
return size, errors.Annotatef(err, "cannot get disk capacity at %s", dir)
Expand Down
Loading

0 comments on commit 609aa00

Please sign in to comment.