util: introduce tracing-based test toolkit

[lysu]

Signed-off-by: lysu [email protected]

What problem does this PR solve?

Problem Summary:

failpoint is powerful, but it's still not easy to write testcase to simulate multiple routines' execution in white-box test.

this simple PR just want to let testcase can control execution path like using "programmable-debugger"(try delve but failed)

and take benefit from opentracing, it support "assert opentracing log" like "assert explain result"

IMHO, it maybe useful to write whitebox-testcast to simulate conflict or event happen in different routines as different order.

What is changed and how it works?

What's Changed, How it Works:

// Package tracingtest provide a simple tracing-based test toolkit.
// with it, white-box-testcase writers can:
//
//  - add/continue breakpoints for routines to simulate different execution orders
//  - ensure routines wait on special breakpoint or ensure it has finished without unstable `time.Sleep(n)`
//  - enable/disable failpoint in routine level easily
//  - collect and assert execution path pattern by tracing-log to ensure routine cover desired code-path.
//
// it's standing on the shoulders of giants, it combines failpoint, opentracing, panicparse...
// it's not a substitute for failpoint, it bases on failpoint and wants to simulate execution flow easier
// and support assert execution path just like assert explain result
//
// Terminology
//
//  - Sandbox: every test using tracingtest need create a Sandbox to manage and monitor test Routines.
//  - Routine: one test can have one or many routines(i.e. different txn), it derived from Sandbox and be managed and monitored by Sandbox, routine's logic need use routine's `context.Context`
//  - Breakpoint: one routine can set several breakpoints, and each breakpoint can be hit or continued individually and also support wait breakpoint be hit
//
// How to Use
//
// 1. create an `Sandbox` at first in test
// 2. divide testcase to several routine and `StartRoutine` for each of them(e.g. one routine for one txn)
// 3. add `CheckBreakpoint` to place that want to wait on
// 4. `EnableBreakpoint` for special routine's `context.Context`
// 5. do testcase and using right routine's `context.Context`
// 6. check whether breakpoint has hit via `EnsureWaitOnBreakpoint`
// 7. continue do testcase or `EnableBreakpoint` for another breakpoint if needed
// 8. `ContinueBreakpoint` to resume waited routine
// 9. use `EnsureGoroutineFinished` to ensure some goroutine has finished
// 10. use `CollectLogs` to collect execution path info and do assert
// 11. build TiDB-server with `make failpoint-enable` then run testcase
//
// see example testcase in: 2pc_test.go => TestCommitSecondarySlow and 1pc_test.go => Test1PCOnRegionSplit
package tracingtest

also write 2 testcase demo in 1pc_test and 2pc_test

Related changes

• n/a

Check List

Tests

• Unit test

Side effects

• n/a

Release note

• No release note.

---

This change is 

[sre-bot]

No release note, Please follow https://github.com/pingcap/community/blob/master/contributors/release-note-checker.md

[lysu]

/rebuild

[tiancaiamao]

The core part are those three operation:

EnableBreakpoint = (bind a break point name with a channel)

CheckBreakpoint = (find the channel by break point name, block reading on that channel)

ContinueBreakpoint = (find the channel by break point name, send on that channel to unblock it)

And then

EnsureWaitOnBreakpoint = dump stack to check the goroutine state

EnsureGoroutineFinished = dump stack to check the goroutine state

Sandbox is used to manage the whole thing, it bind each goroutine with context, each goroutine can get its routine information like the TLS(thread local storage) ...

[ti-chi-bot]

@lysu: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.