Skip to content
This repository has been archived by the owner on Aug 17, 2020. It is now read-only.

Subtests autoinstrumentation #123

Draft
wants to merge 9 commits into
base: master
Choose a base branch
from

Conversation

tonyredondo
Copy link
Contributor

This PR adds auto instrumentation to sub tests and sub benchmarks.

The problem on auto instrumenting sub tests and sub benchmarks is that the call to t.Run and b.Run are recursive from the initial test execution to other sub tests.

An initial try to solve the problem could be the following implementation:

runPatch, err = mpatch.PatchMethodByReflect(tRunMethod, func(t *testing.T, name string, f func(t *testing.T)) bool {
	logOnError(runPatch.Unpatch())
	defer func() { logOnError(runPatch.Patch()) }()
	pc, _, _, _ := runtime.Caller(1)
	return t.Run(name, func(childT *testing.T) {
		logOnError(runPatch.Patch())
		defer func() { logOnError(runPatch.Unpatch()) }()
		addAutoInstrumentedTest(childT)
		childTest := StartTestFromCaller(childT, pc)
		defer childTest.end()
		f(childT)
	})
})

This uses Patch and Unpatch twice to use the original t.Run implementation. The problem with this algorithm is that is not valid on Parallel scenarios, because we don't know the execution order, If we try to use a mutex we end with a deadlock, because each func executed by t.Run runs in a separated goroutine.

The way we solve to solve this issue, is by never Unpatch the patched function, but calling the original implementation in another way.

In this case we copy the implementation of both t.Run and b.Run from the original golang source code, to do that we are forced to create the same testing.T and testing.B structure and use unsafe to have access to the private fields and use some linking to private methods. Note with this implementation we don't need any lock.

With this, we add auto instrumentation support for the following cases:

Tests

func TestWithSubTests(t *testing.T) {
	for i := 0; i < 20; i++ {
		t.Run(fmt.Sprintf("Child:%d", i+1), func(t *testing.T) {
			t.Parallel()

			for i := 0; i < 20; i++ {
				t.Run(fmt.Sprintf("SubChild:%d", i+1), func(t *testing.T) {
					t.Parallel()
				})
			}
		})
	}
}

image

Benchmarks

// Auto instrumented benchmark and sub benchmarks
func Benchmark05(b *testing.B) {
	// Auto instrumented sub benchmarks
	b.Run("Sub01", func(b *testing.B) {
		for i := 0; i < b.N; i++ {
			math.Log(float64(b.N))
		}
	})

	b.Run("Sub02", func(b *testing.B) {
		for i := 0; i < b.N; i++ {
			math.Log(float64(b.N))
		}
	})

	b.Run("Sub03", bench)
	b.Run("Sub04", bench)
}

func bench(b *testing.B) {
	// Auto instrumented sub benchmarks
	b.Run("Sub01", func(b *testing.B) {
		for i := 0; i < b.N; i++ {
			math.Log(float64(b.N))
		}
	})
	b.Run("Sub02", func(b *testing.B) {
		for i := 0; i < b.N; i++ {
			math.Log(float64(b.N))
		}
	})
}

image

@tonyredondo tonyredondo self-assigned this Jan 27, 2020
@tonyredondo tonyredondo force-pushed the subtests-autoinstrumentation branch 2 times, most recently from 0f7eab7 to 673e9ff Compare February 12, 2020 14:25
@tonyredondo tonyredondo force-pushed the subtests-autoinstrumentation branch 2 times, most recently from bedb332 to 69bc25f Compare February 21, 2020 14:30
@tonyredondo tonyredondo removed the WIP label Feb 21, 2020
@tonyredondo tonyredondo force-pushed the subtests-autoinstrumentation branch 3 times, most recently from cc123a3 to 6bcf27a Compare March 16, 2020 17:19
@tonyredondo tonyredondo force-pushed the subtests-autoinstrumentation branch 2 times, most recently from c2db8e9 to 5a1148d Compare March 18, 2020 16:46
@tonyredondo tonyredondo marked this pull request as draft June 10, 2020 10:52
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant