Find a threadsafe way to log in timeout_handler

[bobcatfish]

Expected Behavior

We should be able to emit logs from the timeout_handler in a threadsafe manner. The logic in the timeout_handler is intended to be running in separate goroutines alongside the main reconcile loop execution, to manage actions like timing out and retrying.

Actual Behavior

We were using the same zap sugared logger that we were using everywhere else for logging, e.g.:

pipeline/pkg/reconciler/timeout_handler.go

Lines 276 to 290 in 5e3276d

	func (t *TimeoutSet) setTimer(runObj StatusKey, timeout time.Duration, callback func(interface{})) {
	finished := t.getOrCreateFinishedChan(runObj)
	started := time.Now()
	select {
	case <-t.stopCh:
	t.logger.Infof("stopping timer for %q", runObj.GetRunKey())
	return
	case <-finished:
	t.logger.Infof("%q finished, stopping timer", runObj.GetRunKey())
	return
	case <-time.After(timeout):
	t.logger.Infof("timer for %q has activated after %s", runObj.GetRunKey(), time.Since(started).String())
	callback(runObj)
	}
	}

However when we started running tests with newer versions of go (e.g. 1.13 v 1.11 see #1124 ) and the -race flag, we found there were data races.

Not 100% sure what the root cause is but uber-go/zap#99 suggests that zap logger writes are not protected and could have a race.

So in #1124 we're removing the logging, which is unfortunate b/c we used it previously to help us debug #731 and itd be great to have some logging for debugging any future issues.

Additional Info

(Is "threadsafe" the right word to use when we're talking about goroutines??)