Adding image for debugging and traffic shaping #1394
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description, Motivation and Context
Finding flaky tests that rely on faulty timing assumptions and fast server performance is tricky. Doing it efficiently requires adding artificial delay for server calls. This PR provides ability to start a version of temporal that induces delay for all network calls by introducing
TEMPORAL_DBG_NETWORK_DELAY_MS
variable. Treat this as a starting point, we can iterate on top and add a lot more knobs in the future, for adding packet loss, random lags and database delays and some useful debug tools. For now this is targeted primarily for finding flaky tests in our SDKs.This PR is a refinement of a previous change docker-compose.
Details
To achieve this goal I've used a tool called
tc
, which is a part ofiproute2
package. I didn't want to addiproute2
into our main docker image, since it's needed only for testing, that's why I've created a separatetesting/Dockerfile
that builds temporal image with any test packages needed for debugging and resiliancy testing, also this image runs a configuration script, before starting the server, which actually invokes tc tool.Running