Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding image for debugging and traffic shaping #1394

Merged
merged 2 commits into from
Mar 23, 2021

Conversation

vitarb
Copy link
Contributor

@vitarb vitarb commented Mar 23, 2021

Description, Motivation and Context

Finding flaky tests that rely on faulty timing assumptions and fast server performance is tricky. Doing it efficiently requires adding artificial delay for server calls. This PR provides ability to start a version of temporal that induces delay for all network calls by introducing TEMPORAL_DBG_NETWORK_DELAY_MS variable. Treat this as a starting point, we can iterate on top and add a lot more knobs in the future, for adding packet loss, random lags and database delays and some useful debug tools. For now this is targeted primarily for finding flaky tests in our SDKs.
This PR is a refinement of a previous change docker-compose.

Details

To achieve this goal I've used a tool called tc, which is a part of iproute2 package. I didn't want to add iproute2 into our main docker image, since it's needed only for testing, that's why I've created a separate testing/Dockerfile that builds temporal image with any test packages needed for debugging and resiliancy testing, also this image runs a configuration script, before starting the server, which actually invokes tc tool.

Running

docker run -e TEMPORAL_DBG_NETWORK_DELAY_MS=50 --cap-add=NET_ADMIN <container-id>

Dockerfile Outdated Show resolved Hide resolved
@vitarb vitarb merged commit 698d8a7 into temporalio:master Mar 23, 2021
@vitarb vitarb deleted the debug-image branch March 23, 2021 06:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants