-
Notifications
You must be signed in to change notification settings - Fork 1.6k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
skaffold trace wrapping of critical functions and skaffold trace expo…
…rters (#5854) What is the problem being solved? Part of #5756, adding opentelemetry trace information to skaffold commands. Added trace information to specific performance critical skaffold functions (identified in go/cloud-trace-skaffold). Also added 4 trace exporters - gcp-skaffold, gcp-adc, stdout, and jaeger. This PR uses env var based enabling/disabling for the trace for simplicity and to hide it from users directly. Why is this the best approach? Using opentelemetry tracing is the obvious choice as we use open telemetry libs for metrics and it is becoming the metrics/tracing standard. Using an env var in this PR and later integrating the flag setup was considered optimal as currently skaffold tracing will be used for benchmarking/bottleneck-identifying for select use cases while the user facing UX w/ jaeger, etc. is still being worked out. What other approaches did you consider? There was the possibility of building tracing directly into skaffold events but I think with the current wrapper setup in pkg/skaffold/instrumentation/trace.go (w/ the minimal code required) and the fact that many trace locations will not be event locations (eg: how long to hash a file, etc.) it makes sense to not integrate them. What side effects will this approach have? There shouldn't be any side effects w/ this approach as the default "off" for tracing and the minimal user visibility for now should mean that it used only for select use cases experimentally. I have done timing tests with the no-op/empty trace (SKAFFOLD_TRACE unset) and it does not change the performance of skaffold. What future work remains to be done? Future work includes wiring up a --trace flag through dev, build, deploy, etc. and working on how skaffold might be able to do distributed tracing w/ other tools (minikube, buildpacks, etc.). Additionally the ability to allow for more sporadic sampling (vs AlwaysSample) should be added. Some future work mentioned in PR review included: - OTEL_TRACES_EXPORTER=* support (vs SKAFFOLD_TRACE)
- Loading branch information
1 parent
53b0780
commit f95999f
Showing
34 changed files
with
888 additions
and
42 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| ### Example: Skaffold Command Tracing with Jaeger | ||
|
|
||
|
|
||
| _**WARNING: Skaffold's trace functionality is experimental and may change without notice.**_ | ||
|
|
||
| In this example: | ||
|
|
||
| * Use Skaffold to deploy a local/remote Jaeger instance | ||
| * Enable Skaffold tracing functionality to get trace information about skaffold `dev`, `build`, `deploy`, etc. timings | ||
| * Send Skaffold trace information to Jaeger and have that information be visible in the Jaeger UI, | ||
|
|
||
| In this example, we'll walk through enabling Skaffold trace information that can be used to explore performance bottlenecks and to get a more in depth view of user's local dev loop. | ||
|
|
||
| _**WARNING: If you're running this on a cloud cluster, this example will create a service and expose a webserver. | ||
| It's highly recommended that you only run this example on a local, private cluster like minikube or Kubernetes in Docker for Desktop.**_ | ||
|
|
||
| #### Setting up Jaeger locally | ||
|
|
||
| Use docker to start a local jaeger instance using the Jaeger project's [all-in-one docker setup](https://www.jaegertracing.io/docs/getting-started/#all-in-one): | ||
| ```bash | ||
| docker run -d --name jaeger \ | ||
| -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \ | ||
| -p 5775:5775/udp \ | ||
| -p 6831:6831/udp \ | ||
| -p 6832:6832/udp \ | ||
| -p 5778:5778 \ | ||
| -p 16686:16686 \ | ||
| -p 14268:14268 \ | ||
| -p 14250:14250 \ | ||
| -p 9411:9411 \ | ||
| jaegertracing/all-in-one:1.22 | ||
| ``` | ||
|
|
||
| Now, in a different terminal, go to another Skaffold example (eg: microservices), enable SKAFFOLD_TRACE with Jaeger and start dev session there: | ||
| ```bash | ||
| cd ../microservices | ||
| export SKAFFOLD_TRACE=jaeger | ||
| skaffold dev | ||
| ``` | ||
|
|
||
| Now go to the Jaeger UI that Skaffold will port-forward to localhost at http://127.0.0.1:16686/ | ||
|
|
||
| Select service:`skaffold-trace` in the left bar and then click `Find Traces` on the bottom of the left bar. | ||
|
|
||
| From here you should be able to view all of the relevant traces | ||
|
|
||
| #### Cleaning up | ||
| To cleanup Jaeger all-in-one setup, run the following: | ||
| ``` | ||
| docker kill jaeger # stops the running jaeger container | ||
| docker rm jaeger #removes the container image | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| ### Example: Skaffold Command Tracing with Jaeger | ||
|
|
||
|
|
||
| _**WARNING: Skaffold's trace functionality is experimental and may change without notice.**_ | ||
|
|
||
| In this example: | ||
|
|
||
| * Use Skaffold to deploy a local/remote Jaeger instance | ||
| * Enable Skaffold tracing functionality to get trace information about skaffold `dev`, `build`, `deploy`, etc. timings | ||
| * Send Skaffold trace information to Jaeger and have that information be visible in the Jaeger UI, | ||
|
|
||
| In this example, we'll walk through enabling Skaffold trace information that can be used to explore performance bottlenecks and to get a more in depth view of user's local dev loop. | ||
|
|
||
| _**WARNING: If you're running this on a cloud cluster, this example will create a service and expose a webserver. | ||
| It's highly recommended that you only run this example on a local, private cluster like minikube or Kubernetes in Docker for Desktop.**_ | ||
|
|
||
| #### Setting up Jaeger locally | ||
|
|
||
| Use docker to start a local jaeger instance using the Jaeger project's [all-in-one docker setup](https://www.jaegertracing.io/docs/getting-started/#all-in-one): | ||
| ```bash | ||
| docker run -d --name jaeger \ | ||
| -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \ | ||
| -p 5775:5775/udp \ | ||
| -p 6831:6831/udp \ | ||
| -p 6832:6832/udp \ | ||
| -p 5778:5778 \ | ||
| -p 16686:16686 \ | ||
| -p 14268:14268 \ | ||
| -p 14250:14250 \ | ||
| -p 9411:9411 \ | ||
| jaegertracing/all-in-one:1.22 | ||
| ``` | ||
|
|
||
| Now, in a different terminal, go to another Skaffold example (eg: microservices), enable SKAFFOLD_TRACE with Jaeger and start dev session there: | ||
| ```bash | ||
| cd ../microservices | ||
| export SKAFFOLD_TRACE=jaeger | ||
| skaffold dev | ||
| ``` | ||
|
|
||
| Now go to the Jaeger UI that Skaffold will port-forward to localhost at http://127.0.0.1:16686/ | ||
|
|
||
| Select service:`skaffold-trace` in the left bar and then click `Find Traces` on the bottom of the left bar. | ||
|
|
||
| From here you should be able to view all of the relevant traces | ||
|
|
||
| #### Cleaning up | ||
| To cleanup Jaeger all-in-one setup, run the following: | ||
| ``` | ||
| docker kill jaeger # stops the running jaeger container | ||
| docker rm jaeger #removes the container image | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.