-
Notifications
You must be signed in to change notification settings - Fork 1.6k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
skaffold trace wrapping of critical functions and skaffold trace expo…
…rters What is the problem being solved? Part of #5756, adding opentelemetry trace information to skaffold commands. Added trace information to specific performance critical skaffold functions (identified in go/cloud-trace-skaffold). Also added 4 trace exporters - gcp-skaffold, gcp-adc, stdout, and jaeger. This PR uses env var based enabling/disabling for the trace for simplicity and to hide it from users directly. Why is this the best approach? Using opentelemetry tracing is the obvious choice as we use open telemetry libs for metrics and it is becoming the metrics/tracing standard. Using an env var in this PR and later integrating the flag setup was considered optimal as currently skaffold tracing will be used for benchmarking/bottleneck-identifying for select use cases while the user facing UX w/ jaeger, etc. is still being worked out. What other approaches did you consider? There was the possibility of building tracing directly into skaffold events but I think with the current wrapper setup in pkg/skaffold/instrumentation/trace.go (w/ the minimal code required) and the fact that many trace locations will not be event locations (eg: how long to hash a file, etc.) it makes sense to not integrate them. What side effects will this approach have? There shouldn't be any side effects w/ this approach as the default "off" for tracing and the minimal user visibility for now should mean that it used only for select use cases experimentally. I have done timing tests with the no-op/empty trace (SKAFFOLD_TRACE unset) and it does not change the performance of skaffold. What future work remains to be done? Future work includes wiring up a --trace flag through dev, build, deploy, etc. and working on how skaffold might be able to do distributed tracing w/ other tools (minikube, buildpacks, etc.). Additionally the ability to allow for more sporadic sampling (vs AlwaysSample) should be added.
- Loading branch information
1 parent
a26b947
commit 5893976
Showing
29 changed files
with
951 additions
and
36 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| ### Example: Skaffold Command Tracing w/ Jaeger | ||
|
|
||
| In this example: | ||
|
|
||
| * Use skaffold to deploy a jaeger local/remote jaeger instance | ||
| * Enable skaffold tracing functionality to get trace information about skaffol dev, build, deploy, etc. timings | ||
| * Send skaffold trace information to jaeger and have that information be visible in the jaeger UI, | ||
|
|
||
| In this example, we'll walk through enabling skaffold trace information that can be used to explore performance bottlenecks, etc. in skaffold and to get a more in depth view of user's local dev loop | ||
|
|
||
| **WARNING: If you're running this on a cloud cluster, this example will create a service and expose a webserver. | ||
| It's highly suggested that you only run this example on a local, private cluster like minikube or Kubernetes in Docker for Desktop.** | ||
|
|
||
| #### Running the example on minikube | ||
|
|
||
| Start minikube (or verify that minikube is up and running locally) | ||
| ```bash | ||
| minikube start | ||
| ``` | ||
|
|
||
| From this directory, run | ||
| ```bash | ||
| skaffold deploy --port-forward | ||
| ``` | ||
|
|
||
| Now, in a different terminal, go to another skaffold example (eg: microservices), enable SKAFFOLD_TRACE w/ jaeger and start dev session there: | ||
| ```bash | ||
| cd ../microservices | ||
| export SKAFFOLD_TRACE=jaeger | ||
| skaffold dev | ||
| ``` | ||
|
|
||
| Now go to the jaeger UI that skaffold will port-forward to localhost at http://127.0.0.1:16686/ | ||
|
|
||
| Select service:`skaffold-trace` in the left bar and then click `Find Traces` on the bottom of the left bar. | ||
|
|
||
| From here you should be able to view all of the relevant traces |
161 changes: 161 additions & 0 deletions
161
examples/jaeger-skaffold-trace/jaeger-all-in-one-template.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,161 @@ | ||
| # Copyright 2019 The Skaffold Authors | ||
|
|
||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
|
|
||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
|
|
||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| apiVersion: v1 | ||
| kind: List | ||
| items: | ||
| - apiVersion: apps/v1 | ||
| kind: Deployment | ||
| metadata: | ||
| name: jaeger | ||
| labels: | ||
| app: jaeger | ||
| app.kubernetes.io/name: jaeger | ||
| app.kubernetes.io/component: all-in-one | ||
| spec: | ||
| replicas: 1 | ||
| strategy: | ||
| type: Recreate | ||
| selector: | ||
| matchLabels: | ||
| app: jaeger | ||
| app.kubernetes.io/name: jaeger | ||
| app.kubernetes.io/component: all-in-one | ||
| template: | ||
| metadata: | ||
| labels: | ||
| app: jaeger | ||
| app.kubernetes.io/name: jaeger | ||
| app.kubernetes.io/component: all-in-one | ||
| annotations: | ||
| prometheus.io/scrape: "true" | ||
| prometheus.io/port: "16686" | ||
| spec: | ||
| containers: | ||
| - env: | ||
| - name: COLLECTOR_ZIPKIN_HTTP_PORT | ||
| value: "9411" | ||
| image: jaegertracing/all-in-one | ||
| name: jaeger | ||
| ports: | ||
| - containerPort: 5775 | ||
| protocol: UDP | ||
| - containerPort: 6831 | ||
| protocol: UDP | ||
| - containerPort: 6832 | ||
| protocol: UDP | ||
| - containerPort: 5778 | ||
| protocol: TCP | ||
| - containerPort: 16686 | ||
| protocol: TCP | ||
| - containerPort: 9411 | ||
| protocol: TCP | ||
| readinessProbe: | ||
| httpGet: | ||
| path: "/" | ||
| port: 14269 | ||
| initialDelaySeconds: 5 | ||
| - apiVersion: v1 | ||
| kind: Service | ||
| metadata: | ||
| name: jaeger-query | ||
| labels: | ||
| app: jaeger | ||
| app.kubernetes.io/name: jaeger | ||
| app.kubernetes.io/component: query | ||
| spec: | ||
| ports: | ||
| - name: query-http | ||
| port: 16686 | ||
| protocol: TCP | ||
| targetPort: 16686 | ||
| selector: | ||
| app.kubernetes.io/name: jaeger | ||
| app.kubernetes.io/component: all-in-one | ||
| type: LoadBalancer | ||
| - apiVersion: v1 | ||
| kind: Service | ||
| metadata: | ||
| name: jaeger-collector | ||
| labels: | ||
| app: jaeger | ||
| app.kubernetes.io/name: jaeger | ||
| app.kubernetes.io/component: collector | ||
| spec: | ||
| ports: | ||
| - name: jaeger-collector-tchannel | ||
| port: 14267 | ||
| protocol: TCP | ||
| targetPort: 14267 | ||
| - name: jaeger-collector-http | ||
| port: 14268 | ||
| protocol: TCP | ||
| targetPort: 14268 | ||
| - name: jaeger-collector-zipkin | ||
| port: 9411 | ||
| protocol: TCP | ||
| targetPort: 9411 | ||
| selector: | ||
| app.kubernetes.io/name: jaeger | ||
| app.kubernetes.io/component: all-in-one | ||
| type: ClusterIP | ||
| - apiVersion: v1 | ||
| kind: Service | ||
| metadata: | ||
| name: jaeger-agent | ||
| labels: | ||
| app: jaeger | ||
| app.kubernetes.io/name: jaeger | ||
| app.kubernetes.io/component: agent | ||
| spec: | ||
| ports: | ||
| - name: agent-zipkin-thrift | ||
| port: 5775 | ||
| protocol: UDP | ||
| targetPort: 5775 | ||
| - name: agent-compact | ||
| port: 6831 | ||
| protocol: UDP | ||
| targetPort: 6831 | ||
| - name: agent-binary | ||
| port: 6832 | ||
| protocol: UDP | ||
| targetPort: 6832 | ||
| - name: agent-configs | ||
| port: 5778 | ||
| protocol: TCP | ||
| targetPort: 5778 | ||
| clusterIP: None | ||
| selector: | ||
| app.kubernetes.io/name: jaeger | ||
| app.kubernetes.io/component: all-in-one | ||
| - apiVersion: v1 | ||
| kind: Service | ||
| metadata: | ||
| name: zipkin | ||
| labels: | ||
| app: jaeger | ||
| app.kubernetes.io/name: jaeger | ||
| app.kubernetes.io/component: zipkin | ||
| spec: | ||
| ports: | ||
| - name: jaeger-collector-zipkin | ||
| port: 9411 | ||
| protocol: TCP | ||
| targetPort: 9411 | ||
| clusterIP: None | ||
| selector: | ||
| app.kubernetes.io/name: jaeger | ||
| app.kubernetes.io/component: all-in-one | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| apiVersion: skaffold/v2beta16 | ||
| kind: Config | ||
| metadata: | ||
| name: jaeger-all-in-one | ||
| deploy: | ||
| kubectl: | ||
| manifests: | ||
| - jaeger-all-in-one-template.yaml |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| ### Example: Skaffold Command Tracing w/ Jaeger | ||
|
|
||
| In this example: | ||
|
|
||
| * Use skaffold to deploy a jaeger local/remote jaeger instance | ||
| * Enable skaffold tracing functionality to get trace information about skaffol dev, build, deploy, etc. timings | ||
| * Send skaffold trace information to jaeger and have that information be visible in the jaeger UI, | ||
|
|
||
| In this example, we'll walk through enabling skaffold trace information that can be used to explore performance bottlenecks, etc. in skaffold and to get a more in depth view of user's local dev loop | ||
|
|
||
| **WARNING: If you're running this on a cloud cluster, this example will create a service and expose a webserver. | ||
| It's highly suggested that you only run this example on a local, private cluster like minikube or Kubernetes in Docker for Desktop.** | ||
|
|
||
| #### Running the example on minikube | ||
|
|
||
| Start minikube (or verify that minikube is up and running locally) | ||
| ```bash | ||
| minikube start | ||
| ``` | ||
|
|
||
| From this directory, run | ||
| ```bash | ||
| skaffold deploy --port-forward | ||
| ``` | ||
|
|
||
| Now, in a different terminal, go to another skaffold example (eg: microservices), enable SKAFFOLD_TRACE w/ jaeger and start dev session there: | ||
| ```bash | ||
| cd ../microservices | ||
| export SKAFFOLD_TRACE=jaeger | ||
| skaffold dev | ||
| ``` | ||
|
|
||
| Now go to the jaeger UI that skaffold will port-forward to localhost at http://127.0.0.1:16686/ | ||
|
|
||
| Select service:`skaffold-trace` in the left bar and then click `Find Traces` on the bottom of the left bar. | ||
|
|
||
| From here you should be able to view all of the relevant traces |
Oops, something went wrong.