forked from feast-dev/feast
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
# This is a combination of 7 commits.
# This is the 1st commit message: chore: Bump ws from 7.5.6 to 7.5.10 in /ui (feast-dev#4292) # This is the commit message feast-dev#2: Remove metric checks Signed-off-by: Twinkll Sisodia <[email protected]> # This is the commit message feast-dev#3: Draft different md file # This is the commit message feast-dev#4: Refine opentelemetry.md # This is the commit message feast-dev#5: Refine opentelemetry.md # This is the commit message feast-dev#6: Refine opentelemetry.md # This is the commit message feast-dev#7: Refine opentelemetry.md
- Loading branch information
1 parent
5744d3e
commit f036670
Showing
9 changed files
with
113 additions
and
12 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,107 @@ | ||
## Adding Monitoring | ||
To add monitoring to the Feast Feature Server, follow these steps: | ||
|
||
### Workflow | ||
|
||
Feast instrumentation Using OpenTelemetry and Prometheus - | ||
![Workflow](samples/workflow.png) | ||
|
||
### Deploy Prometheus Operator | ||
Navigate to OperatorHub and install the stable version of the Prometheus Operator | ||
|
||
### Deploy OpenTelemetry Operator | ||
Before installing OTEL Operator, install `cert-manager` and validate the `pods` should spin up -- | ||
``` | ||
kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml | ||
``` | ||
|
||
Then navigate to OperatorHub and install the stable version of the community OpenTelemetry Operator | ||
|
||
|
||
### Configure OpenTelemetry Collector | ||
Add the OpenTelemetry Collector configuration under the metrics section in your values.yaml file. | ||
|
||
Example values.yaml: | ||
|
||
``` | ||
metrics: | ||
enabled: true | ||
otelCollector: | ||
endpoint: "otel-collector.default.svc.cluster.local:4317" #sample | ||
headers: | ||
api-key: "your-api-key" | ||
``` | ||
|
||
### Add instrumentation annotation and environment variables in the deployment.yaml | ||
|
||
``` | ||
template: | ||
metadata: | ||
{{- with .Values.podAnnotations }} | ||
annotations: | ||
{{- toYaml . | nindent 8 }} | ||
instrumentation.opentelemetry.io/inject-python: "true" | ||
``` | ||
|
||
``` | ||
- name: OTEL_EXPORTER_OTLP_ENDPOINT | ||
value: http://{{ .Values.service.name }}-collector.{{ .Release.namespace }}.svc.cluster.local:{{ .Values.metrics.endpoint.port}} | ||
- name: OTEL_EXPORTER_OTLP_INSECURE | ||
value: "true" | ||
``` | ||
|
||
### Add checks | ||
Add metric checks to all manifests and deployment file - | ||
|
||
``` | ||
{{ if .Values.metrics.enabled }} | ||
apiVersion: opentelemetry.io/v1alpha1 | ||
kind: Instrumentation | ||
metadata: | ||
name: feast-instrumentation | ||
spec: | ||
exporter: | ||
endpoint: http://{{ .Values.service.name }}-collector.{{ .Release.Namespace }}.svc.cluster.local:4318 # This is the default port for the OpenTelemetry Collector | ||
env: | ||
propagators: | ||
- tracecontext | ||
- baggage | ||
python: | ||
env: | ||
- name: OTEL_METRICS_EXPORTER | ||
value: console,otlp_proto_http | ||
- name: OTEL_LOGS_EXPORTER | ||
value: otlp_proto_http | ||
- name: OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED | ||
value: "true" | ||
{{end}} | ||
``` | ||
|
||
### Add manifests to the chart | ||
Add Instrumentation, OpenTelemetryCollector, ServiceMonitors, Prometheus Instance and RBAC rules as provided in the samples/ directory. | ||
|
||
For latest updates please refer the official repository - https://github.com/open-telemetry/opentelemetry-operator | ||
|
||
### Deploy Feast | ||
Deploy Feast and set `metrics` value to `true`. | ||
|
||
Example - | ||
``` | ||
helm install feast-release infra/charts/feast-feature-server --set metric=true --set feature_store_yaml_base64="" | ||
``` | ||
|
||
## See logs | ||
Once the opentelemetry is deployed, you can search the logs to see the required metrics - | ||
|
||
``` | ||
oc logs otelcol-collector-0 | grep "Name: feast_feature_server_memory_usage\|Value: 0.*" | ||
oc logs otelcol-collector-0 | grep "Name: feast_feature_server_cpu_usage\|Value: 0.*" | ||
``` | ||
``` | ||
-> Name: feast_feature_server_memory_usage | ||
Value: 0.579426 | ||
``` | ||
``` | ||
-> Name: feast_feature_server_cpu_usage | ||
Value: 0.000000 | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1,3 @@ | ||
{{- if .Values.metrics }} | ||
apiVersion: opentelemetry.io/v1alpha1 | ||
kind: Instrumentation | ||
metadata: | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1,3 @@ | ||
{{- if .Values.metrics }} | ||
apiVersion: monitoring.coreos.com/v1 | ||
kind: ServiceMonitor | ||
metadata: | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1,3 @@ | ||
{{- if .Values.metrics }} | ||
apiVersion: monitoring.coreos.com/v1 | ||
kind: Prometheus | ||
metadata: | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1,3 @@ | ||
{{- if .Values.metrics }} | ||
apiVersion: v1 | ||
kind: ServiceAccount | ||
metadata: | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1,3 @@ | ||
{{- if .Values.metrics }} | ||
apiVersion: monitoring.coreos.com/v1 | ||
kind: ServiceMonitor | ||
metadata: | ||
|
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters