Skip to content

Commit

Permalink
# This is a combination of 7 commits.
Browse files Browse the repository at this point in the history
# This is the 1st commit message:

chore: Bump ws from 7.5.6 to 7.5.10 in /ui (feast-dev#4292)


# This is the commit message feast-dev#2:

Remove metric checks

Signed-off-by: Twinkll Sisodia <[email protected]>

# This is the commit message feast-dev#3:

Draft different md file

# This is the commit message feast-dev#4:

Refine opentelemetry.md

# This is the commit message feast-dev#5:

Refine opentelemetry.md

# This is the commit message feast-dev#6:

Refine opentelemetry.md

# This is the commit message feast-dev#7:

Refine opentelemetry.md
  • Loading branch information
dependabot[bot] authored and tsisodia10 committed Jul 23, 2024
1 parent 5744d3e commit f036670
Show file tree
Hide file tree
Showing 9 changed files with 113 additions and 12 deletions.
107 changes: 107 additions & 0 deletions infra/charts/feast-feature-server/opentelemetry.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
## Adding Monitoring
To add monitoring to the Feast Feature Server, follow these steps:

### Workflow

Feast instrumentation Using OpenTelemetry and Prometheus -
![Workflow](samples/workflow.png)

### Deploy Prometheus Operator
Navigate to OperatorHub and install the stable version of the Prometheus Operator

### Deploy OpenTelemetry Operator
Before installing OTEL Operator, install `cert-manager` and validate the `pods` should spin up --
```
kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml
```

Then navigate to OperatorHub and install the stable version of the community OpenTelemetry Operator


### Configure OpenTelemetry Collector
Add the OpenTelemetry Collector configuration under the metrics section in your values.yaml file.

Example values.yaml:

```
metrics:
enabled: true
otelCollector:
endpoint: "otel-collector.default.svc.cluster.local:4317" #sample
headers:
api-key: "your-api-key"
```

### Add instrumentation annotation and environment variables in the deployment.yaml

```
template:
metadata:
{{- with .Values.podAnnotations }}
annotations:
{{- toYaml . | nindent 8 }}
instrumentation.opentelemetry.io/inject-python: "true"
```

```
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: http://{{ .Values.service.name }}-collector.{{ .Release.namespace }}.svc.cluster.local:{{ .Values.metrics.endpoint.port}}
- name: OTEL_EXPORTER_OTLP_INSECURE
value: "true"
```

### Add checks
Add metric checks to all manifests and deployment file -

```
{{ if .Values.metrics.enabled }}
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: feast-instrumentation
spec:
exporter:
endpoint: http://{{ .Values.service.name }}-collector.{{ .Release.Namespace }}.svc.cluster.local:4318 # This is the default port for the OpenTelemetry Collector
env:
propagators:
- tracecontext
- baggage
python:
env:
- name: OTEL_METRICS_EXPORTER
value: console,otlp_proto_http
- name: OTEL_LOGS_EXPORTER
value: otlp_proto_http
- name: OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED
value: "true"
{{end}}
```

### Add manifests to the chart
Add Instrumentation, OpenTelemetryCollector, ServiceMonitors, Prometheus Instance and RBAC rules as provided in the samples/ directory.

For latest updates please refer the official repository - https://github.com/open-telemetry/opentelemetry-operator

### Deploy Feast
Deploy Feast and set `metrics` value to `true`.

Example -
```
helm install feast-release infra/charts/feast-feature-server --set metric=true --set feature_store_yaml_base64=""
```

## See logs
Once the opentelemetry is deployed, you can search the logs to see the required metrics -

```
oc logs otelcol-collector-0 | grep "Name: feast_feature_server_memory_usage\|Value: 0.*"
oc logs otelcol-collector-0 | grep "Name: feast_feature_server_cpu_usage\|Value: 0.*"
```
```
-> Name: feast_feature_server_memory_usage
Value: 0.579426
```
```
-> Name: feast_feature_server_cpu_usage
Value: 0.000000
```
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
{{- if .Values.metrics }}
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
# API reference https://github.com/open-telemetry/opentelemetry-operator/blob/main/docs/api.md
# Refs for v1beta1 config: https://github.com/open-telemetry/opentelemetry-operator/issues/3011#issuecomment-2154118998
{{- if .Values.metrics }}
apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
Expand Down
1 change: 0 additions & 1 deletion infra/charts/feast-feature-server/samples/otel-sm.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
{{- if .Values.metrics }}
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
Expand Down
1 change: 0 additions & 1 deletion infra/charts/feast-feature-server/samples/prometheus.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
{{- if .Values.metrics }}
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
Expand Down
1 change: 0 additions & 1 deletion infra/charts/feast-feature-server/samples/rbac.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
{{- if .Values.metrics }}
apiVersion: v1
kind: ServiceAccount
metadata:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
{{- if .Values.metrics }}
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
12 changes: 6 additions & 6 deletions ui/yarn.lock
Original file line number Diff line number Diff line change
Expand Up @@ -11640,14 +11640,14 @@ write-file-atomic@^3.0.0:
typedarray-to-buffer "^3.1.5"

ws@^7.4.6:
version "7.5.6"
resolved "https://registry.yarnpkg.com/ws/-/ws-7.5.6.tgz#e59fc509fb15ddfb65487ee9765c5a51dec5fe7b"
integrity sha512-6GLgCqo2cy2A2rjCNFlxQS6ZljG/coZfZXclldI8FB/1G3CCI36Zd8xy2HrFVACi8tfk5XrgLQEk+P0Tnz9UcA==
version "7.5.10"
resolved "https://registry.yarnpkg.com/ws/-/ws-7.5.10.tgz#58b5c20dc281633f6c19113f39b349bd8bd558d9"
integrity sha512-+dbF1tHwZpXcbOJdVOkzLDxZP1ailvSxM6ZweXTegylPny803bFhA+vqBYw4s31NSAk4S2Qz+AKXK9a4wkdjcQ==

ws@^8.1.0:
version "8.4.2"
resolved "https://registry.yarnpkg.com/ws/-/ws-8.4.2.tgz#18e749868d8439f2268368829042894b6907aa0b"
integrity sha512-Kbk4Nxyq7/ZWqr/tarI9yIt/+iNNFOjBXEWgTb4ydaNHBNGgvf2QHbS9fdfsndfjFlFwEd4Al+mw83YkaD10ZA==
version "8.17.1"
resolved "https://registry.yarnpkg.com/ws/-/ws-8.17.1.tgz#9293da530bb548febc95371d90f9c878727d919b"
integrity sha512-6XQFvXTkbfUOZOKKILFG1PDK2NDQs4azKQl26T0YS5CxqWLgXajbPZ+h4gZekJyRqFU8pvnbAbbs/3TgRPy+GQ==

xml-name-validator@^3.0.0:
version "3.0.0"
Expand Down

0 comments on commit f036670

Please sign in to comment.