Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Documentation] add more details about prometheus #427

Merged
merged 1 commit into from
Jan 31, 2024
Merged

[Documentation] add more details about prometheus #427

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
82 changes: 66 additions & 16 deletions doc/integration/integration-prometheus-grafana.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,78 +1,116 @@
# StarRocks Cluster Integration With Prometheus and Grafana Service

This document describes how to integrate StarRocks cluster with Prometheus and Grafana service in a kubernetes environment. From this document, you will learn,
This document describes how to integrate StarRocks cluster with Prometheus and Grafana service in a kubernetes
environment. From this document, you will learn,

* How to turn on prometheus metrics scrape for the StarRocks cluster
* How to import StarRocks Grafana dashboard


## 1. Prerequisites

+ Kubernetes: v1.18+ - v1.26
+ Prometheus helm chart: latest available version
+ Grafana helm chart: latest available version
+ StarRocks operator and helm chart: v1.7.1+

## 2. Deploy Prometheus Service
There are two ways to deploy Prometheus and Grafana service in the Kubernetes environment.

1. Deploy Prometheus and Grafana Service Directly.
2. Deploy Prometheus and Grafana Service by Operator.

Inorder to scrape the metrics by ServiceMonitor CRD, you must install the Prometheus Operator in the Kubernetes. That is
to say, you must choose the second way to deploy Prometheus and Grafana Service.

## 2. Deploy Prometheus and Grafana Service Directly

### 2.1 Deploy Prometheus Service

> SKIP this step if the Prometheus service is already available in the Kubernetes environment.

Follow the instructions from [Prometheus Helm Chart](https://artifacthub.io/packages/helm/prometheus-community/prometheus) to deploy the Prometheus service into the Kubernetes environment.
Follow the instructions
from [Prometheus Helm Chart](https://artifacthub.io/packages/helm/prometheus-community/prometheus) to deploy the
Prometheus service into the Kubernetes environment.

```shell
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update prometheus-community
helm install prometheus prometheus-community/prometheus
```

## 3. Deploy Grafana Service
### 2.2 Deploy Grafana Service

> SKIP this step if the Grafana service is already available in the Kubernetes environment.

Follow the instructions from [Grafana Helm Chart](https://artifacthub.io/packages/helm/grafana/grafana) to deploy the Grafana service into the Kubernetes environment.
Follow the instructions from [Grafana Helm Chart](https://artifacthub.io/packages/helm/grafana/grafana) to deploy the
Grafana service into the Kubernetes environment.

Need override the default value of the grafana service, change the Grafana service type from default `ClusterIP` to `LoadBalancer` in order to access the Grafana service from the outside Kubernetes network.
Need override the default value of the grafana service, change the Grafana service type from default `ClusterIP`
to `LoadBalancer` in order to access the Grafana service from the outside Kubernetes network.

An example `grafana-values.yaml` may look like as the following snippet.

```yaml
service:
enabled: true
type: "LoadBalancer"
```

Install the Grafana service with the following commands.

```shell
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update grafana
helm install grafana -f grafana-values.yaml grafana/grafana
```

## 3. Deploy Prometheus and Grafana Service by Operator

you can also install `kube-prometheus-stack` chart which will install both Prometheus Operator, Prometheus service
and Grafana service.

```shell
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update prometheus-community
helm install prometheus prometheus-community/kube-prometheus-stack
```

## 4. Login And Configure Grafana Service

> SKIP this section if the prometheus data source is already configured in Grafana.

### 4.1 Get Grafana Service External Address

Run the following command to get the external ip address

```shell
kubectl get svc grafana
```

A possible result may look like the follows

```text
$ kubectl get svc grafana
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
grafana LoadBalancer 10.40.6.237 35.239.53.124 80:31362/TCP 4m25s
```

### 4.2 Get Grafana Admin Password

Run the following command to get the login password for admin user.

```shell
kubectl get secrets grafana -o yaml | grep admin-password | awk '{print $2}' | base64 -d
```

### 4.3 Login Grafana and Configure Prometheus Data Source

Access Grafana web GUI with `http://<external_ip>/`, login with `admin:<password>`.

From the navigation path, `"Administration" -> "Configuration" -> "Add Data Source"`, set the HTTP.URL to `http://prometheus-server.default`. Change `default` to the correct namespace if the prometheus service is not deployed in the default namespace. Refer to Grafana doc [here](https://grafana.com/docs/grafana/latest/datasources/prometheus/configure-prometheus-data-source/) for detailed instructions.
From the navigation path, `"Administration" -> "Configuration" -> "Add Data Source"`, set the HTTP.URL
to `http://prometheus-server.default`. Change `default` to the correct namespace if the prometheus service is not
deployed in the default namespace. Refer to Grafana
doc [here](https://grafana.com/docs/grafana/latest/datasources/prometheus/configure-prometheus-data-source/) for
detailed instructions.

## 5. Deploy StarRocks Cluster

Expand All @@ -83,11 +121,13 @@ There are two ways to turn on the prometheus metrics scrape for the StarRocks cl

### 5.1 Turn on the prometheus metrics scrape by adding annotations

Follow the instructions from [StarRocks Helm Chart](https://artifacthub.io/packages/helm/kube-starrocks/kube-starrocks) with some customized values.
Follow the instructions from [StarRocks Helm Chart](https://artifacthub.io/packages/helm/kube-starrocks/kube-starrocks)
with some customized values.

Following is an example of the content of the `sr-values.yaml`.

* For chart v1.7.1 and below,

```yaml
starrocksFESpec:
service:
Expand Down Expand Up @@ -118,6 +158,7 @@ starrocksBESpec:
```

* For chart v1.8.0 and above,

```yaml
starrocks:
starrocksFESpec:
Expand Down Expand Up @@ -148,10 +189,12 @@ starrocks:
memory: 4Gi
```

Note that `"prometheus.io/*` annotations are the must items to be added, this will allow prometheus to auto discover StarRocks PODs and to collect the metrics.
Note that `"prometheus.io/*` annotations are the must items to be added, this will allow prometheus to auto discover
StarRocks PODs and to collect the metrics.
This method will restart the StarRocks cluster.

An equivalent StarRocks CRD may look like,

```yaml
apiVersion: starrocks.com/v1
kind: StarRocksCluster
Expand Down Expand Up @@ -205,8 +248,12 @@ helm install starrocks -f sr-values.yaml starrocks-community/kube-starrocks

### 5.2 Turn on the prometheus metrics scrape by using ServiceMonitor CRD

Compared to the annotation approach, ServiceMonitor allows for more flexible definition of selector and relabeling rules in the future.
Follow the instructions from [StarRocks Helm Chart](https://artifacthub.io/packages/helm/kube-starrocks/kube-starrocks) with some customized values.
Compared to the annotation approach, ServiceMonitor allows for more flexible definition of selector and relabeling rules
in the future.
> Make sure the prometheus operator is installed in the Kubernetes environment.

Follow the instructions from [StarRocks Helm Chart](https://artifacthub.io/packages/helm/kube-starrocks/kube-starrocks)
with some customized values.

```shell
starrocks:
Expand All @@ -217,14 +264,17 @@ starrocks:

Note: This only works for chart v1.8.4 and above.


## 6. Import StarRocks Grafana Dashboard

StarRocks grafana dashboard configuration for kubernetes environment is available at https://github.com/StarRocks/starrocks/blob/main/extra/grafana/kubernetes/StarRocks-Overview-kubernetes-3.0.json
StarRocks grafana dashboard configuration for kubernetes environment is available
at https://github.com/StarRocks/starrocks/blob/main/extra/grafana/kubernetes/StarRocks-Overview-kubernetes-3.0.json

Detailed instruction can be
found [here](https://grafana.com/docs/grafana/latest/dashboards/manage-dashboards/#import-a-dashboard).

Detailed instruction can be found [here](https://grafana.com/docs/grafana/latest/dashboards/manage-dashboards/#import-a-dashboard).
* Click **Dashboards** in the left-side menu.
* Click New and select **Import** in the dropdown menu.
* Paste dashboard JSON text directly into the text area

The import process enables you to change the name of the dashboard, pick the data source you want the dashboard to use, and specify any metric prefixes (if the dashboard uses any).
The import process enables you to change the name of the dashboard, pick the data source you want the dashboard to use,
and specify any metric prefixes (if the dashboard uses any).
4 changes: 3 additions & 1 deletion helm-charts/charts/kube-starrocks/charts/starrocks/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,10 +34,12 @@ datadog:
enabled: false

# This configuration is used to integrate with external system Prometheus.
# You can enable the integration by setting the enabled to true.
metrics:
serviceMonitor:
# Enable a prometheus ServiceMonitor
# Note: make sure the prometheus operator is installed in your cluster.
# If prometheus is not installed by operator, you can add annotations on k8s service to expose metrics.
# see https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/doc/integration/integration-prometheus-grafana.md#51-turn-on-the-prometheus-metrics-scrape-by-adding-annotations for more details.
enabled: false
# Prometheus ServiceMonitor interval
interval: 15s
Expand Down
Loading