Skip to content
This repository has been archived by the owner on Aug 16, 2023. It is now read-only.

Commit

Permalink
Add kafka for milvus cluster message queue (#319)
Browse files Browse the repository at this point in the history
Signed-off-by: Edward Zeng <[email protected]>
  • Loading branch information
LoveEachDay authored Mar 28, 2022
1 parent 1d9e100 commit 3bc7aab
Show file tree
Hide file tree
Showing 7 changed files with 101 additions and 4 deletions.
2 changes: 1 addition & 1 deletion charts/milvus/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ name: milvus
appVersion: "2.0.1"
kubeVersion: "^1.10.0-0"
description: Milvus is an open-source vector database built to power AI applications and vector similarity search.
version: 3.0.13
version: 3.0.14
keywords:
- milvus
- elastic
Expand Down
15 changes: 15 additions & 0 deletions charts/milvus/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,12 @@ Assume the release name is `my-release`:
# Helm v3.x
$ helm upgrade --install my-release milvus/milvus
```
By default, milvus cluster uses `Pulsar` as message queue. You can also use `Kafka` instead of `Pulsar` for milvus cluster:

```bash
# Helm v3.x
$ helm upgrade --install my-release milvus/milvus --set pulsar.enabled=false --set kafka.enabled=true
```

### Upgrade an existing Milvus cluster
E.g. to scale out query node from 1(default) to 2:
Expand Down Expand Up @@ -129,6 +135,8 @@ The following table lists the configurable parameters of the Milvus Service and
| `externalPulsar.enabled` | Enable or disable external Pulsar | `false` |
| `externalPulsar.host` | The host of the external Pulsar | `localhost` |
| `externalPulsar.port` | The port of the external Pulsar | `6650` |
| `externalKafka.enabled` | Enable or disable external Kafka | `false` |
| `externalKafka.brokerList` | The brokerList of the external Kafka separated by comma | `localhost:9092` |

### Milvus Standalone Deployment Configuration

Expand Down Expand Up @@ -316,3 +324,10 @@ This version of the chart includes the dependent Minio chart in the charts/ dire

You can find more information at:
* [https://github.com/minio/charts/blob/master/README.md](https://github.com/minio/charts/blob/master/README.md)

### Kafka Configuration

This version of the chart includes the dependent Kafka chart in the charts/ directory.

You can find more information at:
* [https://artifacthub.io/packages/helm/bitnami/kafka](https://artifacthub.io/packages/helm/bitnami/kafka)
Binary file added charts/milvus/charts/kafka-15.5.1.tgz
Binary file not shown.
7 changes: 5 additions & 2 deletions charts/milvus/requirements.lock
Original file line number Diff line number Diff line change
Expand Up @@ -8,5 +8,8 @@ dependencies:
- name: pulsar
repository: https://pulsar.apache.org/charts
version: 2.7.8
digest: sha256:5a785d4805e5e75f58240a5f683bb0a47f06d2dd24682e503ce01ab475cc73a1
generated: "2022-03-10T15:19:41.299459+08:00"
- name: kafka
repository: https://charts.bitnami.com/bitnami
version: 15.5.1
digest: sha256:6f7dac7840a8b00b76a0bb819094cb5ea287dd0021902f08ba769c12d6b4847e
generated: "2022-03-23T14:45:09.223015+08:00"
6 changes: 6 additions & 0 deletions charts/milvus/requirements.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -17,3 +17,9 @@ dependencies:
condition: pulsar.enabled
tags:
- pulsar
- name: kafka
version: 15.5.1
repository: https://charts.bitnami.com/bitnami
condition: kafka.enabled
tags:
- kafka
19 changes: 18 additions & 1 deletion charts/milvus/templates/config.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -55,11 +55,14 @@ minio:
rootPath: {{ .Values.minio.rootPath }}
{{- end }}

pulsar:
{{- if .Values.externalPulsar.enabled }}

pulsar:
address: {{ .Values.externalPulsar.host }}
port: {{ .Values.externalPulsar.port }}
{{- else if .Values.pulsar.enabled }}

pulsar:
{{- if contains .Values.pulsar.name .Release.Name }}
address: {{ .Release.Name }}-proxy
{{- else }}
Expand All @@ -68,6 +71,20 @@ pulsar:
port: {{ .Values.pulsar.proxy.ports.pulsar }}
{{- end }}

{{- if .Values.externalKafka.enabled }}

kafka:
brokerList: {{ .Values.externalKafka.brokerList }}
{{- else if .Values.kafka.enabled }}

kafka:
{{- if contains .Values.kafka.name .Release.Name }}
brokerList: {{ .Release.Name }}:{{ .Values.kafka.service.ports.client }}
{{- else }}
brokerList: {{ .Release.Name }}-{{ .Values.kafka.name }}:{{ .Values.kafka.service.ports.client }}
{{- end }}
{{- end }}

rocksmq:
path: "{{ .Values.standalone.persistence.mountPath }}/rdb_data"
rocksmqPageSize: "{{ .Values.standalone.rocksmq.rocksmqPageSize }}" # 2 GB
Expand Down
56 changes: 56 additions & 0 deletions charts/milvus/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -616,6 +616,54 @@ pulsar:
tag: 2.8.2


## Configuration values for the kafka dependency
## ref: https://artifacthub.io/packages/helm/bitnami/kafka
##

kafka:
enabled: false
name: kafka
replicaCount: 3
image:
repository: bitnami/kafka
tag: 3.1.0-debian-10-r52

## Kafka Java Heap size
heapOpts: "-Xmx4096m -Xms4096m"
extraEnvVars:
- name: KAFKA_CFG_MAX_PARTITION_FETCH_BYTES
value: "5242880"
- name: KAFKA_CFG_MAX_REQUEST_SIZE
value: "5242880"
- name: KAFKA_CFG_MESSAGE_MAX_BYTES
value: "5242880"
- name: KAFKA_CFG_REPLICA_FETCH_MAX_BYTES
value: "5242880"
- name: KAFKA_CFG_FETCH_MESSAGE_MAX_BYTES
value: "5242880"

persistence:
enabled: true
storageClass:
accessMode: ReadWriteOnce
size: 300Gi

metrics:
kafka:
enabled: false
image:
repository: bitnami/kafka-exporter
tag: 1.4.2-debian-10-r182

service:
type: ClusterIP
ports:
client: 9092

zookeeper:
enabled: true
replicaCount: 3

###################################
# External S3
# - these configs are only used when `externalS3.enabled` is true
Expand Down Expand Up @@ -656,3 +704,11 @@ externalPulsar:
enabled: false
host: localhost
port: 6650

###################################
# External kafka
# - these configs are only used when `externalKafka.enabled` is true
###################################
externalKafka:
enabled: false
brokerList: localhost:9092

0 comments on commit 3bc7aab

Please sign in to comment.