Skip to content

Commit

Permalink
[Doc] add load_data_using_stream_load.md
Browse files Browse the repository at this point in the history
Signed-off-by: yandongxiao <[email protected]>
  • Loading branch information
yandongxiao committed Nov 1, 2023
1 parent 95436e2 commit 38ebcc9
Show file tree
Hide file tree
Showing 8 changed files with 853 additions and 2 deletions.
7 changes: 5 additions & 2 deletions doc/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,15 @@ Table Of Content
- HOWTO
- [Deploy StarRocks With Operator](./deploy_starrocks_with_operator_howto.md)
- [Deploy StarRocks With Helm](./deploy_starrocks_with_helm_howto.md)
- [Change Root Password HOWTO](./change_root_password_howto.md)
- [Init Root Password When First Deploy HOWTO](./initialize_root_password_howto.md)
- [Install StarRocks Locally](./local_installation_how_to.md)
- [Change Root Password](./change_root_password_howto.md)
- [Init Root Password When First Deploy](./initialize_root_password_howto.md)
- [Build Operator Image By Yourself](./build_the_operator_images_by_yourself_howto.md)
- [Mount Persistent Volume](./mount_persistent_volume_howto.md)
- [Mount External ConfigMaps Or Secrets](./mount_external_configmaps_or_secrets_howto.md)
- [Logging and Related Configurations](./logging_and_related_configurations_howto.md)
- [Automatic Scaling For CN Nodes](./automatic_scaling_for_cn_nodes_howto.md)
- [Load Data Using Stream Load](./load_data_using_stream_load_howto.md)
- Integration
- [Prometheus And Grafana](./integration/integration-prometheus-grafana.md)
- [Datadog](./integration/integration-with-datadog.md)
Binary file added doc/images/issues_of_stream_load.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added doc/images/stream_load-fe-proxy.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
760 changes: 760 additions & 0 deletions doc/images/stream_load.drawio

Large diffs are not rendered by default.

Binary file added doc/images/stream_load_interaction.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
88 changes: 88 additions & 0 deletions doc/load_data_using_stream_load_howto.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
# Load Data Using Stream Load

The issue is that when the load client residents in a different network other than FE/BE's private network. FE's HTTP
307 brings BE's private network address to the client who does not recognize and can't process the redirection.

## Issues of Stream Load

Stream Load Interaction:

![Stream Load Interaction](./images/stream_load_interaction.png)

The issues:

![Issues When Client Is Not In the Same Private Network](./images/issues_of_stream_load.png)

## Issues of Transactional Stream Load

Transactional Stream Load Interaction:

![Transactional Stream Load Interaction](./images/transactional_stream_load_interaction.png)

The issues:

![Issues When Client Is Not In the Same Private Network](./images/issues_of_transactional_stream_load.png)

## Using Fe Proxy to Solve the Issues

![Using Fe Proxy to Solve the Issues](./images/stream_load-fe-proxy.png)

## How to Deploy Fe Proxy

### Deploy Fe Proxy Using Helm

If you install StarRocks with Helm, you can add the following configuration to the `values.yaml` file:

For `kube-starrocks` Helm chart:

```yaml
starrocks:
starRocksFeProxySpec:
replicas: 1
# set the resolver for nginx server, default kube-dns.kube-system.svc.cluster.local
resolver: ""
limits:
cpu: 1
memory: 2Gi
requests:
cpu: 1
memory: 2Gi
service:
type: NodePort
ports:
- containerPort: 8080
name: http-port
nodePort: 30180
port: 8080
```
For `starrocks` Helm chart:

```yaml
starRocksFeProxySpec:
replicas: 1
# set the resolver for nginx server, default kube-dns.kube-system.svc.cluster.local
resolver: ""
limits:
cpu: 1
memory: 2Gi
requests:
cpu: 1
memory: 2Gi
service:
type: NodePort
ports:
- containerPort: 8080
name: http-port
nodePort: 30180
port: 8080
```

Please
see https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/helm-charts/charts/kube-starrocks/values.yaml
for more details about how to configure `starRocksFeProxySpec`.

### Deploy Fe Proxy Using Helm

If you install StarRocks with StarRocksCluster CR yaml, please see [deploy_a_starrocks_cluster_with_fe_proxy.md](
../examples/starrocks/deploy_a_starrocks_cluster_with_fe_proxy.yaml)

0 comments on commit 38ebcc9

Please sign in to comment.