Skip to content

Commit

Permalink
Spell out Kubernetes in docs (grafana#10922)
Browse files Browse the repository at this point in the history
Developers tend to abbreviate "Kubernetes" to just K8, but as a best
practice it should be spelled out in the documentation.
  • Loading branch information
JStickler authored and pull[bot] committed Nov 1, 2024
1 parent 972ff59 commit a6922b0
Show file tree
Hide file tree
Showing 4 changed files with 19 additions and 19 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ rejected pushes. Users are recommended to do one of the following:

1. Have a dedicated Promtail instance for receiving pushes. This also applies to
using the syslog target.
1. Have a separatee k8s service that always resolves to the same Promtail pod,
1. Have a separate Kubernetes service that always resolves to the same Promtail pod,
bypassing the load balancing issue.

## Implementation
Expand Down
20 changes: 10 additions & 10 deletions docs/sources/operations/zone-ingesters.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,27 +7,27 @@ weight:

# Zone aware ingesters

Loki's zone aware ingesters are used by Grafana Labs in order to allow for easier rollouts of large Loki deployments. You can think of them as three logical zones, however with some extra k8s config you could deploy them in separate zones.
Loki's zone aware ingesters are used by Grafana Labs in order to allow for easier rollouts of large Loki deployments. You can think of them as three logical zones, however with some extra Kubernetes config you could deploy them in separate zones.

By default, an incoming log stream's logs are replicated to 3 random ingesters. Except in the case of some replica scaling up or down, a given stream will always be replicated to the same 3 ingesters. This means that if one of those ingesters is restarted no data is lost, but two restarting can mean data is lost and also impacts the systems ability to ingest logs because of an unhealthy ring status.

With zone awareness enabled, an incomming log line will be replicated to one ingester in each zone. This means that we're not only concerned if ingesters in multiple zones restart at the same time. We can now rollout, or lose, an entire zone at once and not impact the system. This allows deployments with a large number of ingesters to be deployed too much more quickly.
With zone awareness enabled, an incoming log line will be replicated to one ingester in each zone. This means that we're not only concerned if ingesters in multiple zones restart at the same time. We can now rollout, or lose, an entire zone at once and not impact the system. This allows deployments with a large number of ingesters to be deployed too much more quickly.

We also make use of [rollout-operator](https://github.com/grafana/rollout-operator) to manage rollouts to the 3 StatefulSets gracefully. The rollout-operator looks for labels on StatefulSets to know which StatefulSets are part of a certain rollout group, and coordinate rollouts of pods only from a single StatefulSet in the group at a time. See the README in the rollout-operator repo. for a more in depth explanation.

## Migration

Migrating from a single ingester StatefulSet to 3 zone aware ingester StatefulSets. The migration follows a few general steps, regardless of deployment method.

0. Configure your existing ingesters to be part of a zone, for example `zone-default`, this will allow us to later exclude them from the write path while still allowing for graceful shutdowns.
1. Configure your existing ingesters to be part of a zone, for example `zone-default`, this will allow us to later exclude them from the write path while still allowing for graceful shutdowns.
1. Prep for the increase in active streams (due to the way streams are split between ingesters) by increasing the # of active streams allowed for your tenants.
2. Add and scale up your new zone aware ingester StatefulSets such that each has 1/3 of the total # of replicas you want to run.
3. Enable zone awareness on the write path by setting `distributor.zone-awareness-enabled` to true for distributors and rulers.
4. Wait some time to ensure that the new zone aware ingesters have data for the time period they are queried for (`query_ingesters_within`).
5. Enable zone awareness on the read path by setting `distributor.zone-awareness-enabled` to true for queriers.
6. Configure distributors and rulers to exclude ingesters in the `zone-default` so those ingesters no longer receive write traffic via `distributor.excluded-zones`.
7. Use the shutdown endpoint to flush data from the default ingesters, then scale down and remove the associated StatefulSet.
8. Clean up any config remaining from the migration.
1. Add and scale up your new zone aware ingester StatefulSets such that each has 1/3 of the total # of replicas you want to run.
1. Enable zone awareness on the write path by setting `distributor.zone-awareness-enabled` to true for distributors and rulers.
1. Wait some time to ensure that the new zone aware ingesters have data for the time period they are queried for (`query_ingesters_within`).
1. Enable zone awareness on the read path by setting `distributor.zone-awareness-enabled` to true for queriers.
1. Configure distributors and rulers to exclude ingesters in the `zone-default` so those ingesters no longer receive write traffic via `distributor.excluded-zones`.
1. Use the shutdown endpoint to flush data from the default ingesters, then scale down and remove the associated StatefulSet.
1. Clean up any config remaining from the migration.

### Detailed Migration Steps

Expand Down
8 changes: 4 additions & 4 deletions docs/sources/send-data/fluentd/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -185,7 +185,7 @@ This plugin automatically adds a `fluentd_thread` label with the name of the buf

### `url`

The URL of the Loki server to send logs to. When sending data, the publish path (`../reference/api/loki/v1/push`) will automatically be appended.
The URL of the Loki server to send logs to. When sending data, the publish path (`../reference/api/loki/v1/push`) will automatically be appended.
By default the url is set to `https://logs-prod-us-central1.grafana.net`, the url of the Grafana Labs [hosted Loki](/products/cloud/) service.

#### Proxy Support
Expand All @@ -199,10 +199,10 @@ If using the GrafanaLab's hosted Loki, the username needs to be set to your inst

### tenant

All requests sent to Loki, a multi-tenant log storage platform, must include a tenant. For some installations the tenant will be set automatically by an authenticating proxy. Otherwise you can define a tenant to be passed through.
All requests sent to Loki, a multi-tenant log storage platform, must include a tenant. For some installations the tenant will be set automatically by an authenticating proxy. Otherwise you can define a tenant to be passed through.
The tenant can be any string value.

The tenant field also supports placeholders, allowing it to dynamically change based on tag and record fields. Each placeholder must be added as a buffer chunk key. The following is an example of setting the tenant based on a k8s pod label:
The tenant field also supports placeholders, allowing it to dynamically change based on tag and record fields. Each placeholder must be added as a buffer chunk key. The following is an example of setting the tenant based on a Kubernetes pod label:

```conf
<match **>
Expand Down Expand Up @@ -253,7 +253,7 @@ A flag to disable server certificate verification. By default the `insecure_tls`

### Output format

Loki is intended to index and group log streams using only a small set of labels. It is not intended for full-text indexing. When sending logs to Loki the majority of log message will be sent as a single log "line".
Loki is intended to index and group log streams using only a small set of labels. It is not intended for full-text indexing. When sending logs to Loki the majority of log message will be sent as a single log "line".

Several configuration settings are available to control the output format.

Expand Down
8 changes: 4 additions & 4 deletions docs/sources/setup/upgrade/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -266,19 +266,19 @@ ruler:

### Querier

#### query-frontend k8s headless service changed to load balanced service
#### query-frontend Kubernetes headless service changed to load balanced service

*Note:* This is relevant only if you are using [jsonnet for deploying Loki in Kubernetes](/docs/loki/latest/installation/tanka/)

The `query-frontend` k8s service was previously headless and was used for two purposes:
The `query-frontend` Kubernetes service was previously headless and was used for two purposes:
* Distributing the Loki query requests amongst all the available Query Frontend pods.
* Discover IPs of Query Frontend pods from Queriers to connect as workers.

The problem here is that a headless service does not support load balancing and leaves it up to the client to balance the load.
Additionally, a load-balanced service does not let us discover the IPs of the underlying pods.

To meet both these requirements, we have made the following changes:
* Changed the existing `query-frontend` k8s service from headless to load-balanced to have a fair load distribution on all the Query Frontend instances.
* Changed the existing `query-frontend` Kubernetes service from headless to load-balanced to have a fair load distribution on all the Query Frontend instances.
* Added `query-frontend-headless` to discover QF pod IPs from queriers to connect as workers.

If you are deploying Loki with Query Scheduler by setting [query_scheduler_enabled](https://github.com/grafana/loki/blob/cc4ab7487ab3cd3b07c63601b074101b0324083b/production/ksonnet/loki/config.libsonnet#L18) config to `true`, then there is nothing to do here for this change.
Expand Down Expand Up @@ -1094,7 +1094,7 @@ and the Kubelet.
This commit adds the same to the Loki scrape config. It also removes
the container_name label. It is the same as the container label
and was already added to Loki previously. However, the
container_name label is deprecated and has disappeared in K8s 1.16,
container_name label is deprecated and has disappeared in Kubernetes 1.16,
so that it will soon become useless for direct joining.
````

Expand Down

0 comments on commit a6922b0

Please sign in to comment.