From 950747df14683434bf1a7ff1199e8e8e01fc1b47 Mon Sep 17 00:00:00 2001 From: Rich Loveland Date: Thu, 7 Oct 2021 20:30:32 -0400 Subject: [PATCH] Add docs on automatic ballast files Fixes #11156 Addresses a little bit of #11917 --- v21.2/cluster-setup-troubleshooting.md | 20 ++++++++++++++++++-- v21.2/cockroach-debug-ballast.md | 4 +++- v21.2/cockroach-start.md | 5 +++-- v21.2/operational-faqs.md | 8 ++++++-- v21.2/recommended-production-settings.md | 2 +- 5 files changed, 31 insertions(+), 8 deletions(-) diff --git a/v21.2/cluster-setup-troubleshooting.md b/v21.2/cluster-setup-troubleshooting.md index a85eb4551bd..0b0e4addb41 100644 --- a/v21.2/cluster-setup-troubleshooting.md +++ b/v21.2/cluster-setup-troubleshooting.md @@ -314,11 +314,27 @@ Network capacity | Network Bytes Received
Network Bytes Sent | Consistently ### Disks filling up -Like any database system, if you run out of disk space the system will no longer be able to accept writes. Additionally, a CockroachDB node needs a small amount of disk space (a few GBs to be safe) to perform basic maintenance functionality. For more information about this issue, see: +Like any database system, if you run out of disk space the system will no longer be able to accept writes. Additionally, a CockroachDB node needs a small amount of disk space (a few GiBs to be safe) to perform basic maintenance functionality. For more information about this issue, see: +- [What happens when a node runs out of disk space?](operational-faqs.html#what-happens-when-a-node-runs-out-of-disk-space) - [Why is memory usage increasing despite lack of traffic?](operational-faqs.html#why-is-memory-usage-increasing-despite-lack-of-traffic) - [Why is disk usage increasing despite lack of writes?](operational-faqs.html#why-is-disk-usage-increasing-despite-lack-of-writes) -- [Can I reduce or disable the storage of timeseries data?](operational-faqs.html#can-i-reduce-or-disable-the-storage-of-time-series-data) +- [Can I reduce or disable the storage of timeseries data?](operational-faqs.html#can-i-reduce-or-disable-the-storage-of-time-series-data) + +#### Automatic ballast files + +New in v21.2 CockroachDB automatically creates an emergency ballast file at [node startup](cockroach-start.html). This feature is **on** by default. Note that the [`cockroach debug ballast`](cockroach-debug-ballast.html) command is still available but deprecated. + +The ballast file defaults to 1% of total disk capacity or 1GiB, whichever is smaller. The size of the ballast file may be configured using [the `--store` flag to `cockroach start`](cockroach-start.html#flags-store) with a [`ballast-size` field](cockroach-start.html#fields-ballast-size); this field accepts the same value formats as the `size` field. + +The ballast file is automatically created when either: + +- Available disk space is at least four times the ballast file size. +- Available disk space after creating the ballast file is at least 10 GiB. + +During node startup, if available disk space on at least one store is less than or equal to half the ballast file size, the node will exit immediately with the code `Disk Full (10)`. + +To allow the node to start, you can manually remove the ballast file; this will give you a chance to remedy the disk space exhaustion. The ballast file will automatically be recreated when there is sufficient disk space. ### Disk stalls diff --git a/v21.2/cockroach-debug-ballast.md b/v21.2/cockroach-debug-ballast.md index 6629092028c..7c3762d8dcd 100644 --- a/v21.2/cockroach-debug-ballast.md +++ b/v21.2/cockroach-debug-ballast.md @@ -4,7 +4,9 @@ summary: Create a large, unused file in a node's storage directory that you can toc: true --- -The `cockroach debug ballast` [command](cockroach-commands.html) creates a large, unused file that you can place in a node's storage directory. In the case that a node runs out of disk space and shuts down, you can delete the ballast file to free up enough space to be able to restart the node. +New in v21.2: CockroachDB automatically creates an emergency ballast file at startup time. The `cockroach debug ballast` command is still available but deprecated. For more information about how automatic ballast file creation works, see [automatic ballast files](cluster-setup-troubleshooting.html#automatic-ballast-files). + +The `cockroach debug ballast` [command](cockroach-commands.html) creates a large, unused file that you can place in a node's storage directory. In the case that a node runs out of disk space and shuts down, you can delete the ballast file to free up enough space to be able to restart the node. - Do not run `cockroach debug ballast` with a unix `root` user. Doing so brings the risk of mistakenly affecting system directories or files. - `cockroach debug ballast` now refuses to overwrite the target ballast file if it already exists. This change is intended to prevent mistaken uses of the `ballast` command. Consider adding an `rm` command to scripts that integrate `cockroach debug ballast`, or provide a new file name every time and then remove the old file. diff --git a/v21.2/cockroach-start.md b/v21.2/cockroach-start.md index 5011b195b5d..68f6012ea81 100644 --- a/v21.2/cockroach-start.md +++ b/v21.2/cockroach-start.md @@ -64,7 +64,7 @@ Flag | Description `--max-offset` | The maximum allowed clock offset for the cluster. If observed clock offsets exceed this limit, servers will crash to minimize the likelihood of reading inconsistent data. Increasing this value will increase the time to recovery of failures as well as the frequency of uncertainty-based read restarts.

Note that this value must be the same on all nodes in the cluster and cannot be changed with a [rolling upgrade](upgrade-cockroach-version.html). In order to change it, first stop every node in the cluster. Then once the entire cluster is offline, restart each node with the new value.

**Default:** `500ms` `--max-sql-memory` | The maximum in-memory storage capacity available to store temporary data for SQL queries, including prepared queries and intermediate data rows during query execution. This can be a percentage (notated as a decimal or with `%`) or any bytes-based unit, for example:

`--max-sql-memory=.25`
`--max-sql-memory=25%`
`--max-sql-memory=10000000000 ----> 1000000000 bytes`
`--max-sql-memory=1GB ----> 1000000000 bytes`
`--max-sql-memory=1GiB ----> 1073741824 bytes`

The temporary files are located in the path specified by the `--temp-dir` flag, or in the subdirectory of the first store (see `--store`) by default.

Note: If you use the `%` notation, you might need to escape the `%` sign, for instance, while configuring CockroachDB through `systemd` service files. For this reason, it's recommended to use the decimal notation instead.

**Default:** `25%`

The default SQL memory size is suitable for production deployments but can be raised to increase the number of simultaneous client connections the node allows as well as the node's capacity for in-memory processing of rows when using `ORDER BY`, `GROUP BY`, `DISTINCT`, joins, and window functions. For local development clusters with memory-intensive workloads, reduce this value to, for example, `128MiB` to prevent out of memory errors. `--pid-file` | The file to which the node's process ID will be written as soon as the node is ready to accept connections. When `--background` is used, this happens before the process detaches from the terminal. When this flag is not set, the process ID is not written to file. -`--store`
`-s` | The file path to a storage device and, optionally, store attributes and maximum size. When using multiple storage devices for a node, this flag must be specified separately for each device, for example:

`--store=/mnt/ssd01 --store=/mnt/ssd02`

For more details, see [Store](#store) below. + `--store`
`-s` | The file path to a storage device and, optionally, store attributes and maximum size. When using multiple storage devices for a node, this flag must be specified separately for each device, for example:

`--store=/mnt/ssd01 --store=/mnt/ssd02`

For more details, see [Store](#store) below. `--spatial-libs` | The location on disk where CockroachDB looks for [spatial](spatial-features.html) libraries.

**Defaults:**

`--temp-dir` | The path of the node's temporary store directory. On node start up, the location for the temporary files is printed to the standard output.

**Default:** Subdirectory of the first [store](#store) @@ -145,7 +145,8 @@ Field | Description `type` | For in-memory storage, set this field to `mem`; otherwise, leave this field out. The `path` field must not be set when `type=mem`. `path` | The file path to the storage device. When not setting `attr` or `size`, the `path` field label can be left out:

`--store=/mnt/ssd01`

When either of those fields are set, however, the `path` field label must be used:

`--store=path=/mnt/ssd01,size=20GB`

**Default:** `cockroach-data` `attrs` | Arbitrary strings, separated by colons, specifying disk type or capability. These can be used to influence the location of data replicas. See [Configure Replication Zones](configure-replication-zones.html#replication-constraints) for full details.

In most cases, node-level `--locality` or `--attrs` are preferable to store-level attributes, but this field can be used to match capabilities for storage of individual databases or tables. For example, an OLTP database would probably want to allocate space for its tables only on solid state devices, whereas append-only time series might prefer cheaper spinning drives. Typical attributes include whether the store is flash (`ssd`) or spinny disk (`hdd`), as well as speeds and other specs, for example:

`--store=path=/mnt/hda1,attrs=hdd:7200rpm` -`size` | The maximum size allocated to the node. When this size is reached, CockroachDB attempts to rebalance data to other nodes with available capacity. When there's no capacity elsewhere, this limit will be exceeded. Also, data may be written to the node faster than the cluster can rebalance it away; in this case, as long as capacity is available elsewhere, CockroachDB will gradually rebalance data down to the store limit.

The `size` can be specified either in a bytes-based unit or as a percentage of hard drive space (notated as a decimal or with `%`), for example:

`--store=path=/mnt/ssd01,size=10000000000 ----> 10000000000 bytes`
`--store=path=/mnt/ssd01,size=20GB ----> 20000000000 bytes`
`--store=path=/mnt/ssd01,size=20GiB ----> 21474836480 bytes`
`--store=path=/mnt/ssd01,size=0.02TiB ----> 21474836480 bytes`
`--store=path=/mnt/ssd01,size=20% ----> 20% of available space`
`--store=path=/mnt/ssd01,size=0.2 ----> 20% of available space`
`--store=path=/mnt/ssd01,size=.2 ----> 20% of available space`

**Default:** 100%

For an in-memory store, the `size` field is required and must be set to the true maximum bytes or percentage of available memory, for example:

`--store=type=mem,size=20GB`
`--store=type=mem,size=90%`

Note: If you use the `%` notation, you might need to escape the `%` sign, for instance, while configuring CockroachDB through `systemd` service files. For this reason, it's recommended to use the decimal notation instead. + `size` | The maximum size allocated to the node. When this size is reached, CockroachDB attempts to rebalance data to other nodes with available capacity. When there's no capacity elsewhere, this limit will be exceeded. Also, data may be written to the node faster than the cluster can rebalance it away; in this case, as long as capacity is available elsewhere, CockroachDB will gradually rebalance data down to the store limit.

The `size` can be specified either in a bytes-based unit or as a percentage of hard drive space (notated as a decimal or with `%`), for example:

`--store=path=/mnt/ssd01,size=10000000000 ----> 10000000000 bytes`
`--store=path=/mnt/ssd01,size=20GB ----> 20000000000 bytes`
`--store=path=/mnt/ssd01,size=20GiB ----> 21474836480 bytes`
`--store=path=/mnt/ssd01,size=0.02TiB ----> 21474836480 bytes`
`--store=path=/mnt/ssd01,size=20% ----> 20% of available space`
`--store=path=/mnt/ssd01,size=0.2 ----> 20% of available space`
`--store=path=/mnt/ssd01,size=.2 ----> 20% of available space`

**Default:** 100%

For an in-memory store, the `size` field is required and must be set to the true maximum bytes or percentage of available memory, for example:

`--store=type=mem,size=20GB`
`--store=type=mem,size=90%`

Note: If you use the `%` notation, you might need to escape the `%` sign, for instance, while configuring CockroachDB through `systemd` service files. For this reason, it's recommended to use the decimal notation instead. + New in v21.2 `ballast-size` | Configure the size of the automatically created emergency ballast file. Accepts the same value formats as the [`size` field](#store-size). ### Logging diff --git a/v21.2/operational-faqs.md b/v21.2/operational-faqs.md index 05ad5298b47..bae1eb42b39 100644 --- a/v21.2/operational-faqs.md +++ b/v21.2/operational-faqs.md @@ -106,7 +106,11 @@ If you want all existing time-series data to be deleted, also change both the `t ## What happens when a node runs out of disk space? -When a node runs out of disk space, it shuts down and cannot be restarted until space is freed up. To prepare for this case, place a [ballast file](cockroach-debug-ballast.html) in each node's storage directory that can be deleted to free up enough space to be able to restart the node. If you did not create a ballast file, look for other files that can be deleted, such as [log files](configure-logs.html#logging-directory). +When a node runs out of disk space, it shuts down and cannot be restarted until space is freed up. + +New in v21.2 To prepare for this case, CockroachDB [automatically creates an emergency ballast file](cluster-setup-troubleshooting.html#automatic-ballast-files) in each node's storage directory that can be deleted to free up enough space to be able to restart the node. + +For more information about troubleshooting disk usage issues, see [storage issues](cluster-setup-troubleshooting.html#disks-filling-up). {{site.data.alerts.callout_info}} In addition to using ballast files, it is important to actively [monitor remaining disk space](monitoring-and-alerting.html#events-to-alert-on). @@ -140,4 +144,4 @@ You can also see these metrics in [the Clock Offset graph](ui-runtime-dashboard. - [Production Checklist](recommended-production-settings.html) - [Product FAQs](frequently-asked-questions.html) -- [SQL FAQs](sql-faqs.html) \ No newline at end of file +- [SQL FAQs](sql-faqs.html) diff --git a/v21.2/recommended-production-settings.md b/v21.2/recommended-production-settings.md index a87b54334c9..6019a46a019 100644 --- a/v21.2/recommended-production-settings.md +++ b/v21.2/recommended-production-settings.md @@ -82,7 +82,7 @@ We recommend provisioning volumes with **150 GiB per vCPU**. It's fine to have l We strongly recommend [monitoring](monitoring-and-alerting.html#node-is-running-low-on-disk-space) your storage utilization and rate of growth, and taking action to add capacity well before you hit the limit. -- Place a [ballast file](cockroach-debug-ballast.html) in each node's storage directory. In the unlikely case that a node runs out of disk space and shuts down, you can delete the ballast file to free up enough space to be able to restart the node. +- New in v21.2: CockroachDB will [automatically provision an emergency ballast file](cluster-setup-troubleshooting.html#automatic-ballast-files) at [node startup](cockroach-start.html). In the unlikely case that a node runs out of disk space and shuts down, you can delete the ballast file to free up enough space to be able to restart the node. - Use [zone configs](configure-replication-zones.html) to increase the replication factor from 3 (the default) to 5 (across at least 5 nodes).