Skip to content

Commit

Permalink
Node log (#116)
Browse files Browse the repository at this point in the history
* singlog node log

* update configs

* fix dan and doodle

* fix dan3

* dan4

* fix dan5
  • Loading branch information
amber-moe authored Jul 27, 2020
1 parent 2fd31cd commit caf3f34
Show file tree
Hide file tree
Showing 6 changed files with 77 additions and 72 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -11,68 +11,6 @@
- Restart the services after changing the configuration files to take effect.
- Configuration changes in console take effect in real time.

<!-- ## gflag Parameters
**Nebula Graph** uses `gflags` for run-time configurations. `gflags` parameters see the following table.
Name | Type | Description
--------------------------------- | ------- | -----------
max_edge_returned_per_vertex | MUTABLE | Control the max edges returned by a certain vertex.
minloglevel | MUTABLE | Minimum log level.
v | MUTABLE | Debug log level.
heartbeat_interval_secs | MUTABLE | Heartbeat interval.
meta_client_retry_times | MUTABLE | Meta client retry times.
slow_op_threshhold_ms | MUTABLE | Default threshold for slow operation, set in ms
wal_ttl | MUTABLE | Life time of write-ahead-log in disks. Default value is `14400` secondes. The minimum value is `3600` seconds. When the setting is less than the minimum value, Nebula will ignore the value and regard it as `3600`.
rocksdb_db_options | NESTED | Parameter in json format, and the key and value of them are in string format.
rocksdb_column_family_options | NESTED | Parameter in json format, and the key and value of them are in string format.
rocksdb_block_based_table_options | NESTED | Parameter in json format, and the key and value of them are in string format.
For example, you can set as follows in the conf file of storage:
```text
rocksdb_db_options = {"stats_dump_period_sec":"200", "enable_write_thread_adaptive_yield":"false", "write_thread_max_yield_usec":"600"}
rocksdb_column_family_options = {"max_write_buffer_number":"4", "min_write_buffer_number_to_merge":"2", "max_write_buffer_number_to_maintain":"1"}
rocksdb_block_based_table_options = {"block_restart_interval":"2"}
"max_edge_returned_per_vertex":"INT_MAX"
```
**Nebula Graph** supports changing some rocksdb parameters in storage service as follows:
```text
// rocksdb_column_family_options
disable_auto_compactions
write_buffer_size
max_write_buffer_number
level0_file_num_compaction_trigger
level0_slowdown_writes_trigger
level0_stop_writes_trigger
target_file_size_base
target_file_size_multiplier
max_bytes_for_level_base
max_bytes_for_level_multiplier
// rocksdb_db_options
max_total_wal_size
delete_obsolete_files_period_micros
max_background_jobs
stats_dump_period_sec
compaction_readahead_size
writable_file_max_buffer_size
bytes_per_sync
wal_bytes_per_sync
delayed_write_rate
avoid_flush_during_shutdown
max_open_files
```
For example
```ngql
nebula> UPDATE CONFIGS storage:rocksdb_column_family_options = \
{ disable_auto_compactions = false, level0_file_num_compaction_trigger = 10 }
``` -->

## SHOW CONFIGS

```ngql
Expand Down Expand Up @@ -122,16 +60,18 @@ nebula> GET CONFIGS heartbeat_interval_secs;
-----------------------------------------------------------------
```

## Update CONFIGS
## UPDATE CONFIGS

```ngql
UPDATE CONFIGS [graph|meta|storage :] var = value
```

> The updated CONFIGS will be stored into the `meta service` permanently.
> If the configuration's mode is `MUTABLE`, the change will take effects immediately. Otherwise, if the mode is `REBOOT`, the change will not work until server restart.
> Expression is supported in UPDATE CONFIGS.
For example
> If the configuration's mode is `MUTABLE`, the change will take effect immediately.
> The configurations of some RocksDB parameters take effect after the services are restarted.
> Expressions are supported in the `UPDATE CONFIGS` command.
For example:

```ngql
nebula> UPDATE CONFIGS storage:heartbeat_interval_secs=1;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ max_background_jobs -- Use multi-thread for compacting. The default
```

> The above parameters can either be dynamically modified by UPDATE CONFIGS syntax, or written in the local configuration file. Please refer to the RocksDB manual for specific functions and whether restarting is needed.
The parameters above can either be dynamically modified by the `UPDATE CONFIGS` syntax, or written in the local configuration file. Please refer to the RocksDB manual for specific functions and whether restarting is needed.

### `rocksdb_column_family_options`

Expand All @@ -106,9 +106,9 @@ disable_auto_compactions -- Compact automatically when writing data is sto
```

> The above parameters can either be dynamically modified by UPDATE CONFIGS syntax, or written in the local configuration file. Please refer to the RocksDB manual for specific functions and whether restarting is needed.
> The preceding parameters can either be dynamically modified by UPDATE CONFIGS syntax, or written in the local configuration file. Please refer to the RocksDB manual for specific functions and whether restarting is needed.
The above parameters can be set via the command line as follows:
The preceding parameters can be set via the command line as follows:

```ngql
nebula> UPDATE CONFIGS storage:rocksdb_column_family_options = \
Expand All @@ -120,7 +120,7 @@ nebula> UPDATE CONFIGS storage:rocksdb_db_options = \
nebula> UPDATE CONFIGS storage:max_edge_returned_per_vertex = 10; -- The parameter is explained below
```

Configurations done with the configuration file
We recommend the following configuration:

```text
rocksdb_db_options = {"stats_dump_period_sec":"200", "write_thread_max_yield_usec":"600"}
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# Logs

**Nebula Graph** uses [glog](https://github.com/google/glog) to print logs, uses [gflag](https://gflags.github.io/gflags/) to control the severity level of the log, and provides an HTTP interface to dynamically change the log level at runtime to facilitate tracking.

## Log Directory

The default log directory is `/usr/local/nebula/logs/`.

**Note:** If you deleted the log directory during runtime, the runtime log would not continue to be printed. However, this operation will not affect the services. Restart the services to recover the logs.

## Parameter Description

### Two most commonly used flags in glog

- minloglevel: The scale of minloglevel is 0-4. The numbers of severity levels INFO(DEBUG), WARNING, ERROR, and FATAL are 0, 1, 2, and 3, respectively. Usually specified as 0 for debug, 1 for production. If you set the minloglevel to 4, no logs are printed.
- v: The scale of v is 0-3. When the value is set to 0, you can further set the severity level of the debug log. The greater the value is, the more detailed the log is.

### Configuration Files

The default severity level for the metad, graphd, and storaged logs can be found in the configuration files (usually in `/usr/local/nebula/etc/`).

## Check and Change the Severity Levels Dynamically

Check all the flag values (log values included) of the current gflags with the following command. `curl` is only available when the `local_config` parameter is set to `true`.

```bash
> curl ${ws_ip}:${ws_port}/get_flags
```

In the command:

- `ws_ip` is the IP address for the HTTP service, which can be found in the configuration files above. The default value is 127.0.0.1.
- `ws_port` is the port for the HTTP service, the default values for `metad`, `storaged`, and `graphd` are 11000, 12000,and 13000, respectively.

For example, check the minloglevel for the `storaged` service:

```bash
> curl 127.0.0.1:12000/get_flags | grep minloglevel
```

You can also change the logs' severity level to **the most detailed** with the following command.

```bash
> curl "http://127.0.0.1:12000/set_flags?flag=v&value=4"
> curl "http://127.0.0.1:12000/set_flags?flag=minloglevel&value=0"
```

In the Nebula console, check the severity minloglevel of graphd and set it to **the most detailed** with the following commands.

```ngql
nebula> GET CONFIGS graph:minloglevel;
nebula> UPDATE CONFIGS graph:minloglevel=0;
```

To change the severity of the storage log, replace the `graph` in the preceding command with `storage`.

**Note:** **Nebula Graph** only supports modifying the graph and storage log severity by using console. And the severity level of meta logs can only be modified with the `curl` command.

**Close** all logs print (FATAL only) with the following command.

```bash
> curl "http://127.0.0.1:12000/set_flags?flag=minloglevel&value=4"
```
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,13 @@

This document will walk you through the concept of Compact.

1. Start or stop compact. This method calls the default RocksDB compact. You can use it to merge sst files in small scale during data writing to ensure the read speed in a short time. This kind of compact is usually performed in the day time.
- The default RocksDB compact style. You can use the `UPDATE CONFIG` statement as follows to start or stop it. It merges sst files in small scale during data writing to speed up the data reading in a short time. When enabled, this compaction is automatically performed during the daytime.

```ngql
nebula> UPDATE CONFIG storage:disable_auto_compactions=false/true;
```

2. Start compact with the `SUBMIT JOB COMPACT` command. This method calls the customized compact of **Nebula Graph**. You can use it to perform large scale background operations such as sst files merging in large scale or TTL. This kind of compact is usually performed after midnight.
- The customized compact style for **Nebula Graph**. You can run the `SUBMIT JOB COMPACT` command to start it. You can use it to perform large scale background operations such as sst files merging in large scale or TTL. This kind of compact is usually performed after midnight.

In addition, you can modify the number of threads in both methods by the following command. You can decrease the threads during daytime and increase it at night.

Expand Down
1 change: 1 addition & 0 deletions docs/manual-EN/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,7 @@ It is the optimal solution in the world capable of hosting graphs with dozens of
* [Storaged Configuration](3.build-develop-and-administration/3.configurations/5.storage-config.md)
* [Console Configuration](3.build-develop-and-administration/3.configurations/6.console-config.md)
* [Kernel Configuration](3.build-develop-and-administration/3.configurations/7.kernel-config.md)
* [Single Node Log](3.build-develop-and-administration/3.configurations/8.single-node-log.md)
* Account Management Statement
* [Alter User Syntax](3.build-develop-and-administration/4.account-management-statements/alter-user-syntax.md)
* [Authentication](3.build-develop-and-administration/4.account-management-statements/authentication.md)
Expand Down
1 change: 1 addition & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,7 @@ nav:
- Storaged Configuration: manual-EN/3.build-develop-and-administration/3.configurations/5.storage-config.md
- Console Configuration: manual-EN/3.build-develop-and-administration/3.configurations/6.console-config.md
- Kernel Configuration: manual-EN/3.build-develop-and-administration/3.configurations/7.kernel-config.md
- Single Node Log: manual-EN/3.build-develop-and-administration/3.configurations/8.single-node-log.md
- Account Management Statement:
- Alter User Syntax: manual-EN/3.build-develop-and-administration/4.account-management-statements/alter-user-syntax.md
- Authentication: manual-EN/3.build-develop-and-administration/4.account-management-statements/authentication.md
Expand Down

0 comments on commit caf3f34

Please sign in to comment.