Skip to content

Commit

Permalink
[pkg/stanza/operator/transformer/recombine] docs: add example configu…
Browse files Browse the repository at this point in the history
…rations with max_unmatched_batch_size (#32347)

**Description:** add example configurations for recombine operator with
`max_unmatched_batch_size`

**Link to tracking Issue:** #31653
  • Loading branch information
kasia-kujawa authored Apr 16, 2024
1 parent e4d4bc9 commit 1855b19
Showing 1 changed file with 144 additions and 1 deletion.
145 changes: 144 additions & 1 deletion pkg/stanza/docs/operators/recombine.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ The `recombine` operator combines consecutive logs into single logs based on sim
| `combine_field` | required | The [field](../types/field.md) from all the entries that will recombined. |
| `combine_with` | `"\n"` | The string that is put between the combined entries. This can be an empty string as well. When using special characters like `\n`, be sure to enclose the value in double quotes: `"\n"`. |
| `max_batch_size` | 1000 | The maximum number of consecutive entries that will be combined into a single entry. |
| `max_unmatched_batch_size` | 100 | The maximum number of consecutive entries that will be combined into a single entry before the match occurs (with `is_first_entry` or `is_last_entry`), e.g. `max_unmatched_batch_size=0` - all entries combined, `max_unmatched_batch_size=1` - all entries uncombined until the first match occurs, `max_unmatched_batch_size=100` - entries combined into 100-entry-packages until the first match occurs |
| `max_unmatched_batch_size` | 100 | The maximum number of consecutive entries that will be combined into a single entry before the match occurs (with `is_first_entry` or `is_last_entry`), e.g. `max_unmatched_batch_size=0` - all entries combined, `max_unmatched_batch_size=1` - all entries uncombined until the match occurs, `max_unmatched_batch_size=100` - entries combined into 100-entry-packages until the match occurs |
| `overwrite_with` | `newest` | Whether to use the fields from the `oldest` or the `newest` entry for all the fields that are not combined. |
| `force_flush_period` | `5s` | Flush timeout after which entries will be flushed aborting the wait for their sub parts to be merged with. |
| `source_identifier` | `$attributes["file.path"]` | The [field](../types/field.md) to separate one source of logs from others when combining them. |
Expand Down Expand Up @@ -141,3 +141,146 @@ The following logs will be output:
},
]
```

#### Example configurations with `max_unmatched_batch_size`

##### `max_unmatched_batch_size` set to `0`

When `max_unmatched_batch_size` is set to `0` all logs entries will be combined into a single entry before the match occurs.

Example configuration:

```yaml
- type: recombine
combine_field: body
is_first_entry: body == 'log1'
max_unmatched_batch_size: 0
```
Input log entries:
```
log2
log3
log4
log1
log5
log6
log7
log1
```

Output log entries:

```
log2\nlog3\nlog4
log1\nlog5\nlog6\nlog7
log1
```

##### `max_unmatched_batch_size` set to `1`

When `max_unmatched_batch_size` is set to `1` all logs entries will not be combined before the match occurs.

Example configuration:

```yaml
- type: recombine
combine_field: body
is_first_entry: body == 'log1'
max_unmatched_batch_size: 1
```
Input log entries:
```
log2
log3
log4
log1
log5
log6
log7
log1
```

Output log entries:

```
log2
log3
log4
log1\nlog5\nlog6\nlog7
log1
```

##### `max_unmatched_batch_size` set to `3`

When `max_unmatched_batch_size` is set to `3` entries will be combined into 3-entry-packages until the match occurs.

Example configuration with `is_first_entry`:

```yaml
- type: recombine
combine_field: body
is_first_entry: body == 'log1'
max_unmatched_batch_size: 3
```
Input log entries:
```
log2
log3
log4
log5
log1
log6
log7
log1
log8
log1
```

Output log entries:

```
log2\nlog3\nlog4
log5
log1\nlog6\nlog7
log1\nlog8
log1
```

Example configuration with `is_last_entry`:

```yaml
- type: recombine
combine_field: body
is_last_entry: body == 'log1'
max_unmatched_batch_size: 3
```
Input log entries:
```
log2
log3
log4
log5
log1
log6
log7
log1
log8
log1
```

Output log entries:

```
log2\nlog3\nlog4
log5\nlog1
log6\nlog7\nlog1
log8\nlog1
```

0 comments on commit 1855b19

Please sign in to comment.