Skip to content

Commit

Permalink
docs: fix typos and reformat
Browse files Browse the repository at this point in the history
  • Loading branch information
@fhussonnois
fhussonnois committed Mar 2, 2022
1 parent b8dd10b commit c5cb5a3
Show file tree
Hide file tree
Showing 8 changed files with 132 additions and 132 deletions.
148 changes: 74 additions & 74 deletions docs/content/en/docs/Developer Guide/accessing-data-and-metadata.md
View file

Large diffs are not rendered by default.

28 changes: 14 additions & 14 deletions docs/content/en/docs/Developer Guide/cleaning-completed-files.md
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
date: 2021-06-10
date: 2022-03-02
title: "File Cleanup Policies"
linkTitle: "File Cleanup Policies"
weight: 100
Expand All @@ -11,9 +11,9 @@ The connector can be configured with a specific [FileCleanupPolicy](https://gith

The cleanup policy can be configured with the below connect property :

| Configuration | Description | Type | Default | Importance |
| --------------| --------------|-----------| --------- | ------------- |
|`fs.cleanup.policy.class` | The fully qualified name of the class which is used to cleanup files | class | *-* | high |
| Configuration | Description | Type | Default | Importance |
|---------------------------|----------------------------------------------------------------------|-------|---------|------------|
| `fs.cleanup.policy.class` | The fully qualified name of the class which is used to cleanup files | class | *-* | high |


## Available Cleanup Policies
Expand All @@ -22,30 +22,30 @@ The cleanup policy can be configured with the below connect property :

This policy deletes all files regardless of their final status (completed or failed).

To enable this policy, the property `fs.cleanup.policy.class` must configured to :
To enable this policy, the property `fs.cleanup.policy.class` must be configured to :

```
io.streamthoughts.kafka.connect.filepulse.fs.clean.DeleteCleanupPolicy
```

### `LogCleanPolicy`

This policy prints to logs some information after files completion.
This policy prints into logs some information after files completion.

To enable this policy, the property `fs.cleanup.policy.class` must configured to :
To enable this policy, the property `fs.cleanup.policy.class` must be configured to :

```
io.streamthoughts.kafka.connect.filepulse.fs.clean.LogCleanupPolicy
```

### `LocalMoveCleanPolicy`
### `LocalMoveCleanupPolicy`

This policy attempts to move atomically files to configurable target directories.

To enable this policy, the property `fs.cleanup.policy.class` must configured to :
To enable this policy, the property `fs.cleanup.policy.class` must be configured to :

```
io.streamthoughts.kafka.connect.filepulse.fs.clean.LocalMoveCleanPolicy
io.streamthoughts.kafka.connect.filepulse.fs.clean.LocalMoveCleanupPolicy
```

{{% alert title="Usage" color="warning" %}}
Expand All @@ -54,9 +54,9 @@ This policy only works when using the `LocalFSDirectoryListing`.

#### Configuration

| Configuration | Description | Type | Default | Importance |
| --------------| --------------|-----------| --------- | ------------- |
|`cleaner.output.failed.path` | Target directory for file proceed with failure | string | *.failure* | high |
|`cleaner.output.succeed.path` | Target directory for file proceed successfully | string | *.success* | high |
| Configuration | Description | Type | Default | Importance |
|-------------------------------|------------------------------------------------|--------|------------|------------|
| `cleaner.output.failed.path` | Target directory for file proceed with failure | string | *.failure* | high |
| `cleaner.output.succeed.path` | Target directory for file proceed successfully | string | *.success* | high |

## Implementing your own policy
28 changes: 14 additions & 14 deletions docs/content/en/docs/Developer Guide/conditional-execution.md
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
date: 2020-09-30
date: 2022-03-02
title: "Conditional Execution"
linkTitle: "Conditional Execution"
weight: 60
Expand Down Expand Up @@ -29,19 +29,19 @@ filters.TagSecurityException.values=SecurityAlert

These conditional functions are available for use with the `if` config property :

| Function | Since | Description | Syntax |
| --------------| --------| -------|-----------|
| `and` | `2.4.0` | Checks if all of the given conditional expressions are `true`. | `{{ and(booleanExpression1, booleanExpression2, ...) }}` |
| `contains` | | Returns `true` if an array field's value contains the specified value | `{{ contains(field, value) }}` |
| `ends_with` | | Returns `true` if an a string field's value end with the specified string suffix | `{{ ends_with(field, suffix) }}` |
| `equals` | | Returns `true` if an a string or number fields's value equals the specified value | `{{ equals(field, value) }}` |
| `exists` | | Returns `true` if an the specified field exists | `{{ exists(struct, field) }}` |
| `gt` | `2.4.0` | Executes "*greater than operation*" on two values and returns `true` if the first value is greater than the second value, `false`, otherwise. | `{{ gt(expressionValue1, expressionValue2) }}` |
| `is_null` | | Returns `true` if a field's value is null | `{{ is_null(field) }}` |
| `lt` | `2.4.0` | Executes "*less than operation*" on two values and returns `true` if the first value is less than the second value, `false`, otherwise. | `{{ lt(expressionValue1, expressionValue2) }}` |
| `matches` | | Returns `true` if a field's value match the specified regex | `{{ matches(field, regex) }}` |
| `starts_with` | | Returns `true` if an a string field's value start with the specified string prefix | `{{ starts_with(field, prefix) }}` |
| `or` | `2.4.0` | Checks if at least one of the given conditional expressions is `true`.. | `{{ or(booleanExpression1, booleanExpression2, ...) }}` |
| Function | Since | Description | Syntax |
|---------------|---------|-----------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------|
| `and` | `2.4.0` | Checks if all of the given conditional expressions are `true`. | `{{ and(booleanExpression1, booleanExpression2, ...) }}` |
| `contains` | | Returns `true` if an array field's value contains the specified value | `{{ contains(field, value) }}` |
| `ends_with` | | Returns `true` if an a string field's value end with the specified string suffix | `{{ ends_with(field, suffix) }}` |
| `equals` | | Returns `true` if an a string or number fields's value equals the specified value | `{{ equals(field, value) }}` |
| `exists` | | Returns `true` if an the specified field exists | `{{ exists(struct, field) }}` |
| `gt` | `2.4.0` | Executes "*greater than operation*" on two values and returns `true` if the first value is greater than the second value, `false`, otherwise. | `{{ gt(expressionValue1, expressionValue2) }}` |
| `is_null` | | Returns `true` if a field's value is null | `{{ is_null(field) }}` |
| `lt` | `2.4.0` | Executes "*less than operation*" on two values and returns `true` if the first value is less than the second value, `false`, otherwise. | `{{ lt(expressionValue1, expressionValue2) }}` |
| `matches` | | Returns `true` if a field's value match the specified regex | `{{ matches(field, regex) }}` |
| `starts_with` | | Returns `true` if an a string field's value start with the specified string prefix | `{{ starts_with(field, prefix) }}` |
| `or` | `2.4.0` | Checks if at least one of the given conditional expressions is `true`.. | `{{ or(booleanExpression1, booleanExpression2, ...) }}` |

**Limitations** :
* conditions cannot be used to easily create pipeline branching.
10 changes: 5 additions & 5 deletions docs/content/en/docs/Developer Guide/file-readers.md
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
date: 2022-02-23
date: 2022-03-02
title: "File Readers"
linkTitle: "File Readers"
weight: 40
Expand Down Expand Up @@ -94,10 +94,10 @@ The `<PREFIX>XMLFileInputReader`s can be used to read XML files.
| `reader.xml.exclude.node.attributes.in.namespaces` | `2.4.0` | Specifies that the reader should only exclude node attributes in the defined list of namespaces. | `list` | `false` | Low |
| `reader.xml.data.type.inference.enabled` | `2.3.0` | Specifies that the reader should try to infer the type of data nodes. | `boolean` | `false` | High |
| `reader.xml.attribute.prefix` | `2.4.0` | If set, the name of attributes will be prepended with the specified prefix when they are added to a record. | `string` | `""` | Low |
| `reader.xml.content.field.name` | `2.4.4` | Specifies the name to be used for naming the field that will contain the value of a TextNode element having attributes. | `string` | `value` | Low |
| `reader.xml.field.name.characters.regex.pattern` | `2.4.4` | Specifies the regex pattern to use for matching the characters in XML element name to replace when converting a document to a struct. | `string` | `[.\-]'` | Low |
| `reader.xml.field.name.characters.string.replacement` | `2.4.4` | Specifies the replacement string to be used when converting a document to a struct. | `string` | `_` | Low |
| `reader.xml.force.content.field.for.paths` | `2.4.4` | The comma-separated list of field for which a content-field must be forced. | `List` | - | Low |
| `reader.xml.content.field.name` | `2.5.4` | Specifies the name to be used for naming the field that will contain the value of a TextNode element having attributes. | `string` | `value` | Low |
| `reader.xml.field.name.characters.regex.pattern` | `2.5.4` | Specifies the regex pattern to use for matching the characters in XML element name to replace when converting a document to a struct. | `string` | `[.\-]'` | Low |
| `reader.xml.field.name.characters.string.replacement` | `2.5.4` | Specifies the replacement string to be used when converting a document to a struct. | `string` | `_` | Low |
| `reader.xml.force.content.field.for.paths` | `2.5.4` | The comma-separated list of field for which a content-field must be forced. | `List` | - | Low |

## XxxMetadataFileInputReader

Expand Down
32 changes: 16 additions & 16 deletions docs/content/en/docs/Developer Guide/filters.md
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
date: 2022-21-02
date: 2022-03-02
title: "Processing Filters"
linkTitle: "Processing Filters"
weight: 80
Expand Down Expand Up @@ -780,21 +780,21 @@ The `XmlToStructFilter` parses an XML record-field into STRUCT

### Configuration

| Configuration | Description | Type | Default | Importance |
|-----------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------|-----------|-------------|------------|
| `source` | The input field on which to apply the filter. | string | `"message"` | high |
| `xml.force.array.on.fields` | The comma-separated list of fields for which an array-type must be forced | `List` | `-` | High |
| `xml.parser.validating.enabled` | Specifies that the parser will validate documents as they are parsed. | `boolean` | `false` | Low |
| `xml.parser.namespace.aware.enabled` | Specifies that the XML parser will provide support for XML namespaces. | `boolean` | `false` | Low |
| `xml.exclude.empty.elements` | Specifies that the reader should exclude element having no field. | `boolean` | `false` | Low |
| `xml.exclude.node.attributes` | Specifies that the reader should exclude all node attributes. | `boolean` | `false` | Low |
| `xml.exclude.node.attributes.in.namespaces` | Specifies that the reader should only exclude node attributes in the defined list of namespaces. | `list` | `false` | Low |
| `xml.data.type.inference.enabled` | Specifies that the reader should try to infer the type of data nodes. | `boolean` | `false` | High |
| `xml.attribute.prefix` | If set, the name of attributes will be prepended with the specified prefix when they are added to a record. | `string` | `""` | Low |
| `xml.content.field.name` | Specifies the name to be used for naming the field that will contain the value of a TextNode element having attributes. | `string` | `value` | Low |
| `xml.field.name.characters.regex.pattern` | Specifies the regex pattern to use for matching the characters in XML element name to replace when converting a document to a struct. | `string` | `[.\-]' | Low |
| `xml.field.name.character.string.replacement` | Specifies the replacement string to be used when converting a document to a struct. | `string` | `_` | Low |
| `xml.force.content.field.for.paths` | The comma-separated list of field for which a content-field must be forced. | `List` | - | Low |
| Configuration | Since | Description | Type | Default | Importance |
|-----------------------------------------------|-------|---------------------------------------------------------------------------------------------------------------------------------------|-----------|-------------|------------|
| `source` | | The input field on which to apply the filter. | string | `"message"` | High |
| `xml.force.array.on.fields` | | The comma-separated list of fields for which an array-type must be forced | `List` | | High |
| `xml.parser.validating.enabled` | | Specifies that the parser will validate documents as they are parsed. | `boolean` | `false` | Low |
| `xml.parser.namespace.aware.enabled` | | Specifies that the XML parser will provide support for XML namespaces. | `boolean` | `false` | Low |
| `xml.exclude.empty.elements` | | Specifies that the reader should exclude element having no field. | `boolean` | `false` | Low |
| `xml.exclude.node.attributes` | | Specifies that the reader should exclude all node attributes. | `boolean` | `false` | Low |
| `xml.exclude.node.attributes.in.namespaces` | | Specifies that the reader should only exclude node attributes in the defined list of namespaces. | `List` | `false` | Low |
| `xml.data.type.inference.enabled` | | Specifies that the reader should try to infer the type of data nodes. | `boolean` | `false` | High |
| `xml.attribute.prefix` | | If set, the name of attributes will be prepended with the specified prefix when they are added to a record. | `string` | `""` | Low |
| `xml.content.field.name` | 2.5.0 | Specifies the name to be used for naming the field that will contain the value of a TextNode element having attributes. | `string` | `value` | Low |
| `xml.field.name.characters.regex.pattern` | 2.5.0 | Specifies the regex pattern to use for matching the characters in XML element name to replace when converting a document to a struct. | `string` | `[.\-]' | Low |
| `xml.field.name.character.string.replacement` | 2.5.0 | Specifies the replacement string to be used when converting a document to a struct. | `string` | `_` | Low |
| `xml.force.content.field.for.paths` | 2.5.0 | The comma-separated list of field for which a content-field must be forced. | `List` | | Low |
### Example

**Configuration**
Expand Down
4 changes: 2 additions & 2 deletions docs/content/en/docs/Developer Guide/handling-failures.md
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
date: 2021-03-30
date: 2022-03-02
title: "Handling Failures"
linkTitle: "Handling Failures"
weight: 70
Expand All @@ -9,7 +9,7 @@ description: >

The connector provides some mechanisms to handle failures while executing filters.

By default, the filters chain will immediately failed after an exception is thrown.
By default, the filters chain will immediately fail after an exception is thrown.
But, you can also configure each filter to either ignore errors or to branch to a sub filters-chain.

## Configuration
Expand Down
Loading

0 comments on commit c5cb5a3

Please sign in to comment.