Skip to content
This repository has been archived by the owner on May 25, 2022. It is now read-only.

Kv parser #307

Closed
wants to merge 75 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
75 commits
Select commit Hold shift + click to select a range
bbc73f8
contribute key value parser from observiq
Nov 16, 2021
a6b45c2
document delimiter param
Nov 16, 2021
076fafd
make tidy
Nov 16, 2021
b87a269
Merge branch 'main' into kv-parser
Nov 16, 2021
8219978
rebase
Jan 21, 2022
eba64cd
tidy
Jan 21, 2022
22c34e2
Merge branch 'open-telemetry-main' into kv-parser
Jan 21, 2022
e00181d
tidy
Jan 21, 2022
8f00220
do not support bytes
Jan 21, 2022
5ee14c7
make pair delimiter a configuration option, default to whitespace
Jan 22, 2022
580ce6b
simplify tests and test new pair delimiter option
Jan 22, 2022
373286c
remove benchmark, not required
Jan 22, 2022
427d6ce
handle single quoted values
Jan 22, 2022
0857bf7
fail if delimiter and pair delimiter are identical
Jan 22, 2022
2c6ae2f
finish remaining test coverage
Jan 22, 2022
0b0f8bf
Give recombine timeout test a more margin for error (#355)
djaglowski Jan 22, 2022
45c9011
Bump github.com/onsi/gomega from 1.17.0 to 1.18.0 in /internal/tools …
dependabot[bot] Jan 24, 2022
7559b93
Change log level for missing file case (#357)
djaglowski Jan 26, 2022
2a8dbeb
Bump k8s.io/client-go from 0.23.2 to 0.23.3 (#358)
dependabot[bot] Jan 31, 2022
d386a36
Bump github.com/onsi/gomega from 1.18.0 to 1.18.1 in /internal/tools …
dependabot[bot] Jan 31, 2022
4cecafa
Bump go.opentelemetry.io/collector from 0.42.0 to 0.43.1 (#361)
dependabot[bot] Jan 31, 2022
f25b106
Bump github.com/golangci/golangci-lint in /internal/tools (#362)
dependabot[bot] Jan 31, 2022
f6a5ca2
Bump go.opentelemetry.io/collector from 0.43.1 to 0.44.0 (#365)
dependabot[bot] Feb 7, 2022
fcea3af
Remove agent.WithConfigFiles (#374)
djaglowski Feb 10, 2022
3df09ff
file_input should not attempt to track lost files on windows (#366)
djaglowski Feb 10, 2022
1f8bd51
Remove the Dockerfile (#381)
djaglowski Feb 13, 2022
44270c1
Remove version package (#382)
djaglowski Feb 13, 2022
c929bb9
Bump go.uber.org/zap from 1.20.0 to 1.21.0 (#384)
dependabot[bot] Feb 14, 2022
e039b7c
Remove .dockerignore file (#385)
djaglowski Feb 14, 2022
77b222e
Remove plugins (#377)
djaglowski Feb 14, 2022
a9f0162
Syslog input as wrapper (#376)
djaglowski Feb 15, 2022
fd0fe16
Make operator.Builder.Build return one operator (#386)
djaglowski Feb 15, 2022
48d725b
Remove 'stanza_input' operator and related logger wrapper (#389)
djaglowski Feb 16, 2022
43a325c
Remove notion of operator namespaces (#387)
djaglowski Feb 16, 2022
ee764be
Remove BuildContext (#393)
djaglowski Feb 16, 2022
1cb0229
Merge agent package into pipeline package (#395)
djaglowski Feb 18, 2022
a60a77d
Tidy up operator testdata files (#398)
djaglowski Feb 18, 2022
8453fbc
Remove 'builtin' directory layer in operator packages (#400)
djaglowski Feb 18, 2022
bc0b94b
Bump go.opentelemetry.io/collector from 0.44.0 to 0.45.0 (#407)
dependabot[bot] Feb 21, 2022
8cb8712
Update changelog in preparation for release v0.25.0 (#402)
djaglowski Feb 21, 2022
5e83f90
Update dependencies and related tooling (#409)
djaglowski Feb 22, 2022
b3e4572
change attributes (#401)
chaitanyaphalak Feb 22, 2022
78eb238
Remove write_to setting (#412)
djaglowski Feb 23, 2022
c118cd6
change resources (#411)
chaitanyaphalak Feb 23, 2022
312f08c
fix iteration (#413)
rockb1017 Feb 25, 2022
248eda3
Correctly parse timestamps from 1970 (#417)
djaglowski Feb 25, 2022
e99ae1f
Update changelog ahead of v0.26.0 release (#418)
djaglowski Feb 25, 2022
07cef82
Bump github.com/securego/gosec/v2 in /internal/tools (#419)
dependabot[bot] Mar 7, 2022
8bd9504
feat(operator/recombine): do not combine logs before first_entry matc…
sumo-drosiek Mar 8, 2022
6dace5c
Bump go.opentelemetry.io/collector from 0.45.0 to 0.46.0 (#422)
dependabot[bot] Mar 8, 2022
8064c5b
Update dependencies (#426)
djaglowski Mar 9, 2022
b3a8528
Add support for parsing multiline csv records (#425)
djaglowski Mar 10, 2022
13688f4
Update changelog ahead of v0.27.0 release (#427)
djaglowski Mar 10, 2022
48c8d7d
Update CHANGELOG.md (#432)
djaglowski Mar 10, 2022
e92a761
Remove unused tools (#435)
djaglowski Mar 15, 2022
547e79e
Remove notion of a default timestamp (#436)
djaglowski Mar 15, 2022
f054b8f
Bump gonum to resolve iteration issue in Go 1.18 (#437)
djaglowski Mar 16, 2022
3a4af2e
Revert version of syslog-go (#438)
djaglowski Mar 17, 2022
fd465dc
Bump github.com/stretchr/testify from 1.7.0 to 1.7.1 (#444)
dependabot[bot] Mar 21, 2022
a798d75
Bump github.com/golangci/golangci-lint in /internal/tools (#442)
dependabot[bot] Mar 21, 2022
b59ffe9
fix(helpers/multiline): fix force flushing with multiline (#434)
sumo-drosiek Mar 22, 2022
5f623db
Fix typo in recombine operator docs (#452)
djaglowski Mar 23, 2022
ac38b6d
Remove '$' from field syntax (#364)
djaglowski Mar 24, 2022
636ffc5
Update attribute names used by file_input, to match semantic conventi…
djaglowski Mar 24, 2022
5774206
Remove restructure operator (#371)
djaglowski Mar 24, 2022
28dd828
Remove metadata operator (#429)
djaglowski Mar 24, 2022
ecefbe3
Add ObservedTimestamp to entry (#370)
djaglowski Mar 24, 2022
7bf6948
Add prefixes to many examples in documentation (#453)
djaglowski Mar 24, 2022
90e31d9
Provide a dedicated mechanism for parsing logger name (#397)
djaglowski Mar 24, 2022
104d1b9
Enforce maximum SD-NAME length of 32, per RFC5424 (#439)
djaglowski Mar 24, 2022
b3803d7
Update Changelog ahead of major set of breaking changes (#430)
djaglowski Mar 28, 2022
97c3b23
tidy
Nov 16, 2021
7f8d2e7
newline
Mar 28, 2022
3af7827
remove default value for empty string. remove TODO, it is okay to rep…
Mar 28, 2022
585f1a0
doc feadback
Mar 28, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
4 changes: 0 additions & 4 deletions .dockerignore

This file was deleted.

86 changes: 86 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,92 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [0.28.0] - 2022-03-28

### Breaking Changes
- `$` has been removed from [field](/docs/types/field.md) syntax. ([PR364](https://github.com/open-telemetry/opentelemetry-log-collection/pull/364))
- Use `body` instead of `$body`.
- e.g. `body.foo`.
- Use `attributes` instead of `$attributes`.
- e.g. `attributes.["log.file.name"]`
- Use `resource` instead of `$resource`.
- e.g. `resource.["host.name"]`
- There is no longer a default top-level field.
- i.e. `foo` is no longer equivalent to `$body.foo`. (It is invalid.)
- A top-level field MUST be specified at the beginning of each field.
- e.g. `body.foo`, `attributes.foo`, or `resource.foo`.
- `entry.Entry.Timestamp` field is no longer required and is not initialized by default. ([PR370](https://github.com/open-telemetry/opentelemetry-log-collection/pull/370))
- The value can be set using the `timestamp` block on any parser, or the using the standalone `time_parser` operator.
- Removed `metadata` operator. ([PR429](https://github.com/open-telemetry/opentelemetry-log-collection/pull/429))
- Use `add`, `copy`, or `move` operators instead.
- Removed `restructure` operator. ([PR371](https://github.com/open-telemetry/opentelemetry-log-collection/pull/371))
- Use `add`, `copy`, `flatten`, `move`, `remove`, and `retain` operators instead.
- Changed the names of attributes added by `file_input` operator to match new [semantic conventions](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/semantic_conventions/media.md#log-file). ([PR372](https://github.com/open-telemetry/opentelemetry-log-collection/pull/372))
- Switch to original `go-syslog` library, restoring strict enforcement of SD-NAME length. ([PR439](https://github.com/open-telemetry/opentelemetry-log-collection/pull/439))


## [0.27.2] - 2022-03-17

### Fixed
- Revert version update on go-syslog, which introduced incompatibility with 386 architecture. ([PR438](https://github.com/open-telemetry/opentelemetry-log-collection/pull/438))


## [0.27.1] - 2022-03-16

### Fixed
- Issue where pipelines could fail to build when running on Go 1.18. ([PR347](https://github.com/open-telemetry/opentelemetry-log-collection/pull/347))


## [0.27.0] - 2022-03-10

### Added
- `entry.Entry.ObservedTimestamp` field. This is initialized automatically when an entry is created. ([PR370](https://github.com/open-telemetry/opentelemetry-log-collection/pull/370))
- `entry.Entry.ScopeName` field. This field will be used in the collector to group related log entries. ([PR397](https://github.com/open-telemetry/opentelemetry-log-collection/pull/397))
- A `scope_name` block is now supported by every parser. Alternately, use the standalone `scope_name_parser` operator.


## [0.27.0] - 2022-03-10

### Added
- `csv_parser` can now handle fields containing line breaks. ([PR425](https://github.com/open-telemetry/opentelemetry-log-collection/pull/425))

### Fixed
- Issue where `recombine` operator would combine entire file in certain specific circumstances. ([PR416](https://github.com/open-telemetry/opentelemetry-log-collection/pull/416))


## [0.26.0] - 2022-02-25

### Added
- `source_identifier` setting to `recombine` operator, to ensure partial entries are joined to others from the same file, or other source. ([PR341](https://github.com/open-telemetry/opentelemetry-log-collection/pull/341))
- `max_sources` setting to `recombine` operator, which limits the number of unique sources that may accumulate partial entries. ([PR341](https://github.com/open-telemetry/opentelemetry-log-collection/pull/341))

### Fixed
- Time parsing will now correctly parse timestamps from 1970. ([PR417](https://github.com/open-telemetry/opentelemetry-log-collection/pull/417))
- Issue where `file_input` operator could duplicate a small number of log entries. ([PR413](https://github.com/open-telemetry/opentelemetry-log-collection/pull/413))

### Changed
- `entry.Attributes` data type from `map[string]string` to `map[string]interface{}`. ([PR401](https://github.com/open-telemetry/opentelemetry-log-collection/pull/401))
- `entry.Resource` data type from `map[string]string` to `map[string]interface{}`. ([PR411](https://github.com/open-telemetry/opentelemetry-log-collection/pull/411))

### Removed
- `write_to` configuration setting from all input operators. Use `move` operator instead. ([PR412](https://github.com/open-telemetry/opentelemetry-log-collection/pull/412))


## [0.25.0] - 2022-02-21

This release contains a few minor updates as well as a major cleanup of the codebase. See the [Reduce Complexity](https://github.com/open-telemetry/opentelemetry-log-collection/milestone/2?closed=1) milestone for full details on the cleanup.

### Added
- `source_identifier` setting to `recombine` operator, to ensure partial entries are joined to others from the same file, or other source. ([PR341](https://github.com/open-telemetry/opentelemetry-log-collection/pull/341))
- `max_sources` setting to `recombine` operator, which limits the number of unique sources that may accumulate partial entries. ([PR341](https://github.com/open-telemetry/opentelemetry-log-collection/pull/341))

### Fixed
- On Windows, `file_input` will immediately close files after reading. ([PR366](https://github.com/open-telemetry/opentelemetry-log-collection/pull/366))

### Changed
- When `file_input` cannot open a file, it will print a debug level log instead of an error level log. ([PR357](https://github.com/open-telemetry/opentelemetry-log-collection/pull/357))


## [0.24.0] - 2021-12-21

### Added
Expand Down
14 changes: 0 additions & 14 deletions Dockerfile

This file was deleted.

12 changes: 9 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ ALL_MODULES := $(shell find . -type f -name "go.mod" -exec dirname {} \; | sort
ALL_SRC := $(shell find . -name '*.go' -type f | sort)
ADDLICENSE=addlicense
ALL_COVERAGE_MOD_DIRS := $(shell find . -type f -name 'go.mod' -exec dirname {} \; | egrep -v '^./internal/tools' | sort)
FIELDALIGNMENT_DIRS := ./agent/ ./pipeline/
FIELDALIGNMENT_DIRS := ./pipeline/

TOOLS_MOD_DIR := ./internal/tools
.PHONY: install-tools
Expand Down Expand Up @@ -51,8 +51,14 @@ clean:
.PHONY: tidy
tidy:
$(MAKE) for-all CMD="rm -fr go.sum"
$(MAKE) for-all CMD="go mod tidy -go=1.16"
$(MAKE) for-all CMD="go mod tidy -go=1.17"
$(MAKE) for-all CMD="go mod tidy -compat=1.17"

.PHONY: mod-update-only
mod-update-only:
$(MAKE) for-all CMD="go get -u ./..."

.PHONY: mod-update
mod-update: mod-update-only tidy

.PHONY: listmod
listmod:
Expand Down
54 changes: 0 additions & 54 deletions agent/agent.go

This file was deleted.

85 changes: 0 additions & 85 deletions agent/agent_test.go

This file was deleted.

111 changes: 0 additions & 111 deletions agent/builder.go

This file was deleted.

Loading