feat(logging): support structured logging functionality #6029
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
product-auto-label
bot
added
size: l
minherz
force-pushed
the
structured_logging
branch
3 times, most recently
from
934dd7e
to
d2e20e9
Compare
minherz
force-pushed
the
structured_logging
branch
from
04d8ac9
to
e60cf98
Compare
product-auto-label
bot
added
size: xl
minherz
force-pushed
the
structured_logging
branch
from
598bf4a
to
3412f8b
Compare
minherz
force-pushed
the
structured_logging
branch
from
8743a2f
to
16f49dd
Compare
product-auto-label
bot
added
the
stale: old
minherz
force-pushed
the
structured_logging
branch
from
ac3e517
to
88fb827
Compare
product-auto-label
bot
added
the
stale: old
losalex
approved these changes
Jun 23, 2022
quartzmo
reviewed
Jun 24, 2022
…6022) align resource detection with heuristics in logging libraries for other languages: - use /sys/class/dmi/id/product_name to read a product name of the resource on Linux - change resource detection heuristics in a way that the order of validation is not important - reduce timeout and retrying logic when querying metadata server - demand active metadata server for detecting GCP resources (for GCE, GAE, GKE, CR and CF) - add test to validate resource detection heuristics
…on (#6043) add source location population for ingested log entries. provide designated logger option: `SourceLocationPopulation` to control source population logic. The option can be initiated with the following values: * `SourceLocationPopulation(DoNotPopulateSourceLocation)` - continue with existing behavior when the source location field of the log entry is not populated automatically (default) * `SourceLocationPopulation(PopulateSourceLocationForDebugEntries)` - the source location is auto-populated for log entries with Debug severity * `SourceLocationPopulation(DoNotPopulateSourceLocation)` - the source location is auto-populated for all log entries
Adding a benchmark test to measure execution time for detectResource() to address #5855. Current implementation uses cloud.google.com/go/compute/metadata package to read Google Cloud metadata server attributes. The package implements retrying mechanism to cover on occasional problems of miscommunication with the metadata server. When a new Logger instance is being created, it calls detectResource() to read the monitored resource object. The resource detection algorithm is invoked once per process (regardless of how many Logger instances is created). The algorithm checks whether the environment runs the metadata server as a first step and to avoid further validations. Due to the retrying mechanism, this call may take between 6 to 7 seconds which composes the overall delay of creating a first instance of the Logger. The benchmark demonstrates that only first Logger takes long time: ```bash go test -run=XXX -bench=BenchmarkDetectResource -benchtime 8s goos: linux goarch: amd64 pkg: cloud.google.com/go/logging cpu: 11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz BenchmarkDetectResource-8 1000000000 2.291 ns/op PASS ok cloud.google.com/go/logging 9.955s ``` When the cloud.google.com/go/compute/metadata package will be modified to allow customizable retrying mechanism, this logic will be altered to minimize the number of retries for the initial call.
add support for W3C trace context header: traceparent
implement writing log entries to io.writer instead of ingesting them to Logging service. the written log entries are Jsonified into one line strings following structured logging format. test formatting of log entries for different types of payload. test redirection does not ingest log entries. support protobuf payloads when passing the payload as protobuf.Any. document all available LoggerOption options. add sample of synchronous log writing.
remove example with printing to stderr because the output to stderr cannot be tested.
clarify description of the RedirectAsJSON LoggerOption. add sample of logging protobuf payload.
Capture usage of the package per process by ingesting instrumentation log entry in each process that ingest at least one log entry. The instrumentation log entry will be ingested using the same Logger as the first log entry. If it will be ingested directly using Logging API, the instrumentation entry's Log ID will be set to `diagnostic-log`.
minherz
force-pushed
the
structured_logging
branch
from
73e662f
to
a3e8747
Compare
This was referenced Jun 25, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Align resource detection with heuristics in logging libraries for other languages (PR #6022)
Adds automatic population of the source location for ingested log entries with severity = Debug (PR #6043)
Adds benchmark for detectResource() to validate the long delays of the logger creation in non-GCP environments (PR #6084)
Adds PartialSuccess logger option to set the partialSuccess flag in all write log entries calls to true (PR #6092)
Adds support for W3C trace context header:
traceparentin addition to X-Cloud-Trace-Context (PR #6101)Adds RedirectToStderr and RedirectToStdout logger options to enable printing Jsonified log entries to STDOUT or STDERR instead of ingesting them directly to Cloud Logging (PR #6130)
Adds sending instrumentation info about the package on each use (one time per process) (PR #6210)
Fixes #5829
Fixes #5831
Fixes #5830
Fixes #3824
Fixes #5855
Fixes #6087
Fixes #6086
Fixes #6131
Fixes #6132
BEGIN_COMMIT_OVERRIDE
feat(logging): support structured logging functionality
feat: more robust and quick resource detection heuristics
feat: opted-in population of source location for ingested logs
feat: allow partial success for batched logs ingestions
feat: support W3C trace context header when reading tracing info from http.Request
feat: support out-of-process ingestion by writing Jsonified logs into stdout or stderr
END_COMMIT_OVERRIDE