Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OTLP Logs - remove depth from scope fields #3843

Merged

Conversation

cijothomas
Copy link
Member

Fixes #.

Changes

Please provide a brief description of the changes here.

For significant contributions please make sure you have completed the following items:

  • Appropriate CHANGELOG.md updated for non-trivial changes
  • Design discussion issue #
  • Changes in public API reviewed

@codecov
Copy link

codecov bot commented Oct 29, 2022

Codecov Report

Merging #3843 (a95575a) into main (c154931) will increase coverage by 0.07%.
The diff coverage is 0.00%.

❗ Current head a95575a differs from pull request most recent head 9e40df7. Consider uploading reports for the commit 9e40df7 to get more accurate results

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #3843      +/-   ##
==========================================
+ Coverage   87.45%   87.52%   +0.07%     
==========================================
  Files         280      280              
  Lines       10765    10763       -2     
==========================================
+ Hits         9414     9420       +6     
+ Misses       1351     1343       -8     
Impacted Files Coverage Δ
...etryProtocol/Implementation/LogRecordExtensions.cs 89.15% <0.00%> (+0.92%) ⬆️
...Telemetry/Internal/SelfDiagnosticsEventListener.cs 97.65% <0.00%> (+0.78%) ⬆️
...lementation/SqlClientInstrumentationEventSource.cs 75.00% <0.00%> (+4.16%) ⬆️
...tation/OpenTelemetryProtocolExporterEventSource.cs 95.00% <0.00%> (+10.00%) ⬆️
...porter.OpenTelemetryProtocol/OtlpMetricExporter.cs 86.36% <0.00%> (+13.63%) ⬆️

@cijothomas cijothomas marked this pull request as ready for review October 29, 2022 01:15
@cijothomas cijothomas requested a review from a team October 29, 2022 01:15
@@ -145,18 +146,40 @@ internal static OtlpLogs.LogRecord ToOtlpLog(this LogRecord logRecord, SdkLimitO
otlpLogRecord.Flags = (uint)logRecord.TraceFlags;
}

int scopeDepth = -1;
logRecord.ForEachScope(ProcessScope, otlpLogRecord);

void ProcessScope(LogRecordScope scope, OtlpLogs.LogRecord otlpLog)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit:

  • This can be made static void ProcessScope(LogRecordScope scope, OtlpLogs.LogRecord otlpLog) now that it isn't using a closure. On newer runtimes the delegate will be created the first time it is accessed and then reused.

  • For best perf, make a static field for the delegate and pass it in to logRecord.ForEachScope. That will ensure for all runtimes it is reused and doesn't need an existence check.

Comment on lines +177 to +179
else
{
if (OtlpKeyValueTransformer.Instance.TryTransformTag(scopeItem, out var result, attributeValueLengthLimit))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Looks like you could combine these into a single else if

Copy link
Member

@CodeBlanch CodeBlanch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couple nits, but LGTM.

Comment on lines +173 to +175
// Note: It is possible that we allow users
// to override this exporter feature. So not blocking
// empty/{OriginalFormat} in the SDK itself.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you suggesting that in the future we might add an exporter option to override this behavior?

My preference would be to try and avoid this solution if it becomes a need in the future. Given we have ILogger specific notions into our LogRecord data model we're required to handle them somehow in our exporters. But, I'd prefer to avoid spilling ILogger specific notions into our OTLP exporter options.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

deduping is not going to be Ilogger specific. It'll be same for any source which itself prevent duplicates.
For Activity, we do not produce duplicates, so if we want to have that similar behavior, we need to expose something.
(but it could be in the SDK or Exporter...)

Comment on lines +5 to +7
* Log Exporter modified to no longer prefix scope-depth when exporting ILogger
scopes as attributes. Empty keys and {OriginalFormat} key will be ignored from
scopes.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not opposed to this change, but can you give a little more background on why we're making this change?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Main motivation is the duplication/de-duplication cost, and undesired modification of user attributes.
Explanation for each of the case:

  1. With plain string, the key is null/empty, which is unallowed in OTel. But even if we were to use some magic key like "scope", it'll be duplicated, when user have nested scopes. The .NET official docs are being fixed already to no longer showing scopes with strings.

  2. With {OriginalFormat} - this is an unintentional side effect in Ms.Ext.Logging. It makes sense for Log message, where there is a Format() function, which can be used to get the fully formatted string. There is nothing like that for Scopes. "{OriginalFormat}" key will be duplicated if user has nested scopes, and we do not want to pay for dedup efforts.

The current approach simply prefixed a numeric value (based on scope depth), to solve this problem. But overriding user provide attributes with something else is undesirable (and affects perf too).

The following example can show why this is bad:

using (logger.BeginScope(new List<KeyValuePair<string, object>>
{
new KeyValuePair<string, object>("userid", "cijo")
}))
using (logger.BeginScope(new List<KeyValuePair<string, object>>
{
new KeyValuePair<string, object>("transactionid", "abcd")
}))
{
logger.LogInformation("Hello from {food} {price}.", "artichoke", 3.99);
}

the above would have resulted in attributes ("[0]:userid", "cijo"), ("[1]:transactionid", "abcd") as per current code, instead of ("userid", "cijo"). ("transactionid", "abcd"), per this PR.

@cijothomas cijothomas merged commit d4cca85 into open-telemetry:main Oct 31, 2022
@cijothomas cijothomas deleted the cijothomas/otlp-log-scope-fix branch October 31, 2022 23:30
// key is going to be if user simply
// passes a string as scope.
// To summarize this exporter only allows
// IReadOnlyList<KeyValuePair<string, object?>>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just mention IEnumerable<KeyValuePair<string, object?>>?

IReadOnlyList<KeyValuePair<string, object?>> implements IEnumerable<KeyValuePair<string, object?>> anyway.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants