TraceQL: Error when searching for span attributes that start with `resource.` #2605

LasseHels · 2023-07-03T08:25:29Z

Describe the bug
When executing this query:

{ resource.service.name = "scribe" && span.resource.id = "/subscriptions/05828da8-9a93-4b5c-ae81-11e62e5c8024/resourceGroups/rg-eu-west-iom-pp/providers/Microsoft.DBforPostgreSQL/flexibleServers/iom-db-pp" }

We get this error:

invalid TraceQL query: parse error at line 0, col 44: syntax error: unexpected resource., expecting IDENTIFIER

To Reproduce
Steps to reproduce the behavior:

Start Tempo. We are running grafana/tempo:2.1.1; other versions may be affected.
Ingest a span with a span attribute called resource.id. The value of the attribute (probably) doesn't matter.
Attempt to search for the span: { span.resource.id = "" }

Expected behavior
No error is emitted.

Observed behaviour
An error is emitted:

Environment:

Infrastructure: We run Tempo in a large Kubernetes cluster on AKS.
Deployment tool: Our cluster is continually reconciled with Flux. We use Tempo's jsonnet templates to generate the YAML files that make up our cluster (or at least, the Tempo part of it).

Additional Context
I believe this issue is isolated to span attributes prefixed with resource.. I'm guessing this kind of span attribute name tricks the TraceQL parser into believing that we are trying to search for a resource attribute.

This query works:

{ resource.service.name = "scribe" && span.identifier = "" }

This query fails:

{ resource.service.name = "scribe" && span.resource.id = "" }

Our application only emits a resource.id span attribute, so I've not tested with other resource.* attributes, but I would assume that the error is the same.

This bug is not critical to us; we are likely going to rename the span attribute to resolve the issue. In hindsight, naming a span attribute resource.id was asking for trouble and not my brighest move 😅.

The text was updated successfully, but these errors were encountered:

joe-elliott · 2023-07-05T13:01:27Z

Thanks for the report. This is also failing with a similar issue:

{ resource.span.foo = "bar" }

This seems to work but it is not as performant:

{ .resource.id = "" }

joe-elliott · 2023-07-05T15:12:29Z

The below diff fixes the issue as stated but still fails on weird constructions like:

{ parent.span.resource.id = "" }

I think this is just going to require some better attribute scope parsing.

diff --git a/pkg/traceql/lexer.go b/pkg/traceql/lexer.go
index 830c30912..19266c75d 100644
--- a/pkg/traceql/lexer.go
+++ b/pkg/traceql/lexer.go
@@ -77,15 +77,15 @@ type lexer struct {
        parser *yyParserImpl
        errs   []ParseError
 
-       parsingAttribute bool
+       currentAttribute int
 }
 
 func (l *lexer) Lex(lval *yySymType) int {
        // if we are currently parsing an attribute and the next rune suggests that
        //  this attribute will end, then return a special token indicating that the attribute is
        //  done parsing
-       if l.parsingAttribute && !isAttributeRune(l.Peek()) {
-               l.parsingAttribute = false
+       if l.parsingAttribute() && !isAttributeRune(l.Peek()) {
+               l.resetAttribute()
                return END_ATTRIBUTE
        }
 
@@ -93,13 +93,16 @@ func (l *lexer) Lex(lval *yySymType) int {
 
        // if we are currently parsing an attribute then just grab everything until we find a character that ends the attribute.
        // we will handle parsing this out in ast.go
-       if l.parsingAttribute {
+       if l.parsingAttribute() {
                str := l.TokenText()
-               // parse out any scopes here
-               tok := tokens[str+string(l.Peek())]
-               if tok == RESOURCE_DOT || tok == SPAN_DOT {
-                       l.Next()
-                       return tok
+
+               // parse out any scopes here, but only attempt if we are parsing a parent attribute
+               if l.currentAttribute == PARENT_DOT {
+                       tok := tokens[str+string(l.Peek())]
+                       if tok == RESOURCE_DOT || tok == SPAN_DOT {
+                               l.Next()
+                               return tok
+                       }
                }
 
                // go forward until we find the end of the attribute
@@ -168,12 +171,12 @@ func (l *lexer) Lex(lval *yySymType) int {
        tokStrNext := l.TokenText() + string(l.Peek())
        if tok, ok := tokens[tokStrNext]; ok {
                l.Next()
-               l.parsingAttribute = startsAttribute(tok)
+               l.setAttribute(tok)
                return tok
        }
 
        if tok, ok := tokens[l.TokenText()]; ok {
-               l.parsingAttribute = startsAttribute(tok)
+               l.setAttribute(tok)
                return tok
        }
 
@@ -185,6 +188,20 @@ func (l *lexer) Error(msg string) {
        l.errs = append(l.errs, newParseError(msg, l.Line, l.Column))
 }
 
+func (l *lexer) parsingAttribute() bool {
+       return l.currentAttribute != 0
+}
+
+func (l *lexer) setAttribute(tok int) {
+       if startsAttribute(tok) {
+               l.currentAttribute = tok
+       }
+}
+
+func (l *lexer) resetAttribute() {
+       l.currentAttribute = 0
+}
+
 func tryScanDuration(number string, l *scanner.Scanner) (time.Duration, bool) {
        var sb strings.Builder
        sb.WriteString(number)
diff --git a/pkg/traceql/lexer_test.go b/pkg/traceql/lexer_test.go
index 401fa1229..8d1dd3d78 100644
--- a/pkg/traceql/lexer_test.go
+++ b/pkg/traceql/lexer_test.go
@@ -34,12 +34,14 @@ func TestLexerAttributes(t *testing.T) {
                {`span.foo3`, []int{SPAN_DOT, IDENTIFIER, END_ATTRIBUTE}},
                {`span.foo+bar`, []int{SPAN_DOT, IDENTIFIER, END_ATTRIBUTE}},
                {`span.foo-bar`, []int{SPAN_DOT, IDENTIFIER, END_ATTRIBUTE}},
+               {`span.resource.foo`, []int{SPAN_DOT, IDENTIFIER, END_ATTRIBUTE}},
                // resource attributes
                {`resource.foo`, []int{RESOURCE_DOT, IDENTIFIER, END_ATTRIBUTE}},
                {`resource.count`, []int{RESOURCE_DOT, IDENTIFIER, END_ATTRIBUTE}},
                {`resource.foo3`, []int{RESOURCE_DOT, IDENTIFIER, END_ATTRIBUTE}},
                {`resource.foo+bar`, []int{RESOURCE_DOT, IDENTIFIER, END_ATTRIBUTE}},
                {`resource.foo-bar`, []int{RESOURCE_DOT, IDENTIFIER, END_ATTRIBUTE}},
+               {`resource.span.foo`, []int{RESOURCE_DOT, IDENTIFIER, END_ATTRIBUTE}},
                // parent span attributes
                {`parent.span.foo`, []int{PARENT_DOT, SPAN_DOT, IDENTIFIER, END_ATTRIBUTE}},
                {`parent.span.count`, []int{PARENT_DOT, SPAN_DOT, IDENTIFIER, END_ATTRIBUTE}},

github-actions · 2023-09-04T00:03:16Z

This issue has been automatically marked as stale because it has not had any activity in the past 60 days.
The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity.
Please apply keepalive label to exempt this Issue.

mghildiy · 2023-12-17T03:33:20Z

I am working on it.

mghildiy · 2023-12-23T13:12:58Z

To confirm my understanding:

span.xxx.yyy represents a span attribute xxx.yyy, and when we have a span attribute starting with resource. (like, span.resource.foo), this problem arises.

mghildiy · 2023-12-23T14:04:25Z

tok := tokens[str+string(l.Peek())]

The below diff fixes the issue as stated but still fails on weird constructions like:

{ parent.span.resource.id = "" }

I think this is just going to require some better attribute scope parsing.

diff --git a/pkg/traceql/lexer.go b/pkg/traceql/lexer.go
index 830c30912..19266c75d 100644
--- a/pkg/traceql/lexer.go
+++ b/pkg/traceql/lexer.go
@@ -77,15 +77,15 @@ type lexer struct {
        parser *yyParserImpl
        errs   []ParseError
 
-       parsingAttribute bool
+       currentAttribute int
 }
 
 func (l *lexer) Lex(lval *yySymType) int {
        // if we are currently parsing an attribute and the next rune suggests that
        //  this attribute will end, then return a special token indicating that the attribute is
        //  done parsing
-       if l.parsingAttribute && !isAttributeRune(l.Peek()) {
-               l.parsingAttribute = false
+       if l.parsingAttribute() && !isAttributeRune(l.Peek()) {
+               l.resetAttribute()
                return END_ATTRIBUTE
        }
 
@@ -93,13 +93,16 @@ func (l *lexer) Lex(lval *yySymType) int {
 
        // if we are currently parsing an attribute then just grab everything until we find a character that ends the attribute.
        // we will handle parsing this out in ast.go
-       if l.parsingAttribute {
+       if l.parsingAttribute() {
                str := l.TokenText()
-               // parse out any scopes here
-               tok := tokens[str+string(l.Peek())]
-               if tok == RESOURCE_DOT || tok == SPAN_DOT {
-                       l.Next()
-                       return tok
+
+               // parse out any scopes here, but only attempt if we are parsing a parent attribute
+               if l.currentAttribute == PARENT_DOT {
+                       tok := tokens[str+string(l.Peek())]
+                       if tok == RESOURCE_DOT || tok == SPAN_DOT {
+                               l.Next()
+                               return tok
+                       }
                }
 
                // go forward until we find the end of the attribute
@@ -168,12 +171,12 @@ func (l *lexer) Lex(lval *yySymType) int {
        tokStrNext := l.TokenText() + string(l.Peek())
        if tok, ok := tokens[tokStrNext]; ok {
                l.Next()
-               l.parsingAttribute = startsAttribute(tok)
+               l.setAttribute(tok)
                return tok
        }
 
        if tok, ok := tokens[l.TokenText()]; ok {
-               l.parsingAttribute = startsAttribute(tok)
+               l.setAttribute(tok)
                return tok
        }
 
@@ -185,6 +188,20 @@ func (l *lexer) Error(msg string) {
        l.errs = append(l.errs, newParseError(msg, l.Line, l.Column))
 }
 
+func (l *lexer) parsingAttribute() bool {
+       return l.currentAttribute != 0
+}
+
+func (l *lexer) setAttribute(tok int) {
+       if startsAttribute(tok) {
+               l.currentAttribute = tok
+       }
+}
+
+func (l *lexer) resetAttribute() {
+       l.currentAttribute = 0
+}
+
 func tryScanDuration(number string, l *scanner.Scanner) (time.Duration, bool) {
        var sb strings.Builder
        sb.WriteString(number)
diff --git a/pkg/traceql/lexer_test.go b/pkg/traceql/lexer_test.go
index 401fa1229..8d1dd3d78 100644
--- a/pkg/traceql/lexer_test.go
+++ b/pkg/traceql/lexer_test.go
@@ -34,12 +34,14 @@ func TestLexerAttributes(t *testing.T) {
                {`span.foo3`, []int{SPAN_DOT, IDENTIFIER, END_ATTRIBUTE}},
                {`span.foo+bar`, []int{SPAN_DOT, IDENTIFIER, END_ATTRIBUTE}},
                {`span.foo-bar`, []int{SPAN_DOT, IDENTIFIER, END_ATTRIBUTE}},
+               {`span.resource.foo`, []int{SPAN_DOT, IDENTIFIER, END_ATTRIBUTE}},
                // resource attributes
                {`resource.foo`, []int{RESOURCE_DOT, IDENTIFIER, END_ATTRIBUTE}},
                {`resource.count`, []int{RESOURCE_DOT, IDENTIFIER, END_ATTRIBUTE}},
                {`resource.foo3`, []int{RESOURCE_DOT, IDENTIFIER, END_ATTRIBUTE}},
                {`resource.foo+bar`, []int{RESOURCE_DOT, IDENTIFIER, END_ATTRIBUTE}},
                {`resource.foo-bar`, []int{RESOURCE_DOT, IDENTIFIER, END_ATTRIBUTE}},
+               {`resource.span.foo`, []int{RESOURCE_DOT, IDENTIFIER, END_ATTRIBUTE}},
                // parent span attributes
                {`parent.span.foo`, []int{PARENT_DOT, SPAN_DOT, IDENTIFIER, END_ATTRIBUTE}},
                {`parent.span.count`, []int{PARENT_DOT, SPAN_DOT, IDENTIFIER, END_ATTRIBUTE}},

I pulled latest code, and it seems code has undergone changes since this set of change.

mapno · 2024-01-02T12:47:29Z

To confirm my understanding:

span.xxx.yyy represents a span attribute xxx.yyy, and when we have a span attribute starting with resource. (like, span.resource.foo), this problem arises.

Exactly this, yes!

mghildiy · 2024-01-06T10:26:09Z

parent.resource.xxx.
I understand above one is a wrong combination.

We can have following though:
parent.span.xxx
parent.span.resource.xxx

Correct me if I am wrong.

joe-elliott · 2024-01-06T15:20:45Z

technically parent.resource.xxx is allowed just like parent.span.xxx. We don't actually support this scope in code though so it's ok if this PR doesn't fix that.

mghildiy · 2024-01-08T14:59:49Z

I think we don't need check for resource scope,
And only resolving span scope when starting scope is parent.
This has made all tests pass.

mghildiy · 2024-01-09T13:59:38Z

What is expected for:
parent.span.resource.id

PARENT_DOT, SPAN_DOT, IDENTIFIER, END_ATTRIBUTE
OR
PARENT_DOT, SPAN_DOT, RESOURCE_DOT, ATTRIBUTE, END_ATTRIBUTE

joe-elliott · 2024-01-09T17:55:41Z

According to the spec this would be most correct:

PARENT_DOT, SPAN_DOT, IDENTIFIER, END_ATTRIBUTE

mghildiy · 2024-01-11T13:28:30Z

PR

joe-elliott added type/bug Something isn't working traceql labels Jul 5, 2023

github-actions bot added the stale Used for stale issues / PRs label Sep 4, 2023

joe-elliott added keepalive Label to exempt Issues / PRs from stale workflow and removed stale Used for stale issues / PRs labels Sep 5, 2023

mapno assigned mghildiy Dec 18, 2023

This was referenced Jan 11, 2024

Fix-2605:Making span.resource.xyz... work #3282

Closed

FIX-2605: Make span.resource.xyz attributes work #3284

Merged

joe-elliott closed this as completed in #3284 Jan 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TraceQL: Error when searching for span attributes that start with `resource.` #2605

TraceQL: Error when searching for span attributes that start with `resource.` #2605

LasseHels commented Jul 3, 2023 •

edited

Loading

joe-elliott commented Jul 5, 2023

joe-elliott commented Jul 5, 2023

github-actions bot commented Sep 4, 2023

mghildiy commented Dec 17, 2023

mghildiy commented Dec 23, 2023

mghildiy commented Dec 23, 2023

mapno commented Jan 2, 2024

mghildiy commented Jan 6, 2024

joe-elliott commented Jan 6, 2024

mghildiy commented Jan 8, 2024

mghildiy commented Jan 9, 2024

joe-elliott commented Jan 9, 2024

mghildiy commented Jan 11, 2024

TraceQL: Error when searching for span attributes that start with resource. #2605

TraceQL: Error when searching for span attributes that start with resource. #2605

Comments

LasseHels commented Jul 3, 2023 • edited Loading

joe-elliott commented Jul 5, 2023

joe-elliott commented Jul 5, 2023

github-actions bot commented Sep 4, 2023

mghildiy commented Dec 17, 2023

mghildiy commented Dec 23, 2023

mghildiy commented Dec 23, 2023

mapno commented Jan 2, 2024

mghildiy commented Jan 6, 2024

joe-elliott commented Jan 6, 2024

mghildiy commented Jan 8, 2024

mghildiy commented Jan 9, 2024

joe-elliott commented Jan 9, 2024

mghildiy commented Jan 11, 2024

TraceQL: Error when searching for span attributes that start with `resource.` #2605

TraceQL: Error when searching for span attributes that start with `resource.` #2605

LasseHels commented Jul 3, 2023 •

edited

Loading