Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

error decode response: invalid character 'x' in string escape code #24

Closed
Proxymiity opened this issue Jun 11, 2024 · 20 comments
Closed
Assignees
Labels
bug Something isn't working

Comments

@Proxymiity
Copy link

Hi,

When using the latest version of VictoriaLogs (0.18.0 as of today, after upgrading from 0.7.0), the datasource stops working.
A simple _time:5m or host:test-server returns a decoding error in the panel: error decode response: invalid character 'x' in string escape code

No specific error logged in the VictoriaLogs server itself, but the Grafana log has some errors:

GRAFANA  | logger=plugin.victorialogs-datasource t=2024-06-11T12:50:04.627607298Z level=error msg="error decode response: invalid character 'x' in string escape code"
GRAFANA  | logger=context userId=2 orgId=1 uname=*** t=2024-06-11T12:50:04.627862706Z level=info msg="Request Completed" method=POST path=/api/ds/query status=400 remote_addr=*** time_ms=105 duration=105.622792ms size=139 referer="http://***/d/bdoghm58zwdmof/test-panel?editPanel=2&orgId=1" handler=/api/ds/query status_source=downstream

The query itself works fine in the vmui. Datasource is on 0.2.1, and when added replies with "This datasource is working".
Rolling back to our known working vlogs version (0.7.0) works well, however we lose new features of the query language.

@rkaracic
Copy link

rkaracic commented Jun 17, 2024

Hi,

I'm having same issue as you.
Latest working version for us is 0.15, DS is on latest version (0.2.1).

Maybe this additional info will help someone:
E.g. query "instance:hostname" will not work, but if I query it like "hostname AND instance:hostname" this will work. It's ugly workaround, but maybe someone will find it useful.

@hagen1778 hagen1778 added the bug Something isn't working label Jun 24, 2024
@hagen1778
Copy link
Contributor

@dmitryk-dk is this related to the backend of the plugin?

@dmitryk-dk
Copy link
Contributor

Hi @Proxymiity ! Can you share what version of the VictoriaLogs you have? Can you share an example of your response from the VMUI you have?
This information can help us to detect the problem

@Proxymiity
Copy link
Author

Hi,

Sure. The version that does not work is 0.18.0, but the same thing happens with 0.23.0. Here's a screenshot of the VMUI with the request:
image

And here's what's in the Grafana panel request:
image

The Grafana variable is defined as follows:
image

(replacing the Grafana $host variable with the actual host:xxxxxx-test does not change a thing)

@Proxymiity
Copy link
Author

Proxymiity commented Jun 26, 2024

FYI, I have a working panel with that version. Some are returning "no data" because there is no data in the timerange (which is expected), one is returning data correctly, and some are erroring with the same error you'd find within the first issue comment

@dmitryk-dk
Copy link
Contributor

dmitryk-dk commented Jun 26, 2024

FYI, I have a working panel with that version. Some are returning "no data" because there is no data in the timerange (which is expected), one is returning data correctly, and some are erroring with the same error you'd find within the first issue comment

It would be helpful if you could share the response in JSON and remove sensitive data when you get that error.
I will try to reproduce the issue. There are used standard JSON decoders, and I think the problem is with parsing response from the victoria logs.
But it is only my assumption

@Proxymiity
Copy link
Author

It would be helpful if you could share the response in JSON and remove sensitive data when you get that error. I will try to reproduce the issue. There are used standard JSON decoders, and I think the problem is with parsing response from the victoria logs. But it is only my assumption

By response, do you mean intercept the response from VLogs back to Grafana?

@dmitryk-dk
Copy link
Contributor

It would be helpful if you could share the response in JSON and remove sensitive data when you get that error. I will try to reproduce the issue. There are used standard JSON decoders, and I think the problem is with parsing response from the victoria logs. But it is only my assumption

By response, do you mean intercept the response from VLogs back to Grafana?

Yes, or response from vmlogs to vmui. They should be identical.

@Proxymiity
Copy link
Author

Proxymiity commented Jun 27, 2024

Here are the responses from both VictoriaLogs versions (mind the dot at the end of the url):
0.7.0: https://m.iya.at/hLfWw.
0.21.0: https://m.iya.at/hHZ76.

@dmitryk-dk
Copy link
Contributor

Here are the responses from both VictoriaLogs versions (mind the dot at the end of the url): 0.7.0: https://m.iya.at/hLfWw. 0.21.0: https://m.iya.at/hHZ76.

Thank you for providing examples. I will check it soon

@dmitryk-dk
Copy link
Contributor

Hi @Proxymiity , I think I found the problem and made a draft fix; if you can build a source from my PR and test it, that would be great. I will check the fix on my own

@Proxymiity
Copy link
Author

Hi @Proxymiity , I think I found the problem and made a draft fix; if you can build a source from my PR and test it, that would be great. I will check the fix on my own

Hi, thanks for the quick draft. I will look into building the plugin and will keep you informed.

@Proxymiity
Copy link
Author

Hi, just compiled and tested your fix. It does not error out anymore, but instead the color has vanished.

Here's vlogs 0.27.0 and your patch:
image

Here's vlogs 0.7.0 and your patch:
image

@dmitryk-dk
Copy link
Contributor

dmitryk-dk commented Jul 4, 2024

@Proxymiity, the logs sent from vlogs only proxied to the UI part of the data source, I will think about how it is better to handle those values with colors. I think it should be handle on the UI part of the plugin @Loori-R , @hagen1778 wdyt?

@Proxymiity, can you also share the Grafana version that you are using?

valyala added a commit to VictoriaMetrics/VictoriaMetrics that referenced this issue Jul 4, 2024
….AppendQuote for encoding JSON strings

The strconv.AppendQuote improperly encodes special chars such as \x1b . They must be encoded as \u001b .

See VictoriaMetrics/victorialogs-datasource#24
valyala added a commit to VictoriaMetrics/VictoriaMetrics that referenced this issue Jul 4, 2024
….AppendQuote for encoding JSON strings

The strconv.AppendQuote improperly encodes special chars such as \x1b . They must be encoded as \u001b .

See VictoriaMetrics/victorialogs-datasource#24
@valyala
Copy link

valyala commented Jul 5, 2024

This issue must be fixed in VictoriaLogs v0.27.1. The bugfix is at VictoriaMetrics/VictoriaMetrics@c0caa69 .

@Proxymiity , @rkaracic , could you upgrade VictoriaLogs to v0.27.1 and verify whether the issue is fixed there?

@clrxbl
Copy link

clrxbl commented Jul 5, 2024

This issue must be fixed in VictoriaLogs v0.27.1. The bugfix is at VictoriaMetrics/VictoriaMetrics@c0caa69 .

@Proxymiity , @rkaracic , could you upgrade VictoriaLogs to v0.27.1 and verify whether the issue is fixed there?

I've had this issue aswell but after upgrading to 0.27.1, it went away.

@rkaracic
Copy link

rkaracic commented Jul 5, 2024

I can confirm that issue is gone in 0.27.1

hagen1778 added a commit that referenced this issue Jul 5, 2024
fix issue when response contains ANSI escape sequences
#24
@hagen1778
Copy link
Contributor

The problem was fixed on the backend side by VictoriaLogs.
Additionally, plugin now understands unicode and ANSI chars thanks to #38.

@dmitryk-dk
Copy link
Contributor

Hi! The issue was fixed by two separate improvement for VictoriaLogs itself and in the datasource.

Please check the release.
Close issue as resolved.

@Proxymiity
Copy link
Author

Hi! Apologies for the response delay.

I can confirm it is working as expected as of the latest version of vlogs and the patch you made.

Thank you!

valyala added a commit to VictoriaMetrics/VictoriaMetrics that referenced this issue Jul 17, 2024
…rings with fmt.* functions instead of using "%q" formatter

The %q formatter may result in incorrectly formatted JSON string if the original string
contains special chars such as \x1b . They must be encoded as \u001b , otherwise the resulting JSON string
cannot be parsed by JSON parsers.

This is a follow-up for c0caa69

See VictoriaMetrics/victorialogs-datasource#24
valyala added a commit to VictoriaMetrics/VictoriaMetrics that referenced this issue Jul 17, 2024
…rings with fmt.* functions instead of using "%q" formatter

The %q formatter may result in incorrectly formatted JSON string if the original string
contains special chars such as \x1b . They must be encoded as \u001b , otherwise the resulting JSON string
cannot be parsed by JSON parsers.

This is a follow-up for c0caa69

See VictoriaMetrics/victorialogs-datasource#24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

6 participants