JSON parser fix

[naithar]

Related issue: #40794

Tested with:

func _ready():
	_check('"text"')
	_check('   "text"   ')
	_check('"10"')
	_check('10')
	_check('{}')
	_check('["a", 0]')
	_check('["a", 0, null]')
	_check(' { "a": "b" }')
	_check(' { "a": null }')
	_check(' { "a": 0 }')
	_check(' { "a": ["b"] }')
	_check(' { "a": {} }')
	print("should fail")
	_check('"a": 0 }')
	_check('{ "a": 0 ')
	_check('"a')
	_check('"a"0')
	_check('{}{}')
	_check('["a", 0')
	_check('["a", 0}')
	_check('{ ["a", 0] }')
	_check('true, false, [1,2,3]')


func _check(jsonString):
	var a = validate_json(jsonString)
	print('validation error: "{error}", result: "{json}"'.format({'error' : a, 'json' : parse_json(jsonString)}))

Some test cases could probably be added to the test added in #40795

Result:

2020-07-28 22:16:20.077650+0300 godot_4_test[14675:6335126] validation error: "", result: "text"
2020-07-28 22:16:20.077687+0300 godot_4_test[14675:6335126] validation error: "", result: "text"
2020-07-28 22:16:20.077716+0300 godot_4_test[14675:6335126] validation error: "", result: "10"
2020-07-28 22:16:20.077745+0300 godot_4_test[14675:6335126] validation error: "", result: "10"
2020-07-28 22:16:20.077775+0300 godot_4_test[14675:6335126] validation error: "", result: "{}"
2020-07-28 22:16:20.077811+0300 godot_4_test[14675:6335126] validation error: "", result: "[a, 0]"
2020-07-28 22:16:20.077855+0300 godot_4_test[14675:6335126] validation error: "", result: "[a, 0, Null]"
2020-07-28 22:16:20.077895+0300 godot_4_test[14675:6335126] validation error: "", result: "{a:b}"
2020-07-28 22:16:20.077939+0300 godot_4_test[14675:6335126] validation error: "", result: "{a:Null}"
2020-07-28 22:16:20.077978+0300 godot_4_test[14675:6335126] validation error: "", result: "{a:0}"
2020-07-28 22:16:20.078020+0300 godot_4_test[14675:6335126] validation error: "", result: "{a:[b]}"
2020-07-28 22:16:20.078063+0300 godot_4_test[14675:6335126] validation error: "", result: "{a:{}}"
2020-07-28 22:16:20.078077+0300 godot_4_test[14675:6335126] should fail
2020-07-28 22:16:20.078155+0300 godot_4_test[14675:6335126] validation error: "0:Expected 'EOF'", result: "Null"
2020-07-28 22:16:20.078232+0300 godot_4_test[14675:6335126] validation error: "0:Expected '}' or ','", result: "Null"
2020-07-28 22:16:20.078304+0300 godot_4_test[14675:6335126] validation error: "0:Unterminated String", result: "Null"
2020-07-28 22:16:20.078379+0300 godot_4_test[14675:6335126] validation error: "0:Expected 'EOF'", result: "Null"
2020-07-28 22:16:20.078450+0300 godot_4_test[14675:6335126] validation error: "0:Expected 'EOF'", result: "Null"
2020-07-28 22:16:20.078525+0300 godot_4_test[14675:6335126] validation error: "0:Expected ']'", result: "Null"
2020-07-28 22:16:20.078598+0300 godot_4_test[14675:6335126] validation error: "0:Expected ','", result: "Null"
2020-07-28 22:16:20.078667+0300 godot_4_test[14675:6335126] validation error: "0:Expected key", result: "Null"
2020-07-28 22:16:20.078748+0300 godot_4_test[14675:6335126] validation error: "0:Expected 'EOF'", result: "Null"

Testing same test cases with macOS's JSONSerializer produces same success/failure results.

Bugsquad edit: Fixes #40794.

[Xrayez]

Nice, ideally this is where #40795 could've been merged to master at this point, so you could rebase and see if my tests pass. If not, perhaps there might be something wrong with the test itself, so you'll may make changes in your PR to the test if necessary. 🙂

[Xrayez]

This fixes the first assert in #40795 (detected as parse error now)!

But this assert fails:

D:\src\godot\tests\test_json.h(52): ERROR: CHECK( parsed.get_type() != Variant::STRING ) is NOT correct!
  values: CHECK( 4 != 4 )
  logged: Should not prematurely parse as a string.

Do you think it should return the string or return empty Variant instead?

[naithar]

This fixes the first assert in #40795 (detected as parse error now)!

But this assert fails:

D:\src\godot\tests\test_json.h(52): ERROR: CHECK( parsed.get_type() != Variant::STRING ) is NOT correct!
  values: CHECK( 4 != 4 )
  logged: Should not prematurely parse as a string.

Do you think it should return the string or return empty Variant instead?

I'm not really sure, how this should be handled, but I would expect an empty or null result in case of malformed json. So it can be actually tested on == null in scripts.

[naithar]

@Xrayez I've added results reset for malformed json, test case should not be failing now

[Xrayez]

Yeah just tested now, I confirm it resets the value to Variant() (aka null in GDScript), all assertions in #40795 pass!

This is the case where I'm not really sure whether returning a "half-way" token even on parse error is intended, but I'm not sure if this "bug" was used as a "feature" before. 🙂

[Calinou]

I think we should merge this only in master for now, and then cherry-pick this to the 3.x branch after 3.3 is released to avoid introducing regressions in 3.3.

[akien-mga]

Thanks!

[akien-mga]

Cherry-picked for 3.4.