json deserialization into int64 can fail using patch, UnstructuredJSONScheme, or json.Unmarshal

[smarterclayton]

https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin/6517/consoleText

E1030 21:17:44.127691   25818 errors.go:62] apiserver received an error that is not an unversioned.Status: json: cannot unmarshal number 1.00009e+09 into Go value of type int64
E1030 21:17:44.297969   25818 errors.go:62] apiserver received an error that is not an unversioned.Status: json: cannot unmarshal number 1.00009e+09 into Go value of type int64
E1030 21:17:44.468171   25818 errors.go:62] apiserver received an error that is not an unversioned.Status: json: cannot unmarshal number 1.00009e+09 into Go value of type int64
E1030 21:17:44.640398   25818 errors.go:62] apiserver received an error that is not an unversioned.Status: json: cannot unmarshal number 1.00009e+09 into Go value of type int64
E1030 21:17:44.844421   25818 errors.go:62] apiserver received an error that is not an unversioned.Status: json: cannot unmarshal number 1.00009e+09 into Go value of type int64
E1030 21:17:45.016131   25818 errors.go:62] apiserver received an error that is not an unversioned.Status: json: cannot unmarshal number 1.00009e+09 into Go value of type int64
E1030 21:17:45.186275   25818 errors.go:62] apiserver received an error that is not an unversioned.Status: json: cannot unmarshal number 1.00009e+09 into Go value of type int64
E1030 21:17:45.356693   25818 errors.go:62] apiserver received an error that is not an unversioned.Status: json: cannot unmarshal number 1.00009e+09 into Go value of type int64
E1030 21:17:45.531035   25818 errors.go:62] apiserver received an error that is not an unversioned.Status: json: cannot unmarshal number 1.00009e+09 into Go value of type int64
E1030 21:17:45.701843   25818 errors.go:62] apiserver received an error that is not an unversioned.Status: json: cannot unmarshal number 1.00009e+09 into Go value of type int64
E1030 21:17:45.872073   25818 errors.go:62] apiserver received an error that is not an unversioned.Status: json: cannot unmarshal number 1.00009e+09 into Go value of type int64
E1030 21:17:46.041894   25818 errors.go:62] apiserver received an error that is not an unversioned.Status: json: cannot unmarshal number 1.00009e+09 into Go value of type int64
E1030 21:17:46.211810   25818 errors.go:62] apiserver received an error that is not an unversioned.Status: json: cannot unmarshal number 1.00009e+09 into Go value of type int64
E1030 21:17:46.381676   25818 errors.go:62] apiserver received an error that is not an unversioned.Status: json: cannot unmarshal number 1.00009e+09 into Go value of type int64
E1030 21:17:46.551402   25818 errors.go:62] apiserver received an error that is not an unversioned.Status: json: cannot unmarshal number 1.00009e+09 into Go value of type int64
E1030 21:17:46.720659   25818 errors.go:62] apiserver received an error that is not an unversioned.Status: json: cannot unmarshal number 1.00009e+09 into Go value of type int64
E1030 21:17:46.891358   25818 errors.go:62] apiserver received an error that is not an unversioned.Status: json: cannot unmarshal number 1.00009e+09 into Go value of type int64
E1030 21:17:47.061610   25818 errors.go:62] apiserver received an error that is not an unversioned.Status: json: cannot unmarshal number 1.00009e+09 into Go value of type int64
E1030 21:17:47.232388   25818 errors.go:62] apiserver received an error that is not an unversioned.Status: json: cannot unmarshal number 1.00009e+09 into Go value of type int64
2015-10-30 21:17:47.240892 N | osutil: received terminated signal, shutting down...

[smarterclayton]

Might just be 15651

[smarterclayton]

Aka #5536

[danmcp]

@smarterclayton Please provide the text of the problem in this issue.

[smarterclayton]

Happened again https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin/6818/console

[pweil-]

@smarterclayton

I can reproduce this locally using a template and the new-app command. In my case it appears that when the processedTemplate is retrieved it is being sent back to the client as a floating point number which then cannot unmarshal.

"securityContext":{"supplementalGroups":[1.000030003e+09]}

template and request/response log: https://gist.github.com/pweil-/1cb8cf06e19f2cc0a102

I found this issue golang/go#5562

cc @pmorie

[pweil-]

@liggitt - word on the street is you are a Marshal expert ^

[liggitt]

I am skilled in marshal arts... will take a look asap

[liggitt]

is this actually p2? if so, I'll take a look later

[pmorie]

@liggitt @pweil- Seems like this should be a p1 or p0 to me...

[pmorie]

Looks like decoder.UseNumber() is what we want:

http://play.golang.org/p/Zvcu2BMBJf

[smarterclayton]

p1 because it can wedge the server, unless we eliminate the possibility that it doesn't happen again.

[pmorie]

Actually, the problem is that the encoder is producing the wrong output. From @pweil-'s gist, the response from the API server from a POST is wrong.

[liggitt]

I can recreate this in process template, might be related to RawExtension

[liggitt]

Issue is this line:

runtime.UnstructuredJSONScheme.Decode(obj.RawJSON)

which decodes all numbers as float64

[smarterclayton]

Go JSON doesn't understand int64, right?

On Fri, Nov 6, 2015 at 4:21 PM, Jordan Liggitt [email protected]
wrote:

Issue is this line:

runtime.UnstructuredJSONScheme.Decode(obj.RawJSON)

which decodes all numbers as float64

—
Reply to this email directly or view it on GitHub
#5557 (comment).

[liggitt]

When deserializing to an unstructured map, it either puts all numbers into float64 (today's behavior, which flips integers to floating point in weird cases), or you can set UseNumber() on the json decoder, which stores all numbers internally as strings, gives you Int64,Float64 and String accessors, and by default breaks round tripping by outputting all numbers as quoted strings when re-marshalling

[pmorie]

The root issue is that the unstructured serialization doesn't have the type information to make the right serialization choices. It's tough to imagine a good way to solve this, unless you basically look up the type information by picking through the template output (which might not even be an object).

[smarterclayton]

We can normalize all float64 into their compact representation (1e3 ->
1000)? Or is the case here that we cannot?

On Nov 6, 2015, at 4:38 PM, Jordan Liggitt [email protected] wrote:

When deserializing to an unstructured map, it either puts all numbers into
float64 (today's behavior), or you can set UseNumber() on the json decoder,
which stores all numbers internally as strings, and by default breaks round
tripping by outputting all numbers as quoted strings.

—
Reply to this email directly or view it on GitHub
#5557 (comment).

[liggitt]

Doing anything to all the number values requires walking the JSON tree. If I was going to do that, I'd rather deserialize losslessly (jsonDecoder.UseNumber()) and walk the tree converting to either int64 or float64 as appropriate

[smarterclayton]

If you deserialize with UseNumber don't you lose whether the original value
was string or number (which is the point of unstructured)?

On Nov 6, 2015, at 5:05 PM, Jordan Liggitt [email protected] wrote:

Doing anything to all the number values requires walking the JSON tree. If
I was going to do that, I'd rather deserialize losslessly
(jsonDecoder.UseNumber()) and walk the tree converting to either int64 or
float64 as appropriate

—
Reply to this email directly or view it on GitHub
#5557 (comment).

[liggitt]

no, strings are always strings in JSON, UseNumber turns unquoted numbers into json.Number

[smarterclayton]

Ok sorry. Yes, that is better.

On Nov 6, 2015, at 5:10 PM, Jordan Liggitt [email protected] wrote:

no, strings are always strings in JSON, UseNumber turns unquoted numbers
into json.Number

—
Reply to this email directly or view it on GitHub
#5557 (comment).

[liggitt]

This impacts patch (not used a lot now, but will be with label and annotate), and unstructured unmarshaling (generic List handling and templates)