--swagger_out plugin translates proto type int64 to string in Swagger specification

[tartale]

The --swagger_out plugin yields a swagger definition that translates an int64 protobuf type to a string, even though the Swagger 2.0 OpenAPI specification allows for a signed 64-bit type (type integer; format int64). This might be an oversight and/or copy/paste error, as this code section seems to indicate that this was done intentionally for uint64 values.

Example proto file:

syntax = "proto2";
package testmessage;

import "google/api/annotations.proto";

message TestMessage {
    optional int32 int32Field = 1;
    optional int64 int64Field = 2;
    optional uint32 uint32Field = 3;
    optional uint64 uint64Field = 4;
}

service TestService {
  rpc DoItNow(TestMessage) returns (TestMessage) {
    option (google.api.http) = {
      post: "/do/it/now"
      body: "*"
    };
  }
}

Running protoc with the grpc-gateway plugin and the --swagger_out output currently yields this swagger definition:

{
  "swagger": "2.0",
  "info": {
    "title": "TestService.proto",
    "version": "version not set"
  },
  "schemes": [
    "http",
    "https"
  ],
  "consumes": [
    "application/json"
  ],
  "produces": [
    "application/json"
  ],
  "paths": {
    "/do/it/now": {
      "post": {
        "operationId": "DoItNow",
        "responses": {
          "200": {
            "description": "",
            "schema": {
              "$ref": "#/definitions/testmessageTestMessage"
            }
          }
        },
        "parameters": [
          {
            "name": "body",
            "in": "body",
            "required": true,
            "schema": {
              "$ref": "#/definitions/testmessageTestMessage"
            }
          }
        ],
        "tags": [
          "TestService"
        ]
      }
    }
  },
  "definitions": {
    "testmessageTestMessage": {
      "type": "object",
      "properties": {
        "int32Field": {
          "type": "integer",
          "format": "int32"
        },
        "int64Field": {
          "type": "string",
          "format": "int64"
        },
        "uint32Field": {
          "type": "integer",
          "format": "int64"
        },
        "uint64Field": {
          "type": "string",
          "format": "uint64"
        }
      }
    }
  }
}

[tamalsaha]

Hi,
We are facing the same issue. I can confirm that this was not a problem in earlier versions.

[tamalsaha]

@tartale, gomodules@f7d82dd fixes (reverses?) this behavior. We generate a json schema by post processing swagger spec. So, we had to revert this to generate proper json schema.

[daved]

Will this be addressed?

[achew22]

Sorry, I thought I'd commented on this before. This is working as intended. gRPC gateway follows the serialization rules of proto3 and the proto3 spec says that int64, fixed64, and uint64 are all strings.

This is the case because of IEEE 754 Double precision which JS implements using a 52 bit mantissa (1 for sign, and 11 for exponent). This means the largest number that JS can represent without losing precision is 2^52. Try it out:

> (Math.pow(2,53) + 0) == (Math.pow(2,53) + 1)
true

So, rather than pretending that Javascript can handle it (and lose data) proto3 prefers to have the client interact with strings. Fewer footguns == better libraries.

If you want to interact with int64 as numbers, there are a bunch of JS bigint libraries out there that you can use to parse the string, perform whatever operations you want and get back a string. That would probably be a better way to handle this.

Note that int32 doesn't suffer from the same issue. If you don't have the pressure to operate on numbers > 2^32 that might be a good fix.

[daved]

Thanks for the clarity!

[tamalsaha]

@achew22 , Thanks for your explanation. We generate a json schema that is used for frontend validation. In that scenario, the data format is more important than actual data. So, in that case, we should be validating it against an "integer". Am I right?

[achew22]

Unfortunately, that is not the case. Let's walk through a simple example and see what happens.

Here is a non comprehensive list of my assumptions:

1. You are using grpc-gateway
2. You're using a fairly recent version of grpc-gateway
3. You haven't overridden the serializer

Okay, let's make a proto like this:

message TestProto {
  int64 a = 1;
  int64 b = 2;
}

Now let's send some data through grpc-gateway using that proto. [Code left as an exercise for the reader] You should get out on the other side assuming you transmit TestProto.newBuilder().setA(1L).setB(2L).build() (using Java syntax but any language would be fine).

{"a":"1","b":"2"}

And when you transmit stuff back to the server

{"a":"1","b":"9007199254740993"}

On your server you should get a response that is an int64 and contains all the bits of precision without losing anything. If you get something else, can you please reopen this so I can take a look at it because it is a bug.

[tamalsaha]

@achew22 , Will grpc-gateway fail if my frontend sends {"a":"1","b":2} ? or will it auto convert it into string? I understand that frontend is sending incorrectly formatted data according to protoc3 spec.

Edit: After reading the spec, the answer is it will work ok. For json -> protoc, string or int both are ok. For protoc->json, protoc will only product strings.

[achew22]

I don't know. It's not specified behavior. Wanna try it out and see what happens? It might work, but I can't promise even if it does that it'll work forever.

[tamalsaha]

@achew22 , it seems to be specified int the notes from your link above.

[achew22]

Cool! Then go for it. If it works that's wonderful, but don't forget the aforementioned imprecision issues.

[achew22]

If you really want to have the type as number and you're 100% sure the data isn't ever going to be > 2^32, why not use an int32?

[tamalsaha]

The main source of int64 is timestamp. The other places we used int64 could probably be changed to int32. We will probably do that now that we are aware of this issue.

[achew22]

Ah, have you considered using a google.protobuf.Timestamp? That will get detected by the swagger generator and you'll be able to interact with it directly as a Date object. What do you think about that?

[tamalsaha]

@achew22 , How does google.protobuf.Timestamp convert into Json? My guess is that it becomes 2 fields (sec, nano)? If that is the case, it is unacceptable for us. We use PHP, Java, Javascript and GO. Using epoch sec as int64 is the easiest way to keep our sanity.

[achew22]

Timestamp is special and very nice. It's actually specified on that same page we keep sending back and forth to each other 😉.

---

Timestamp string "1972-01-01T10:00:20.021Z" Uses RFC 3339, where generated output will always be Z-normalized and uses 0, 3, 6 or 9 fractional digits.

I just switched my thing to use it a couple of days ago. Works like a charm!

[dreaming1237]

hi, anyone could help to explain why swagger translates proto type uint32 to int64

[johanbrandhorst]

Could you join the #grpc-gateway channel on Gophers slack? https://invite.slack.golangbridge.org/. This isn't the right place for this sort of debugging.