Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal: Expand OpenAPI to include RPC APIs #801

Closed
timburks opened this issue Sep 29, 2016 · 19 comments
Closed

Proposal: Expand OpenAPI to include RPC APIs #801

timburks opened this issue Sep 29, 2016 · 19 comments
Labels
Moved to Moonwalk Issues that can be closed or migrated as being addressed in Moonwalk

Comments

@timburks
Copy link
Contributor

The OpenAPI 2.0 Specification states that its goal is “to define a standard, language-agnostic interface to REST APIs”. REST is an extremely popular style for implementing APIs that run over HTTP and related protocols (HTTPS, HTTP/2, QUIC). But other transport mechanisms are common and APIs are often defined in ways that correspond to procedure calls. In some ways, procedure calls are more fundamental, as REST APIs are often implemented by systems that convert REST requests into procedure calls.

At times it is desirable to use remote procedure calls (RPCs) as the primary form of an API. RPC systems allow expression of semantics that go outside the range of the HTTP verbs used by REST. RPC systems map naturally to the function calls that are used to implement most software systems, and RPC provides a design pattern that can be use to guide system implementations. The RPC abstraction is also much older than REST, and new systems such as gRPC and Thrift show its enduring usefulness.

With this in mind, we think it would be beneficial to expand the scope of the OpenAPI specification to include remote procedure call semantics. Here we propose an expanded specification that builds on our experience using RPCs to connect very large distributed systems while being general enough to apply to a broad range of variations.

Background

Protocol Buffers

We give special consideration to the Protocol Buffer message representation. At Google, Protocol Buffers are used to send and receive tens of billions of messages each second. Protocol Buffers are also used for messages in the open-source gRPC framework and other more specialized RPC frameworks [1] [2] [3].

Protocol Buffers are typically represented in a binary form, and the latest version (proto3) also allows mapping to representations in JSON and YAML formats. Messages can be generally treated as collections of typed fields, just as they are in other serialization formats including Thrift, Avro, and Cap’n Proto. Thus message description can be treated similarly for all of these representations, including the JSON and YAML that OpenAPI uses to describe request and response objects.

Protocol Buffer interfaces are defined using a special description language (.proto) that describes messages and the services that use them. This language is different from the JSON and YAML used for OpenAPI specifications. To keep the specification and tooling simple, this proposal replaces this language with extensions to OpenAPI that carry the same meanings. It seeks to express as much as possible of the .proto language within OpenAPI so that if desired, .proto files can be converted into OpenAPI RPC descriptions and vice versa.

RPC Elements

Messages

Remote procedure calls are described in terms of the messages that they send and receive. Messages contain fields that have names, types, and other attributes, often including an integer field number that is used in message serialization.

Services

In RPC terminology, "Services" are collections of remote procedure calls that send and receive messages. Each call has a single input and output message, and some RPC implementations will allow input and output messages to be streamed in a single call. Thus each remote procedure call is described by its name, input type, output type, and whether or not the input and output messages are streamed.

Example

Here we show an example .proto description of an API. For illustration, we've specified the GetBook RPC as a streaming call that accepts a stream of book requests and returns a stream of books. This is indicated with the stream keyword that appears before the request and response types.

syntax = "proto3";

package examples.bookstore;

import "google/protobuf/empty.proto";
import "google/protobuf/struct.proto";

service Bookstore {
  rpc ListShelves(google.protobuf.Empty) returns (ListShelvesResponse) {}
  rpc CreateShelf(CreateShelfRequest) returns (Shelf) {}
  rpc GetShelf(GetShelfRequest) returns (Shelf) {}
  rpc DeleteShelf(DeleteShelfRequest) returns (google.protobuf.Value) {}
  rpc ListBooks(ListBooksRequest) returns (ListBooksResponse) {}
  rpc CreateBook(CreateBookRequest) returns (Book) {}
  rpc GetBook(stream GetBookRequest) returns (stream Book) {}
  rpc DeleteBook(DeleteBookRequest) returns (google.protobuf.Value) {}
}

message Shelf {
  string name = 2;
  string theme = 3;
}

message Book {
  string author = 2;
  string name = 3;
  string title = 4;
}

message ListShelvesResponse {
  repeated Shelf shelves = 1;
}

message CreateShelfRequest {
  Shelf shelf = 1;
}

message GetShelfRequest {
  int64 shelf = 1;
}

message DeleteShelfRequest {
  int64 shelf = 1;
}

message ListBooksRequest {
  int64 shelf = 1;
}

message ListBooksResponse {
  repeated Book books = 1;
}

message CreateBookRequest {
  int64 shelf = 1;
  Book book = 2;
}

message GetBookRequest {
  int64 shelf = 1;
  int64 book = 2;
}

message DeleteBookRequest {
  int64 shelf = 1;
  int64 book = 2;
}

Open API Representation

Messages

RPC messages are described in the OpenAPI definitions section. To these we add a few new fields to provide protocol buffer semantics. New fields are prefixed x- but in the accepted proposal, we would expect all x- prefixes to be deleted.

Properties of a message correspond to fields in a protocol buffer message. x-field-number (or fieldNumber) is a required integer property that associates a unique field number with each property. x-repeated is an optional boolean property (by default false) that, when true, indicates that a field may occur more than once.

Here we show an example OpenAPI representation of the messages in the Bookstore API.

 definitions:
  "Shelf":
    type: object
    properties:
      name: 
        type: string
        x-field-number: 2
      theme:
        type: string
        x-field-number: 3
  "Book":
    type: object
    properties:
      author: 
        type: string
        x-field-number: 2
      name:
        type: string
        x-field-number: 3
      title:
        type: string
        x-field-number: 4
  "ListShelvesResponse":
    type: object
    properties:
      shelves:
        allOf:
          - $ref: "#/definitions/Shelf"
          - x-repeated: true
          - x-field-number: 1
  "CreateShelfRequest":
    type: object
    properties:
      shelf:
        allOf:
          - $ref: "#/definitions/Shelf"
          - x-field-number: 1
  "GetShelfRequest":
    type: object
    properties:
      shelf:
        type: integer
        format: int64
        x-field-number: 1
  "DeleteShelfRequest":
    type: object
    properties:
      shelf:
        type: integer
        format: int64
        x-field-number: 1
  "ListBooksRequest":
    type: object
    properties:
      shelf:
        type: integer
        format: int64
        x-field-number: 1
  "ListBooksResponse":
    type: object
    properties:
      books:
        allOf:
          - $ref: "#/definitions/Book"
          - x-repeated: true
          - x-field-number: 1
  "CreateBookRequest":
    type: object
    properties:
      shelf:
        type: integer
        format: int64
        x-field-number: 1
      book:
        allOf:
          - $ref: "#/definitions/Book"
          - x-field-number: 2
  "GetBookRequest": 
    type: object
    properties:
      shelf:
        type: integer
        format: int64
        x-field-number: 1
      book:
        type: integer
        format: int64
        x-field-number: 2
  "DeleteBookRequest": 
    type: object
    properties:
      shelf:
        type: integer
        format: int64
        x-field-number: 1
      book:
        type: integer
        format: int64
        x-field-number: 2

Services

Services are a distinct new entity that we represent using the x-services key at the top level of the OpenAPI description. This allows an API to include both RPC and REST representations side-by-side.

Services are described by their name and the procedures they contain. Procedures are described by the objects they accept and return. Streamed values are indicated with x-streaming, an optional boolean property (by default false).

Here is an example OpenAPI representation of the services in the Bookstore API (in this case, a single service named "Bookstore").

x-services:
  "Bookstore":
    x-procedures:
      "ListShelves":
        x-accepts: 
          $ref: "google-protobuf.yaml#/Empty"
        x-returns:
          $ref: "#/definitions/ListShelvesResponse"
      "CreateShelf":
        x-accepts:
          $ref: "#/definitions/CreateShelfRequest"
        x-returns:
          $ref: "#/definitions/Shelf"
      "GetShelf":
        x-accepts:
          $ref: "#/definitions/GetShelfRequest"
        x-returns:
          $ref: "#/definitions/Shelf"
      "DeleteShelf":
        x-accepts:
          $ref: "#/definitions/DeleteShelfRequest"
        x-returns:
          $ref: "google-protobuf.yaml/#Value"
      "ListBooks": 
        x-accepts:
          $ref: "#/definitions/ListBooksRequest"
        x-returns:
          $ref: "#/definitions/ListBooksResponse"
      "CreateBook": 
        x-accepts:
          $ref: "#/definitions/CreateBookRequest"
        x-returns:
          $ref: "#/definitions/Book"
      "GetBook": 
        x-accepts:
          allOf:
            - $ref: "#/definitions/GetBookRequest"
            - streaming: true
        x-returns:
          allOf:
            - $ref: "#/definitions/Book"
            - streaming: true
      "DeleteBook": 
        x-accepts:
          $ref: "#/definitions/DeleteBookRequest"
        x-returns:
          $ref: "#/definitions/Shelf"

Discussion

Our intent is to make it possible to write tools that convert back-and-forth between .proto and OpenAPI representations and to build RPC flows that completely replace .proto inputs with OpenAPI RPC specifications.

OpenAPI representations are more verbose than .proto, but are more amenable to automated processing and custom editors like swagger-editor. Our hope is that this representation will lead more editors and other tools to support RPC APIs.

Types in OpenAPI and Protocol Buffers

Protocol Buffers contain many finely-differentiated scalar types, while the OpenAPI spec uses general types such as number and integer. In OpenAPI, an additional format field supplements the type field with additional representation detail, so our proposal uses this field to include the full name of the corresponding Protocol Buffer type.

.proto field type OpenAPI type field value OpenAPI format field value
double number double
float number float
int64 integer int64
uint64 integer uint64
int32 integer int32
fixed64 integer fixed64
fixed32 integer fixed64
bool boolean -
string string -
group - -
message - -
bytes binary -
uint32 integer uint32
enum ? ?
sfixed32 integer sfixed32
sfixed64 integer sfixed64
sint32 integer sint32
sint64 integer sint64

In the above table, the bool, string, and bytes types directly correspond to OpenAPI types and need no additional detail in the format field. The group type in .proto is deprecated and unsupported, and the message type corresponds to inclusion of another message, which is represented in OpenAPI with the $ref property.

The enum type is represented in .proto as an integer and in OpenAPI as a string. This difference is unresolved in this proposal.

Gaps between OpenAPI and Protocol Buffers

There are some Protocol Buffer features that aren’t yet covered by this proposal. Gaps that are significant and unresolved may be addressed in future proposals.

Default values

Default field values could be represented with the existing OpenAPI default property, but in some message representations (such as proto3, default values are specified for each type and are not modifiable.

Enumerations

Enumerations are represented with strings in OpenAPI and integers in .proto. Resolution of this is a high priority future proposal.

Maps

The .proto format allows fields to have map types which include type specifications for map keys and values. When serialized, these maps are represented with special automatically-created messages.

Here we omit further discussion of maps, leaving it as a more general question about the OpenAPI Specification.

Extensions

Extensions are defined in proto2 and allow additional fields to be added to messages and for ranges of field numbers to be reserved for third-party extensions. Extensions are not supported in proto3 and are omitted from this proposal.

Nested messages

The .proto language allows messages to be defined inside other messages. Nested types can’t be written directly in OpenAPI, but we can define messages with hierarchical names similar to the ones that would be implied for nested .proto messages.

Options

In .proto, options are predefined annotations that can be added in various places in a .proto file. Common options configure code generators (by specifying package names or class name prefixes) or map RPC procedures to REST methods. Options are not addressed here but are a priority for inclusion in a future proposal.

Oneof

In .proto, the oneof statement groups fields to indicate that only one of the grouped fields can be included in a message. There is no corresponding concept in OpenAPI 2.0, but this appears to be addressed by a pending pull request.

Packages and API Versions

.proto descriptions optionally include a package name. This has a similar purpose as the basePath field in the OpenAPI root object and we suggest that either a new field named package or the existing basePath be used for this.

API Versions are commonly indicated by the last segment in a package name (segments are separated by periods). When the last segment looks like a version name (beginning with a 'v' and following with a possibly-dotted number), it is used as the version name. We omit this convention from this proposal and leave the representation of API versions to the existing version field of the info object.

@fehguy
Copy link
Contributor

fehguy commented Sep 29, 2016

RPC semantics is quite impossible for 3.0 but protobufs as payload may be doable and would certainly help out. Will look at your comments about the missing fields in our extended json schema for supporting protos.

@cfineman
Copy link

cfineman commented Oct 6, 2016

@fehguy "quite impossible" because it goes beyond the standard CRUD REST verbs or is there some other reason it's untenable?

@fehguy
Copy link
Contributor

fehguy commented Oct 7, 2016

@cfineman because we're closed on major features for 3.0.

@cfineman
Copy link

cfineman commented Oct 7, 2016

acknowledged

@RobDolinMS RobDolinMS added this to the v3.Next milestone Apr 21, 2017
@fmvilas
Copy link

fmvilas commented May 24, 2017

It might be a good fit for the AsyncAPI spec, given it's message-driven instead of request/response.

@smyrman
Copy link

smyrman commented Nov 13, 2019

If we look at RPC APIs over JSON/YAML, I feel the "discriminator" feature in OpenAPI should allow describing the requests side of things pretty accurately as a single path item. However, there is lacking a way to associate the right requests with the right response when doing this.

E.g. if we look at JSON RPC 2.0 for instance, requests have two main formats.

  • {"jsonrpc": "2.0", "id": 1, "method": "service.functionCall", "params: {...}} named parameters.
  • {"jsonrpc": "2.0", "id": 1, "method": "service.functionCall", "params: [...]} positional parameters.

Where a large collection of different methods exist.

The responses in JSON RPC are of format:

  • {"jsonrpc": "2.0", "id":"1", "result": ..} // On success
  • {"jsonrpc": "2.0", "id":1, "error": ...} // On error

Where the schema of the result (and in principal error) are generally determined by the request method.

At least for the first variant, there is no problem describing this as a set of "<functionCall>Request" body definitions and have a single Open API path item for them (describing the HTTP protocol specific RPC endpoint) that uses a discriminator object to select between them. If we want to describe e.g. JSON batch requests as well (an array of the described request bodies), we could probably do it with an outer $anyOf.

The only missing feature for a correct implementation with this approach, is to be able to provide a similar discriminator object for the response type that instead of switching on a field in the response payload, switches on a field at a particular path in the request (A discriminator object for batch requests is probably not something we could pull of though).

I am not stating that this is the best way of documenting a RPC API.

But I think it's important that the scope should be to describe the HTTP Protocol for the RPC API, and not the RPC API in general (which could be protocol agnostic).

@philsturgeon
Copy link
Contributor

philsturgeon commented Jul 3, 2020

Hey @timburks, the goal of OpenAPI Specification has swapped from REST to HTTP so that's one roadblock gone. #1946

image

I know protobuf support is wanted by @darrelmiller and it's come up several times at Stoplight. Maybe it's time to progress. 😎

@orubel
Copy link

orubel commented Aug 23, 2021

You are basically over complicating and mixing FUNCTION with DATA; the idea is to abstract these two. These docs are meant to reference data that is shared across the architecture with request/response for proper checking of endpoint compliance.

Adding function names complicates the referencing; not all resources/services in the architecture will have these functions available to them... but they will have the names of all params as sent in the request/response. Thus it is better to create a common object in the doc that is a reference to all variables sent/returned.

By creating the OBJECT in the spec, you can easily create all endpoints in said doc by referencing parm KEY. This allows for JOINED DATABASE QUERIES as well as other complex datasets to EASILY be referenced and checked at all points in the architecture.

@Cahl-Dee
Copy link

Hey team. I come from a web2 background where "REST is the way" and have always leaned on OpenAPI for defining our API specifications. I now work at a blockchain infrastructure provider where our APIs are mostly JSON-RPC with some REST here and there. I'd love to leverage OAS for our documentation and may even be able to contribute some development resources to the effort if we could come together and agree on a path forward.

@smyrman
Copy link

smyrman commented Jun 2, 2022

@Cahl-Dee, OpenAPI isn't particularly suitable for describing RPC APIs, and for JSON RPC 2.0 in particular, where all requests generally happen against the same endpoint, you are probably better off looking at https://open-rpc.org/ and associated tools.

@orubel
Copy link

orubel commented Jun 2, 2022

@smyrman "where all requests generally happen against the same endpoint"???

The endpoint is 'controller/method'; for each separate controller/method, its a separate endpoint . This is where the request/response meet on the backend and thus is the central version of truth. This is why all BINDINGS in an API build are put into the controllers.

What are you talking about???

@smyrman
Copy link

smyrman commented Jun 3, 2022

I mean all requests for JSON RPC 2.0 generally (or often) happen against the same URL / URL path. E.g. all method calls might be done against a single URL https://example.com/rpc. I.e. same endpoint from an HTTP (protocol) perspective.

@orubel
Copy link

orubel commented Jun 3, 2022

@smyrman Thats because they are sending 'method' in the passed data (which is unusual for RPC); look at how JSON-RPC passes data.

gRPC encodes the controller/method in the URI just like all other RPC. JSON-RPC seems to be odd in this respect.

ex. http:// localhost/controller/method
ie http://localhost/user/getByName

The reason it is done in the URI is because request uri will ALWAYS be passed with redirects in your network and cannot be overwritten. The request variables can be removed/overwritten.

ALSO... would point out ticket stated 'RPC' so is not specific to any one RPC implementation so as long as you stick with a controller/method approach, this will work with JSON-RPC, gRPC and pretty much all RPC

@Cahl-Dee
Copy link

Cahl-Dee commented Aug 24, 2022

@Cahl-Dee, OpenAPI isn't particularly suitable for describing RPC APIs, and for JSON RPC 2.0 in particular, where all requests generally happen against the same endpoint, you are probably better off looking at https://open-rpc.org/ and associated tools.

@smyrman I'm not familiar with the inner workings of OAS so I have no clue how big of a change this would be, but are methods and endpoints really that different of concepts? Has there been, or is there openness to discussing how the primitive leveraged by OAS could be abstracted to cover both?

@orubel
Copy link

orubel commented Aug 24, 2022

@Cahl-Dee Thats how it is used in Spring and Springboot https://www.baeldung.com/spring-rest-openapi-documentation. So I don't see how it isn't good at it when it is doing it fine. That's an RPC implementation.

@Oct17th
Copy link

Oct17th commented Jul 13, 2023

@smyrman Thats because they are sending 'method' in the passed data (which is unusual for RPC); look at how JSON-RPC passes data.

gRPC encodes the controller/method in the URI just like all other RPC. JSON-RPC seems to be odd in this respect.

ex. http:// localhost/controller/method ie http://localhost/user/getByName

The reason it is done in the URI is because request uri will ALWAYS be passed with redirects in your network and cannot be overwritten. The request variables can be removed/overwritten.

ALSO... would point out ticket stated 'RPC' so is not specific to any one RPC implementation so as long as you stick with a controller/method approach, this will work with JSON-RPC, gRPC and pretty much all RPC

+1 use http:// localhost/controller/method can mapped to a unique server implementation just like http endpoint.
btw i have another question, rpc methods do not have http verbs and http status, so we seems like need a new tag like paths and webhooks to describe the rpc methods, otherwise it seems to be odd when use paths to describe the rpc methods, it always use 'post' verb and always defined request in 'requestBody' and always defined response in '200' status

  /controller/method:
    post:
      requestBody:
        - name: name
          xxx
        - name: key
          xxx
      responses:
        '200':
          description: An array of rpc resp
          schema:
            type: array
            items:
              $ref: '#/definitions/RPCstruct'

@orubel
Copy link

orubel commented Jul 13, 2023 via email

@handrews handrews added the Moved to Moonwalk Issues that can be closed or migrated as being addressed in Moonwalk label Jan 27, 2024
@handrews
Copy link
Member

OAS 4 Moonwalk will support synchronous HTTP RPC APIs under the big tent principle. Please join the discussions in that repository if you are still interested.

For asynchronous RPC, there is AsyncAPI.

@orubel
Copy link

orubel commented Jan 27, 2024 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Moved to Moonwalk Issues that can be closed or migrated as being addressed in Moonwalk
Projects
None yet
Development

No branches or pull requests