Skip to content

Commit

Permalink
TypeScript Definition Generator
Browse files Browse the repository at this point in the history
Adds a generator that produces TypeScript definition files for a JavaScript client. As with JavaScript, there are two generators: `tsd_types` and `tsd_client`.

The definition files require TypeScript 2.0, as they rely upon TypeScript tagged unions.

Overview: Mapping Stone Types to TypeScript
===========================================

Below, I will summarize how we map Stone types to TypeScript.

Basic Types
-----------

TypeScript's basic types match JSDoc, so there is no difference from the `js_types` generator.

Alias
-----

Aliases are emitted as `type`s:

``` typescript
type AliasName = ReferencedType;
```

Struct
------

Structs are emitted as `interface`s, which support inheritance. Thus, if a struct `A` extends struct `B`, it will be emitted as:

``` typescript
interface A extends B {
  // fields go here
}
```

Nullable fields and fields with default values are emitted as _optional_ fields. In addition, the generator adds a field description with the default field value, if the field has one:

``` typescript
interface A {
  // Defaults to False
  recur?: boolean;
}
```

Unions
------

Unions are emitted as a `type` that is the disjunction of all possible union variants (including those from parent types!). Each variant is emitted as an individual `interface`.

```
union Shape
    point
    square Float64
        "The value is the length of a side."
    circle Float64
        "The value is the radius."
```

``` typescript
interface ShapeCircle {
  .tag: 'circle';
  circle: number;
}

interface ShapeSquare {
  .tag: 'square';
  square: number;
}

interface ShapePoint {
  .tag: 'point';
}

type Shape = ShapePoint | ShapeSquare | ShapeCircle;
```

TypeScript 2.0 supports [tagged union types](https://github.com/Microsoft/TypeScript/wiki/What%27s-new-in-TypeScript#tagged-union-types) like these ones, so the compiler should automatically infer the type of a shape when the developer writes code like so (and statically check that all cases are covered!):

``` typescript
var shape: Shape = getShape();
switch (shape['.tag']) {
  case 'point':
      console.log('point');
      break;
   case 'square':
       // Compiler knows this is a ShapeSquare, so .square field is visible.
      console.log('square ' + shape.square);
      break;
    // No 'circle' case! If developer enables the relevant compiler option, compilation will fail.
}
```

Unfortunately, [there is a bug that prevents this from happening](microsoft/TypeScript#10530) when you use bracket notation to access a field. It will be fixed in a future version of TypeScript. Until then, developers will need to cast:

``` typescript
var shape: Shape = getShape();
switch (shape['.tag']) {
  case 'point':
      console.log('point');
      break;
   case 'square':
      console.log('square ' + (<ShapeSquare> shape).square);
      break;
}
```

Struct Polymorphism
-------------------

When a struct explicitly enumerates its subtypes, direct references to the struct will have a `.tag` field to indicate which subtype it is. Direct references to the struct's subtypes will omit this field.

To capture this subtlety, the generator emits an interface that represents a direct reference to a struct with enumerated subtypes:

```
struct Resource
    union
        file File
        folder Folder

    path String

struct File extends Resource
    ...

struct Folder extends Resource
    ...
```

``` typescript
interface Resource {
  path: string;
}
interface File extends Resource {
}
interface Folder extends Resource {
}
interface ResourceReference extends Resource {
  '.tag': 'file' | 'folder';
}
interface FileReference extends File {
  '.tag': 'file';
}
interface FolderReference extends Folder {
  '.tag': 'folder';
}
```

Direct references to `Resource` will be typed as `FileReference | FolderReference | ResourceReference` if the union is open, or `FileReference | FolderReference` if the union is closed. A direct reference to `File` will be typed as `File`, since the `.tag` field will not be present.

TypeScript 2.0's tagged union support should work on these types once the previously-discussed bug is fixed.

Nullable Types
--------------

Nullable types are emitted as optional fields when referenced from structs.

Routes
------

Routes are emitted in the same manner as the JavaScript generators, **except** that TypeScript's type system is unable to type `Promise`-based errors. The generator adds text to the route's documentation that explicitly mentions the data type the developer should expect when an error occurs.

Example:

``` typescript
type DropboxError = DropboxTypes.Error;
db.filesListFolder({path: ''}).then((response) => {
  // TypeScript knows the type of response, so no type annotation is needed.
}).catch(
  // Add explicit annotation on err.
  (err: DropboxError<DropboxTypes.files.ListFolderError>) => {

  });
```

Import / Namespaces
-------------------

Stone namespaces are mapped directly to TypeScript namespaces:

```
namespace files;

import common;

struct Metadata
    parent_shared_folder_id common.SharedFolderId?
```

``` typescript
namespace files {
  interface Metadata {
    parent_shared_folder_id?: common.SharedFolderId;
  }
}
```

Using the Generator
===================

Both `tsd_types` and `tsd_client` consume a template file, which contains a skeleton around the types they omit. This skeleton is unavoidable, as SDKs may augment SDK classes (like `Dropbox` or `DropboxTeam`) with additional methods not described in stone.

The "templates" simply have a comment string that marks where the generator should insert code. For example, the following template has markers for route definitions and type definitions:

``` typescript
class Dropbox {
  // This is an SDK-specific method which isn't described in stone.
  getClientId(): string;

  // All of the routes go here:
  /*ROUTES*/
}

// All of the stone data types are defined here:
/*TYPES*/
```

In the above template, the developer would need to run the `tsd_types` generator to produce an output file, and then run the `tsd_client` generator on that output to insert the routes (or vice-versa).

The developer may also choose to have separate template files for types and routes:

``` typescript
// in types.d.ts
namespace DropboxTypes {
  /*TYPES*/
}
```

``` typescript
/// <reference path="./types.d.ts" />
// ^ this will "import" the types from the other file.
// in dropbox.d.ts
namespace DropboxTypes {
  class Dropbox {
    /*ROUTES*/
  }
}
```

Developers can customize the template string used for `tsd_client` with a command line parameter, in case they have multiple independent sets of routes:

``` typescript
namespace DropboxTypes {
  class Dropbox {
    /*ROUTES*/
  }
  class DropboxTeam {
    /*TEAM_ROUTES*/
  }
}
```

Generator Usage in Dropbox SDK
==============================

For Dropbox's JavaScript SDK, I've defined the following templates.

**dropbox.d.tstemplate**: Contains a template for the `Dropbox` class.

``` typescript
/// <reference path="./dropbox_types.d.ts" />
declare module DropboxTypes {
  class Dropbox extends DropboxBase {
    /**
     * The Dropbox SDK class.
     */
    constructor(options: DropboxOptions);

/*ROUTES*/
  }
}
```

**dropbox_team.d.tstemplate**: Contains a template for the `DropboxTeam` class.

``` typescript
/// <reference path="./dropbox_types.d.ts" />
/// <reference path="./dropbox.d.ts" />
declare module DropboxTypes {
  class DropboxTeam extends DropboxBase {
    /**
     * The DropboxTeam SDK class.
     */
    constructor(options: DropboxOptions);

    /**
     * Returns an instance of Dropbox that can make calls to user api endpoints on
     * behalf of the passed user id, using the team access token. Only relevant for
     * team endpoints.
     */
    actAsUser(userId: string): Dropbox;

/*ROUTES*/
  }
}
```

**dropbox_types.d.ts**: Contains a template for the Stone data types, as well as the `DropboxBase` class (which is shared by both `Dropbox` and `DropboxTeam`).

``` typescript
declare module DropboxTypes {
  interface DropboxOptions {
    // An access token for making authenticated requests.
    accessToken?: string;
    // The client id for your app. Used to create authentication URL.
    clientId?: string;
    // Select user is only used by team endpoints. It specifies which user the team access token should be acting as.
    selectUser?: string;
  }

  class DropboxBase {
    /**
     * Get the access token.
     */
    getAccessToken(): string;

    /**
     * Get a URL that can be used to authenticate users for the Dropbox API.
     * @param redirectUri A URL to redirect the user to after authenticating.
     *   This must be added to your app through the admin interface.
     * @param state State that will be returned in the redirect URL to help
     *   prevent cross site scripting attacks.
     */
    getAuthenticationUrl(redirectUri: string, state?: string): string;

    /**
     * Get the client id
     */
    getClientId(): string;

    /**
     * Set the access token used to authenticate requests to the API.
     * @param accessToken An access token.
     */
    setAccessToken(accessToken: string): void;

    /**
     * Set the client id, which is used to help gain an access token.
     * @param clientId Your app's client ID.
     */
    setClientId(clientId: string): void;
  }

/*TYPES*/
}
```

Then, I defined simple definition files for each of the ways you package up the SDK, which references these types. These can be readily distributed alongside your libraries.

`DropboxTeam-sdk.min.d.ts` (`DropboxTeam` class in a UMD module):

``` typescript
/// <reference path="./dropbox_team.d.ts" />
export = DropboxTypes.DropboxTeam;
export as namespace DropboxTeam;
```

`Dropbox-sdk.min.d.ts` (`Dropbox` class in a UMD module):

``` typescript
/// <reference path="./dropbox.d.ts" />
export = DropboxTypes.Dropbox;
export as namespace Dropbox;
```

`dropbox-sdk.js` (`Dropbox` class in a CommonJS module -- not sure why you distribute this when you have a UMD version!):

``` typescript
/// <reference path="./dropbox.d.ts" />
export = DropboxTypes.Dropbox;
```

Finally, for your Node module, there's `src/index.d.ts` which goes alongside `src/index.js` and defines all of your Node modules together. After adding a `typings` field to `package.json` that points to `src/index`, the TypeScript compiler _automatically_ picks up the definitions from the NPM module:

``` typescript
/// <reference path="../dist/dropbox.d.ts" />
/// <reference path="../dist/dropbox_team.d.ts" />

declare module "dropbox/team" {
  export = DropboxTypes.DropboxTeam;
}

declare module "dropbox" {
  export = DropboxTypes.Dropbox;
}
```

To properly bundle things, I added a `typescript-copy.js` script that NPM calls when you run `npm run build`. The script simply copies the TypeScript typings to the `dist` folder.

These are the files that must be maintained to provide complete TypeScript typings for all of your distribution methods. The files that are likely to change in the future are the templates, as you add/modify/remove SDK-specific interfaces.
  • Loading branch information
John Vilk committed Jan 21, 2017
1 parent ce64166 commit 459c59c
Show file tree
Hide file tree
Showing 5 changed files with 656 additions and 1 deletion.
3 changes: 2 additions & 1 deletion doc/builtin_generators.rst
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,8 @@ command-line interface (CLI)::
a generator module. Paths to generator modules must
end with a .stoneg.py extension. The following
generators are built-in: js_client, js_types,
python_types, python_client, swift_client
tsd_client, tsd_types, python_types, python_client,
swift_client
output The folder to save generated files to.
spec Path to API specifications. Each must have a .stone
extension. If omitted or set to "-", the spec is read
Expand Down
2 changes: 2 additions & 0 deletions stone/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@
_builtin_generators = (
'js_client',
'js_types',
'tsd_client',
'tsd_types',
'python_types',
'python_client',
'swift_types',
Expand Down
127 changes: 127 additions & 0 deletions stone/target/tsd_client.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
from __future__ import absolute_import, division, print_function, unicode_literals

import os
import re

# Hack to get around some of Python 2's standard library modules that
# accept ascii-encodable unicode literals in lieu of strs, but where
# actually passing such literals results in errors with mypy --py2. See
# <https://github.com/python/typeshed/issues/756> and
# <https://github.com/python/mypy/issues/2536>.
import importlib
import typing # noqa: F401 # pylint: disable=unused-import
argparse = importlib.import_module(str('argparse')) # type: typing.Any

from stone.generator import CodeGenerator
from stone.target.tsd_helpers import (
fmt_error_type,
fmt_func,
fmt_tag,
fmt_type,
)


_cmdline_parser = argparse.ArgumentParser(prog='tsd-client-generator')
_cmdline_parser.add_argument(
'template',
help=('A template to use when generating the TypeScript definition file.')
)
_cmdline_parser.add_argument(
'filename',
help=('The name to give the single TypeScript definition file to contain '
'all of the emitted types.'),
)
_cmdline_parser.add_argument(
'-t',
'--template-string',
type=str,
default='ROUTES',
help=('The name of the template string to replace with route definitions. '
'Defaults to ROUTES, which replaces the string /*ROUTES*/ with route '
'definitions.')
)
_cmdline_parser.add_argument(
'-i',
'--indent-level',
type=int,
default=1,
help=('Indentation level to emit types at. Routes are automatically '
'indented one level further than this.')
)
_cmdline_parser.add_argument(
'-s',
'--spaces-per-indent',
type=int,
default=2,
help=('Number of spaces to use per indentation level.')
)

_header = """\
// Auto-generated by Stone, do not modify.
"""

class TSDClientGenerator(CodeGenerator):
"""Generates a TypeScript definition file with routes defined."""

cmdline_parser = _cmdline_parser

preserve_aliases = True

def generate(self, api):
spaces_per_indent = self.args.spaces_per_indent
indent_level = self.args.indent_level
template_path = os.path.join(self.target_folder_path, self.args.template)
template_string = self.args.template_string

with self.output_to_relative_path(self.args.filename):
if os.path.isfile(template_path):
with open(template_path, 'r') as template_file:
template = template_file.read()
else:
exit('TypeScript template file does not exist.')

# /*ROUTES*/
r_match = re.search("/\\*%s\\*/" % (template_string), template)
if not r_match:
exit('Missing /*%s*/ in TypeScript template file.' % template_string)

r_start = r_match.start()
r_end = r_match.end()
r_ends_with_newline = template[r_end - 1] == '\n'
t_end = len(template)
t_ends_with_newline = template[t_end - 1] == '\n'

self.emit_raw(template[0:r_start] + ('\n' if not r_ends_with_newline else ''))
self._generate_routes(api, spaces_per_indent, indent_level)
self.emit_raw(template[r_end + 1:t_end] + ('\n' if not t_ends_with_newline else ''))

def _generate_routes(self, api, spaces_per_indent, indent_level):
with self.indent(dent=spaces_per_indent * (indent_level + 1)):
for namespace in api.namespaces.values():
for route in namespace.routes:
self._generate_route(
namespace, route)

def _generate_route(self, namespace, route):
function_name = fmt_func(namespace.name + '_' + route.name)
self.emit()
self.emit('/**')
if route.doc:
self.emit_wrapped_text(self.process_doc(route.doc, self._docf), prefix=' * ')
self.emit(' * ')
self.emit_wrapped_text('When an error occurs, the route rejects the promise with type %s.'
% fmt_error_type(route.error_data_type), prefix=' * ')
if route.deprecated:
self.emit(' * @deprecated')

self.emit(' * @param arg The request parameters.')
self.emit(' */')

self.emit('public %s(arg: %s): Promise<%s>;' %
(function_name, fmt_type(route.arg_data_type), fmt_type(route.result_data_type)))

def _docf(self, tag, val):
"""
Callback to process documentation references.
"""
return fmt_tag(None, tag, val)
130 changes: 130 additions & 0 deletions stone/target/tsd_helpers.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
from __future__ import absolute_import, division, print_function, unicode_literals

from stone.data_type import (
Boolean,
Bytes,
Float32,
Float64,
Int32,
Int64,
List,
String,
Timestamp,
UInt32,
UInt64,
Void,
is_alias,
is_list_type,
is_struct_type,
is_user_defined_type,
)
from stone.target.helpers import (
fmt_camel,
)

_base_type_table = {
Boolean: 'boolean',
Bytes: 'string',
Float32: 'number',
Float64: 'number',
Int32: 'number',
Int64: 'number',
List: 'Array',
String: 'string',
UInt32: 'number',
UInt64: 'number',
Timestamp: 'Timestamp',
Void: 'void',
}


def fmt_error_type(data_type, inside_namespace=None):
"""
Converts the error type into a TypeScript type.
inside_namespace should be set to the namespace that the reference
occurs in, or None if this parameter is not relevant.
"""
return 'Error<%s>' % fmt_type(data_type, inside_namespace)

def fmt_type_name(data_type, inside_namespace=None):
"""
Produces a TypeScript type name for the given data type.
inside_namespace should be set to the namespace that the reference
occurs in, or None if this parameter is not relevant.
"""
if is_user_defined_type(data_type) or is_alias(data_type):
if data_type.namespace == inside_namespace:
return data_type.name
else:
return '%s.%s' % (data_type.namespace.name, data_type.name)
else:
fmted_type = _base_type_table.get(data_type.__class__, 'Object')
if is_list_type(data_type):
fmted_type += '<' + fmt_type(data_type.data_type, inside_namespace) + '>'
return fmted_type

def fmt_polymorphic_type_reference(data_type, inside_namespace=None):
"""
Produces a TypeScript type name for the meta-type that refers to the given
struct, which belongs to an enumerated subtypes tree. This meta-type contains the
.tag field that lets developers discriminate between subtypes.
"""
# NOTE: These types are not properly namespaced, so there could be a conflict
# with other user-defined types. If this ever surfaces as a problem, we
# can defer emitting these types until the end, and emit them in a
# nested namespace (e.g., files.references.MetadataReference).
return fmt_type_name(data_type, inside_namespace) + "Reference"

def fmt_type(data_type, inside_namespace=None):
"""
Returns a TypeScript type annotation for a data type.
May contain a union of enumerated subtypes.
inside_namespace should be set to the namespace that the type reference
occurs in, or None if this parameter is not relevant.
"""
if is_struct_type(data_type) and data_type.has_enumerated_subtypes():
possible_types = []
possible_subtypes = data_type.get_all_subtypes_with_tags()
for _, subtype in possible_subtypes:
possible_types.append(fmt_polymorphic_type_reference(subtype, inside_namespace))
if data_type.is_catch_all():
possible_types.append(fmt_polymorphic_type_reference(data_type, inside_namespace))
return fmt_union(possible_types)
else:
return fmt_type_name(data_type, inside_namespace)

def fmt_union(type_strings):
"""
Returns a union type of the given types.
"""
return '|'.join(type_strings) if len(type_strings) > 1 else type_strings[0]

def fmt_func(name):
return fmt_camel(name)

def fmt_var(name):
return fmt_camel(name)

def fmt_tag(cur_namespace, tag, val):
"""
Processes a documentation reference.
"""
if tag == 'type':
fq_val = val
if '.' not in val and cur_namespace is not None:
fq_val = cur_namespace.name + '.' + fq_val
return fq_val
elif tag == 'route':
return fmt_func(val) + "()"
elif tag == 'link':
anchor, link = val.rsplit(' ', 1)
# There's no way to have links in TSDoc, so simply use JSDoc's formatting.
# It's entirely possible some editors support this.
return '[%s]{@link %s}' % (anchor, link)
elif tag == 'val':
# Value types seem to match JavaScript (true, false, null)
return val
elif tag == 'field':
return val
else:
raise RuntimeError('Unknown doc ref tag %r' % tag)
Loading

0 comments on commit 459c59c

Please sign in to comment.