This is a guide to writing a client for TabNine.
TabNine is invoked by the text editor plugin (the "client") as a subprocess.
The client communicates with TabNine through standard input and output. TabNine does not write to standard error.
Each request to TabNine is a JSON object followed by a newline, encoded as UTF-8. There must be no newlines in the JSON object.
TabNine will produce exactly one response for each request. A response consists of a JSON object followed by a newline.
Each line of input to TabNine corresponds to exactly one line of output. If a line of input is malformed, the corresponding output will be the JSON object null.
It's helpful when debugging to view TabNine's log output. You can enable logging by passing --log-file-path as an argument to the TabNine binary.
For example, this lets you see error messages explaining why the request is malformed.
Run dl_binaries.sh (in this repository) to download the most recent version of TabNine.
Find the TabNine binary in binaries/<version>/<platform>.
Run TabNine in your terminal and paste the following command as input:
{"version": "1.0.0", "request": {"Autocomplete": {"before": "Hello H", "after": "", "region_includes_beginning": true, "region_includes_end": true, "filename": null, "correlation_id": 1}}}
You should see the following output:
{"old_prefix":"H","results":[{"new_prefix":"Hello","old_suffix":"","new_suffix":""}],"user_message":[],"correlation_id":1}
A few things to note:
- The protocol is versioned. The protocol versions are the same as TabNine versions. To guarantee forward compatibility with future versions of TabNine, pass the current TabNine version (or any previous version) as the protocol version.
- The completion position is specified by giving the strings before and after the cursor. If these strings are very long, you can truncate them. In this case you should set
region_includes_beginningorregion_includes_endtofalseto indicate that the strings do not extend to the beginning or end of the file, respectively. - The recommended threshold for truncation is 100 KB.
- Autocomplete responses contain a field
user_messagewhich is a message that should be displayed to the user. For example, this is used to inform the user when a language server fails to start, or when TabNine hits the index size limit. - The
correlation_idfield can be passed as a verification token that will be returned in response.
You must preserve the directory structure created by dl_binaries.sh, or else TabNine's automatic updating will not work.
When TabNine updates, it downloads the new version in the same location as the current binary but with a different version directory. For example, if the current binary is at bin/1.0.5/x86_64-apple-darwin/TabNine, and TabNine downloads version 1.0.7, it will be installed at bin/1.0.7/x86_64-apple-darwin/TabNine.
Once TabNine downloads an update, it terminates. You should restart TabNine when it terminates, up to some maximum number of restarts (say, 10).
In recent versions, TabNine also creates a .active file in parallel to the version folders. This file contains the version the plugin should run.
To start TabNine, read the .active file content and run the binary under that version. If no such file exists, list the binaries directory and choose the most recent version. Here is Python code similar to the Sublime Text client which does this:
def parse_semver(s):
try:
return [int(x) for x in s.split('.')]
except ValueError:
return []
def get_arch():
if is_apple_m1():
return "arm64"
return sublime.arch()
def get_tabnine_path(binary_dir):
def join_path(*args):
return os.path.join(binary_dir, *args)
translation = {
("linux", "x64"): "x86_64-unknown-linux-musl/TabNine",
("osx", "x64"): "x86_64-apple-darwin/TabNine",
("osx", "arm64"): "aarch64-apple-darwin/TabNine",
("windows", "x32"): "i686-pc-windows-gnu/TabNine.exe",
("windows", "x64"): "x86_64-pc-windows-gnu/TabNine.exe",
}
platform_key = sublime.platform(), get_arch()
platform = translation[platform_key]
versions = []
# if a .active file exists and points to an existing binary than use it
active_path = join_path(binary_dir, ".active")
if os.path.exists(active_path):
version = open(active_path).read().strip()
version_path = join_path(binary_dir, version)
active_tabnine_path = join_path(version_path, platform)
if os.path.exists(active_tabnine_path):
versions = [version_path]
# if no .active file then fallback to taking the latest
if len(versions) == 0:
versions = os.listdir(binary_dir)
versions.sort(key=parse_semver, reverse=True)
for version in versions:
path = join_path(version, platform)
if os.path.isfile(path):
add_execute_permission(path)
print("Tabnine: starting version", version)
return pathAs of late 2020, apple released their new M1 processors, based on the arm64 architecture. When running on this platform, it is advised to run the aarch64-apple-darwin binary. Running the x86_64 binary will work, using the Rosetta translation environment. However, Tabnine will not be able to download and load the local deep model, because it relies on some intel specific cpu intrinsics (FMA, AVX2) that do not exist when running under Rosetta.
Some editors already have native support for arm64, while others rely on rosetta to function. Either way, it is highly
recommended to run the aarch64 binary on m1 platforms. It can get a little tricky correctly detecting being on m1 when
running under rosetta. It usually requires calling some form of uname or similar. Here's how we do it in sublime:
import platofrm
if sublime.platform() == "osx":
if "ARM64" in platform.version().upper():
return "arm64"Even running uname on rosetta can get a little tricky, so test it well.
Each request to TabNine must be a JSON object followed by a newline. The JSON object must be a dictionary containing the fields version and request. version should be a string corresponding to a TabNine version. The field request must be a dictionary with a single key. The key must be one of the following:
AutocompletePrefetchGetIdentifierRegex
The value associated with the key must be of the corresponding type. For example, if the key is Autocomplete, the value must have type AutocompleteArgs.
TabNine's response will be of the corresponding type. For example, if the key was Autocomplete, the response will be of type AutocompleteResponse.
null fields can be omitted in requests.
AutocompleteArgs {
before: string,
after: string,
filename: string | null,
region_includes_beginning: bool,
region_includes_end: bool,
max_num_results: int | null,
correlation_id: int | null,
}
max_num_results must be positive. More information about Autocomplete requests is in the "Getting Started" section.
correlation_id field can be passed as a verification token that will be returned in response.
PrefetchArgs {
filename: string
}
You can use this API call to make TabNine add a file to its index even if the user hasn't requested completions in the file yet.
GetIdentifierRegexArgs {
filename: string | null
}
This gives the regex used by TabNine to parse identifiers for the provided file.
AutocompleteResponse {
old_prefix: string,
results: ResultEntry[],
user_message: string[],
correlation_id: int | null,
}
ResultEntry {
new_prefix: string,
old_suffix: string,
new_suffix: string,
kind: CompletionItemKind | null,
detail: string | null,
documentation: Documentation | null,
deprecated: bool | null
}
CompletionItemKind and Documentation are specified by the Language Server Protocol.
The Language Server Protocol also specifies the meanings of kind, detail, documentation, and deprecated. Each of these fields will be omitted from the response if they are null.
The behavior of autocompletion is as follows: when the user selects the result, the text before the cursor should be old_prefix, and it should be replaced by new_prefix. The text after the cursor should be old_suffix, and it should be replaced by new_suffix.
For example, suppose the current state of the editor is as follows (where | represents the cursor):
if (x == |)
Suppose TabNine wants to suggest if (x == 0) {. Then the fields will be as follows:
old_prefix = ""new_prefix = "0) {"old_suffix = ")"(we need to delete the closing bracket after the cursor because it's already included innew_prefix)new_suffix = "}"(inserting a matching closing bracket for the{)
After the completion is accepted, the editor will look like this:
if (x == 0) {|}
The responses to the other queries are:
PrefetchResponse = null
GetIdentifierRegexResponse = string