Skip to content

Latest commit

 

History

History
285 lines (208 loc) · 10.2 KB

go-target.md

File metadata and controls

285 lines (208 loc) · 10.2 KB
Raw

ANTLR4 Language Target, Runtime for Go

Changes from ANTLR 4.12.0

Please see Changes in ANTLR Go runtimes, but in summary:

  • The Go runtime is now stored in the repo antlr4-go/antlr - change your import, remove the old location from go.mod and use go get github.com/antlr4-go/antlr
  • There are some new @actions for adding to the generated import statements and recognizer structure
  • The recognizer rules are no longer called via an interface, for performance reasons
  • Memory usage improvements
  • Performance improvements
  • Documentation in true Go format
  • Git tags now work correctly with go tools

Removal of non v4 code

Prior to the release of the v4 tagged runtime, the source code for the Go runtime module existed at runtime/Go/antlr, which is the pre-v4 version of the code, and also under runtime/Go/antlr/v4. If your project was not using modules, you could merely sync to the latest hash in the master branch and use the code. This has changed.

As of the current release, the source code for the Go runtime module has been moved to its own repo in its own GitHub organization. As of now, you can still use the code without modules, but you must use the code in the repo at https://github.com/antlr4-go/antlr instead of the code in the main ANTLR repo.

This is for historic reasons as the code was originally written before modules were a thing, and the go runtime source was - and the maintainer's version still is - a part of the monorepo that is antlr/antlr4/....

Note that I am unable to properly deprecate the go.mod in the non-V4 directory, for hte same reason that I cannot use tag the v4 module at this depth in the source tree.

We strongly advise you to use modules, though it is not required. See below for more information.

ANTLR Go Maintainer: Jim Idle - Email: [email protected]

First steps

1. Install ANTLR4

See: The getting started guide.

2. Get the Go ANTLR runtime

Each target language for ANTLR has a runtime package for running a recognizer generated by ANTLR4. The runtime provides a common set of tools for using your parser/lexer. Note that if you have existing projects and have yet to replace the v1.x.x modules with the v4 modules, then you can skip ahead to the section Upgrading to v4 from earlier versions

The Go runtime uses modules and has a version path of /v4 to stay in sync with the runtime versions of all the other runtimes and the tool itself.

Setup is the same as any other module based project:

$ cd mymodproject
$ go mod init mymodproject

After which, you can use go get, to get the latest release version of the ANTLR v4 runtime using:

go get github.com/antlr4-go/antlr

If your project was already using the v4 runtime from the main ANTLR repo, then you can upgrade to the latest release by removing the github.com/antlr/antlr4/runtime/Go/antlr/v4 reference in your module, and changing the associated import in your project code. The following script may be useful in changing your imports:

find . -type f \
    -name '*.go' \
    -exec sed -i -e 's,github.com/antlr/antlr4/runtime/Go/antlr/v4,github.com/antlr4-go/antlr/v4,g' {} \;

Note that the import package still imports with the final path as antlr, so only the import statement itself needs to change.

If you are already using the repo and import github.com/antlr4-go/antlr/v4 then you can upgrade to the latest version using the standard.

go get -u github.com/antlr4-go/antlr

If you have not yet upgraded existing projects to the /v4 module path, consult the section Upgrading to v4 from earlier versions

The ANTLR runtime has only one external transient dependency, and that is part of the go system itself:

golang.org/x/exp

A complete list of releases can be found on the release page. The Go runtime will be tagged using standard Go tags, so release 4.13.2 in the antlr4-go/antlr repo, will be tagged with v4.13.2 and go get will pick that up from the ANTLR repo.

3. Configuring go generate in your project

In order to promote the use of repeatable builds, it is often useful to add the latest tool jar to your project's repo and configure a generate.sh and generate.go file. You can of course globally alias the java command required to run the tool. Your own CI and dev environment will guide you.

Here is how you can configure go generate for your project, assuming that you follow the general recommendation to place the ANTLR grammar files in their own package in your project structure. Here is a general template as a starting point:

	.
	├── myproject
	├── parser
	│     ├── mygrammar.g4
	│     ├── antlr-4.13.2-complete.jar
	│     ├── generate.go
	│     └── generate.sh
	├── parsing  # Generated code goes here
	│     └── error_listeners.go
	├── go.mod
	├── go.sum
	├── main.go
	└── main_test.go

Make sure that the package statement in your grammar file(s) reflects the go package the go code will be generated in. The generate.go file then looks like this:

	package parser

	//go:generate ./generate.sh

And the generate.sh file will look similar to this:

	#!/bin/sh

	alias antlr4='java -Xmx500M -cp "./antlr-4.13.2-complete.jar:$CLASSPATH" org.antlr.v4.Tool'
	antlr4 -Dlanguage=Go -no-visitor -package parsing *.g4

From the command line at the root of your package - the location of the go.mod file - you can then simply issue the command:

	go generate ./...

If you have not yet run a go get, you can now run go mod tidy and update your

4. Generate your parser manually

You use the ANTLR4 "tool" to generate a parser. These will reference the ANTLR runtime, installed above.

Suppose you're using a UNIX system and have set up an alias for the ANTLR4 tool as described in the getting started guide.

To generate your go parser, you'll need to invoke:

    antlr4 -Dlanguage=Go MyGrammar.g4

For a full list of antlr4 tool options, please visit the tool documentation page.

Upgrading to /v4 from the default path

NB: While switching to new module path would normally imply that the public interface for the runtime has changed, this is not actually the case - you will not need to change your existing code to upgrade. The main point of the repo change is so that git tagging works with the ANTLR Go runtime and the go tools.

Prior to release v4.11.0 the Go runtime shipped with a module but the module had no version path. This meant that the tags in the ANTLR repo did not work, as any tag above v1 must refer to a matching module path. So the command go get github.com/antlr/antlr4/runtime/Go/antlr would just bring in whatever was the HEAD of the master branch. While this kind of worked, it is obviously subject to problems and does not fit properly with the idiomatic ways of Go.

As of v4.13.0 the runtime code exists in its own repo, github.com/antlr4-go/antlr, and is correctly tagged. However, this means you need to perform a few simple actions in order to upgrade to the /v4 path.

  • Firstly, make sure that you are using an ANTLR tool jar with a version number of 4.13.0 or greater.
  • Next you replace any mention of the old (default) path to ANTLR in your go source files.
  • If using modules, remove any existing reference to the ANTLR Go runtime
  • Now regenerate your grammar files either manually or using go generate ./... (see above)
  • Consider whether you can move to using modules in your project

A quick way to replace the original module path references is to use this script from your module's base directory:

find . -type f \
    -name '*.go' \
    -exec sed -i -e 's,github.com/antlr/antlr4/runtime/Go/antlr,github.com/antlr4-go/antlr/v4,g' {} \;

After performing the steps above, and you are using modules issuing:

go mod tidy

Should fix up your go.mod file to reference only the v4 version of the ANTLR Go runtime:

require github.com/antlr/antlr4/runtime/Go/antlr/v4 v4.13.0

From this point on, your go mod commands will work correctly with the ANTLR repo and upgrades and downgrades will work as you expect. As will branch version such as @dev

Referencing the Go ANTLR runtime

You can reference the go ANTLR runtime package like this:

import "github.com/antlr4-go/antlr/v4"

Complete example

Suppose you're using the JSON grammar from https://github.com/antlr/grammars-v4/tree/master/json placed in the parser directory and have initialized your go mod file.

Then, invoke antlr4 -Dlanguage=Go JSON.g4. The result of this is a collection of .go files in the parser directory including:

json_parser.go
json_base_listener.go
json_lexer.go
json_listener.go

Another common option to the ANTLR tool is -visitor, which generates a parse tree visitor, but we won't be doing that here. For a full list of antlr4 tool options, please visit the tool documentation page.

We'll write a small main func to call the generated parser/lexer (assuming they are separate). This one writes out the encountered ParseTreeContext's. Assuming the generated parser code is in the parser directory relative to this code:

package main

import (
	"github.com/antlr4-go/antlr/v4"
	"./parser"  // Note that with modules you may not be able to use a relative immport path
	"os"
	"fmt"
)

type TreeShapeListener struct {
	*parser.BaseJSONListener
}

func NewTreeShapeListener() *TreeShapeListener {
	return new(TreeShapeListener)
}

func (this *TreeShapeListener) EnterEveryRule(ctx antlr.ParserRuleContext) {
	fmt.Println(ctx.GetText())
}

func main() {
	input, _ := antlr.NewFileStream(os.Args[1])
	lexer := parser.NewJSONLexer(input)
	stream := antlr.NewCommonTokenStream(lexer,0)
	p := parser.NewJSONParser(stream)
	p.AddErrorListener(antlr.NewDiagnosticErrorListener(true))
	tree := p.Json()
	antlr.ParseTreeWalkerDefault.Walk(NewTreeShapeListener(), tree)
}

Fix up your go.mod file:

go mod tidy

This one expects the input to be passed on the command line:

go run test.go input

The output is:

{"a":1}
{"a":1}
"a":1
1