Compile / edit time pluggable analyzers like C#

[Ciantic]

It would be nice to have similar plugabble compile time / editing time analyzer plugins as introduced in C# in Build 2015, see the video.

Analyzer is a library that can do additional syntax checks, tag lines as errors / warnings, and are interpreted by language service.

I can imagine one could do similar things with it as Facebook's flow does e.g. own nullable checking just by writing own analyzer. This would be configurable from tsconfig, maybe one could whitelist the files each analyzer works on. This way the nullability check analyzer would only complain about the code I want.

[DickvdBrink]

I would definitely use this if it became available!
For know I always create a simple nodejs application which uses the TypeScript compiler to analyse the things I care about, but this would be a nice improvement!

[jbondc]

Thanks for sharing, seems like all the building blocks are almost in place. async may be a blocker though for TypeScript.
You'd need a cancellation token or something similar for this to work with threads.

+1 to Visual Studio debugging Visual Studio #2656 😮
This mixing of syntax model + semantic model is very nifty stuff.

[Deathspike]

This would be great. It would also open good integration for things like design-by-contract and of course nullable checking. I would love this feature.

[CyrusNajmabadi]

As someone who was heavily involved in this work in https://github.com/dotnet/roslyn, i'm definitely very interested in enhancing TypeScript in this space.

Unfortunately, there are a lot of hard issues here that make this a non-trivial for us. For one thing, our compiler has been designed for raw speed on a JS engine, rather than extensibility. To be very fast on JS engines, you need to avoid lots of abstractions and complex layering in your objects. So, for example, our AST nodes work very differently than Roslyn's do. This includes:

1. Our ASTs are not full fidelity. You don't get all information from the source code in the AST, and it's not possible to fully recreate the source code from the AST. This is problematic for analyzers that want to have fine-grained understanding of your code syntax, as well as fine-grained ability to manipulate it.
2. Our ASTs do not follow a consistent model. You can get things like delimited lists without delimiters in our model. In Roslyn, this can't happen.
3. Our ASTs are not persistent. This is problematic for things like building analyzers/fixers. You want immutability so that you can create speculative forks of your code that you can then format/simplify/continue-to-analyze/etc.
4. Our ASTs are not immutable and do not prevent you from performing operations on them that will break our compiler. Indeed, in our compiler almost everything in our data model is a public mutable field. This is great for perf, and really bad for a safe API. In Roslyn, everything is immutable. When you change a tree, you're not affecting any existing tree, or any compilation that depends on that tree. Everything in the system remains consistent and sane. in our model if you go an edit our trees, then many things will break. All our assumptions about how the tree relates to the source code will no longer be true. Data we've stored in the tree may or may not be valid post your edit, and we'll have no idea. Our incremental parser will likely be missing very important information it needs to work with this tree. Someone may change some piece of data out from underneath them (esp. if we did this in an async manner). etc. etc.

Furthermore, the JavaScript language and engines are missing many useful features that Roslyn heavily leans on for performance. For example, Roslyn takes huge advantage of structs in order to provide their dual Red Green Trees. By using structs they can effectively have two different tree representations, with only marginal memory/heap/gc overhead.

All of these deviations we made from Roslyn (which we understood and acknowledged when we did this work), were for performance reasons. They allowed us to provide a fast and lean compiler initially. Though they do now impact our ability to better supply things like analyzers/fixers in as simple a manner as Roslyn provides them. My hope is that as time moves on we can get these same sorts of features, while still maintaining our goal of a fast and lean compiler.

[nbellowe]

I just wanted to post something I posted in the closed post here, just to give a reference of a couple possible uses. Begin quote ->

Here are a few examples of possible extensions:

Add language support for defining properties with getters and setters.

Too big of a change, possibly not useful enough to warrant a change in Typescript, but it would be super cool if I could make a seperate npm package called typescript-properties, and anyone could pull that, and utilize it in there apps. This could maybe be an example that decorators could take care of, not sure.
property hello:T
compiles to
_hello:T
get hello():T { return this._hello }
set hello(v:T){ this._hello = v }
Add preprocessing to the code that doesn't break source mapping.

    #IF(DEBUG)
       expr;
    #ENDIF

While this can be accomplished with external tools, it breaks source mapping. With a complex build, like:

Original TS
Post-pre-processed TS
Compiled TS (Javascript) and sourcemaps to PPP-TS
Single-file JS (may need to be ordered in optimal way) & source map to PPP-TS if the tool supports input source mapping.
Minified Javascript & source map & source map to PPP-TS if the tool supports it.
-- note(UglifyJS can optimize out constants, so defining a const DEBUG=true|false can optimize out debug code as well. if(DEBUG) expr;)

[alexeagle]

👍
Check out http://error-prone.info where we did this for the Java compiler. I think we can find lots of statically analyzable bugs, and doing so in the compiler is much better than a standalone tool because you are guaranteed that everyone on the team complies with the rules.

[andersekdahl]

Just wanted to chim in with a use case: We use CSS modules which means that we import CSS files in TypeScript. We get an object map back from that import where the keys are class names from the CSS file. It would be incredible if it was possible to extend TypeScript to give autocompletion and compile safety for this.

[alexeagle]

I think tslint would be better architected as a compiler plugin. It better supports the ability to share the typecheckers understanding of the program without duplicate logic to read tsconfig.json and possibility of getting different answer from the tsc run by the build system. It also allows tslint to produce errors that appear in the editor and block the regular compilation, so the user experience is the same as first-party checks like --noImplicitReturn.

Can you guys discuss a bit whether this is possible?

[mhegazy]

Can you guys discuss a bit whether this is possible?

it is possible. we have had a prototype for this 2 years back. we have just been busy with other things since.

[alexeagle]

Okay, what needs to happen? Would it help to pick up that prototype, or is
the core team not yet sold on the idea to add this now?

On Thu, Mar 31, 2016 at 12:19 PM Mohamed Hegazy [email protected]
wrote:

Can you guys discuss a bit whether this is possible?

it is possible. we have had a prototype for this 2 years back. we have
just been busy with other things since.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#3003 (comment)

[vladima]

Prototype was built against the old codebase so it cannot be directly applied now. There are plans to support user-defined diagnostics, at this point it is mostly the question of prioritization.

[chrisber]

I think this topic is related to #6508 which relates to Roadmap
2.0 Investigate Language Service extensibility

[alexeagle]

My understanding is that there are four extension points (at least):

• Listing sources. Angular 2 has an offline template compiler that produces extra .ts files as inputs to the compilation, based on the Decorators appearing in the presented inputs. This is very much like Java's annotation processors: https://docs.oracle.com/javase/7/docs/api/javax/annotation/processing/Processor.html - I don't think there is an issue filed for this yet.
• Static-checking. (this issue) Allow programs like tslint, tsvet (Lint Programs as a Whole palantir/tslint#680 (comment)), http://github.com/alexeagle/tsetse to hook after the type checker and produce additional diagnostics
• Language service. (TypeScript extensibility #6508) which answers queries for code completion, jump to def'n, and so on. Arguably these want to produce diagnostics as well, but that can use the static-checking extension point above.
• Emit pipeline. When https://github.com/Microsoft/TypeScript/tree/transforms is merged to master, we'll want to be able to add AST transformations both before and after the pipeline executes. We already do both of these with https://github.com/angular/sickle which produces Closure Compiler-friendly ES6 code using the goog.module module syntax.

As I see it, we need a higher-level discussion of how the compiler could be configured to know about plugins, how the actual loading of the plugin occurs, how the plugin declares which of the 4+ extension points it wants to hook, how to defend against abusive plugins (intentional or unintentional), and so on.

As it stands, Angular is writing multiple independent programs which wrap tsc, which is hard for users to configure and inefficient.

@mhegazy would it be helpful to write up a rough proposal for this, or do you already have some partly-formed ideas about it?

[mhegazy]

@billti has a write-up and a prototype on the basic extensibility story. i.e. discovery, loading and creating extensions in the compiler and the language service in #6508. Once that is in, we should be able to add more extension points as you mentioned and expose them to the extensions.

[giggio]

Was this completed when #6508 was delivered? I still can't find guidance on how to write analyzers and fixes for TypeScript projects. Is it at all possible?
I want to build a terminal app which will load a project, run some custom analyzers that will produce diagnostics, run some code fixes for those diagnostics that will edit the code file and exit. How would that be done?

[weswigham]

Yes and no depending on your usecase. If you want to provide an alternative language service like angular and vue do, along with any additional diagnostics that entails; then yes - you have hooks available to wrap and reexpose the LS. If you wanted to, say, add lint-like errors on the command line, then no; we've yet to merge any plugin model that supports that.