Rewrite

[nonara]

Update (latest status)

Date: April 19th 2022

Much has happened in life and the world since planning this! I regret that I have not been able to get to this yet. To answer the immediate question — I still very much intend to make the new architecture.

At the moment, I am working to scale up my business to a point where I will be able to set aside some personal time to devote to this. As of this point, I cannot determine when that will be, however I am still actively maintaining the current architecture, as I know that many (including my own) build processes rely on it!

Background

I believe that major improvements can be gained by a new architecture for extending the typescript compiler. Additionally, it would be useful to move beyond the limitations of the current architecture.

For simplicity and visibility, we will keep the ts-patch name and the rewrite will be housed under a new major version.

Highlights of New Proposal

• Will not directly modify ts files (better for yarn)
• Will allow persistent alias overwriting (ie. can name the compiler tsc or something else, default is tspc)
• Supports custom patches which allow actual AST transformation of typescript libraries themselves, opening the door for packaged middleware
• Uses smart caching for core patch and custom patches applied to typescript modules (improves speed)
• Single plugin package can contain multiple transformers, patches, and language service plugins
• Will maintain support for current ttypescript style transformer packages

Updates will be posted as they happen.

Active Branch

https://github.com/nonara/ts-patch/tree/next

[craigphicks]

Note: This will be in beta until we can get a few testers to help check it out, so please say hello in this thread if you plan to test the beta! Help is appreciated!

Definitely want to try the beta. Please let me know.

Will not directly modify ts files (better for yarn)

Good move. Expect increased uptake of ts-patch.

Supports custom patches which allow actual AST transformation of typescript libraries themselves, opening the door for packaged middleware

Hot bug fixes too. Nice!

[nonara]

Updates

Have a look at what's new, below. Would greatly appreciate feedback!

Also — at first, I'll be devoting about 1-3 days per week on this. Hopefully I can increase in a few weeks.

🎉 Initial config is up!

Check out:

• TspConfig - Config file format
• TsModule - Config for individual ts modules

Everything marked @configurable is configurable from the config file, which will be <rootDir>/tsp.config.json. It will run without a config file, using defaults.

Caching Strategy

• A directory called .tsp will be created in the root directory. This will contain a cache folder, which will house cached patched versions of typescript.
• Cache will be intelligently updated when necessary (ie, new typescript installed, new plugins added or installed, etc).

Linking Strategy

• Direct links will be added to package.json (devDependencies or dependencies, based on where the original module is specified)

The strategy for this is based on configuration. By default the config for tsModule looks like this (see notes below for why this may change):

{
  "tsModule": {
    "module": "typescript"
  }
}

Assuming we have:

package.json

{
  // ...
  "devDependencies": {
    "typescript": "^4.3.5"
  }
}

Our package file is automatically transformed to be the following:

package.json

{
  // ...
  "devDependencies": {
    "@ts-patch/patched-typescript": "file:./tsp/cache/typescript",
    "typescript": "^4.3.5",
  }
}

Also, this does point the tsc and tsserver bin links to our patched version.

Note: For those of you thinking we should use link: instead, I agree. In effort to support both NPM and Yarn, we need to use file, however, tsp will magically fix the node_modules path to be a symlink after install. We also do this to ensure the binary links for our patched version take precedence.

What if we want to use the typescript module name?

For more advanced users, you can configure your module to expropriate the module name by setting patchedModuleName to the same name as module.

In this example, we want to patch all libraries, and we want node_modules/typescript to point to our patched installation.

{
  "tsModule": {
    "module": "typescript",
    "patchedModuleName": "typescript"
    "libraries": [ "tsserver", "tsserverlibrary", "typescript", "tsc", "typescriptServices" ]
  }
}

With the same starting package.json as before, our package file is transformed to be:

package.json

{
  // ...
  "devDependencies": {
    "typescript": "file:./tsp/cache/typescript",
    "@ts-patch/base-typescript": "typescript"
  }
}

Note

I have been back and forth on the idea of making expropriation of the module name the default action.

On the one hand, it might get a few grumbles from people for changing the typescript name, but it will ultimately make it much easier for less experienced users to simply plug and play, as opposed to having to learn how to separately configure vs-code, webpack, webstorm, etc. etc.

If you'd like to push back on this, feel free to speak up. I'm glad to hear all arguments for or against.

[nonara]

It just occurred to me that I built the ability to patch other modules into v1, in case the need ever arose.

I can't promise that it will work, but try ts-patch patch tsserver.js in your project dir. I just checked and it did apply the patch. Theoretically, it should work as it would with ttypescript. If you give it a go, let me know if it works or crashes! If it's good, I'll add it in early next week as an officially supported library.

I'd really like to try your beta and see if I can patch the language service in vs2019

I'd definitely appreciate the beta testing, thanks! Though, I'm not entirely sure I understand what you mean by "patching the language service".

As a bit of background info, ttypescript / ts-patch apply a patch to createProgram in order to hook into the emit process. Essentially, it makes it so that you can supply AST transformers in the tsconfig.json -> plugins section, which is otherwise reserved for language service plugins. These transformers only get run during the file emit process that occurs after compilation.

A language service plugin is primarily created to extend hints, code completion, etc. It can also be used to remove diagnostics. Where that becomes powerful is in that it could allow you to augment the language itself. (Example: making decorators work on functions, object variable declarations, and object properties)

This is actually technically possible already with v1 by using a program transformer and a language service plugin, together, but the configuration would be a little cumbersome. v2 will mitigate that, as one package can contain config for multiple language service plugins, transformers, and even transformers for the TS compiler libraries as well (called patches).

Can tspatch patch in language plugins as well as ast transformers?

TS natively supports language service plugins! You can just add them to the plugins section of tsconfig.json. You can have both tsp and LS plugins at the same time.

I'm assuming that the transformers need to be specified in the config when building the compiler, but do not need to be specified when using the compiler, is this correct?

Not sure I understand what you mean, here. If you mean for v2, we'll have single modules (packages) which house any number of language service plugins, transformers, and/or patches.

We are also going to move away from specifying these packages in tsconfig (though legacy support will remain for v1 and tts packages). I'm planning to specify them in tsp.config.json -> extensions, and have cli commands like tsp add <module> / tsp remove <module>

In many ways tsp will function like the yarn cli app, keeping everything clean and wired up properly when executed. It will also track file modification dates, config hashes, etc, to make sure that it only rebuilds cached ts libs when necessary.

Hope that helps answer your questions!

[Griffork]

I think I've misunderstood something basic about using ts-patch - although this doesn't change my ability to use it.

Essentially I was hoping to be able to ship a version of tsc (internally) that is not only patched to be able to use transformers, but is also shipped with several transformers, a language service plugin and an addition to lib.d.ts built-in.

It seems that ts-patch actually generates a version of tsc that can then be extended using a config file. While this is fine, it does mean that I need to copy my transformers and configuration settings into every project that's using my custom compiler setup.

As for "patching the language service in VS2019" patch might have been the wrong term. I essentially was going to make a custom sdk featuring a version of tsc.js and tsserver.js that have hooks for the ast transformers and language plugin following this suggestion from a typescript dev: microsoft/TypeScript#41355 (comment).

Would there be a way for me to copy my transformers and config into the outputted cache so that it's automatically included when using the generated tsc.js or tsserver.js of the sdk I intend to make?

[craigphicks]

@Giffork @nonara

Giffork -

It sounds like you a progrmatic interface for ts-patch would suit your needs.
A conceptual suggestion

import tsPatch from `ts-patch`

const origReadConfig = tsPatch.readConfig

tsPatch.readConfing=(...)=>{ /* put your transforms first in line */ }

tsPatch.run()

[nonara]

@Griffork Sounds like v2 would suit that use case nicely.

You could do it the standard way with a package that contained all of your transformers and your language plugin. In your package, you can include a setup script, which is triggered post-install. The script would init tsp config, perform the patch, and add your lib.d.ts.

With that route, you simply install ts, then install your package, and your project will be ready to go. For most, I would recommend that route, because it allows TS to update, and the patching persists.

However, if you wanted to bake it in directly, you could also create a module with a patch. This would let you build your transformers and LS directly into the ts libraries. You could then publish the end result to a private repository, which would be your customized version of TS. I could see this making sense for some organizations! It is also very automatable. You could setup a cron script on a build server to automate building when a new version of TS comes out. You could probably use the code at ts-expose-internals as a starting point for that.

The second route will take a little know-how in the compiler, as patches hand off the ability to modify the TS source to you.

[Griffork]

The patch is what I need, I need the basic config to be hard-baked into the output .tsc and .tsserver since the visual studio typescript plugin is closed source and I can't edit it, and it's hard-coded to use tsc and tsserver directly.

I do however agree with the fully-built-in transforms and language plugins being useful for organisations as if I can get everything working well with the visual studio typescript service I'd actually be considering proposing using my transformers at work (at the moment the process is too hacky to scale-up to more developers).

I'll look into that link and see what I can do.

[AlCalzone]

I stumbled across this while researching ttypescript and finding out that it is unmaintained. I would be interested in trying the v2 beta, because there are a few existing transforms which I could really use in a lot of places.

I have some requirements though where I'm not sure that ts-patch fulfills them:

1. support for yarn v2+ with PnP
2. (more or less) drop-in replacement for typescript

[nonara]

I would be interested in trying the v2 beta

Great! I would appreciate it!

1. support for yarn v2+ with PnP

I looked over the config a bit. It looks like all packages are stored in cache in zip files. I'm not sure if yarn2 supports link: or file: prefixes for dependencies, however, it does seem we can use resolutions to specify a path to a local dependency. Bottom line is, it looks possible, so I plan to support it. I will build it with support in mind. That said, I'll likely get the beta built before adding y2 support.

2. (more or less) drop-in replacement for typescript

v1 has this. v2+ will also! Two steps. Add dev dep, add prepare script, and it's ready to go.

[AlCalzone]

I think the complicated part of supporting yarn v2 is that it patches typescript itself to add PnP support. I think this only affects the language server for VSCode, but maybe this interferes with what you do.

Depending on what ts-patch does, either directly implementing the PnP API or using the patch: protocol to automatically patch the typescript dependency seem like a good approach.

[nonara]

FYI — I was able to get v1 working with yarn3 via patch protocol.

Steps here: #60

[geroyche]

i was trying to use ts-patch as a replacement for ttypescript today.

given that this seems to be a stated goal, please consider the Webpack use case via ts-loader
https://github.com/cevek/ttypescript#webpack

there is no such configuration info in this repo and simply swapping out ttypescript there with ts-patch did not work

ts-loader: Using ts-patch@undefined

and a bit later

Module build failed (from ./node_modules/ts-loader/index.js):
TypeError: Cannot read property 'fileExists' of undefined

[nonara]

Tsp directly patches typescript. You should not need any special configuration info. Simply patch typescript and make sure webpack uses typescript as its compiler.

[yankeeinlondon]

@nonara I have really loved using ttypescript along with typescript-transform-paths to address path aliases in code and types. I noticed today that while it still works for building, it now fails when ttsc is in watch mode. Anyway, do you think ts-patch would be a good suitable replacement for this use case? If so, are there any "reciepes" yet that folks are using to do their magic?

[nonara]

The goal of v2 is to allow for new and more advanced types of plugins. v1 is perfectly suited for tstp! No special config needed. You just use the same config as you would for ttypescript.

So to get going follow:

1. Add dev dep ts-patch
2. Run ts-patch install
3. Add prepare script to package.json -> ts-patch install -s (this keeps it persisted after updates, yarn install, etc)

After that, tsc will natively process the transformers in your tsconfig.json

[yankeeinlondon]

thanks for bringing it down to my level ... i love the idea of parsing, lexing, and AST but i have a tendency to footgun things pretty quickly when I dig into it. :)

[nonara]

Sure thing! Hope it works for you! And I understand. It gets better with patience and practice, but you can do a lot of cool things, so it's a great tool to have in the toolbox!

[pmrotule]

I'm in the same situation, but I have to say, I prefer the way ttypescript works: instead of patching, you just use a different cli (ttsc). I don't want to add an extra project setup step and would rather have it ready to be called after running yarn install.

It's unclear in the README whether I can skip the second step (ts-patch install) or not. Is it sufficient to add ts-patch to our dependencies together with the prepare script? Will my team members be able to run tsc right after running yarn install without having to run ts-patch install?

And thanks for your work, this is incredibly useful!

[nonara]

I don't want to add an extra project setup step and would rather have it ready to be called after running yarn install.

The new version is planned to have the ability to write a separate cli and preserve the original typescript install. I am considering adding a feature that allows on-the-fly patching to mimic ttsc. I haven't made a decision on that yet. It adds drag to compile time and does a lot of unnecessary work each time, so it's generally something I prefer to avoid, but I'm still mulling it over.

Is it sufficient to add ts-patch to our dependencies together with the prepare script?

prepare script executes after every yarn install. This ensures it stays up to date. If you have the devDependency set and prepare script in place, it will patch automatically after yarn install, so you won't need to run it manually.

And thanks for your work, this is incredibly useful!

Thank you! I'm very glad to hear it's helpful!

[yankeeinlondon]

I've found ts-patch works better in every way to ttypescript even if patching tsc initially made me a little nervous.

[joshAg]

hey, are you still looking for beta testers? I can't tell if this shipped in the v2 or this is ongoing since the readme says the patch is still coming.

I'm using ttypescript for type-script-is and ran into an issue with incremental builds failing like the bug mentioned here, so it looks like we finally have to move off that project if we want incremental builds to keep working. In addition to that we run our tests using ts-node configured to use ttypescript as the compiler.

[nonara]

@MohamedLamineAllal It's actually in process. I have it speced out. Due to TS 5 changes, we'll need to have it soon, so it's in my near queue. I had a few people who said they'd help, but unfortunately they dropped out.

[MohamedLamineAllal]

I see, my bad. And thank you a lot for all the great work. I wish i was able to help right now. I'm in a similar situation. Just about to finish the re-write (re-design) of laravel-mix-glob (a tool that I don't even use). struggling with time. And have to focus on starting some new business projects I was preparing for.

[nonara]

No problem. OSS is certainly challenging, but it's rewarding. Good luck with your rewrite!

[MohamedLamineAllal]

@nonara Thank you. About to publish. May you share the link to the spec if it's available? I can help but at a slower pace. I can work efficiently. Once I read the spec and check typescript architecture. I can commit drafts for the parts that i'll be handling. And ill be sharp about any notes.

[MohamedLamineAllal]

Thank you again. Email sent.