ESM support

[mattwynne]

Switch over to import from require.

I followed https://jestjs.io/docs/ecmascript-modules for Jest config.

[vercel]

@mattwynne is attempting to deploy a commit to the Probot Team on Vercel.

A member of the Team first needs to authorize it.

[travi]

i like where this is headed. it looks like there are quite a few formatting changes included that are unrelated to the ESM transition, though. we dont have the project set up with a great lint or auto-formatting story at this point, but improving that situation seems separate enough from this effort that i think i would prefer those types of changes be in a separate PR. would backing those out of this change be a huge pain?

[mattwynne]

Gah, sorry about that. I thought I'd configured my editor to follow the standard formatter but perhaps some of the files weren't already formatted, or I failed in my configuration.

I agree that's going to be confusing to try and review. Let me take a look.

[mattwynne]

@travi dunno if you approve of the approach, but if you think #571 is a 👍 then I can just use the same rules to auto-format this branch and all the irellevant changes should disappear.

[travi]

perhaps some of the files weren't already formatted, or I failed in my configuration

since we dont have things configured to be very strict yet, i expect that there are likely things that are not currently formatted consistently. i agree with the approach of getting a better handle on that and using that improvement to simplify this PR

[travi]

a question about the settings export and a small cleanup item

[travi]

@mattwynne would you prefer i squash the history of this branch, or do you think it is worth keeping the story of the progression?

[mattwynne]

👍 for a squash

[mattwynne]

I can also rebase and tidy it up - all those dependabot commits are just from a sloppy merge.

[travi]

either way is ok with me. we can go with whichever you'd be most happy with

[mattwynne]

There you go, there's one nice clean commit for the main branch.

In general I'm totally happy to have my commits from a branch squashed when they go into main. I tend to make a lot of small commits when I'm working on something, but that kind of detail usually just creates clutter in the narrative of the main branch. In Cucumber we only ever merge with squash.

[jftanner]

This change broke my deployment and I'm not sure how to get it working again.

I was running a very simple Docker image, deployed to my own Kube cluster, that contains this simple package.json:

{
  "name": "@taas/github-settings",
  "version": "0.0.0-semantic-release-0",
  "private": true,
  "scripts": {
    "start": "probot run probot-settings"
  },
  "dependencies": {
    "probot": "12.2.2",
    "probot-settings": "probot/settings"
  }
}

After this PR, though, my deployment started to fail with the following error:

Error [ERR_REQUIRE_ESM]: Must use import to load ES Module: /home/jftanner/code/taas/github-settings/node_modules/probot-settings/index.js
require() of ES modules is not supported.

I think because Probot itself isn't an ESM module. So, the probot run PACKAGE doesn't work. Any suggestions on how else I should run it?

I tried creating a simple index.js file to handle the imports:

import Probot from 'probot'
import settingsBot from 'probot-settings'

await Probot.run(settingsBot)

But that failed with this error:

Error [ERR_MODULE_NOT_FOUND]: Cannot find module '/REDACTED/github-settings/node_modules/probot-settings/lib/mergeArrayByName' imported f
rom /REDACTED/github-settings/node_modules/probot-settings/index.js

The only solution I've found, so far, is to pin probot/settings at an older commit.

[mattwynne]

Oh dear!

I wonder if you need to change your package.json to include type: "module"?

I'm inclined to think Probot does work OK with import modules, as we can run npm start in this project and it boots up a Probot server OK. Maybe it's not working in your project because your package.json hasn't told Probot to use import?

[jftanner]

Ah, thank you!

I added "type": "module" when added the index.js, but didn't try it with the original start script. That appears to have worked. :)

[mattwynne]

Great! Thanks for checking back to let us know.

[jftanner]

Apparently I spoke too soon. I suspect I didn't properly re-install modules after I made that change, so I was still on the pre-ESM version.

In my deployed Kubernetes container, I get this error:

This is my package.json, which is the only thing in the container besides node_modules:

{
  "name": "@taas/github-settings",
  "version": "0.0.0-semantic-release-0",
  "type": "module",
  "private": true,
  "description": "Probot App that manages settings for TaaS Github repos",
  "scripts": {
    "start": "probot run probot-settings"
  },
  "dependencies": {
    "probot": "12.2.2",
    "probot-settings": "probot/settings"
  }
}

(Our release process updates the version field, which is why the number is different in the kubernetes cluster.)

Looking in node_modules, I think the issue is that Probot itself isn't an ESM module. So when it tries to load probot-settings (via lib/helpers/resolve-app-function.js) it fails. I'm not sure why though. Looking at that code, it looks like it should handle ESM just fine. Perhaps the typescript transpilation isn't configured correctly?

(I'm sorry to be a pain about this. I wouldn't be surprised if it's user-error on my part somewhere. I'm just a bit stumped.)

[mattwynne]

(I'm sorry to be a pain about this. I wouldn't be surprised if it's user-error on my part somewhere. I'm just a bit stumped.)

No worries, I can imagine this isn't the yak you intended to be shaving today 😁

I tried to make a minimal reproducible example, here, which seems to work OK.

Any thoughts on how that setup is different to yours?

[jftanner]

Definitely not the anticipated yak, no. 😂

I cloned your reproduction, added my .env and private key, and ran it. Same problem:

However, if I don't include an .env file, it starts up in setup mode:

The .env in question is pretty straightforward:

APP_ID=<redacted>
WEBHOOK_SECRET=<redacted>
LOG_LEVEL=debug
PRIVATE_KEY_PATH=private-key.pem
GHE_HOST=https://github.<redacted>/api/v3

Maybe try adding a dummy config to your reproduction and see what happens?

[mattwynne]

Maybe try adding a dummy config to your reproduction and see what happens?

Sounds like the next logical step. 👍

If you had any time to fork it and do that I'd appreciate it.

[jftanner]

Happily done!

Edit: From the check run, it looks like it's happy on Node 12 and 14, but fails on 16.

[jftanner]

Very strangely, though, I still end up with this error even if I use the node:14 docker image as my base. :/

[mattwynne]

I think those are false positives. If you look at the log output there's still an error but it's buried in an unhandled promise:

https://github.com/mattwynne/probot-settings-deployment-demo/runs/6313470886?check_suite_focus=true

[jftanner]

Oh, right, of course. Because unhandled rejections didn't become fatal until 16.

That explains why my kube deployment still errors out: the readiness check fails.

[mattwynne]

OK @jftanner I think I have a fix (#581), but we need to see what @travi and @gr2m have to say. Hopefully I've missed something and there's a way to get Probot to load ESM code directly.

It certainly seems to work with your tests on my machine.

[jftanner]

That ought to do it. It's really strange to me that this is a problem, though. Your PR was two weeks ago -- it seems bizarre that I'd be the only one complaining if this the breaking change that it looks like. But maybe people don't deploy their own settings bot very often?

I really like the changes in ESM, but the migration process has been an intermittent headache for months now. 😢