All ember-source modules missing from a build

[vlascik]

Seems that all modules from ember-source end up missing from /assets/my-app.js and the build fails with errors like these:

ERROR in ./components/-dynamic-element-alt.js 1:0-41
Module not found: Error: Can't resolve '@ember/component' in '$TMPDIR\embroider\889da1\components'
 @ ./assets/my-app.js 383:13-62
 
ERROR in ./components/badge.js 1:0-67
Module not found: Error: Can't resolve '@ember/component/template-only' in '$TMPDIR\embroider\889da1\components'
 @ ./assets/my-app.js 395:13-47

I understand that's a pretty vague error description, so maybe someone familiar with the source code could provide some pointers as to what to check, e.g. where this list of modules in my-app.js gets created, so that I can start debugging from there.

Thanks.

compat build, require('@embroider/compat').compatBuild(app, Webpack);
ember-source: 4.3.0-alpha.3
ember-cli: 4.2.0-beta.1
node: 14.16.1
os: win32 x64

[mehulkar]

May be related, I also see this:

Module not found: Error: Can't resolve '@ember/string' in '$TMPDIR/embroider/84850c/components/blahblah.js'

In my case since @ember/string is a real package published to npm, I was able to fix by installing it for real.

[ef4]

Neither of these should happen, all of these imports are supposed to get rewritten to point at things coming from ember-source before webpack sees them. Can somebody share a reproduction that does this?

[ef4]

(Maybe this is a canary thing, our CI is not currently running against canary so it could be broken and I wouldn't necessarily have noticed yet.)

[mehulkar]

Just tried in a brand new app (3.28) and was unable to repro (embroider v1.2.0). Must be something else in my main app. Sadly decided not to hold onto that play branch so I'll have to repro and look at debug logs again later!

[NullVoxPopuli]

I have a reproduction -- kinda happened by accident -- but ran in to this error:

Module not found: Error: Can't resolve '@ember/component' in '<repo>/ember-popperjs/dist/index.js'}

NullVoxPopuli/ember-popperjs#142

test-app has ember-source 4.1.0

[NullVoxPopuli]

Another repro: ember-polyfills/ember-functions-as-helper-polyfill#111

🤔 Surely there is something common between all these 🤔

[ef4]

There are probably at least two different bugs being reported in this thread now. It matters whether the place that is failing to import from ember is code in the app or code in an addon (and whether that addon is a v2 addon).

I looked at the ember-functions-as-helper-polyfill reproduction and that is a failure coming from inside ember-welcome-page (a v2 addon in the test-app).

It's probably a different bug from the original report.

[MichalBryxi]

I had success by removing ember-welcome-page when migrating to v2 addon format ember-intl-changeset-validations. Feel free to grab the branch & investigate.

[mehulkar]

I've gotten my app back into this state and have saved the branch this time (now with @embroider/*@1.4.0. It reproduces consistently, and I see errors on every reference of @ember/string.

Build Error (PackagerRunner) in components/we-modal.js
Module not found: Error: Can't resolve '@ember/string' in '$TMPDIR/embroider/84850c/components/we-modal.js'

It matters whether the place that is failing to import from ember is code in the app or code in an addon

In my case, the import is in a component in my app (not from an addon). I tried running with DEBUG=ember-cli:*,embroider:* , but seems the error is in webpack land.

[ef4]

Does it keep reproducing after you rm -rf $TMPDIR/embroider/webpack-babel-loader?

[ef4]

Also, do you have @ember/string and which packages are consuming it? yarn why @ember/string

[mehulkar]

Does it keep reproducing after you rm -rf $TMPDIR/embroider/webpack-babel-loader?

Yeah.

Side note: Does $TMPDIR remain the same between clean builds?

I used the one where the full stack trace from the build¹ (before rm -rf), and it turned out that it was the same again in the second build where I repro'd, but I'm not sure if that's a coincidence or not.)

Also, do you have @ember/string and which packages are consuming it?

Output of `npm ls @ember/string` shows ember-data as the only consumer

─┬ [email protected]
  ├─┬ @ember-data/[email protected]
  │ └── @ember/[email protected]  deduped
  ├─┬ @ember-data/[email protected]
  │ └── @ember/[email protected]  deduped
  ├─┬ @ember-data/[email protected]
  │ └── @ember/[email protected]  deduped
  ├─┬ @ember-data/[email protected]
  │ └── @ember/[email protected]  deduped
  └── @ember/[email protected] 

I tried npm uninstall --save ember-data (and confirmed that npm ls @ember/string is empty, and it still reproduces, so I think ember-data is a red herring here. (Another tip off to that is that I actually have addons.blacklist = ['ember-data'] configured for my build, but for Reasons™️, it's still it's in my dependencies.)

Footnotes

1. (Stack Trace and Error Report: /var/folders/dw/l307h_b16vzc0wys2f_nbmm00000gn/T/error.dump.2ac43f103a9bc69ff15ce572ccd476ed.log, ↩

[NullVoxPopuli]

I'm gonna log how I'm trying to debug this in this comment. (I'll be updating the comment instead of adding more posts as I figure things out).

---

The original error I got ended with

16 errors have detailed information that is not shown.
Use 'stats.errorDetails: true' resp. '--stats-error-details' to show it.

But adding that to the webpack config (in my autoImport config at the time of debugging) had no effect, so I assume the stats option isn't being merged.
Instead, I catd the full error log and placed a breakpoint in webpack here:

For me, this error appears when I'm running node_modules/.bin/ember test.
So I must use VSCode's JavaScript Debug Terminal, which attaches itself to all spawn sub-processes. Very nice. (Though, I have to close the default terminal and re-open JavaScript Debug Terminal from the command pallet when I start VS Code.

Success:

breakpoint hit, now I can walk the stack backwards and poke around for useful information.

The main thing I'm looking for is. Where did webpack even look for these packages?
On disk, they all exist at node_modules/ember-source/dist/packages/... so.. why didn't webpack look there?

---

Going up the stack, I only make it to here

before the stack devolves into only enhanced-resolve frames. So, I've set a new breakpoint so I don't have to crawl back up here, and I can poke around in this file for what resolver is and figure that out (I need to find where the callback is called without all the enhanced-resolve stuff).

I see that at the top of the function with this new breakpoint, resolver is passed in. So I'm placing a breakpoint at the top in case I can catch it before enhanced-reslove steps in:

At this point though, I need to kill my current ember test execution and start over, clicking "continue" in the debugger controls kept hitting my other breakpoints, so I think this _resolveResourceErrorHints is called earlier.
The name also implies the error has still already happened, and all this code is just for printing the error (and trying to help out the dev)

Hitting that breakpoint, and going up the stack a frame brings me here:

So now I'm going to repeat the process and see what's calling resolveResource's resolver.resolve callback. Variables that look potentially interesting right now are context, contextInfo, and resolveContext

I'm also gonna start needing conditional breakpoints based of the missing package name, because I'm getting in to area of webpack where the success path is the common path, and I don't want to debug what's working (yet?)

I've gotten to the top of where NormalModuleFafctory starts its thing, and I don't feel like I'm able to discern anything meaningful. So now I need to do this some process but for a project that is resolving @ember/template-factory (and most of the @glimmer/* and @ember/* packages) -- but my wrist hurts, so I gotta take a break.

Hope this helps someone!

[NullVoxPopuli]

Got another occurrence here: CrowdStrike/ember-aria-utilities#6

Though, not sure what the difference is between that an ember-resources.
Both are v2 addons, yet ember-resources' test app boots.
More diffing required ahead!

[NullVoxPopuli]

alright, so my issue was that the v2 addon didn't have "keywords": [ "ember-addon" ]

(at least most immediately, I have other things to figure out as well)

[NullVoxPopuli]

@vlascik do you still have this issue / did keywoards: ['ember-addon'] in the package.json resolve it?

[vlascik]

@NullVoxPopuli well, my original problem (all ember-source modules missing) seems to be resolved - the app builds and boots.

However, some modules are still missing - as far as I can tell the scenario is this:

1. app depends on an ember addon
2. addon depends on a generic JS library, not an addon

I was able to create a reproduction here https://github.com/vlascik/repro-embroider-missing-modules

Vanilla ember new app depends on ember-changeset, which in turn depends on validated-changeset. App builds, but fails to boot with:

Uncaught Error: Could not find module `validated-changeset` imported from `(require)`
    at missingModule (node_modules\@embroider\synthesized-vendor\vendor\loader\loader.js:247:1)
    at findModule (node_modules\@embroider\synthesized-vendor\vendor\loader\loader.js:258:1)
    at requireModule (node_modules\@embroider\synthesized-vendor\vendor\loader\loader.js:24:1)
    at eval (validated-changeset.js:1:18)
    at ../externals/validated-changeset.js (chunk.2c1f9ffb38eea73b66f3.js:721:1)
    at __webpack_require__ (chunk.6c61298b501950d139e4.js:31:41)
    at eval (application.js:5:88)
    at ./routes/application.js (chunk.2c1f9ffb38eea73b66f3.js:260:1)
    at __webpack_require__ (chunk.6c61298b501950d139e4.js:31:41)
    at Module.eval [as callback] (repro-embroider-missing-modules.js:103:147)

because of an import from validated-changeset at routes/application.js.

Adding validated-changeset to app's devDependencies "fixes" the problem.

[NullVoxPopuli]

Adding validated-changeset to app's devDependencies "fixes" the problem.

This is the correct behavior, in modern packagers, you cannot import from things you do not, yourself, declare

[NullVoxPopuli]

Gonna close this for now -- if anyone is having a similar error, let's open a new issue so we don't get distracted / confused (I would!) with all the things in this thread.

[vlascik]

@NullVoxPopuli Got any link for that? Because as far as I was able to find, webpack shouldn't care where was dependency declared, as long as it is in node_modules, and builds its own dependency graph based on import statements https://webcache.googleusercontent.com/search?q=cache:xN1si4ZIZLwJ:https://blog.jakoblind.no/how-webpack-decides-what-to-bundle/ and webpack/webpack#6547 (comment)

Also, the behavior of classic ember build system, where addon's dependencies went into the host app, is well known and relied upon, so if embroider changes that, users should be either be notified in build/boot output, or the change needs to be well documented, otherwise you'll create a lot of confusion.

[NullVoxPopuli]

as it is in node_modules

this is the kicker, you can't guarantee it is, because how node_modules is organized is an implementation detail of the package manager. in both yarn@v1 and npm (which have broken dependency management techniques, though npm@8 is quite a bit better), the will shove everything flatly into node_modules -- this gives you the illusion that you can import your transient dependencies. What yarn3, pnpm, and npm@8 (with strict settings still enabled) do is manage deps correctly, in that you can only import what is declared in your package.json (be that via nested node_modules, or whatever other technique they're implementing).

This isn't an emborider issue -- embroider is following the correct resolution algorithm, in that it doesn't re-implement bugs that yarn@v1 and older npm contain