JS Source Maps support #506

DeMoorJasper · 2018-01-07T11:11:12Z

Adds source map support to Javascript, Typescript & Coffeescript.
Performance impact appears to be very limited to my surprise (as every transpiler was probably already creating a sourcemap in the background)

Closes #68

TODO:

Get source maps to work
~~Generate custom source maps instead of using babel (so we can continu using babylon)~~ EDIT: Used babel-generator as it's already being used
Find a better way to offset (Possibly offset on JSPackage, as this will be most reliable? For this i'd have to update the code to package main asset type before the sibling bundles)
Support source maps in typescript, coffeescript,...
Fix the tests (to expect source maps)
Write source maps tests
Make it blazing fast! 🚀
SourceMaps in Production mode? [RFC]

Add new improvements

Add all new commits

a

new features

Bugfixes

devongovett

Good start! Thanks for working on this. 🎉

devongovett · 2018-01-08T16:02:23Z

src/packagers/JSPackager.js

@@ -20,6 +21,7 @@ class JSPackager extends Packager {
  async start() {
    this.first = true;
    this.dedupe = new Map();
+    this.sourcemapPackager = new SourcemapPackager(this.bundle, this.bundler);


If you register the SourcemapPackager in here you won't need to construct it from here. addAsset will automatically get called as well, you don't need to call it from here. This way the source map packager will automatically work for non-JS langs as well if we generate maps.

How do i go about this without overwriting the jsPackager?

You'd register for type map not type js so it shouldn't overwrite.

this.add('map', SourceMapPackager);

Parcel should already be creating a bundle of type map since you return one from JSAsset#generate.

I've tried this out and it appears bundles are created based on asset.type not asset.generated.

oh right, I changed this recently. See https://github.com/parcel-bundler/parcel/blob/master/src/Bundler.js#L413-L420. Looks like your branch doesn't have that change.

devongovett · 2018-01-08T16:03:35Z

src/packagers/JSPackager.js

-    await this.dest.end('},{},' + JSON.stringify(entry) + ')');
+    await this.dest.write('},{},' + JSON.stringify(entry) + ')');
+    await this.dest.write(
+      `\n//# sourceMappingURL=${this.sourcemapPackager.url}`


Since you won't have access to the source map packager, the URL will be the name of the JS bundle with .js replaced with .map.

Alright, will fix this

devongovett · 2018-01-08T16:04:45Z

src/assets/JSAsset.js

@@ -116,7 +116,8 @@ class JSAsset extends Asset {
    }

    return {
-      js: code
+      js: code,
+      map: this.sourcemap


We'll also need to generate 1:1 source maps for files that we don't process with babel.

devongovett · 2018-01-08T16:05:55Z

src/packagers/SourcemapPackager.js

+          column: mapping.originalColumn
+        },
+        generated: {
+          line: mapping.generatedLine,


don't we need to offset this line by the current line number in the file since we're concatenating a bunch of things together? and what about the extra lines added by the JSPackager... hmm

We probably should, i'll dive deeper into this

Add new features from master branch

…into feature/source-maps

DeMoorJasper · 2018-01-09T10:35:23Z

@devongovett I got the big portions of source maps done, my main concern now is the offsets.
I was thinking of adding this to bundle.js code:

constructor() {
  ...
  this.offsets = new Map();
}

addOffset(assetname, line, column) {
  this.offsets.set(assetname, {line, column});
}

And this to jsPackager.js

addAsset(asset) {
  ...
  this.bundle.addOffset(asset.relativename, this.lineCount, 0);
  this.writeModule(...);
  ...
}

This way i could just extract the perfectly accurate offsets from the bundle, this would also support css in the future, if cssPackager just adds offsets as well.

The current issue with this is that the map packager is triggered before the js packager, which would require a few edits before this to work, so i'm wondering if this is the best approach before changing/adding all this code and if you have any tips on how to approach triggering the packaging of asset.type before triggering the packager of the other asset.generated values.

…into feature/source-maps

TimNZ · 2018-01-17T19:14:02Z

@devongovett With the env issue I commented above needing clearing of cache, something you should catch and show a message? Error handling in general ... :)

TimNZ · 2018-01-18T00:35:42Z

@DeMoorJasper Did a git pull on your fork and rm node_modules and reinstall and seems to have sorted my issues at quick glance. Thanks.

DeMoorJasper · 2018-01-19T16:38:20Z

After people have mentioned the new improved source-map and reading a blog post about the many performance improvements i decided to branch of this pr and make a source-map 0.7 compatible version for when it lands: https://github.com/DeMoorJasper/parcel/tree/source-map-0.7
The performance impact is pretty limited due to this PR already being maximised for performance and preventing mapping parsing whenever possible, this was the main performance issue with source-map 0.6.1, however 0.7 will have a huge benefit for coffeescript, typescript or any other library that doesn't expose rawMappings like babel

EDIT: source-map v0.7 has been officially released, now this PR depends on v0.7

source-maps 0.7 update

…into feature/source-maps

devongovett · 2018-01-21T20:11:49Z

One problem: source-map 0.7 requires Node 8. I know we're not there yet, but I think we are aiming to support Node 6 in parcel as well...

DeMoorJasper · 2018-01-21T20:56:08Z

@devongovett Should i downgrade back to source-map 0.6.1? I see no way around this as webassembly is not supported below node.js 8.0

devongovett · 2018-01-21T21:03:20Z

Yeah for now. Filed an issue to see if they can include the JS only version for Node 6 users: mozilla/source-map#313

devongovett · 2018-01-21T23:46:56Z

src/assets/JSAsset.js

    if (this.globals.size > 0) {
      code = Array.from(this.globals.values()).join('\n') + '\n' + code;
+      if (this.options.sourcemaps) {
+        if (!SourceMap.isSourceMapInstance(this.sourcemap)) {


Why not this.sourcemap instanceof SourceMap)? The current test for sources.length seems like it might be somewhat brittle.

devongovett · 2018-01-21T23:58:30Z

src/assets/JSAsset.js

+      if (this.options.sourcemaps) {
+        if (!SourceMap.isSourceMapInstance(this.sourcemap)) {
+          let sourcemap = new SourceMap();
+          if (!this.sourcemap.version) {


What is this check for? When will a sourcemap not have a version?

devongovett · 2018-01-22T00:16:58Z

src/SourceMap.js

+  }
+
+  isConsumer(map) {
+    return map && map.computeColumnSpans;


Can we just do map instanceof sourceMap.SourceMapConsumer? This check currently seems somewhat brittle since they could change the internal implementation.

Speeds things up slightly in JSPackager since we can use the line count calculated in JSAsset rather than re-computing.

devongovett

This is amazing work @DeMoorJasper! 🥇 Only refactored a few very minor things. The performance work you did in particular was very impressive. 🔥

Going to merge! Let's follow up with production support in a separate PR (will require uglify sourcemap support). 🚀

* run prettier * initial sourcemap * improve sourcemap constructor * also support SourceMapConsumer as addMap input * some cleanup and fixes * add lineOffset and 1:1 sourcemaps * add lineOffset and 1:1 sourcemaps * slight sourcemaputil improvement * use babel-generator instead of double generating * add 1:1 sourcemap for untranspiled assets * update offset * fix source paths * Better offset solution * fix sourceContent not being set by babel-generator * fix small offset bug * add sourcemap option and offset when globals are added * add cli option to disable sourcemaps * ts & coffeescript support * fix most tests * All tests (except hmr) fixed + bugfix for ts * fix hmr tests * comment out error throwing in tests * fix windows tests * add sourcemap tests * add comment to why throwing errors is commented out in tests * update with circular fix, now throws on test failures * fix generator for invalid maps * fix tests * fix tiny issues * small performance improvement * rewrite sourcemap handling, less generator constructing overhead and some more improvements * extendSourceMap bugfix * Use babel rawMappings to remove encoding step * small performance fixes * improve linecounter * small performance improvement and bugfix * small improvement * change for source-map 0.7 compatibility, improves performance a lil * switch to official npm release * tiny improvement * only install source-map 0.7 if possible * remove optional 0.7 dep * Minor refactorings * Store precomputed line count as part of source map * Clean up * Last cleanup

Jasper De Moor and others added 15 commits December 9, 2017 11:01

Merge pull request #1 from parcel-bundler/master

fedbd52

Add new improvements

Merge branch 'master' of github.com:DeMoorJasper/parcel

3f31fc5

Merge pull request #4 from parcel-bundler/master

b09d15f

Add all new commits

Merge branch 'master' of github.com:DeMoorJasper/parcel

80b3431

run prettier

392bbc9

Merge pull request #7 from parcel-bundler/master

e616cc2

a

stuff

1d90425

Merge pull request #9 from parcel-bundler/master

51bfe22

new features

Merge pull request #12 from parcel-bundler/master

2d393d2

new features

Merge branch 'master' of github.com:DeMoorJasper/parcel

85c7587

Merge pull request #14 from parcel-bundler/master

3c3cb77

Bugfixes

Merge branch 'master' of github.com:DeMoorJasper/parcel

9e7e46b

initial sourcemap

44f7a89

improve sourcemap constructor

f24bdd7

also support SourceMapConsumer as addMap input

a5b5ce2

devongovett reviewed Jan 8, 2018

View reviewed changes

DeMoorJasper changed the title ~~[WIP] Basic Source Maps support~~ [WIP] JS Source Maps support Jan 8, 2018

DeMoorJasper and others added 11 commits January 8, 2018 19:31

some cleanup and fixes

243d543

Merge pull request #16 from parcel-bundler/master

a2056df

Add new features from master branch

Merge branch 'feature/source-maps' of github.com:DeMoorJasper/parcel …

c3f40bd

…into feature/source-maps

add lineOffset and 1:1 sourcemaps

f6cadb5

add lineOffset and 1:1 sourcemaps

a6a0d9a

fix conflicts

cef8a8b

slight sourcemaputil improvement

5157b02

use babel-generator instead of double generating

ca829e5

add 1:1 sourcemap for untranspiled assets

6d8f4da

update offset

d101c48

fix source paths

7d9532b

fix conflicts

35a1b6f

Merge branch 'feature/source-maps' of github.com:DeMoorJasper/parcel …

56e6ea6

…into feature/source-maps

small performance improvement and bugfix

5567ee5

DeMoorJasper changed the title ~~[WIP] JS Source Maps support~~ JS Source Maps support Jan 17, 2018

DeMoorJasper added 2 commits January 18, 2018 10:04

small improvement

c4eb40f

change for source-map 0.7 compatibility, improves performance a lil

310d22f

DeMoorJasper and others added 4 commits January 21, 2018 11:01

switch to official npm release

be42812

Merge pull request #23 from DeMoorJasper/source-map-0.7

0e57597

source-maps 0.7 update

Merge branch 'feature/source-maps' of github.com:DeMoorJasper/parcel …

2abb918

…into feature/source-maps

tiny improvement

f39c21e

devongovett mentioned this pull request Jan 21, 2018

Document how to depend on source-map 0.6.X for old nodes and 0.7.0 for new nodes mozilla/source-map#313

Closed

DeMoorJasper added 2 commits January 21, 2018 22:41

only install source-map 0.7 if possible

90b85d1

remove optional 0.7 dep

8c81116

devongovett reviewed Jan 22, 2018

View reviewed changes

devongovett added 4 commits January 21, 2018 19:02

Minor refactorings

0b95aae

Store precomputed line count as part of source map

9908e9e

Speeds things up slightly in JSPackager since we can use the line count calculated in JSAsset rather than re-computing.

Clean up

01db40c

Last cleanup

11b89c6

devongovett approved these changes Jan 22, 2018

View reviewed changes

devongovett merged commit 5c5d5f8 into parcel-bundler:master Jan 22, 2018

DeMoorJasper deleted the feature/source-maps branch January 22, 2018 08:47

mindplay-dk mentioned this pull request Feb 7, 2018

Documentation for sourcemaps? parcel-bundler/website#110

Closed

Mickyfen17 mentioned this pull request Feb 25, 2018

🐛 bug report: parcel does not load existing sourcemaps of pre-transformed JS assets #741

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

JS Source Maps support #506

JS Source Maps support #506

DeMoorJasper commented Jan 7, 2018 •

edited

Loading

devongovett left a comment

devongovett Jan 8, 2018

DeMoorJasper Jan 8, 2018

devongovett Jan 8, 2018

DeMoorJasper Jan 8, 2018

devongovett Jan 8, 2018

devongovett Jan 8, 2018

DeMoorJasper Jan 8, 2018

devongovett Jan 8, 2018

DeMoorJasper Jan 11, 2018

devongovett Jan 8, 2018

DeMoorJasper Jan 8, 2018

DeMoorJasper commented Jan 9, 2018 •

edited

Loading

TimNZ commented Jan 17, 2018

TimNZ commented Jan 18, 2018

DeMoorJasper commented Jan 19, 2018 •

edited

Loading

devongovett commented Jan 21, 2018

DeMoorJasper commented Jan 21, 2018

devongovett commented Jan 21, 2018

devongovett Jan 21, 2018

devongovett Jan 21, 2018

devongovett Jan 22, 2018

devongovett left a comment

JS Source Maps support #506

JS Source Maps support #506

Conversation

DeMoorJasper commented Jan 7, 2018 • edited Loading

devongovett left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DeMoorJasper commented Jan 9, 2018 • edited Loading

TimNZ commented Jan 17, 2018

TimNZ commented Jan 18, 2018

DeMoorJasper commented Jan 19, 2018 • edited Loading

devongovett commented Jan 21, 2018

DeMoorJasper commented Jan 21, 2018

devongovett commented Jan 21, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

devongovett left a comment

Choose a reason for hiding this comment

DeMoorJasper commented Jan 7, 2018 •

edited

Loading

DeMoorJasper commented Jan 9, 2018 •

edited

Loading

DeMoorJasper commented Jan 19, 2018 •

edited

Loading