Triple backticks to allow creation of JavaScript blocks #4357

GeoffreyBooth · 2016-11-14T07:28:34Z

Closes #1504 and #3760. Per Jeremy’s comment from 2011, this adds support for “heredoc” JavaScript blocks, e.g.:

```
var a = 1;
var b = 2;
```
c = 3
```var d = 4;```
eq a + b + c + d, 10

Escaped backticks are output as just backticks:

```var e = \`hello, world${'!'}\````
eq e, 'hello, world!'

… ```)

greghuc · 2016-11-14T08:01:20Z

test/javascript_literals.coffee

+  ```
+  // This is a comment with `backticks`
+  var a = 42;
+  var b = `foo ${'bar'}`;


There's no 'eq' for variable b in the ""block inline JavaScript containing backticks" test. Is this correct?

Yes, as I didn't want to rely on ES2015 in the 1.x branch. Though I guess I am already for it to parse the b line. Hmm...

greghuc · 2016-11-14T10:07:50Z

src/lexer.coffee

@@ -900,7 +907,7 @@ CODE       = /^[-=]>/

 MULTI_DENT = /^(?:\n[^\n\S]*)+/

-JSTOKEN    = /^`[^\\`]*(?:\\.[^\\`]*)*`/
+JSTOKEN    = /^```([\s\S]*?)```|^`[^\\`]*(?:\\.[^\\`]*)*`/


I read the added regex part as "capture everything between starting and end triple backquotes, stopping at first-seen ending triple quotes", which seems sane.

The original regex is rather obtuse. I read it as "greedily read everything between starting and ending single backquotes, and make sure every backslash has a character following it". I'm thinking the backslash+character part exists to stop an escaped backslash preemptively closing the Javascript block. E.g:

OK: `a\`` Not OK: `a\`

The new triple backticks regex will behave differently:

OK: ```a\```` Also OK: ```a````

I found this website useful: https://regex101.com/

#3760 is actually complaining about just what you describe: a backslash not followed by a character (in that case, it's followed by a newline). Perhaps the original regex is incorrect here?

What would you use as the regex, if you don't mind taking the time to play with it?

Parsing this way is inconsistent with heredocs and heregexen, where backslash can be used to escape characters adjacent to the delimeters:

"""\"hello\"""" # '"hello"' ///\/hello\//// # /\/hello\// ```\`hello\```` # ?

Per my last comment, I’m suggesting we convert \`` to ``` wherever `` appears inside a single- or triple-backtick block. That should cover this case, yes?

jashkenas · 2016-11-14T16:10:40Z

This is great! I'd only suggest adding some complicated tests involving nested levels of backticks / interpolation. That's usually where things get tricky to lex.

GeoffreyBooth · 2016-11-14T16:59:31Z

@jashkenas more complicated than this?

lydell · 2016-11-14T17:40:33Z

Could somebody explain with words:

When a ``` JS block ends.
When a ````` JS block ends.

Do backslashes inside JS block affect where the block ends?

jashkenas · 2016-11-14T17:49:59Z

I'm probably not the expert in the details of it — but I'd imagine that the idea is to be sensible about escaping the backticks in the same way that we try to be sensible about escaping quotes in a heredoc.

So: A JS block ends at the first unescaped and uninterpolated that follows.

GeoffreyBooth · 2016-11-14T17:55:33Z

@jashkenas What do you mean by uninterpolated? I would’ve answered this way:

A JS block ends at the first unescaped that follows.
A JS block ends at the first that follows.

lydell · 2016-11-14T18:04:24Z

`\`a\n\``

↓

`a\n`;

Is the above correct? Just checking if I understand the rules or not.

jashkenas · 2016-11-14T18:08:37Z

@lydell — The above looks correct to me.

@GeoffreyBooth: By uninterpolated, I mean this:

before #{"str`ing"} after

The backtick inside the interpolation doesn't kill the expression halfway. Ideally the same would be true for triple backticks.

Edit: Hah, nevermind. I just tested it and I guess we don't allow interpolations in JS. Forget it.

GeoffreyBooth · 2016-11-14T18:29:21Z

@jashkenas I wasn’t aware that we supported interpolation between backticks, or wanted to.

@lydell Your example, if saved into a file that is compiled via coffee -bp test.coffee, outputs:

\`a\n\`;

So clearly somehow my “escaping works!” test is flawed. I guess some escaping is happening in the test runner itself? Anyway it appears we’re not escaping backticks (that is, outputting just ``` for ```), we’re just skipping over escaped backticks until we find a non-escaped one to end the token on. I can’t see any benefit to our current behavior.

jashkenas · 2016-11-14T18:41:58Z

@jashkenas I wasn’t aware that we supported interpolation between backticks, or wanted to.

Yes. We don't. My mistake.

connec · 2016-11-14T18:43:09Z

Might be worth noting that the main problem in the issues you linked was for tagged template literals, which will hopefully soon be merged.

GeoffreyBooth · 2016-11-14T18:55:04Z

@connec Yes, but this should get fixed anyway. This future-proofs us against any other weird things ECMAScript wants to use backticks for.

I figured out why my test passed when you wouldn’t expect it to: JavaScript converts ``` into ``` wherever it sees it:

So the test should compare a string against the expected JavaScript output, not the executed JavaScript.

I’ll update this PR soon, but just so we’re clear on the intended behavior:

A JS block ends at the first unescaped that follows. Escaped backticks are converted to simple backticks (i.e. ``` becomes ```).
A JS block ends at the first that follows. Escaped single backticks are converted to simple backticks (i.e. \`` becomes ```) so if you really want a triple-backtick in the generated JavaScript, you would write ````.

?

connec · 2016-11-14T19:42:52Z

That sounds fine to me. I wonder if we can leverage the code paths used to handle heredocs and heregexen? This is an area of the parser I've not looked at.

GeoffreyBooth · 2016-11-15T01:41:10Z

Okay, it took quite some doing to get `````hello````` to match out to the last backtick, and not one before it (i.e. treating the ``` as not part of the closing triple backticks). @connec or anyone else, can you please check the new regex?

Escaped backticks are now replaced with just backticks in the JavaScript code, so you can write code like this:

`let greeting = \`hello, ${name}\``

I looked into somehow getting dangling backslashes at the end of a line to not stop the capture (the specific bug in #3760) but I ultimately don’t think it’s worth the effort: a backslash before a newline is invalid JavaScript, so it will never compile correctly.

I think this is everything?

lydell · 2016-11-15T07:10:28Z

I think I'd approach this problem something like this:

JSTOKEN      = ///^ `(?!``) ((?: [^`\\] | \\[\s\S]           )*) `   ///
HERE_JSTOKEN = ///^ ```     ((?: [^`\\] | \\[\s\S] | `(?!``) )*) ``` ///

test = (chunk) ->
  match = HERE_JSTOKEN.exec(chunk) or JSTOKEN.exec(chunk)
  console.log chunk, match
  if match
    script = match[1].replace(/\\+(`|$)/g, (string) ->
      string[-Math.ceil(string.length / 2)..]
    )
    return script
  return null

console.log([
  test('``````') # == ''
  test('`\\\\`') # == '\\'
  test('`\\\\\\`a`') # == '\\`a'
  test('```\\```') # == null
])

GeoffreyBooth · 2016-11-15T07:41:27Z

@lydell I'm not sure I follow; aside from more readable regexes, what problems with the PR does your version solve?

lydell · 2016-11-15T07:57:18Z

You should be able to output a backslash before a backtick, or at the end. For example \\\``` → `\`;` and \`` → \;. (We should add tests for even crazier numbers of backslashes as well, and tests for invalid stuff like \.)

GeoffreyBooth · 2016-11-15T18:05:00Z

@lydell I would think that \\\``` is supposed to become `\\`;`, and it does in this PR. Likewise, I think \`` should become \\;, as it does. It’s not our job to escape all JavaScript escaped characters; if the user types `\` between backticks, it should just be output as `\`. That’s the current behavior before this PR, so changing it could break backward compatibility. A backslash is normally escaped in JavaScript, so it’s probably pretty common to find `\` in backticked code.

Technically speaking, converting ``` to ``` is also breaking backward compatibility, but I think this is okay since the failure to do this conversion is more of a bug than a feature. The only time a backtick is ever escaped in JavaScript is within a template literal, but people basically haven’t been able to write those yet in CoffeeScript because of the lack of any way to escape backticks.

lydell · 2016-11-15T18:28:13Z

I only meant that backslashes should be handled specially before backticks and at the end of the block (which my code does). In other words, \\ and \n, for example, are untouched. Otherwise you wouldn't be able to express anything inside a JS block (which is the point of this PR, isn't it?). For example, a template literal ending in a backslash: \`a\\\``` → a```

GeoffreyBooth · 2016-11-15T22:40:21Z

@lydell can you please explain

script = match[1].replace /\\+(`|$)/g, (string) ->
      string[-Math.ceil(string.length / 2)..]

? I replaced my regex with yours and adjusted the code as appropriate, but when I add this new replace function the first JavaScript literal test fails.

lydell · 2016-11-16T06:45:21Z

src/lexer.coffee

+    return 0 unless @chunk.charAt(0) is '`' and
+      (match = HERE_JSTOKEN.exec(@chunk) or JSTOKEN.exec(@chunk))
+    [js] = match
+    script = if js[0..2] is '```' then js[3...-3] else js[1...-1]


Why not use the capturing group instead of this trickery? match[1]

The old regex didn’t have a capturing group (see the original code that this PR revises). But yes, match[1] is better.

Can you please explain your replace function? Or rewrite it to itself use a lot less sorcery? What exactly are we trying to replace, and with what, when it comes to escaped backticks?

Considering the raw text inside a JSTOKEN: \` → ` \\` → can't happen \\\` → \` \\\\` → can't happen \\\\\` → \\` ... Go from left to right. Take a backslash and the character after it. Replace with the second of those two. That's the rule. Or, observe the pattern: We simply end up with the second half of a \\\\\\\` sequence! And, for completeness, at the end of the JSTOKEN: \ → can't happen \\ → \ \\\ → can't happen \\\\ → \\ \\\\\ → can't happen ...

So double backslashes are left alone except when followed by an escaped backtick or the end of the JS block. Seems a bit convoluted to try to explain in documentation. Can we come up with a simpler rule?

Currently the PR always converts ``` to a backtick, and that's it. So there's no way to render an escaped backtick inside a template literal. One way to solve that problem is to convert the escaped backticks only in the single-backtick tokens, and leave the triple-backtick ones alone. If some needs an escaped backtick in the JS, they use triple backticks. If someone needs three consecutive backticks in their JS, they use single backticks and escape the triplet.

https://xkcd.com/1638/ I guess

This behaviour makes sense to me, I'd describe it as:

A backslash (\) can be used to escape a backtick (```) or another backslash in a JS literal, and will otherwise be passed through.

`hello` # hello; `\`hello\`` # `hello`; `\`Escaping backticks in JS: \\\`hello\\\`\`` # `Escaping backticks in JS: \`hello\``; `Single backslash: \ ` # Single backslash: \ ; `Single backslash at EOS: \\` # Single backslash at EOS: \; `Double backslash at EOS: \\\\` # Double backslash at EOS: \\; `Double backslash: \\\ ` # Double backslash: \\ ;

If we start requiring backslashes to be escaped, that could break existing code. I think it's better to only convert ```, only within single-backtick blocks. That preserves backward compatibility, and gives people a way to output an escaped backtick (via a triple-backtick block).

They're only required to be escaped if they appear before a ```, so that shouldn't impact backward compatibility too much.

I think there's hesitation about releasing a syntax that would prevent users from rendering something (in this case, ``` in single-backticks, and ````` in triple-backticks) when there's a familiar, established solution.

Edit: to be clear, I'm not proposing to require that backslashes are escaped, just that you can escape them if you want to render a literal backslash before a backtick.

lydell · 2016-11-16T06:51:29Z

test/javascript_literals.coffee

+  eq '\\`', `
+    // Inline JS
+    "\\\`"
+  `


With my proposed escaping rules, this test needs to be written like this:

eq '\\`', ` // Inline JS "\\\\\`" `

or like this:

eq '\\a`', ` // Inline JS "\\a\`" `

…e backticks; additional tests

GeoffreyBooth · 2016-11-17T06:52:26Z

Okay, I’ll concede to the preferences of the crowd. I’ve updated the replace function per @lydell.

@connec, I added your examples as a new test, though I had to escape all the backslashes for them to be parsed correctly as strings. Since that made them hard to understand, I included all the before-and-afters as a big comment just above the test. To double-check it was really working, I created a test.coffee file and pasted your code in there:

`hello`                                       # hello;
`\`hello\``                                   # `hello`;
`\`Escaping backticks in JS: \\\`hello\\\`\`` # `Escaping backticks in JS: \`hello\``;
`Single backslash: \ `                        # Single backslash: \ ;
`Single backslash at EOS: \\`                 # Single backslash at EOS: \;
`Double backslash at EOS: \\\\`               # Double backslash at EOS: \\;
`Double backslash: \\\ `                      # Double backslash: \\ ;

It becomes:

hello;
`hello`;
`Escaping backticks in JS: \`hello\``;
Single backslash: \ ;
Single backslash at EOS: \;
Double backslash at EOS: \\;
Double backslash: \\\ ;

The only difference is the last one: you had it outputting only two backslashes, instead of the three I get. But there’s no backtick following these backslashes, so I don’t think the lexer should be affecting them.

Is everyone happy with this?

lydell · 2016-11-17T07:58:33Z

`Double backslash: \\\ `                      # Double backslash: \\ ;

The above is nothing but wrong and misleading.

Do we have any tests for that '\\\n' becomes '\\\n' (in other words, making sure that we don't mess up other backslashes)?

GeoffreyBooth · 2016-11-17T08:16:08Z

@lydell Other backslashes are unaffected. I added a test.

connec · 2016-11-17T11:17:04Z

OK, cool.

~~I would definitely remove the Double backslash: \\\ ; test, or rename it to Triple backslash: \\\ ;, because as @lydell says it's currently misleading~~ - Whoops, looks like that's already changed.

This seems good to me. I'm still slightly dubious about only escaping ```s, but it makes sense that the literal JS should be as close as possible to what's written.

jashkenas · 2016-11-18T15:49:21Z

I don't think that I can mentally expand these regexes well enough to comment on them materially — but if @lydell and @connec sign off on this, then I'm cool with it too.

GeoffreyBooth · 2016-11-18T18:16:01Z

@lydell and @connec, do you have any other notes or should we merge this in?

vendethiel · 2016-11-18T18:22:22Z

LGTM

connec

LGTM

lydell · 2016-11-19T08:41:48Z

LGTM

GeoffreyBooth added 2 commits November 13, 2016 23:01

Support JavaScript code blocks set apart by triple backticks (``` ...…

27e8a62

… ```)

Add test for escaped backticks

107d8af

GeoffreyBooth added the enhancement label Nov 14, 2016

Remove TODOs for things we’re never going to support

edd36b5

greghuc reviewed Nov 14, 2016

View reviewed changes

GeoffreyBooth added 2 commits November 14, 2016 16:41

Convert escaped backticks to backticks; update tests

b9dd310

Block inline JavaScript can end with an escaped backtick character

3200259

GeoffreyBooth force-pushed the triple-backticks branch from 482bc46 to 45b7193 Compare November 15, 2016 22:41

Updated JavaScript token regexes per @lydell

0d5449f

GeoffreyBooth force-pushed the triple-backticks branch from 45b7193 to 0d5449f Compare November 15, 2016 22:42

GeoffreyBooth mentioned this pull request Nov 16, 2016

CS2 Discussion: Project: Progress as of November 2016 coffeescript6/discuss#52

Closed

lydell reviewed Nov 16, 2016

View reviewed changes

In JavaScript blocks, escape backslashes when they immediately preced…

7689620

…e backticks; additional tests

Test that we don’t break backslash escaping in JavaScript literals

9945e68

Merge branch 'master' into triple-backticks

621d9d5

lydell approved these changes Nov 18, 2016

View reviewed changes

connec approved these changes Nov 18, 2016

View reviewed changes

GeoffreyBooth merged commit 073e147 into jashkenas:master Nov 19, 2016

GeoffreyBooth deleted the triple-backticks branch November 19, 2016 19:13

This was referenced Nov 20, 2016

Docs for triple-backticks and escaping backticks #4369

Merged

Add support for CoffeeScript triple backticks highlightjs/highlight.js#1358

Merged

GeoffreyBooth mentioned this pull request Dec 2, 2016

CS2 Discussion: Project: Progress as of December 2016 coffeescript6/discuss#56

Closed

GeoffreyBooth mentioned this pull request Feb 22, 2017

CS2 Discussion: Project: Progress as of February 2017: 2.0.0-alpha1! coffeescript6/discuss#71

Closed

GeoffreyBooth mentioned this pull request Apr 14, 2017

CS2 Discussion: Project: Progress as of April 2017: 2.0.0-beta1! coffeescript6/discuss#80

Closed

GeoffreyBooth mentioned this pull request Sep 10, 2017

Lexer documentation typo #4694

Closed

Triple backticks to allow creation of JavaScript blocks #4357

Triple backticks to allow creation of JavaScript blocks #4357

Conversation

GeoffreyBooth commented Nov 14, 2016 • edited Loading

greghuc Nov 14, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

greghuc Nov 14, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jashkenas commented Nov 14, 2016

GeoffreyBooth commented Nov 14, 2016

lydell commented Nov 14, 2016

jashkenas commented Nov 14, 2016 • edited Loading

GeoffreyBooth commented Nov 14, 2016

lydell commented Nov 14, 2016

jashkenas commented Nov 14, 2016 • edited Loading

GeoffreyBooth commented Nov 14, 2016 • edited Loading

jashkenas commented Nov 14, 2016

connec commented Nov 14, 2016

GeoffreyBooth commented Nov 14, 2016

connec commented Nov 14, 2016

GeoffreyBooth commented Nov 15, 2016 • edited Loading

lydell commented Nov 15, 2016

GeoffreyBooth commented Nov 15, 2016

lydell commented Nov 15, 2016

GeoffreyBooth commented Nov 15, 2016

lydell commented Nov 15, 2016

GeoffreyBooth commented Nov 15, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lydell Nov 16, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

connec Nov 16, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

connec Nov 16, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GeoffreyBooth commented Nov 17, 2016

lydell commented Nov 17, 2016

GeoffreyBooth commented Nov 17, 2016

connec commented Nov 17, 2016 • edited Loading

jashkenas commented Nov 18, 2016

GeoffreyBooth commented Nov 18, 2016

vendethiel commented Nov 18, 2016

connec left a comment

Choose a reason for hiding this comment

lydell commented Nov 19, 2016

GeoffreyBooth commented Nov 14, 2016 •

edited

Loading

greghuc Nov 14, 2016 •

edited

Loading

greghuc Nov 14, 2016 •

edited

Loading

jashkenas commented Nov 14, 2016 •

edited

Loading

jashkenas commented Nov 14, 2016 •

edited

Loading

GeoffreyBooth commented Nov 14, 2016 •

edited

Loading

GeoffreyBooth commented Nov 15, 2016 •

edited

Loading

lydell Nov 16, 2016 •

edited

Loading

connec Nov 16, 2016 •

edited

Loading

connec Nov 16, 2016 •

edited

Loading

connec commented Nov 17, 2016 •

edited

Loading