Neither VM nor dart2js accepts null character in source code.

[lrhn]

The Dart specification allows all Unicode characters as string content (depending on the string type and delimiters). There is no exception made for the null character (ASCII NUL, U+0000).

However, neither implementation actually allow a program that contains a literal NUL inside a string literal. The Analyzer does allow it.

Example code (you have to replace the visual NUL with a real NUL):

main() {
 print("␀");
}

Posting as single bug for both implementations in case it is really a spec change that is needed.

[gbracha]

Is there a reason why NUL is special? Is this disallowed in the browser for example?

[iposva-google]

Please do not file multi-area bugs. Making this the dart2js bug.

---

Removed Area-VM label.

[iposva-google]

Filed issue #18100 for the VM version of this bug.

[peter-ahe-google]

We use the NUL character to detect end of file. It is a dummy value that acts as a sentinel, which is a standard optimization technique described by Knuth. I consider it a bug in dart2js that it has this limitation, but I never got around to fixing it.

Specifying this limitation should have little impact on Dart programmers: there are other, better ways to create a string with a NUL.

I'm not particularly attached to either option. The advantage of disallowing NUL would be that it is easier to manipulate Dart source code in a C program. I believe the so-called "modified UTF-8" of the JVM is exactly a codification of this limitation.

[gbracha]

I don't have a very strong position on the substance of the matter. That said we have a specification that is effectively frozen - TC52 voting is already in progress. The specification doubtless has bugs that will have to be corrected in future revisions, but I don't think this falls into that category.

[lrhn]

JavaScript does not have a problem with ASCII NULL in script code.
It seems browsers may have a problem with it in HTML code, but in a separate script file, it works fine.

The "modified UTF-8" has the disadvantage of not being valid UTF-8. It is not accepted by our own UTF8.decode:

    import "dart:convert";
    main() { print(UTF8.decode([0xc0, 0x80]).codeUnitAt(0)); }

gives: FormatException: Overlong encoding of 0x0
   

[rakudrama]

@johnniwinther does the CFE accept \u0000 in string literals and comments?

[lrhn]

Most likely it does not. If I create a file containing a zero byte inside a string literal,

void main() => print("�");

as tests/language/zero_byte_test.dart, and run it with the fasta target, it fails to see the bytes after the zero-byte.
The errors are all over the place:

tests/language/zero_byte_test.dart:1:22: Error: String starting with " must end with ".
void main() => print("");
                     ^
tests/language/zero_byte_test.dart:1:22: Error: The control character U+0000 can only be used in strings and comments.
void main() => print("");
                     ^
tests/language/zero_byte_test.dart:1:24: Error: String starting with " must end with ".
void main() => print("");
                       ^^^
tests/language/zero_byte_test.dart:1:21: Error: Can't find ')' to match '('.
void main() => print("");
                    ^
tests/language/zero_byte_test.dart:1:24: Error: Expected ';' after this.
void main() => print("");
                       ^^^
tests/language/zero_byte_test.dart:2:1: Error: Unexpected token ''.

So, the parser appears to not be able to handle zero-bytes in the source (even though its error message suggests that it should).

Also, from utf8_bytes_scanner.dart:

class Utf8BytesScanner extends AbstractScanner {
  /**
   * The file content.
   *
   * The content is zero-terminated.
   */
  final List<int> bytes;

which doesn't prove that it's actually used as terminator, but it is slightly worrying.

[johnniwinther]

I guess that if this is a parser issue, the analyzer should fail similarly on it.

[lrhn]

The analyzer does fail:

unexpected analysis errors in zero_byte_test.dart:
- Line 1, column 22: SYNTACTIC_ERROR.ILLEGAL_CHARACTER
  Illegal character '0'.

- Line 1, column 22: SYNTACTIC_ERROR.UNTERMINATED_STRING_LITERAL
  Unterminated string literal.

- Line 1, column 24: SYNTACTIC_ERROR.EXPECTED_TOKEN
  Expected to find ';'.

- Line 1, column 26: SYNTACTIC_ERROR.UNTERMINATED_STRING_LITERAL
  Unterminated string literal.

- Line 2, column 1: SYNTACTIC_ERROR.EXPECTED_TOKEN
  Expected to find ')'.