SpanScanner doesn't handle UTF-16 surrogate pairs

[leonsenft]

The following example demonstrates the issue:

import 'package:string_scanner/string_scanner.dart';

void main() {
  final text = '\u{12345}';
  final scanner = new SpanScanner(text);
  final start = scanner.state;

  while (!scanner.isDone) {
    scanner.readChar();
  }

  print(scanner.spanFrom(start));
}

This code throws:

RangeError: End 2 must not be greater than the number of characters in the file, 1.

I believe the issue is that package:string_scanner operates on code units, whereas package:source_span operates on code points (runes), resulting in this index mismatch for surrogate pairs.

I found a workaround by decoding the string first:

import 'package:source_span/source_span.dart';
import 'package:string_scanner/string_scanner.dart';

void main() {
  final text = '\u{12345}';
  final file = new SourceFile.decoded(text.codeUnits);
  final scanner = new SpanScanner.within(file.span(0));
  final start = scanner.state;

  while (!scanner.isDone) {
    scanner.readChar();
  }

  print(scanner.spanFrom(start));
}

Is the code unit and code point distinction here intentional? If so is there a better way to ensure proper UTF-16 support? The workaround seems reasonable, although having to provide a span rather than the decoded file itself is clunky. Would adding a constructor to SpanScanner which accepts a decoded file or list of code units be a suitable solution? Or could SpanScanner optionally operate on code points instead of code units?

[nex3]

I think the real issue here is that new SourceFile() should never have operated on code points in the first place. It's contrary to the rest of string handling throughout Dart, and I'd bet that all the code that actually handles spans assumes they refer to code units rather than code points.

[leonsenft]

I agree the issue stems from a choice made within SourceFile; however I still think operating at a code point level is an entirely valid use case. I'll post my reasoning on dart-lang/source_span#16.