clarify newline behavior of regex

[danmoseley]

Fixes #35498

Attempt to clarify behavior of $ in regex, to address confusion in dotnet/runtime#87368.

---

Internal previews

📄 File	🔗 Preview link
docs/standard/base-types/anchors-in-regular-expressions.md	Anchors in Regular Expressions
docs/standard/base-types/backtracking-in-regular-expressions.md	Backtracking in Regular Expressions
docs/standard/base-types/character-classes-in-regular-expressions.md	Character Classes in .NET Regular Expressions
docs/standard/base-types/regular-expression-options.md	Regular expression options
docs/standard/base-types/regular-expression-source-generators.md	docs/standard/base-types/regular-expression-source-generators

[danmoseley]

cc @Tisit

[danmoseley]

Is anyone available to review..

[stephentoub]

Negative lookbehind inside of a positive lookahead is a really complicated construct. Is it necessary? If the first branch of the alternation is \r\n\z, why can't the later branch just be \n\z, or why not just have the \r be optional inside of the lookahead, e.g. (?=\r\n\z|\n\z|\z) or (?=(?:\r?\n)?\z) or some such thing?

[danmoseley]

The assumption is that we want to recognize \r\n but not treat it like two lines (no editor would treat that as two line breaks)

For example given "abc\n" in SingleLine mode the pattern ".\Z" will match at position 2 (before the newline) and position 3 (before the end). Now take "abc\r\n". If we want \r\n to be treated like a newline, then we want to match at 2 (before newline) and 4 (before end)

(?=\r\n\z|\z|(?<!\r)(?=\n\z)) does this but (?=\r\n\z|\n\z|\z) and (?=(?:\r?\n)?\z) will add a third match at position 3.

[stephentoub]

Recommending lookarounds is problematic because lookarounds don't work with non-backtracking, nevermind their being complicated. It'd be better, for example, to recommend other patterns that would match the newlines and then have a capture group for just the piece you want.

[danmoseley]

That would be a more extended discussion. I think there is value in including this subexpression for reference and for those that need it. Mind if I leave as is for now? I feel like what we have here is a net improvement and I can do more to the docs another day.

Adding support for AnyNewLine is the most useful thing we can do here.

[stephentoub]

If you recommend such patterns then you need to call out that they will prevent NonBacktracking from working / will fail with NonBacktracking.

[stephentoub]

Amongst engines that do support backtracking it seems to be fairly common to support \r\n as an atomic unit

Can you elaborate? What is the syntax someone uses with PCRE for example to achieve the "treat \n or \r\n as a newline" semantic? Is that the \R?

Does PCRE support that with its non-backtracking functions?

We could support it as part of matching, but without changing the matching logic I don't think we can do it as written just during parsing.

[danmoseley]

\R consumes a "generic line break". In PCRE what \R considers that to be can be adjusted. Similarly, what $ and \Z consider a line break can be separately adjusted, at compile time or within the pattern.

https://www.pcre.org/current/doc/html/pcre2pattern.html#newlines https://www.pcre.org/current/doc/html/pcre2pattern.html#newlineseq
https://www.pcre.org/current/doc/html/pcre2api.html#newlines

eg \R might be equivalent to (?>\r\n|\n|\x0b|\f|\r|\x85). And you might have (*ANYCRLF) in your pattern to mean that you want to recognize any of CR, LF, or CRLF.

It makes clear that if CRLF is recognized as a newline, then it will be treated atomically:

When the newline convention (see "Newline conventions" below) recognizes the two-character sequence CRLF as a newline, this is preferred, even if the single characters CR and LF are also recognized as newlines. For example, if the newline convention is "any", a multiline mode circumflex matches before "xyz" in the string "abc\r\nxyz" rather than after CR, even though CR on its own is a valid newline. (It also matches at the very start of the string, of course.)

[danmoseley]

In Perl, \R matches generic newline: it's not clear whether you can change what $ recognizes.

In Java, $ and \Z and . as appropriate recognize a variety of line terminators including the atomic \r\n. But there's an option UNIX_LINES to make them only recognize \n. In other words, they have AnyNewLine as proposed, but on by default.
https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html (see Line terminators)

\R in Java seems to always be matching a variety, like Perl.

\R Any Unicode linebreak sequence, is equivalent to \u000D\u000A|[\u000A\u000B\u000C\u000D\u0085\u2028\u2029]

[stephentoub]

@veanes / @olsaarik, I assume for the lookarounds support we discussed, we could by default only permit lookarounds with a max length limited by the pattern rather than by the input, eg no unbounded loops. That would enable the patterns discussed here.

[veanes]

Right. So a straightforward fragment is essentially the one where unbounded loops do not occur, perhaps even stricter -- no loops (either bounded or nonbounded). One other issue is that in general NFA mode might still be needed during lookaraounds, but that should be orthogonal to the rest and happen automatically. My understanding is also that all captures are disabled during lookarounds and not stored.

[danmoseley]

@stephentoub I removed the lookahead here so we can merge the rest - does it look OK?