panicked at 'not a char boundary' when using Regex::replace

[WIZeaz]

What version of regex are you using?

v1.8.3

Describe the bug at a high level.

I have fuzzed the regex package with afl.rs, and afl reports some cases panicked at "char boundary":

thread 'main' panicked at 'byte index 1 is not a char boundary; it is inside 'ο' (bytes 0..2) of `ο00000000000`', <my_project_path>/regex/src/re_unicode.rs:574:31

It seems replace function have some problems with utf-8 character. I have found such panic in several fuzz drivers.

What are the steps to reproduce the behavior?

fn main() {
    let re = regex::Regex::new(r"\B|00(?-u)\B").unwrap();
    let text = r"𐾁00000000";
    let rep = r"0𐾁Ű000ο";
    let _ = re.replace(text, rep);
}

fn main(){
    let re = regex::Regex::new(r"\B|00(?-u)0\B").unwrap();
    let text = "ο";
    let rep = "000";
    let _ = re.replace(text, rep);
}

fn main(){
    let re = regex::Regex::new(r"(()(?-u)\B)0|\B").unwrap();
    let _ = re.replace("Ԑ0000000000000" ,"000000000000000");
}

fn main(){
    let re = regex::Regex::new(r"0()|\B|(?-u)\B0").unwrap();
    let _ = re.replace("ⳅ000000000000" ,"000000000000000");
}

fn main(){
    let re = regex::Regex::new(r"\B|(?-u)\B0").unwrap();
    let _ = re.replace_all("00000^睓00" ,"00000000000");
}

I also found that if I replace "replace_all" with "replace", the last case will not panic. The rest of these will panic after being replaced. It seems weird.

What is the actual behavior?

Here is an error message sample; the error messages in these cases are similar.

thread 'main' panicked at 'byte index 1 is not a char boundary; it is inside 'ο' (bytes 0..2) of `ο00000000000`', <my_project_path>/regex/src/re_unicode.rs:574:31

What is the expected behavior?

Regex::replace should deal with utf-8 character correctly.

[BurntSushi]

It looks like this is a new bug introduced by regex 1.8. Prior versions return an error for such regexes:

$ cargo run
   Compiling aho-corasick v0.7.20
   Compiling regex-syntax v0.6.29
   Compiling regex v1.7.3
   Compiling i1006 v0.1.0 (/home/andrew/tmp/issues/regex/i1006)
    Finished dev [unoptimized + debuginfo] target(s) in 1.50s
     Running `target/debug/i1006`
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Syntax(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
regex parse error:
    \B|00(?-u)0\B
               ^^
error: pattern can match invalid UTF-8
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
)', main.rs:2:50
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

This is fixed by #978 more generally by allowing the regex to compile but not reporting a match anywhere. In older versions of this crate, things like (?-u:\B) were forbidden in Unicode regexes because it can match between code units in UTF-8 encoded text. But that's technically also true of .* as well (since it can match any empty string). So #978 takes a more robust perspective on this issue and generally permits regexes that could match between code units, but adjusts the matching engines to not report such matches.

[BurntSushi]

I put up #1007 to make regex 1.8 also return a regex compilation error, like in previous versions.

[addisoncrump]

Can we add replace to the general fuzz harness for this case? Seems to only happen in replace, which I didn't write a fuzzer for.

[BurntSushi]

Probably just making sure that Match::as_str works is enough. But adding replace seems fine.

I'd prefer is we waited until #978 was merged to do more work on the fuzzers.

[BurntSushi]

This is fixed in regex 1.8.4 on crates.io.