Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement the complete reverse case for the two way algorithm #27474

Merged
merged 4 commits into from
Aug 18, 2015

Commits on Aug 2, 2015

  1. StrSearcher: Add tests for rfind(&str)

    Add tests for .rfind(&str), using the reverse searcher case for
    substring search.
    bluss committed Aug 2, 2015
    Configuration menu
    Copy the full SHA
    c5a1d8c View commit details
    Browse the repository at this point in the history
  2. StrSearcher: Implement the full two way algorithm in reverse for rfind

    Fix quadratic behavior in StrSearcher in reverse search with periodic
    needles.
    
    This commit adds the missing pieces for the "short period" case in
    reverse search. The short case will show up when the needle is literally
    periodic, for example "abababab".
    
    Two way uses a "critical factorization" of the needle: x = u v.
    
    Searching matches v first, if mismatch at character k, skip k forward.
    Matching u, if mismatch, skip period(x) forward.
    
    To avoid O(mn) behavior after mismatch in u, memorize the already
    matched prefix.
    
    The short period case requires that |u| < period(x).
    
    For the reverse search we need to compute a different critical
    factorization x = u' v' where |v'| < period(x), because we are searching
    for the reversed needle. A short v' also benefits the algorithm in
    general.
    
    The reverse critical factorization is computed quickly by using the same
    maximal suffix algorithm, but terminating as soon as we have a location
    with local period equal to period(x).
    
    This adds extra fields crit_pos_back and memory_back for the reverse
    case. The new overhead for TwoWaySearcher::new is low, and additionally
    I think the "short period" case is uncommon in many applications of
    string search.
    
    The maximal_suffix methods were updated in documentation and the
    algorithms updated to not use !0 and wrapping add, variable left is now
    1 larger, offset 1 smaller.
    
    Use periodicity when computing byteset: in the periodic case, just
    iterate over one period instead of the whole needle.
    
    Example before (rfind) after (twoway_rfind) benchmark shows the removal
    of quadratic behavior.
    
    needle: "ab" * 100, haystack: ("bb" + "ab" * 100) * 100
    
    ```
    test periodic::rfind           ... bench:   1,926,595 ns/iter (+/- 11,390) = 10 MB/s
    test periodic::twoway_rfind    ... bench:      51,740 ns/iter (+/- 66) = 386 MB/s
    ```
    bluss committed Aug 2, 2015
    Configuration menu
    Copy the full SHA
    7ebae85 View commit details
    Browse the repository at this point in the history

Commits on Aug 7, 2015

  1. StrSearcher: Improve inner loop in TwoWaySearcher::next, next_back

    The innermost loop of TwoWaySearcher checks the boundary of the haystack
    vs position + needle.len(), and it checks the last byte of the needle
    against the byteset.
    
    If these two steps are combined by using the indexing of the last
    needle byte's position as bounds check, the algorithm improves its
    throughput. We improve the innermost loop by reducing the number of
    instructions used, and elminating the panic case for the checked
    indexing that was previously used.
    
    Selected benchmarks from the external/workspace testsuite. Benchmarks
    improve across the board.
    
    ```
    before:
    
    test bb_in_aa::twoway_find                  ... bench:       4,229 ns/iter (+/- 1,305) = 23646 MB/s
    test bb_in_aa::twoway_rfind                 ... bench:       3,873 ns/iter (+/- 101) = 25819 MB/s
    test short_1let_long::twoway_find           ... bench:       7,075 ns/iter (+/- 29) = 360 MB/s
    test short_1let_long::twoway_rfind          ... bench:       6,640 ns/iter (+/- 79) = 384 MB/s
    test short_2let_long::twoway_find           ... bench:       3,823 ns/iter (+/- 16) = 667 MB/s
    test short_2let_long::twoway_rfind          ... bench:       3,774 ns/iter (+/- 44) = 675 MB/s
    test short_3let_long::twoway_find           ... bench:       3,582 ns/iter (+/- 47) = 712 MB/s
    test short_3let_long::twoway_rfind          ... bench:       3,616 ns/iter (+/- 34) = 705 MB/s
    
    with this commit:
    
    test bb_in_aa::twoway_find                  ... bench:       2,952 ns/iter (+/- 20) = 33875 MB/s
    test bb_in_aa::twoway_rfind                 ... bench:       2,939 ns/iter (+/- 99) = 34025 MB/s
    test short_1let_long::twoway_find           ... bench:       4,593 ns/iter (+/- 4) = 555 MB/s
    test short_1let_long::twoway_rfind          ... bench:       4,592 ns/iter (+/- 76) = 555 MB/s
    test short_2let_long::twoway_find           ... bench:       2,804 ns/iter (+/- 3) = 909 MB/s
    test short_2let_long::twoway_rfind          ... bench:       2,807 ns/iter (+/- 40) = 908 MB/s
    test short_3let_long::twoway_find           ... bench:       3,105 ns/iter (+/- 120) = 821 MB/s
    test short_3let_long::twoway_rfind          ... bench:       3,019 ns/iter (+/- 50) = 844 MB/s
    ```
    
    - `bb_in_aa`: fast skip due to byteset filter loop improves.
    - 1/2/3let: Searches for 1, 2, or 3 ascii bytes improves.
    bluss committed Aug 7, 2015
    Configuration menu
    Copy the full SHA
    2b82c07 View commit details
    Browse the repository at this point in the history

Commits on Aug 16, 2015

  1. StrSearcher: Additional comments and small code moves

    Break out a separate static method to create the "byteset".
    bluss committed Aug 16, 2015
    Configuration menu
    Copy the full SHA
    01e8812 View commit details
    Browse the repository at this point in the history