Commits on Mar 12, 2017

1. Add support for additional text encodings. …

   This includes, but is not limited to, UTF-16, latin-1, GBK, EUC-JP and
   Shift_JIS. (Courtesy of the `encoding_rs` crate.)
   
   Specifically, this feature enables ripgrep to search files that are
   encoded in an encoding other than UTF-8. The list of available encodings
   is tied directly to what the `encoding_rs` crate supports, which is in
   turn tied to the Encoding Standard. The full list of available encodings
   can be found here: https://encoding.spec.whatwg.org/#concept-encoding-get
   
   This pull request also introduces the notion that text encodings can be
   automatically detected on a best effort basis. Currently, the only
   support for this is checking for a UTF-16 bom. In all other cases, a
   text encoding of `auto` (the default) implies a UTF-8 or ASCII
   compatible source encoding. When a text encoding is otherwise specified,
   it is unconditionally used for all files searched.
   
   Since ripgrep's regex engine is fundamentally built on top of UTF-8,
   this feature works by transcoding the files to be searched from their
   source encoding to UTF-8. This transcoding only happens when:
   
   1. `auto` is specified and a non-UTF-8 encoding is detected.
   2. A specific encoding is given by end users (including UTF-8).
   
   When transcoding occurs, errors are handled by automatically inserting
   the Unicode replacement character. In this case, ripgrep's output is
   guaranteed to be valid UTF-8 (excluding non-UTF-8 file paths, if they
   are printed).
   
   In all other cases, the source text is searched directly, which implies
   an assumption that it is at least ASCII compatible, but where UTF-8 is
   most useful. In this scenario, encoding errors are not detected. In this
   case, ripgrep's output will match the input exactly, byte-for-byte.
   
   This design may not be optimal in all cases, but it has some advantages:
   
   1. In the happy path ("UTF-8 everywhere") remains happy. I have not been
      able to witness any performance regressions.
   2. In the non-UTF-8 path, implementation complexity is kept relatively
      low. The cost here is transcoding itself. A potentially superior
      implementation might build decoding of any encoding into the regex
      engine itself. In particular, the fundamental problem with
      transcoding everything first is that literal optimizations are nearly
      negated.
   
   Future work should entail improving the user experience. For example, we
   might want to auto-detect more text encodings. A more elaborate UX
   experience might permit end users to specify multiple text encodings,
   although this seems hard to pull off in an ergonomic way.
   
   Fixes #1

   

   BurntSushi committed Mar 12, 2017
   Configuration menu

   • View commit details
   • Copy full SHA for 92f1422
   • Browse repository at this point

   Copy the full SHA

   92f1422 View commit details

   Browse the repository at this point in the history