Add support for custom tokenizers ngram and regex.

[fmassot]

Fix #3056 and #3392

This PR adds support for custom analyzers, aka tokenizer + filters.

Doc mapping config with custom tokenizers

doc_mapping:
  tokenizers:
    - name: service_regex
      type: regex
      pattern: "\\w*"
  field_mappings:
    - name: service
      type: text
      tokenizer: service_regex

Endpoint to test a custom tokenizer

POST /analyze

with payload

{
  "type": "ngram",
  "min_gram": 3,
  "max_gram": 5,
  "text": "hello"
}

curl -XPOST http://localhost:7280/api/v1/analyze -H "content-type: application/json" --data '{"type": "regex", "pattern": "\\w+", "text":"hello world"}'
[
  {
    "offset_from": 0,
    "offset_to": 5,
    "position": 0,
    "position_length": 1,
    "text": "hello"
  },
  {
    "offset_from": 6,
    "offset_to": 11,
    "position": 1,
    "position_length": 1,
    "text": "world"
  }
]

Follow-up in PRs to come

• Add documentation.
• Add multilang tokenizer
• What else?

[fulmicoton]

Why are those public? Is it for the rest endpoint?

[fmassot]

Good catch, only "analyze_text, TokenizerConfig, TokenizerEntry" should be public and the others should be pub(crate) for the OpenAPI stuff.

[fulmicoton]

I only had minor comments, so I approved.