Analysis ICU Plugin

[kimchy]

A plugin using ICU (http://icu-project.org/) to allow for unicode normalization, collation and folding. The plugin includes the following analysis token files:

ICU Normalization:

Normalizes characters as explained here: http://userguide.icu-project.org/transforms/normalization. it registeres itself by default under icu_normalizer or icuNormalizer using the default settings. Allows for the name parameter to be provided which can include the following values: nfc, nfkc, and nfkc_cf.

Sample setting:

curl -XPUT 'http://localhost:9200/test_1/'  -d '
{
    "index" : {
        "analysis" : {
            "analyzer" : {
                "collation" : {
                    "tokenizer" : "keyword",
                    "filter" : ["icu_normalizer"]
                }
            }
        }
    }
}
'

ICU Folding:

Folding of unicode characters based on UTR#30. It registeres itself under icu_folding and icuFolding names. Sample setting:

curl -XPUT 'http://localhost:9200/test_1/'  -d '
{
    "index" : {
        "analysis" : {
            "analyzer" : {
                "collation" : {
                    "tokenizer" : "keyword",
                    "filter" : ["icu_folding"]
                }
            }
        }
    }
}
'

ICU Collation:

Uses collation token filter. Allows to either specify the rules for collation (defined here http://www.icu-project.org/userguide/Collate_Customization.html) using the rules parameter (can point to a location or expressed in the settings, location can be relative to config location), or using the language parameter (further specialized by country and variant). By default registers under icu_collation or icuCollation and uses the default locale.

Sample settings:

curl -XPUT 'http://localhost:9200/test_1/'  -d '
{
    "index" : {
        "analysis" : {
            "analyzer" : {
                "collation" : {
                    "tokenizer" : "keyword",
                    "filter" : ["icu_collation"]
                }
            }
        }
    }
}
'

Sample custom collation:

curl -XPUT 'http://localhost:9200/test_1/'  -d '
{
    "index" : {
        "analysis" : {
            "analyzer" : {
                "collation" : {
                    "tokenizer" : "keyword",
                    "filter" : ["icu_collation"]
                }
            },
            "filter" : {
                "myCollator" : {
                    type : "icu_collator",
                    language : "en"
                }
            }
        }
    }
}
'

[kimchy]

Analysis ICU Plugin, closed by 11e4ad9