Language specific sorting

[Fronix]

We're having some issues using the alphabetical sorting function in meilisearch, I've gotten an explaination from @many in discord and he says the following:

For the sort feature and the asc/desc ranking rule we decompose the characters containing diacritics (NFKD), for instance:
å (U+00E5) is decomposed in a (U+0061) - ◌̊ (U+030A).
And the sort will be a lexicographic sort based on this decomposition.
This means that å and ä will be sorted this way:

1. ä (U+0061 - U+0308)
2. å (U+0061 - U+030A)
   because the diacritic ◌̈ (U+0308) is under ◌̊ (U+030A).

Related discord thread

This is quite problematic for languages like Swedish where å ä ö are the last 3 letters in the alphabet and in meilisearch they are treated as a a and o which fall under very different sorting rules.

Here's and example of how meilisearch sorts today

"hits": [
    {
        "namn": "Aland hejåala"
    },
    {
        "namn": "Ämanprocess" // <-- This is treated as A
    },
    {
        "namn": "Åbersprocess" // <-- This is treated as A
    },
    {
        "namn": "Jabbaprocess"
    },
    {
        "namn": "Ömödaprocess" // <-- This is treated as O
    },
    {
        "namn": "Palaprocess"
    },
    {
        "namn": "Zprocess Värl"
    }
]

But in reality this is how it should be sorted

"hits": [
    {
        "namn": "Aland hejåala"
    },
    {
        "namn": "Jabbaprocess"
    },
    {
        "namn": "Palaprocess"
    },
    {
        "namn": "Zprocess Värl"
    },
    {
        "namn": "Åbersprocess" // <-- Treated as the 3rd last letter in the Swedish alphabet
    },
    {
        "namn": "Ämanprocess" // <-- Treated as the 2nd last letter in the Swedish alphabet
    },
    {
        "namn": "Ömödaprocess"// <-- Treated as the last letter in the Swedish alphabet
    }
]

What I would like is a way to maybe configure how to handle this type of sorting, either by configuration or at searchtime.
So you could add sortLanguage: 'sv' in the index config or at searchtime.

[macraig]

Hi @Fronix , thank you for taking the time to explain your use case, it is very valuable. We're currently busy working on other priorities, but I'll keep this discussion open to revisit in the future.

[curquiza]

Related issue here: meilisearch/meilisearch#4133