Unicode support / word tokens

[4kimov]

Carrying over discussion of unicode from https://github.com/orgs/hashids/discussions/92#discussioncomment-6223192

@niieani, I want to be sure I understand the problem correctly first. If the alphabet was 1 string of many emojis of different lengths, was the issue that the length of the string did not match the number of emojis (which I assume broke encoding/decoding)?

[niieani]

Yes, that's correct. The problem is that most emojis are actually multi-byte Unicode characters. This is also the case with many non-English alphabets. My proposal for a solution to this is that instead of treating the alphabet as a long string, it might be favorable to request the use to input the alphabet as a list of strings, i.e. ['a', 'b', 'c', ...] instead of 'abc...'.
At least in JavaScript, this also turned out to be a lot more performant, hence in the latest version of Hashids I've rewritten it to internally use arrays.

It's easy to transform a plain language alphabet string to an Array by invoking 'abc...'.split(''), and the opposite is also true: ['a', 'b', 'c', ...].join(''). This also enables us to support words as elements of the alphabet essentially for free.

The only caveat is that in such a case one of two statements about the alphabet need to be true:

• each element of the alphabet cannot be a substring of another element of the alphabet (that would cause ambiguity - i.e. you cannot use both 'cat' and 'caterpillar' as separate "letters" of the alphabet), or
• when using alphabets that contain multi-character elements, they must be separated in the output by using a separator character in the sqid output, e.g. a dash.

In practice, this is not a problem for regular alphabets as used with Hashids, or even with alphabets containing any multibyte unicode characters, which guarantee uniqueness.

If we don't care about multi-character "letters", the simplest way to solve this is to just disallow them.

Alternatively, if we want to support multi-character letters e.g. for token usage, we could add a simple validateAlphabet method that is called during initialization of sqids, and checks whether the alphabet is valid. Alphabet could be configured as an object with an optional separator. For example:

const sqids = new Sqids({ alphabet: { tokens: ANIMAL_NAMES, separator: '-' } })
const result = sqids.encode([1, 2, 3])
result === 'snake-octopus-zebra-elephant-rabbit'

or

const sqids = new Sqids({ alphabet: EMOJIS })
const result = sqids.encode([1, 2, 3])
result === `🔥🐈‍⬛✈️😮🧠`

[4kimov]

@niieani, I see, thank you for clarifying.

At first glance, I don't see it as a difficult change to accept an array of chars/strings in the constructor, as opposed to a single string of alphabet (other than typing/formatting for the custom alphabet end users).

I guess my concerns are regarding the possibilities + complexities that this opens up. But first, I did a quick search to see if anyone had similar issues, so far I've found these:

• Unicode alphabet unsupported? vinkla/hashids#87 [cc @vinkla]
• Unicode alphabets not supported? davidaurelio/hashids-python#22 [cc @davidaurelio]
• Provide an API that support wide characters tzvetkoff/hashids.c#5 [cc @tzvetkoff]

[Maybe there's more?]

So far I see that encoding/decoding emojis would be possible, but I'm wondering what the practical use-case for this is.

Right now, v2 advertises support for avoiding most common profanity words (because it'd be unfortunate if they showed up randomly in the URL). I'm a bit worried that if we claim to officially support Unicode characters, people might assume we have baked in a solution to treat characters like û, ü, ù, ú, ū as u (right now we don't). And if we were to add support for this, it would make Sqids library bigger (nothing necessarily wrong with that, but something to keep in mind).

Regarding word tokens: if we support IDs like snake-octopus-zebra-elephant-rabbit, I'm not sure users would need this library anymore (no need for blacklist filtering + min length padding). They might be better off writing a simple function for that.

Generally, I do see it as a plus when random codebases support Unicode, but for this library I'm a bit concerned for the complexities this opens up and the use-cases it encourages.

Therefore, let me ask this - what's the one practical use-case where supporting Unicode would make this library better (& justify the complexity)?

[tzvetkoff]

I would personally NOT put UTF8 as a direct requirement to the library, since it's quirky and cumbersome to implement in "classic" programming languages, and most people will never actually use it.

It would additionally add more ambiguity (i.e. blah vs blаh (the second one includes a Cyrillic а.)
Additionally, if we decide to put it in the spec, we'll need to add a LOT more curse-words in the default blacklist.

Using custom tokens (multichar strings) and separators instead of just characters is one other thing I'd happily avoid - again, no library in the world can (and SHOULDN'T) accommodate all possible needs. Think of it the UNIX way - do one thing but do it good.

As for the general UTF8 support, I plan to actually have it by using a few preprocessor directives, which would allow for enabling/disabling it on compile time (I ported a micro-UTF8 library a few years ago while working on a Lua propject.)

[4kimov]

@tzvetkoff Thank you for the feedback. Good points. What do you think about the constructor API: accepting a string vs an array of chars?

[tzvetkoff]

@4kimov, sorry for the late response -- I'd rather keep it as a regular string, since it's the classic way of doing things.

[4kimov]

@niieani thoughts on any of the above? My impression is that unicode requests are mainly for emoji shuffling. I'm trying to figure out if there's an important use-case I'm missing...

[niieani]

Unicode is not a must if we want to keep things very simple. Emojis are a nice-to-have.

If we make the alphabet an array with characters, rather than a long string (which forces the implicit assumption that 1 byte = 1 character = 1 element of the alphabet), we could potentially get unicode support cheaply, without actually having to specifically "support unicode" (we treat strings as arbitrary groups of bytes). I can try to hack something together to see how it could work in practice.

[4kimov]

Ok, thank you for clarifying.

At this point I'm leaning more towards leaving as, because emoji encoding reminds me of trying to hide information, and I wouldn't want this library to encourage that (Hashids had that pushback before). Internally I think the libraries should pick whatever storage they want, but I do think it's easier to pass a string for now as is.

Plus later if there is a nice use-case where an array of chars is needed, I think it should be backwards-compatible with the default alphabet, so we could introduce it without breaking things.

[4kimov]

Closing for now. Let's revisit when we get more feedback.