Switch to Cython language level 3

[pabs3]

Python 2 is not supported by gensim, so
switching to language level 3 should be fine.

Silences a warning from Cython:

FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release!

[piskvorky]

Thanks. Where is that warning shown? If some extensions are being built with an incorrect language level, that sounds dangerous.

CI is failing though.

[pabs3]

The warning is shown during the build of the Debian package of gensim (amd64 log). Side note: the log also shows various GCC warnings, some test warnings and a warning about setup.py being deprecated within the Python community.

Unfortunately based on the CI results for this PR, it looks like the Cython code in gensim isn't yet compatible with Cython language version 3, so I'm going to mark this PR as draft and work on fixing the issues as I get time.

[pabs3]

Hmm, maybe I should instead mark those files as language level 2 for now, to silence the warnings.

[piskvorky]

language_level is about the level of Python (2 vs 3), right? Not Cython, which works with either.

So we should have language_level=3 everywhere in Gensim. Unless I'm missing something.

[pabs3]

I thought so too, but the CI fails when I add that everywhere.

…

-- bye, pabs https://bonedaddy.net/pabs3/

[piskvorky]

Hmm, the CI errors complain about syntax stuff in the Cython code like gensim/models/fasttext_inner.pyx:187:9: 'REAL_t' is not a type identifier. Which is unrelated to py2/3 IMO, weird. Will need a deeper look.

[piskvorky]

The problem seems to stem from this code:

https://github.com/RaRe-Technologies/gensim/blob/5a19dd0407577001bcc177166582da7a5a4e5f70/gensim/models/word2vec_inner.pxd#L125-L126

which seems indeed syntactically broken:

cdef init_w2v_config(Word2VecConfig *c, model, alpha, compute_loss, _work, _neu1=*)
                                                                                 ^
------------------------------------------------------------

gensim/models/word2vec_inner.pxd:126:82: Expected an identifier or literal

The code comes from 3c3506d in #2127.

Or is it some special Cython syntax for PXD? I don't get it. CC @persiyanov @mpenkov – any idea why Cython compilation with language_level=3 fails?

[piskvorky]

My hypothesis: error is related to the compilation order of Gensim extensions.

1. fasttext_inner.pyx depends on word2vec_inner.pyx
2. But with language_level=3, the fasttext_inner.pyx gets cythonized first => failure.
3. Under language_level=2, word2vec_inner.pyx is cythonized first and everything works.

Checking my hypothesis now :)

[piskvorky]

OK I think I figured it out. The issue is with relative imports (from word2vec_inner import …) in these pyx/pdx files. After changing them to absolute (from gensim.models.word2vec_inner import …) the compilation works even with language_level=3.

@pabs3 I'll push the fix to your branch, OK?

[piskvorky]

Hm, I get a permission error when I try to push to your branch directly (usually possible for maintainers across forks but maybe you disabled this).

And github doesn't let me open a PR against your fork (doesn't even list it??). So can you please apply 0e70635 yourself @pabs3? Cheers.

[pabs3]

I usually disable maintainer edits of my patches; had some suboptimal experiences with that in the past. I think it would be best to merge your patch first, since otherwise git bisects will have to skip my patch since it will be unbuildable, but if your patch goes in first, then both patches will be bisect buildable. Anyway, I've added your patch to the top of my branch too.

…

-- bye, pabs https://bonedaddy.net/pabs3/

[mpenkov]

Thank you @pabs3 !