import error Symbol not found: ____chkstk_darwin

[sumitjha4321]

I am getting below error when importing tokenizers.

Details:

• Platform: MacOS 10.13.6
• Installed using pip inside conda env.

ImportError: dlopen(/Users/sumit.jha/.conda/envs/kaggle/lib/python3.7/site-packages/tokenizers/tokenizers.cpython-37m-darwin.so, 2): Symbol not found: ____chkstk_darwin
Referenced from: /Users/sumit.jha/.conda/envs/kaggle/lib/python3.7/site-packages/tokenizers/tokenizers.cpython-37m-darwin.so (which was built for Mac OS X 10.15)
Expected in: /usr/lib/libSystem.B.dylib
in /Users/sumit.jha/.conda/envs/kaggle/lib/python3.7/site-packages/tokenizers/tokenizers.cpython-37m-darwin.so

[n1t0]

This is weird, I recently tested it on Yosemite to make sure it works, and everything was working fine indeed.

Would you mind trying in an env you create using python -m venv .env and install it with pip. I'd like to make sure this is not related to conda.

[chenyaoysu]

hi, i have the same issue here. Any help? Thanks

[ECon87]

Same issue here!

[ECon87]

This is weird, I recently tested it on Yosemite to make sure it works, and everything was working fine indeed.

Would you mind trying in an env you create using python -m venv .env and install it with pip. I'd like to make sure this is not related to conda.

I tried the above solution with Mac High Sierra and python 3.8. The problem persists

[n1t0]

Thank you for testing this. Can you try to install the 0.8.0rc1 or 0.8.0rc2? I suspect that it starts to fail in the more recent versions.
And if it doesn't work, would you mind trying a few other earlier versions to see if you find where it started to fail? (You can find a list of versions available on pip here: https://pypi.org/project/tokenizers/#history)

[HelloJocelynLu]

Got the same problem. Using Mac High Sierra and python 3.6. The problem should come with later versions for I installed transformers before without any errors. I reinstalled it today and got the error.
The attempt trying to pip install pip install tokenizers==0.8.0rc2 gives another error:

ERROR: transformers 3.0.1 has requirement tokenizers==0.8.0-rc4, but you'll have tokenizers 0.8.0rc2 which is incompatible.

Thank you for testing this. Can you try to install the 0.8.0rc1 or 0.8.0rc2? I suspect that it starts to fail in the more recent versions.
And if it doesn't work, would you mind trying a few other earlier versions to see if you find where it started to fail? (You can find a list of versions available on pip here: https://pypi.org/project/tokenizers/#history)

[n1t0]

This is happening because you try to install it with transformers, in the same environment. Can you try in a fresh env, while installing tokenizers only with pip install tokenizers==0.8.0rc2?

[HelloJocelynLu]

I tried with another environment:

Python 3.6.10 |Anaconda, Inc.| (default, May  7 2020, 23:06:31) 
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import tokenizers
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jocelynlu/anaconda3/envs/help/lib/python3.6/site-packages/tokenizers/__init__.py", line 17, in <module>
    from .tokenizers import Tokenizer, Encoding, AddedToken
ImportError: dlopen(/Users/jocelynlu/anaconda3/envs/help/lib/python3.6/site-packages/tokenizers/tokenizers.cpython-36m-darwin.so, 2): Symbol not found: ____chkstk_darwin
  Referenced from: /Users/jocelynlu/anaconda3/envs/help/lib/python3.6/site-packages/tokenizers/tokenizers.cpython-36m-darwin.so (which was built for Mac OS X 10.15)
  Expected in: /usr/lib/libSystem.B.dylib
 in /Users/jocelynlu/anaconda3/envs/help/lib/python3.6/site-packages/tokenizers/tokenizers.cpython-36m-darwin.so

It was true that I installed it via transformer, and I got this error when I import it.

This is happening because you try to install it with transformers, in the same environment. Can you try in a fresh env, while installing tokenizers only with pip install tokenizers==0.8.0rc2?

[mahowak]

I am having the same problem, with Python 3.7.6 on High Sierra. I tried to upgrade to the latest transformers package using pip and now have this issue.

[n1t0]

I understand that this is happening to a lot of persons. Unfortunately, I don't have a mac with a version before Catalina these days, so I can't debug this. If one of you could tell me what is the first version of tokenizers where this bug exists, then I would be able to understand why it happened and how to fix it.

So, if someone can try to install the versions below and report if they are working, that would be extremely helpful. Here is a list of the versions to test, in order:

• 0.7.0
• 0.8.0.dev0
• 0.8.0.dev1
• 0.8.0.dev2
• 0.8.0rc1
• 0.8.0rc2

[glf1030]

same issue here, try pip install tokenizers==0.7.0, then it works

[HelloJocelynLu]

It only happens in High Sierra, I tried the same thing in a linux system and everything went well

[ECon87]

It only happens in High Sierra, I tried the same thing in a linux system and everything went well

I can confirm this. I have both a linux distro and MacOS High Sierra. I haven't tried all the different version mentioned above though.

[sepal]

I understand that this is happening to a lot of persons. Unfortunately, I don't have a mac with a version before Catalina these days, so I can't debug this. If one of you could tell me what is the first version of tokenizers where this bug exists, then I would be able to understand why it happened and how to fix it.

So, if someone can try to install the versions below and report if they are working, that would be extremely helpful. Here is a list of the versions to test, in order:

* 0.7.0

* 0.8.0.dev0

* 0.8.0.dev1

* 0.8.0.dev2

* 0.8.0rc1

* 0.8.0rc2

Just tested it in a seperate virtual env: The error starts appearing with 0.8.0.dev1. With 0.8.0.dev0 I can at least still import tokenizers.

[Narsil]

Could it be linked to that ? nodegui/nodegui#391 (comment) Seems building with github actions by defaults make binaries incompatible with older macos versions.

[n1t0]

Awesome! Thank you @sepal, this is extremely helpful.

@Narsil Yes, totally related! I think in our case it is due to our dependencies. If we use only full rust crates, then it seems to work fine for older versions of macOS even when building from the last one. It probably gets funky when we start linking to external libs because I guess the linker does only include what's necessary for the last version of macOS.

So I guess, this is related to the addition of the onig dependency, as it was working right before we added it. Maybe libc would cause the same thing actually. We should be able to fix it with MACOSX_DEPLOYMENT_TARGET=10.9 added in the env, at least I hope so! (Like here: rust-lang/rustup#24)

[n1t0]

Version 0.8.1.rc2 is out and should work with macOS 10.11+.

@sepal Could you try and see if it works now?

[sepal]

@n1t0 just tested it the issue seems fixed with 0.8.1.rc2. Was able to import tokenizers and tokenize a text with BertWordPieceTokenizer. Thanks for the quick fix!

I guess we now have to also update the dependency in transformers in a PR, right?

[n1t0]

Great! I'm happy that it fixed it. Thank you for your help @sepal!

[ballwang]

i still encounter this issue, with transformer 3.0.2 and tokenizer 0.8.0.rc1, i tried 0.8.0.rc2, but it is not compatible

[n1t0]

Yes, it is actually compatible, but pip says otherwise because it thinks only 0.8.1.rc1 works with transformers. Anyway, you can install 0.8.1 which I am sure will work!

[ballwang]

[image: Screen Shot 2020-07-23 at 5.52.06 PM.png] Still. not working

…

On Thu, Jul 23, 2020 at 5:27 PM Anthony MOI ***@***.***> wrote: Yes, it is actually compatible, but pip says otherwise because it thinks only 0.8.1.rc1 works with transformers. Anyway, you can install 0.8.1 which I am sure will work! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#321 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APZV3LMSC6Z6H4ULNFUYPMTR5DIQ5ANCNFSM4OKR4AWQ> .

[ballwang]

[image: Screen Shot 2020-07-23 at 5.52.43 PM.png] sorry about the last posting. This one. it still incompatible

…

On Thu, Jul 23, 2020 at 5:52 PM Tianxing Wang ***@***.***> wrote: [image: Screen Shot 2020-07-23 at 5.52.06 PM.png] Still. not working On Thu, Jul 23, 2020 at 5:27 PM Anthony MOI ***@***.***> wrote: > Yes, it is actually compatible, but pip says otherwise because it thinks > only 0.8.1.rc1 works with transformers. Anyway, you can install 0.8.1 > which I am sure will work! > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <#321 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/APZV3LMSC6Z6H4ULNFUYPMTR5DIQ5ANCNFSM4OKR4AWQ> > . >

[n1t0]

Can't see any of your screenshots. Anyway, you can install tokenizers even if pip says otherwise:

$ pip install transformers
Collecting transformers
  Using cached https://files.pythonhosted.org/packages/27/3c/91ed8f5c4e7ef3227b4119200fc0ed4b4fd965b1f0172021c25701087825/transformers-3.0.2-py3-none-any.whl
Collecting filelock (from transformers)
  Using cached https://files.pythonhosted.org/packages/93/83/71a2ee6158bb9f39a90c0dea1637f81d5eef866e188e1971a1b1ab01a35a/filelock-3.0.12-py3-none-any.whl
Collecting sentencepiece!=0.1.92 (from transformers)
  Using cached https://files.pythonhosted.org/packages/f2/e2/813dff3d72df2f49554204e7e5f73a3dc0f0eb1e3958a4cad3ef3fb278b7/sentencepiece-0.1.91-cp37-cp37m-manylinux1_x86_64.whl
Collecting regex!=2019.12.17 (from transformers)
  Using cached https://files.pythonhosted.org/packages/c7/10/a5f40a296a199dda90432ba0af4dc4221aa5cde31c62b6088c88df971e9e/regex-2020.7.14-cp37-cp37m-manylinux2010_x86_64.whl
Collecting tokenizers==0.8.1.rc1 (from transformers)
  Using cached https://files.pythonhosted.org/packages/02/59/68c7e3833f535615fb97d33ffcb7b30bbf62bc7477a9c59cd19ad8535d72/tokenizers-0.8.1rc1-cp37-cp37m-manylinux1_x86_64.whl
Collecting tqdm>=4.27 (from transformers)
  Using cached https://files.pythonhosted.org/packages/af/88/7b0ea5fa8192d1733dea459a9e3059afc87819cb4072c43263f2ec7ab768/tqdm-4.48.0-py2.py3-none-any.whl
Collecting packaging (from transformers)
  Using cached https://files.pythonhosted.org/packages/46/19/c5ab91b1b05cfe63cccd5cfc971db9214c6dd6ced54e33c30d5af1d2bc43/packaging-20.4-py2.py3-none-any.whl
Collecting requests (from transformers)
  Using cached https://files.pythonhosted.org/packages/45/1e/0c169c6a5381e241ba7404532c16a21d86ab872c9bed8bdcd4c423954103/requests-2.24.0-py2.py3-none-any.whl
Collecting numpy (from transformers)
  Using cached https://files.pythonhosted.org/packages/50/8f/29d5688614f9bba59931683d5d353d738d4a3007833219ee19c455732753/numpy-1.19.1-cp37-cp37m-manylinux2010_x86_64.whl
Collecting sacremoses (from transformers)
  Using cached https://files.pythonhosted.org/packages/7d/34/09d19aff26edcc8eb2a01bed8e98f13a1537005d31e95233fd48216eed10/sacremoses-0.0.43.tar.gz
Collecting pyparsing>=2.0.2 (from packaging->transformers)
  Using cached https://files.pythonhosted.org/packages/8a/bb/488841f56197b13700afd5658fc279a2025a39e22449b7cf29864669b15d/pyparsing-2.4.7-py2.py3-none-any.whl
Collecting six (from packaging->transformers)
  Using cached https://files.pythonhosted.org/packages/ee/ff/48bde5c0f013094d729fe4b0316ba2a24774b3ff1c52d924a8a4cb04078a/six-1.15.0-py2.py3-none-any.whl
Collecting idna<3,>=2.5 (from requests->transformers)
  Using cached https://files.pythonhosted.org/packages/a2/38/928ddce2273eaa564f6f50de919327bf3a00f091b5baba8dfa9460f3a8a8/idna-2.10-py2.py3-none-any.whl
Collecting chardet<4,>=3.0.2 (from requests->transformers)
  Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests->transformers)
  Using cached https://files.pythonhosted.org/packages/5e/c4/6c4fe722df5343c33226f0b4e0bb042e4dc13483228b4718baf286f86d87/certifi-2020.6.20-py2.py3-none-any.whl
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests->transformers)
  Using cached https://files.pythonhosted.org/packages/9f/f0/a391d1463ebb1b233795cabfc0ef38d3db4442339de68f847026199e69d7/urllib3-1.25.10-py2.py3-none-any.whl
Collecting click (from sacremoses->transformers)
  Using cached https://files.pythonhosted.org/packages/d2/3d/fa76db83bf75c4f8d338c2fd15c8d33fdd7ad23a9b5e57eb6c5de26b430e/click-7.1.2-py2.py3-none-any.whl
Collecting joblib (from sacremoses->transformers)
  Using cached https://files.pythonhosted.org/packages/51/dd/0e015051b4a27ec5a58b02ab774059f3289a94b0906f880a3f9507e74f38/joblib-0.16.0-py3-none-any.whl
Installing collected packages: filelock, sentencepiece, regex, tokenizers, tqdm, pyparsing, six, packaging, idna, chardet, certifi, urllib3, requests, numpy, click, joblib, sacremoses, transformers
  Running setup.py install for sacremoses ... done
Successfully installed certifi-2020.6.20 chardet-3.0.4 click-7.1.2 filelock-3.0.12 idna-2.10 joblib-0.16.0 numpy-1.19.1 packaging-20.4 pyparsing-2.4.7 regex-2020.7.14 requests-2.24.0 sacremoses-0.0.43 sentencepiece-0.1.91 six-1.15.0 tokenizers-0.8.1rc1 tqdm-4.48.0 transformers-3.0.2 urllib3-1.25.10
You are using pip version 19.0.3, however version 20.2b1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

$ pip install tokenizers==0.8.1
Collecting tokenizers==0.8.1
  Using cached https://files.pythonhosted.org/packages/25/fc/775a802d0d59707a0a6448b0457c7222bd3d65a8232fbd02d904bae255b5/tokenizers-0.8.1-cp37-cp37m-manylinux1_x86_64.whl
transformers 3.0.2 has requirement tokenizers==0.8.1.rc1, but you'll have tokenizers 0.8.1 which is incompatible.
Installing collected packages: tokenizers
  Found existing installation: tokenizers 0.8.1rc1
    Uninstalling tokenizers-0.8.1rc1:
      Successfully uninstalled tokenizers-0.8.1rc1
Successfully installed tokenizers-0.8.1
You are using pip version 19.0.3, however version 20.2b1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

$ python
Python 3.7.3 (default, Apr 15 2020, 10:51:02)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tokenizers
>>> tokenizers.__version__
'0.8.1'
>>> import transformers
>>> transformers.__version__
'3.0.2'

See above, it says it is incompatible, and yet it works 🙂