Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Having trouble building python-zstandard for pypy2 on windows #61

Closed
nma-ro opened this issue Nov 2, 2018 · 2 comments
Closed

Having trouble building python-zstandard for pypy2 on windows #61

nma-ro opened this issue Nov 2, 2018 · 2 comments

Comments

@nma-ro
Copy link

nma-ro commented Nov 2, 2018 •
edited
Loading

Recently i was unsuccessfully trying to build module for pypy2-v6.0.0-win32 on windows 10 x64:

C:\Program Files (x86)\Microsoft Visual C++ Build Tools>c:\pypy2-v6.0.0-win32\pypy.exe -m pip install zstandard
Collecting zstandard
  Using cached https://files.pythonhosted.org/packages/29/21/4612a4b9e628d61aa045558ff008452378b5a333e5a64f7de29dee8b1e77/zstandard-0.10.1.tar.gz
Requirement already satisfied: cffi>=1.11 in c:\pypy2-v6.0.0-win32\lib_pypy (from zstandard) (1.11.5)
Installing collected packages: zstandard
  Running setup.py install for zstandard ... error
    Complete output from command c:\pypy2-v6.0.0-win32\pypy.exe -u -c "import setuptools, tokenize;__file__='c:\\users\\admin\\appdata\\local\\temp\\pip-install-sjrjic\\zstandard\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record c:\users\admin\appdata\local\temp\pip-record-2zhkwh\install-record.txt --single-version-externally-managed --compile:
    Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 15.00.30729.01 for 80x86
    Copyright (C) Microsoft Corporation.  All rights reserved.

    tmp_ppnnx.h
    Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 15.00.30729.01 for 80x86
    Copyright (C) Microsoft Corporation.  All rights reserved.

    tmpbltlc_.h
    generating build\_zstd_cffi.c
    (already up-to-date)
    not modified: 'build\\_zstd_cffi.c'
    running install
    running build
    running build_py
    creating build\lib.win32-2.7
    creating build\lib.win32-2.7\zstandard
    copying zstandard\__init__.py -> build\lib.win32-2.7\zstandard
    running build_ext
    building 'zstd' extension
    creating build\temp.win32-2.7
    creating build\temp.win32-2.7\Release
    creating build\temp.win32-2.7\Release\zstd
    creating build\temp.win32-2.7\Release\zstd\common
    creating build\temp.win32-2.7\Release\c-ext
    creating build\temp.win32-2.7\Release\zstd\compress
    creating build\temp.win32-2.7\Release\zstd\decompress
    creating build\temp.win32-2.7\Release\zstd\dictBuilder
    C:\Users\admin\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -Ic-ext -Izstd\common -Izstd -Izstd\compress -Izstd\decompress -Izstd\dictBuilder -Ic:\pypy2-v6.0.0-win32\include /Tczstd\common\pool.c /Fobuild\temp.win32-2.7\Release\zstd\common\pool.obj -DZSTD_MULTITHREAD -DZSTDLIB_VISIBILITY= -DZDICTLIB_VISIBILITY= -DZSTDERRORLIB_VISIBILITY=
    pool.c
    C:\Users\admin\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -Ic-ext -Izstd\common -Izstd -Izstd\compress -Izstd\decompress -Izstd\dictBuilder -Ic:\pypy2-v6.0.0-win32\include /Tczstd\common\threading.c /Fobuild\temp.win32-2.7\Release\zstd\common\threading.obj -DZSTD_MULTITHREAD -DZSTDLIB_VISIBILITY= -DZDICTLIB_VISIBILITY= -DZSTDERRORLIB_VISIBILITY=
    threading.c
    C:\Users\admin\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -Ic-ext -Izstd\common -Izstd -Izstd\compress -Izstd\decompress -Izstd\dictBuilder -Ic:\pypy2-v6.0.0-win32\include /Tczstd.c /Fobuild\temp.win32-2.7\Release\zstd.obj -DZSTD_MULTITHREAD -DZSTDLIB_VISIBILITY= -DZDICTLIB_VISIBILITY= -DZSTDERRORLIB_VISIBILITY=
    zstd.c
    C:\Users\admin\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -Ic-ext -Izstd\common -Izstd -Izstd\compress -Izstd\decompress -Izstd\dictBuilder -Ic:\pypy2-v6.0.0-win32\include /Tcc-ext\bufferutil.c /Fobuild\temp.win32-2.7\Release\c-ext\bufferutil.obj -DZSTD_MULTITHREAD -DZSTDLIB_VISIBILITY= -DZDICTLIB_VISIBILITY= -DZSTDERRORLIB_VISIBILITY=
    bufferutil.c
    C:\Users\admin\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -Ic-ext -Izstd\common -Izstd -Izstd\compress -Izstd\decompress -Izstd\dictBuilder -Ic:\pypy2-v6.0.0-win32\include /Tcc-ext\compressiondict.c /Fobuild\temp.win32-2.7\Release\c-ext\compressiondict.obj -DZSTD_MULTITHREAD -DZSTDLIB_VISIBILITY= -DZDICTLIB_VISIBILITY= -DZSTDERRORLIB_VISIBILITY=
    compressiondict.c
    C:\Users\admin\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -Ic-ext -Izstd\common -Izstd -Izstd\compress -Izstd\decompress -Izstd\dictBuilder -Ic:\pypy2-v6.0.0-win32\include /Tcc-ext\compressobj.c /Fobuild\temp.win32-2.7\Release\c-ext\compressobj.obj -DZSTD_MULTITHREAD -DZSTDLIB_VISIBILITY= -DZDICTLIB_VISIBILITY= -DZSTDERRORLIB_VISIBILITY=
    compressobj.c
    c-ext\compressobj.c(62) : error C2065: 'ssize_t' : undeclared identifier
    c-ext\compressobj.c(62) : error C2146: syntax error : missing ')' before identifier 'input'
    c-ext\compressobj.c(62) : error C2059: syntax error : ')'
    c-ext\compressobj.c(62) : warning C4018: '<' : signed/unsigned mismatch
    c-ext\compressobj.c(62) : error C2143: syntax error : missing ';' before '{'
    c-ext\compressobj.c(62) : warning C4552: '<' : operator has no effect; expected operator with side-effect
    error: command 'C:\\Users\\admin\\AppData\\Local\\Programs\\Common\\Microsoft\\Visual C++ for Python\\9.0\\VC\\Bin\\cl.exe' failed with exit status 2

    ----------------------------------------
Command "c:\pypy2-v6.0.0-win32\pypy.exe -u -c "import setuptools, tokenize;__file__='c:\\users\\admin\\appdata\\local\\temp\\pip-install-sjrjic\\zstandard\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record c:\users\admin\appdata\local\temp\pip-record-2zhkwh\install-record.txt --single-version-externally-managed --compile" failed with error code 1 in c:\users\admin\appdata\local\temp\pip-install-sjrjic\zstandard\

Please, help me to understand what`s went wrong and how to fix it.
@indygreg
Copy link
Owner

indygreg commented Nov 3, 2018

There isn't explicit test coverage of python-zstandard on PyPy on Windows in CI. However, there is test coverage of 32-bit Windows on CPython 2.7. An example build for that is at https://ci.appveyor.com/project/indygreg/python-zstandard/builds/20028053/job/5xa633pei1siq87s.

If we compare the failing command invocation for compiling compressobj.c, the differences are:

good: -IC:\Python27\include -IC:\projects\python-zstandard\.tox\py27\PC /WX
bad: -Ic:\pypy2-v6.0.0-win32\include

/WX enables all compiler warnings as errors. So that's harmless.

The other differences boil down to include paths for CPython versus PyPy.

Since the compilation is complaining about missing ssize_t, my guess is CPython's headers include a header defining this symbol (or define it themselves) and PyPy's don't.

I reckon this can be fixed by adding a #include somewhere in python-zstandard.

indygreg added a commit that referenced this issue Nov 3, 2018
@indygreg
c-ext: change casts between Py_ssize_t and size_t
d2228c4 
In all these cases, we are comparing a Py_ssize_t (from Python
obviously) with a size_t in zstd. The Py_ssize_t type is signed.
But in these cases, the value refers to the length of a Py_buffer.
The length of a Py_buffer can never be negative. And the size
of Py_ssize_t must be the same as size_t. So it follows that casting
a Py_buffer to size_t should be safe.

This avoids the use of ssize_t in our code. And this should hopefully
unblock building the extension with PyPy on Windows.

Fixes #61.
@indygreg
Copy link
Owner

indygreg commented Nov 3, 2018

I can't guarantee things are fixed or will stay fixed since there isn't CI coverage of PyPy on Windows, but I think the upcoming 0.10.2 release will fix this ssize_t issue.

You should definitely run the tests to make sure things fail. And you may want to file an issue to track running PyPy on Windows in this project's CI. It looks to be possible with AppVeyor...

@indygreg indygreg closed this as completed Nov 3, 2018
indygreg added a commit that referenced this issue Nov 3, 2018
@indygreg
c-ext: change casts between Py_ssize_t and size_t
256a976 
In all these cases, we are comparing a Py_ssize_t (from Python
obviously) with a size_t in zstd. The Py_ssize_t type is signed.
But in these cases, the value refers to the length of a Py_buffer.
The length of a Py_buffer can never be negative. And the size
of Py_ssize_t must be the same as size_t. So it follows that casting
a Py_buffer to size_t should be safe.

This avoids the use of ssize_t in our code. And this should hopefully
unblock building the extension with PyPy on Windows.

Fixes #61.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@indygreg @nma-ro