MSVC build environment. #2085
MSVC build environment. #2085
Conversation
The way the current build system worked it limited the build to only being able to be performed on a Windows 10+ system when compiling on Windows. This would require a user to have the Unix subsystem feature that is apart of Windows installed. Most users that are going to build wxPython will have Git installed and Git comes with bash.exe so why not use it?
|
HEY!!! Finally.. all windows builds are passing. This should make it a bit easier when removing distutils as the mechanism to handle the compiling. Because the environment gets set any creation of new processes by the build script will cause the environment to get carried over into the new process... The msvc script does detect cmake and ninja and I can add a property to the VisualCInfo class so the build script will be able to know if ninja is available and if it is then ninja can be used for compiling wxWidgets which would trim down the time it takes to compile. ninja and cmake are easier to install and use because they are apart of VisualStudio tho I would have to check and see if they get installed by default. Right now the comtypes dependency is pointing to a reporitory I made that has a couple of fixes in it. When a new version of comtypes gets released that has these fixes included then we can change that dependency so it will download directly from pypi |
|
Happy to see is working. Waiting for numpy release version for Python 3.11 and see if this work. If is possible keeping notes |
|
I am as well. It shouldn't break at all if a newer version of Python is used. What will break is the whole wxPython build system when Python 3.12 if released because of the removal if distutils. I do not feel that distutils should be removed with a minor version change as the entire packaging system has been built around distutils for how long now? They are not giving enough time to make the migration and they don't even have a suitable replacement in place yet!!! |
|
@kdschlosser is there anything wxPython-specific about the modules in buildtools/msvc? If not, have you considered publishing those modules separately? |
|
I think |
Oh, great! Yeah I think that would be preferred over bundling the code in wxPython. Thanks @kdschlosser @oleksis for your work. |
|
WARNING LONG READ There is nothing that is wxPythonspecific in the build environment code. I could release it on pypi but the problems would be as in wxPython is collecting it programmatically and not installing it into a users site-packages when it is only going to get used for a single time. setuptools offers a way to collect packages that are only used for the purposes of building a library and puts them into a temporary folder and not into a users site-packages. Unfortunatly this feature is now deprecated and there is no mechanism available to do this. In older versions of setuptools the wheel package was not needed in order to use that feature. Now wheel is needed and that in and of it's self now makes it impossible to use without installing one time use packages into the users site-packages. The build system in wxPython is it's own animal, it is not apart of the setup program so requirements,txt does not get parsed unless it is specifically told to. So that has to be known if compiling wxPython. Say a user compiles wxPython using virtualenv and then wants to install wxPython into their production environment. Fantastic!! so the user runs this command to install the compiled wxPython. See where the issue is?? setuptools has a mechanism to handle this and is slated for removal with no replacement because it has been slated for removal it has not been maintained and no longer functions properly. you have pyproject.toml which is supposed to handle the build system. It is defined in PEPs 517, 518, 621 and 660. It has been around for 6 years and doesn't get used by much. I can understand as to why because this is the limit of the documentation on how to use it. which is exceedingly limited and leave you more lost on how to use it then when you initially started reading the documentation. In PEP518 There is specific mention as to various file type parsers and which ones are apart of the standard library. It states there is no toml parser that is apart of the standard library that that "pytoml" is only 300 lines of code. It does not state if pytoml needs to be installed or not. It does not mention the inclusion of pytoml into pip or any other build system for that matter. And when visiting pytoml's repository it is soon discovered that the library has been deprecated and is no longer being developed. This leaves a large ? as to whether or not the pyproject.toml file is supported properly. The lack of information and the information that is provided leads to deprecated modules leads one to believe it is an idea that has become stale because it never gained any traction. One of the PEPs mentions that setuptools "setup_requires" parameter uses a hook that is built into pip in order to utilize the section of the pyproject.toml file that installs build modules. After inspection of the setuptools code that handles setup_requires there is no hook into pip that is used, in fact subprocess is used to call pip to install the package into an alternate directory. It does not use any hook located in pip. The people over at Python have not thought things through with coming up with a unified build/install ecosystem. They have jumped the gun and started to deprecate things with no replacement available or even thought of. Here is the warning that setup tools outputs when using setup_requires. "setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer." That leaves a person with a giant ? over their head. It provides no indication as to what the replacement is. This is the reason why thee are 3rd party build systems that exist like poetry. They have slated distutils for removal in < 2 years. with the replacement being setuptools and setuptools has a slew of things that have been deprecated in it with no replacements available. setuptools also relies on external modules to work. To complicate things further neither setuptools nor the modules it requires are apart of the stdlib. Everything that is needed in order to build and install a package for python should be included in the stdlib and if it is something like an external compiler say MSVC then the build system needs to be able to locate those external programs. This is where a lot of the failure comes in is that the majority of the people that develop and contribute to Python are the minority OS users and they do not use Windows and have no clue about writing code to set up a proper MSVC build environment. They also do not accept PRs that have been made to correct problems in Windows and bug reports pertaining to Windows go largely unanswered/ignored/unfixed. Now being that in one of the PEPs regarding pyproject.toml and it stating that setuptools uses a hook to facilitate package installs using the mechanisms put into place in pip and setuptools having a deprecated warning about that mechanism it leads one to believe that pyproject.toml files are on the out and are not going to be supported, If that is not the case then why would it be deprecated in setuptools? So now do we install packages that are only needed to compile wxPython and not have them get installed into the user normal Python environment when they are not needed to run wxPython??? If someone can answer that question for me then I will entertain the idea of releasing it to pypi. If there is no way then the code needs to be added directly to a packages build system which would make it a moot point to release it on pypi. To top it off I am having to battle with comtypes not being maintained like it should be and it not building properly due to it's reliance on build_py2to3 which is apart of distutils prior to python 3.9. This component was never deprecated and it was removed with zero notice. |
|
The last job pipeline don't run Pipeline working OK ✅##[section]Starting: build wxWidgets
==============================================================================
Task : Bash
Description : Run a Bash script on macOS, Linux, or Windows
Version : 3.195.0
Author : Microsoft Corporation
Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/bash
==============================================================================
Generating script.
[command]"C:\Program Files\Git\bin\bash.exe" -c pwd
/d/a/_temp
========================== Starting Command Output ===========================
[command]"C:\Program Files\Git\bin\bash.exe" /d/a/_temp/4c7790eb-a9a2-4bab-b040-41f24396f18f.sh
which: no cl.exe in (/mingw64/bin:/usr/bin:/c/Users/VssAdministrator/bin:/c/Users/VssAdministrator/AppData/Roaming/Python/Python39/Scripts:/c/hostedtoolcache/windows/Python/3.9.9/x86/Scripts:/c/hostedtoolcache/windows/Python/3.9.9/x86:/c/agents/2.196.2/externals/git/cmd:/c/agents/2.196.2/externals/git/mingw64/bin:/c/Program Files/MongoDB/Server/5.0/bin:/c/aliyun-cli:/c/vcpkg:/c/cf-cli:/c/Program Files (x86)/NSIS:/c/tools/zstd:/c/Program Files/Mercurial:/c/hostedtoolcache/windows/stack/2.7.3/x64:/c/cabal/bin:/c/ghcup/bin:/c/tools/ghc-9.2.1/bin:/c/Program Files/dotnet:/c/mysql/bin:/c/Program Files/R/R-4.1.2/bin/x64:/c/SeleniumWebDrivers/GeckoDriver:/c/Program Files (x86)/sbt/bin:/c/Program Files (x86)/GitHub CLI:/bin:/c/Program Files (x86)/pipx_bin:/c/hostedtoolcache/windows/go/1.15.15/x64/bin:/c/hostedtoolcache/windows/Python/3.7.9/x64/Scripts:/c/hostedtoolcache/windows/Python/3.7.9/x64:/c/hostedtoolcache/windows/Ruby/2.5.9/x64/bin:/c/tools/kotlinc/bin:/c/hostedtoolcache/windows/Java_Temurin-Hotspot_jdk/8.0.312-7/x64/bin:/c/npm/prefix:/c/Program Files (x86)/Microsoft SDKs/Azure/CLI2/wbin:/c/ProgramData/kind:/c/Program Files/Eclipse Foundation/jdk-8.0.302.8-hotspot/bin:/c/Windows/system32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0:/c/Windows/System32/OpenSSH:/c/ProgramData/Chocolatey/bin:/c/Program Files/Docker:/c/Program Files/PowerShell/7:/c/Program Files/Microsoft/Web Platform Installer:/c/Program Files/dotnet:/c/Program Files/Microsoft SQL Server/130/Tools/Binn:/c/Program Files/Microsoft SQL Server/Client SDK/ODBC/170/Tools/Binn:/c/Program Files (x86)/Windows Kits/10/Windows Performance Toolkit:/c/Program Files (x86)/Microsoft SQL Server/110/DTS/Binn:/c/Program Files (x86)/Microsoft SQL Server/120/DTS/Binn:/c/Program Files (x86)/Microsoft SQL Server/130/DTS/Binn:/c/Program Files (x86)/Microsoft SQL Server/140/DTS/Binn:/c/Program Files (x86)/Microsoft SQL Server/150/DTS/Binn:/c/Program Files/nodejs:/c/Program Files/OpenSSL/bin:/c/Strawberry/c/bin:/c/Strawberry/perl/site/bin:/c/Strawberry/perl/bin:/c/ProgramData/chocolatey/lib/pulumi/tools/Pulumi/bin:/c/Program Files/TortoiseSVN/bin:/c/Program Files/CMake/bin:/c/ProgramData/chocolatey/lib/maven/apache-maven-3.8.4/bin:/c/Program Files/Microsoft Service Fabric/bin/Fabric/Fabric.Code:/c/Program Files/Microsoft SDKs/Service Fabric/Tools/ServiceFabricLocalClusterManager:/cmd:/mingw64/bin:/usr/bin:/c/tools/php:/c/Program Files (x86)/sbt/bin:/c/SeleniumWebDrivers/ChromeDriver:/c/SeleniumWebDrivers/EdgeDriver:/c/Program Files/Amazon/AWSCLIV2:/c/Program Files/Amazon/SessionManagerPlugin/bin:/c/Program Files/Amazon/AWSSAMCLI/bin:/c/Program Files (x86)/Google/Cloud SDK/google-cloud-sdk/bin:/c/Program Files (x86)/Microsoft BizTalk Server:/c/Program Files/LLVM/bin:/c/Users/VssAdministrator/.dotnet/tools:/c/Users/VssAdministrator/.cargo/bin:/c/Users/VssAdministrator/AppData/Local/Microsoft/WindowsApps)
Will build using: "C:\hostedtoolcache\windows\Python\3.9.9\x86\python.exe"
3.9.9 (tags/v3.9.9:ccb0e6a, Nov 15 2021, 17:50:04) [MSC v.1929 32 bit (Intel)]
Python's architecture is 32bit
cfg.VERSION: 4.1.2a1
Running command: build_wx
Machine architecture: x64
Build architecture: x86
== Python =====================================================
version: 3.9.9.final.0
architecture: x86
library: Python39.lib
libs: ['C:\\hostedtoolcache\\windows\\Python\\3.9.9\\x86\\libs']
includes: ['C:\\hostedtoolcache\\windows\\Python\\3.9.9\\x86\\include', 'C:\\hostedtoolcache\\windows\\Python\\3.9.9\\x86\\include\\cpython', 'C:\\hostedtoolcache\\windows\\Python\\3.9.9\\x86\\include\\internal']
== Visual Studio Enterprise 2019==============================================
description: Scalable, end-to-end solution for teams of any size
version: 16.11.32106.194
version (friendly): 2019
display version: 16.11.9
path: C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise
executable: C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\devenv.exe
is complete: True
is prerelease: False
is launchable: True
state: local, registered, no reboot required
== Visual C ===================================================
version: 14.29.30139
path: C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC
-- Tools ---------------------------------------------------
version: 14.29.30133
path: C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133
redist path: C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Redist\MSVC\14.29.30133\
-- F# ------------------------------------------------------
path: C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\CommonExtensions\Microsoft\FSharp\Tools
-- DLL -----------------------------------------------------
version: v142-14.29.30139.0
path: C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Redist\MSVC\14.29.30133\x86\Microsoft.VC142.CRT
== MSBuild ====================================================
version: 16.11.2.50704
path: C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Current\Bin\MSBuild.exe
== HTML Help ==================================================
path: C:\Program Files (x86)\HTML Help Workshop
== ATLMFC =====================================================
path: C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\ATLMFC
include path: C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\ATLMFC\include
lib path: C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\ATLMFC\lib\x86
== Windows SDK ================================================
version: 10.0
sdk version: 10.0.22000.0
path: C:\Program Files (x86)\Windows Kits\10\
== Universal CRT ==============================================
version: 10.0.22000.0
path: C:\Program Files (x86)\Windows Kits\10\
lib directory: C:\Program Files (x86)\Windows Kits\10\Lib\10.0.22000.0\ucrt\x86\
headers directory: C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\ucrt\
== Extension SDK ==============================================
path: C:\Program Files (x86)\Microsoft SDKs\Windows Kits\10\ExtensionSDKs
== TypeScript =================================================
path: C:\Program Files (x86)\Microsoft SDKs\TypeScript\4.3
== .NET =======================================================
version: v4.7.3062.2
-- x86 -----------------------------------------------------
version: v4.7.3062.2
path: C:\Windows\Microsoft.NET\Framework
-- x64 -----------------------------------------------------
version: v4.7.3062.2
path:
-- NETFX ---------------------------------------------------
path: None
x86 exe path: None
x64 exe path: None
Downloading microsoft.web.webview2 1.0.622.22...
Connection successful...
Data downloaded...
wxWidgets build options: ['--wxpython', '--unicode', '--jobs=4']
Updating wx/msw/setup.h
setting build options...
nmake.exe -f makefile.vc UNICODE=1 OFFICIAL_BUILD=1 COMPILER_VERSION=140 SHARED=1 MONOLITHIC=0 USE_OPENGL=1 USE_GDIPLUS=1 BUILD=release
Microsoft (R) Program Maintenance Utility Version 14.29.30139.0
Copyright (C) Microsoft Corporation. All rights reserved.
if not exist vc140_mswudll mkdir vc140_mswudll
if not exist ..\..\lib\vc140_dll mkdir ..\..\lib\vc140_dll
if not exist ..\..\lib\vc140_dll\mswu mkdir ..\..\lib\vc140_dll\mswu
if not exist ..\..\lib\vc140_dll\mswu\wx mkdir ..\..\lib\vc140_dll\mswu\wx
if not exist ..\..\lib\vc140_dll\mswu\wx\setup.h copy ..\..\include\wx\msw\setup.h ..\..\lib\vc140_dll\mswu\wx\setup.h
1 file(s) copied.Pipeline failed ❌##[section]Starting: build wxWidgets
==============================================================================
Task : Bash
Description : Run a Bash script on macOS, Linux, or Windows
Version : 3.195.0
Author : Microsoft Corporation
Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/bash
==============================================================================
Generating script.
[command]"C:\Program Files\Git\bin\bash.exe" -c pwd
/d/a/_temp
========================== Starting Command Output ===========================
[command]"C:\Program Files\Git\bin\bash.exe" /d/a/_temp/480453ed-f1f1-424b-b8de-96364d8549c5.sh
which: no cl.exe in (/mingw64/bin:/usr/bin:/c/Users/VssAdministrator/bin:/c/Users/VssAdministrator/AppData/Roaming/Python/Python39/Scripts:/c/hostedtoolcache/windows/Python/3.9.9/x86/Scripts:/c/hostedtoolcache/windows/Python/3.9.9/x86:/c/agents/2.196.2/externals/git/cmd:/c/agents/2.196.2/externals/git/mingw64/bin:/c/Program Files/MongoDB/Server/5.0/bin:/c/aliyun-cli:/c/vcpkg:/c/cf-cli:/c/Program Files (x86)/NSIS:/c/tools/zstd:/c/Program Files/Mercurial:/c/hostedtoolcache/windows/stack/2.7.3/x64:/c/cabal/bin:/c/ghcup/bin:/c/tools/ghc-9.2.1/bin:/c/Program Files/dotnet:/c/mysql/bin:/c/Program Files/R/R-4.1.2/bin/x64:/c/SeleniumWebDrivers/GeckoDriver:/c/Program Files (x86)/sbt/bin:/c/Program Files (x86)/GitHub CLI:/bin:/c/Program Files (x86)/pipx_bin:/c/hostedtoolcache/windows/go/1.15.15/x64/bin:/c/hostedtoolcache/windows/Python/3.7.9/x64/Scripts:/c/hostedtoolcache/windows/Python/3.7.9/x64:/c/hostedtoolcache/windows/Ruby/2.5.9/x64/bin:/c/tools/kotlinc/bin:/c/hostedtoolcache/windows/Java_Temurin-Hotspot_jdk/8.0.312-7/x64/bin:/c/npm/prefix:/c/Program Files (x86)/Microsoft SDKs/Azure/CLI2/wbin:/c/ProgramData/kind:/c/Program Files/Eclipse Foundation/jdk-8.0.302.8-hotspot/bin:/c/Windows/system32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0:/c/Windows/System32/OpenSSH:/c/ProgramData/Chocolatey/bin:/c/Program Files/Docker:/c/Program Files/PowerShell/7:/c/Program Files/Microsoft/Web Platform Installer:/c/Program Files/dotnet:/c/Program Files/Microsoft SQL Server/130/Tools/Binn:/c/Program Files/Microsoft SQL Server/Client SDK/ODBC/170/Tools/Binn:/c/Program Files (x86)/Windows Kits/10/Windows Performance Toolkit:/c/Program Files (x86)/Microsoft SQL Server/110/DTS/Binn:/c/Program Files (x86)/Microsoft SQL Server/120/DTS/Binn:/c/Program Files (x86)/Microsoft SQL Server/130/DTS/Binn:/c/Program Files (x86)/Microsoft SQL Server/140/DTS/Binn:/c/Program Files (x86)/Microsoft SQL Server/150/DTS/Binn:/c/Program Files/nodejs:/c/Program Files/OpenSSL/bin:/c/Strawberry/c/bin:/c/Strawberry/perl/site/bin:/c/Strawberry/perl/bin:/c/ProgramData/chocolatey/lib/pulumi/tools/Pulumi/bin:/c/Program Files/TortoiseSVN/bin:/c/Program Files/CMake/bin:/c/ProgramData/chocolatey/lib/maven/apache-maven-3.8.4/bin:/c/Program Files/Microsoft Service Fabric/bin/Fabric/Fabric.Code:/c/Program Files/Microsoft SDKs/Service Fabric/Tools/ServiceFabricLocalClusterManager:/cmd:/mingw64/bin:/usr/bin:/c/tools/php:/c/Program Files (x86)/sbt/bin:/c/SeleniumWebDrivers/ChromeDriver:/c/SeleniumWebDrivers/EdgeDriver:/c/Program Files/Amazon/AWSCLIV2:/c/Program Files/Amazon/SessionManagerPlugin/bin:/c/Program Files/Amazon/AWSSAMCLI/bin:/c/Program Files (x86)/Google/Cloud SDK/google-cloud-sdk/bin:/c/Program Files (x86)/Microsoft BizTalk Server:/c/Program Files/LLVM/bin:/c/Users/VssAdministrator/.dotnet/tools:/c/Users/VssAdministrator/.cargo/bin:/c/Users/VssAdministrator/AppData/Local/Microsoft/WindowsApps)
Will build using: "C:\hostedtoolcache\windows\Python\3.9.9\x86\python.exe"
3.9.9 (tags/v3.9.9:ccb0e6a, Nov 15 2021, 17:50:04) [MSC v.1929 32 bit (Intel)]
Python's architecture is 32bit
cfg.VERSION: 4.1.2a1
Running command: build_wx
Machine architecture: x64
Build architecture: x86
== Python =====================================================
version: 3.9.9.final.0
architecture: x86
library: Python39.lib
libs: ['C:\\hostedtoolcache\\windows\\Python\\3.9.9\\x86\\libs']
includes: ['C:\\hostedtoolcache\\windows\\Python\\3.9.9\\x86\\include', 'C:\\hostedtoolcache\\windows\\Python\\3.9.9\\x86\\include\\cpython', 'C:\\hostedtoolcache\\windows\\Python\\3.9.9\\x86\\include\\internal']
== Visual Studio Enterprise 2019 ==============================
description: Scalable, end-to-end solution for teams of any size
install date: Sun Jan 16 22:12:32 2022
version: 16.11.32106.194
version (friendly): 2019
display version: 16.11.9
path: C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise
executable: C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\devenv.exe
is complete: True
is prerelease: False
is launchable: True
state: local, registered, no reboot required
== Visual C ===================================================
version: 14.29.30139
path: C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC
has cmake: True
has ninja: True
-- Tools ---------------------------------------------------
version: 14.29.30133
path: C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133
redist path: C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Redist\MSVC\14.29.30133\
-- F# ------------------------------------------------------
path: C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\CommonExtensions\Microsoft\FSharp\Tools
-- DLL -----------------------------------------------------
version: v142-14.29.30139.0
path: C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Redist\MSVC\14.29.30133\x86\Microsoft.VC142.CRT
== MSBuild ====================================================
version: 16.11.2.50704
path: C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Current\Bin\MSBuild.exe
== HTML Help ==================================================
path: C:\Program Files (x86)\HTML Help Workshop
== ATLMFC =====================================================
path: C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\ATLMFC
include path: C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\ATLMFC\include
lib path: C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\ATLMFC\lib\x86
== Windows SDK ================================================
version: 10.0
sdk version: 10.0.22000.0
path: C:\Program Files (x86)\Windows Kits\10\
== Universal CRT ==============================================
version: 10.0.22000.0
path: C:\Program Files (x86)\Windows Kits\10\
lib directory: C:\Program Files (x86)\Windows Kits\10\Lib\10.0.22000.0\ucrt\x86\
headers directory: C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\ucrt\
== Extension SDK ==============================================
path: C:\Program Files (x86)\Microsoft SDKs\Windows Kits\10\ExtensionSDKs
== TypeScript =================================================
path: C:\Program Files (x86)\Microsoft SDKs\TypeScript\4.3
== .NET =======================================================
version: v4.7.3062.2
-- x86 -----------------------------------------------------
version: v4.7.3062.2
path: C:\Windows\Microsoft.NET\Framework
-- x64 -----------------------------------------------------
version: v4.7.3062.2
path:
-- NETFX ---------------------------------------------------
path: None
x86 exe path: None
x64 exe path: None
Downloading microsoft.web.webview2 1.0.622.22...
Connection successful...
Data downloaded...
wxWidgets build options: ['--wxpython', '--unicode', '--jobs=4']
Updating wx/msw/setup.h
setting build options...
Command 'cl.exe' failed with exit code 1.
Finished command: build_wx (0m2.523s)
##[error]Bash exited with code '1'.
##[section]Finishing: build wxWidgets |
|
The build environments are identical so I am not sure why it didn't compile properly. I will set the distutils logging to capture the full output to see what is going on. |
|
found the problem and fixed it. |
I am not 100% if this is going to work properly. It does compile correctly and it is substantially faster. for the wxWidgets compilation I use the msvc build system I added to check if cmake and ninja are available and if they are I use that if JOM is not being used. the --jobs command line switch sets the number of parallel files that can be compiled. Windows builds on Azure typically take around 45 minutes, we will see how much that gets cut down. It took 1 minutes 30 seconds to compile wxWidgets on my 6 core using 6 jobs. I don't remember off hand how long it took previously but I sure know it didn't take a minute and 30 seconds!!
|
Heh didn't think about that being an issue. Gonna have to make sure the old path statements are at the end of the line. The last commit hopefully will hopefully trim down the time it takes to compile on Windows without using JOM. |
|
It took about 15 minutes to compile on my antique AMD FX 6 core @ 3.2ghz. |
buildtools/build_wxwidgets.py
Outdated
| if sys.platform.startswith("win"): | ||
| if nmakeCommand.startswith('cmake'): |
There was a problem hiding this comment.
can do:
if sys.platform.startswith("win") and nmakeCommand.startswith('cmake'):There was a problem hiding this comment.
No I cannot.
if sys.platform.startswith("win"):
if nmakeCommand.startswith('cmake'):
args.insert(0, '-GNinja')
else:
args.append("--jobs=" + options.jobs)because if cmake or ninja is not available then cmake.exe doesn't get set to the nmake command so if the platform is windows we do not want to ad a --jobs to the nmake.exe command bcause it doesn't support this switch.
…end of the new path
|
OK path is now fixed. I had to do a little tinkering with it because of the subprocess calls that the build system uses and it running the msvc script multiple times. I didn't want to end up with multiple entries for the same thing like the path. |
|
How glad I am that you can use a new and better PC. As soon as you have an update you notify me to go testing. I wish you a better experience and happy hacking code! |
|
Here you can see the HTML report output.html with Comparation build system nmake vs (cmake and ninja)
$ diffoscope --html output.html --exclude '__pycache__' venv/Lib/site-packages/wx/ ./wx/ |
|
Comparation build system nmake vs (cmake and ninja)
$ diffoscope --html output.html --exclude '__pycache__' ./wx/ ./wx-cmake-ninja/
|
|
@kdschlosser, thanks for this PR. I've been fighting a little with trying to rely more on setuptools and kept running into issues. Then I remembered that you had already worked on this and it looks like exactly what we need. I'll be working on understanding it a bit better and maybe experimenting or tinkering a bit, and then I'll try to get it committed in the next day or two. |
My concern with this is that it seems like an awful lot of code (6000 lines!) that would now have to be maintained in wxPython. Aren't there other Python projects that run into trouble locating MSVC? Seems like this would be a common problem that others would have run into. |
|
Yes, that's a good point. There's some refactoring that could be done to reduce and simplify somewhat. Also I wonder if I could almost get what I need from setuptools, but I keep running into this bug no matter how I install newer versions of MSVC: pypa/setuptools#1902. It's been around for a few years and the fix seems simple so I'm not sure why they haven't done it already. If I could get my build modifications working reliably with setuptools that would be my preference. Maybe I should just monkey-patch it... |
|
You can vswhere.exe that being said you are calling an executable using subprocess which can have issues in and of itself like an orphaned process if the program crashes for some reason. The other things is vswhere doesn't expose the entire COM API where as the vswhere script I wrote does. vswhere.exe uses the exact same COM API to collect information that the vswhere script does. I have also organized the information collected so it can be accessed in a more user friendly manner. There is definitely cleanup that can be done. The MSVC script I wrote works and it works for every version of the MSVC compiler that has been used to compile Python since version 2.6. locates the C compiler for all versions in between as well. All windows SDKs that are supported by those compilers is also located. The MSVC script has a built in backup so if for some reason the COM API doesn't work or is not available because of permissions or any other reason the script will still be able to locate what is needed. It is a slower then using the COM API but it will still work. |
|
I have opened up an issue on setup tools about updating the compiler detection on Windows using the same mechanisms I am. It has fallen on deaf ears like any other issue with Python running on Windows does. The developers only seem to care about it working 100% on a Posix system. I don't understand why the greater market share has the least amount of importance, |
|
Don't commit this as it is far from finished an will introduce other problems I am sure. The last thing I was messing with was using Ninja to speed up the time it takes to compile on Windows. |
|
Phoenix has an extremely complex build system and I am pretty sure that if you and I put our heads together a far simpler build system can be written. |
I'll just drop this breadcrumb here. I'm a KiCad core developer and we use wxPython, I'm not a fan but I needed true compatibility with our Windows CI tooling and build pipeline and python tooling is grossly incompatible for that. The entire build chain can be simplified to a combination of python for some of the preprocessing like SIP and then the build work under CMake that can parallelize builds and be done in seconds. This is the "preprocess" portion https://gitlab.com/kicad/packaging/kicad-vcpkg-registry/-/blob/main/ports/wxpython/portfile.cmake And here is the CMakeList for the pyd compilation and packaging https://gitlab.com/kicad/packaging/kicad-vcpkg-registry/-/blob/main/ports/wxpython/CMakeLists.txt As you notice, it's actually a vcpkg port. However, upstream microsoft vcpkg isn't meant for python plugins so it rests in our kicad registry instead |
|
I use KiCad for my PCB designs. I thought you guys use QT for the GUI framework. You can use Python to compile your entire program. I never check to see if KiCAD is cross platform or not. If it is it can still be compiled using python. I have a framework that I wrote that does a really good job of locating compilers on different operating systems. I did know KiCAD has Python scripting support but never investigated how much KiCAD relied on Python. I am really curious, probably because I use KiCAD. If you want to have a more in depth conversation about build systems or if you want to know anything about this MSVC script I wrote let me know. Tell me where to go to have have the conversation and I would be more then happy to. |
|
Using the /MP cl.exe switch does not do what most people think it does. It does not run CL.exe in parallel technically speaking. It is going to compile solutions in parallel. This is bad in most case because the memory consumption ends up being enormous and there can be dependency problems. What If I told you I could compile a single solution using say a threadripper 16 core process and all 32 threads (logical CPUs) would be used when compiling that single solution.. Far less RAM gets used and it is faster then compiling solutions in parallel. It also solves the possible issue of a build failing due to a solution getting compiled that is dependent on another solution that has not finished compiling. Only one solution at a time would be compiled any dependencies become a non issue. This would work cross platform as well and would not be just for Windows. |
Nope, we can't use QT for legacy reasons, mainly so many hard dependencies. Long long term, maybe years of manhours we could move to QT but at the same time, we are currently happy being able to maintain wxWidgets. QT in comparison would be a nightmare and in my personal experience in the last 10 years, I rather headdesk myself than try and upstream another 1 line bugfix to qt. python's build support is funky. As is, if MSVC updates in advance of hardcoded expectations in python, it will basically refuse to compile. This is why we use CMake, all the problems are handled, and with But that's not the topic and reason I commented here.
I think you misunderstand entirely what I was commenting on. I brought up the fact that wxpython can be thought as two steps, "preprocessing" which needs python and the actual "build" which does not. With that logic, the build chain can be simplified quite a bit with the usage of CMake. Now on the /MP topic, when CMake does parallelization for MSVC, it does not use /MP. It uses Ninja. Even Microsoft officially uses Ninja under CMake by default now. /MP and MSVC msbuild/slns/cprojs are ancient history ;) |
|
I agree with you as far as the CMAKE and Ninja goes. unfortunately wxWidgets is set up to compile using NMAKE and I don't believe the CMAKE files have been kept up to date. I have tried to get it to compile properly using CMAKE and Ninja and have not had any success in doing so. |
wx 3.1.4 and up had cmake support added and is actively maintained. |
|
NMAKE is what gets used when compiling wxWidgets in wxPython. If I change to using CMAKE the compile fails. |
|
This pull request has been mentioned on Discuss wxPython. There might be relevant details there: https://discuss.wxpython.org/t/building-phoenix-on-windows/36435/4 |
|
I wanted to make a note of something. This "bug" in Setuptools will NEVER be fixed because in their eyes it is not a bug. You have to read the "fine print" with setuptools. They documentation just plain stinks so one has to resort to looking at the source code for it. This is what I actually did recently and stumbled upon this. read that carefully. It states that setuptools DOES NOT support Visual Studio or Visual C++. It only supports Build Tools.. Build tools is a different animal than Visual Studio and because setuptools relies on hard coded paths for both the registry and filesystem it is not going to be able to locate Visual Studio properly, especially the new versions. Using vswhere that comes with Visual Studio is not a viable solution in my opinion because once again the reliance of hard coded paths in order to locate where it is installed to be able to run it. If you are able to locate vswhere then you have already found the installation of Visual Studio. That in and of itself makes it pointless to use., If the decision was made to only allow using VS 2017 or newer then quite a bit of the code is able to be removed from the script I wrote. I did also want to note that I added the code to pypi as pyMSVC. It can be added to the build system using the toml file and it will be downloaded when a windows build is being done. This means the build system that is currently in place needs to have a makeover so it gets run from setup.py directly and not through a subprocess. Me personally I will avoid using subprocess as much as possible. The output may change from the process being run and or errors are not handled properly inside of the subprocess which can cause issues. With windows systems you have orphaned process issues on top of that. Using subprocess has caused me some headache in the past. For whatever reason the program freezes or crashes the subprocess could end up getting detached from the main process. This has caused hung network ports that can only be released by rebooting the computer. It is also no mystery that the majority of the developers for Python and the build systems for python are mostly Linux folks. They do not put importance to getting the Windows side of things up to snuff on the top priority list. There are bugs in the socket module that have been there for over a decade. So they don't exactly rush to get Windows based issues fixed, a fantastic example is the locale module that still doesn't work properly. My script still works with zero bug reports in the last year. It is being used by people that compile Cython code, There is a link to it in their documentation. So it's not a question of reliability or if the code is sound or not. The question of maintainability pretty much went out the window as well. a full year of being on pypi and not one thing has needed to be done to it in terms of fixing a bug or updating it to work with new versions of VS. I am going to update it to add python 3.11 to the automatic compiler detection. The script does still work with Python 3.11 it just has to have the MSVC compiler versions passed to the setup function. |
|
I did also want to mention the importance of using the exact same MSVC compiler version that was used to compile CPython to compile extension modules. This needs to be done because of changes to the C runtime code not being 100% compatible between MSVC versions. 3.6 = 14.0 - ? What is important to keep the same is the c runtime version. https://en.wikipedia.org/wiki/Microsoft_Visual_C%2B%2B when this code gets run import platform
print(platform.python_compiler())the output is along the lines of it tells you the MSVC version used to compile. Here is is 1928 which uses the the 14.28 c runtime. That means to make sure the compilation works properly for an extension that these VS versions need to be used 16.8.1, 16.8.2 and 16.9.2 which is VS2019. But not all VS2019 versions use MSVC 1928. the funny thing is this is how not up to date information is for Python. Right in the python documentation it states this..
https://devguide.python.org/getting-started/setup-building/#windows-compiling It is pulling information form the waybackmachine so that tells you how up to date it is. But they specifically mention Victor Stinner. It just so happens he has some information that is more up to date. https://pythondev.readthedocs.io/windows.html#python-and-visual-studio-version-matrix and even that is really far off. Look at what it says for Python 3.8.. VS 2017. Yet here we have 3.8.10 that has been compiled using VS2019. While Python is able to be compiled using VS2017 or newer an extension module should be compiled using the exact same compiler that Python was compiled with. This is the only way of ensuring 100% compatibility between the extension module and python. setuptools is never going to be set up to do this properly. Because the authors feel that they have provided enough support for compiling with only supporting build tools versions 2015, 2017 AND 2019 even tho the binary release of Python is being compiled using VS 2022. |
This hasn't been true since MSVC 2015. https://devblogs.microsoft.com/cppblog/the-great-c-runtime-crt-refactoring/ The 2015+ now use the UCRT (universal c runtime). Basically in the past vcruntime1XX was a thing. Ever since 2015, vcruntime140 is now the only one and the name nor version ever changes. Any "old" apps built in MSVC 2015 will happily use a newer 2023 updated version of the same vcruntime140. |
|
https://learn.microsoft.com/en-us/cpp/porting/binary-compat-2015-2017?view=msvc-170
If Python is compiled using Visual Studio 2022 (which it is) you CANNOT compile the extension modules using VS 2019 you MUST use VS 2022 or newer. If Python was compiled using VS 2017 then you CAN use VS 2022 to compile the extension modules. Backwards compatibility exists not forward. So the rule of thumb is the same or newer version than what Python has been compiled with. Python binaries are compiled using version 14.3 of the c runtime. The terms build tools and tool sets are used interchangeably and is referring to the c runtime version. https://wiki.python.org/moin/WindowsCompilers
14.3 is the first release of VS 2022 so only VS2022 should be used to compile extension modules for Python 3.11. IDK about any of the other versions of Python as I have not checked to see if they are being compiled using a different version of MSVC |
using setuptools or distutils to set up an MSVC build environment has been flaky at best. It doesn't always work correctly because of Microsoft's inability to stick to a single kind of an installation ecosystem. Starting with Visual Studio 2017 Microsoft added a series of COM interfaces to Visual Studio that allows for programmatically enumerating all Visual Studio installations. This coupled with registry keys and known locations on the system drive a proper build environment is able to be set up without using any of the vcvars*.bat files. Those bat files do not always work properly and they should not be used to set up the build environment. This is what setuptools and distutils use to create the build environment and that is the reason why it does not work 100% of the time.
wxPython is now able to be built on a Windows 7 computer that is running Visual Studio 2019+ where as it was limited to Windows 10+ computers before. This was due to it using bash.exe which is apart of the WSL (Windows Subsystem for Linux) which is only available on Windows 10+. There is another way... Seeing as how most users that would want to build wxPython on a Windows computer are probably going to have Git installed why not use the bash.exe that is included with that??? So I added in detecting if Git is installed and collecting the installation path from the registry and use that bash.exe if WSL is not available.