Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Packaging] Bump MSI embedded Python to 3.10.3 #21746

Merged
merged 2 commits into from
Mar 24, 2022
Merged

[Packaging] Bump MSI embedded Python to 3.10.3 #21746

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ae37f8a
msi python
jiasli Mar 22, 2022
f29bdbd
arguments
jiasli Mar 23, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 53 additions & 24 deletions build_scripts/windows/scripts/build.cmd
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
@echo off
SetLocal EnableDelayedExpansion

REM Double colon :: should not be used in parentheses blocks, so we use REM.
REM See https://stackoverflow.com/a/12407934/2199657
Comment on lines +4 to +5
Copy link
Member Author

@jiasli jiasli Mar 22, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using double colons in a block results in failure:

test.cmd:

@echo off
SetLocal EnableDelayedExpansion

for /f %%f in ('dir /b /s *.pyc') do (
     ::echo test
     ::echo test
)
>test.cmd
The system cannot find the drive specified.


echo build a msi installer using local cli sources and python executables. You need to have curl.exe, unzip.exe and msbuild.exe available under PATH
echo.

Expand All @@ -9,13 +13,16 @@ if "%CLI_VERSION%"=="" (
echo Please set the CLI_VERSION environment variable, e.g. 2.0.13
goto ERROR
)
set PYTHON_VERSION=3.8.9
set PYTHON_VERSION=3.10.3

set WIX_DOWNLOAD_URL="https://azurecliprod.blob.core.windows.net/msi/wix310-binaries-mirror.zip"
set PYTHON_DOWNLOAD_URL="https://azurecliprod.blob.core.windows.net/util/Python389-32.zip"
set PYTHON_DOWNLOAD_URL="https://www.python.org/ftp/python/3.10.3/python-3.10.3-embed-win32.zip"
set PROPAGATE_ENV_CHANGE_DOWNLOAD_URL="https://azurecliprod.blob.core.windows.net/util/propagate_env_change.zip"

:: Set up the output directory and temp. directories
REM https://pip.pypa.io/en/stable/installation/#get-pip-py
set GET_PIP_DOWNLOAD_URL="https://bootstrap.pypa.io/get-pip.py"

REM Set up the output directory and temp. directories
echo Cleaning previous build artifacts...
set OUTPUT_DIR=%~dp0..\out
if exist %OUTPUT_DIR% rmdir /s /q %OUTPUT_DIR%
Expand All @@ -26,14 +33,14 @@ mkdir %ARTIFACTS_DIR%
set TEMP_SCRATCH_FOLDER=%ARTIFACTS_DIR%\cli_scratch
set BUILDING_DIR=%ARTIFACTS_DIR%\cli
set WIX_DIR=%ARTIFACTS_DIR%\wix
set PYTHON_DIR=%ARTIFACTS_DIR%\Python389-32
set PYTHON_DIR=%ARTIFACTS_DIR%\Python
set PROPAGATE_ENV_CHANGE_DIR=%~dp0..\propagate_env_change

set REPO_ROOT=%~dp0..\..\..

::reset working folders
REM reset working folders
if exist %BUILDING_DIR% rmdir /s /q %BUILDING_DIR%
::rmdir always returns 0, so check folder's existence
REM rmdir always returns 0, so check folder's existence
if exist %BUILDING_DIR% (
echo Failed to delete %BUILDING_DIR%.
goto ERROR
Expand All @@ -51,35 +58,50 @@ if exist %REPO_ROOT%\privates (
copy %REPO_ROOT%\privates\*.whl %TEMP_SCRATCH_FOLDER%
)

::ensure wix is available
REM ensure wix is available
if exist %WIX_DIR% (
echo Using existing Wix at %WIX_DIR%
)
if not exist %WIX_DIR% (
mkdir %WIX_DIR%
pushd %WIX_DIR%
echo Downloading Wix.
curl -o wix-archive.zip %WIX_DOWNLOAD_URL% -k
unzip -q wix-archive.zip
curl --output wix-archive.zip %WIX_DOWNLOAD_URL%
unzip wix-archive.zip
if %errorlevel% neq 0 goto ERROR
del wix-archive.zip
echo Wix downloaded and extracted successfully.
popd
)

::ensure Python is available
REM ensure Python is available
if exist %PYTHON_DIR% (
echo Using existing Python at %PYTHON_DIR%
)
if not exist %PYTHON_DIR% (
echo Setting up Python and pip
mkdir %PYTHON_DIR%
pushd %PYTHON_DIR%
echo Downloading Python.
curl -o Python389-32.zip %PYTHON_DOWNLOAD_URL% -k
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-k, --insecure was introduced by #4523 allowing non-secure HTTPS, which we don't know why either.

unzip -q Python389-32.zip

echo Downloading Python
curl --output python-archive.zip %PYTHON_DOWNLOAD_URL%
unzip python-archive.zip
if %errorlevel% neq 0 goto ERROR
del Python389-32.zip
echo Python downloaded and extracted successfully.
del python-archive.zip
echo Python downloaded and extracted successfully

REM Delete _pth file so that Lib\site-packages is included in sys.path
REM https://github.com/pypa/pip/issues/4207#issuecomment-297396913
REM https://docs.python.org/3.10/using/windows.html#finding-modules
del python*._pth
Comment on lines +93 to +96
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding python*._pth, see:


echo Installing pip
curl --output get-pip.py %GET_PIP_DOWNLOAD_URL%
%PYTHON_DIR%\python.exe get-pip.py
del get-pip.py
echo Pip set up successful

dir .
popd
)
set PYTHON_EXE=%PYTHON_DIR%\python.exe
Expand All @@ -93,7 +115,7 @@ for %%a in (%CLI_SRC%\azure-cli %CLI_SRC%\azure-cli-core %CLI_SRC%\azure-cli-tel
%BUILDING_DIR%\python.exe -m pip install --no-warn-script-location --no-cache-dir --no-deps .
popd
)
%BUILDING_DIR%\python.exe -m pip install -r %CLI_SRC%\azure-cli\requirements.py3.windows.txt
%BUILDING_DIR%\python.exe -m pip install --no-warn-script-location --requirement %CLI_SRC%\azure-cli\requirements.py3.windows.txt

if %errorlevel% neq 0 goto ERROR

Expand All @@ -111,37 +133,44 @@ copy %REPO_ROOT%\build_scripts\windows\resources\CLI_LICENSE.rtf %BUILDING_DIR%
copy %REPO_ROOT%\build_scripts\windows\resources\ThirdPartyNotices.txt %BUILDING_DIR%
copy %REPO_ROOT%\NOTICE.txt %BUILDING_DIR%

:: Remove .py and only deploy .pyc files
REM Remove .py and only deploy .pyc files
pushd %BUILDING_DIR%\Lib\site-packages
for /f %%f in ('dir /b /s *.pyc') do (
set PARENT_DIR=%%~df%%~pf..
echo !PARENT_DIR! | findstr /C:\Lib\site-packages\pip\ 1>nul
if !errorlevel! neq 0 (
REM Only take the file name without 'pyc' extension: e.g., (same below) __init__.cpython-310
set FILENAME=%%~nf
set BASE_FILENAME=!FILENAME:~0,-11!
REM Truncate the '.cpython-310' postfix which is 12 chars long: __init__
REM https://stackoverflow.com/a/636391/2199657
set BASE_FILENAME=!FILENAME:~0,-12!
REM __init__.pyc
set pyc=!BASE_FILENAME!.pyc
REM Delete ..\__init__.py
del !PARENT_DIR!\!BASE_FILENAME!.py
REM Copy to ..\__init__.pyc
copy %%~f !PARENT_DIR!\!pyc! >nul
REM Delete __init__.pyc
del %%~f
) ELSE (
echo --SKIP !PARENT_DIR! under pip
)
)
popd

:: Remove __pycache__
REM Remove __pycache__
echo remove pycache
for /d /r %BUILDING_DIR%\Lib\site-packages\pip %%d in (__pycache__) do (
if exist %%d rmdir /s /q "%%d"
)

:: Remove aio
REM Remove aio
echo remove aio
for /d /r %BUILDING_DIR%\Lib\site-packages\azure\mgmt %%d in (aio) do (
if exist %%d rmdir /s /q "%%d"
)

:: Remove dist-info
REM Remove dist-info
echo remove dist-info
pushd %BUILDING_DIR%\Lib\site-packages
for /d %%d in ("azure*.dist-info") do (
Expand All @@ -151,14 +180,14 @@ popd

if %errorlevel% neq 0 goto ERROR

::ensure propagate_env_change.exe is available
REM ensure propagate_env_change.exe is available
if exist "%PROPAGATE_ENV_CHANGE_DIR%\propagate_env_change.exe" (
echo Using existing propagate_env_change.exe at %PROPAGATE_ENV_CHANGE_DIR%
) else (
pushd %PROPAGATE_ENV_CHANGE_DIR%
echo Downloading propagate_env_change.exe.
curl -o propagate_env_change.zip %PROPAGATE_ENV_CHANGE_DOWNLOAD_URL% -k
unzip -q propagate_env_change.zip
curl --output propagate_env_change.zip %PROPAGATE_ENV_CHANGE_DOWNLOAD_URL%
unzip propagate_env_change.zip
if %errorlevel% neq 0 goto ERROR
del propagate_env_change.zip
echo propagate_env_change.exe downloaded and extracted successfully.
Expand Down