Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Docker Image on Windows Base Image #439 #2332

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Docker Image on Windows Base Image #439 #2332

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
3e8584d
Dockerfile for Windows container
jwdb Jan 3, 2024
c79f97f
Add build scripts to package.json
jwdb Jan 4, 2024
f1488ab
Add build step for windows docker container
jwdb Jan 4, 2024
51de33f
Rename jobname because the name has to be a letter, number or underscore
jwdb Jan 4, 2024
729f43e
changing the Node-Windows stage name to nodewindows.
jwdb Jan 4, 2024
0ac3aff
Add cmd to run command to validate docker image in line with the wind…
jwdb Jan 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions ChangeLog.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@

## Upcoming Release

General:

- Added support for docker image based on Windows Base Image.

Blob:

- Fixed issue of setting blob tag should not update Blob Etag and LastModified. (issue #2327)
Expand Down
63 changes: 63 additions & 0 deletions Dockerfile.Windows
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
#
# Base-Node
#
FROM mcr.microsoft.com/windows/nanoserver:ltsc2022-amd64 AS nodewindows
# Install dependencies first
RUN mkdir c:\node
WORKDIR c:\\node
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Try to build docker image on my win10 machine failed:

D:\code\azurite>npm run docker:build-windows

> [email protected] docker:build-windows D:\code\azurite
> npm run docker:prebuild && cross-var docker build --no-cache --rm -f "Dockerfile.Windows" -t xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version . && cross-var docker tag xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version xstoreazurite.azurecr.io/public/azure-storage/azurite:latest

npm WARN config cache-max This option has been deprecated in favor of `--prefer-online`
npm WARN config cache-min This option has been deprecated in favor of `--prefer-offline`.
npm WARN config optional Use `--omit=optional` to exclude optional dependencies, or
npm WARN config `--include=optional` to include them.
npm WARN config
npm WARN config     Default value does install optional deps unless otherwise omitted.
npm WARN config shrinkwrap Use the --package-lock setting instead.
npm WARN config tmp This setting is no longer used.  npm stores temporary files in a special
npm WARN config location in the cache, and they are managed by
npm WARN config     [`cacache`](http://npm.im/cacache).

> [email protected] docker:prebuild D:\code\azurite
> echo skip

skip
[+] Building 14.0s (6/23)                                                                                                                                                                                                                                                                                                                  docker:default
 => [internal] load build definition from Dockerfile.Windows                                                                                                                                                                                                                                                                                         0.1s
 => => transferring dockerfile: 1.22kB                                                                                                                                                                                                                                                                                                               0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                                                                                    0.1s
 => => transferring context: 282B                                                                                                                                                                                                                                                                                                                    0.0s
 => [internal] load metadata for mcr.microsoft.com/windows/nanoserver:ltsc2022-amd64                                                                                                                                                                                                                                                                 0.9s
 => [nodewindows 1/6] FROM mcr.microsoft.com/windows/nanoserver:ltsc2022-amd64@sha256:505fd350350a443863a2f8e827878216343389429572ec5e7b89aab4d18dce10                                                                                                                                                                                              12.7s
 => => resolve mcr.microsoft.com/windows/nanoserver:ltsc2022-amd64@sha256:505fd350350a443863a2f8e827878216343389429572ec5e7b89aab4d18dce10                                                                                                                                                                                                           0.0s
 => => sha256:505fd350350a443863a2f8e827878216343389429572ec5e7b89aab4d18dce10 429B / 429B                                                                                                                                                                                                                                                           0.0s
 => => sha256:46a42778ee1d5da936c5a65c66528eb6c727cd190a7935fe60cb33017bd29f45 638B / 638B                                                                                                                                                                                                                                                           0.0s
 => => sha256:3221147efd2bf92b8a3ff6e6b8206dfab879e886d1b97b07845f9eb5ddd0ab50 116.99MB / 116.99MB                                                                                                                                                                                                                                                   8.7s
 => => extracting sha256:3221147efd2bf92b8a3ff6e6b8206dfab879e886d1b97b07845f9eb5ddd0ab50                                                                                                                                                                                                                                                            3.9s
 => [internal] load build context                                                                                                                                                                                                                                                                                                                    5.2s
 => => transferring context: 23.58MB                                                                                                                                                                                                                                                                                                                 5.2s
 => ERROR [nodewindows 2/6] RUN mkdir c:node                                                                                                                                                                                                                                                                                                         0.2s
------
 > [nodewindows 2/6] RUN mkdir c:node:
------
Dockerfile.Windows:6
--------------------
   4 |     FROM mcr.microsoft.com/windows/nanoserver:ltsc2022-amd64 AS nodewindows
   5 |     # Install dependencies first
   6 | >>> RUN mkdir c:\node
   7 |     WORKDIR c:\\node
   8 |
--------------------
ERROR: failed to solve: process "cmd /S /C mkdir c:\\node" did not complete successfully: unable to find user ContainerUser: invalid argument
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] docker:build-windows: `npm run docker:prebuild && cross-var docker build --no-cache --rm -f "Dockerfile.Windows" -t xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version . && cross-var docker tag xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version xstoreazurite.azurecr.io/public/azure-storage/azurite:latest`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] docker:build-windows script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\weiwei\AppData\Roaming\npm-cache\_logs\2024-01-08T05_44_24_027Z-debug.log

My machine has:
Runtime Environment:
OS Name: Windows
OS Version: 10.0.19045
OS Platform: Windows
RID: win10-x64
Base Path: C:\Program Files\dotnet\sdk\7.0.404\

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What type of containers are you running? The error seems to be related with running Linux Containers instead of Windows Containers. (You can check with docker info -f "{{.OSType}}" and switch using the context menu of docker desktop Switch to Windows containers).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have switched docker on my machine to Windows.
However, run npm run docker:build-windows will still fail.
See failure log:
2024-01-08T09_30_37_462Z-debug.log

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not see the actual exception in the logfile. From what I see one of the child processes spawned by the npm command died with an error, do you have a log of the subcommand that failed?
Running the command with the same node & npm versions as specified in the logfile does not trigger any error at my side (nor on DevOps).

Copy link
Member

@blueww blueww Jan 9, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After I upgrade node to LTS version, and try again, I meet following error:

ltsc2022-amd64: Pulling from windows/nanoserver
a Windows version 10.0.20348-based image is incompatible with a 10.0.19045 host

It looks my windows version is lower than the image and cause the docker image build failure.
I will try to upgrade my machine or find a new machine and try build it. However, it might need some time.
Will update you later.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can successfully build the docker image on a win11 machine with node 18.19.0.
However, get following warning.
So to get the new docker image in good shape, we should increasing node version.

npm WARN notsup Unsupported engine for [email protected]: wanted: {"node":">=16"} (current: {"node":"14.21.3","npm":"6.14.18"})
npm WARN notsup Not compatible with your version of node/npm: [email protected]
npm verb notsup Not compatible with your version of node/npm: [email protected]
npm verb notsup Required: {"node":">=16"}
npm verb notsup Actual:   {"npm":"6.14.18","node":"14.21.3"}
npm WARN notsup Unsupported engine for @azure/[email protected]: wanted: {"node":">=16.0.0"} (current: {"node":"14.21.3","npm":"6.14.18"})
npm WARN notsup Not compatible with your version of node/npm: @azure/[email protected]
npm verb notsup Not compatible with your version of node/npm: @azure/[email protected]
npm verb notsup Required: {"node":">=16.0.0"}
npm verb notsup Actual:   {"npm":"6.14.18","node":"14.21.3"}
npm WARN notsup Unsupported engine for [email protected]: wanted: {"node":">=16.14"} (current: {"node":"14.21.3","npm":"6.14.18"})
npm WARN notsup Not compatible with your version of node/npm: [email protected]
npm verb notsup Not compatible with your version of node/npm: [email protected]
npm verb notsup Required: {"node":">=16.14"}
npm verb notsup Actual:   {"npm":"6.14.18","node":"14.21.3"}
npm WARN notsup Unsupported engine for @azure/[email protected]: wanted: {"node":">=18.0.0"} (current: {"node":"14.21.3","npm":"6.14.18"})
npm WARN notsup Not compatible with your version of node/npm: @azure/[email protected]
npm verb notsup Not compatible with your version of node/npm: @azure/[email protected]
npm verb notsup Required: {"node":">=18.0.0"}
npm verb notsup Actual:   {"npm":"6.14.18","node":"14.21.3"}
npm WARN notsup Unsupported engine for @azure/[email protected]: wanted: {"node":">=16"} (current: {"node":"14.21.3","npm":"6.14.18"})
npm WARN notsup Not compatible with your version of node/npm: @azure/[email protected]
npm verb notsup Not compatible with your version of node/npm: @azure/[email protected]
npm verb notsup Required: {"node":">=16"}
npm verb notsup Actual:   {"npm":"6.14.18","node":"14.21.3"}


RUN curl.exe -o Node.zip https://nodejs.org/dist/v14.21.3/node-v14.21.3-win-x64.zip
RUN tar -xf Node.zip -C c:\node
RUN del Node.zip

ENV PATH="$PATH;C:\Node\node-v14.21.3-win-x64"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we use node with a higher version?

Copy link
Author

@jwdb jwdb Jan 8, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could, but then this Dockerfile would deviate from the linux containers version (which also is using Node 14). Also the build process would be slightly different (options like npm config set unsafe-perm=true are no longer available in Node 16+).

To me it feels like upgrading the node version should be a seperate issue, since the current Dockerfile also uses Node 14 and there will be changes needed in the build commands.

Copy link
Member

@blueww blueww Jan 8, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the new added docker image, hope it's adding with a good shape. So the node version should be in the recommended version list (better with LTS version): https://nodered.org/docs/faq/node-versions

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've just tried updating to a newer version of Node (16/18/20) and it seems like I'm hitting npm/cli#4027. The current prepare script executes tsc which is unavailable and unwanted during the install phase. Changing this would require either a rather dirty workaround (setting the prepare step to "" in the docker file) or changes in the normal build process specified in the package.json. If desired I could implement these changes, but it feels like it's going rather out of scope.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update!

Yes, for the new added docker image, we would like it be in the good shape.
It would be great if you could add the change to make it on the node LTS version.


#
# Builder
#
FROM nodewindows AS builder

RUN mkdir c:\azurite
WORKDIR c:\\azurite

COPY *.json LICENSE NOTICE.txt ./

# Copy the source code and build the app
COPY src ./src
COPY tests ./tests
RUN npm config set unsafe-perm=true && \
npm ci
RUN npm run build && \
npm install -g --loglevel verbose


#
# Production image
#
FROM nodewindows

ENV NODE_ENV=productions

RUN mkdir c:\azurite
WORKDIR c:\\azurite

# Default Workspace Volume
VOLUME [ "c:/data" ]

COPY package*.json LICENSE NOTICE.txt ./

COPY --from=builder c:/azurite/dist/ dist/

RUN dir c:\azurite

RUN npm config set unsafe-perm=true && \
npm install -g --loglevel verbose

# Blob Storage Port
EXPOSE 10000
# Queue Storage Port
EXPOSE 10001
# Table Storage Port
EXPOSE 10002

CMD "azurite -l c:/data --blobHost 0.0.0.0 --queueHost 0.0.0.0 --tableHost 0.0.0.0"
22 changes: 22 additions & 0 deletions azure-pipelines.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -547,6 +547,28 @@ jobs:
docker run xstoreazurite.azurecr.io/public/azure-storage/azurite:latest azurite-table -v
workingDirectory: "./"
displayName: "Validate docker image"
- job: dockerwindows
displayName: Docker Windows Build
pool:
vmImage: "windows-latest"
steps:
- script: |
npm ci --legacy-peer-deps
workingDirectory: "./"
displayName: "npm ci --legacy-peer-deps"

- script: |
npm run docker:build-windows
workingDirectory: "./"
displayName: "Build docker image"

- script: |
docker run xstoreazurite.azurecr.io/public/azure-storage/azurite:latest "cmd /s /c azurite" -v
docker run xstoreazurite.azurecr.io/public/azure-storage/azurite:latest "cmd /s /c azurite-blob" -v
docker run xstoreazurite.azurecr.io/public/azure-storage/azurite:latest "cmd /s /c azurite-queue" -v
docker run xstoreazurite.azurecr.io/public/azure-storage/azurite:latest "cmd /s /c azurite-table" -v
workingDirectory: "./"
displayName: "Validate docker image"

- job: governance
displayName: Component Governance Component Detection
Expand Down
8 changes: 6 additions & 2 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -272,16 +272,20 @@
"vscode:pack": "vsce package",
"docker:prebuild": "echo skip",
"docker:build": "npm run docker:prebuild && cross-var docker build --no-cache --rm -f \"Dockerfile\" -t xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version . && cross-var docker tag xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version xstoreazurite.azurecr.io/public/azure-storage/azurite:latest",
"docker:build-windows": "npm run docker:prebuild && cross-var docker build --no-cache --rm -f \"Dockerfile.Windows\" -t xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version . && cross-var docker tag xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version xstoreazurite.azurecr.io/public/azure-storage/azurite:latest",
"docker:build:internal": "npm run docker:prebuild && cross-var docker build --no-cache --rm -f \"Dockerfile\" -t xstoreazurite.azurecr.io/internal/azure-storage/azurite:$npm_package_version . && cross-var docker tag xstoreazurite.azurecr.io/internal/azure-storage/azurite:$npm_package_version xstoreazurite.azurecr.io/internal/azure-storage/azurite:latest",
"docker:build-windows:internal": "npm run docker:prebuild && cross-var docker build --no-cache --rm -f \"Dockerfile.Windows\" -t xstoreazurite.azurecr.io/internal/azure-storage/azurite:$npm_package_version . && cross-var docker tag xstoreazurite.azurecr.io/internal/azure-storage/azurite:$npm_package_version xstoreazurite.azurecr.io/internal/azure-storage/azurite:latest",
"docker:publish": "cross-var docker push xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version",
"docker:publish:internal": "cross-var docker push xstoreazurite.azurecr.io/internal/azure-storage/azurite:$npm_package_version",
"docker:init-multi-platform-builder": "docker buildx create --name multi-platform-builder --use",
"docker:build-amd64": "cross-var docker buildx build --platform linux/amd64 --load --no-cache --rm -f \"Dockerfile\" -t xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version-amd64 .",
"docker:build-windows-amd64": "cross-var docker buildx build --platform windows/amd64 --load --no-cache --rm -f \"Dockerfile.Windows\" -t xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version-windows-amd64 .",
"docker:build-arm64": "cross-var docker buildx build --platform linux/arm64 --load --no-cache --rm -f \"Dockerfile\" -t xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version-arm64 .",
"docker:publish-amd64": "cross-var docker push xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version-amd64",
"docker:publish-windows-amd64": "cross-var docker push xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version-windows-amd64",
"docker:publish-arm64": "cross-var docker push xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version-arm64",
"docker:create-manifest-versioned": "cross-var docker manifest create xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version-amd64 xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version-arm64",
"docker:create-manifest-latest": "cross-var docker manifest create xstoreazurite.azurecr.io/public/azure-storage/azurite:latest xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version-amd64 xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version-arm64",
"docker:create-manifest-versioned": "cross-var docker manifest create xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version-amd64 xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version-windows-amd64 xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version-arm64",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to check what's the effect after publish the manifest.
With the originally manifest, when user run Azurite docker image without specifying platform, docker will choose the docker image most match the current platform (ARM64 platform will choose AMD64 docker image, AMD64 platform will choose AMD64 docker image). After the windows-amd64 image is added, what will happen when user run Azurite docker image without specifying platform?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The platform is chooses includes the os name, hence it currently throwing an error on windows ( no matching manifest for windows/amd64 in the manifest list entries) adding this image will add that specific platform to the manifest.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With current Azurite release (before the change), I will install AMD64 docker image in my machine (my machine is windows with AMD64).
So after your change, what will happen on different platform/OS?

And for the error, do you mean you will resolve it, or it's already resolved?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you are running Docker in Windows Container mode (this is an option within Docker-Desktop), with the current release this will throw an error.
After the change, the other platforms will be unaffected (since they will use the linux/amd64 image).

You can view the platform docker is running internally on with docker info -f "{{.OSType}}". For me this gives windows.

Copy link
Member

@blueww blueww Jan 4, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Get it. Thanks!

And on my windows machine run docker info -f "{{.OSType}}", will get result "linux".

"docker:create-manifest-latest": "cross-var docker manifest create xstoreazurite.azurecr.io/public/azure-storage/azurite:latest xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version-amd64 xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version-windows-amd64 xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version-arm64",
"docker:publish-manifest-versioned": "cross-var docker manifest push xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version",
"docker:publish-manifest-latest": "cross-var docker manifest push xstoreazurite.azurecr.io/public/azure-storage/azurite:latest",
"prepare": "npm run build",
Expand Down
Loading