-
Notifications
You must be signed in to change notification settings - Fork 178
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
docs: pixi-pack, docker, devcontainer (#2220)
Co-authored-by: Ruben Arts <[email protected]>
- Loading branch information
1 parent
989838c
commit e6b0533
Showing
5 changed files
with
253 additions
and
5 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,153 @@ | ||
| # Bringing pixi to production | ||
|
|
||
| You can bring pixi projects into production by either containerizing it using tools like Docker or by using [`quantco/pixi-pack`](https://github.com/quantco/pixi-pack). | ||
|
|
||
| !!!tip "" | ||
| [@pavelzw](https://github.com/pavelzw) from [QuantCo](https://quantco.com) wrote a blog post about bringing pixi to production. You can read it [here](https://tech.quantco.com/blog/pixi-production). | ||
|
|
||
| ## Docker | ||
|
|
||
| <!-- Keep in sync with https://github.com/prefix-dev/pixi-docker/blob/main/README.md --> | ||
|
|
||
| We provide a simple docker image at [`pixi-docker`](https://github.com/prefix-dev/pixi-docker) that contains the pixi executable on top of different base images. | ||
|
|
||
| The images are available on [ghcr.io/prefix-dev/pixi](https://ghcr.io/prefix-dev/pixi). | ||
|
|
||
| There are different tags for different base images available: | ||
|
|
||
| - `latest` - based on `ubuntu:jammy` | ||
| - `focal` - based on `ubuntu:focal` | ||
| - `bullseye` - based on `debian:bullseye` | ||
| - `jammy-cuda-12.2.2` - based on `nvidia/cuda:12.2.2-jammy` | ||
| - ... and more | ||
|
|
||
| !!!tip "All tags" | ||
| For all tags, take a look at the [build script](https://github.com/prefix-dev/pixi-docker/blob/main/.github/workflows/build.yml). | ||
|
|
||
| ### Example usage | ||
|
|
||
| The following example uses the pixi docker image as a base image for a multi-stage build. | ||
| It also makes use of `pixi shell-hook` to not rely on pixi being installed in the production container. | ||
|
|
||
| !!!tip "More examples" | ||
| For more examples, take a look at [pavelzw/pixi-docker-example](https://github.com/pavelzw/pixi-docker-example). | ||
|
|
||
| ```Dockerfile | ||
| FROM ghcr.io/prefix-dev/pixi:0.31.0 AS build | ||
|
|
||
| # copy source code, pixi.toml and pixi.lock to the container | ||
| WORKDIR /app | ||
| COPY . . | ||
| # install dependencies to `/app/.pixi/envs/prod` | ||
| # use `--locked` to ensure the lockfile is up to date with pixi.toml | ||
| RUN pixi install --locked -e prod | ||
| # create the shell-hook bash script to activate the environment | ||
| RUN pixi shell-hook -e prod -s bash > /shell-hook | ||
| RUN echo "#!/bin/bash" > /app/entrypoint.sh | ||
| RUN cat /shell-hook >> /app/entrypoint.sh | ||
| # extend the shell-hook script to run the command passed to the container | ||
| RUN echo 'exec "$@"' >> /app/entrypoint.sh | ||
|
|
||
| FROM ubuntu:24.04 AS production | ||
| WORKDIR /app | ||
| # only copy the production environment into prod container | ||
| # please note that the "prefix" (path) needs to stay the same as in the build container | ||
| COPY --from=build /app/.pixi/envs/prod /app/.pixi/envs/prod | ||
| COPY --from=build --chmod=0755 /app/entrypoint.sh /app/entrypoint.sh | ||
| # copy your project code into the container as well | ||
| COPY ./my_project /app/my_project | ||
|
|
||
| EXPOSE 8000 | ||
| ENTRYPOINT [ "/app/entrypoint.sh" ] | ||
| # run your app inside the pixi environment | ||
| CMD [ "uvicorn", "my_project:app", "--host", "0.0.0.0" ] | ||
| ``` | ||
|
|
||
| ## pixi-pack | ||
|
|
||
| <!-- Keep in sync with https://github.com/quantco/pixi-pack/blob/main/README.md --> | ||
|
|
||
| [`pixi-pack`](https://github.com/quantco/pixi-pack) is a simple tool that takes a pixi environment and packs it into a compressed archive that can be shipped to the target machine. | ||
|
|
||
| It can be installed via | ||
|
|
||
| ```bash | ||
| pixi global install pixi-pack | ||
| ``` | ||
|
|
||
| Or by downloading our pre-built binaries from the [releases page](https://github.com/quantco/pixi-pack/releases). | ||
|
|
||
| Instead of installing pixi-pack globally, you can also use pixi exec to run `pixi-pack` in a temporary environment: | ||
|
|
||
| ```bash | ||
| pixi exec pixi-pack pack | ||
| pixi exec pixi-pack unpack environment.tar | ||
| ``` | ||
|
|
||
|  | ||
|  | ||
|
|
||
| You can pack an environment with | ||
|
|
||
| ```bash | ||
| pixi-pack pack --manifest-file pixi.toml --environment prod --platform linux-64 | ||
| ``` | ||
|
|
||
| This will create a `environment.tar` file that contains all conda packages required to create the environment. | ||
|
|
||
| ```plain | ||
| # environment.tar | ||
| | pixi-pack.json | ||
| | environment.yml | ||
| | channel | ||
| | ├── noarch | ||
| | | ├── tzdata-2024a-h0c530f3_0.conda | ||
| | | ├── ... | ||
| | | └── repodata.json | ||
| | └── linux-64 | ||
| | ├── ca-certificates-2024.2.2-hbcca054_0.conda | ||
| | ├── ... | ||
| | └── repodata.json | ||
| ``` | ||
|
|
||
| ### Unpacking an environment | ||
|
|
||
| With `pixi-pack unpack environment.tar`, you can unpack the environment on your target system. This will create a new conda environment in `./env` that contains all packages specified in your `pixi.toml`. It also creates an `activate.sh` (or `activate.bat` on Windows) file that lets you activate the environment without needing to have `conda` or `micromamba` installed. | ||
|
|
||
| ### Cross-platform packs | ||
|
|
||
| Since `pixi-pack` just downloads the `.conda` and `.tar.bz2` files from the conda repositories, you can trivially create packs for different platforms. | ||
|
|
||
| ```bash | ||
| pixi-pack pack --platform win-64 | ||
| ``` | ||
|
|
||
| !!!note "" | ||
| You can only unpack a pack on a system that has the same platform as the pack was created for. | ||
|
|
||
| ### Inject additional packages | ||
|
|
||
| You can inject additional packages into the environment that are not specified in `pixi.lock` by using the `--inject` flag: | ||
|
|
||
| ```bash | ||
| pixi-pack pack --inject local-package-1.0.0-hbefa133_0.conda --manifest-pack pixi.toml | ||
| ``` | ||
|
|
||
| This can be particularly useful if you build the project itself and want to include the built package in the environment but still want to use `pixi.lock` from the project. | ||
|
|
||
| ### Unpacking without pixi-pack | ||
|
|
||
| If you don't have `pixi-pack` available on your target system, you can still install the environment if you have `conda` or `micromamba` available. | ||
| Just unarchive the `environment.tar`, then you have a local channel on your system where all necessary packages are available. | ||
| Next to this local channel, you will find an `environment.yml` file that contains the environment specification. | ||
| You can then install the environment using `conda` or `micromamba`: | ||
|
|
||
| ```bash | ||
| tar -xvf environment.tar | ||
| micromamba create -p ./env --file environment.yml | ||
| # or | ||
| conda env create -p ./env --file environment.yml | ||
| ``` | ||
|
|
||
| !!!note "" | ||
| The `environment.yml` and `repodata.json` files are only for this use case, `pixi-pack unpack` does not use them. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,76 @@ | ||
| # Use pixi inside of a devcontainer | ||
|
|
||
| [VSCode Devcontainers](https://code.visualstudio.com/docs/devcontainers/containers) are a popular tool to develop on a project with a consistent environment. | ||
| They are also used in [GitHub Codespaces](https://github.com/features/codespaces) which makes it a great way to develop on a project without having to install anything on your local machine. | ||
|
|
||
| To use pixi inside of a devcontainer, follow these steps: | ||
|
|
||
| Create a new directory `.devcontainer` in the root of your project. | ||
| Then, create the following two files in the `.devcontainer` directory: | ||
|
|
||
| ```dockerfile title=".devcontainer/Dockerfile" | ||
| FROM mcr.microsoft.com/devcontainers/base:jammy | ||
|
|
||
| ARG PIXI_VERSION=v0.31.0 | ||
|
|
||
| RUN curl -L -o /usr/local/bin/pixi -fsSL --compressed "https://github.com/prefix-dev/pixi/releases/download/${PIXI_VERSION}/pixi-$(uname -m)-unknown-linux-musl" \ | ||
| && chmod +x /usr/local/bin/pixi \ | ||
| && pixi info | ||
|
|
||
| # set some user and workdir settings to work nicely with vscode | ||
| USER vscode | ||
| WORKDIR /home/vscode | ||
|
|
||
| RUN echo 'eval "$(pixi completion -s bash)"' >> /home/vscode/.bashrc | ||
| ``` | ||
|
|
||
| ```json title=".devcontainer/devcontainer.json" | ||
| { | ||
| "name": "my-project", | ||
| "build": { | ||
| "dockerfile": "Dockerfile", | ||
| "context": "..", | ||
| }, | ||
| "customizations": { | ||
| "vscode": { | ||
| "settings": {}, | ||
| "extensions": ["ms-python.python", "charliermarsh.ruff", "GitHub.copilot"] | ||
| } | ||
| }, | ||
| "features": { | ||
| "ghcr.io/devcontainers/features/docker-in-docker:2": {} | ||
| }, | ||
| "mounts": ["source=${localWorkspaceFolderBasename}-pixi,target=${containerWorkspaceFolder}/.pixi,type=volume"], | ||
| "postCreateCommand": "sudo chown vscode .pixi && pixi install" | ||
| } | ||
| ``` | ||
|
|
||
| !!!tip "Put `.pixi` in a mount" | ||
| In the above example, we mount the `.pixi` directory into a volume. | ||
| This is needed since the `.pixi` directory shouldn't be on a case insensitive filesystem (default on macOS, Windows) but instead in its own volume. | ||
| There are some conda packages (for example [ncurses-feedstock#73](https://github.com/conda-forge/ncurses-feedstock/issues/73)) that contain files that only differ in case which leads to errors on case insensitive filesystems. | ||
|
|
||
| ## Secrets | ||
|
|
||
| If you want to authenticate to a private conda channel, you can add secrets to your devcontainer. | ||
|
|
||
| ```json title=".devcontainer/devcontainer.json" | ||
| { | ||
| "build": "Dockerfile", | ||
| "context": "..", | ||
| "options": [ | ||
| "--secret", | ||
| "id=prefix_dev_token,env=PREFIX_DEV_TOKEN", | ||
| ], | ||
| // ... | ||
| } | ||
| ``` | ||
|
|
||
| ```dockerfile title=".devcontainer/Dockerfile" | ||
| # ... | ||
| RUN --mount=type=secret,id=prefix_dev_token,uid=1000 \ | ||
| test -s /run/secrets/prefix_dev_token \ | ||
| && pixi auth login --token "$(cat /run/secrets/prefix_dev_token)" https://repo.prefix.dev | ||
| ``` | ||
|
|
||
| These secrets need to be present either as an environment variable when starting the devcontainer locally or in your [GitHub Codespaces settings](https://github.com/settings/codespaces) under `Secrets`. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters