diff --git a/docs/Community.md b/docs/Community.md index faf510c10..4837beafa 100644 --- a/docs/Community.md +++ b/docs/Community.md @@ -38,14 +38,23 @@ When you want to show your users and contributors that they can use pixi in your - [MODFLOW 6](https://github.com/MODFLOW-USGS/modflow6): USGS modular hydrological model - QuantCo: - [glum](https://github.com/quantco/glum): High performance Python GLMs with all the features! + - [tabmat](https://github.com/quantco/tabmat): Efficient matrix representations for working with tabular data - [pixi-pack](https://github.com/quantco/pixi-pack): A tool to pack and unpack conda environments created with pixi - - [polarify](https://github.com/Quantco/polarify): Simplifying conditional Polars Expressions with Python 🐍 πŸ»β€β„οΈ - - [datajudge](https://github.com/Quantco/datajudge): Assessing whether data from database complies with reference information - - [ndonnx](https://github.com/Quantco/ndonnx): ONNX-backed array library that is compliant with the Array API standard - - [multiregex](https://github.com/Quantco/multiregex): Quickly match many regexes against a string + - [polarify](https://github.com/quantco/polarify): Simplifying conditional Polars Expressions with Python 🐍 πŸ»β€β„οΈ + - [copier-template-python-open-source](https://github.com/quantco/copier-template-python-open-source): Copier template for python projects using pixi + - [datajudge](https://github.com/quantco/datajudge): Assessing whether data from database complies with reference information + - [ndonnx](https://github.com/quantco/ndonnx): ONNX-backed array library that is compliant with the Array API standard + - [multiregex](https://github.com/quantco/multiregex): Quickly match many regexes against a string - [slim-trees](https://github.com/quantco/slim-trees): Pickle your ML models more efficiently for deployment πŸš€ - - [metalearners](https://github.com/Quantco/metalearners): MetaLearners for CATE estimation + - [sqlcompyre](https://github.com/quantco/sqlcompyre): Compare SQL tables and databases + - [metalearners](https://github.com/quantco/metalearners): MetaLearners for CATE estimation + - [ndonnx](https://github.com/quantco/ndonnx): ONNX-backed array library that is compliant with the Array API standard + - [tabulardelta](https://github.com/quantco/tabulardelta): Simplify table comparisons + - [pydiverse.pipedag](https://github.com/pydiverse/pydiverse.pipedag): A library for data pipeline orchestration optimizing high development iteration speed + - [pydiverse.transform](https://github.com/pydiverse/pydiverse.transform): Pipe based dataframe manipulation library that can also transform data on SQL databases +- [pixi-pycharm](https://github.com/pavelzw/pixi-pycharm): Conda shim for PyCharm that proxies pixi +- [pixi-diff-to-markdown](https://github.com/pavelzw/pixi-diff-to-markdown): Generate markdown summaries from pixi update - [jiaxiyang/cpp_project_guideline](https://github.com/jiaxiyang/cpp_project_guideline): Guide the way beginners make their c++ projects. - [karelze/tclf](https://github.com/KarelZe/tclf): A python library for trade classification⚑ - [hex-inc/vegafusion](https://github.com/hex-inc/vegafusion): Serverside scaling of Vega and Altair visualizations in Rust, Python, WASM, and Java diff --git a/docs/advanced/production_deployment.md b/docs/advanced/production_deployment.md new file mode 100644 index 000000000..45af04cf2 --- /dev/null +++ b/docs/advanced/production_deployment.md @@ -0,0 +1,153 @@ +# Bringing pixi to production + +You can bring pixi projects into production by either containerizing it using tools like Docker or by using [`quantco/pixi-pack`](https://github.com/quantco/pixi-pack). + +!!!tip "" + [@pavelzw](https://github.com/pavelzw) from [QuantCo](https://quantco.com) wrote a blog post about bringing pixi to production. You can read it [here](https://tech.quantco.com/blog/pixi-production). + +## Docker + + + +We provide a simple docker image at [`pixi-docker`](https://github.com/prefix-dev/pixi-docker) that contains the pixi executable on top of different base images. + +The images are available on [ghcr.io/prefix-dev/pixi](https://ghcr.io/prefix-dev/pixi). + +There are different tags for different base images available: + +- `latest` - based on `ubuntu:jammy` +- `focal` - based on `ubuntu:focal` +- `bullseye` - based on `debian:bullseye` +- `jammy-cuda-12.2.2` - based on `nvidia/cuda:12.2.2-jammy` +- ... and more + +!!!tip "All tags" + For all tags, take a look at the [build script](https://github.com/prefix-dev/pixi-docker/blob/main/.github/workflows/build.yml). + +### Example usage + +The following example uses the pixi docker image as a base image for a multi-stage build. +It also makes use of `pixi shell-hook` to not rely on pixi being installed in the production container. + +!!!tip "More examples" + For more examples, take a look at [pavelzw/pixi-docker-example](https://github.com/pavelzw/pixi-docker-example). + +```Dockerfile +FROM ghcr.io/prefix-dev/pixi:0.31.0 AS build + +# copy source code, pixi.toml and pixi.lock to the container +WORKDIR /app +COPY . . +# install dependencies to `/app/.pixi/envs/prod` +# use `--locked` to ensure the lockfile is up to date with pixi.toml +RUN pixi install --locked -e prod +# create the shell-hook bash script to activate the environment +RUN pixi shell-hook -e prod -s bash > /shell-hook +RUN echo "#!/bin/bash" > /app/entrypoint.sh +RUN cat /shell-hook >> /app/entrypoint.sh +# extend the shell-hook script to run the command passed to the container +RUN echo 'exec "$@"' >> /app/entrypoint.sh + +FROM ubuntu:24.04 AS production +WORKDIR /app +# only copy the production environment into prod container +# please note that the "prefix" (path) needs to stay the same as in the build container +COPY --from=build /app/.pixi/envs/prod /app/.pixi/envs/prod +COPY --from=build --chmod=0755 /app/entrypoint.sh /app/entrypoint.sh +# copy your project code into the container as well +COPY ./my_project /app/my_project + +EXPOSE 8000 +ENTRYPOINT [ "/app/entrypoint.sh" ] +# run your app inside the pixi environment +CMD [ "uvicorn", "my_project:app", "--host", "0.0.0.0" ] +``` + +## pixi-pack + + + +[`pixi-pack`](https://github.com/quantco/pixi-pack) is a simple tool that takes a pixi environment and packs it into a compressed archive that can be shipped to the target machine. + +It can be installed via + +```bash +pixi global install pixi-pack +``` + +Or by downloading our pre-built binaries from the [releases page](https://github.com/quantco/pixi-pack/releases). + +Instead of installing pixi-pack globally, you can also use pixi exec to run `pixi-pack` in a temporary environment: + +```bash +pixi exec pixi-pack pack +pixi exec pixi-pack unpack environment.tar +``` + +![pixi-pack demo](https://raw.githubusercontent.com/quantco/pixi-pack/refs/heads/main/.github/assets/demo/demo-light.gif#only-light) +![pixi-pack demo](https://raw.githubusercontent.com/quantco/pixi-pack/refs/heads/main/.github/assets/demo/demo-dark.gif#only-dark) + +You can pack an environment with + +```bash +pixi-pack pack --manifest-file pixi.toml --environment prod --platform linux-64 +``` + +This will create a `environment.tar` file that contains all conda packages required to create the environment. + +```plain +# environment.tar +| pixi-pack.json +| environment.yml +| channel +| β”œβ”€β”€ noarch +| | β”œβ”€β”€ tzdata-2024a-h0c530f3_0.conda +| | β”œβ”€β”€ ... +| | └── repodata.json +| └── linux-64 +| β”œβ”€β”€ ca-certificates-2024.2.2-hbcca054_0.conda +| β”œβ”€β”€ ... +| └── repodata.json +``` + +### Unpacking an environment + +With `pixi-pack unpack environment.tar`, you can unpack the environment on your target system. This will create a new conda environment in `./env` that contains all packages specified in your `pixi.toml`. It also creates an `activate.sh` (or `activate.bat` on Windows) file that lets you activate the environment without needing to have `conda` or `micromamba` installed. + +### Cross-platform packs + +Since `pixi-pack` just downloads the `.conda` and `.tar.bz2` files from the conda repositories, you can trivially create packs for different platforms. + +```bash +pixi-pack pack --platform win-64 +``` + +!!!note "" + You can only unpack a pack on a system that has the same platform as the pack was created for. + +### Inject additional packages + +You can inject additional packages into the environment that are not specified in `pixi.lock` by using the `--inject` flag: + +```bash +pixi-pack pack --inject local-package-1.0.0-hbefa133_0.conda --manifest-pack pixi.toml +``` + +This can be particularly useful if you build the project itself and want to include the built package in the environment but still want to use `pixi.lock` from the project. + +### Unpacking without pixi-pack + +If you don't have `pixi-pack` available on your target system, you can still install the environment if you have `conda` or `micromamba` available. +Just unarchive the `environment.tar`, then you have a local channel on your system where all necessary packages are available. +Next to this local channel, you will find an `environment.yml` file that contains the environment specification. +You can then install the environment using `conda` or `micromamba`: + +```bash +tar -xvf environment.tar +micromamba create -p ./env --file environment.yml +# or +conda env create -p ./env --file environment.yml +``` + +!!!note "" + The `environment.yml` and `repodata.json` files are only for this use case, `pixi-pack unpack` does not use them. diff --git a/docs/ide_integration/devcontainer.md b/docs/ide_integration/devcontainer.md new file mode 100644 index 000000000..1c572338c --- /dev/null +++ b/docs/ide_integration/devcontainer.md @@ -0,0 +1,76 @@ +# Use pixi inside of a devcontainer + +[VSCode Devcontainers](https://code.visualstudio.com/docs/devcontainers/containers) are a popular tool to develop on a project with a consistent environment. +They are also used in [GitHub Codespaces](https://github.com/features/codespaces) which makes it a great way to develop on a project without having to install anything on your local machine. + +To use pixi inside of a devcontainer, follow these steps: + +Create a new directory `.devcontainer` in the root of your project. +Then, create the following two files in the `.devcontainer` directory: + +```dockerfile title=".devcontainer/Dockerfile" +FROM mcr.microsoft.com/devcontainers/base:jammy + +ARG PIXI_VERSION=v0.31.0 + +RUN curl -L -o /usr/local/bin/pixi -fsSL --compressed "https://github.com/prefix-dev/pixi/releases/download/${PIXI_VERSION}/pixi-$(uname -m)-unknown-linux-musl" \ + && chmod +x /usr/local/bin/pixi \ + && pixi info + +# set some user and workdir settings to work nicely with vscode +USER vscode +WORKDIR /home/vscode + +RUN echo 'eval "$(pixi completion -s bash)"' >> /home/vscode/.bashrc +``` + +```json title=".devcontainer/devcontainer.json" +{ + "name": "my-project", + "build": { + "dockerfile": "Dockerfile", + "context": "..", + }, + "customizations": { + "vscode": { + "settings": {}, + "extensions": ["ms-python.python", "charliermarsh.ruff", "GitHub.copilot"] + } + }, + "features": { + "ghcr.io/devcontainers/features/docker-in-docker:2": {} + }, + "mounts": ["source=${localWorkspaceFolderBasename}-pixi,target=${containerWorkspaceFolder}/.pixi,type=volume"], + "postCreateCommand": "sudo chown vscode .pixi && pixi install" +} +``` + +!!!tip "Put `.pixi` in a mount" + In the above example, we mount the `.pixi` directory into a volume. + This is needed since the `.pixi` directory shouldn't be on a case insensitive filesystem (default on macOS, Windows) but instead in its own volume. + There are some conda packages (for example [ncurses-feedstock#73](https://github.com/conda-forge/ncurses-feedstock/issues/73)) that contain files that only differ in case which leads to errors on case insensitive filesystems. + +## Secrets + +If you want to authenticate to a private conda channel, you can add secrets to your devcontainer. + +```json title=".devcontainer/devcontainer.json" +{ + "build": "Dockerfile", + "context": "..", + "options": [ + "--secret", + "id=prefix_dev_token,env=PREFIX_DEV_TOKEN", + ], + // ... +} +``` + +```dockerfile title=".devcontainer/Dockerfile" +# ... +RUN --mount=type=secret,id=prefix_dev_token,uid=1000 \ + test -s /run/secrets/prefix_dev_token \ + && pixi auth login --token "$(cat /run/secrets/prefix_dev_token)" https://repo.prefix.dev +``` + +These secrets need to be present either as an environment variable when starting the devcontainer locally or in your [GitHub Codespaces settings](https://github.com/settings/codespaces) under `Secrets`. diff --git a/mkdocs.yml b/mkdocs.yml index b3c72ad9d..7b50c7339 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -106,6 +106,7 @@ nav: - JupyterLab: ide_integration/jupyterlab.md - PyCharm: ide_integration/pycharm.md - RStudio: ide_integration/r_studio.md + - VSCode Devcontainer: ide_integration/devcontainer.md - Tutorials: - Python: tutorials/python.md - ROS 2: tutorials/ros2.md @@ -129,6 +130,7 @@ nav: - Channel Logic: advanced/channel_priority.md - GitHub Actions: advanced/github_actions.md - Updates using GitHub Actions: advanced/updates_github_actions.md + - Production Deployment: advanced/production_deployment.md - Pyproject.toml: advanced/pyproject_toml.md - Reference: - Project Configuration: reference/project_configuration.md diff --git a/tbump.toml b/tbump.toml index 934bc9ae3..83ccde09a 100644 --- a/tbump.toml +++ b/tbump.toml @@ -43,6 +43,14 @@ src = "Cargo.toml" search = "pixi-version: v{current_version}" src = "docs/advanced/github_actions.md" +[[file]] +search = "FROM ghcr.io/prefix-dev/pixi:{current_version} AS build" +src = "docs/advanced/production_deployment.md" + +[[file]] +search = "ARG PIXI_VERSION=v{current_version}" +src = "docs/ide_integration/devcontainer.md" + [[file]] search = "/pixi.sh/v{current_version}/" src = "schema/schema.json"