Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nix installable packages #508

Draft
wants to merge 1 commit into
base: nix-packages
Choose a base branch
from

Conversation

Brikaa
Copy link
Member

@Brikaa Brikaa commented Aug 14, 2022

No description provided.

@Brikaa
Copy link
Member Author

Brikaa commented Aug 20, 2022

@HexF I get dependency problems when trying to inject some packages into the environment. For example this happens when installing python requests and adding its site-packages PYTHONPATH

Traceback (most recent call last):
  File \"/piston/jobs/351474a4-a3bd-4127-a0e9-cedf32106275/file0.code\", line 4, in <module>
    import requests
  File \"/nix/store/23x8xw10z0slsdvy9jjnblvf0hyl78yh-python3.9-requests-2.26.0/lib/python3.9/site-packages/requests/__init__.py\", line 100, in <module>
    check_compatibility(urllib3.__version__, chardet_version, charset_normalizer_version)
  File \"/nix/store/23x8xw10z0slsdvy9jjnblvf0hyl78yh-python3.9-requests-2.26.0/lib/python3.9/site-packages/requests/__init__.py\", line 85, in check_compatibility
    raise Exception(\"You need either charset_normalizer or chardet installed\")
Exception: You need either charset_normalizer or chardet installed

@Brikaa
Copy link
Member Author

Brikaa commented Aug 20, 2022

Creating a ready environment using pkgs.python3.withPackages does not have this problem.

@Brikaa
Copy link
Member Author

Brikaa commented Aug 20, 2022

But Python libraries you would like to use for development cannot be installed, at least not individually, because they won't be able to find each other resulting in import errors. Instead, it is possible to create an environment with python.buildEnv or python.withPackages where the interpreter and other executables are wrapped to be able to find each other and all of the modules.

@HexF
Copy link
Collaborator

HexF commented Aug 22, 2022

What if piston.mkRuntime in the nix flake takes a function, which allows us to pass in all the package names? That way we could dynamically construct the environment from nix. The only issue I can think of is the speed at which some environments will load.

{pkgs, piston, ...}:
let
    basePkg = pkgs.python3;
in piston.mkRuntime libraries: 
    let
pkg = basePkg.withPackages libraries;
in
    {
    language = "python3";
    version = pkg.version;

    aliases = [
        "py3"
        "py"
        "python"
    ];

    run = ''
    ${pkg}/bin/python3 "$@"
    '';

    packages = pkg.pkgs;

    tests = [
        (piston.mkTest {
            files = {
                "test.py" = ''
                    print("OK")
                '';
            };
        })
    ];
}

@Brikaa
Copy link
Member Author

Brikaa commented Aug 28, 2022

Flakes don't seem to support command-line arguments so how will we do the ad-hoc package installations?

@HexF
Copy link
Collaborator

HexF commented Aug 29, 2022

in piston.mkRuntime {
{ pkgs, piston, ... }:
let basePkg = pkgs.python3;
in piston.mkRuntime (libraries:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we possibly make this piston.mkRuntimePackages or similar, just so if packages aren't required we can use the old format?

@Brikaa
Copy link
Member Author

Brikaa commented Sep 3, 2022

Error: Command failed: nix eval --json /piston/src#pistonRuntimeSets.all.python3.metadata --arg sf true
error: '--arg' and '--argstr' are incompatible with flakes
Try 'nix --help' for more information.

@Brikaa
Copy link
Member Author

Brikaa commented Sep 3, 2022

There seems to be this: https://nixos.org/manual/nix/stable/language/builtins.html#builtins-getFlake if we export the runtimes as a function that takes the packages as an argument. But it seems to copy the entire working directory into the nix store every time and it is slow.

@Brikaa
Copy link
Member Author

Brikaa commented Sep 3, 2022

Maybe we can copy flake.nix into the job directory and inject the packages into it lol

@Brikaa
Copy link
Member Author

Brikaa commented Sep 4, 2022

The reason builtins.getFlake is slow is that it is ignoring the version control system and copying everything to the nix store every time. Putting git+file// before the path solves the problem.

@HexF
Copy link
Collaborator

HexF commented Sep 7, 2023

Finally got back around to looking at this... wow its been a while.
The reason why --arg isn't supported is kinda weird. It's to do with nix's pure builds, and if you can suddenly start throwing arguments at an eval call, the actual build outputs can change.
While this is what we want, nix doesn't actually support this for flakes.

What I'm thinking currently is just exporting the packages key to the API, calling nix build on all the paths we get out, then passing these paths into the run and compile scripts.
This way the run and compile scripts can deal with how to setup the environment correctly for the package set.
I think this was the way you were originally going to implement it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants