CLI extracted from Polykey and migrated to Polykey-CLI

[tegefaulkes]

Description

In this PR we are pulling out all of the bin related code from Polykey and transplanting it here. This enables our Polykey repo to be simpler and also closer to a library that can be used in downstream applications. The Polykey's CI/CD should work faster, and not be blocked on integration problems. This repo can be more focused on CLI workflows and integration problems and cross-platform packaging.

• Related: CLI migration Polykey#534
• Realted: Cross-Platform Standardisation Polykey#422

Tasks

• 1. New repo needs to be created, use the ts-demo-lib as a base
• 2. All CLI code needs to be transplanted.
• 3. All Bin related tests need to be transplanted.
• 4. All CLI and integration CI jobs need to be transplanted.

Final checklist

• Domain specific tests
• Full tests
• Updated inline-comment documentation
• Lint fixed
• Squash and rebased
• Sanity check the final build

[tegefaulkes]

I don't know if we want to explore bundling the source code just yet. I'll create/move an issue here for that. It should be done in a separate PR.

[CMCDragonkai]

You mean bundling before we feed it all into pkg?

We can do that later. Because we have to upgrade to ES modules first, and then figure out which bundler can support all we want properly.

[tegefaulkes]

Then we'll need to explore using ES modules in the next stage.

[tegefaulkes]

I think we're ready to start testing CI now.

[tegefaulkes]

We'll need to set up the Gitlab repo and mirroring.

[CMCDragonkai]

Is MatrixAI/Polykey#534 also completed? I mean it's pending the first PR, but that would have to extract everything so PK can be used as a library right?

[tegefaulkes]

The Polykey side changes haven't been done yet. It can be used as a library already except for two points.

1. We need to merge and publish the agent migration changes if we want to run CI testing here.
2. We need to review how Polykey does exports. Right now I'm doing a lot of Polykey/dist/whatever paths when importing and it's not ideal.

I'm blocked here at this stage. I need to move onto the Polykey side of CLI migration an merge the agent migration PR before moving forward here.

[CMCDragonkai]

I've setup the CI/CD configuration for this repo and also in gitlab.

Note that branches are now protected, so before merging, make sure to squash properly.

Furthermore, I've changed the default branches to be staging. Make sure that this PR is targeting staging not master.

Also remember to replicate all the scaffolding files like .github/workflows/codesee-arch-diagram.yml and all the other relevant files at the project root and testing architecture.

[CMCDragonkai]

• github
• .env
• .env.example
• .eslintignore
• .eslintrc
• .gitignore
• .gitlab-ci.yml
• .npmignore
• nodemon.json
• .npmrc
• .prettierrc
• tsconfig.json

Make sure tests structure and benches structure are the same, and also generated docs too.

[CMCDragonkai]

Oh and the images/cli_demo.gif should be moved over too!

The README.md inside Polykey should have a link to Polykey-CLI. And the demo gif can be moved here.

[CMCDragonkai]

Because Polykey is no longer an application, there's no need to have default.nix, nor utils.nix, nor release.nix. The only thing necessary is pkg.nix and shell.nix.

[CMCDragonkai]

The .github will bring in codesee.

Note that one thing is the container path needs to change too. We are now uploading containers to the polykey-cli repo in Gitlab:

[CMCDragonkai]

Regarding benchmarks, make sure that you have a dummy structure there for later, as can create benchmarks for our CLI too later.

[CMCDragonkai]

Synchronised issue labels too now.

[CMCDragonkai]

ETA on merge?

Make sure to test out the CI jobs locally too inside your own shell.nix.

[CMCDragonkai]

Some problems I had worked on earlier that will be relevant here:

• ci: merge staging to master TypeScript-Demo-Lib#94 (comment)
• Update node2nix with the upstream fixes to building bin links #21

Pretty much all the issues in TS demo lib is relevant here!

[tegefaulkes]

Tests are passing locally now. We're importing a pre-release version of polykey for this so that's all working.

The Container stuff still needs to be updated.

CI job is failing, namely nix-dry and the tests. Tests are acting odd, they're failing to import polykey and I'm not sure what's happening there.

The dry run nix-build is failing with the following error.

...
...
trace: warning: in docker image polykey-cli-0.0.1: The contents parameter is deprecated. Change to copyToRoot if the contents are designed to be copied to the root filesystem, such as when you use `buildEnv` or similar between contents and your packages. Use copyToRoot = buildEnv { ... }; or similar if you intend to add packages to /bin.
error (ignored): error: end of string reached
error: getting status of '/nix/store/Polykey': No such file or directory

I still need to look into this.

[tegefaulkes]

ETA on merge is hard to say. All that is left is to get CI passing but there are some odd bugs i'm dealing with.

The main one is that the peculiar crypto stuff is not working when using polykey as a library. I'm pretty sure it's a mismatch in dependency versions. I see two different versions for some peculiar modules so I'm trying to resolve that.

I still need to get the nix build working, there are some minor bugs there i'm investigating still.

The CI is failing with imports missing. Everything was working locally. I'm still looking into this as well.

Hopefully I can get it done today. But there are a lot of oddities with using Polykey as a library that is cropping up at this stage.

[tegefaulkes]

I think all of the above is resolved now. Fixed mostly by re-generating the package-lock and fixing some dependencies in Polykey.

[tegefaulkes]

I think this step can be merged. the next step is testing the staging and master step CI jobs. I may need to work from a staging PR for that. I think I can merge this PR for now and work in the merge to master PR that gets auto-generated.

[tegefaulkes]

Testing if pkg works right now.

Looks like pkg is trying to download the prebuilt node file and failing. Looking into this, these are meant to be provided by nix-shell.

It's trying to download version 18.5.0 while nix-shell is providing 18.15.0. I tried to update pkg/pkg-fetch to the version that would fetch 18.15.0 instead, but there is no released version that does this. So I'll have to modify the env to provide 18.5.0.

Note that our version of node is 18.15.0, not sure if this will cause problems with building but we'll see.

[tegefaulkes]

pkg is working now. There may still be runtime dependencies that are missing, should be seen in tests.

[tegefaulkes]

I'm merging this, further fixes will be done in the staging branch.