[POC] Validation of zip packages using javascript

[jsoriano]

What does this PR do?

Validates packages in zip format from JavaScript, using WebAssembly.

Why is it important?

To use the same validation code in Go-based tooling and JavaScript components as Fleet.

Checklist

• I have added test packages to test/packages that prove my change is effective.
• I have added an entry in versions/N/changelog.yml.

Related issues

[elasticmachine]

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-02-22T15:59:24.186+0000

• Duration: 2 min 25 sec

Steps errors

Expand to view the steps failures

Check

• Took 0 min 6 sec . View more details here
• Description: make check

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[joshdover]

Thanks for experimenting with this! One question I have is whether or not it would be possible to validate a package without having to save a zip file on disk. Ideally we could pass in a Buffer of bytes for the zip file contents. We can then stream these directly from the EPR response or from an uploaded package.

This could also be used as the basis for supporting development of bundled packages directly in the Kibana repo if we could easily access the ability to build a package from a directory. This could be done by using the CLI directly as well in build steps, but it'd be nice to only have way of interacting with elastic-package from our JS code in Kibana. cc @kpollich

In terms of packaging and consuming from the Kibana repository, it'd be ideal if elastic-package distributed a @elastic/package NPM module that contained the built wasm and JS interface for accessing it in Node.js code. Would this be feasible to accommodate within the CI pipeline for elastic-package?

[jsoriano]

One question I have is whether or not it would be possible to validate a package without having to save a zip file on disk. Ideally we could pass in a Buffer of bytes for the zip file contents. We can then stream these directly from the EPR response or from an uploaded package.

There are some options for this, the main challenge would be to see how we can plug something like a buffer of bytes in JS to Go. But for example the zip package in the Go stdlib can read zip files from stream readers, and in Go it is possible to create readers from arrays of bytes. It should be possible to pass an array of bytes from JS to Go.

This could also be used as the basis for supporting development of bundled packages directly in the Kibana repo if we could easily access the ability to build a package from a directory. This could be done by using the CLI directly as well in build steps, but it'd be nice to only have way of interacting with elastic-package from our JS code in Kibana.

Do you expect some use case where this would be needed apart of in build time? In build and development time I guess that it is ok to use the elastic-package binary.

In terms of packaging and consuming from the Kibana repository, it'd be ideal if elastic-package distributed a @elastic/package NPM module that contained the built wasm and JS interface for accessing it in Node.js code. Would this be feasible to accommodate within the CI pipeline for elastic-package?

I have tried to build elastic-package for wasm, and it fails for some dependency in golang.org/x/sys/unix, this would need more investigation.

[joshdover]

Do you expect some use case where this would be needed apart of in build time? In build and development time I guess that it is ok to use the elastic-package binary.

Right this would be a build-time step, so using the elastic-package binary directly might be fine. Thought it might be simpler from a maintenance perspective if we had a single WASM target/NPM package that would support both use cases, but happy to explore using the binary directly.

[jsoriano]

@joshdover I have added to the POC package validation using a Uint8Array as input. This allows to do client-side validation, I have also added a simple web server to show this.

[jsoriano]

This doesn't work.

[joshdover]

@joshdover I have added to the POC package validation using a Uint8Array as input. This allows to do client-side validation, I have also added a simple web server to show this.

Very nice, that should make consuming this from Kibana very easy. Note that we won't be able to start using this validation until #23 is resolved since the Endpoint package has a transform and index template that aren't valid in the spec currently.

[joshdover]

I got this working in Kibana, but it required a small adjustment to the interface to be able to use this from Node.js.

Here's the adjustments to this PR: jsoriano#1
Here's the POC PR for using this build from Kibana: elastic/kibana#128076

[jsoriano]

Great, thanks @joshdover for giving it a try :)

We would have to think about how to package this functionality for node, and what functionality to expose. Do you think it can be a problem to include wasm in Kibana?

[joshdover]

We would have to think about how to package this functionality for node, and what functionality to expose. Do you think it can be a problem to include wasm in Kibana?

Got this preliminary answer, and expecting @azasypkin to chime in as well. No immediate concerns for server-side usage, but client-side may need some research, but I don't think it's necessary: elastic/kibana#128076 (comment)

[jsoriano]

We had a discussion about this topic in an internal meeting of the Fleet team, some points we discussed:

• As result of the package validation, it would be nice to have structured errors, or a json, instead of the current string.
• Current wasm is about 12MB in size, check if there are ways to reduce it (there are some tips in the docs).
• Exporting through globals may not be such a problem, but we need confirmation from Kibana team. We may give another though to explore options to expose the functionality.
• Package as npm module.
• Keeping the go instance running instead of starting and stopping it on every call may be ok.
• Expose information from packages, like for example the data stream information.

[jsoriano]

If we manage to build with tinygo we may solve some of the problems mentioned above: smaller artifact, local exports, no need to keep the go instance running.