Adds browser (WASM) support #34

gregnr · 2023-11-03T07:26:01Z

Adds WASM support using the incredible emnapi project that implements Node's N-API for Emscripten. This means no new bindings needed to be built - we can reuse the existing Node-API bindings and simply target WASM.

Some implementation notes:

Build method

I've added a new script yarn build:wasm which uses a Makefile under the hood. emnapi experimentally supports node-gyp, but node-gyp doesn't yet support static library linking when cross-compiling. The Makefile also gives us a cleaner way to load and build the libpg_query source.
yarn build:wasm runs via Docker to make the build process easier for devs who don't have emsdk installed.
I needed to patch one line of code in libpg_query's Makefile to get libpg_query.a to build properly for Emscripten, hence the reference to my fork at https://github.com/gregnr/libpg_query.git. Once that PR is merged I will link back to the original repo. This is now merged.

Output format

I opted for Emscripten's SINGLE_FILE=1 flag for now which bundles the .wasm output into the accompanying .js file as base64. The .js file ends up just under 2MB, but this makes NPM distribution and bundling much simpler (otherwise devs will need to add special rules to their web bundler configs to import the .wasm file properly). I tested serving the .js file over a web server that uses standard gzip compression, and the compressed file size went down to ~450Kb, which is more approachable.
Instead of SINGLE_FILE=1, we now use EXPORT_ES6=1 which outputs the .js and .wasm separately, where the .js file is built as an ES Module that includes a import.meta.url reference to the WASM file, as noted below by @RReverser. Bundlers like webpack have first class support for import.meta.url, meaning no extra configuration is required to load the .wasm file.

Async only + single threaded for now

I only included async methods in the web export since the module itself also needs to be loaded asynchronously. We can add the sync methods back, but this will require devs to await some sort of init() command before they can use them.
The WASM module is compiled without pthread and uses emnapi's emnapi-basic lib. This means that asynchronous logic (ie. AsyncWorker, etc) isn't truly multithreaded - it's just mocked in the main thread. It is possible to enable pthread which is implemented as WebWorkers under the hood, but this method requires SharedArrayBuffer which has much stricter security requirements and makes deploying much more difficult/nuanced. If we are still interested in executing WASM in a different thread, I suggest we manually wrap the current output in WebWorkers and manage the back-and-forth communication ourselves without using SharedArrayBuffer.

Consumption

I've added conditional exports to package.json which should tell each runtime/bundler which entry point to use - meaning Node.js will continue to use the native NAPI bindings and web bundlers will use the WASM build. ~~I haven't yet tested publishing this to NPM and using it, but plan to shortly.~~ This has now been tested when imported as a published NPM module: @gregnr/[email protected]. Works in both Node (uses native NAPI) and browser (uses WASM).

Webpack POC

I added a webpack demo project under ./test as a means to test the lib in the browser.

RReverser · 2023-11-03T11:52:22Z

(otherwise devs will need to add special rules to their web bundler configs to import the .wasm file properly

Ideally they shouldn't, but you need to emit .mjs not .js (or set -sEXPORT_ES6). I implemented bundler-friendly static import patterns in Emscripten a while back.

Although it depends on the bundler and whether they support new URL(..., import.meta.url) scheme for assets. Webpack does out of the box, Rollup has plugins for that, other bundlers might need some tweaks.

Either way, if possible, I'd really recommend to avoid SINGLE_FILE and keep Wasm separate as otherwise you're losing on compiled code caching optimisation, which only works for Wasm retrieved via its own fetch request: https://v8.dev/blog/wasm-code-caching#stream

gregnr · 2023-11-03T13:38:47Z

This is great to know @RReverser - I'll look into updating this.

gregnr · 2023-11-03T20:07:29Z

This has now been updated as an ES Module via EXPORT_ES6=1 and removes SINGLE_FILE=1 so the .js and .wasm output as separate files. Confirmed it is working out of the box with no extra configuration on webpack 5.

gregnr · 2023-11-06T05:43:42Z

@RReverser by chance, have you ever gotten Emscripten outputs to work seamlessly with Next.js? So far it has been a bit challenging:

Next.js can execute the same React code on both server (SSR) and browser, so ideally I'd like to get the lib to work in both situations with no extra configuration from the end user

We can use package.json conditional exports to target WASM (-s ENVIRONMENT=web) for browser and native NAPI for node:

{
  "exports": {
    ".": {
      "types": "./index.d.ts",
      "browser": "./wasm/index.js",
      "node": "./index.js",
      "default": "./index.js"
    }
  }
}

This works, but since Next.js runs all code through webpack (targeting both browser and server), the end user needs to install something like nextjs-node-loader to tell webpack how to deal with *.node binaries during the server build:

const nextConfig = {
  webpack: (config, { dev, isServer, webpack, nextRuntime }) => {
    config.module.rules.push({
      test: /\.node$/,
      use: [
        {
          loader: 'nextjs-node-loader',
        },
      ],
    })
    return config
  },
}

Otherwise Webpack (understandably) fails because it doesn't know how to handle *.node binaries. The above works, and it's not the end of the world, but it's no longer turn-key / configure-less.

Alternatively we could try to just use WASM for both client & server by omitting -s ENVIRONMENT=web (or set to -s ENVIRONMENT=web,node). Unfortunately I could never get this to work on either browser or server because:
- Browser: Emscripten adds a bunch of logic to the .js output that detects which environment it's in, and in the process includes a const { createRequire } = await import('module') which Next.js doesn't like during its tracing/static analysis: module not found: can't resolve 'module'
- Server: Somewhere in Next.js' process of bundling the server code, new URL(..., import.meta.url) ends up as /_next/static/media/libpg-query.a03578ad.wasm which is correct for the browser but not the server: ENOENT: no such file or directory, open '/_next/static/media/libpg-query.a03578ad.wasm'

As of now my best solution for Next.js is the one above requiring developers to install nextjs-node-loader if they want it working server side.

RReverser · 2023-11-06T13:13:07Z

Hm no I didn't do anything Next.js-specific, I'm afraid you'll need to play with it yourself.

2. Otherwise Webpack (understandably) fails because it doesn't know how to handle *.node binaries. The above works, and it's not the end of the world, but it's no longer turn-key / configure-less.

If it's about the file extension, you can just change it to .js when targeting Wasm. In particular, one trick I used in my initial porting was to make Emscripten's output end in .node.js. That way, when Node code does require('.../some-native-addon.node') it will either find a native module (which actually ends in .node) or the Wasm one (which ends in .node.js, but Node implicitly adds .js and finds it anyway).

I forgot to mention one other thing, but it might help here too - instead of custom Makefile, you can integrate Wasm building into binding.gyp. See here how I did that in Sharp / feel free to borrow the common.gypi until we find a way to upstream it to emnapi (the blocker was that it won't work on Windows yet): https://github.com/lovell/sharp/pull/3522/files#diff-61fe5790e62b9d1af82b5332536fb4bdc899528b095c22f43f9f10068e91eeb7

karlhorky · 2023-12-08T13:59:05Z

@gregnr I saw that libpg_query is being used in the Supabase PostgreSQL language server, nice!

https://twitter.com/kiwicopple/status/1733096971949298003

How much more work is needed on this PR? (and approximately how hard is each thing?)

We at @upleveled would love to test SafeQL by @Newbie012 with our next cohort of bootcamp students (some of whom use Windows) in mid-February 2024, and I think that this PR would offer us a chance to do so!

If there's an early version that @Eprince-hub or other people on our team can try out, we'd love to give it a shot, even before this PR is merged :)

karlhorky · 2023-12-17T09:08:39Z

Elsewhere in the ecosystem, another WebAssembly implementation of libpg_query by @anuraaga:

Wasm-based codepath to run without cgo pganalyze/pg_query_go#103

ivasilov · 2024-02-16T13:37:48Z

@pyramation @karlhorky We've been using the WASM library at Supabase in a forked package for a while and would like to merge this PR. Are there any outstanding issues (beside the merge conflicts) that needs to be resolved?

karlhorky · 2024-02-16T19:26:49Z

@ivasilov does that mean that @gregnr is done with this PR now? From the last comments it sounded like there was still something open to be resolved...

gregnr · 2024-02-20T19:22:08Z

@karlhorky This PR is good to go now. The challenges above with Next.js are solvable using nextjs-node-loader so shouldn't prevent this PR from merging.

I've updated the base branch to v15 since the WASM build in the PR has been intended for v15. If supporting WASM in v13 and v14 is important, I can dig into these, though with v13 specifically I had encountered a rabbit hole of issues when targeting WASM.

I couldn't find any build/deploy CI automation in this repo so didn't set anything up there on the WASM side (manual wasm build + NPM deploy for now). Let me know if there's a better way to do this.

pyramation · 2024-02-21T03:39:15Z

Thank you for this amazing work :)

oops, I deleted master and a few others in an attempt to have more clarity around branches, keep similar naming convention as libqg-query and it automatically closed this PR — re-opening :)

pyramation · 2024-02-21T04:46:16Z

OK, I've published an rc version [email protected] — in case the wasm changes any behavior, devs can remove the -wasm.0.1 from the version

Once we verify on a few machines this is good, sounds like we can merge the wasm-feature branch into 15-latest and publish 15.0.4 or later

npm notice === Tarball Details ===
npm notice name:          libpg-query
npm notice version:       15.0.3-wasm.0.1
npm notice filename:      libpg-query-15.0.3-wasm.0.1.tgz
npm notice package size:  8.0 MB
npm notice unpacked size: 28.7 MB
npm notice shasum:        ee416b6ed5ce6c3b146364d75e0d013466c59242
npm notice integrity:     sha512-FWCvNK7IDdh8H[...]QHv5FwLz24fdQ==
npm notice total files:   23


+ [email protected]

julessre · 2024-02-21T08:39:34Z

I tried to install this on a Windows machine today using pnpm add --save-dev [email protected] and received the following failure:

$ pnpm add --save-dev @ts-safeql/eslint-plugin [email protected]
Downloading registry.npmjs.org/libpg-query/15.0.3-wasm.0.1: 7,97 MB/7,97 MB, done
 WARN  1 deprecated subdependencies found: @npmcli/[email protected]
Packages: +95
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Progress: resolved 856, reused 730, downloaded 56, added 95, done
node_modules/.pnpm/[email protected]/node_modules/esbuild: Running postinstall script, done in 1.3s
node_modules/.pnpm/[email protected]/node_modules/libpg-query: Running install script, failed in 32.7s
.../node_modules/libpg-query install$ node-pre-gyp install --fallback-to-build
│ node-pre-gyp info it worked if it ends with ok
│ node-pre-gyp info using [email protected]
│ node-pre-gyp info using [email protected] | win32 | x64
│ node-pre-gyp info check checked for "C:\Users\HUAWEI\projects\nextjs-ecommerce-store\node_modules\.pnpm\[email protected]\node_modules\li…  
│ node-pre-gyp http GET https://supabase-public-artifacts-bucket.s3.amazonaws.com/libpg-query-node/queryparser-v15.0.3-wasm.0.1-node-v115-win32-x64.t…  
│ node-pre-gyp ERR! install response status 404 Not Found on https://supabase-public-artifacts-bucket.s3.amazonaws.com/libpg-query-node/queryparser-v…  
│ node-pre-gyp WARN Pre-built binaries not installable for [email protected] and [email protected] (node-v115 ABI, unknown) (falling back to sou…  
│ node-pre-gyp WARN Hit error response status 404 Not Found on https://supabase-public-artifacts-bucket.s3.amazonaws.com/libpg-query-node/queryparser…  
│ gyp info it worked if it ends with ok
│ gyp info using [email protected]
│ gyp info using [email protected] | win32 | x64
│ gyp info ok
│ gyp info it worked if it ends with ok
│ gyp info using [email protected]
│ gyp info using [email protected] | win32 | x64
│ gyp info find Python using Python version 3.12.1 found at "C:\Python312\python.exe"
│ gyp http GET https://nodejs.org/download/release/v20.11.1/node-v20.11.1-headers.tar.gz
│ gyp http 200 https://nodejs.org/download/release/v20.11.1/node-v20.11.1-headers.tar.gz
│ gyp http GET https://nodejs.org/download/release/v20.11.1/SHASUMS256.txt
│ gyp http GET https://nodejs.org/download/release/v20.11.1/win-x64/node.lib
│ gyp http 200 https://nodejs.org/download/release/v20.11.1/SHASUMS256.txt
│ gyp http 200 https://nodejs.org/download/release/v20.11.1/win-x64/node.lib
│ gyp info find VS using VS2017 (15.9.34118.181) found at:
│ gyp info find VS "C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools"
│ gyp info find VS run with --verbose for detailed information
│ gyp info spawn C:\Python312\python.exe
│ gyp info spawn args [
│ gyp info spawn args   'C:\\Users\\HUAWEI\\AppData\\Local\\node\\corepack\\pnpm\\8.15.3\\dist\\node_modules\\node-gyp\\gyp\\gyp_main.py',
│ gyp info spawn args   'binding.gyp',
│ gyp info spawn args   '-f',
│ gyp info spawn args   'msvs',
│ gyp info spawn args   '-I',
│ gyp info spawn args   'C:\\Users\\HUAWEI\\projects\\nextjs-ecommerce-store\\node_modules\\.pnpm\\[email protected]\\node_modules\\libpg-q…  
│ gyp info spawn args   '-I',
│ gyp info spawn args   'C:\\Users\\HUAWEI\\AppData\\Local\\node\\corepack\\pnpm\\8.15.3\\dist\\node_modules\\node-gyp\\addon.gypi',
│ gyp info spawn args   '-I',
│ gyp info spawn args   'C:\\Users\\HUAWEI\\AppData\\Local\\node-gyp\\Cache\\20.11.1\\include\\node\\common.gypi',
│ gyp info spawn args   '-Dlibrary=shared_library',
│ gyp info spawn args   '-Dvisibility=default',
│ gyp info spawn args   '-Dnode_root_dir=C:\\Users\\HUAWEI\\AppData\\Local\\node-gyp\\Cache\\20.11.1',
│ gyp info spawn args   '-Dnode_gyp_dir=C:\\Users\\HUAWEI\\AppData\\Local\\node\\corepack\\pnpm\\8.15.3\\dist\\node_modules\\node-gyp',
│ gyp info spawn args   '-Dnode_lib_file=C:\\\\Users\\\\HUAWEI\\\\AppData\\\\Local\\\\node-gyp\\\\Cache\\\\20.11.1\\\\<(target_arch)\\\\node.lib',      
│ gyp info spawn args   '-Dmodule_root_dir=C:\\Users\\HUAWEI\\projects\\nextjs-ecommerce-store\\node_modules\\.pnpm\\[email protected]\\nod…  
│ gyp info spawn args   '-Dnode_engine=v8',
│ gyp info spawn args   '--depth=.',
│ gyp info spawn args   '--no-parallel',
│ gyp info spawn args   '--generator-output',
│ gyp info spawn args   'C:\\Users\\HUAWEI\\projects\\nextjs-ecommerce-store\\node_modules\\.pnpm\\[email protected]\\node_modules\\libpg-q…  
│ gyp info spawn args   '-Goutput_dir=.'
│ gyp info spawn args ]
│ gyp info ok
│ gyp info it worked if it ends with ok
│ gyp info using [email protected]
│ gyp info using [email protected] | win32 | x64
│ gyp info spawn C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\MSBuild\15.0\Bin\MSBuild.exe
│ gyp info spawn args [
│ gyp info spawn args   'build/binding.sln',
│ gyp info spawn args   '/clp:Verbosity=minimal',
│ gyp info spawn args   '/nologo',
│ gyp info spawn args   '/p:Configuration=Release;Platform=x64'
│ gyp info spawn args ]
│ Die Projekte in dieser Projektmappe werden nacheinander erstellt. Um eine parallele Erstellung zu ermöglichen, müssen Sie den Schalter "/m" hinzufü…  
│   nothing.c
│   win_delay_load_hook.cc
│   nothing.vcxproj -> C:\Users\HUAWEI\projects\nextjs-ecommerce-store\node_modules\.pnpm\[email protected]\node_modules\libpg-query\build\…  
│   prebuild_dependencies
│   Das System kann den angegebenen Pfad nicht finden.
│ C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\Common7\IDE\VC\VCTargets\Microsoft.CppCommon.targets(209,5): error MSB6006: "cmd.exe…  
│ gyp ERR! build error
│ gyp ERR! stack Error: `C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\MSBuild\15.0\Bin\MSBuild.exe` failed with exit code: 1
│ gyp ERR! stack     at ChildProcess.onExit (C:\Users\HUAWEI\AppData\Local\node\corepack\pnpm\8.15.3\dist\node_modules\node-gyp\lib\build.js:203:23)    
│ gyp ERR! stack     at ChildProcess.emit (node:events:518:28)
│ gyp ERR! stack     at ChildProcess._handle.onexit (node:internal/child_process:294:12)
│ gyp ERR! System Windows_NT 10.0.22631
│ gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Users\\HUAWEI\\AppData\\Local\\node\\corepack\\pnpm\\8.15.3\\dist\\node_modules\\node-g…  
│ gyp ERR! cwd C:\Users\HUAWEI\projects\nextjs-ecommerce-store\node_modules\.pnpm\[email protected]\node_modules\libpg-query
│ gyp ERR! node -v v20.11.1
│ gyp ERR! node-gyp -v v9.4.1
│ gyp ERR! not ok
│ node-pre-gyp ERR! build error
│ node-pre-gyp ERR! stack Error: Failed to execute 'C:\Program Files\nodejs\node.exe C:\Users\HUAWEI\AppData\Local\node\corepack\pnpm\8.15.3\dist\nod…  
│ node-pre-gyp ERR! stack     at ChildProcess.<anonymous> (C:\Users\HUAWEI\projects\nextjs-ecommerce-store\node_modules\.pnpm\@mapbox+node-pre-gyp@1.…  
│ node-pre-gyp ERR! stack     at ChildProcess.emit (node:events:518:28)
│ node-pre-gyp ERR! stack     at maybeClose (node:internal/child_process:1105:16)
│ Failed to execute 'C:\Program Files\nodejs\node.exe C:\Users\HUAWEI\AppData\Local\node\corepack\pnpm\8.15.3\dist\node_modules\node-gyp\bin\node-gyp…  
│ node-pre-gyp ERR! stack     at ChildProcess._handle.onexit (node:internal/child_process:305:5)
│ node-pre-gyp ERR! System Windows_NT 10.0.22631
│ node-pre-gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Users\\HUAWEI\\projects\\nextjs-ecommerce-store\\node_modules\\.pnpm\\@mapbox+…  
│ node-pre-gyp ERR! cwd C:\Users\HUAWEI\projects\nextjs-ecommerce-store\node_modules\.pnpm\[email protected]\node_modules\libpg-query
│ node-pre-gyp ERR! node -v v20.11.1
│ node-pre-gyp ERR! node-pre-gyp -v v1.0.11
│ node-pre-gyp ERR! not ok
└─ Failed in 32.7s at C:\Users\HUAWEI\projects\nextjs-ecommerce-store\node_modules\.pnpm\[email protected]\node_modules\libpg-query
 ELIFECYCLE  Command failed with exit code 1.

karlhorky · 2024-02-21T10:35:42Z

With the npm latest version [email protected] it also also still compiles the native version on Linux:

$ pnpm add --save-dev [email protected]
...
.../node_modules/libpg-query install$ node-pre-gyp install --fallback-to-build
.../node_modules/libpg-query install: node-pre-gyp info it worked if it ends with ok
.../node_modules/libpg-query install: node-pre-gyp info using [email protected]
.../node_modules/libpg-query install: node-pre-gyp info using [email protected] | linux | x64
.../node_modules/libpg-query install: node-pre-gyp info check checked for "/home/runner/work/next-js-example-winter-2024-atvie/next-js-example-winter-2024-atvie/node_modules/.pnpm/[email protected]/node_modules/libpg-query/build/Release/queryparser.node" (not found)
.../node_modules/libpg-query install: node-pre-gyp http GET https://supabase-public-artifacts-bucket.s3.amazonaws.com/libpg-query-node/queryparser-v15.0.3-wasm.0.1-node-v115-linux-x64.tar.gz
.../node_modules/libpg-query install: node-pre-gyp ERR! install response status 404 Not Found on https://supabase-public-artifacts-bucket.s3.amazonaws.com/libpg-query-node/queryparser-v15.0.3-wasm.0.1-node-v115-linux-x64.tar.gz 
.../node_modules/libpg-query install: node-pre-gyp WARN Pre-built binaries not installable for [email protected] and [email protected] (node-v115 ABI, glibc) (falling back to source compile with node-gyp) 
.../node_modules/libpg-query install: node-pre-gyp WARN Hit error response status 404 Not Found on https://supabase-public-artifacts-bucket.s3.amazonaws.com/libpg-query-node/queryparser-v15.0.3-wasm.0.1-node-v115-linux-x64.tar.gz 
.../node_modules/libpg-query install: gyp info it worked if it ends with ok
.../node_modules/libpg-query install: gyp info using [email protected]
.../node_modules/libpg-query install: gyp info using [email protected] | linux | x64
.../node_modules/libpg-query install: gyp info ok 
.../node_modules/libpg-query install: gyp info it worked if it ends with ok
.../node_modules/libpg-query install: gyp info using [email protected]
.../node_modules/libpg-query install: gyp info using [email protected] | linux | x64
.../node_modules/libpg-query install: gyp info find Python using Python version 3.10.12 found at "/usr/bin/python3"
.../node_modules/libpg-query install: gyp http GET https://nodejs.org/download/release/v20.11.1/node-v20.11.1-headers.tar.gz
.../node_modules/libpg-query install: gyp http 200 https://nodejs.org/download/release/v20.11.1/node-v20.11.1-headers.tar.gz
.../node_modules/libpg-query install: gyp http GET https://nodejs.org/download/release/v20.11.1/SHASUMS256.txt
.../node_modules/libpg-query install: gyp http 200 https://nodejs.org/download/release/v20.11.1/SHASUMS256.txt
.../node_modules/libpg-query install: gyp info spawn /usr/bin/python3
.../node_modules/libpg-query install: gyp info spawn args [
.../node_modules/libpg-query install: gyp info spawn args   '/home/runner/setup-pnpm/node_modules/.pnpm/[email protected]/node_modules/pnpm/dist/node_modules/node-gyp/gyp/gyp_main.py',
.../node_modules/libpg-query install: gyp info spawn args   'binding.gyp',
.../node_modules/libpg-query install: gyp info spawn args   '-f',
.../node_modules/libpg-query install: gyp info spawn args   'make',
.../node_modules/libpg-query install: gyp info spawn args   '-I',
.../node_modules/libpg-query install: gyp info spawn args   '/home/runner/work/next-js-example-winter-2024-atvie/next-js-example-winter-2024-atvie/node_modules/.pnpm/[email protected]/node_modules/libpg-query/build/config.gypi',
.../node_modules/libpg-query install: gyp info spawn args   '-I',
.../node_modules/libpg-query install: gyp info spawn args   '/home/runner/setup-pnpm/node_modules/.pnpm/[email protected]/node_modules/pnpm/dist/node_modules/node-gyp/addon.gypi',
.../node_modules/libpg-query install: gyp info spawn args   '-I',
.../node_modules/libpg-query install: gyp info spawn args   '/home/runner/.cache/node-gyp/20.11.1/include/node/common.gypi',
.../node_modules/libpg-query install: gyp info spawn args   '-Dlibrary=shared_library',
.../node_modules/libpg-query install: gyp info spawn args   '-Dvisibility=default',
.../node_modules/libpg-query install: gyp info spawn args   '-Dnode_root_dir=/home/runner/.cache/node-gyp/20.11.1',
.../node_modules/libpg-query install: gyp info spawn args   '-Dnode_gyp_dir=/home/runner/setup-pnpm/node_modules/.pnpm/[email protected]/node_modules/pnpm/dist/node_modules/node-gyp',
.../node_modules/libpg-query install: gyp info spawn args   '-Dnode_lib_file=/home/runner/.cache/node-gyp/20.11.1/<(target_arch)/node.lib',
.../node_modules/libpg-query install: gyp info spawn args   '-Dmodule_root_dir=/home/runner/work/next-js-example-winter-2024-atvie/next-js-example-winter-2024-atvie/node_modules/.pnpm/[email protected]/node_modules/libpg-query',
.../node_modules/libpg-query install: gyp info spawn args   '-Dnode_engine=v8',
.../node_modules/libpg-query install: gyp info spawn args   '--depth=.',
.../node_modules/libpg-query install: gyp info spawn args   '--no-parallel',
.../node_modules/libpg-query install: gyp info spawn args   '--generator-output',
.../node_modules/libpg-query install: gyp info spawn args   'build',
.../node_modules/libpg-query install: gyp info spawn args   '-Goutput_dir=.'
.../node_modules/libpg-query install: gyp info spawn args ]
.../node_modules/libpg-query install: gyp info ok 
.../node_modules/libpg-query install: gyp info it worked if it ends with ok
.../node_modules/libpg-query install: gyp info using [email protected]
.../node_modules/libpg-query install: gyp info using [email protected] | linux | x64
.../node_modules/libpg-query install: gyp info spawn make
.../node_modules/libpg-query install: gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build' ]
.../node_modules/libpg-query install: make: Entering directory '/home/runner/work/next-js-example-winter-2024-atvie/next-js-example-winter-2024-atvie/node_modules/.pnpm/[email protected]/node_modules/libpg-query/build'
.../node_modules/libpg-query install:   CC(target) Release/obj.target/nothing/../../../[email protected]/node_modules/node-addon-api/nothing.o
.../node_modules/libpg-query install: rm -f Release/obj.target/../../../[email protected]/node_modules/node-addon-api/nothing.a Release/obj.target/../../../[email protected]/node_modules/node-addon-api/nothing.a.ar-file-list; mkdir -p `dirname Release/obj.target/../../../[email protected]/node_modules/node-addon-api/nothing.a`
.../node_modules/libpg-query install: ar crs Release/obj.target/../../../[email protected]/node_modules/node-addon-api/nothing.a @Release/obj.target/../../../[email protected]/node_modules/node-addon-api/nothing.a.ar-file-list
.../node_modules/libpg-query install:   COPY Release/nothing.a
.../node_modules/libpg-query install:   ACTION binding_gyp_queryparser_target_prebuild_dependencies libpg_query/include/pg_query.h
.../node_modules/libpg-query install: Cloning into 'libpg_query'...
.../node_modules/libpg-query install: Note: switching to 'db39825bc7c1ddd45962ec6a626d740b7f8f027a'.
.../node_modules/libpg-query install: You are in 'detached HEAD' state. You can look around, make experimental
.../node_modules/libpg-query install: changes and commit them, and you can discard any commits you make in this
.../node_modules/libpg-query install: state without impacting any branches by switching back to a branch.
.../node_modules/libpg-query install: If you want to create a new branch to retain commits you create, you may
.../node_modules/libpg-query install: do so (now or later) by using -c with the switch command. Example:
.../node_modules/libpg-query install:   git switch -c <new-branch-name>
.../node_modules/libpg-query install: Or undo this operation with:
.../node_modules/libpg-query install:   git switch -
.../node_modules/libpg-query install: Turn off this advice by setting config variable advice.detachedHead to false
.../node_modules/libpg-query install: HEAD is now at db39825 Update changelog, prepare `15-4.2.4` release (#227)
.../node_modules/libpg-query install: make[1]: Entering directory '/tmp/tmp.CL1Ce5Osfp/libpg_query'
.../node_modules/libpg-query install: compiling src/pg_query.c
.../node_modules/libpg-query install: compiling src/pg_query_deparse.c
.../node_modules/libpg-query install: compiling src/pg_query_fingerprint.c
.../node_modules/libpg-query install: compiling src/pg_query_json_plpgsql.c
.../node_modules/libpg-query install: compiling src/pg_query_normalize.c
.../node_modules/libpg-query install: compiling src/pg_query_outfuncs_json.c
.../node_modules/libpg-query install: compiling src/pg_query_outfuncs_protobuf.c
.../node_modules/libpg-query install: compiling src/pg_query_parse.c
.../node_modules/libpg-query install: compiling src/pg_query_parse_plpgsql.c
.../node_modules/libpg-query install: compiling src/pg_query_readfuncs_protobuf.c
.../node_modules/libpg-query install: Warning: protoc-gen-c not found, skipping protocol buffer regeneration
.../node_modules/libpg-query install: compiling src/pg_query_scan.c
.../node_modules/libpg-query install: compiling src/pg_query_split.c
.../node_modules/libpg-query install: compiling src/postgres_deparse.c
.../node_modules/libpg-query install: compiling src/postgres/src_backend_catalog_namespace.c
.../node_modules/libpg-query install: compiling src/postgres/src_backend_catalog_pg_proc.c
.../node_modules/libpg-query install: compiling src/postgres/src_backend_commands_define.c
.../node_modules/libpg-query install: compiling src/postgres/src_backend_nodes_bitmapset.c
.../node_modules/libpg-query install: compiling src/postgres/src_backend_nodes_copyfuncs.c
.../node_modules/libpg-query install: compiling src/postgres/src_backend_nodes_equalfuncs.c
.../node_modules/libpg-query install: compiling src/postgres/src_backend_nodes_extensible.c
.../node_modules/libpg-query install: compiling src/postgres/src_backend_nodes_list.c
.../node_modules/libpg-query install: compiling src/postgres/src_backend_nodes_makefuncs.c
.../node_modules/libpg-query install: compiling src/postgres/src_backend_nodes_nodeFuncs.c
.../node_modules/libpg-query install: compiling src/postgres/src_backend_nodes_value.c
.../node_modules/libpg-query install: compiling src/postgres/src_backend_parser_gram.c
.../node_modules/libpg-query install: compiling src/postgres/src_backend_parser_parser.c
.../node_modules/libpg-query install: compiling src/postgres/src_backend_parser_scan.c
.../node_modules/libpg-query install: compiling src/postgres/src_backend_parser_scansup.c
.../node_modules/libpg-query install: compiling src/postgres/src_backend_postmaster_postmaster.c
.../node_modules/libpg-query install: compiling src/postgres/src_backend_storage_ipc_ipc.c
.../node_modules/libpg-query install: compiling src/postgres/src_backend_storage_lmgr_s_lock.c
.../node_modules/libpg-query install: compiling src/postgres/src_backend_tcop_postgres.c
.../node_modules/libpg-query install: compiling src/postgres/src_backend_utils_activity_pgstat_database.c
.../node_modules/libpg-query install: compiling src/postgres/src_backend_utils_adt_datum.c
.../node_modules/libpg-query install: compiling src/postgres/src_backend_utils_adt_expandeddatum.c
.../node_modules/libpg-query install: compiling src/postgres/src_backend_utils_adt_format_type.c
.../node_modules/libpg-query install: compiling src/postgres/src_backend_utils_adt_ruleutils.c
.../node_modules/libpg-query install: compiling src/postgres/src_backend_utils_error_assert.c
.../node_modules/libpg-query install: compiling src/postgres/src_backend_utils_error_elog.c
.../node_modules/libpg-query install: compiling src/postgres/src_backend_utils_fmgr_fmgr.c
.../node_modules/libpg-query install: compiling src/postgres/src_backend_utils_hash_dynahash.c
.../node_modules/libpg-query install: compiling src/postgres/src_backend_utils_init_globals.c
.../node_modules/libpg-query install: compiling src/postgres/src_backend_utils_mb_mbutils.c
.../node_modules/libpg-query install: compiling src/postgres/src_backend_utils_misc_guc.c
.../node_modules/libpg-query install: compiling src/postgres/src_backend_utils_mmgr_aset.c
.../node_modules/libpg-query install: compiling src/postgres/src_backend_utils_mmgr_mcxt.c
.../node_modules/libpg-query install: compiling src/postgres/src_common_encnames.c
.../node_modules/libpg-query install: compiling src/postgres/src_common_hashfn.c
.../node_modules/libpg-query install: compiling src/postgres/src_common_keywords.c
.../node_modules/libpg-query install: compiling src/postgres/src_common_kwlookup.c
.../node_modules/libpg-query install: compiling src/postgres/src_common_pg_prng.c
.../node_modules/libpg-query install: compiling src/postgres/src_common_psprintf.c
.../node_modules/libpg-query install: compiling src/postgres/src_common_string.c
.../node_modules/libpg-query install: compiling src/postgres/src_common_stringinfo.c
.../node_modules/libpg-query install: compiling src/postgres/src_common_wchar.c
.../node_modules/libpg-query install: compiling src/postgres/src_pl_plpgsql_src_pl_comp.c
.../node_modules/libpg-query install: compiling src/postgres/src_pl_plpgsql_src_pl_funcs.c
.../node_modules/libpg-query install: compiling src/postgres/src_pl_plpgsql_src_pl_gram.c
.../node_modules/libpg-query install: compiling src/postgres/src_pl_plpgsql_src_pl_handler.c
.../node_modules/libpg-query install: compiling src/postgres/src_pl_plpgsql_src_pl_scanner.c
.../node_modules/libpg-query install: compiling src/postgres/src_port_pg_bitutils.c
.../node_modules/libpg-query install: compiling src/postgres/src_port_pgsleep.c
.../node_modules/libpg-query install: compiling src/postgres/src_port_pgstrcasecmp.c
.../node_modules/libpg-query install: compiling src/postgres/src_port_qsort.c
.../node_modules/libpg-query install: compiling src/postgres/src_port_snprintf.c
.../node_modules/libpg-query install: compiling src/postgres/src_port_strerror.c
.../node_modules/libpg-query install: compiling src/postgres/src_port_strnlen.c
.../node_modules/libpg-query install: compiling vendor/protobuf-c/protobuf-c.c
.../node_modules/libpg-query install: compiling vendor/xxhash/xxhash.c
.../node_modules/libpg-query install: Warning: protoc-gen-c not found, skipping protocol buffer regeneration
.../node_modules/libpg-query install: compiling protobuf/pg_query.pb-c.c
.../node_modules/libpg-query install: ar: creating libpg_query.a
.../node_modules/libpg-query install: make[1]: Leaving directory '/tmp/tmp.CL1Ce5Osfp/libpg_query'
.../node_modules/libpg-query install:   CXX(target) Release/obj.target/queryparser/src/addon.o
.../node_modules/libpg-query install:   CXX(target) Release/obj.target/queryparser/src/helpers.o
.../node_modules/libpg-query install:   CXX(target) Release/obj.target/queryparser/src/sync.o
.../node_modules/libpg-query install:   CXX(target) Release/obj.target/queryparser/src/async.o
.../node_modules/libpg-query install:   SOLINK_MODULE(target) Release/obj.target/queryparser.node
.../node_modules/libpg-query install:   COPY Release/queryparser.node
.../node_modules/libpg-query install: make: Leaving directory '/home/runner/work/next-js-example-winter-2024-atvie/next-js-example-winter-2024-atvie/node_modules/.pnpm/[email protected]/node_modules/libpg-query/build'
.../node_modules/libpg-query install: gyp info ok 
.../node_modules/libpg-query install: node-pre-gyp info ok 
.../node_modules/libpg-query install: Done
...
devDependencies:
...
+ libpg-query 15.0.3-wasm.0.1

ivasilov · 2024-02-21T10:38:50Z

The package is not published onto Supabase S3 bucket for some reason. You need to run npm run binary:publish at the end, right? Are you missing the proper credentials?

pyramation · 2024-02-22T19:47:05Z

ok, so sounds like this works for linux, but not windows yet, and we need to get the binary publish

gregnr mentioned this pull request Nov 3, 2023

Window support (depends on llbpg_query) #22

Closed

This was referenced Nov 3, 2023

Windows Support pganalyze/libpg_query#44

Closed

Enable SafeQL on Windows upleveled/eslint-config-upleveled#297

Closed

SafeQL installation fails on Windows ts-safeql/safeql#80

Closed

karlhorky mentioned this pull request Nov 11, 2023

Failed to npm install - gyp Error supabase/postgres-meta#387

Open

karlhorky mentioned this pull request Dec 17, 2023

Wasm-based codepath to run without cgo pganalyze/pg_query_go#103

Open

gregnr added 11 commits February 20, 2024 11:57

feat: wasm support using emnapi + emscripten

7eaac50

feat: wasm webpack test

7597744

chore(wasm): better source caching and make commands

dc5904a

feat(makefile): support multiple platform-arch targets

97f2f63

feat(wasm): convert to esm + separate wasm file

f6ce4e2

fix(wasm): yarn clean scripts

b810a52

chore: remove packageManager from package.json

9639d9b

fix: published files and conditional exports

fec1ae2

fix: add .node extension to native import

26fd7c9

fix: support c++ exceptions

649d850

feat: better makefile dependency management

8b5a22e

gregnr force-pushed the feat/wasm branch from 674e983 to 8b5a22e Compare February 20, 2024 19:12

gregnr changed the base branch from master to v15 February 20, 2024 19:13

feat(wasm): lock libpg_query to specific version tag

d8038de

gregnr added 2 commits February 20, 2024 17:56

refactor(wasm): makefile build/clean commands

256f406

refactor(wasm): makefile command location

0f42732

pyramation deleted the branch launchql:wasm-feature February 21, 2024 03:09

pyramation closed this Feb 21, 2024

pyramation reopened this Feb 21, 2024

pyramation changed the base branch from v15 to wasm-feature February 21, 2024 03:50

pyramation merged commit 1c8617a into launchql:wasm-feature Feb 21, 2024

karlhorky mentioned this pull request Apr 6, 2024

Upgrade (or remove) node-addon-api? #47

Closed

gregnr mentioned this pull request May 21, 2024

chore: upgrade libpg-query supabase/supabase#26618

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds browser (WASM) support #34

Adds browser (WASM) support #34

gregnr commented Nov 3, 2023 •

edited

Loading

RReverser commented Nov 3, 2023 •

edited

Loading

gregnr commented Nov 3, 2023

gregnr commented Nov 3, 2023 •

edited

Loading

gregnr commented Nov 6, 2023 •

edited

Loading

RReverser commented Nov 6, 2023

karlhorky commented Dec 8, 2023

karlhorky commented Dec 17, 2023

ivasilov commented Feb 16, 2024

karlhorky commented Feb 16, 2024

gregnr commented Feb 20, 2024

pyramation commented Feb 21, 2024 •

edited

Loading

pyramation commented Feb 21, 2024

julessre commented Feb 21, 2024 •

edited

Loading

karlhorky commented Feb 21, 2024 •

edited

Loading

ivasilov commented Feb 21, 2024

pyramation commented Feb 22, 2024

Adds browser (WASM) support #34

Adds browser (WASM) support #34

Conversation

gregnr commented Nov 3, 2023 • edited Loading

Build method

Output format

Async only + single threaded for now

Consumption

Webpack POC

RReverser commented Nov 3, 2023 • edited Loading

gregnr commented Nov 3, 2023

gregnr commented Nov 3, 2023 • edited Loading

gregnr commented Nov 6, 2023 • edited Loading

RReverser commented Nov 6, 2023

karlhorky commented Dec 8, 2023

karlhorky commented Dec 17, 2023

ivasilov commented Feb 16, 2024

karlhorky commented Feb 16, 2024

gregnr commented Feb 20, 2024

pyramation commented Feb 21, 2024 • edited Loading

pyramation commented Feb 21, 2024

julessre commented Feb 21, 2024 • edited Loading

karlhorky commented Feb 21, 2024 • edited Loading

ivasilov commented Feb 21, 2024

pyramation commented Feb 22, 2024

gregnr commented Nov 3, 2023 •

edited

Loading

RReverser commented Nov 3, 2023 •

edited

Loading

gregnr commented Nov 3, 2023 •

edited

Loading

gregnr commented Nov 6, 2023 •

edited

Loading

pyramation commented Feb 21, 2024 •

edited

Loading

julessre commented Feb 21, 2024 •

edited

Loading

karlhorky commented Feb 21, 2024 •

edited

Loading