Fixes a number of issues preventing Kepler from building on fresh checkout #2596

alex-bain · 2024-07-25T00:53:35Z

When I perform a fresh clone, a number of build issues prevent Kepler from starting successfully.

This collection of changes ensures that the application can be cloned / installed / built without error.

Testing

Clone this PR into a fresh installation
Run yarn bootstrap
Run yarn start

Everything should build successfully and the demo app should load without error.

May be related to #2594, #2593, and earlier reports.

netlify · 2024-07-25T00:53:52Z

✅ Deploy Preview for keplergl2 ready!

Name	Link
🔨 Latest commit	`e60abef`
🔍 Latest deploy log	https://app.netlify.com/sites/keplergl2/deploys/66b473794688980008c2ed5b
😎 Deploy Preview	https://deploy-preview-2596--keplergl2.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

echo292 · 2024-07-25T08:30:24Z

Ran the following in git-bash on Windows 10.
It breaks right when it tries to creates the subfolders a second time @kepler.gl/constants/dist

Run yarn
Run yarn start

Error results:

[5/5] Building fresh packages...
success Saved lockfile.
$ yarn fix-dependencies && yarn workspaces run stab && yarn workspaces run prepublish && yarn add-licence &&  yarn build:umd && yarn build:types
yarn run v1.22.19
$ ./scripts/fix-dependencies.sh
Done in 3.71s.
yarn workspaces v1.22.19

> @kepler.gl/types
yarn run v1.22.19
$
Done in 0.23s.

> @kepler.gl/constants
yarn run v1.22.19
$ mkdir -p dist && touch dist/index.js
A subdirectory or file -p already exists.
Error occurred while processing: -p.
A subdirectory or file dist already exists.
Error occurred while processing: dist.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed.
Exit code: 1
Command: H:\nodejs\node.exe
Arguments: C:\Users\foo\AppData\Roaming\npm\node_modules\yarn\lib\cli.js run stab
Directory: C:\dev\kepler.gl-build-fixes\src\constants
Output:

info Visit https://yarnpkg.com/en/docs/cli/workspaces for documentation about this command.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

echo292 · 2024-07-25T19:38:48Z

Ran in /examples/demo-app
yarn
yarn stat

yarn_output.txt is attached.
yarn_output.txt

alex-bain · 2024-07-25T19:46:01Z

The stab command isn't something I touched, but I can modify all instances to redirect errors to /dev/null to mask errors if that is preferable by the maintainers.

Example:

    "stab": "mkdir -p dist && touch dist/index.js >/dev/null 2>&1"

The errors you're seeing are the ones that should be resolved by this PR, I'm not able to reproduce those locally from this branch.

Can you tell me more about your environment?

echo292 · 2024-07-25T19:54:56Z

@alex-bain Hi Alex, thank you for the response. My first comment was based on running the demo from kepler.gl root. My bad!
I reviewed your commit and realized it applies to the examples/demo-app.

My second comment is from the /examples/demo-app.
I cloned your branch and ran yarn, then yarn start.

My environment is on Windows 10 with running the commands in git-bash..
I also have a linux workstation I can try as well. I will report back once I have the results.

alex-bain · 2024-07-25T20:35:17Z

@echo292 I've pushed up a fix that will silently consume the "directory already exists" error, which should get multiple invocations of npm run stab to complete without error.

echo292 · 2024-07-26T02:57:29Z

@alex-bain
I re-cloned, ran the build on both Windows 10 Pro 22H2 and Linux Ubuntu 20.04.2 LTS and received the same errors. See attached errors.
yarn_output.txt

I tried it with node versions (v18.18.2 and v22.3.0).

kepler.gl/examples/demo-app.
yarn
yarn start

Let me know if there's anything you want me to try different on my end.

henryreed · 2024-07-26T03:09:17Z

@alex-bain You're fantastic. With the latest commit, I am able to successfully compile Kepler.gl. My Dockerfile is as follows:

FROM node:18

COPY kepler.gl /kepler.gl

WORKDIR /kepler.gl

RUN apt update && apt full-upgrade -y
RUN yarn

CMD ["yarn", "start"]

Only thing I had to change was to add --host 0.0.0.0 to kepler.gl/examples/demo-app/package.json on line 5, like so:
"start-local": "webpack-dev-server --host 0.0.0.0 --mode development --env.es6 --progress --hot --open",

For whatever reason, webpack-dev-server listens on IPv6 only. Maybe it's a Docker quirk? Anyway, thanks again!

pablos1rvent · 2024-07-26T17:32:21Z

I'm beginning to believe that the developers of this started on mac/unix. Gonna try clean without branches on Debian and try again. In all branches and tags cross env is not applied, some env vars are just setted hard way like Unix, so I think it's not compatible to dev on Windows.

Maybe with some changes, it's possible, but I'm not gonna fix it due to my personal situation.

When trying to checkout and run Kepler a number of build issues prevent the application from starting successfully. This collection of changes ensures that the application can be cloned / installed / built without error. Signed-off-by: Alex Bain <[email protected]>

Signed-off-by: Alex Bain <[email protected]>

…ady-exists error Signed-off-by: Alex Bain <[email protected]>

…R feedback Signed-off-by: Alex Bain <[email protected]>

alex-bain · 2024-07-26T17:44:48Z

@echo292 I tried adding all of the Babel transform plugins to each babel.config.js and .babelrc file in the repository. That may help with your build issues, but since I cannot reproduce those failures locally I'm not certain if this will help. Please let me know if this resolves the issues for you?

Signed-off-by: Alex Bain <[email protected]>

pablos1rvent · 2024-07-26T17:56:13Z

@echo292 I tried adding all of the Babel transform plugins to each babel.config.js and .babelrc file in the repository. That may help with your build issues, but since I cannot reproduce those failures locally I'm not certain if this will help. Please let me know if this resolves the issues for you?

I can try.

Clone fresh, yarn and yarn start, any extra steps maybe on your branch? Thanks

echo292 · 2024-07-26T17:59:06Z

@alex-bain Thank you! I will give this a try now.

@psirdev - As far as I know, this branch commit is specific to the kepler.gl/examples/demo-app.

pablos1rvent · 2024-07-26T19:14:27Z

@alex-bain Thank you! I will give this a try now.

@psirdev - As far as I know, this branch commit is specific to the kepler.gl/examples/demo-app.

Yes, I'm on the same point. Clean fresh, yarn, yarn start and everything crashes trying the demo app.

Tried Windows 10, Debian, branches that I tried: master, alex-bain:build-fixes, igr/fix-yarn-start...

Your commit also doesn't work on Windows. Gonna try again on Debian...

pablos1rvent · 2024-07-26T19:44:01Z

Yikes, failing... What can we do? Maybe switch to another commit that works?

* Adds missing babel transform deps to package.jsons * Resolves dep version mismatches contributing to build failures * Adds ts-ignore statements to workaround TS failures after making these changes Signed-off-by: Alex Bain <[email protected]>

pablos1rvent · 2024-07-26T20:39:29Z

Hello @alex-bain , I'm on the Slack, if you wanna join and try to resolve with me (pair programming), I'm here all day D:

I tried some old branches and it happens, maybe is some NPM dependencies (react, redux or babel) that broke all the code

P.D: It happened with other repositories too that are not affiliated with kepler

Signed-off-by: Alex Bain <[email protected]>

alex-bain · 2024-07-26T21:09:19Z

@psirdev - Thank you. I believe the latest commit should resolve the issues that @echo292 reported, however the scope of this PR has increased as I was tracking down and trying to resolve those errors.

pablos1rvent · 2024-07-26T21:11:02Z

@psirdev - Thank you. I believe the latest commit should resolve the issues that @echo292 reported, however the scope of this PR has increased as I was tracking down and trying to resolve those errors.

Thanks for everything. It looks like the pipeline is working. I tried to install on Windows and it shows:

But I think it's something on my own. Tried to delete those folders, change the stab scripts...

Gonna try on Debian and I will tell you in a couple minutes. Thanks.

pablos1rvent · 2024-07-26T21:20:05Z

Looks like mkdir -p (stab script on every folder in src) on unix works like a charm, but in Git Bash of Windows it fails due to yarn executing like CMD style.

Maybe It's because I have WSL subsystem. Gonna try Cygwin or another bash and try again.

EDIT: it fails too with Cygwin. Gonna change workaround and see if I can use another command.

echo292 · 2024-07-26T21:58:47Z

@psirdev
I also successfully got it working on Ubuntu 20.04.2 LTS.
Window returned the same @loader.gl errors in Git Bash on my end.
Are you running from the kepler.gl root or examples/demo-app?

pablos1rvent · 2024-07-26T22:08:54Z

@psirdev I also successfully got it working on Ubuntu 20.04.2 LTS. Window returned the same @loader.gl errors in Git Bash on my end. Are you running from the kepler.gl root or examples/demo-app?

root, if you look the yarn start it redirects to the example, and it loads from the node_modules from the root folder

I'm slow because I had problems with Virtualbox and switched everything to Hyper-V, so I'm gonna confirm in 10 mins at least

pablos1rvent · 2024-07-26T22:24:35Z

@echo292 @alex-bain I have a fix for the stab script (for cross compatibility on Windows & Unix) on every folder on src:

 "scripts": {
    // bla bla bla... another scripts...
    "stab": "bash -c 'mkdir -p dist && touch dist/index.js'"
  },

But.... It fails later resolving the ts (it's empty) dist folder:


> @kepler.gl/utils
yarn run v1.22.22
$ babel-node ../../scripts/license-header/bin --license ../../FILE-HEADER && yarn build && yarn build:types
$ rm -fr dist && babel src --out-dir dist --source-maps inline --extensions '.ts,.tsx,.js,.jsx' --ignore '**/*.d.ts'
Successfully compiled 0 files with Babel (8ms).
$ tsc --project ./tsconfig.production.json
Done in 4.85s.

> @kepler.gl/styles
yarn run v1.22.22
$ babel-node ../../scripts/license-header/bin --license ../../FILE-HEADER && yarn build && yarn build:umd && yarn build:types
$ rm -fr dist && babel src --out-dir dist --source-maps inline --extensions '.ts,.tsx,.js,.jsx' --ignore '**/*.d.ts'
Successfully compiled 0 files with Babel (4ms).
$ cross-env NODE_OPTIONS=--openssl-legacy-provider webpack --config ./webpack/umd.js --progress --env.prod
Hash: 102f7846386cdaca8fc9
Version: webpack 4.47.0
Time: 1224ms
Built at: 26/07/2024 23:31:13
 1 asset
Entrypoint KeplerGl = keplergl.min.js
[3] external {"commonjs":"styled-components","commonjs2":"styled-components","amd":"styled-components","root":"styled"} 42 bytes {0} [built]   
[4] ./src/index.ts 784 bytes {0} [built]
[5] ./src/base.ts 78.9 KiB {0} [built]
[9] ./src/media-breakpoints.ts 2.54 KiB {0} [built]
    + 6 hidden modules

ERROR in ./src/base.ts
Module not found: Error: Can't resolve '@kepler.gl/constants' in 'C:\Users\TERMINUS\Desktop\fix2\kepler.gl\src\styles\src'
 @ ./src/base.ts 14:17-48
 @ ./src/index.ts
error Command failed with exit code 2.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 2.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed.

So I give up, throw the keyboard and tried on Debian. And it works!!!

Just thinking Windows is not supported or maybe nobody programs on windows anymore lol

echo292 · 2024-07-28T22:50:32Z

@alex-bain @psirdev
I had Kepler.gl 3.0 demo working in windows in December and I can't remember how I got it to work. Since then I corrupted the demo folder.
I do have a working version of the /custom-reducer in windows. If I recall it was specific to how I ran the build maybe.

I attached the yarn.lock.txt hoping this can provide some clues.

yarn.lock.txt

echo292 · 2024-07-29T02:12:22Z

@alex-bain @psirdev

Got the demo to work in WSL/VS Code in Windows 10.

wsl --install in the powershell
sudo apt update && sudo apt upgrade
sudo apt-get install curl
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/master/install.sh | bash
nvm install --lts
I switched to node 18.18.2. --> npm use 18.18.2
In VS Code install the WSL extension
Switch the terminal to WSL Ubuntu
cd to the /demo-app folder
yarn
yarn start

Signed-off-by: Alex Bain <[email protected]>

alex-bain · 2024-07-30T16:10:55Z

I've reverted the >/dev/null 2>&1 change since that did not fix the problem with the stab script.

I'll also call out I had to add a number of @ts-ignore statements to deal with TS related build issues, but it appears the repo is now cleanly building.

echo292 · 2024-08-01T02:43:45Z

@alex-bain Thank you! Your latest commit worked on the first try.
This branch works on WSL 2 / windows 10, however, it's incredibly slow to build.

heshan0131 · 2024-08-06T22:43:44Z

examples/webpack.config.local.js

+                /node_modules\/@loaders\.gl/,
+                /node_modules\/@probe\.gl/,
+                /node_modules\/@math\.gl/,
+                /node_modules\/@kepler\.gl/


I don't think you need to include @kepler.gl here, the babel loaders of kepler.gl src is handled by makeBabelRule

examples/webpack.config.local.js

alex-bain added 4 commits July 26, 2024 10:42

Adds a @ts-ignore to resolve a build failure related to an unknown type

133b58e

Signed-off-by: Alex Bain <[email protected]>

Running 'stab' multiple times will no longer produce a directory-alre…

6069c82

…ady-exists error Signed-off-by: Alex Bain <[email protected]>

Updating all babel configs to apply the latest transforms, based on P…

4e85ef9

…R feedback Signed-off-by: Alex Bain <[email protected]>

alex-bain force-pushed the build-fixes branch from 04c4a33 to 4e85ef9 Compare July 26, 2024 17:43

Adds missing dependencies to all package.json files

25e39db

Signed-off-by: Alex Bain <[email protected]>

Another attempt at resolving build issues

791777f

* Adds missing babel transform deps to package.jsons * Resolves dep version mismatches contributing to build failures * Adds ts-ignore statements to workaround TS failures after making these changes Signed-off-by: Alex Bain <[email protected]>

Few small fixes that were uncovered in PR

4d73bcb

Signed-off-by: Alex Bain <[email protected]>

Reverts dev/null redirect for stab command

332e796

Signed-off-by: Alex Bain <[email protected]>

alex-bain force-pushed the build-fixes branch from 2d11c3f to 332e796 Compare July 30, 2024 16:09

chrisgervang requested review from ibgreen and igorDykhta August 1, 2024 17:29

ibgreen requested a review from heshan0131 August 1, 2024 17:57

heshan0131 reviewed Aug 6, 2024

View reviewed changes

heshan0131 approved these changes Aug 6, 2024

View reviewed changes

heshan0131 reviewed Aug 7, 2024

View reviewed changes

examples/webpack.config.local.js Outdated Show resolved Hide resolved

heshan0131 reviewed Aug 7, 2024

View reviewed changes

examples/webpack.config.local.js Outdated Show resolved Hide resolved

heshan0131 added 2 commits August 8, 2024 00:27

Update examples/webpack.config.local.js

89b3ef5

Update examples/webpack.config.local.js

e60abef

heshan0131 merged commit e5111da into keplergl:master Aug 8, 2024
9 of 10 checks passed

alex-bain deleted the build-fixes branch August 12, 2024 16:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixes a number of issues preventing Kepler from building on fresh checkout #2596

Fixes a number of issues preventing Kepler from building on fresh checkout #2596

alex-bain commented Jul 25, 2024 •

edited

Loading

netlify bot commented Jul 25, 2024 •

edited

Loading

echo292 commented Jul 25, 2024

echo292 commented Jul 25, 2024

alex-bain commented Jul 25, 2024

echo292 commented Jul 25, 2024

alex-bain commented Jul 25, 2024

echo292 commented Jul 26, 2024

henryreed commented Jul 26, 2024

pablos1rvent commented Jul 26, 2024

alex-bain commented Jul 26, 2024

pablos1rvent commented Jul 26, 2024 •

edited

Loading

echo292 commented Jul 26, 2024

pablos1rvent commented Jul 26, 2024 •

edited

Loading

pablos1rvent commented Jul 26, 2024

pablos1rvent commented Jul 26, 2024 •

edited

Loading

alex-bain commented Jul 26, 2024

pablos1rvent commented Jul 26, 2024 •

edited

Loading

pablos1rvent commented Jul 26, 2024 •

edited

Loading

echo292 commented Jul 26, 2024

pablos1rvent commented Jul 26, 2024 •

edited

Loading

pablos1rvent commented Jul 26, 2024

echo292 commented Jul 28, 2024

echo292 commented Jul 29, 2024

alex-bain commented Jul 30, 2024

echo292 commented Aug 1, 2024

heshan0131 Aug 6, 2024

Fixes a number of issues preventing Kepler from building on fresh checkout #2596

Fixes a number of issues preventing Kepler from building on fresh checkout #2596

Conversation

alex-bain commented Jul 25, 2024 • edited Loading

Testing

netlify bot commented Jul 25, 2024 • edited Loading

✅ Deploy Preview for keplergl2 ready!

echo292 commented Jul 25, 2024

echo292 commented Jul 25, 2024

alex-bain commented Jul 25, 2024

echo292 commented Jul 25, 2024

alex-bain commented Jul 25, 2024

echo292 commented Jul 26, 2024

henryreed commented Jul 26, 2024

pablos1rvent commented Jul 26, 2024

alex-bain commented Jul 26, 2024

pablos1rvent commented Jul 26, 2024 • edited Loading

echo292 commented Jul 26, 2024

pablos1rvent commented Jul 26, 2024 • edited Loading

pablos1rvent commented Jul 26, 2024

pablos1rvent commented Jul 26, 2024 • edited Loading

alex-bain commented Jul 26, 2024

pablos1rvent commented Jul 26, 2024 • edited Loading

pablos1rvent commented Jul 26, 2024 • edited Loading

echo292 commented Jul 26, 2024

pablos1rvent commented Jul 26, 2024 • edited Loading

pablos1rvent commented Jul 26, 2024

echo292 commented Jul 28, 2024

echo292 commented Jul 29, 2024

alex-bain commented Jul 30, 2024

echo292 commented Aug 1, 2024

heshan0131 Aug 6, 2024

Choose a reason for hiding this comment

alex-bain commented Jul 25, 2024 •

edited

Loading

netlify bot commented Jul 25, 2024 •

edited

Loading

pablos1rvent commented Jul 26, 2024 •

edited

Loading

pablos1rvent commented Jul 26, 2024 •

edited

Loading

pablos1rvent commented Jul 26, 2024 •

edited

Loading

pablos1rvent commented Jul 26, 2024 •

edited

Loading

pablos1rvent commented Jul 26, 2024 •

edited

Loading

pablos1rvent commented Jul 26, 2024 •

edited

Loading