Unable to build the project due to the private pxt-arcade-sim dependency

[iaroslavn]

Describe the bug
As a developer external to Microsoft organization I am no longer able to build and run pxt-arcade due to the private npm dependency introduced in #684.

To Reproduce
Steps to reproduce the behavior:

1. I am following steps described in https://github.com/Microsoft/pxt-arcade#setup

	git clone https://github.com/Microsoft/pxt-arcade.git 
	cd pxt-arcade
	npm install

2. I got the error:

3573 verbose stack Error: exited with error code: 128
3573 verbose stack     at ChildProcess.<anonymous> (C:\Program Files\nodejs\node_modules\npm\node_modules\pacote\lib\util\finished.js:12:19)
3573 verbose stack     at ChildProcess.emit (events.js:182:13)
3573 verbose stack     at maybeClose (internal/child_process.js:962:16)
3573 verbose stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:251:5)
3574 verbose cwd C:\java\projects\pxt-arcade
3575 verbose Windows_NT 10.0.17134
3576 verbose argv "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "install"
3577 verbose node v10.15.0
3578 verbose npm  v6.4.1
3579 error Error while executing:
3579 error C:\Program Files\Git\mingw64\bin\git.EXE ls-remote -h -t https://github.com/Microsoft/pxt-arcade-sim.git
3579 error
3579 error remote: Invalid username or password.
3579 error fatal: Authentication failed for 'https://github.com/Microsoft/pxt-arcade-sim.git/'
3579 error
3579 error exited with error code: 128
3580 verbose exit [ 1, true ]

Expected behavior
I think npm install should work for people outside of Microsoft organization as well.
At least it was working prior to the changes introduced in #684.

Desktop (please complete the following information):

• OS: Windows 10
• Node v10.15.0
• npm 6.4.1
• Git Bash

Additional context
I think the root cause is that pxt-arcade-sim dependency mentioned in package.json resides in Microsoft private repository (https://github.com/Microsoft/pxt-arcade-sim).
To fix the issue please either publish it to the public NPM registry or make pxt-arcade-sim repository public.

P.S.
It would be really great if you could also publish pxt-arcade to NPM registry as well (e.g. how it was done for pxt-microbit), so 3rd party developers no longer need to build pxt-arcade locally prior to building their pxt game packages.

[abchatra]

Are you trying to build an extension? Can you try steps here: https://makecode.com/extensions/github-authoring ? This requires no installation.

[iaroslavn]

@abchatra
I am working on a simple arcade game.
The link you posted above is useful, thank you!
I was hoping to get some kind of build automation outside of https://arcade.makecode.com editor, merely to check if the code I push to my GitHub repo compiles.

I got this Travis config generated by the MakeCode website, but it fails on the pxt target arcade step link, that's why I had an intention to build pxt-arcade locally, but run into more troubles and opened this issue.

[iaroslavn]

Seems like the fix that made pxt-arcade-sim dependency optional fixed npm install, but now the website started with pxt serve doesn't work (probably the dependency is still required at run time?):

$ git clone https://github.com/Microsoft/pxt-arcade.git
$ cd pxt-arcade/
$ npm install
npm WARN deprecated [email protected]: Browserslist 2 could fail on reading Browserslist >3.0 config used in other tools.
npm WARN deprecated [email protected]: All versions below 4.0.1 of Nodemailer are deprecated. See https://nodemailer.com/status/
npm WARN deprecated [email protected]: CircularJSON is in maintenance only, flatted is its successor.
npm WARN deprecated [email protected]: This version is no longer maintained. Please upgrade to the latest version.
npm WARN deprecated [email protected]: Use uuid module instead
npm WARN deprecated [email protected]: This version is no longer maintained. Please upgrade to the latest version.
npm WARN deprecated [email protected]: This version is no longer maintained. Please upgrade to the latest version.
npm WARN deprecated [email protected]: This version is no longer maintained. Please upgrade to the latest version.
npm WARN deprecated [email protected]: If using 2.x branch, please upgrade to at least 2.1.6 to avoid a serious bug with socket data flow and an import issue introduced in 2.1.0
npm WARN deprecated [email protected]: This project is unmaintained
npm WARN deprecated [email protected]: This project is unmaintained
npm WARN deprecated [email protected]: stop using this version

> [email protected] install C:\Temp\pxt-arcade\node_modules\uws
> node-gyp rebuild > build_log.txt 2>&1 || exit 0

npm WARN optional SKIPPING OPTIONAL DEPENDENCY: pxt-arcade-sim@git+https://github.com/Microsoft/pxt-arcade-sim.git#8d3641ecec6d355d1f51029f4dae6c8a518dadd9 (node_modules\pxt-arcade-sim):
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: Error while executing:
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: C:\Program Files\Git\mingw64\bin\git.EXE ls-remote -h -t https://github.com/Microsoft/pxt-arcade-sim.git
npm WARN optional SKIPPING OPTIONAL DEPENDENCY:
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: remote: Invalid username or password.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fatal: Authentication failed for 'https://github.com/Microsoft/pxt-arcade-sim.git/'
npm WARN optional SKIPPING OPTIONAL DEPENDENCY:
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: exited with error code: 128
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules\fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"win32","arch":"x64"})

added 796 packages from 1418 contributors and audited 8649 packages in 39.669s
found 20 vulnerabilities (2 low, 18 moderate)
  run `npm audit fix` to fix them, or `npm audit` for details
$ pxt serve -p 3131
Using target arcade with build engine codal
  target: v C:\Temp\pxt-arcade
  pxt-core: v C:\Temp\pxt-arcade\node_modules\pxt-core
No sim/tsconfig.json nor sim/public/; assuming npm installed package
Starting server in C:\Temp\pxt-arcade
With pxt core at C:\Temp\pxt-arcade\node_modules\pxt-core
starting local ws server at 3233...
package "serialport" failed to load, run "pxt npminstallnative" to install native depencencies
---------------------------------------------

To launch the editor, open this URL:
http://localhost:3131/#local_token=5a137011-c9de-4ef4-7db1-7debe88304a5&wsport=3233

[abchatra]

We are working on a way to solve the local serve dependency. Meanwhile please let us know web based extension works fine

[iaroslavn]

I am able to use http://arcade.makecode.com/beta for my purposes, however if I am not able to use code from pxt-arcade public repository as pxt-serve inside pxt-arcade doesn't launch a working website due to the missing pxt-arcade-sim dependency (see the log output in my message above).

With this being said, the code in pxt-arcade public repo can't be run locally by developers outside of Microsoft organization (due to the private pxt-arcade-sim dependency). This was not the case before #684 changes were introduced and I was able to get pxt-arcade website running locally by following these steps:

1. https://github.com/Microsoft/pxt-arcade#setup
2. https://github.com/Microsoft/pxt-arcade#launching-the-server

To summarize, the problem described in this ticket only affects potential external users of the https://github.com/Microsoft/pxt-arcade repository.

[triptych]

I'm having the same issue:

npm install
npm ERR! Error while executing:
npm ERR! /usr/bin/git ls-remote -h -t https://github.com/Microsoft/pxt-arcade-sim.git
npm ERR!
npm ERR! remote: Repository not found.
npm ERR! fatal: repository 'https://github.com/Microsoft/pxt-arcade-sim.git/' not found
npm ERR!
npm ERR! exited with error code: 128

It's impossible to get this working without some microsoft internal dependency then this is a broken repo. Please fix this so that external users are able to build and run this.

[abchatra]

Try the latest bits. You don't need pxt-arcade-sim repository to build locally.

[triptych]

Sorry, I just tried this a few seconds ago:

git clone https://github.com/Microsoft/pxt-arcade
cd pxt-arcade
npm install
npm ERR! Error while executing:
npm ERR! /usr/bin/git ls-remote -h -t https://github.com/Microsoft/pxt-arcade-sim.git
npm ERR!
npm ERR! remote: Repository not found.
npm ERR! fatal: repository 'https://github.com/Microsoft/pxt-arcade-sim.git/' not found
npm ERR!
npm ERR! exited with error code: 128

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/awooldridge/.npm/_logs/2019-02-01T23_16_59_307Z-debug.log

[abchatra]

@guillaumejenkins This is an optional dependency right? Why is this erroring for users?

"optionalDependencies": {
"pxt-arcade-sim": "git+https://github.com/Microsoft/pxt-arcade-sim.git#v0.6.0"
},

[abchatra]

@triptych As a workaround, just remove the this dependency from package.json, until we fix it.

[abchatra]

#727 should fix this.

[riknoll]

@triptych or @iaroslavn could you try out the build with the latest commits when you get a chance? I removed the sim as a required dependency from the package-lock.json and it hopefully fixed the issue.

[triptych]

Sorry, I tried just now - I see your changes, however the error still happens:

[22:41][awooldridge@LM-SJC-11007915:~/MyHome/MyWork/pubgit]$ rm -rf pxt-arcade
[22:42][awooldridge@LM-SJC-11007915:~/MyHome/MyWork/pubgit]$ git clone https://github.com/Microsoft/pxt-arcade
Cloning into 'pxt-arcade'...
remote: Enumerating objects: 36, done.
remote: Counting objects: 100% (36/36), done.
remote: Compressing objects: 100% (30/30), done.
remote: Total 6380 (delta 11), reused 13 (delta 5), pack-reused 6344
Receiving objects: 100% (6380/6380), 46.27 MiB | 13.98 MiB/s, done.
Resolving deltas: 100% (3754/3754), done.
[22:42][awooldridge@LM-SJC-11007915:~/MyHome/MyWork/pubgit]$ cd pxt-arcade
[22:42][awooldridge@LM-SJC-11007915:~/MyHome/MyWork/pubgit/pxt-arcade(master)]$ npm install
npm ERR! Error while executing:
npm ERR! /usr/bin/git ls-remote -h -t https://github.com/Microsoft/pxt-arcade-sim.git
npm ERR!
npm ERR! remote: Repository not found.
npm ERR! fatal: repository 'https://github.com/Microsoft/pxt-arcade-sim.git/' not found
npm ERR!
npm ERR! exited with error code: 128

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/awooldridge/.npm/_logs/2019-02-02T06_43_00_433Z-debug.log

[triptych]

Using node v8.9.4

[iaroslavn]

I pulled changes form the latest revision in mainline, but I see the same issue - npm install gives the SKIPPING OPTIONAL DEPENDENCY: pxt-arcade-sim warning, but pxt serve will not open a working website.
It doesn't seem like the pxt-arcade-sim dependency is optional from the run-time perspective.

[guillaumejenkins]

@iaroslavn are there any errors in the terminal output, or inside the browser console?

[guillaumejenkins]

Ah, CLI is skipping build entirely - I'm investigating

In the mean time, you can unblock yourselves by creating a sim folder at the root of pxt-arcade, with the following files:

• foo.ts: an empty .ts file with any name
• tsconfig.json: a JSON file with the following content:

{
    "compilerOptions": {
        "target": "es5",
        "out": "../built/sim.js",
        "rootDir": "."
    }
}

[guillaumejenkins]

This will be fixed when we next bump the pxt-core dependency; for now the above workaround will unblock you

Sorry for the disruption!