Download manifest seems to not be cached, which leads to failures when builds are run in parallel

[CosmoMyzrailGorynych]

Issue Type

• Bug Report
• Feature Request
• Other

Current/Missing Behaviour

Running several nwbuilder calls in parallel sometimes fails with this error:

The version manifest does not exist/was not downloaded. Please try again in some time.
TypeError: Cannot read properties of undefined (reading 'components')
    at bld (file:///builds/CoMiGo/ct-js/node_modules/nw-builder/src/bld.js:170:35)
    at async nwbuild (file:///builds/CoMiGo/ct-js/node_modules/nw-builder/src/index.js:116:7)
    at async Promise.all (index 1)
    at async bakePackages (file:///builds/CoMiGo/ct-js/gulpfile.mjs:504:5)

I do run the first build separately so the manifest can be downloaded without overwrites:

/**
 * Array of tuples with platform — arch — itch.io channel name in each element.
 * Note how win32 platform is written as just 'win' (that's how nw.js binaries are released).
 */
let platforms = [
    ['linux', 'ia32', 'linux32'],
    ['linux', 'x64', 'linux64'],
    ['osx', 'x64', 'osx64'],
    ['osx', 'arm64', 'osxarm'],
    ['win', 'ia32', 'win32'],
    ['win', 'x64', 'win64']
];
if (process.platform === 'win32') {
    platforms = platforms.filter(p => p[0] !== 'osx');
    console.warn('⚠️  Building packages for MacOS is not supported on Windows. This platform will be skipped.');
}
const nwBuilderOptions = {
    version: nwVersion,
    flavor: 'sdk',
    srcDir: './app/',
    glob: false
};
/* ... */
export const bakePackages = async () => {
    /* ... */
    const builder = pf => {
        const [platform, arch, itchChannel] = pf;
        log.info(`'bakePackages': Building for ${platform}-${arch}…`);
        return nwBuilder({
            ...nwBuilderOptions,
            mode: 'build',
            platform,
            arch,
            outDir: `./build/ctjs - v${pack.version}/${itchChannel}`,
            zip: false
        });
    };
    // Run first build separately so it fetches manifest.json with all nw.js versions
    // without occasional rewrites and damage. ⬅️ wishful thinking here 😢 
    await builder(platforms[0]);
    await Promise.all(platforms.slice(1).map(builder));
    /* ... */
}

From what happens, I guess the downloads manifest is re-downloaded on each nwbuilder run.

Expected/Proposed Behaviour

It should be possible to run builds in parallel.

Additional Info

It would also be cool if there was a separate call to fetch download manifest only, like mode: 'fetch'

• Package version: 4.7.4
• Operating System: custom node:lts-bullseye + Wine container, Windows 11
• Node version: v20.13.1
• NW.js version: 0.87.0
• Logs from a sample failed pipeline: https://gitlab.com/CoMiGo/ct-js/-/jobs/6850357327#L1904

[ayushmanchhabra]

Will look into it. Usually afk during the week but will try to work on the weekend.

[ayushmanchhabra]

On every build, the versions manifest (nwjs.io/versions.json) is redownloaded (in the event it is updated in between. The error is most probably because two build processes are trying to write to the same location.

Possible solution is to cache versions manifest and remove it after some time has passed (set a timeout for 5 minutes for example). One edge case that comes to mind: if cache is invalidated but no network connection present, then it should use the existing cache and not delete it.

If you're running parallel builds, then both functions should atleast have the same timeout.