Create DLLs for the vendor modules

[mistic]

After dedicating some time understanding how we can kill/drastically reduce the optimizer, with the findings from this investigation available on #7322, we decided to get this step as the first one into the desired direction.

Tasks

• Collect stats from this new solution and compare them to the old one (results section below)
• Code review and refactor all the code
• Assure that we are handling styles properly with the DLLs
• Update webpack and the loaders to the last versions
• Run a fork porting the dynamic dll plugin to webpack 3 in order to understand the differences from using webpack 4 (result: the compilation with webpack 4 is faster).
• Parse server side code in order to identify the server only used dependencies and be able to not exclude them from the final bundle.
• Investigate if we can improve the caches (webpack cache, add more cache loaders, HardSourceWebpackPlugin)
• Investigate if we can benefit from running webpack loaders in parallel (happypack or webpack new parallelism option)
• Create a DLL Bundler module on the optimizer who runs the DLL config in another webpack running instance (possibly even in another process)
• Run the DLL Bundler, if needed, as a step of the current optimizer.
• Auto generate the modules list to bundle in the DLL during the generation process (we want to exclude server modules and others that throws errors).
• Automatically understands when we need to rebuild the DLL, while in dev, and trigger the rebuild.
• Use the DLL manifest file or emit a simpler and straightforward DLL compilation result file, with the modules bundled into the DLL, in order to delete those modules from the node_modules folder during our production build process.

Goals

• Improve initial and rebuild optimizations' performance both on development and production.
• Reduce the amount of files in the final build ( this will reduce the file count of the distributable decompressed and the size both compressed and decompressed).
• Improve development and user experience.

Results

The following results were obtained comparing a new build for kibana using the branch who implements the DLL work vs the current oficial release for kibana-6.4.0 when installing a simple plugin called notification_center in the distributable.

All the mentioned times are a reference obtained running kibana in the following hardware:

Computer: MacBookPro 14,3
OS: macOS 10.13.6
Processor: 2,9 GHz Intel Core i7
Memory: 16 GB 2133 MHz LPDDR3
Graphics Card (in use): Intel HD Graphics 630 1536 MB

The memory and file descriptors data were obtained using the following code (or the equivalent on webpack 4 in order to work into the dll branch):
(NOTE: the after-optimize-modules returns more than one time, so I'm using the absolute maximum found both for resident set mem usage and opened
file descriptors. This webpack hook seems quite good to measure those things because is called during the intermidiate steps of a compilation.)

this.compiler.plugin('compilation', compilation => {
  compilation.plugin('after-optimize-modules', modules => {
    // resident set mem usage
    const memoryUsage = process.memoryUsage();
    const rawResidentSetMemUsed = process.memoryUsage().rss / 1024 / 1024;
    const residentSetMemUsed = Math.round(rawResidentSetMemUsed * 100) / 100;
    process.stdout.write('residentSetMemUsed(MB): ' + residentSetMemUsed + '\n');

    // file descriptors count
    // used counters function from
    // the following node module: https://www.npmjs.com/package/lsof
    counters((data) => {
      process.stdout.write('fdsOpened: ' + data.open + '\n'); 
    });
  });
});

Stats (on distributable)	Without Dll	With Dll	Difference
Compile Time	114s	63s	-44.7%
Resident Set Memory Usage	874.29 MB	840.8 MB	-3.8%
File Descriptors	637	31	-95.1%
Compressed Size	201.4 MB	168.5 MB	-16.3%
Uncompressed Size	620.6 MB	442.8 MB	-28.6%
File Count	65355	44549	-31.8%

Note: This tasks are being developed in the mistic/kibana#optimizer-spike-single-process-v3 branch and have resulted in the following opened PR #22618.