More customizable build process for Docker images (#11176)

* Allows more customizations for image building.

This is the third (and not last) part of making the Production
image more corporate-environment friendly. It's been prepared
for the request of one of the big Airflow user (company) that
has rather strict security requirements when it comes to
preparing and building images. They are committed to
synchronizing with the progress of Apache Airflow 2.0 development
and making the image customizable so that they can build it using
only sources controlled by them internally was one of the important
requirements for them.

This change adds the possibilty of customizing various steps in
the build process:

* adding custom scripts to be run before installation of both
  build image and runtime image. This allows for example to
  add installing custom GPG keys, and adding custom sources.

* customizing the way NodeJS and Yarn are installed in the
  build image segment - as they might rely on their own way
  of installation.

* adding extra packages to be installed during both build and
  dev segment build steps. This is crucial to achieve the same
  size optimizations as the original image.

* defining additional environment variables (for example
  environment variables that indicate acceptance of the EULAs
  in case of installing proprietary packages that require
  EULA acceptance - both in the build image and runtime image
  (again the goal is to keep the image optimized for size)

The image build process remains the same when no customization
options are specified, but having those options increases
flexibility of the image build process in corporate environments.

This is part of #11171.

This change also fixes some of the issues opened and raised by
other users of the Dockerfile.

Fixes: #10730
Fixes: #10555
Fixes: #10856

Input from those issues has been taken into account when this
change was designed so that the cases described in those issues
could be implemented. Example from one of the issue landed as
an example way of building highly customized Airflow Image
using those customization options.

Depends on #11174

* Update IMAGES.rst

Co-authored-by: Kamil Breguła <[email protected]>
(cherry picked from commit ebd7150)

BREEZE.rst

@@ -542,7 +542,7 @@ or ``apache/airflow:v1-10-test-python3.6``).
 
 However in many cases you want to add your own custom version of the image - with added apt dependencies,
 python dependencies, additional Airflow extras. Breeze's ``build-image`` command helps to build your own,
-customised variant of the image that contains everything you need.
+customized variant of the image that contains everything you need.
 
 You can switch to building the production image by adding ``--production-image`` flag to the ``build_image``
 command. Note, that the images can also be build using ``docker build`` command by passing appropriate
@@ -566,8 +566,8 @@ This install additional pypi dependency - torchio in specified version.
 
 .. code-block:: bash
 
-     ./breeze build-image --production-image --additional-dev-deps "libasound2-dev" \
-        --additional-runtime-deps "libasound2"
+     ./breeze build-image --production-image --additional-dev-apt-deps "libasound2-dev" \
+        --additional-runtime-apt-deps "libasound2"
 
 This install additional apt dependencies - ``libasound2-dev`` in build image and ``libasound`` in the
 final image. Those are development dependencies that might be needed to build and use python packages added
@@ -580,7 +580,7 @@ suffix and they need to also be paired with corresponding runtime dependency add
 .. code-block:: bash
 
      ./breeze build-image --production-image --python 3.7 --additional-dev-deps "libasound2-dev" \
-        --additional-runtime-deps "libasound2"
+        --additional-runtime-apt-deps "libasound2"
 
 Same as above but uses python 3.7.
 
@@ -1197,6 +1197,8 @@ This is the current syntax for  `./breeze <./breeze>`_:
           images are pulled by default only for the first time you run the
           environment, later the locally build images are used as cache.
 
+  Customization options:
+
   -E, --extras EXTRAS
           Extras to pass to build images The default are different for CI and production images:
 
@@ -1210,20 +1212,49 @@ This is the current syntax for  `./breeze <./breeze>`_:
   --image-tag TAG
           Additional tag in the image.
 
+  --disable-pip-cache
+          Disables GitHub PIP cache during the build. Useful if github is not reachable during build.
+
   --additional-extras ADDITIONAL_EXTRAS
           Additional extras to pass to build images The default is no additional extras.
 
   --additional-python-deps ADDITIONAL_PYTHON_DEPS
           Additional python dependencies to use when building the images.
 
-  --additional-dev-deps ADDITIONAL_DEV_DEPS
+  --dev-apt-command DEV_APT_COMMAND
+          The basic command executed before dev apt deps are installed.
+
+  --additional-dev-apt-command ADDITIONAL_DEV_APT_COMMAND
+          Additional command executed before dev apt deps are installed.
+
+  --additional-dev-apt-deps ADDITIONAL_DEV_APT_DEPS
           Additional apt dev dependencies to use when building the images.
 
-  --additional-runtime-deps ADDITIONAL_RUNTIME_DEPS
+  --dev-apt-deps DEV_APT_DEPS
+          The basic apt dev dependencies to use when building the images.
+
+  --additional-dev-apt-deps ADDITIONAL_DEV_DEPS
+          Additional apt dev dependencies to use when building the images.
+
+  --additional-dev-apt-envs ADDITIONAL_DEV_APT_ENVS
+          Additional environment variables set when adding dev dependencies.
+
+  --runtime-apt-command RUNTIME_APT_COMMAND
+          The basic command executed before runtime apt deps are installed.
+
+  --additional-runtime-apt-command ADDITIONAL_RUNTIME_APT_COMMAND
+          Additional command executed before runtime apt deps are installed.
+
+  --runtime-apt-deps ADDITIONAL_RUNTIME_APT_DEPS
+          The basic apt runtime dependencies to use when building the images.
+
+  --additional-runtime-apt-deps ADDITIONAL_RUNTIME_DEPS
           Additional apt runtime dependencies to use when building the images.
 
-  --disable-pip-cache
-          Disables GitHub PIP cache during the build. Useful if github is not reachable during build.
+  --additional-runtime-apt-envs ADDITIONAL_RUNTIME_APT_DEPS
+          Additional environment variables set when adding runtime dependencies.
+
+  Build options:
 
   --disable-mysql-client-installation
           Disables installation of the mysql client which might be problematic if you are building
@@ -1654,6 +1685,8 @@ This is the current syntax for  `./breeze <./breeze>`_:
           images are pulled by default only for the first time you run the
           environment, later the locally build images are used as cache.
 
+  Customization options:
+
   -E, --extras EXTRAS
           Extras to pass to build images The default are different for CI and production images:
 
@@ -1667,20 +1700,49 @@ This is the current syntax for  `./breeze <./breeze>`_:
   --image-tag TAG
           Additional tag in the image.
 
+  --disable-pip-cache
+          Disables GitHub PIP cache during the build. Useful if github is not reachable during build.
+
   --additional-extras ADDITIONAL_EXTRAS
           Additional extras to pass to build images The default is no additional extras.
 
   --additional-python-deps ADDITIONAL_PYTHON_DEPS
           Additional python dependencies to use when building the images.
 
-  --additional-dev-deps ADDITIONAL_DEV_DEPS
+  --dev-apt-command DEV_APT_COMMAND
+          The basic command executed before dev apt deps are installed.
+
+  --additional-dev-apt-command ADDITIONAL_DEV_APT_COMMAND
+          Additional command executed before dev apt deps are installed.
+
+  --additional-dev-apt-deps ADDITIONAL_DEV_APT_DEPS
+          Additional apt dev dependencies to use when building the images.
+
+  --dev-apt-deps DEV_APT_DEPS
+          The basic apt dev dependencies to use when building the images.
+
+  --additional-dev-apt-deps ADDITIONAL_DEV_DEPS
           Additional apt dev dependencies to use when building the images.
 
-  --additional-runtime-deps ADDITIONAL_RUNTIME_DEPS
+  --additional-dev-apt-envs ADDITIONAL_DEV_APT_ENVS
+          Additional environment variables set when adding dev dependencies.
+
+  --runtime-apt-command RUNTIME_APT_COMMAND
+          The basic command executed before runtime apt deps are installed.
+
+  --additional-runtime-apt-command ADDITIONAL_RUNTIME_APT_COMMAND
+          Additional command executed before runtime apt deps are installed.
+
+  --runtime-apt-deps ADDITIONAL_RUNTIME_APT_DEPS
+          The basic apt runtime dependencies to use when building the images.
+
+  --additional-runtime-apt-deps ADDITIONAL_RUNTIME_DEPS
           Additional apt runtime dependencies to use when building the images.
 
-  --disable-pip-cache
-          Disables GitHub PIP cache during the build. Useful if github is not reachable during build.
+  --additional-runtime-apt-envs ADDITIONAL_RUNTIME_APT_DEPS
+          Additional environment variables set when adding runtime dependencies.
+
+  Build options:
 
   --disable-mysql-client-installation
           Disables installation of the mysql client which might be problematic if you are building
@@ -1974,6 +2036,8 @@ This is the current syntax for  `./breeze <./breeze>`_:
           images are pulled by default only for the first time you run the
           environment, later the locally build images are used as cache.
 
+  Customization options:
+
   -E, --extras EXTRAS
           Extras to pass to build images The default are different for CI and production images:
 
@@ -1987,20 +2051,49 @@ This is the current syntax for  `./breeze <./breeze>`_:
   --image-tag TAG
           Additional tag in the image.
 
+  --disable-pip-cache
+          Disables GitHub PIP cache during the build. Useful if github is not reachable during build.
+
   --additional-extras ADDITIONAL_EXTRAS
           Additional extras to pass to build images The default is no additional extras.
 
   --additional-python-deps ADDITIONAL_PYTHON_DEPS
           Additional python dependencies to use when building the images.
 
-  --additional-dev-deps ADDITIONAL_DEV_DEPS
+  --dev-apt-command DEV_APT_COMMAND
+          The basic command executed before dev apt deps are installed.
+
+  --additional-dev-apt-command ADDITIONAL_DEV_APT_COMMAND
+          Additional command executed before dev apt deps are installed.
+
+  --additional-dev-apt-deps ADDITIONAL_DEV_APT_DEPS
+          Additional apt dev dependencies to use when building the images.
+
+  --dev-apt-deps DEV_APT_DEPS
+          The basic apt dev dependencies to use when building the images.
+
+  --additional-dev-apt-deps ADDITIONAL_DEV_DEPS
           Additional apt dev dependencies to use when building the images.
 
-  --additional-runtime-deps ADDITIONAL_RUNTIME_DEPS
+  --additional-dev-apt-envs ADDITIONAL_DEV_APT_ENVS
+          Additional environment variables set when adding dev dependencies.
+
+  --runtime-apt-command RUNTIME_APT_COMMAND
+          The basic command executed before runtime apt deps are installed.
+
+  --additional-runtime-apt-command ADDITIONAL_RUNTIME_APT_COMMAND
+          Additional command executed before runtime apt deps are installed.
+
+  --runtime-apt-deps ADDITIONAL_RUNTIME_APT_DEPS
+          The basic apt runtime dependencies to use when building the images.
+
+  --additional-runtime-apt-deps ADDITIONAL_RUNTIME_DEPS
           Additional apt runtime dependencies to use when building the images.
 
-  --disable-pip-cache
-          Disables GitHub PIP cache during the build. Useful if github is not reachable during build.
+  --additional-runtime-apt-envs ADDITIONAL_RUNTIME_APT_DEPS
+          Additional environment variables set when adding runtime dependencies.
+
+  Build options:
 
   --disable-mysql-client-installation
           Disables installation of the mysql client which might be problematic if you are building