Prepare a base image for backend to speed up current process of building application for deployment #246

placek · 2024-02-21T11:08:38Z

As a Backend Developer,

I want to prepare a base image that includes all necessary pre-compiled dependencies,

So that the build process is expedited by only compiling the project-specific code on demand.

Summary:

Prepare and maintain a base Docker image that includes all the pre-compiled dependencies required by the application.

This will significantly reduce the build and deployment time, allowing for quicker iterations and deployments.

Acceptance Criteria:

Repo has been created to store the base image.
A base Dockerfile is created that includes all the project's dependencies, pre-compiled.
The backend application's Dockerfile is updated to use this base image.
The build process should verify that only the application-specific code is compiled during the build process, leveraging the pre-compiled dependencies in the base image.
Deployment times are measured before and after the implementation, showing a significant reduction in build and deployment time.

Introduces a new Dockerfile.base within the govtool/backend directory, aimed at creating a base image for the Haskell backend project. This base image includes all necessary pre-compiled dependencies, following the initiative to expedite the build process for deployment. The Dockerfile is based on the `haskell:9.2-buster` image and sets up an environment optimized for the GovTool project by pre-installing all dependencies. By removing the source code after dependency installation, we ensure that the image only contains what's necessary for future builds, thus aligning with our goal to reduce build and deployment times significantly. This change addresses part of the acceptance criteria for speeding up the deployment process by preparing and maintaining a base image with all pre-compiled dependencies required by the application.

Enhances the infrastructure by adding a new AWS ECR repository specifically for storing the base Docker image. This addition is crucial for the initiative to speed up the backend build and deployment process by utilizing a pre-compiled dependencies image. Modifications include updates to the `infra/terraform/modules/ecr/main.tf` to create a new ECR repository named with the suffix `-base` to differentiate it from application-specific repositories. The repository is configured with KMS encryption for enhanced security. Corresponding outputs for the base repository URL are added to `infra/terraform/modules/ecr/outputs.tf`, ensuring easy access and integration with CI/CD pipelines. This update directly supports the acceptance criteria for the project by ensuring a dedicated storage location for the base image, facilitating quicker iterations and deployments through the use of pre-compiled dependencies.

This commit introduces a new Terraform module instantiation for creating an AWS ECR repository dedicated to the backend base Docker image. The repository, named `backend-base`, is aimed at storing pre-compiled dependencies for the backend, in line with the project's goal to speed up build and deployment times. Changes to `infra/terraform/main.tf` include the addition of the `govtool-ecr-backend-base` module, specifying the `backend-base` as the repository name. This ensures a separate and organized repository for the base image, facilitating better management and access control. Furthermore, the AWS IAM policy `cicd_ecr` is updated to include permissions for the new `backend-base` repository. This adjustment allows CI/CD pipelines to push and pull images, ensuring seamless integration with existing workflows. The outputs are also expanded to include the URL of the newly created `backend-base` ECR repository, enhancing accessibility for deployment scripts and CI/CD pipelines. This setup supports the acceptance criteria by establishing the necessary infrastructure to store and manage the base image, ultimately contributing to reduced deployment times.

Introduces a new Dockerfile.base within the govtool/backend directory, aimed at creating a base image for the Haskell backend project. This base image includes all necessary pre-compiled dependencies, following the initiative to expedite the build process for deployment. The Dockerfile is based on the `haskell:9.2-buster` image and sets up an environment optimized for the GovTool project by pre-installing all dependencies. By removing the source code after dependency installation, we ensure that the image only contains what's necessary for future builds, thus aligning with our goal to reduce build and deployment times significantly. This change addresses part of the acceptance criteria for speeding up the deployment process by preparing and maintaining a base image with all pre-compiled dependencies required by the application.

This commit introduces a new Terraform module instantiation for creating an AWS ECR repository dedicated to the backend base Docker image. The repository, named `backend-base`, is aimed at storing pre-compiled dependencies for the backend, in line with the project's goal to speed up build and deployment times. Changes to `infra/terraform/main.tf` include the addition of the `govtool-ecr-backend-base` module, specifying the `backend-base` as the repository name. This ensures a separate and organized repository for the base image, facilitating better management and access control. Furthermore, the AWS IAM policy `cicd_ecr` is updated to include permissions for the new `backend-base` repository. This adjustment allows CI/CD pipelines to push and pull images, ensuring seamless integration with existing workflows. The outputs are also expanded to include the URL of the newly created `backend-base` ECR repository, enhancing accessibility for deployment scripts and CI/CD pipelines. This setup supports the acceptance criteria by establishing the necessary infrastructure to store and manage the base image, ultimately contributing to reduced deployment times.

This commit updates the Dockerfile for the GovTool backend to utilize the newly created base image, which contains all necessary pre-compiled Haskell dependencies. By changing the base image from `haskell:9.2-buster` to the custom `backend-base` image hosted on AWS ECR, the build process for the backend Docker image is significantly optimized. The key changes include: - Modifying the FROM directive to point to the `backend-base` image, ensuring that the environment is pre-setup with all the dependencies required for the backend build. - Simplifying the build command to `cabal build` since `cabal update`, `cabal configure`, and dependency installations are no longer necessary, being already handled in the base image creation process. This adjustment ensures that the backend Dockerfile leverages the pre-compiled dependencies in the base image, directly contributing to a faster build and deployment process. This change aligns with the project's acceptance criteria by ensuring that only the application-specific code is compiled during the build process, leveraging the efficiencies offered by the base image.

This commit enhances the `scripts/govtool/Makefile` by introducing new targets `build-backend-base` and `push-backend-base`. These additions facilitate the process of building and pushing the base Docker image for the backend, which contains all necessary pre-compiled dependencies to expedite subsequent builds. - The `build-backend-base` target is responsible for building the base image using the `Dockerfile.base` located in the `govtool/backend` directory. It tags the image with the commit hash, providing a clear linkage between the image and the source code state from which it was built. - The `push-backend-base` target, dependent on a successful Docker login (`docker-login`), pushes the newly built base image to the configured repository. This ensures that the image is available for use by the CI/CD pipelines and developers for building the actual backend application image. These targets are crucial for implementing the strategy to speed up the backend build and deployment process by leveraging a Docker image with pre-compiled dependencies. This approach aligns with the acceptance criteria set forth for the project, aiming at reducing build and deployment times through efficient dependency management.

This commit introduces a significant enhancement to the Docker build process for the backend by making the base image versioning dependent on the `vva-be.cabal` file. This change ensures that the base Docker image is closely tied to the project's Haskell dependencies, as specified in the cabal file. - In `govtool/backend/Dockerfile`, an `ARG BASE_IMAGE_TAG` is introduced, allowing for dynamic specification of the base image tag during the build process. This facilitates using different versions of the base image as dependencies evolve. - The `scripts/govtool/Makefile` is updated to calculate the `base_backend_image_tag` by hashing the `vva-be.cabal` file. This hash becomes the tag for the base image, ensuring that any changes in the cabal file result in a new base image version. This mechanism automates the versioning process, ensuring that the backend is always built against the correct set of pre-compiled dependencies. By linking the base image version directly to the cabal file's hash, this approach guarantees that any changes in dependencies are automatically accounted for, thereby streamlining the build process and enhancing consistency across builds.

Introduces a new Dockerfile.base within the govtool/backend directory, aimed at creating a base image for the Haskell backend project. This base image includes all necessary pre-compiled dependencies, following the initiative to expedite the build process for deployment. The Dockerfile is based on the `haskell:9.2-buster` image and sets up an environment optimized for the GovTool project by pre-installing all dependencies. By removing the source code after dependency installation, we ensure that the image only contains what's necessary for future builds, thus aligning with our goal to reduce build and deployment times significantly. This change addresses part of the acceptance criteria for speeding up the deployment process by preparing and maintaining a base image with all pre-compiled dependencies required by the application.

This commit introduces a new Terraform module instantiation for creating an AWS ECR repository dedicated to the backend base Docker image. The repository, named `backend-base`, is aimed at storing pre-compiled dependencies for the backend, in line with the project's goal to speed up build and deployment times. Changes to `infra/terraform/main.tf` include the addition of the `govtool-ecr-backend-base` module, specifying the `backend-base` as the repository name. This ensures a separate and organized repository for the base image, facilitating better management and access control. Furthermore, the AWS IAM policy `cicd_ecr` is updated to include permissions for the new `backend-base` repository. This adjustment allows CI/CD pipelines to push and pull images, ensuring seamless integration with existing workflows. The outputs are also expanded to include the URL of the newly created `backend-base` ECR repository, enhancing accessibility for deployment scripts and CI/CD pipelines. This setup supports the acceptance criteria by establishing the necessary infrastructure to store and manage the base image, ultimately contributing to reduced deployment times.

This commit updates the Dockerfile for the GovTool backend to utilize the newly created base image, which contains all necessary pre-compiled Haskell dependencies. By changing the base image from `haskell:9.2-buster` to the custom `backend-base` image hosted on AWS ECR, the build process for the backend Docker image is significantly optimized. The key changes include: - Modifying the FROM directive to point to the `backend-base` image, ensuring that the environment is pre-setup with all the dependencies required for the backend build. - Simplifying the build command to `cabal build` since `cabal update`, `cabal configure`, and dependency installations are no longer necessary, being already handled in the base image creation process. This adjustment ensures that the backend Dockerfile leverages the pre-compiled dependencies in the base image, directly contributing to a faster build and deployment process. This change aligns with the project's acceptance criteria by ensuring that only the application-specific code is compiled during the build process, leveraging the efficiencies offered by the base image.

This commit enhances the `scripts/govtool/Makefile` by introducing new targets `build-backend-base` and `push-backend-base`. These additions facilitate the process of building and pushing the base Docker image for the backend, which contains all necessary pre-compiled dependencies to expedite subsequent builds. - The `build-backend-base` target is responsible for building the base image using the `Dockerfile.base` located in the `govtool/backend` directory. It tags the image with the commit hash, providing a clear linkage between the image and the source code state from which it was built. - The `push-backend-base` target, dependent on a successful Docker login (`docker-login`), pushes the newly built base image to the configured repository. This ensures that the image is available for use by the CI/CD pipelines and developers for building the actual backend application image. These targets are crucial for implementing the strategy to speed up the backend build and deployment process by leveraging a Docker image with pre-compiled dependencies. This approach aligns with the acceptance criteria set forth for the project, aiming at reducing build and deployment times through efficient dependency management.

This commit introduces a significant enhancement to the Docker build process for the backend by making the base image versioning dependent on the `vva-be.cabal` file. This change ensures that the base Docker image is closely tied to the project's Haskell dependencies, as specified in the cabal file. - In `govtool/backend/Dockerfile`, an `ARG BASE_IMAGE_TAG` is introduced, allowing for dynamic specification of the base image tag during the build process. This facilitates using different versions of the base image as dependencies evolve. - The `scripts/govtool/Makefile` is updated to calculate the `base_backend_image_tag` by hashing the `vva-be.cabal` file. This hash becomes the tag for the base image, ensuring that any changes in the cabal file result in a new base image version. This mechanism automates the versioning process, ensuring that the backend is always built against the correct set of pre-compiled dependencies. By linking the base image version directly to the cabal file's hash, this approach guarantees that any changes in dependencies are automatically accounted for, thereby streamlining the build process and enhancing consistency across builds.

This commit optimizes the build process for the backend base Docker image in the `scripts/govtool/Makefile`. It introduces conditional logic to build or push the base image only if it doesn't already exist in the repository, significantly reducing unnecessary builds and pushes, thereby saving time and resources. Key changes include: - The `build-backend` target now depends on `build-backend-base`, ensuring the base image is available before building the backend image. However, the base image will only be built if it is not already present, as checked by `docker manifest inspect`. - Similarly, `push-backend-base` incorporates a condition to push the base image only if it is not already available in the repository, using `docker manifest inspect` to verify its existence. This logic prevents redundant builds and uploads of the base image when no changes have occurred to the `vva-be.cabal` file, making the CI/CD pipeline more efficient. By building the base image only when necessary, we align with best practices for Docker image management and optimize our deployment workflow.