A push to master will trigger infrastructure deployment to AWS using Github acitons defined in apply.yml
Also its possible to trigger the deployment manually using Github Actions
If the "Destroy the infra" option is selected the infra will be destroyed instead of applying the changes
To deploy to AWS from local manually follow the create-aws-infra.sh script
to make localstack work with terraform we need to use terraform-local
more info here
https://docs.localstack.cloud/user-guide/integrations/terraform/
Also, for convenience we'll install awslocal
command
https://docs.localstack.cloud/user-guide/integrations/aws-cli/
Conda environment created in the main repo needs to be activated (eg it has pipenv already installed and uses a proper python version):
conda activate capitalbikeshare-mlops
To have both tflocal
and awslocal
available:
pipenv install
pipenv shell
To test everything locally you can just run
Important
For this to run you need to have a docker image capitalbikeshare-service-prediction-service:latest
defined in the service repo built locally
Also make sure that the port 80 is available
./test-infra.sh
It will execute all the following steps:
To be able to use localstack-pro image that supports ECR and ECS one needs to sign up for
localstack free trial and have an environment variable LOCALSTACK_API_KEY
set.
On Mac/Linux:
export LOCALSTACK_API_KEY=<your_localstack_api_key>
Then we need to spin up container with localstack
docker compose up -d
First we need to make a bucket manually to hold the tfstate
Make bucket
awslocal s3 mb s3://terraform-state-mlops-zoomcamp
Check that the bucket is there just in case
awslocal s3 ls
Export environment name that we wish to work with. For instance, stg
export TF_ENV="stg"
To init tflocal with the corresponding s3 backend configuration (for staging/prod) Eg the following will do the init with the staging backend
tflocal init -reconfigure -backend-config=environments/${TF_ENV}/backend.config
First we need to create only ECR repo to be able to push an image
tflocal plan -target=aws_ecr_repository.prediction_service_ecr_repo -var-file="environments/${TF_ENV}/environment.tfvars" -out=ecr_repo.tfplan
tflocal apply ecr_repo.tfplan
Tag existing image (built in the other repo) and push to the ECR
image_name_with_tag="$(tflocal output -raw ecr_repository_url):latest"
docker tag capitalbikeshare-service-prediction-service:latest $image_name_with_tag
docker push $image_name_with_tag
See the plan
tflocal plan -var-file="environments/${TF_ENV}/environment.tfvars"
Apply (needs confirmation)
tflocal apply -var-file="environments/${TF_ENV}/environment.tfvars"
Run healthcheck
curl http://localhost/healthcheck
Run prediction
curl -X POST http://localhost/predict -H "Content-Type: application/json" -d '{"start_station_id": "31205","end_station_id": "31239","rideable_type": "docked_bike","member_casual": "member","started_at": "2020-04-06 07:54:59"}'
Destroy (needs confirmation)
tflocal destroy -var-file="environments/${TF_ENV}/environment.tfvars"
The following requirements are needed by this module:
The following providers are used by this module:
- aws (5.13.1)
The following resources are used by this module:
- aws_alb.application_load_balancer (resource)
- aws_default_subnet.default_subnet_a (resource)
- aws_default_subnet.default_subnet_b (resource)
- aws_default_vpc.default_vpc (resource)
- aws_ecr_repository.prediction_service_ecr_repo (resource)
- aws_ecs_cluster.ecs_cluster (resource)
- aws_ecs_service.app_service (resource)
- aws_ecs_task_definition.service_task (resource)
- aws_iam_role.ecsTaskExecutionRole (resource)
- aws_iam_role_policy_attachment.ecsTaskExecutionRole_policy (resource)
- aws_lb_listener.listener (resource)
- aws_lb_target_group.target_group (resource)
- aws_security_group.load_balancer_security_group (resource)
- aws_security_group.service_security_group (resource)
The following input variables are required:
Description: ECS CPU to set to the tasks.
Type: number
Description: How many ECS Tasks to initialy deploy per service.
Type: number
Description: Environment name
Type: string
Description: ECS memory to set to the tasks.
Type: number
Description: AWS Region where to create the resources.
Type: string
The following input variables are optional (have default values):
Description: ECR repo name
Type: string
Default: "capitalbikeshare-dv-model-pipeline"
The following outputs are exported:
Description: Log the load balancer app url
Description: n/a
Description: n/a
Description: n/a
Description: n/a
Description: n/a