Skip to content

An industrial strength, best practice reference architecture showing how to prevent credential sprawl with Vault on AWS

License

Notifications You must be signed in to change notification settings

mike-goodwin/aws-vault-reference-infrastructure

Repository files navigation

A reference architecture for Hashicorp Vault on AWS

This project aims to show how to build an secure and resilient impementation of Vault on the AWS platform, consumed by a simple web application that includes some good security coding practises. Hopefully this will help you get the best out of your Vault out of your projects. The project is licensed under the Apache v2.0 license, so you are free to use or modify it however you like, but we don't offer any kind of warranty!

Overview

Vault is a good fit for a few use cases. The main one we are looking at is for a running application. This solves a few problems:

  • It protects agains the insider threat of rogue admins/operators abusing DB credentials. This is a hard issue to solve in traditional ways and in the end the mitigation comes down to auditing DB logs for unusual activity. That is obviously a bit late because the damage has already been done by the time you look at the logs..
  • It automates credential rotation, making it easy to comply with your crypto key management best practises
  • It removes the need to store credentials in potentially insecure config files or similar

Here is the overall architecture:

Architecure Overview

Some key features:

  • If the Vault fails, any dependent application fails too, so it is multi AZ and multi-server in each AZ
  • The storage backend must also support HA (we have chosen DynamoDB)
  • The MySQL secret backend generates credentials on the fly so it needs access to the DB. We are using VPG Gateways to lock down access to the DB
  • We are using the AWS EC2 Auth backend so that we can make use of IAM roles to control access to the Vault
  • For better isolation and defense-in-depth, the architecture supports applications in different AWS accounts and the Vault servers in yet another AWS account

Infrastructure

To do:

  • Network
  • Components
  • Deployment/CloudFormation

Vault config

To do:

  • Initialisation
  • Storage backend
  • Audit backend
  • MySQL secret backend

IAM roles and policies and Vault permissions

The overall identitiy flow is like this: Architecure Overview

IAM for Application Servers

Vault will be configured to use the AWS EC2 Auth backend so each application server will launch into an instance profile IAM role. We have called this AppNInstanceProfile - a different profile is needed for each application if they are in different accounts. There are no special IAM polciy requirements for this role. It just needs whatever is required for the application to run. In the case of the sampel application, this is nothing.

IAM for Vault Servers

The Vault servers are launched using an IAM role called VaultServer. In order to authenticate requests for credentials from the application servers, it needs to make calls to IAM for each of the application AWS accounts. In each application account this needs a role AppNVault with permissions ec2:DescribeInstances and iam:GetInstanceProfile. The VaultServer role then needs sts:AssumeRole for each of these roles.

Vault Permissions

For better isolation, each application has a dedicated secret backend mounted in Vault. Only the appropriate application server instance profile is allowed to call each secret backend.

The client application

To complement this reference infrastructure, there is a sample Vault-aware application. The app shows how to dynamically fetch DB credentials and handle Vault lease renewals.

About the authors

We are not affiliated to Hashicorp or the Vault project in any way and any recommendations made are our own and not endorsed by Hashicorp. We just like Vault, AWS, security and messing about with cool tech :-)


Installation Prerequisites

There are a couple of things you need to satisfy before you can deploy the Vault Reference Infrastructure in your own AWS environment;

  • An AWS Account - You can set up an account by following the instructions listed here. At the moment one account will suffice, but the final deployment will utilise two separate AWS accounts with one VPC in each. The account creation is the only thing that cannot be automated.
  • AWS CLI - This deployment uses the CLI to automate the build process, so you will need to install and configure this before attempting the deploy. Full instructions to install the CLI tools are located here, with configuration instructions located here.

Once you have created the AWS account, you will need to create an IAM user (with ability to run the Cloudformation templates and create AWS services) and also download the associated SSH key (which is used to SSH onto the Bastion and Vault Servers.)

You will have two files located in the ~/.aws/ directory called config and credentials. The contents of the credentials file will look similar to;

[default]
aws_access_key_id = AKIGHSJJKKKLKLLAKHJK
aws_secret_access_key = k/wKqksafjejfhkhadflkjlixjklqnekdm;lw8Xd

You don't need to know anything in this file, other than what is between the square brackets, in this case [default]. This is the profile name and will be used later when the setup script is run.

Installation Files

The files required for the installation (and are located in this GitHub repository) are listed below;

File Name Description
setup.sh A Bash script which makes the set up of variables much easier. It's been tested on Linux but should also work on all macOS and OSX versions. A Powershell script will also be added for ease of Installation on Windows machines.
vault-core-networking.json This is the first Cloudformation template which generates all of the basic networking required (VPCs, subnets and gateways.)
vault-core-networking-parameters.json This file contains all of the parameters which you will need to change to deploy the system within your AWS infrastructure. These include the VPC CIDR address, Choice of AWS Region Availability Zones, Subnet CIDR addresses and your IP address for connecting into the Vault servers if required.
vault-core-security.json The Cloudformation template that handles Security Groups and Network ACLs.
vault-core-instances.json The Cloudformation template which generates the EC2 instances, AutoScaling Groups and ELBs.
config.hcl Basic config file for DynamoDB creation. You will only need to change the AWS Region to where you require DynamoDB to be created (The default is EU-West-1.)
supervisord.conf This is the Supervisord config file which allows Vault to run automatically upn deploy.
supervisord The init.d script for Supervisord to start.

Download and Configuration

The first step to deploy, is to clone the GitHub repository to your local machine, using the following command (you will need git installed and running locally, which is out of scope for this document);

git clone https://github.com/mike-goodwin/aws-vault-reference-infrastructure.git

Once this has been cloned locally, all of the parameters that you may possibly need to change are in one file called vault-core-networking-parameters.json. Although all options can be changed, the ones you especially need to pay attention to, are listed at the end of the file;

Parameter Key Parameter Value Description
VaultIngressCIDR 77.97.82.48/32 In order to SSH into the Vault servers, you are required to use a Bastion server sitting in a public subnet to 'hop' onto and from there, connect to the Vault servers. This Bastion server is locked down to a single IP address which is the address that you are connecting to the Internet from (Go to https://ifconfig.co to get this value) and replace the existing CIDR address.
VaultCoreEC2AMI ami-70edb016 If you are deploying to any region other than EU-WEST-1, you will need to change this to reflect an AMI in your region
VaultEC2KeyPair palindrome_2017 This is the name of your AWS SSH key which you should have downloaded when creating your AWS IAM User Account (described above)

Installation Process

The final step is to actually deploy the full Stack. To do this, locate the file called setup.sh ( This will be in the local git repository you cloned earlier and you may need to set the file executable bit using chmod +x setup.sh) and then run it. You will see the following;

---------------------------------------------------------------------------------------
AWS VAULT REFERENCE INSTALLATION
---------------------------------------------------------------------------------------
Select from the following options...

1. VALIDATE CLOUDFORMATION TEMPLATES
2. BUILD CLOUDFORMATION STACK
0. Quit

Enter Selection [0-2]

Press 1, if you want to check that all of the templates downloaded are syntactically correct before deploying.

To deploy the whole Cloudformation Stack, press 2 and you will see;

---------------------------------------------------------------------------------------
BUILD CLOUDFORMATION STACK
---------------------------------------------------------------------------------------
Enter the name of the AWS profile you wish to use (Leave blank for 'default' profile)...

default

Enter the name of the Cloudformation Stack...

Vault Reference Stack

Type the name of the profile in the first field (the one in the square brackets of the credentials file) and in the second field, type in any name or phrase you wish to call the Cloudformation Stack. The Stack will then build (this will take some time) and providing everything goes well, the Vault side of the infrastructure will be built and will be running.

About

An industrial strength, best practice reference architecture showing how to prevent credential sprawl with Vault on AWS

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published