Lambda function to cleanup index's according to age
- Admin Acess to: AWS S3, Elasticsearch, Lambda, IAM
- aws cli
- python 2.7+
- boto3
- virtualenv
- jq
- create the lambda IAM role
aws iam create-role --role-name lambda-elasticsearch-index-cleaner --assume-role-policy-document="$(cat aws_role_policies/trust-policy.json|jq -c '.' )"
- modify the role so that it can assume itself for STS token generation
aws iam update-assume-role-policy --policy-document="$(cat aws_role_policies/trust-policy-mod.json|jq -c '.')" --role-name lambda-elasticsearch-index-cleaner
- Add access policies to allow
- S3 readonly
- ES access
- Basic Lambda execution (aka cloudwatch logs)
- create the bucket where the lambda function config will be stored
aws s3 mb s3://lambda-elasticsearch-index-cleaner-config --region eu-west-1
- create the bucket where lambda function deployment zip will be stored
aws s3 mb s3://lambda-elasticsearch-index-cleaner --region eu-west-1
- Create 4 folders to hold config files for different deployment stages thorugh the AWS S3 console:
$LATEST
DEV
STAGE
PROD
Permissions policy should allow calls from the lamda role, however in my case I have this open to the domain. You will need to get your ES endpoint URL
- install needed python dep's
pip install virtualenv boto3
- clone the repo
git clone https://github.com/dbnegative/lambda-elasticsearch-index-cleaner
cd lambda-elasticsearch-index-cleaner
- edit the config/deployment-config.json if needed.
{
"S3_CONFIG_BUCKET":"lambda-elasticsearch-index-cleaner",
"LAMBDA_DEPLOY_BUCKET": "lambda-elasticsearch-index-cleaner",
"CONFIG_FILE":"config.json",
"LAMBDA_FUNC_NAME" :"lambda-elasticsearch-index-cleaner",
"LAMBDA_HANDLER":"lambda_function.lambda_function",
"LAMBDA_ROLE_ARN":"arn:aws:iam::<YOURAWSACCOUNTID>:role/lambda-elasticsearch-index-cleaner",
"LAMBDA_TIMEOUT":"120",
"LAMBDA_MEMORY_SIZE":"128"
}
- setup the build enviroment
deploy-wrapper.py setup
- edit the config/config.json with your own settings, at the minimum the following:
- set expiry age in days, logs older than this will be deleted
- string-match should be the log index prefix that will be deleted. i.e a string match of "cloudfrontlog" would match indexes called cloudfrontlog-2016-10-21
{
"es_host": "search-cloudfront-logging-sfgveenrrv6a2d52t5svjyvnzy.eu-west-1.es.amazonaws.com",
"es_region": "eu-west-1",
"es_connection_timeout": 60,
"sts_role_arn": "arn:aws:iam::688390856993:role/lambda-elasticsearch-index-clean",
"sts_session_name": "lambda-elasticsearch-index-clean",
"string-match": "cloudfrontlog",
"expiry-age": "4"
}
- create the initial version of the function using the deploy-wrapper.sh
deploy-wrapper.py init
- create 3 lambda alias for continous deploments and tests
aws lambda create-alias --name DEV --function-name lambda-elasticsearch-index-cleaner --function-version=1
aws lambda create-alias --name STAGE --function-name lambda-elasticsearch-index-cleaner --function-version=1
aws lambda create-alias --name PROD --function-name lambda-elasticsearch-index-cleanerr --function-version=1
-
create s3 trigger on PROD alias. You can now deploy and test to DEV and STAGE without affecting your production version
- go to the lambda console
- select the lambda-elasticsearch-index-cleaner fucntion
- press the "Qualifiers" button and select the PROD alias
- select the "Triggers" tab
- add an Cloudwatch Scheduled event trigger
- name the rule to something appropriate
- set the event to trigger every day
- enable the trigger and save
-
deploying a new build to DEV alias
deploy-wrapper.py deploy --env DEV
- promoting that version to STAGE alias
deploy-wrapper.py promote DEV STAGE
Deploy and manipulate lambda function
positional arguments:
{promote,deploy,config,clean,setup}
[CMDS...]
promote promote <source enviroment> version to <target
enviroment>
deploy deploy function to s3
config deploy config to s3
init creates the base lambda function
clean clean local build enviroment
setup create local build enviroment
optional arguments:
-h, --help show this help message and exit
- aws policy files - S3, ELASTICSEARCH, LOG
- improve instructions aka this file