Reduce AWS costs while maintaining stable operations using this one weird trick!
- Troy Dieter
- Aws
- September 20, 2020
Table Of Contents
aws-auto-cleanup
Functional Requirements
- Reduce operational run-time of resources used within an AWS account for testing\development
- Reduce cost due to deployed resources
- Ability to whitelist AWS resources that need to be retained
Operating Cost
< $2.00/mo for the following:
- AWS::Events::Rule
- AWS::Lambda::Function
- AWS::Lambda::Permission
- AWS::Logs::LogGroup
- AWS::IAM::Role
- AWS::S3::Bucket
- AWS::S3::Bucket
- AWS::DynamoDB::Table
- AWS::DynamoDB::Table
| Node | Type | Unit | Usage | Total |
|---|---|---|---|---|
| auto-cleanup-settings-prod (DynamoDB-Table) | provisioned read | $0.00013 per hour for units of read capacity | 1 units per month | $0.10 |
| auto-cleanup-settings-prod (DynamoDB-Table) | provisioned write | $0.00065 per hour for units of write capacity | 1 units per month | $0.48 |
| auto-cleanup-settings-prod (DynamoDB-Table) | storage | $0.25 per GB-month | 1 GB | $0.25 |
| auto-cleanup-whitelist-prod (DynamoDB-Table) | provisioned read | $0.00013 per hour for units of read capacity | 1 units per month | $0.10 |
| auto-cleanup-whitelist-prod (DynamoDB-Table) | provisioned write | $0.00065 per hour for units of write capacity | 1 units per month | $0.48 |
| auto-cleanup-whitelist-prod (DynamoDB-Table) | storage | $0.25 per GB-month | 1 GB | $0.25 |
| auto-cleanup-prod (Lambda-Function) | 1000 | 128MB * AvgTime * Invocations per month (Usage) | 10000 invocations | $0.02 |
| ServerlessDeploymentBucket (S3-Bucket) | Standard | $0.023 per GB - first 50 TB / month of storage used | 10 GB | $0.23 |
| auto-cleanup-prod-resourcetreebucket-troydieter (S3-Bucket) | Standard | $0.023 per GB - first 50 TB / month of storage used | 10 GB | $0.23 |
| Dynamo Storage | Discount | First 25GB Free (-$0.25 per GB) | -2 | ($0.50) |
| Dynamo Backup | Point In Time Recovery | $0.20 per GB-month (Continuous backups) | 0 | $0.00 |
| Dynamo Backup | OnDemand | $0.10 per GB-month (On Demand) | 0 | $0.00 |
| Dynamo Network Outbound Traffic | Transfer | $0.09 per GB (Data Transfer Out) | 1 | $0.09 |
| Lambda Service | Number of invocations | Invocation call for a Lambda function | 10000 | $0.00 |
Diagram
Stack Requirements
- Serverless Stack
Framework Core: 2.1.1 Plugin: 4.0.4 SDK: 2.3.2 Components: 3.1.3 - AWS CLI
- NPM
Repository
Utilize the GitHub repository for aws-auto-cleanup
Notable commands
serverless deploy --region region-example --aws-profile profile-example
serverless invoke --function AutoCleanup --region region-example --aws-profile profile-example --type Event
Configuration
After you clone the aws-auto-cleanup repository as shown in the previous section, you will have the opportunity to change the parameters of the serverless-stack deployment. A few notable changes:
Change the serverless.yml file line 4, to your company name or project name.
Within the
auto_cleanup/data/auto-cleanup-settings.jsonfile, you will find the default parameters for the function. I have (obviously) found that the TTL (time-to-live) of 7 days to be too low. I have raised that to 120 days. Example:"ttl": { "N": "120" }
Outputs
- Under CloudWatch, you’ll see under Logs > Log Groups the following log group (if you kept the default Lambda function name):
/aws/lambda/auto-cleanup-prod
With a log stream of the latest Serverless invocation (or scheduled Lambda execution):
| 1600638849212 | [INFO] S3 Bucket ‘outbound-email-send-dev-serverlessdeploymentbucke-46346’ was created 234 days ago and has been deleted. (s3_cleanup.py, buckets(), line 168)
| | 1600638849212 | [INFO] S3 Bucket ‘outbound-email-send-dev-serverlessdeploymentbucke-46346’ was created 234 days ago and has been deleted. (s3_cleanup.py, buckets(), line 168)
| | 1600638849212 | [INFO] S3 Bucket ‘outbound-email-send-hand-serverlessdeploymentbuck-46346’ was created 234 days ago and has been deleted. (s3_cleanup.py, buckets(), line 168)
| | 1600638849212 | [INFO] S3 Bucket ‘outbound-email-send-hand-serverlessdeploymentbuck-46346’ was created 234 days ago and has been deleted. (s3_cleanup.py, buckets(), line 168)
| | 1600638849212 | [INFO] S3 Bucket ‘46346-ai-chat-bot-dev-serverlessdeploymentbucket-46346’ was created 230 days ago and has been deleted. (s3_cleanup.py, buckets(), line 168)
| | 1600638849268 | [INFO] S3 Bucket ‘46346-artifacts’ was created 505 days ago and has been deleted. (s3_cleanup.py, buckets(), line 168)
| | 1600638849269 | [INFO] S3 Bucket ‘46346-cpds-infra’ was created 494 days ago and has been deleted. (s3_cleanup.py, buckets(), line 168)
| | 1600638849269 | [INFO] S3 Bucket ‘46346-sree52-testbkt’ was created 430 days ago and has been deleted. (s3_cleanup.py, buckets(), line 168)
| | 1600638849269 | [INFO] S3 Bucket ‘python-myservice-bucket’ was created 158 days ago and has been deleted. (s3_cleanup.py, buckets(), line 168)
| | 1600638849269 | [INFO] S3 Bucket ‘python-poc-deployment-bucket’ was created 166 days ago and has been deleted. (s3_cleanup.py, buckets(), line 168)
| | 1600638849269 | [INFO] S3 Bucket ‘s3-to-sns’ was created 264 days ago and has been deleted. (s3_cleanup.py, buckets(), line 168)
| | 1600638849269 | [INFO] S3 Bucket ‘serverless-telegram-bot-serverlessdeploymentbuck-46346’ was created 236 days ago and has been deleted. (s3_cleanup.py, buckets(), line 168)
| | 1600638849269 | [INFO] S3 Bucket ‘swagger-bucket-1’ was created 199 days ago and has been deleted. (s3_cleanup.py, buckets(), line 168)
| | 1600638849269 | [INFO] S3 Bucket ’terraform-bucket-lokesh’ was created 243 days ago and has been deleted. (s3_cleanup.py, buckets(), line 168)
| | 1600638849270 | [INFO] S3 Bucket ’twilio-voice-test-dev-serverlessdeploymentbucket-46346’ was created 230 days ago and has been deleted. (s3_cleanup.py, buckets(), line 168)
Performing clean-up
Change the
dry_runitem in the DynamoDB table (auto-cleanup-settings-prod- if you kept the default settings name) to false to perform clean-up:{ "dry_run" : { "BOOL" : false }}Invoke using:
serverless invoke --function AutoCleanup --region region-example --aws-profile profile-example --type Event
Recap
aws-auto-cleanup helps to keep multi-region AWS accounts clean. Resources are either white-listed, or deleted after specified amount of days. This reduces monthly expenses and conflicting resources within the same region!
