Reduce AWS costs while maintaining stable operations using this one weird trick!

Share on:

aws-auto-cleanup

Functional Requirements

  • Reduce operational run-time of resources used within an AWS account for testing\development
  • Reduce cost due to deployed resources
  • Ability to whitelist AWS resources that need to be retained

Operating Cost

< $2.00/mo for the following:

  • AWS::Events::Rule
  • AWS::Lambda::Function
  • AWS::Lambda::Permission
  • AWS::Logs::LogGroup
  • AWS::IAM::Role
  • AWS::S3::Bucket
  • AWS::S3::Bucket
  • AWS::DynamoDB::Table
  • AWS::DynamoDB::Table
Node Type Unit Usage Total
auto-cleanup-settings-prod (DynamoDB-Table) provisioned read $0.00013 per hour for units of read capacity 1 units per month $0.10
auto-cleanup-settings-prod (DynamoDB-Table) provisioned write $0.00065 per hour for units of write capacity 1 units per month $0.48
auto-cleanup-settings-prod (DynamoDB-Table) storage $0.25 per GB-month 1 GB $0.25
auto-cleanup-whitelist-prod (DynamoDB-Table) provisioned read $0.00013 per hour for units of read capacity 1 units per month $0.10
auto-cleanup-whitelist-prod (DynamoDB-Table) provisioned write $0.00065 per hour for units of write capacity 1 units per month $0.48
auto-cleanup-whitelist-prod (DynamoDB-Table) storage $0.25 per GB-month 1 GB $0.25
auto-cleanup-prod (Lambda-Function) 1000 128MB * AvgTime * Invocations per month (Usage) 10000 invocations $0.02
ServerlessDeploymentBucket (S3-Bucket) Standard $0.023 per GB - first 50 TB / month of storage used 10 GB $0.23
auto-cleanup-prod-resourcetreebucket-troydieter (S3-Bucket) Standard $0.023 per GB - first 50 TB / month of storage used 10 GB $0.23
Dynamo Storage Discount First 25GB Free (-$0.25 per GB) -2 ($0.50)
Dynamo Backup Point In Time Recovery $0.20 per GB-month (Continuous backups) 0 $0.00
Dynamo Backup OnDemand $0.10 per GB-month (On Demand) 0 $0.00
Dynamo Network Outbound Traffic Transfer $0.09 per GB (Data Transfer Out) 1 $0.09
Lambda Service Number of invocations Invocation call for a Lambda function 10000 $0.00

Diagram

diagram

Stack Requirements

  1. Serverless Stack Framework Core: 2.1.1 Plugin: 4.0.4 SDK: 2.3.2 Components: 3.1.3
  2. AWS CLI
  3. NPM

Repository

Utilize the GitHub repository for aws-auto-cleanup

Notable commands

serverless deploy --region region-example --aws-profile profile-example
serverless invoke --function AutoCleanup --region region-example --aws-profile profile-example --type Event

Configuration

After you clone the aws-auto-cleanup repository as shown in the previous section, you will have the opportunity to change the parameters of the serverless-stack deployment. A few notable changes:

  1. Change the serverless.yml file line 4, to your company name or project name.

  2. Within the auto_cleanup/data/auto-cleanup-settings.json file, you will find the default parameters for the function. I have (obviously) found that the TTL (time-to-live) of 7 days to be too low. I have raised that to 120 days. Example:

         "ttl": {
           "N": "120"
         }
    

Outputs

  • Under CloudWatch, you’ll see under Logs > Log Groups the following log group (if you kept the default Lambda function name):

/aws/lambda/auto-cleanup-prod

With a log stream of the latest Serverless invocation (or scheduled Lambda execution):

| 1600638849212 | [INFO] S3 Bucket ‘outbound-email-send-dev-serverlessdeploymentbucke-46346’ was created 234 days ago and has been deleted. (s3_cleanup.py, buckets(), line 168)
| | 1600638849212 | [INFO] S3 Bucket ‘outbound-email-send-dev-serverlessdeploymentbucke-46346’ was created 234 days ago and has been deleted. (s3_cleanup.py, buckets(), line 168)
| | 1600638849212 | [INFO] S3 Bucket ‘outbound-email-send-hand-serverlessdeploymentbuck-46346’ was created 234 days ago and has been deleted. (s3_cleanup.py, buckets(), line 168)
| | 1600638849212 | [INFO] S3 Bucket ‘outbound-email-send-hand-serverlessdeploymentbuck-46346’ was created 234 days ago and has been deleted. (s3_cleanup.py, buckets(), line 168)
| | 1600638849212 | [INFO] S3 Bucket ‘46346-ai-chat-bot-dev-serverlessdeploymentbucket-46346’ was created 230 days ago and has been deleted. (s3_cleanup.py, buckets(), line 168)
| | 1600638849268 | [INFO] S3 Bucket ‘46346-artifacts’ was created 505 days ago and has been deleted. (s3_cleanup.py, buckets(), line 168)
| | 1600638849269 | [INFO] S3 Bucket ‘46346-cpds-infra’ was created 494 days ago and has been deleted. (s3_cleanup.py, buckets(), line 168)
| | 1600638849269 | [INFO] S3 Bucket ‘46346-sree52-testbkt’ was created 430 days ago and has been deleted. (s3_cleanup.py, buckets(), line 168)
| | 1600638849269 | [INFO] S3 Bucket ‘python-myservice-bucket’ was created 158 days ago and has been deleted. (s3_cleanup.py, buckets(), line 168)
| | 1600638849269 | [INFO] S3 Bucket ‘python-poc-deployment-bucket’ was created 166 days ago and has been deleted. (s3_cleanup.py, buckets(), line 168)
| | 1600638849269 | [INFO] S3 Bucket ‘s3-to-sns’ was created 264 days ago and has been deleted. (s3_cleanup.py, buckets(), line 168)
| | 1600638849269 | [INFO] S3 Bucket ‘serverless-telegram-bot-serverlessdeploymentbuck-46346’ was created 236 days ago and has been deleted. (s3_cleanup.py, buckets(), line 168)
| | 1600638849269 | [INFO] S3 Bucket ‘swagger-bucket-1’ was created 199 days ago and has been deleted. (s3_cleanup.py, buckets(), line 168)
| | 1600638849269 | [INFO] S3 Bucket ‘terraform-bucket-lokesh’ was created 243 days ago and has been deleted. (s3_cleanup.py, buckets(), line 168)
| | 1600638849270 | [INFO] S3 Bucket ‘twilio-voice-test-dev-serverlessdeploymentbucket-46346’ was created 230 days ago and has been deleted. (s3_cleanup.py, buckets(), line 168)

Performing clean-up

  1. Change the dry_run item in the DynamoDB table (auto-cleanup-settings-prod - if you kept the default settings name) to false to perform clean-up:

    { "dry_run" : { "BOOL" : false }}

  2. Invoke using:

    serverless invoke --function AutoCleanup --region region-example --aws-profile profile-example --type Event

Recap

aws-auto-cleanup helps to keep multi-region AWS accounts clean. Resources are either white-listed, or deleted after specified amount of days. This reduces monthly expenses and conflicting resources within the same region!