Cleaning up files from AWS S3 Automatically !!!

PublishedFebruary 24, 2023

Hey there! 👋 I'm Hardeep Jethwani (HJ), your resident cloud aficionado and code maestro, proudly navigating the ever-changing seas of AWS Cloud and Full Stack Development for ~5 glorious years and counting. ☁️💻

Currently, I'm orchestrating the tech symphony as part of Team HSBC Bank, where I'm on a mission to enhance the banking experience through the magic of technology. 🚀💼

In my past life at Capgemini, I led exciting adventures like migrating critical applications to the cloud (18 and counting!). I had databases waltzing into the AWS Cloud, sprinkling a bit of containerization magic along the way. AWS managed services like RDS, Lambda, ECS, and friends? They were my trusty sidekicks. 🎩🔧

When not automating deployments with CI/CD finesse (think AWS CodePipeline, CodeBuild, and CodeDeploy), you might find me designing infrastructure like a digital architect using AWS CloudFormation. Security is my jam – I've got WAF, Security Groups, MFA, Cognito, and even a secret club in private subnets to keep things safe. 🔒💂‍♂️

On top of all that, I'm on a mission to reduce carbon footprints because, why not? HSBC's commitment to sustainability is my heart and soul. We're going for NET ZERO carbon footprint, and I'm leading the charge, one container at a time! 🌍🌱

And yes, the fun doesn't stop at work. In my past life at Tata Consultancy Services, I co-created a multi-tier Point of Sale application with a global footprint, touching the lives of billions. My automation tools were so efficient that even Father Time was left scratching his head. ⏳💡

If you're in need of a cloud-savvy comedian or a code deployment magician, look no further. Let's chat about tech, swap automation tales, or share some coding humor over a virtual coffee. Oh, and don't worry; I promise not to write code in my sleep (well, most of the time). Cheers to cloud adventures! ☕🚀

S3 is a cheap and convenient form of cloud storage where we can store data and access the same from anywhere in the world.

It might be you storing the data in S3 or might be your application storing logs or your Database servers storing backups. But what if you just store and store and store but you don't filter or clean it up? Your S3 bucket will be flooded with old log files and backups. OR you will have some outdated data bulked in your S3 that is of No use now.

And also, the more data you have the more you will have to pay. So in these cases you have to clean up the files that you don't need anymore. But wait why do it manually when we have **"Lifecycle Rules"** available for us?

Yes, we have a feature called "Lifecycle Rules" which can be used for either cleaning up the files automatically or moving the files to another storage class of S3. Let's start the action -

Auto delete Life Cycle Rule over the entire S3 Bucket

Jump into the S3 bucket where you want to apply the LifeCycle policy. Here I have a bucket named hardeepjethwani-logs which I did use for storing some log files.

Move to Management section. There you will see Lifecycle Rules :

Click on Create lifecycle rule. You will land to a form where you need to configure your lifecycle policy.

As we are applying the policy to all the objects of our bucket, just select "Apply to all objects in the bucket" under Rule scope section.

As we need to delete the objects automatically. Select "Expire current versions of objects" under lifecycle rule actions.

Enter the number of days after which you want to delete the files and click on Create Rule.

So now all of your files/objects in your S3 bucket are going to be deleted after 30 days.

Auto delete Lifecycle Rule for a specific folder

What if I have subsequent folders in my S3 bucket and I want to delete files after n days for specific folder only? No worries, we have an option for that as well.

You just need to select the prefix based filtering where you need to add the folder name as prefix. Suppose I have to delete files from Logs folder.

So If you just want to delete the files inside Logs/ folder, then you need to "Limit the Scope" in Rule scope while creating the Lifecycle Rule. And add folder path in the prefix section as below: