Cleaning up files from AWS S3 Automatically  !!!

Cleaning up files from AWS S3 Automatically !!!


3 min read

S3 is a cheap and convenient form of cloud storage where we can store data and access the same from anywhere in the world.

It might be you storing the data in S3 or might be your application storing logs or your Database servers storing backups. But what if you just store and store and store but you don't filter or clean it up? Your S3 bucket will be flooded with old log files and backups. OR you will have some outdated data bulked in your S3 that is of No use now.

And also, the more data you have the more you will have to pay. So in these cases you have to clean up the files that you don't need anymore. But wait why do it manually when we have **"Lifecycle Rules"** available for us?

Yes, we have a feature called "Lifecycle Rules" which can be used for either cleaning up the files automatically or moving the files to another storage class of S3. Let's start the action -

Auto delete Life Cycle Rule over the entire S3 Bucket

Jump into the S3 bucket where you want to apply the LifeCycle policy. Here I have a bucket named hardeepjethwani-logs which I did use for storing some log files.

Move to Management section. There you will see Lifecycle Rules :

Click on Create lifecycle rule. You will land to a form where you need to configure your lifecycle policy.

As we are applying the policy to all the objects of our bucket, just select "Apply to all objects in the bucket" under Rule scope section.

As we need to delete the objects automatically. Select "Expire current versions of objects" under lifecycle rule actions.

Enter the number of days after which you want to delete the files and click on Create Rule.

So now all of your files/objects in your S3 bucket are going to be deleted after 30 days.

Auto delete Lifecycle Rule for a specific folder

What if I have subsequent folders in my S3 bucket and I want to delete files after n days for specific folder only? No worries, we have an option for that as well.

You just need to select the prefix based filtering where you need to add the folder name as prefix. Suppose I have to delete files from Logs folder.

So If you just want to delete the files inside Logs/ folder, then you need to "Limit the Scope" in Rule scope while creating the Lifecycle Rule. And add folder path in the prefix section as below:

All the other steps are similar in this case as the previous case.

Woo hooo !!!!! you saved some space and some $$$ toooo ;) .... (when you are sending me those bucks :D)

Hope you found this article helpful and I hope this helped you save some bucks and saved you from data flooding !!!

Do share this blog and add your comments/feedback.

You can connect with me on Linkedin, Instagram or twitter. For supporting this blog, you can sponsor or buy me a coffee ;)

For more interesting topics check out my other Blogs !!!

Did you find this article valuable?

Support Learn with HJ by becoming a sponsor. Any amount is appreciated!