How do I mass delete 'Delete-Markers' inside a bucket?
Sometimes due to the S3 versioning feature, a bucket can accumulate a large number of Delete Markers (DM). These DMs along with current & non-current objects are part of LIST operation calls to any S3 storage system (AWS S3, Wasabi etc.) If a bucket comprises a large number of DMs, the LIST queries can take a longer time to go through all keys in the buckets and can show slower response times.
There are some s3 applications/tools which can recognize versioning feature and allow you to delete one DM at a time as shown here. This is helpful when you have fewer DMs to delete. However, we can also use a scripted method to delete a large set of DMs using the following python code.
Note: You may also use s3undelete which is a simple utility created by ARQ that removes delete markers from S3 (and S3-compatible) objects in buckets that have versioning enabled. The details of this utility are shown at the end of this document.
- Make sure you have installed AWS SDK boto3 for python on your CLI and turned off the versioning feature on your bucket before running the script
- Install Python 3+ version to run this script
Executions and Details of the Script (output & screenshot attached):
1. When you execute the script, it will prompt you to select the profile or enter the API keys of the admin who is executing this script.
$ starting script...
$ Press 1 and enter to select existing profile
$ Press 2 and enter to enter Access Key and Secret Key
$ Press 3 to exit:
(i) If you already have a profile configured on your CLI, you may Press 1
NOTE It is optional for you to use credential files to run your code but it is always a best practice to use such implementation wherein your credential keys are in a file stored on your local machine rather than being part of your actual code or entering Keys at runtime prompt
In this example, we have set the profile name as "profilename" in the "~/.aws/credentials" file.
(ii) If you do not wish to use the existing profile, you may press 2 and enter your API Keys.
2. You will be prompted to select an option from the menu. Select Option 2:
3. Enter your own Bucket Name (The region of the bucket will be automatically detected)
NOTE: If you are specifying a prefix, please be sure to enter FULL PREFIX PATH (bucket name NOT included)
4. Before the deletion starts, the script will perform complete pagination of your bucket and show you
- Total Number of Delete-Markers which will be deleted
- Total Number of Current Objects which will NOT be deleted
- Total Number of Non-Current Objects which will NOT be deleted
Each log entry will represent the deletion of 1000 DMs at a time
$ Please select what you want to do:
$ Versioned bucket
$ 1- Delete only non-current versions (old versions) (Versioned bucket)
$ 2- Delete only Delete markers (Versioned bucket)
$ 3- Delete current and non current versions (this will clear everything) (Versioned bucket)
$ Non Versioned bucket
$ 4- Delete all content (Non versioned bucket)
$ Bucket operation
$ 5- Delete a bucket (consider clearing the bucket data before)
$ 6- Exit the script
$ Choice [1, 2, 3, 4, 5, 6]: 2
$ Please enter the name of the bucket: 0-versioning-1
$ Please enter a prefix (leave blank if you don't need one)
$ Location of the bucket: us-east-1
$ Deleting only delete markers
$ Calculating, please wait... this may take a while
$ listing object versions: page 1
$ Analyze completed.
$ Total number of delete markers: 12
$ Are you sure you want to delete these delete markers [y/n]: y
$ Deleting batch N° 1 - 12 object version
$ Press Enter to resume!
The script is also attached to this KB document.
You may also make use of the s3undelete utility as an alternative to delete Delete-Markers (DMs). This utility is macOS and Windows Compatible.
Download the packages here:
s3undelete, set 2 environment variables,
AWS_SECRET_KEY, to your S3 access key ID and secret access key respectively.
1. List Delete-Marker(s)
s3undelete list us-east-2 https://s3.us-east-2.wasabisys.com/username-delete-marker-demo-bucket
2. Removing Delete-Marker(s):
s3undelete undelete us-east-2 https://s3.us-east-2.wasabisys.com/username-delete-marker-demo-bucket
Note: This config example discusses the use of Wasabi's us-east-2 storage region. To use other Wasabi storage regions, please use the appropriate Wasabi service URL as described in this article