How do I clean up my failed multipart uploads?

When a file larger than 5 GB is uploaded, it does so in parts. The parts of a multipart upload will range in size from 5 MB to 5 GB (last part can be < 5 MB)

When you complete a multipart upload, the Amazon S3 API used by Wasabi creates an object by concatenating the parts in ascending order based on the part number

When all the parts are uploaded, the system creates an object by concatenating the parts in ascending order based on the part number and the object committed. If the object is not committed, Wasabi keeps the parts around for 31 days, so that you can resume the upload of any missing parts, with only the remaining parts being uploaded.  And then, the process completes with the commit. 

Note that the parts of your upload that are retained are counted as active storage and are included in any charges for your account. The following instructions are aimed to help with cleaning up failed multipart uploads:

First you can list out the multipart uploads you have with the AWS CLI command:

aws s3api list-multipart-uploads --bucket <Bucketname> 
$ aws s3api list-multipart-uploads --bucket mikeo-status-test-east --endpoint-url=https://s3.wasabisys.com
{
    "Uploads": [
        {
            "Initiator": {
                "DisplayName": "mikeo",
                "ID": "64FC53EBEA41CD91234567890E793B721167132B88F80CB6D63D39A3C91182CD"
            },
            "Initiated": "2019-07-23T19:36:30.000Z",
            "UploadId": "KCFf6Rkrv2B7aBcDeFgHjlHv8WwC-r-qUV61jMXuMBMh-iObwi6do3xjVX9myirJIhZ2ez0-CnB1knnmpfYpwdSCwPGKzmVxz7oQzQJVPjXkbZdQ0wlEn_FVd_NtMgq",
            "StorageClass": "STANDARD",
            "Key": "10GB-samplefile.bin",
            "Owner": {
                "DisplayName": "mikeo",
                "ID": "64FC53EBEA41CD91234567890E793B721167132B88F80CB6D63D39A3C91182CD"
            }
        },

From that you need to obtain the key and the upload id, that use that in a second command to abort the multipart upload so that the object parts are deleted.  

$ aws s3api abort-multipart-upload --bucket mikeo-status-test-east --endpoint-url=https://s3.wasabisys.com --key 10GB-samplefile.bin --upload-idKCFf6Rkrv2B7aBcDeFgHjlHv8WwC-r-qUV61jMXuMBMh-iObwi6do3xjVX9myirJIhZ2ez0-CnB1knnmpfYpwdSCwPGKzmVxz7oQzQJVPjXkbZdQ0wlEn_FVd_NtMgq

Please note that in this example we use us-east-1 region and the us-east-1 endpoint URL accordingly. You can find the endpoint URLs of our other regions in this support article.

Unfortunately, there is no AWS CLI command to do that in one step and no way to do this from the Wasabi console. 

But there is a Minio client that indicates that they can do this with one command: 
The  Minio Client aka mc is Open Source and compatible with S3.
To list all the incomplete upload on an associated bucket you use the following command
(note that s3 here refers to the name of the host YOU define in your configuration file):

$ mc ls -I s3/mybucketname

And to remove all incomplete uploads to a bucket:

$ mc rm -I -r --force s3/mybucketname

I = incomplete r = recursive f = with force option

For example:

mc ls s3/mikeo-status-test-east -I
[2019-07-23 15:35:21 EDT]     0B UP/10GB-aws.bin
[2019-07-23 16:16:29 EDT]   16MiB UP/10GB-aws.bin
[2019-09-12 17:25:25 EDT] 2.3GiB UP/10GB-aws.bin
[2019-08-04 08:18:56 EDT]   88MiB UP/10GB-aws.bin
[2019-08-20 11:38:40 EDT] 576MiB UP/10GB-aws.bin
[2019-08-24 12:49:12 EDT] 688MiB UP/10GB-aws.bin
[2019-08-04 10:21:37 EDT] 272MiB UP/10GB-packet.bin
[2019-07-18 12:23:42 EDT]   80MiB UP/10GB-packet.bin
[2019-07-23 15:35:25 EDT]     0B UP/10GB-packet.bin
[2019-07-23 16:32:06 EDT]   64MiB UP/10GB-packet.bin
[2019-08-04 08:19:42 EDT] 1.3GiB UP/10GB-packet.bin
[2019-06-05 16:08:51 EDT]     0B UP/10GB.bin

 and to remove them: 

mc rm -I -r --force s3/mikeo-status-test-east
Removing `s3/mikeo-status-test-east/UP/10GB-aws.bin`.
Removing `s3/mikeo-status-test-east/UP/10GB-aws.bin`.
Removing `s3/mikeo-status-test-east/UP/10GB-aws.bin`.
Removing `s3/mikeo-status-test-east/UP/10GB-aws.bin`.
Removing `s3/mikeo-status-test-east/UP/10GB-aws.bin`.
Removing `s3/mikeo-status-test-east/UP/10GB-aws.bin`.
Removing `s3/mikeo-status-test-east/UP/10GB-packet.bin`.
Removing `s3/mikeo-status-test-east/UP/10GB-packet.bin`.
Removing `s3/mikeo-status-test-east/UP/10GB-packet.bin`.
Removing `s3/mikeo-status-test-east/UP/10GB-packet.bin`.
Removing `s3/mikeo-status-test-east/UP/10GB-packet.bin`.
Removing `s3/mikeo-status-test-east/UP/10GB.bin`.

 

Have more questions? Submit a request