Amazon S3 Storage Driver Documentation

Amazon Simple Storage Service (Amazon S3) is an online cloud storage service from Amazon Web Services.

Multipart uploads

Amazon S3 driver supports multipart uploads which means you can upload objects with a total size of up to 5 TB.

Multipart upload works by splitting an object in multiple parts and uploading those parts to S3. After all the parts of the object are uploaded, Amazon S3 assembles those parts and creates an object.

If you use libcloud.storage.base.StorageDriver.upload_object_via_stream() method, Libcloud transparently handles all the splitting and uploading of the parts for you.

By default, to prevent excessive buffering and use of memory, each part is 5 MB in size. This is also the smallest size of a part you can use with the multi part upload.

Examples

1. Uploading a very large file using upload_object_via_stream method

This approach shows how you can upload a very large file using upload_object_via_stream method.

Keep in mind that exactly the same approach and method can also be used for uploading other arbitrary sized files. There is no minimum size limit and you can even upload / create empty objects.

from libcloud.storage.types import Provider
from libcloud.storage.providers import get_driver

# Path to a very large file you want to upload
FILE_PATH = '/home/user/myfile.tar.gz'

cls = get_driver(Provider.S3)
driver = cls('api key', 'api secret key')

container = driver.get_container(container_name='my-backups-12345')

# This method blocks until all the parts have been uploaded.
extra = {'content_type': 'application/octet-stream'}

with open(FILE_PATH, 'rb') as iterator:
    obj = driver.upload_object_via_stream(iterator=iterator,
                                          container=container,
                                          object_name='backup.tar.gz',
                                          extra=extra)

2. Specifying canned ACL when uploading an object

If you want to specify custom ACL when uploading an object, you can do so by passing extra argument with the acl attribute to the upload methods.

Valid values for this attribute are:

  • private (default)
  • public-read
  • public-read-write
  • authenticated-read
  • bucket-owner-read
  • bucket-owner-full-control

For example:

from libcloud.storage.types import Provider
from libcloud.storage.providers import get_driver

FILE_PATH = '/home/user/myfile.tar.gz'

cls = get_driver(Provider.S3)
driver = cls('api key', 'api secret key')

container = driver.get_container(container_name='my-backups-12345')

# This method blocks until all the parts have been uploaded.
extra = {'content_type': 'application/octet-stream',
         'acl': 'public-read'}

with open(FILE_PATH, 'rb') as iterator:
    obj = driver.upload_object_via_stream(iterator=iterator,
                                          container=container,
                                          object_name='backup.tar.gz',
                                          extra=extra)

For more information about the canned ACLs, please refer to the Canned ACL section of the Amazon S3 documentation.