libcloud.storage package

Subpackages

• libcloud.storage.drivers package
  • Submodules
  • libcloud.storage.drivers.atmos module
  • libcloud.storage.drivers.auroraobjects module
  • libcloud.storage.drivers.azure_blobs module
  • libcloud.storage.drivers.backblaze_b2 module
  • libcloud.storage.drivers.cloudfiles module
  • libcloud.storage.drivers.digitalocean_spaces module
  • libcloud.storage.drivers.dummy module
  • libcloud.storage.drivers.google_storage module
  • libcloud.storage.drivers.ktucloud module
  • libcloud.storage.drivers.local module
  • libcloud.storage.drivers.minio module
  • libcloud.storage.drivers.nimbus module
  • libcloud.storage.drivers.ninefold module
  • libcloud.storage.drivers.oss module
  • libcloud.storage.drivers.ovh module
  • libcloud.storage.drivers.rgw module
  • libcloud.storage.drivers.s3 module
  • libcloud.storage.drivers.scaleway module
  • Module contents

Submodules

libcloud.storage.base module

Provides base classes for working with storage

class libcloud.storage.base.Object(name, size, hash, extra, meta_data, container, driver)

Bases: object

Represents an object (BLOB).

Parameters:

• name (str) – Object name (must be unique per container).
• size (int) – Object size in bytes.
• hash (str) – Object hash.
• container (libcloud.storage.base.Container) – Object container.
• extra (dict) – Extra attributes.
• meta_data (dict) – Optional object meta data.
• driver (libcloud.storage.base.StorageDriver) – StorageDriver instance.

as_stream(chunk_size=None)

delete()

download(destination_path, overwrite_existing=False, delete_on_failure=True)

download_range(destination_path, start_bytes, end_bytes=None, overwrite_existing=False, delete_on_failure=True)

enable_cdn()

get_cdn_url()

range_as_stream(start_bytes, end_bytes=None, chunk_size=None)

class libcloud.storage.base.Container(name, extra, driver)

Bases: object

Represents a container (bucket) which can hold multiple objects.

Parameters:

• name (str) – Container name (must be unique).
• extra (dict) – Extra attributes.
• driver (libcloud.storage.base.StorageDriver) – StorageDriver instance.

delete()

delete_object(obj)

download_object(obj, destination_path, overwrite_existing=False, delete_on_failure=True)

download_object_as_stream(obj, chunk_size=None)

download_object_range(obj, destination_path, start_bytes, end_bytes=None, overwrite_existing=False, delete_on_failure=True)

download_object_range_as_stream(obj, start_bytes, end_bytes=None, chunk_size=None)

enable_cdn()

get_cdn_url()

get_object(object_name)

iterate_objects(prefix=None, ex_prefix=None)

list_objects(prefix=None, ex_prefix=None)

upload_object(file_path, object_name, extra=None, verify_hash=True, headers=None)

upload_object_via_stream(iterator, object_name, extra=None, headers=None)

class libcloud.storage.base.StorageDriver(key, secret=None, secure=True, host=None, port=None, api_version=None, region=None, **kwargs)

Bases: libcloud.common.base.BaseDriver

A base StorageDriver to derive from.

Parameters:

• key (str) – API key or username to be used (required)
• secret (str) – Secret password to be used (required)
• secure (bool) – Whether to use HTTPS or HTTP. Note: Some providers only support HTTPS, and it is on by default.
• host (str) – Override hostname used for connections.
• port (int) – Override port used for connections.
• api_version (str) – Optional API version. Only used by drivers which support multiple API versions.
• region (str) – Optional driver region. Only used by drivers which support multiple regions.

Return type:

None

connectionCls

alias of libcloud.common.base.ConnectionUserAndKey

create_container(container_name)

Create a new container.

Parameters:container_name (str) – Container name. Returns:Container instance on success. Return type:libcloud.storage.base.Container

delete_container(container)

Delete a container.

Parameters:container (libcloud.storage.base.Container) – Container instance Returns:True on success, False otherwise. Return type:bool

delete_object(obj)

Delete an object.

Parameters:obj (libcloud.storage.base.Object) – Object instance. Returns:bool True on success. Return type:bool

download_object(obj, destination_path, overwrite_existing=False, delete_on_failure=True)

Download an object to the specified destination path.

Parameters:

• obj (libcloud.storage.base.Object) – Object instance.
• destination_path (str) – Full path to a file or a directory where the incoming file will be saved.
• overwrite_existing (bool) – True to overwrite an existing file, defaults to False.
• delete_on_failure (bool) – True to delete a partially downloaded file if the download was not successful (hash mismatch / file size).

Returns:

True if an object has been successfully downloaded, False otherwise.

Return type:

bool

download_object_as_stream(obj, chunk_size=None)

Return a iterator which yields object data.

Parameters:

• obj (libcloud.storage.base.Object) – Object instance
• chunk_size (int) – Optional chunk size (in bytes).

Return type:

iterator of bytes

download_object_range(obj, destination_path, start_bytes, end_bytes=None, overwrite_existing=False, delete_on_failure=True)

Download part of an object.

Parameters:

• obj (libcloud.storage.base.Object) – Object instance.
• destination_path (str) – Full path to a file or a directory where the incoming file will be saved.
• start_bytes (int) – Start byte offset (inclusive) for the range download. Offset is 0 index based so the first byte in file file is “0”.
• end_bytes (int) – End byte offset (non-inclusive) for the range download. If not provided, it will default to the end of the file.
• overwrite_existing (bool) – True to overwrite an existing file, defaults to False.
• delete_on_failure (bool) – True to delete a partially downloaded file if the download was not successful (hash mismatch / file size).

Returns:

True if an object has been successfully downloaded, False otherwise.

Return type:

bool

download_object_range_as_stream(obj, start_bytes, end_bytes=None, chunk_size=None)

Return a iterator which yields range / part of the object data.

Parameters:

• obj (libcloud.storage.base.Object) – Object instance
• start_bytes (int) – Start byte offset (inclusive) for the range download. Offset is 0 index based so the first byte in file file is “0”.
• end_bytes (int) – End byte offset (non-inclusive) for the range download. If not provided, it will default to the end of the file.
• chunk_size (int) – Optional chunk size (in bytes).

Return type:

iterator of bytes

enable_container_cdn(container)

Enable container CDN.

Parameters:container (libcloud.storage.base.Container) – Container instance Return type:bool

enable_object_cdn(obj)

Enable object CDN.

Parameters:obj (libcloud.storage.base.Object) – Object instance Return type:bool

get_container(container_name)

Return a container instance.

Parameters:container_name (str) – Container name. Returns:Container instance. Return type:libcloud.storage.base.Container

get_container_cdn_url(container)

Return a container CDN URL.

Parameters:container (libcloud.storage.base.Container) – Container instance Returns:A CDN URL for this container. Return type:str

get_object(container_name, object_name)

Return an object instance.

Parameters:

• container_name (str) – Container name.
• object_name (str) – Object name.

Returns:

Object instance.

Return type:

libcloud.storage.base.Object

get_object_cdn_url(obj)

Return an object CDN URL.

Parameters:obj (libcloud.storage.base.Object) – Object instance Returns:A CDN URL for this object. Return type:str

hash_type = 'md5'

iterate_container_objects(container, prefix=None, ex_prefix=None)

Return a iterator of objects for the given container.

Parameters:

• container (libcloud.storage.base.Container) – Container instance
• prefix (str) – Filter objects starting with a prefix.
• ex_prefix (str) – (Deprecated.) Filter objects starting with a prefix.

Returns:

A iterator of Object instances.

Return type:

iterator of libcloud.storage.base.Object

iterate_containers()

Return a iterator of containers for the given account

Returns:A iterator of Container instances. Return type:iterator of libcloud.storage.base.Container

list_container_objects(container, prefix=None, ex_prefix=None)

Return a list of objects for the given container.

Parameters:

• container (libcloud.storage.base.Container) – Container instance.
• prefix (str) – Filter objects starting with a prefix.
• ex_prefix (str) – (Deprecated.) Filter objects starting with a prefix.

Returns:

A list of Object instances.

Return type:

list of libcloud.storage.base.Object

list_containers()

Return a list of containers.

Returns:A list of Container instances. Return type:list of Container

name = None

strict_mode = False

supports_chunked_encoding = False

upload_object(file_path, container, object_name, extra=None, verify_hash=True, headers=None)

Upload an object currently located on a disk.

Parameters:

• file_path (str) – Path to the object on disk.
• container (libcloud.storage.base.Container) – Destination container.
• object_name (str) – Object name.
• verify_hash (bool) – Verify hash
• extra (dict) – Extra attributes (driver specific). (optional)
• headers (dict) – (optional) Additional request headers, such as CORS headers. For example: headers = {‘Access-Control-Allow-Origin’: ‘http://mozilla.com’}

Return type:

libcloud.storage.base.Object

upload_object_via_stream(iterator, container, object_name, extra=None, headers=None)

Upload an object using an iterator.

If a provider supports it, chunked transfer encoding is used and you don’t need to know in advance the amount of data to be uploaded.

Otherwise if a provider doesn’t support it, iterator will be exhausted so a total size for data to be uploaded can be determined.

Note: Exhausting the iterator means that the whole data must be buffered in memory which might result in memory exhausting when uploading a very large object.

If a file is located on a disk you are advised to use upload_object function which uses fs.stat function to determine the file size and it doesn’t need to buffer whole object in the memory.

Parameters:

• iterator (object) – An object which implements the iterator interface.
• container (libcloud.storage.base.Container) – Destination container.
• object_name (str) – Object name.
• extra (dict) – (optional) Extra attributes (driver specific). Note: This dictionary must contain a ‘content_type’ key which represents a content type of the stored object.
• headers (dict) – (optional) Additional request headers, such as CORS headers. For example: headers = {‘Access-Control-Allow-Origin’: ‘http://mozilla.com’}

Return type:

libcloud.storage.base.Object

libcloud.storage.providers module

libcloud.storage.providers.get_driver(provider)

libcloud.storage.providers.set_driver(provider, module, klass)

libcloud.storage.types module

class libcloud.storage.types.Provider

Bases: object

Defines for each of the supported providers

Non-Dummy drivers are sorted in alphabetical order. Please preserve this ordering when adding new drivers.

Variables:

• DUMMY – Example provider
• ALIYUN_OSS – Aliyun OSS storage driver
• AURORAOBJECTS – AuroraObjects storage driver
• AZURE_BLOBS – Azure Blob Storage driver
• BACKBLAZE_B2 – Backblaze B2 Cloud Storage driver
• CLOUDFILES – CloudFiles
• DIGITALOCEAN_SPACES – Digital Ocean Spaces driver

:cvar GOOGLE_STORAGE Google Storage :cvar KTUCLOUD: KT UCloud Storage driver :cvar LOCAL: Local storage driver :cvar NIMBUS: Nimbus.io driver :cvar NINEFOLD: Ninefold :cvar OPENSTACK_SWIFT: OpenStack Swift driver :cvar S3: Amazon S3 US :cvar S3_AP_NORTHEAST: Amazon S3 Asia North East (Tokyo) :cvar S3_AP_NORTHEAST1: Amazon S3 Asia North East (Tokyo) :cvar S3_AP_NORTHEAST2: Amazon S3 Asia North East (Seoul) :cvar S3_AP_SOUTH: Amazon S3 Asia South (Mumbai) :cvar S3_AP_SOUTHEAST: Amazon S3 Asia South East (Singapore) :cvar S3_AP_SOUTHEAST2: Amazon S3 Asia South East 2 (Sydney) :cvar S3_CA_CENTRAL: Amazon S3 Canada (Central) :cvar S3_CN_NORTH: Amazon S3 CN North (Beijing) :cvar S3_EU_WEST: Amazon S3 EU West (Ireland) :cvar S3_EU_WEST2: Amazon S3 EU West 2 (London) :cvar S3_EU_CENTRAL: Amazon S3 EU Central (Frankfurt) :cvar S3_EU_NORTH1: Amazon S3 EU North 1 (Stockholm) :cvar S3_SA_EAST: Amazon S3 South America East (Sao Paulo) :cvar S3_US_EAST2: Amazon S3 US East 2 (Ohio) :cvar S3_US_WEST: Amazon S3 US West (Northern California) :cvar S3_US_WEST_OREGON: Amazon S3 US West 2 (Oregon) :cvar S3_US_GOV_WEST: Amazon S3 GovCloud (US) :cvar S3_RGW: S3 RGW :cvar S3_RGW_OUTSCALE: OUTSCALE S3 RGW

ALIYUN_OSS = 'aliyun_oss'

AURORAOBJECTS = 'auroraobjects'

AZURE_BLOBS = 'azure_blobs'

BACKBLAZE_B2 = 'backblaze_b2'

CLOUDFILES = 'cloudfiles'

CLOUDFILES_SWIFT = 'cloudfiles_swift'

CLOUDFILES_UK = 'cloudfiles_uk'

CLOUDFILES_US = 'cloudfiles_us'

DIGITALOCEAN_SPACES = 'digitalocean_spaces'

DUMMY = 'dummy'

GOOGLE_STORAGE = 'google_storage'

KTUCLOUD = 'ktucloud'

LOCAL = 'local'

MINIO = 'minio'

NIMBUS = 'nimbus'

NINEFOLD = 'ninefold'

OPENSTACK_SWIFT = 'openstack_swift'

OVH = 'ovh'

S3 = 's3'

S3_AP_NORTHEAST = 's3_ap_northeast'

S3_AP_NORTHEAST1 = 's3_ap_northeast_1'

S3_AP_NORTHEAST2 = 's3_ap_northeast_2'

S3_AP_SOUTH = 's3_ap_south'

S3_AP_SOUTHEAST = 's3_ap_southeast'

S3_AP_SOUTHEAST2 = 's3_ap_southeast2'

S3_CA_CENTRAL = 's3_ca_central'

S3_CN_NORTH = 's3_cn_north'

S3_CN_NORTHWEST = 's3_cn_northwest'

S3_EU_CENTRAL = 's3_eu_central'

S3_EU_NORTH1 = 's3_eu_north_1'

S3_EU_WEST = 's3_eu_west'

S3_EU_WEST2 = 's3_eu_west_2'

S3_RGW = 's3_rgw'

S3_RGW_OUTSCALE = 's3_rgw_outscale'

S3_SA_EAST = 's3_sa_east'

S3_US_EAST2 = 's3_us_east_2'

S3_US_GOV_WEST = 's3_us_gov_west'

S3_US_WEST = 's3_us_west'

S3_US_WEST_OREGON = 's3_us_west_oregon'

SCALEWAY = 'scaleway'

exception libcloud.storage.types.ContainerError(value, driver, container_name)

Bases: libcloud.common.types.LibcloudError

error_type = 'ContainerError'

exception libcloud.storage.types.ObjectError(value, driver, object_name)

Bases: libcloud.common.types.LibcloudError

error_type = 'ContainerError'

exception libcloud.storage.types.ContainerAlreadyExistsError(value, driver, container_name)

Bases: libcloud.storage.types.ContainerError

error_type = 'ContainerAlreadyExistsError'

exception libcloud.storage.types.ContainerDoesNotExistError(value, driver, container_name)

Bases: libcloud.storage.types.ContainerError

error_type = 'ContainerDoesNotExistError'

exception libcloud.storage.types.ContainerIsNotEmptyError(value, driver, container_name)

Bases: libcloud.storage.types.ContainerError

error_type = 'ContainerIsNotEmptyError'

exception libcloud.storage.types.ObjectDoesNotExistError(value, driver, object_name)

Bases: libcloud.storage.types.ObjectError

error_type = 'ObjectDoesNotExistError'

exception libcloud.storage.types.ObjectHashMismatchError(value, driver, object_name)

Bases: libcloud.storage.types.ObjectError

error_type = 'ObjectHashMismatchError'

exception libcloud.storage.types.InvalidContainerNameError(value, driver, container_name)

Bases: libcloud.storage.types.ContainerError

error_type = 'InvalidContainerNameError'

Module contents

Module for working with Storage