libcloud.storage package¶
Subpackages¶
- libcloud.storage.drivers package
- Submodules
- libcloud.storage.drivers.atmos module
- libcloud.storage.drivers.auroraobjects module
- libcloud.storage.drivers.azure_blobs module
- libcloud.storage.drivers.backblaze_b2 module
- libcloud.storage.drivers.cloudfiles module
- libcloud.storage.drivers.digitalocean_spaces module
- libcloud.storage.drivers.dummy module
- libcloud.storage.drivers.google_storage module
- libcloud.storage.drivers.ktucloud module
- libcloud.storage.drivers.local module
- libcloud.storage.drivers.minio module
- libcloud.storage.drivers.nimbus module
- libcloud.storage.drivers.ninefold module
- libcloud.storage.drivers.oss module
- libcloud.storage.drivers.rgw module
- libcloud.storage.drivers.s3 module
- libcloud.storage.drivers.scaleway module
- Module contents
Submodules¶
libcloud.storage.base module¶
Provides base classes for working with storage
- class libcloud.storage.base.Container(name, extra, driver)[source]¶
Bases:
object
Represents a container (bucket) which can hold multiple objects.
- Parameters
name (
str
) – Container name (must be unique).extra (
dict
) – Extra attributes.driver (
libcloud.storage.base.StorageDriver
) – StorageDriver instance.
- class libcloud.storage.base.Object(name, size, hash, extra, meta_data, container, driver)[source]¶
Bases:
object
Represents an object (BLOB).
- Parameters
name (
str
) – Object name (must be unique per container).size (
int
) – Object size in bytes.hash (
str
) – Object hash.container (
libcloud.storage.base.Container
) – Object container.extra (
dict
) – Extra attributes.meta_data (
dict
) – Optional object meta data.driver (
libcloud.storage.base.StorageDriver
) – StorageDriver instance.
- class libcloud.storage.base.StorageDriver(key, secret=None, secure=True, host=None, port=None, api_version=None, region=None, **kwargs)[source]¶
Bases:
libcloud.common.base.BaseDriver
A base StorageDriver to derive from.
- Parameters
key (
str
) – API key or username to be used (required)secret (
str
) – Secret password to be used (required)secure (
bool
) – Whether to use HTTPS or HTTP. Note: Some providers only support HTTPS, and it is on by default.host (
str
) – Override hostname used for connections.port (
int
) – Override port used for connections.api_version (
str
) – Optional API version. Only used by drivers which support multiple API versions.region (
str
) – Optional driver region. Only used by drivers which support multiple regions.
- Return type
None
- connectionCls¶
- create_container(container_name)[source]¶
Create a new container.
- Parameters
container_name (
str
) – Container name.- Returns
Container instance on success.
- Return type
- delete_container(container)[source]¶
Delete a container.
- Parameters
container (
libcloud.storage.base.Container
) – Container instance- Returns
True
on success,False
otherwise.- Return type
bool
- delete_object(obj)[source]¶
Delete an object.
- Parameters
obj (
libcloud.storage.base.Object
) – Object instance.- Returns
bool
True on success.- Return type
bool
- download_object(obj, destination_path, overwrite_existing=False, delete_on_failure=True)[source]¶
Download an object to the specified destination path.
- Parameters
obj (
libcloud.storage.base.Object
) – Object instance.destination_path (
str
) – Full path to a file or a directory where the incoming file will be saved.overwrite_existing (
bool
) – True to overwrite an existing file, defaults to False.delete_on_failure (
bool
) – True to delete a partially downloaded file if the download was not successful (hash mismatch / file size).
- Returns
True if an object has been successfully downloaded, False otherwise.
- Return type
bool
- download_object_as_stream(obj, chunk_size=None)[source]¶
Return a iterator which yields object data.
- Parameters
obj (
libcloud.storage.base.Object
) – Object instancechunk_size (
int
) – Optional chunk size (in bytes).
- Return type
iterator
ofbytes
- download_object_range(obj, destination_path, start_bytes, end_bytes=None, overwrite_existing=False, delete_on_failure=True)[source]¶
Download part of an object.
- Parameters
obj (
libcloud.storage.base.Object
) – Object instance.destination_path (
str
) – Full path to a file or a directory where the incoming file will be saved.start_bytes (
int
) – Start byte offset (inclusive) for the range download. Offset is 0 index based so the first byte in file file is “0”.end_bytes (
int
) – End byte offset (non-inclusive) for the range download. If not provided, it will default to the end of the file.overwrite_existing (
bool
) – True to overwrite an existing file, defaults to False.delete_on_failure (
bool
) – True to delete a partially downloaded file if the download was not successful (hash mismatch / file size).
- Returns
True if an object has been successfully downloaded, False otherwise.
- Return type
bool
- download_object_range_as_stream(obj, start_bytes, end_bytes=None, chunk_size=None)[source]¶
Return a iterator which yields range / part of the object data.
- Parameters
obj (
libcloud.storage.base.Object
) – Object instancestart_bytes (
int
) – Start byte offset (inclusive) for the range download. Offset is 0 index based so the first byte in file file is “0”.end_bytes (
int
) – End byte offset (non-inclusive) for the range download. If not provided, it will default to the end of the file.chunk_size (
int
) – Optional chunk size (in bytes).
- Return type
iterator
ofbytes
- enable_container_cdn(container)[source]¶
Enable container CDN.
- Parameters
container (
libcloud.storage.base.Container
) – Container instance- Return type
bool
- enable_object_cdn(obj)[source]¶
Enable object CDN.
- Parameters
obj (
libcloud.storage.base.Object
) – Object instance- Return type
bool
- get_container(container_name)[source]¶
Return a container instance.
- Parameters
container_name (
str
) – Container name.- Returns
Container
instance.- Return type
- get_container_cdn_url(container)[source]¶
Return a container CDN URL.
- Parameters
container (
libcloud.storage.base.Container
) – Container instance- Returns
A CDN URL for this container.
- Return type
str
- get_object(container_name, object_name)[source]¶
Return an object instance.
- Parameters
container_name (
str
) – Container name.object_name (
str
) – Object name.
- Returns
Object
instance.- Return type
- get_object_cdn_url(obj)[source]¶
Return an object CDN URL.
- Parameters
obj (
libcloud.storage.base.Object
) – Object instance- Returns
A CDN URL for this object.
- Return type
str
- hash_type = 'md5'¶
- iterate_container_objects(container, prefix=None, ex_prefix=None)[source]¶
Return a iterator of objects for the given container.
- Parameters
container (
libcloud.storage.base.Container
) – Container instanceprefix (
str
) – Filter objects starting with a prefix.ex_prefix (
str
) – (Deprecated.) Filter objects starting with a prefix.
- Returns
A iterator of Object instances.
- Return type
iterator
oflibcloud.storage.base.Object
- iterate_containers()[source]¶
Return a iterator of containers for the given account
- Returns
A iterator of Container instances.
- Return type
iterator
oflibcloud.storage.base.Container
- list_container_objects(container, prefix=None, ex_prefix=None)[source]¶
Return a list of objects for the given container.
- Parameters
container (
libcloud.storage.base.Container
) – Container instance.prefix (
str
) – Filter objects starting with a prefix.ex_prefix (
str
) – (Deprecated.) Filter objects starting with a prefix.
- Returns
A list of Object instances.
- Return type
list
oflibcloud.storage.base.Object
- list_containers()[source]¶
Return a list of containers.
- Returns
A list of Container instances.
- Return type
list
ofContainer
- name = None¶
- strict_mode = False¶
- supports_chunked_encoding = False¶
- upload_object(file_path, container, object_name, extra=None, verify_hash=True, headers=None)[source]¶
Upload an object currently located on a disk.
- Parameters
file_path (
str
) – Path to the object on disk.container (
libcloud.storage.base.Container
) – Destination container.object_name (
str
) – Object name.verify_hash (
bool
) – Verify hashextra (
dict
) – Extra attributes (driver specific). (optional)headers (
dict
) – (optional) Additional request headers, such as CORS headers. For example: headers = {‘Access-Control-Allow-Origin’: ‘http://mozilla.com’}
- Return type
- upload_object_via_stream(iterator, container, object_name, extra=None, headers=None)[source]¶
Upload an object using an iterator.
If a provider supports it, chunked transfer encoding is used and you don’t need to know in advance the amount of data to be uploaded.
Otherwise if a provider doesn’t support it, iterator will be exhausted so a total size for data to be uploaded can be determined.
Note: Exhausting the iterator means that the whole data must be buffered in memory which might result in memory exhausting when uploading a very large object.
If a file is located on a disk you are advised to use upload_object function which uses fs.stat function to determine the file size and it doesn’t need to buffer whole object in the memory.
- Parameters
iterator (
object
) – An object which implements the iterator interface.container (
libcloud.storage.base.Container
) – Destination container.object_name (
str
) – Object name.extra (
dict
) – (optional) Extra attributes (driver specific). Note: This dictionary must contain a ‘content_type’ key which represents a content type of the stored object.headers (
dict
) – (optional) Additional request headers, such as CORS headers. For example: headers = {‘Access-Control-Allow-Origin’: ‘http://mozilla.com’}
- Return type
libcloud.storage.base.Object
libcloud.storage.providers module¶
libcloud.storage.types module¶
- exception libcloud.storage.types.ContainerAlreadyExistsError(value, driver, container_name)[source]¶
Bases:
libcloud.storage.types.ContainerError
- error_type = 'ContainerAlreadyExistsError'¶
- exception libcloud.storage.types.ContainerDoesNotExistError(value, driver, container_name)[source]¶
Bases:
libcloud.storage.types.ContainerError
- error_type = 'ContainerDoesNotExistError'¶
- exception libcloud.storage.types.ContainerError(value, driver, container_name)[source]¶
Bases:
libcloud.common.types.LibcloudError
- error_type = 'ContainerError'¶
- exception libcloud.storage.types.ContainerIsNotEmptyError(value, driver, container_name)[source]¶
Bases:
libcloud.storage.types.ContainerError
- error_type = 'ContainerIsNotEmptyError'¶
- exception libcloud.storage.types.InvalidContainerNameError(value, driver, container_name)[source]¶
Bases:
libcloud.storage.types.ContainerError
- error_type = 'InvalidContainerNameError'¶
- exception libcloud.storage.types.ObjectDoesNotExistError(value, driver, object_name)[source]¶
Bases:
libcloud.storage.types.ObjectError
- error_type = 'ObjectDoesNotExistError'¶
- exception libcloud.storage.types.ObjectError(value, driver, object_name)[source]¶
Bases:
libcloud.common.types.LibcloudError
- error_type = 'ContainerError'¶
- exception libcloud.storage.types.ObjectHashMismatchError(value, driver, object_name)[source]¶
Bases:
libcloud.storage.types.ObjectError
- error_type = 'ObjectHashMismatchError'¶
- class libcloud.storage.types.Provider[source]¶
Bases:
object
Defines for each of the supported providers
Non-Dummy drivers are sorted in alphabetical order. Please preserve this ordering when adding new drivers.
- Variables
DUMMY – Example provider
ALIYUN_OSS – Aliyun OSS storage driver
AURORAOBJECTS – AuroraObjects storage driver
AZURE_BLOBS – Azure Blob Storage driver
BACKBLAZE_B2 – Backblaze B2 Cloud Storage driver
CLOUDFILES – CloudFiles
DIGITALOCEAN_SPACES – Digital Ocean Spaces driver
:cvar GOOGLE_STORAGE Google Storage :cvar KTUCLOUD: KT UCloud Storage driver :cvar LOCAL: Local storage driver :cvar NIMBUS: Nimbus.io driver :cvar NINEFOLD: Ninefold :cvar OPENSTACK_SWIFT: OpenStack Swift driver :cvar S3: Amazon S3 US :cvar S3_AP_NORTHEAST: Amazon S3 Asia North East (Tokyo) :cvar S3_AP_NORTHEAST1: Amazon S3 Asia North East (Tokyo) :cvar S3_AP_NORTHEAST2: Amazon S3 Asia North East (Seoul) :cvar S3_AP_SOUTH: Amazon S3 Asia South (Mumbai) :cvar S3_AP_SOUTHEAST: Amazon S3 Asia South East (Singapore) :cvar S3_AP_SOUTHEAST2: Amazon S3 Asia South East 2 (Sydney) :cvar S3_CA_CENTRAL: Amazon S3 Canada (Central) :cvar S3_CN_NORTH: Amazon S3 CN North (Beijing) :cvar S3_EU_WEST: Amazon S3 EU West (Ireland) :cvar S3_EU_WEST2: Amazon S3 EU West 2 (London) :cvar S3_EU_CENTRAL: Amazon S3 EU Central (Frankfurt) :cvar S3_EU_NORTH1: Amazon S3 EU North 1 (Stockholm) :cvar S3_SA_EAST: Amazon S3 South America East (Sao Paulo) :cvar S3_US_EAST2: Amazon S3 US East 2 (Ohio) :cvar S3_US_WEST: Amazon S3 US West (Northern California) :cvar S3_US_WEST_OREGON: Amazon S3 US West 2 (Oregon) :cvar S3_US_GOV_WEST: Amazon S3 GovCloud (US) :cvar S3_RGW: S3 RGW :cvar S3_RGW_OUTSCALE: OUTSCALE S3 RGW
- ALIYUN_OSS = 'aliyun_oss'¶
- AURORAOBJECTS = 'auroraobjects'¶
- AZURE_BLOBS = 'azure_blobs'¶
- BACKBLAZE_B2 = 'backblaze_b2'¶
- CLOUDFILES = 'cloudfiles'¶
- CLOUDFILES_SWIFT = 'cloudfiles_swift'¶
- CLOUDFILES_UK = 'cloudfiles_uk'¶
- CLOUDFILES_US = 'cloudfiles_us'¶
- DIGITALOCEAN_SPACES = 'digitalocean_spaces'¶
- DUMMY = 'dummy'¶
- GOOGLE_STORAGE = 'google_storage'¶
- KTUCLOUD = 'ktucloud'¶
- LOCAL = 'local'¶
- MINIO = 'minio'¶
- NIMBUS = 'nimbus'¶
- NINEFOLD = 'ninefold'¶
- OPENSTACK_SWIFT = 'openstack_swift'¶
- S3 = 's3'¶
- S3_AP_NORTHEAST = 's3_ap_northeast'¶
- S3_AP_NORTHEAST1 = 's3_ap_northeast_1'¶
- S3_AP_NORTHEAST2 = 's3_ap_northeast_2'¶
- S3_AP_SOUTH = 's3_ap_south'¶
- S3_AP_SOUTHEAST = 's3_ap_southeast'¶
- S3_AP_SOUTHEAST2 = 's3_ap_southeast2'¶
- S3_CA_CENTRAL = 's3_ca_central'¶
- S3_CN_NORTH = 's3_cn_north'¶
- S3_CN_NORTHWEST = 's3_cn_northwest'¶
- S3_EU_CENTRAL = 's3_eu_central'¶
- S3_EU_NORTH1 = 's3_eu_north_1'¶
- S3_EU_WEST = 's3_eu_west'¶
- S3_EU_WEST2 = 's3_eu_west_2'¶
- S3_RGW = 's3_rgw'¶
- S3_RGW_OUTSCALE = 's3_rgw_outscale'¶
- S3_SA_EAST = 's3_sa_east'¶
- S3_US_EAST2 = 's3_us_east_2'¶
- S3_US_GOV_WEST = 's3_us_gov_west'¶
- S3_US_WEST = 's3_us_west'¶
- S3_US_WEST_OREGON = 's3_us_west_oregon'¶
- SCALEWAY = 'scaleway'¶
Module contents¶
Module for working with Storage