Google Storage Driver Documentation¶
Current version of the Google Storage driver in Libcloud uses S3 compatibility layer and as such, only supports XML API v1.0.
If you are a new Google Cloud Storage customers, you need to enable API v1.0 access and choose a default project in the Google Cloud Console for driver to work.
For information on how to do that, please see the official documentation.
If you don’t do that, you will get a message that the request is missing a project id header.
Known limitations¶
1. Only lower case values are supported for metadata keys¶
Google Storage driver is implemented on top of the S3 compatible XML API. In the XML API (and per AWS API docs), metadata values are sent as part of HTTP request headers which means mixed casing is not supported for metadata keys ( all the metadata keys are lower cased).
Native Google Storage JSON API does support mixed casing for metadata keys, but that API is not supported by Libcloud.
To make migrating across different providers easier and for compatibility reasons, you are encouraged to not rely on mixed casing for metadata header keys even with the providers which support it.
API Docs¶
- class libcloud.storage.drivers.google_storage.GoogleStorageDriver(key, secret=None, project=None, **kwargs)[source]¶
Driver for Google Cloud Storage.
Can authenticate via standard Google Cloud methods (Service Accounts, Installed App credentials, and GCE instance service accounts)
Examples:
Service Accounts:
driver = GoogleStorageDriver(key=client_email, secret=private_key, ...)
Installed Application:
driver = GoogleStorageDriver(key=client_id, secret=client_secret, ...)
From GCE instance:
driver = GoogleStorageDriver(key=foo, secret=bar, ...)
Can also authenticate via Google Cloud Storage’s S3 HMAC interoperability API. S3 user keys are 20 alphanumeric characters, starting with GOOG.
Example:
driver = GoogleStorageDriver(key='GOOG0123456789ABCXYZ', secret=key_secret)
- Parameters:
key (
str
) – API key or username to be used (required)secret (
str
) – Secret password to be used (required)secure (
bool
) – Whether to use HTTPS or HTTP. Note: Some providers only support HTTPS, and it is on by default.host (
str
) – Override hostname used for connections.port (
int
) – Override port used for connections.api_version (
str
) – Optional API version. Only used by drivers which support multiple API versions.region (
str
) – Optional driver region. Only used by drivers which support multiple regions.
- Return type:
None
- connectionCls¶
alias of
GoogleStorageConnection
- create_container(container_name)¶
Create a new container.
- Parameters:
container_name (
str
) – Container name.- Returns:
Container instance on success.
- Return type:
- delete_container(container)¶
Delete a container.
- Parameters:
container (
libcloud.storage.base.Container
) – Container instance- Returns:
True
on success,False
otherwise.- Return type:
bool
- delete_object(obj)¶
Delete an object.
- Parameters:
obj (
libcloud.storage.base.Object
) – Object instance.- Returns:
bool
True on success.- Return type:
bool
- download_object(obj, destination_path, overwrite_existing=False, delete_on_failure=True)¶
Download an object to the specified destination path.
- Parameters:
obj (
libcloud.storage.base.Object
) – Object instance.destination_path (
str
) – Full path to a file or a directory where the incoming file will be saved.overwrite_existing (
bool
) – True to overwrite an existing file, defaults to False.delete_on_failure (
bool
) – True to delete a partially downloaded file if the download was not successful (hash mismatch / file size).
- Returns:
True if an object has been successfully downloaded, False otherwise.
- Return type:
bool
- download_object_as_stream(obj, chunk_size=None)¶
Return a iterator which yields object data.
- Parameters:
obj (
libcloud.storage.base.Object
) – Object instancechunk_size (
int
) – Optional chunk size (in bytes).
- Return type:
iterator
ofbytes
- download_object_range(obj, destination_path, start_bytes, end_bytes=None, overwrite_existing=False, delete_on_failure=True)¶
Download part of an object.
- Parameters:
obj (
libcloud.storage.base.Object
) – Object instance.destination_path (
str
) – Full path to a file or a directory where the incoming file will be saved.start_bytes (
int
) – Start byte offset (inclusive) for the range download. Offset is 0 index based so the first byte in file file is “0”.end_bytes (
int
) – End byte offset (non-inclusive) for the range download. If not provided, it will default to the end of the file.overwrite_existing (
bool
) – True to overwrite an existing file, defaults to False.delete_on_failure (
bool
) – True to delete a partially downloaded file if the download was not successful (hash mismatch / file size).
- Returns:
True if an object has been successfully downloaded, False otherwise.
- Return type:
bool
- download_object_range_as_stream(obj, start_bytes, end_bytes=None, chunk_size=None)¶
Return a iterator which yields range / part of the object data.
- Parameters:
obj (
libcloud.storage.base.Object
) – Object instancestart_bytes (
int
) – Start byte offset (inclusive) for the range download. Offset is 0 index based so the first byte in file file is “0”.end_bytes (
int
) – End byte offset (non-inclusive) for the range download. If not provided, it will default to the end of the file.chunk_size (
int
) – Optional chunk size (in bytes).
- Return type:
iterator
ofbytes
- enable_container_cdn(container: Container) bool ¶
Enable container CDN.
- Parameters:
container (
libcloud.storage.base.Container
) – Container instance- Return type:
bool
- enable_object_cdn(obj: Object) bool ¶
Enable object CDN.
- Parameters:
obj (
libcloud.storage.base.Object
) – Object instance- Return type:
bool
- ex_cleanup_all_multipart_uploads(container, prefix=None)¶
Extension method for removing all partially completed S3 multipart uploads.
- Parameters:
container (
Container
) – The container holding the uploadsprefix (
str
) – Delete only uploads of objects with this prefix
- ex_delete_permissions(container_name, object_name=None, entity=None)[source]¶
Delete permissions for an ACL entity on a container or object.
- Parameters:
container_name (
str
) – The container name.object_name (
str
) – The object name. Optional. Not providing an object will delete a container permission.entity (
str
orNone
) – The entity to whose permission will be deleted. Optional. If not provided, the role will be applied to the authenticated user, if using an OAuth2 authentication scheme.
- ex_get_permissions(container_name, object_name=None)[source]¶
Return the permissions for the currently authenticated user.
- Parameters:
container_name (
str
) – The container name.object_name (
str
orNone
) – The object name. Optional. Not providing an object will return only container permissions.
- Returns:
A tuple of container and object permissions.
- Return type:
tuple
of (int
,int
orNone
) from ContainerPermissions and ObjectPermissions, respectively.
- ex_iterate_multipart_uploads(container, prefix=None, delimiter=None)¶
Extension method for listing all in-progress S3 multipart uploads.
Each multipart upload which has not been committed or aborted is considered in-progress.
- Parameters:
container (
Container
) – The container holding the uploadsprefix (
str
) – Print only uploads of objects with this prefixdelimiter (
str
) – The object/key names are grouped based on being split by this delimiter
- Returns:
A generator of S3MultipartUpload instances.
- Return type:
generator
ofS3MultipartUpload
- ex_set_permissions(container_name, object_name=None, entity=None, role=None)[source]¶
Set the permissions for an ACL entity on a container or an object.
- Parameters:
container_name (
str
) – The container name.object_name (
str
) – The object name. Optional. Not providing an object will apply the acl to the container.entity (
str
) – The entity to which apply the role. Optional. If not provided, the role will be applied to the authenticated user, if using an OAuth2 authentication scheme.role (
int
from ContainerPermissions or ObjectPermissions orstr
.) – The permission/role to set on the entity.
- Raises:
ValueError – If no entity was given, but was required. Or if the role isn’t valid for the bucket or object.
- get_container(container_name)¶
Return a container instance.
- Parameters:
container_name (
str
) – Container name.- Returns:
Container
instance.- Return type:
- get_container_cdn_url(container: Container) str ¶
Return a container CDN URL.
- Parameters:
container (
libcloud.storage.base.Container
) – Container instance- Returns:
A CDN URL for this container.
- Return type:
str
- get_object(container_name, object_name)¶
Return an object instance.
- Parameters:
container_name (
str
) – Container name.object_name (
str
) – Object name.
- Returns:
Object
instance.- Return type:
- get_object_cdn_url(obj: Object) str ¶
Return an object CDN URL.
- Parameters:
obj (
libcloud.storage.base.Object
) – Object instance- Returns:
A CDN URL for this object.
- Return type:
str
- iterate_container_objects(container, prefix=None, ex_prefix=None)¶
Return a generator of objects for the given container.
- Parameters:
container (
Container
) – Container instanceprefix (
str
) – Only return objects starting with prefixex_prefix (
str
) – Only return objects starting with ex_prefix
- Returns:
A generator of Object instances.
- Return type:
generator
ofObject
- iterate_containers()¶
Return a iterator of containers for the given account
- Returns:
A iterator of Container instances.
- Return type:
iterator
oflibcloud.storage.base.Container
- jsonConnectionCls¶
alias of
GoogleStorageJSONConnection
- list_container_objects(container: Container, prefix: str | None = None, ex_prefix: str | None = None) List[Object] ¶
Return a list of objects for the given container.
- Parameters:
container (
libcloud.storage.base.Container
) – Container instance.prefix (
str
) – Filter objects starting with a prefix.ex_prefix (
str
) – (Deprecated.) Filter objects starting with a prefix.
- Returns:
A list of Object instances.
- Return type:
list
oflibcloud.storage.base.Object
- list_containers() List[Container] ¶
Return a list of containers.
- Returns:
A list of Container instances.
- Return type:
list
ofContainer
- upload_object(file_path, container, object_name, extra=None, verify_hash=True, headers=None, ex_storage_class=None)¶
@inherits:
StorageDriver.upload_object
- Parameters:
ex_storage_class (
str
) – Storage class
- upload_object_via_stream(iterator, container, object_name, extra=None, headers=None, ex_storage_class=None)¶
@inherits:
StorageDriver.upload_object_via_stream
- Parameters:
ex_storage_class (
str
) – Storage class