Package org.nuxeo.ecm.core.storage.sql
Class S3BinaryManager
- java.lang.Object
-
- org.nuxeo.ecm.core.blob.binary.AbstractBinaryManager
-
- org.nuxeo.ecm.core.blob.binary.CachingBinaryManager
-
- org.nuxeo.ecm.blob.AbstractCloudBinaryManager
-
- org.nuxeo.ecm.core.storage.sql.S3BinaryManager
-
- All Implemented Interfaces:
S3ManagedTransfer
,BinaryManager
,BlobProvider
- Direct Known Subclasses:
CloudFrontBinaryManager
public class S3BinaryManager extends AbstractCloudBinaryManager implements S3ManagedTransfer
A Binary Manager that stores binaries as S3 BLOBsThe BLOBs are cached locally on first access for efficiency.
Because the BLOB length can be accessed independently of the binary stream, it is also cached in a simple text file if accessed before the stream.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static class
S3BinaryManager.S3BinaryGarbageCollector
Garbage collector for S3 binaries that stores the marked (in use) binaries in memory.class
S3BinaryManager.S3FileStorage
-
Field Summary
-
Fields inherited from class org.nuxeo.ecm.blob.AbstractCloudBinaryManager
CACHE_COUNT_PROPERTY, CACHE_MIN_AGE_PROPERTY, CACHE_SIZE_PROPERTY, DEFAULT_CACHE_COUNT, DEFAULT_CACHE_MIN_AGE, DEFAULT_CACHE_SIZE, DEFAULT_DIRECTDOWNLOAD, DEFAULT_DIRECTDOWNLOAD_EXPIRE, DIGEST_ALGORITHM_PROPERTY, directDownload, DIRECTDOWNLOAD_EXPIRE_PROPERTY, DIRECTDOWNLOAD_PROPERTY, directDownloadExpire, namespace, transientFlag
-
Fields inherited from class org.nuxeo.ecm.core.blob.binary.CachingBinaryManager
cachedir, fileCache, fileStorage
-
Fields inherited from class org.nuxeo.ecm.core.blob.binary.AbstractBinaryManager
blobProviderId, DEFAULT_DEPTH, DEFAULT_DIGEST, descriptor, digestPattern, DIGESTS_BY_LENGTH, garbageCollector, MAX_BUF_SIZE, MD5_DIGEST, MD5_DIGEST_LENGTH, MIN_BUF_SIZE, properties, SHA1_DIGEST, SHA1_DIGEST_LENGTH, SHA256_DIGEST, SHA256_DIGEST_LENGTH
-
Fields inherited from interface org.nuxeo.ecm.core.blob.binary.BinaryManager
PROP_KEY, PROP_PATH
-
-
Constructor Summary
Constructors Constructor Description S3BinaryManager()
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Deprecated Methods Modifier and Type Method Description protected void
abortOldMultipartUploadsInternal()
protected void
abortOldUploads()
Aborts uploads that crashed and are older than 1 day.boolean
canAccessBucket()
Used in the healthCheck; the transferManager should be initialized and the bucket accessiblevoid
clear()
INTERNAL (TESTS).void
close()
Closes the binary manager and releases all resources and temporary objects held by it.protected String
copyBlob(S3BinaryManager sourceBlobProvider, String digest)
Copies a blob.com.amazonaws.services.s3.AmazonS3
getAmazonS3()
Gets AmazonS3.com.amazonaws.auth.AWSCredentialsProvider
getAwsCredentialsProvider()
Gets the AWSCredentialsProvider.String
getBucketName()
Gets the bucket name.String
getBucketPrefix()
Gets the bucket prefix.protected FileStorage
getFileStorage()
protected URI
getRemoteUri(String digest, ManagedBlob blob, javax.servlet.http.HttpServletRequest servletRequest)
protected String
getSystemPropertyPrefix()
Gets the prefix used for configuration using system properties.com.amazonaws.services.s3.transfer.TransferManager
getTransferManager()
Returns theTransferManager
.protected BinaryGarbageCollector
instantiateGarbageCollector()
static boolean
isMD5(String digest)
Deprecated.since 11.1, now unusedprotected static boolean
isMissingKey(com.amazonaws.AmazonClientException e)
long
lengthOfBlob(String digest)
void
removeBinaries(Collection<String> digests)
Remove definitively a set of binariesprotected void
removeBinary(String digest)
protected void
setupCloudClient()
Configure Cloud client using propertiesprotected void
waitForCopyCompletion(com.amazonaws.services.s3.transfer.Copy copy)
String
writeBlob(Blob blob)
Writes aBlob
to storage and returns information about it.-
Methods inherited from class org.nuxeo.ecm.blob.AbstractCloudBinaryManager
getBinaryManager, getBooleanProperty, getContentDispositionHeader, getContentTypeHeader, getDefaultDigestAlgorithm, getIntProperty, getIntProperty, getProperties, getProperty, getProperty, getSystemPropertyName, getURI, initialize, isDirectDownload, isTransient, performsExternalAccessControl, readBlob, supportsUserUpdate, supportsUserUpdateDefaultTrue
-
Methods inherited from class org.nuxeo.ecm.core.blob.binary.CachingBinaryManager
getBinary, getBinary, getFile, initializeCache, initializeCache, initializeCache
-
Methods inherited from class org.nuxeo.ecm.core.blob.binary.AbstractBinaryManager
computeDigestPattern, getBinary, getDescriptor, getDigestAlgorithm, getGarbageCollector, isValidDigest, setDescriptor, storeAndDigest, toHexString
-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
Methods inherited from interface org.nuxeo.ecm.core.blob.BlobProvider
allowByteRange, allowDirectDownload, canConvert, deleteBlob, getAppLinks, getAvailableConversions, getBinaryGarbageCollector, getFile, getStatus, getStream, getStream, getThumbnail, hasCreateFromKeyPermission, isColdStorageMode, isRecordMode, isTransactional, isVersion, readBlob, supportsSync, updateBlob, writeBlob
-
-
-
-
Field Detail
-
SYSTEM_PROPERTY_PREFIX
public static final String SYSTEM_PROPERTY_PREFIX
- See Also:
- Constant Field Values
-
BUCKET_NAME_PROPERTY
public static final String BUCKET_NAME_PROPERTY
- See Also:
- Constant Field Values
-
BUCKET_PREFIX_PROPERTY
public static final String BUCKET_PREFIX_PROPERTY
- See Also:
- Constant Field Values
-
BUCKET_REGION_PROPERTY
public static final String BUCKET_REGION_PROPERTY
- See Also:
- Constant Field Values
-
AWS_ID_PROPERTY
public static final String AWS_ID_PROPERTY
- See Also:
- Constant Field Values
-
AWS_SECRET_PROPERTY
public static final String AWS_SECRET_PROPERTY
- See Also:
- Constant Field Values
-
AWS_SESSION_TOKEN_PROPERTY
public static final String AWS_SESSION_TOKEN_PROPERTY
- Since:
- 10.10
- See Also:
- Constant Field Values
-
CONNECTION_MAX_PROPERTY
public static final String CONNECTION_MAX_PROPERTY
AWS ClientConfiguration default 50- See Also:
- Constant Field Values
-
CONNECTION_RETRY_PROPERTY
public static final String CONNECTION_RETRY_PROPERTY
AWS ClientConfiguration default 3 (with exponential backoff)- See Also:
- Constant Field Values
-
CONNECTION_TIMEOUT_PROPERTY
public static final String CONNECTION_TIMEOUT_PROPERTY
AWS ClientConfiguration default 50*1000 = 50s- See Also:
- Constant Field Values
-
SOCKET_TIMEOUT_PROPERTY
public static final String SOCKET_TIMEOUT_PROPERTY
AWS ClientConfiguration default 50*1000 = 50s- See Also:
- Constant Field Values
-
KEYSTORE_FILE_PROPERTY
public static final String KEYSTORE_FILE_PROPERTY
- See Also:
- Constant Field Values
-
KEYSTORE_PASS_PROPERTY
public static final String KEYSTORE_PASS_PROPERTY
- See Also:
- Constant Field Values
-
SERVERSIDE_ENCRYPTION_PROPERTY
public static final String SERVERSIDE_ENCRYPTION_PROPERTY
- See Also:
- Constant Field Values
-
SERVERSIDE_ENCRYPTION_KMS_KEY_PROPERTY
public static final String SERVERSIDE_ENCRYPTION_KMS_KEY_PROPERTY
- See Also:
- Constant Field Values
-
PRIVKEY_ALIAS_PROPERTY
public static final String PRIVKEY_ALIAS_PROPERTY
- See Also:
- Constant Field Values
-
PRIVKEY_PASS_PROPERTY
public static final String PRIVKEY_PASS_PROPERTY
- See Also:
- Constant Field Values
-
ENDPOINT_PROPERTY
public static final String ENDPOINT_PROPERTY
- See Also:
- Constant Field Values
-
PATHSTYLEACCESS_PROPERTY
public static final String PATHSTYLEACCESS_PROPERTY
- Since:
- 10.3
- See Also:
- Constant Field Values
-
ACCELERATE_MODE_PROPERTY
public static final String ACCELERATE_MODE_PROPERTY
- Since:
- 11.1
- See Also:
- Constant Field Values
-
DIRECTDOWNLOAD_PROPERTY_COMPAT
public static final String DIRECTDOWNLOAD_PROPERTY_COMPAT
- See Also:
- Constant Field Values
-
DIRECTDOWNLOAD_EXPIRE_PROPERTY_COMPAT
public static final String DIRECTDOWNLOAD_EXPIRE_PROPERTY_COMPAT
- See Also:
- Constant Field Values
-
USER_AGENT_PREFIX_PROPERTY
public static final String USER_AGENT_PREFIX_PROPERTY
- Since:
- 2021.15
- See Also:
- Constant Field Values
-
USER_AGENT_SUFFIX_PROPERTY
public static final String USER_AGENT_SUFFIX_PROPERTY
- Since:
- 2021.15
- See Also:
- Constant Field Values
-
DELIMITER
public static final String DELIMITER
- See Also:
- Constant Field Values
-
bucketName
protected String bucketName
-
bucketNamePrefix
protected String bucketNamePrefix
-
awsCredentialsProvider
protected com.amazonaws.auth.AWSCredentialsProvider awsCredentialsProvider
-
clientConfiguration
protected com.amazonaws.ClientConfiguration clientConfiguration
-
encryptionMaterials
protected com.amazonaws.services.s3.model.EncryptionMaterials encryptionMaterials
-
isEncrypted
protected boolean isEncrypted
-
cryptoConfiguration
protected com.amazonaws.services.s3.model.CryptoConfiguration cryptoConfiguration
-
useServerSideEncryption
protected boolean useServerSideEncryption
-
serverSideKMSKeyID
protected String serverSideKMSKeyID
-
amazonS3
protected com.amazonaws.services.s3.AmazonS3 amazonS3
-
transferManager
protected com.amazonaws.services.s3.transfer.TransferManager transferManager
-
-
Method Detail
-
close
public void close()
Description copied from interface:BinaryManager
Closes the binary manager and releases all resources and temporary objects held by it.- Specified by:
close
in interfaceBinaryManager
- Specified by:
close
in interfaceBlobProvider
- Overrides:
close
in classCachingBinaryManager
-
abortOldUploads
protected void abortOldUploads()
Aborts uploads that crashed and are older than 1 day.- Since:
- 7.2
-
abortOldMultipartUploadsInternal
protected void abortOldMultipartUploadsInternal()
-
setupCloudClient
protected void setupCloudClient() throws IOException
Description copied from class:AbstractCloudBinaryManager
Configure Cloud client using properties- Specified by:
setupCloudClient
in classAbstractCloudBinaryManager
- Throws:
IOException
-
getTransferManager
public com.amazonaws.services.s3.transfer.TransferManager getTransferManager()
Description copied from interface:S3ManagedTransfer
Returns theTransferManager
.- Specified by:
getTransferManager
in interfaceS3ManagedTransfer
-
removeBinary
protected void removeBinary(String digest)
-
clear
public void clear()
INTERNAL (TESTS). Clears the binary manager of all its data.- Since:
- 11.5
-
getSystemPropertyPrefix
protected String getSystemPropertyPrefix()
Description copied from class:AbstractCloudBinaryManager
Gets the prefix used for configuration using system properties.- Specified by:
getSystemPropertyPrefix
in classAbstractCloudBinaryManager
-
instantiateGarbageCollector
protected BinaryGarbageCollector instantiateGarbageCollector()
- Specified by:
instantiateGarbageCollector
in classAbstractCloudBinaryManager
-
removeBinaries
public void removeBinaries(Collection<String> digests)
Description copied from interface:BinaryManager
Remove definitively a set of binaries- Specified by:
removeBinaries
in interfaceBinaryManager
- Specified by:
removeBinaries
in classAbstractCloudBinaryManager
- Parameters:
digests
- a set of digests, must not benull
.
-
lengthOfBlob
public long lengthOfBlob(String digest)
- Returns:
- object length, or -1 if missing
-
isMissingKey
protected static boolean isMissingKey(com.amazonaws.AmazonClientException e)
-
isMD5
@Deprecated public static boolean isMD5(String digest)
Deprecated.since 11.1, now unused
-
canAccessBucket
public boolean canAccessBucket()
Used in the healthCheck; the transferManager should be initialized and the bucket accessible- Since:
- 9.3
-
getFileStorage
protected FileStorage getFileStorage()
- Specified by:
getFileStorage
in classAbstractCloudBinaryManager
-
getAwsCredentialsProvider
public com.amazonaws.auth.AWSCredentialsProvider getAwsCredentialsProvider()
Gets the AWSCredentialsProvider.- Since:
- 10.2
-
getAmazonS3
public com.amazonaws.services.s3.AmazonS3 getAmazonS3()
Gets AmazonS3.- Since:
- 10.2
-
getBucketName
public String getBucketName()
Gets the bucket name.- Since:
- 11.1
-
getBucketPrefix
public String getBucketPrefix()
Gets the bucket prefix.- Since:
- 11.1
-
writeBlob
public String writeBlob(Blob blob) throws IOException
Description copied from interface:BlobProvider
Writes aBlob
to storage and returns information about it.Called to store a user-created blob.
- Specified by:
writeBlob
in interfaceBlobProvider
- Overrides:
writeBlob
in classAbstractCloudBinaryManager
- Parameters:
blob
- the blob- Returns:
- the blob key
- Throws:
IOException
-
copyBlob
protected String copyBlob(S3BinaryManager sourceBlobProvider, String digest) throws IOException
Copies a blob. Returnsnull
if the copy was not possible.- Parameters:
sourceBlobProvider
- the source blob providerdigest
- the source blob key- Returns:
- the copied blob key, or
null
if the copy was not possible - Throws:
IOException
- Since:
- 10.1
-
waitForCopyCompletion
protected void waitForCopyCompletion(com.amazonaws.services.s3.transfer.Copy copy)
- Since:
- 2021.15
-
getRemoteUri
protected URI getRemoteUri(String digest, ManagedBlob blob, javax.servlet.http.HttpServletRequest servletRequest) throws IOException
- Overrides:
getRemoteUri
in classAbstractCloudBinaryManager
- Throws:
IOException
-
-