Package org.nuxeo.ecm.core.storage.sql
Class S3BinaryManager
- java.lang.Object
-
- org.nuxeo.ecm.core.blob.binary.AbstractBinaryManager
-
- org.nuxeo.ecm.core.blob.binary.CachingBinaryManager
-
- org.nuxeo.ecm.blob.AbstractCloudBinaryManager
-
- org.nuxeo.ecm.core.storage.sql.S3BinaryManager
-
- All Implemented Interfaces:
S3ManagedTransfer,BinaryManager,BlobProvider
- Direct Known Subclasses:
CloudFrontBinaryManager
public class S3BinaryManager extends AbstractCloudBinaryManager implements S3ManagedTransfer
A Binary Manager that stores binaries as S3 BLOBsThe BLOBs are cached locally on first access for efficiency.
Because the BLOB length can be accessed independently of the binary stream, it is also cached in a simple text file if accessed before the stream.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static classS3BinaryManager.S3BinaryGarbageCollectorGarbage collector for S3 binaries that stores the marked (in use) binaries in memory.classS3BinaryManager.S3FileStorage
-
Field Summary
-
Fields inherited from class org.nuxeo.ecm.blob.AbstractCloudBinaryManager
CACHE_COUNT_PROPERTY, CACHE_MIN_AGE_PROPERTY, CACHE_SIZE_PROPERTY, DEFAULT_CACHE_COUNT, DEFAULT_CACHE_MIN_AGE, DEFAULT_CACHE_SIZE, DEFAULT_DIRECTDOWNLOAD, DEFAULT_DIRECTDOWNLOAD_EXPIRE, DIGEST_ALGORITHM_PROPERTY, directDownload, DIRECTDOWNLOAD_EXPIRE_PROPERTY, DIRECTDOWNLOAD_PROPERTY, directDownloadExpire, namespace, transientFlag
-
Fields inherited from class org.nuxeo.ecm.core.blob.binary.CachingBinaryManager
cachedir, fileCache, fileStorage
-
Fields inherited from class org.nuxeo.ecm.core.blob.binary.AbstractBinaryManager
blobProviderId, DEFAULT_DEPTH, DEFAULT_DIGEST, descriptor, digestPattern, DIGESTS_BY_LENGTH, garbageCollector, MAX_BUF_SIZE, MD5_DIGEST, MD5_DIGEST_LENGTH, MIN_BUF_SIZE, properties, SHA1_DIGEST, SHA1_DIGEST_LENGTH, SHA256_DIGEST, SHA256_DIGEST_LENGTH
-
Fields inherited from interface org.nuxeo.ecm.core.blob.binary.BinaryManager
PROP_KEY, PROP_PATH
-
-
Constructor Summary
Constructors Constructor Description S3BinaryManager()
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Deprecated Methods Modifier and Type Method Description protected voidabortOldMultipartUploadsInternal()protected voidabortOldUploads()Aborts uploads that crashed and are older than 1 day.booleancanAccessBucket()Used in the healthCheck; the transferManager should be initialized and the bucket accessiblevoidclear()INTERNAL (TESTS).voidclose()Closes the binary manager and releases all resources and temporary objects held by it.protected StringcopyBlob(S3BinaryManager sourceBlobProvider, String digest)Copies a blob.com.amazonaws.services.s3.AmazonS3getAmazonS3()Gets AmazonS3.com.amazonaws.auth.AWSCredentialsProvidergetAwsCredentialsProvider()Gets the AWSCredentialsProvider.StringgetBucketName()Gets the bucket name.StringgetBucketPrefix()Gets the bucket prefix.protected FileStoragegetFileStorage()protected URIgetRemoteUri(String digest, ManagedBlob blob, javax.servlet.http.HttpServletRequest servletRequest)protected StringgetSystemPropertyPrefix()Gets the prefix used for configuration using system properties.com.amazonaws.services.s3.transfer.TransferManagergetTransferManager()Returns theTransferManager.protected BinaryGarbageCollectorinstantiateGarbageCollector()static booleanisMD5(String digest)Deprecated.since 11.1, now unusedprotected static booleanisMissingKey(com.amazonaws.AmazonClientException e)longlengthOfBlob(String digest)voidremoveBinaries(Collection<String> digests)Remove definitively a set of binariesprotected voidremoveBinary(String digest)protected voidsetupCloudClient()Configure Cloud client using propertiesprotected voidwaitForCopyCompletion(com.amazonaws.services.s3.transfer.Copy copy)StringwriteBlob(Blob blob)Writes aBlobto storage and returns information about it.-
Methods inherited from class org.nuxeo.ecm.blob.AbstractCloudBinaryManager
getBinaryManager, getBooleanProperty, getContentDispositionHeader, getContentTypeHeader, getDefaultDigestAlgorithm, getIntProperty, getIntProperty, getProperties, getProperty, getProperty, getSystemPropertyName, getURI, initialize, isDirectDownload, isTransient, performsExternalAccessControl, readBlob, supportsUserUpdate, supportsUserUpdateDefaultTrue
-
Methods inherited from class org.nuxeo.ecm.core.blob.binary.CachingBinaryManager
getBinary, getBinary, getFile, initializeCache, initializeCache, initializeCache
-
Methods inherited from class org.nuxeo.ecm.core.blob.binary.AbstractBinaryManager
computeDigestPattern, getBinary, getDescriptor, getDigestAlgorithm, getGarbageCollector, isValidDigest, setDescriptor, storeAndDigest, toHexString
-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
Methods inherited from interface org.nuxeo.ecm.core.blob.BlobProvider
allowByteRange, allowDirectDownload, canConvert, deleteBlob, getAppLinks, getAvailableConversions, getBinaryGarbageCollector, getFile, getStatus, getStream, getStream, getThumbnail, hasCreateFromKeyPermission, isColdStorageMode, isRecordMode, isTransactional, isVersion, readBlob, supportsSync, updateBlob, writeBlob
-
-
-
-
Field Detail
-
SYSTEM_PROPERTY_PREFIX
public static final String SYSTEM_PROPERTY_PREFIX
- See Also:
- Constant Field Values
-
BUCKET_NAME_PROPERTY
public static final String BUCKET_NAME_PROPERTY
- See Also:
- Constant Field Values
-
BUCKET_PREFIX_PROPERTY
public static final String BUCKET_PREFIX_PROPERTY
- See Also:
- Constant Field Values
-
BUCKET_REGION_PROPERTY
public static final String BUCKET_REGION_PROPERTY
- See Also:
- Constant Field Values
-
AWS_ID_PROPERTY
public static final String AWS_ID_PROPERTY
- See Also:
- Constant Field Values
-
AWS_SECRET_PROPERTY
public static final String AWS_SECRET_PROPERTY
- See Also:
- Constant Field Values
-
AWS_SESSION_TOKEN_PROPERTY
public static final String AWS_SESSION_TOKEN_PROPERTY
- Since:
- 10.10
- See Also:
- Constant Field Values
-
CONNECTION_MAX_PROPERTY
public static final String CONNECTION_MAX_PROPERTY
AWS ClientConfiguration default 50- See Also:
- Constant Field Values
-
CONNECTION_RETRY_PROPERTY
public static final String CONNECTION_RETRY_PROPERTY
AWS ClientConfiguration default 3 (with exponential backoff)- See Also:
- Constant Field Values
-
CONNECTION_TIMEOUT_PROPERTY
public static final String CONNECTION_TIMEOUT_PROPERTY
AWS ClientConfiguration default 50*1000 = 50s- See Also:
- Constant Field Values
-
SOCKET_TIMEOUT_PROPERTY
public static final String SOCKET_TIMEOUT_PROPERTY
AWS ClientConfiguration default 50*1000 = 50s- See Also:
- Constant Field Values
-
KEYSTORE_FILE_PROPERTY
public static final String KEYSTORE_FILE_PROPERTY
- See Also:
- Constant Field Values
-
KEYSTORE_PASS_PROPERTY
public static final String KEYSTORE_PASS_PROPERTY
- See Also:
- Constant Field Values
-
SERVERSIDE_ENCRYPTION_PROPERTY
public static final String SERVERSIDE_ENCRYPTION_PROPERTY
- See Also:
- Constant Field Values
-
SERVERSIDE_ENCRYPTION_KMS_KEY_PROPERTY
public static final String SERVERSIDE_ENCRYPTION_KMS_KEY_PROPERTY
- See Also:
- Constant Field Values
-
PRIVKEY_ALIAS_PROPERTY
public static final String PRIVKEY_ALIAS_PROPERTY
- See Also:
- Constant Field Values
-
PRIVKEY_PASS_PROPERTY
public static final String PRIVKEY_PASS_PROPERTY
- See Also:
- Constant Field Values
-
ENDPOINT_PROPERTY
public static final String ENDPOINT_PROPERTY
- See Also:
- Constant Field Values
-
PATHSTYLEACCESS_PROPERTY
public static final String PATHSTYLEACCESS_PROPERTY
- Since:
- 10.3
- See Also:
- Constant Field Values
-
ACCELERATE_MODE_PROPERTY
public static final String ACCELERATE_MODE_PROPERTY
- Since:
- 11.1
- See Also:
- Constant Field Values
-
DIRECTDOWNLOAD_PROPERTY_COMPAT
public static final String DIRECTDOWNLOAD_PROPERTY_COMPAT
- See Also:
- Constant Field Values
-
DIRECTDOWNLOAD_EXPIRE_PROPERTY_COMPAT
public static final String DIRECTDOWNLOAD_EXPIRE_PROPERTY_COMPAT
- See Also:
- Constant Field Values
-
USER_AGENT_PREFIX_PROPERTY
public static final String USER_AGENT_PREFIX_PROPERTY
- Since:
- 2021.15
- See Also:
- Constant Field Values
-
USER_AGENT_SUFFIX_PROPERTY
public static final String USER_AGENT_SUFFIX_PROPERTY
- Since:
- 2021.15
- See Also:
- Constant Field Values
-
DELIMITER
public static final String DELIMITER
- See Also:
- Constant Field Values
-
bucketName
protected String bucketName
-
bucketNamePrefix
protected String bucketNamePrefix
-
awsCredentialsProvider
protected com.amazonaws.auth.AWSCredentialsProvider awsCredentialsProvider
-
clientConfiguration
protected com.amazonaws.ClientConfiguration clientConfiguration
-
encryptionMaterials
protected com.amazonaws.services.s3.model.EncryptionMaterials encryptionMaterials
-
isEncrypted
protected boolean isEncrypted
-
cryptoConfiguration
protected com.amazonaws.services.s3.model.CryptoConfiguration cryptoConfiguration
-
useServerSideEncryption
protected boolean useServerSideEncryption
-
serverSideKMSKeyID
protected String serverSideKMSKeyID
-
amazonS3
protected com.amazonaws.services.s3.AmazonS3 amazonS3
-
transferManager
protected com.amazonaws.services.s3.transfer.TransferManager transferManager
-
-
Method Detail
-
close
public void close()
Description copied from interface:BinaryManagerCloses the binary manager and releases all resources and temporary objects held by it.- Specified by:
closein interfaceBinaryManager- Specified by:
closein interfaceBlobProvider- Overrides:
closein classCachingBinaryManager
-
abortOldUploads
protected void abortOldUploads()
Aborts uploads that crashed and are older than 1 day.- Since:
- 7.2
-
abortOldMultipartUploadsInternal
protected void abortOldMultipartUploadsInternal()
-
setupCloudClient
protected void setupCloudClient() throws IOExceptionDescription copied from class:AbstractCloudBinaryManagerConfigure Cloud client using properties- Specified by:
setupCloudClientin classAbstractCloudBinaryManager- Throws:
IOException
-
getTransferManager
public com.amazonaws.services.s3.transfer.TransferManager getTransferManager()
Description copied from interface:S3ManagedTransferReturns theTransferManager.- Specified by:
getTransferManagerin interfaceS3ManagedTransfer
-
removeBinary
protected void removeBinary(String digest)
-
clear
public void clear()
INTERNAL (TESTS). Clears the binary manager of all its data.- Since:
- 11.5
-
getSystemPropertyPrefix
protected String getSystemPropertyPrefix()
Description copied from class:AbstractCloudBinaryManagerGets the prefix used for configuration using system properties.- Specified by:
getSystemPropertyPrefixin classAbstractCloudBinaryManager
-
instantiateGarbageCollector
protected BinaryGarbageCollector instantiateGarbageCollector()
- Specified by:
instantiateGarbageCollectorin classAbstractCloudBinaryManager
-
removeBinaries
public void removeBinaries(Collection<String> digests)
Description copied from interface:BinaryManagerRemove definitively a set of binaries- Specified by:
removeBinariesin interfaceBinaryManager- Specified by:
removeBinariesin classAbstractCloudBinaryManager- Parameters:
digests- a set of digests, must not benull.
-
lengthOfBlob
public long lengthOfBlob(String digest)
- Returns:
- object length, or -1 if missing
-
isMissingKey
protected static boolean isMissingKey(com.amazonaws.AmazonClientException e)
-
isMD5
@Deprecated public static boolean isMD5(String digest)
Deprecated.since 11.1, now unused
-
canAccessBucket
public boolean canAccessBucket()
Used in the healthCheck; the transferManager should be initialized and the bucket accessible- Since:
- 9.3
-
getFileStorage
protected FileStorage getFileStorage()
- Specified by:
getFileStoragein classAbstractCloudBinaryManager
-
getAwsCredentialsProvider
public com.amazonaws.auth.AWSCredentialsProvider getAwsCredentialsProvider()
Gets the AWSCredentialsProvider.- Since:
- 10.2
-
getAmazonS3
public com.amazonaws.services.s3.AmazonS3 getAmazonS3()
Gets AmazonS3.- Since:
- 10.2
-
getBucketName
public String getBucketName()
Gets the bucket name.- Since:
- 11.1
-
getBucketPrefix
public String getBucketPrefix()
Gets the bucket prefix.- Since:
- 11.1
-
writeBlob
public String writeBlob(Blob blob) throws IOException
Description copied from interface:BlobProviderWrites aBlobto storage and returns information about it.Called to store a user-created blob.
- Specified by:
writeBlobin interfaceBlobProvider- Overrides:
writeBlobin classAbstractCloudBinaryManager- Parameters:
blob- the blob- Returns:
- the blob key
- Throws:
IOException
-
copyBlob
protected String copyBlob(S3BinaryManager sourceBlobProvider, String digest) throws IOException
Copies a blob. Returnsnullif the copy was not possible.- Parameters:
sourceBlobProvider- the source blob providerdigest- the source blob key- Returns:
- the copied blob key, or
nullif the copy was not possible - Throws:
IOException- Since:
- 10.1
-
waitForCopyCompletion
protected void waitForCopyCompletion(com.amazonaws.services.s3.transfer.Copy copy)
- Since:
- 2021.15
-
getRemoteUri
protected URI getRemoteUri(String digest, ManagedBlob blob, javax.servlet.http.HttpServletRequest servletRequest) throws IOException
- Overrides:
getRemoteUriin classAbstractCloudBinaryManager- Throws:
IOException
-
-