Addons

Nuxeo Aspera Connector

Updated: January 20, 2021

The Nuxeo Aspera connector enables users to upload/download binaries with the Aspera Connect client. It is designed to be used with Aspera On Cloud instances, however, if you use a self-hosted Aspera instance, you will still be able to use the Nuxeo Aspera Connector. Contact your IBM Aspera team for more information and assistance.

Nuxeo University
Watch the related courses on Nuxeo University:

Version 3.0.0

Requirements

If you are upgrading from a previous version of the Nuxeo Aspera connector, please see the upgrade notes for critical information related to potential breaking code.

Version 3.0.0 of the Nuxeo Aspera Connector is aligned with IBM Aspera Connect 3.9.9 SDK and, by default, points to the IBM Aspera Connect SDK 3.9.9 CDN URL. If you would like to host the SDK on your own servers or point to another version, please see our entry on configuring the SDK url. If you are using a version newer than this, functionality might be affected.

Installation

Installation is comprised of two steps:

  1. Install the Nuxeo Package available from the marketplace.
  2. Install the Aspera Connect desktop client (you will be prompted to install the necessary Aspera software when you try to use the Connector, if you do not already have it).

Configuration

Aspera Configuration

We need to configure 2 Aspera nodes; one for upload and one for download.

Each node will be attached to one S3 bucket in Nuxeo:

Follow this Aspera documentation to attach S3 buckets to Aspera.

Please note that, in the documentation above, for Download the IAM role used by Aspera only needs READ permissions on the bucket.

The policy attached to this role can be added as shown in this sample:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetObject",
        "s3:GetBucketLocation"
      ],
      "Resource": [
        "arn:aws:s3:::bucket-name",
        "arn:aws:s3:::bucket-name/\*"
      ]
    }
  ]
}

The policy for Upload must be able to put and get objects from the S3 bucket, for example:

{
  "Version": "2012-10-17",
  "Statement": [
  {
    "Sid": "VisualEditor0",
    "Effect": "Allow",
    "Action": [
    "s3:PutObject",
    "s3:GetObject",
        "s3:AbortMultipartUpload",
        "s3:DeleteObject",
        "s3:ListMultipartUploadParts"
      ],
      "Resource": "arn:aws:s3:::bucket-name/*"
    },
    {
      "Sid": "VisualEditor1",
      "Effect": "Allow",
      "Action": [
        "s3:ListBucketMultipartUploads",
        "s3:ListBucket",
        "s3:GetBucketLocation"
      ],
      "Resource": "arn:aws:s3:::bucket-name"
    }
  ]
}

Nuxeo Configuration

Transient Store on AWS

To use S3 direct upload with Nuxeo, you will need to add another policy to a new role for example:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets"
            ],
            "Resource": "arn:aws:s3:::*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts",
                "s3:ListBucketMultipartUploads"
            ],
            "Resource": "arn:aws:s3:::yourbucketname"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts",
                "s3:ListBucketMultipartUploads"
            ],
            "Resource": "arn:aws:s3:::yourbucketname/*"
        }
    ]
}

Make sure that all configuration values are set in the nuxeo.conf:

nuxeo.s3storage.transient.bucket=
nuxeo.aws.accessKeyId=
nuxeo.aws.secretKey=
nuxeo.aws.region=
nuxeo.s3storage.bucket=

New in this release, nuxeo.s3storage.useDirectUpload=true and nuxeo.s3storage.transient.roleArn= (documented as part of the previous version configuration) are no longer necessary for the Nuxeo Aspera Connector and can be removed, if you only use these values for the Nuxeo Aspera Connector, and not for any other reason. See upgrade notes.

Aspera Nuxeo Configuration

Add the Aspera access keys in nuxeo.conf:

Example:

## is the same for the 2 nodes
aspera.node.url=
aspera.node.port=

### ACCESS KEY ON UPLOAD NODE LINKED TO TS S3 BUCKET
aspera.access.key.id=
### SECRET KEY ON UPLOAD NODE LINKED TO TS S3 BUCKET
aspera.access.key.secret=

### ACCESS KEY ON DOWNLOAD NODE LINKED TO MAIN S3 BUCKET
aspera.download.access.key.id=
### SECRET KEY ON DOWNLOAD NODE LINKED TO MAIN S3 BUCKET
aspera.download.access.key.secret=

SDK URL

The Aspera Connect SDK version used with this version of the Nuxeo Aspera Connector defaults to Aspera Connect SDK 3.9.9. If you would like to use a different SDK, you can modify this in one of two ways:

Functional Overview

The Nuxeo Aspera Connector uses Transfers to add content to the Nuxeo system. Transfers are “wrappers” or “buckets” for the files/folders you add to the Nuxeo system.

Transfer Steps

The process of adding files and folders using the Nuxeo Aspera Connector is divided into 2 steps: Upload and Create.

In the first step, the user uploads files and folders to the transfer bucket. While in this step, it may be possible, depending on the chosen “transition type”, for a user to add items to the bucket and add properties that will apply (as needed/desired).

Once content and any metadata have been added, the transfer can be completed and the content is then created in the system.

Transfer Transition Types

Transitioning between the “upload” and “create” steps can be achieved in 2 ways: automatically (auto-create) or manually.

Users can choose which option to use when creating the transfer from the transfer dashboard/transfer view. When creating a transfer from a "folderish" document in the browsing tree, the transition type defaults to manual. You can toggle it to auto-create before adding content. It is not possible to change from manual to auto-create (or vice versa) after you start adding content to the transfer. Once you set the auto-create, some limitations around modifying the transfer will apply. As long as the transfer is empty, you can change the transition type, but once you start adding content, the transition type is "locked".

When auto-create is toggled/selected to be “on”, the user needs only to start the upload of the content to the transfer. Once the upload is completed, the system will automatically start the process to create the content in the system; no need to wait around simply to push a button.

Before adding content to the transfer, you can edit the “common metadata” which would apply to all items in the transfer. Users can, of course, decide not to include any metadata at all, and simply add content, too.

There are some caveats when using auto-create (auto-create is set to “on”). Because of some asynchronous behavior, the ability to modify the transfer/file metadata or to add/remove content once your initial "add to transfer" has begun, is not allowed. If you discover that you have erred in the content/information, you will need to let the transfer process complete before addressing within the system; where the content was added (not the transfer).

If a user has content that needs to be “tagged” or “catalogued” with metadata before creation in the system, the “manual” transition option might be a better choice.

To use the manual option, choose “no” for auto-create or toggle the auto-create to “off” when creating a new transfer (you can only turn off auto-create if there is no content in the transfer). This will require someone manually start the creation process by clicking the Complete transfer button once the upload is finished (the button is only visible once there is no longer an upload in progress).

When using the manual transition, the user uploads content to the transfer bucket. While in this step, a user can add content to the transfer and add properties that apply to the items. Properties can be added that "apply to all" using the “common metadata” panel; more file-specific properties can be added either to multiple files at once (bulk edit of metadata) and/or individually, using a single-file action.

Once the metadata has been edited to fit the user's needs, the transfer can be completed and the content is then created in the system.

Transfers have 3 functional states:

Using Aspera with Nuxeo

Once in your instance, Nuxeo Aspera Upload can be accessed two different ways:

aspera-image3
aspera-image3

aspera-image19

The content uploaded by Aspera will be accessible in this folderish document. If the transfer is "started" from the Aspera Dashboard or from the Transfer View using the "+ New Upload" button, the target location is the user's personal workspace.

Once on the Transfer View (as you begin to add content), you will be able to download the Aspera Desktop client via the following banner at the top (if not already installed or if a new version is needed):

aspera-image14

Once on the Aspera menu, there are 2 main views available:

Aspera Dashboard

This screen shows the status of all current transfers in your Nuxeo application.

You can:

All of these actions are available via the “transfer” view, as well. You can find information on the actions in the “transfer view” section below.

aspera-image20

“Delete” does remove the content from the Aspera Node in most cases, however, we have seen some inconsistencies. It is advised that you regularly clean your Aspera Transfer Node. Deleting the transfer does not remove the content from the repository if the transfer has already been completed. This ONLY deletes the transfer “bucket” or “wrapper”)

Transfer View

To create a transfer from the dashboard, simply click + New Upload in the upper right-hand corner. By default, this will set the “target document” (or destination) as the user’s personal workspace. This can be edited using the “common metadata” edit; "target location" value.

On this screen, different actions are available:

aspera-image1

By default, when clicking + New Upload from the Aspera Dashboard or the transfer view, the "target location" for the upload is the user's personal workspace. If you use the Upload with Aspera action from a folderish document, the "target location" will be set to that folderish document, by default.

aspera-image7

aspera-image15
aspera-image15

aspera-image4

aspera-image6
aspera-image6

aspera-image24

aspera-image25

10-aspera.png
10-aspera.png

Aspera Download

The Nuxeo Aspera Download action is accessible via a button displayed when selecting one or several documents in a "folderish" document view:

3-aspera.png

While we distinguish between transfers used for “upload” purposes from “download” purposes, we currently only show the “upload” transfers on the transfer dashboard. We have improvements planned for displaying both.

How It Works

File and Folder Type Definition

The Connector relies on the CreateFolder operation to create content in the Nuxeo repository. By default the Folder doctype is created for "folders" and File for "files".

If these types are acceptable, be sure that you have defined your content model to allow for appropriate parent/children document types as shown in our Nuxeo Studio documentation. However, if you have written a contribution to apply logic to assign types, then that will be used when using the Nuxeo Aspera Connector.

If your content model needs are more advanced, you may also contribute a custom FolderImporter class for your custom Folder and File doctypes. See FileManagerService on Nuxeo Explorer.

Aspera Upload

aspera-image27

Aspera Download

aspera-image26

Version 2.0.8

Requirements

Installation

Installation is made of two steps:

  1. Install the Nuxeo Package available from the marketplace.
  2. Install the Aspera desktop client.

Configuration

Aspera Configuration

We need to configure 2 Aspera nodes; one for upload and one for download.

Each node will be attached to one S3 bucket in Nuxeo:

Follow this Aspera documentation to attach S3 buckets to Aspera.

Please note that, in the documentation above, for Download the IAM role used by Aspera only needs READ permissions on the bucket.

The policy attached to this role can be added as shown in this sample:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetObject",
        "s3:GetBucketLocation"
      ],
      "Resource": [
        "arn:aws:s3:::bucket-name",
        "arn:aws:s3:::bucket-name/\*"
      ]
    }
  ]
}

The policy for Upload must be able to put and get objects from the S3 bucket:

{
  "Version": "2012-10-17",
  "Statement": [
  {
    "Sid": "VisualEditor0",
    "Effect": "Allow",
    "Action": [
    "s3:PutObject",
    "s3:GetObject",
        "s3:AbortMultipartUpload",
        "s3:DeleteObject",
        "s3:ListMultipartUploadParts"
      ],
      "Resource": "arn:aws:s3:::bucket-name/*"
    },
    {
      "Sid": "VisualEditor1",
      "Effect": "Allow",
      "Action": [
        "s3:ListBucketMultipartUploads",
        "s3:ListBucket",
        "s3:GetBucketLocation"
      ],
      "Resource": "arn:aws:s3:::bucket-name"
    }
  ]
}

Nuxeo Configuration

Transient Store on AWS

To use S3 direct upload with Nuxeo, you will need to add another policy to a new role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets"
            ],
            "Resource": "arn:aws:s3:::*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts",
                "s3:ListBucketMultipartUploads"
            ],
            "Resource": "arn:aws:s3:::yourbucketname"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts",
                "s3:ListBucketMultipartUploads"
            ],
            "Resource": "arn:aws:s3:::yourbucketname/*"
        }
    ]
}

Make sure that all configuration values are set in the nuxeo.conf:

nuxeo.s3storage.useDirectUpload=true
nuxeo.s3storage.transient.bucket=
nuxeo.s3storage.transient.roleArn=
nuxeo.aws.accessKeyId=
nuxeo.aws.secretKey=
nuxeo.aws.region=
nuxeo.s3storage.bucket=
Aspera Nuxeo Configuration

Add the Aspera access keys in nuxeo.conf:

Example:

## is the same for the 2 nodes
aspera.node.url=https://ats-aws-us-east-1.aspera.io
aspera.node.port=443

### ACCESS KEY ON UPLOAD NODE LINKED TO TS S3 BUCKET
aspera.acess.key.id=
### SECRET KEY ON UPLOAD NODE LINKED TO TS S3 BUCKET
aspera.acess.key.secret=

### ACCESS KEY ON DOWNLOAD NODE LINKED TO TMAIN S3 BUCKET
aspera.download.acess.key.id=
### SECRET KEY ON DOWNLOAD NODE LINKED TO MAIN S3 BUCKET
aspera.download.acess.key.secret=

Functional Overview

The Nuxeo Aspera Connector uses "Transfers" to add files to the system.

The process is divided into 2 steps: Upload and Create.

In the first step, the user uploads files to the transfer bucket. While in this step, a user can add files to the bucket and add properties that will apply to them. Properties can be added to all files at once (common metadata), to multiple files at once (bulk edit of metadata) and one at a time.

Once the metadata has been edited to fit the user's needs, the transfer can be completed and the content is then created in the system.

Transfers have 3 states:

Aspera Upload

Once in your instance, Nuxeo Aspera Upload can be accessed from two different ways:

1-aspera.png
1-aspera.png

2-aspera.png

The content uploaded by Aspera will be accessible in this folderish document (by default the target location is the user personal workspace)

Once on the Aspera Upload screen, you will be able to download the Aspera Desktop client via the following banner at the top:

aspera-setup.png

Once on the Aspera menu, 2 tabs are available:

Transfers

This screen shows the status of all current transfers in your Nuxeo application.

You can:

4-aspera.png
4-aspera.png

Upload To Nuxeo

On this screen, different actions are available; you can:

0-aspera.png
0-aspera.png

5-aspera.png
5-aspera.png

6-aspera.png
6-aspera.png

7-aspera.png
7-aspera.png

8-aspera.png
8-aspera.png

9-aspera.png
9-aspera.png

10-aspera.png
10-aspera.png

When completing a transfer, the "common metadata" is propagated to all Nuxeo documents (except all single/bulk metadata edits override them).

Aspera Download

The Nuxeo Aspera Download action is accessible via a button displayed when selecting one or several documents:

3-aspera.png
3-aspera.png

How It Works

Aspera Upload

aspera-upload.png
aspera-upload.png

Aspera download

aspera-download.png
aspera-download.png

Customization

Model

The Transfer document type can be overridden to edit custom metadata needed for your Nuxeo documents created after Aspera uploads.

To do so:

  1. In Nuxeo Studio, go to Settings > Registries and add those schemas and this lifecycle.
  2. Create a new document type, name it Transfer (id and label) that extends File document type.
  3. Check the Hidden in navigation facet for this new document type and add the schemas transfer-dc and common-aspera.
  4. Select the life cycle transfer_lifecyle.
  5. Save and commit: you will be able now to add different schemas to your new Transfer document.

UI Layouts

After having overridden the Transfer document type, you can now override the different UI layouts in the Nuxeo View Designer to be able to edit those metadata:

All those layouts can be found on GitHub.

Just copy/paste those layouts and you will be able to add or remove (custom) metadata.

The metadata inside the single/bulk edit layout is related to the ca:files/*/properties metadata of the Transfer document (the value is a JSON containing the properties you want to set).
It means that you can add any metadata that you want (without creating any additional schemas inside Nuxeo Studio). It will propagate those properties to the different created files (except if those metadata doesn't exist in the schemas of those newly created files).

When overriding, be careful to put back the actions and other HTML elements that are not related to metadata

Polymer UI Custom Example

If you want to develop your custom UI rather than using the Nuxeo addon, you can build and deploy this maven project.

When running your instance, go to NUXEO_URL/nuxeo/app/ to get a simple UI example on how to use the Aspera connector:

Details:

We'd love to hear your thoughts!

All fields required