Addons

Nuxeo Aspera Connector

Updated: August 19, 2019

The Nuxeo Aspera connector enables users to download/upload binaries with Aspera.

Requirements

  • Aspera Desktop Client - latest version.
  • Nuxeo Server LTS 2019 (10.10) with access to AWS S3 Storage

Installation

Installation is made of two steps:

  1. Install the Nuxeo Package available from the marketplace.
  2. Install the Aspera desktop client.

Configuration

Aspera Configuration

We need to configure 2 Aspera nodes; one for upload and one for download.

Each node will be attached to one S3 bucket in Nuxeo:

  • The main Nuxeo S3 bucket in Nuxeo for download purpose
  • The S3 transient store bucket for upload purpose

Follow this Aspera documentation to attach S3 buckets to Aspera.

Please note that, in the documentation above, for Download the IAM role used by Aspera only needs READ permissions on the bucket.

The policy attached to this role can be added as shown in this sample:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetObject",
        "s3:GetBucketLocation"
      ],
      "Resource": [
        "arn:aws:s3:::bucket-name",
        "arn:aws:s3:::bucket-name/\*"
      ]
    }
  ]
}

The policy for Upload must be able to put and get objects from the S3 bucket:

{
  "Version": "2012-10-17",
  "Statement": [
  {
    "Sid": "VisualEditor0",
    "Effect": "Allow",
    "Action": [
    "s3:PutObject",
    "s3:GetObject",
        "s3:AbortMultipartUpload",
        "s3:DeleteObject",
        "s3:ListMultipartUploadParts"
      ],
      "Resource": "arn:aws:s3:::bucket-name/*"
    },
    {
      "Sid": "VisualEditor1",
      "Effect": "Allow",
      "Action": [
        "s3:ListBucketMultipartUploads",
        "s3:ListBucket",
        "s3:GetBucketLocation"
      ],
      "Resource": "arn:aws:s3:::bucket-name"
    }
  ]
}

Nuxeo Configuration

Transient Store on AWS

To use S3 direct upload with Nuxeo, you will need to add another policy to a new role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets"
            ],
            "Resource": "arn:aws:s3:::*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts",
                "s3:ListBucketMultipartUploads"
            ],
            "Resource": "arn:aws:s3:::yourbucketname"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts",
                "s3:ListBucketMultipartUploads"
            ],
            "Resource": "arn:aws:s3:::yourbucketname/*"
        }
    ]
}

Make sure that all configuration values are set in the nuxeo.conf:

nuxeo.s3storage.useDirectUpload=true
nuxeo.s3storage.transient.bucket=
nuxeo.s3storage.transient.roleArn=
nuxeo.aws.accessKeyId=
nuxeo.aws.secretKey=
nuxeo.aws.region=
nuxeo.s3storage.bucket=

Aspera Nuxeo Configuration

Add the aspera access keys in nuxeo.conf:

Example:

## is the same for the 2 nodes
aspera.node.url=https://ats-aws-us-east-1.aspera.io
aspera.node.port=443

### ACCESS KEY ON UPLOAD NODE LINKED TO TS S3 BUCKET
aspera.acess.key.id=
### SECRET KEY ON UPLOAD NODE LINKED TO TS S3 BUCKET
aspera.acess.key.secret=

### ACCESS KEY ON DOWNLOAD NODE LINKED TO TMAIN S3 BUCKET
aspera.download.acess.key.id=
### SECRET KEY ON DOWNLOAD NODE LINKED TO MAIN S3 BUCKET
aspera.download.acess.key.secret=

Functional Overview

The Nuxeo Aspera Connector uses "Transfers" to add content to the system.

The process is divided into 2 steps: Upload and Create.

In the first step, the user uploads content to the transfer bucket. While in this step, he can add files to the bucket and add properties that will apply to the content. Properties can be added to all files at once (common metadata), to multiple files at once (bulk edit of metadata) and one at a time.

Once the metadata has been edited to fit the user's needs, the transfer can be completed and the content is then created in the system.

Transfers have 3 states: Draft (no content added yet), In Progress (content added) and Completed (the transfer has been completed and the content has been created in the system).

Aspera Upload

Once in your instance, Nuxeo Aspera Upload can be accessed from two different ways:

  • From the User Settings menu located on the left side:

1-aspera.png
1-aspera.png

  • By clicking on the Aspera upload button displayed on every folderish document (workspace, folder, etc.)

2-aspera.png
2-aspera.png

The files uploaded by Aspera will be accessible in this folderish document (by default the target location is the user personal workspace)

Once on the Aspera Upload screen, you will be able to download the Aspera Desktop client via the following banner at the top:

aspera-setup.png
aspera-setup.png

Once on the Aspera menu, 2 tabs are available:

Transfers

This screen shows the status of all current transfers in your Nuxeo application.

You can:

  • Access the transfer metadata and files properties
  • Complete transfers
  • Share transfers
  • Delete transfers (as long as they have not been completed)

4-aspera.png
4-aspera.png

Upload To Nuxeo

On this screen, different actions are available; you can:

  • Drag and drop (or click to select) file(s) to upload with Aspera and follow the status of the uploads (whether you have the Aspera desktop client or not).

0-aspera.png
0-aspera.png

  • Define/edit the common metadata of the current transfer (set of uploaded files) by clicking on the Edit button.

5-aspera.png
5-aspera.png

6-aspera.png
6-aspera.png

  • Modify the permissions to share the transfer with other user(s) (i.e. another user is responsible for modifying the metadata).

7-aspera.png
7-aspera.png

  • Edit/Delete each file.

8-aspera.png
8-aspera.png

  • Bulk edit selected files metadata in once.

9-aspera.png
9-aspera.png

10-aspera.png
10-aspera.png

  • Complete transfer means to create the related documents in the Nuxeo application (once all files have been uploaded).

When completing a transfer, the "common metadata" is propagated to all Nuxeo documents (except all single/bulk metadata edits override them).

Aspera Download

The Nuxeo Aspera Download action is accessible via a button displayed when selecting one or several documents:

3-aspera.png
3-aspera.png

How it Works

Aspera Upload

aspera-upload.png
aspera-upload.png

Aspera download

aspera-download.png
aspera-download.png

Customization

Model

The Transfer document type can be overridden to edit custom metadata needed for your Nuxeo documents created after Aspera uploads.

To do so:

  1. In Nuxeo Studio, go to Settings > Registries and add those schemas and this lifecycle.
  2. Create a new document type, name it Transfer (id and label) that extends File document type.
  3. Check the Hidden in navigation facet for this new document type and add the schemas transfer-dc and common-aspera.
  4. Select the life cycle transfer_lifecyle.
  5. Save and commit: you will be able now to add different schemas to your new Transfer document.

UI Layouts

After having overridden the Transfer document type, you can now override the different UI layouts in the Nuxeo View Designer to be able to edit those metadata:

  • The metadata layout nuxeo-transfer-metadata-layout.html

    12-aspera.png
    12-aspera.png

  • The edit layout nuxeo-transfer-edit-layout.html

    6-aspera.png
    6-aspera.png

  • The import layout (for single/bulk metadata edition) nuxeo-transfer-import-layout.html

    10-aspera.png
    10-aspera.png

  • The view layout nuxeo-transfer-view-layout.html

    13-aspera.png
    13-aspera.png

All those layouts can be found on GitHub.

Just copy/paste those layouts and you will be able to add or remove (custom) metadata.

The metadata inside the single/bulk edit layout is related to the ca:files/*/properties metadata of the Transfer document (the value is a JSON containing the properties you want to set).
It means that you can add any metadata that you want (without creating any additional schemas inside Nuxeo Studio). It will propagate those properties to the different created files (except if those metadata doesn't exist in the schemas of those newly created files).

When overriding, be careful to put back the actions and other HTML elements that are not related to metadata

Polymer UI Custom Example

If you want to develop your custom UI rather than using the Nuxeo addon, you can build and deploy this maven project.

When running your instance, go to NUXEO_URL/nuxeo/app/ to get a simple UI example on how to use the Aspera connector:

  • To upload a file in a given location via the connector
  • To upload and add a file to a given document via the connector
  • To list all documents with binaries and download them via the connector

Details:

  • my-app is the main page containing all pages folder pages.
  • In each page, aspera-connector is called to set Aspera authentication in place.
  • aspera-connector.html is wrapping the Aspera API to be used for upload/download via the connector.
  • All Nuxeo operations used in this sample are used in the Addon itself

We'd love to hear your thoughts!

All fields required