Choosing How to Import Data in the Nuxeo Platform

Different Tools for Different Use Cases

The Nuxeo Platform provides several tools for managing imports. Choosing the right tool will depend on your exact use cases:

What amount of data do you need to import? Hundreds, thousands, millions?
Do you need to do the import while the application is running? Initial / one time import vs everyday import.
How complex is your import? How many business rules are integrated in your import?
What is the source you want to import from? Database, XML, files, ...
What are your skills? SQL, ETL/ESB, Java dev, ...

This page will walk you through the different import options and give you the pros and cons of each approach.

Possible Approaches

User Imports

By default, the Nuxeo Platform allows users to import several documents at a time via:

Import criteria details

Criteria	Value	Comment
Average import speed	Low	A few documents.
Custom logic handling	Built-in	All custom logic will be called.
Ability to handle huge volume	No	No transaction batch management.
Production interruption	No
Blob upload	In transaction	The blob upload is part of the import transaction.
Post import tasks	None

The key point is that all these user import systems are designed to be easy to use, but are not designed for high performance and huge volume.

For more information

Content Creation User Documentation

</ul>
<p>

HTTP API

Nuxeo HTTP Automation API can be used to run imports inside the Nuxeo Platform.

You can use Automation from custom code, custom scripting or from tools like:

ETL: see the Talend Connector
ESB: see the Mule Connector

Using the API allows you to easily define import custom logic on the client side, but:

blobs upload will be part of the process,
doing transaction batch is not easy since it requires to create custom chains.

Import criteria details

Criteria	Value	Comment
Average import speed	Low / Medium	Several documents (between 5 and 20 docs).
Custom logic handling	Built-in	All custom logic will be called.
Ability to handle huge volume	No	No easy transaction batch management.
Production interruption	No
Blob upload	In process	The blob upload is part of the import process.
Post import tasks	None

For more information

REST API

Automation

</ul>
<p>

Platform Importer

The Platform importer is a framework that can be used to build custom importers that use the Java API.

Unlike the previous methods:

The Java API is directly used: no network and marshaling overhead.
Blobs are read from a local filesystem: no network cost.

The importer does handle several aspects that are important for managing performances:

transaction batch,
de-activating some listeners,
process event handles in bulk-mode.

Import criteria details

Criteria	Value	Comment
Average import speed	High	Several hundreds of documents (between 50 and 500 docs).
Custom logic handling	Built-in	Most custom logic will be called: depending on which listeners are removed.
Ability to handle huge volume	Yes	Native handling of transaction batch + bulk event handler mode.
Production interruption	Yes	The bulk mode is not adapted for a normal usage: at least a dedicated Nuxeo node should be allocated. High speed import is likely to saturate the database: this will slow down all interactive usages.
Blob upload	Separated	Blobs are directly read on the server side FileSystem.
Post import tasks	May need to restart full text indexing. May need to restart process for listeners that were by-passed .	In a lot of cases, the full text indexing is deactivated during processing, as well as other slow processes like video conversation, thumbnails generation, etc. After import, these processes need to be restarted.

For more information

Nuxeo Bulk Document Importer Developer Documentation

</ul>
<p>

SQL Import

Thanks to the VCS Repository clear and clean SQL structure, you can directly use SQL injection.

Of course, this is by far the fastest technique, but since you will bypass all the Java business layer, you will need to do some checks and post processing. In addition, if you want the SQL import to be really fast, you may want to deactivate some of the integrity constraints and triggers.

Import criteria details

Criteria	Value	Comment
Average import speed	Very high	Several thousands of Documents (between 500 and 5000 docs).
Custom logic handling	Bypass	All Java Layer is by-passed.
Ability to handle huge volume	Yes	Native handling of transaction batch + bulk event handler mode.
Production interruption	Yes	Usually, the database server configuration is changed to make the bulk insert more efficient.
Blob upload	Not handled	Blobs needs to be managed by a separated process.
Post import tasks	May need to restart full text indexing. May need to restart some triggers.	Rebuild full text. Rebuild ancestors cache. Rebuild read-ACLs

For more information

</ul>
<p>