Home > Server > Administration > Search Setup
Server

Search Setup

Updated: November 15, 2024

This page provides several configuration use cases for Elasticsearch and Opensearch.

Setting up an Elasticsearch or OpenSearch Cluster

Nuxeo supports Elasticsearch and OpenSearch clusters.

OpenSearch 1 is a fork of Elasticsearch 7, except some advanced features (not used by Nuxeo) they are fully compatible.

Nuxeo 2023 is defining index settings, mappings and uses the Rest API according to Elasticsearch 7 version (equivalent to OpenSearch 1 version). Nuxeo is relying on OpenSearch client library to access the search cluster.

Note that for historical reasons, Nuxeo 2023 continues to name "Elasticsearch" its search service and configuration options. Elasticsearch and OpenSearch could be used interchangeably in the documentation.

In addition to OpenSearch 1 and Elasticsearch 7, Nuxeo also works with Elasticsearch 8 cluster, as Elasticsearch 8 being backward compatible and able to honor Elasticsearch 7 API.

Please refer to Compatibility Matrix page for more information on the exact supported versions.

Embedded Mode

The default configuration uses an embedded OpenSearch instance that runs in the same JVM as the Nuxeo Platform's.

This embedded mode is only for testing purpose and should not be used in production, neither OpenSearch nor Nuxeo can support an embedded installation.

For production you need to setup a Search cluster.

Installing an Elasticsearch Cluster

Refer to the Elasticsearch documentation to install and secure your cluster. Basically:

An example on how to create a role:

curl -XPOST -u elastic 'localhost:9200/_security/role/nuxeo_role' -H "Content-Type: application/json" -d '{
  "cluster" : [
    "all"
  ],
 "indices" : [
   {
     "names" : [ "nuxeo*" ],
     "privileges" : [ "all" ]
   }
 ]
}'

An example on how to create a user for that role:

curl -XPOST -u elastic 'localhost:9200/_security/user/nuxeo_user' -H "Content-Type: application/json" -d '{
  "password" : "nuxeo_secret_password",
  "full_name" : "Nuxeo User",
  "roles" : [ "nuxeo_role" ]
}'

If you have a large number of documents or if you use Nuxeo in cluster you may reach the default configuration limitation, here are some recommended tuning:

Consider disabling the OS swapping or using other Elasticsearch option to prevent the heap to be swapped.

In /etc/default/elasticsearch file you can increase the JVM heap to half of the available OS memory:

# For a dedicated node with 12g of RAM
ES_JAVA_OPTS="-Xms6g -Xmx6g"

Installing an OpenSearch Cluster

Refer to the OpenSearch documentation to install OpenSearch. Basically:

An example on how to create a role:

curl -XPUT -u admin http://localhost:9200/_plugins/_security/api/roles/nuxeo_role -H "Content-Type: application/json" -d '{
  "cluster_permissions" : [
    "all"
  ],
 "index_permissions" : [
   {
     "index_patterns" : [ "nuxeo*" ],
     "allowed_actions" : [ "all" ]
   }
 ]
}'

An example on how to create a user for that role:

curl -XPUT -u admin http://localhost:9200/_plugins/_security/api/internalusers/nuxeo_user -H "Content-Type: application/json" -d '{
  "password" : "nuxeo_secret_password",
  "description" : "Nuxeo User",
  "backend_roles" : [ "nuxeo_role" ]
}'

If you have a large number of documents or if you use Nuxeo in cluster you may reach the default configuration limitation, here are some recommended tuning OpenSearch options

You can increase the JVM heap to half of the available OS memory:

# For a dedicated node with 12g of RAM
OPENSEARCH_JAVA_OPTS=-Xms6g -Xmx6g

Configuring Nuxeo to Access the Search Cluster

Nuxeo uses the Rest client protocol, you have to configure the access:

elasticsearch.addressList=http://somenode:9200,https://anothernode:443

Where:

  • elasticsearch.addressList is a comma separated list of URL.

Basic Authentication

If you have chosen to configure Basic Authentication then you can setup Nuxeo using nuxeo.conf with the follow properties:

elasticsearch.restClient.username=your_username
elasticsearch.restClient.password=your_password

TLS/SSL Configuration

If you have chosen to configure Elasticsearch TLS/SSL or OpenSearch TLS/SSL then you can set up Nuxeo using nuxeo.conf with the following properties:

elasticsearch.restClient.truststore.path
elasticsearch.restClient.truststore.password
elasticsearch.restClient.truststore.type
elasticsearch.restClient.keystore.path
elasticsearch.restClient.keystore.password
elasticsearch.restClient.keystore.type

If you are using TLS/SSL then the elasticsearch.addressList will need to be updated to include https.

See the Trust Store and Key Store Configuration page for more.

Index Names

Nuxeo manages 3 Elasticsearch indexes:

  • The repository index used to index document content, this index can be rebuild from scratch by extracting content from the repository.
  • The audit logs index to store audit entries, this index is a primary storage and can not be rebuild.
  • A sequence index used to serve unique value that can be used as primary keys, this index is also a primary storage.

To make the connection between the Nuxeo Platform instance and the ES cluster check the following options in the nuxeo.conf file and edit if you need to change the default value:

elasticsearch.indexName=nuxeo
elasticsearch.indexNumberOfReplicas=0
audit.elasticsearch.indexName=${elasticsearch.indexName}-audit
seqgen.elasticsearch.indexName=${elasticsearch.indexName}-uidgen

Where

  • elasticsearch.indexName is the name of the Elasticsearch index for the default document repository.
  • elasticsearch.indexNumberOfReplicas is the number of replicas. By default you have 1 shard and 1 replica. If you have a single node in your cluster you should set the indexNumberOfReplicasto 0. Visit the Elasticsearch Scalability documentation for more information on shards and replicas.
  • audit.elasticsearch.indexName is the name of the Elasticsearch index for audit logs.
  • seqgen.elasticsearch.indexName is the name of the Elasticsearch index for the uid sequencer, extensively used for audit logs.

You can find all the available options in the nuxeo.defaults.

Index Aliases and Reindexing without Service Interruption

Reindexing the repository can be a long operation depending on the size of the repository. This is an administrative procedure that is required in order to apply a new Elastic mapping or setting.

By default, when Nuxeo performs a reindex of the repository, it deletes and re-creates the Elastic index then it submits all the documents for indexation. During this operation only the indexed documents are searchable, this would impact strongly the user experience and, it requires a service interruption.

To avoid this Nuxeo can manage 2 indexes at the same time, the current one with continue to serve queries and index new document modifications, while the new one is going to reindex the entire repository (including the new updates). On completion Nuxeo will switch to the new index.

Nuxeo leverages Elastic Aliases to do this, it manages 2 aliases: one for searching using the name of the contrib (default to nuxeo), one for writing with a -write suffix (default to nuxeo-write), both aliases will point to the same index (nuxeo-0000 at the beginning). The index name ends with a number and is automatically incremented when reindexing.

Here is how to proceed:

  1. Nuxeo must be configured to manage Elastic aliases, add the elasticsearch.manageAlias.enabled=true in your nuxeo.conf
    Note that if you are switching an existing instance to use managed aliases it will require a service interruption. Stop Nuxeo and drop the existing nuxeo index, then activate the manage aliases option and, start Nuxeo, proceed to a repository reindexing while service is interrupted. The next reindexing will not require service interruption.
  2. Perform a full reindexing using the Bulk Service (the legacy reindexing will not work properly during reindexing).
  3. On completion, you have to delete the old unused index.

Note that using managed aliases requires more disk space on Elastic nodes because you have multiple indexes of the repository. Also, you have to manually delete old repository indexes when reindexing is completed.

Translog Tuning

To reduce disk IO you should consider changing the default translog durability from request to async. This can be done from nuxeo.conf:

elasticsearch.index.translog.durability=async

If your indexes are already created you need some manual operation to change the translog:

curl -H "Content-Type: application/json" -XPUT "http://localhost:9200/nuxeo-uidgen/_settings" -d '{
  "index.translog.durability" : "async"
}'

curl -H "Content-Type: application/json" -XPUT "http://localhost:9200/nuxeo-audit/_settings" -d '{
  "index.translog.durability" : "async"
}'

curl -H "Content-Type: application/json" -XPUT "http://localhost:9200/nuxeo/_settings" -d '{
  "index.translog.durability" : "async"
}'

Disabling Elasticsearch

Elasticsearch is enabled by default, if you want to disable Elasticsearch indexing and search you can simply add the following option to the nuxeo.conf:

elasticsearch.enabled=false

Disabling Elasticsearch for Audit Logs

When Elasticsearch is enabled and the audit.elasticsearch.enabled property is set to true in nuxeo.conf which is the case by default, Elasticsearch is used as a backend for audit logs.

This improves scalability, especially when using Nuxeo Drive with a large set of users.

When Elasticsearch is used as a backend for audit logs it becomes the reference (no more SQL backend as it was the case in Nuxeo versions lower than 7.3).

For this purpose make sure you read the Backing Up and Restoring the Audit Elasticsearch Index page.

If you want to disable Elasticsearch and use the SQL database as the default backend for audit logs you can simply update this property in nuxeo.conf:

audit.elasticsearch.enabled=false

Rebuilding the Repository Index

If you need to reindex the whole repository, you have different possibilities:

Re-index the Repository Using the WorkManager (the legacy way)

There are 3 ways to run it:

  1. From the Nuxeo Dev Tool Browser Extension.

  2. From JSF UI (DEPRECATED) > Admin center > Elasticsearch > Admin

  3. Using curl

curl -X POST "<NUXEO_URL>/nuxeo/site/automation/Elasticsearch.Index" -u Administrator:<PASSWORD> -H 'content-type: application/json' -d '{"params":{},"context":{}}'

Look at the server.log you should have 3 WARNs in the logs:

# start of re-indexing
WARN  [http-nio-0.0.0.0-8080-exec-31] [org.nuxeo.elasticsearch.web.admin.ElasticSearchManager] Re-indexing the entire repository: default
...
# all the repository have been scrolled we know how much document are going to be re-indexed
WARN  [Nuxeo-Work-elasticSearchIndexing-1:785116626625974.1486048658] [org.nuxeo.elasticsearch.work.ScrollingIndexingWorker] Re-indexing job: /elasticSearchIndexing:785116626625974.1486048658 has submited 270197 documents in 541 bucket workers
...
# end of the re-indexing
WARN  [Nuxeo-Work-elasticSearchIndexing-1:785120666169686.1890981267] [org.nuxeo.elasticsearch.work.BucketIndexingWorker] Re-indexing job: /elasticSearchIndexing:785116626625974.1486048658 completed.

You can fine tune the WorkManager indexing process using the following options:

  • Sizing the indexing worker thread pool. The default size is 4, using more threads will crawl the repository faster:

      elasticsearch.indexing.maxThreads=4

  • Tuning the number of documents per worker and the number of document submitted using the Elasticsearch bulk API:

      # Reindexing option, number of documents to process per worker
  elasticsearch.reindex.bucketReadSize=500
  # Reindexing option, number of documents to submit to Elasticsearch per bulk command
  elasticsearch.reindex.bucketWriteSize=50

Re-index Repository Using Bulk Service

Run a bulk command to re-index the repository, the command id is returned:

curl -s -X POST "<SERVER_URL>/nuxeo/site/automation/Elasticsearch.BulkIndex" -u Administrator:<PASSWORD> -H 'content-type: application/json' -d '{"params":{},"context":{}}'

{"commandId": "21aeaea1-0ef0-4a89-a92d-fa8f679361de"}

At any time, you can request the status of the re-indexing using the previous command id:

curl -s -X GET "<SERVER_URL>/nuxeo/api/v1/bulk/21aeaea1-0ef0-4a89-a92d-fa8f679361de" -u Administrator:<PASSWORD> -H 'content-type: application/json'

{
  "entity-type": "bulkStatus",
  "commandId": "21aeaea1-0ef0-4a89-a92d-fa8f679361de",
  "state": "RUNNING",
  "processed": 200,
  "error": false,
  "errorCount": 0,
  "total": 42932,
  "action": "index",
  "username": "Administrator",
  "submitted": "2020-11-16T15:26:50.346Z",
  "scrollStart": "2020-11-16T15:26:50.432Z",
  "scrollEnd": "2020-11-16T15:26:50.446Z",
  "processingStart": null,
  "processingEnd": null,
  "completed": null,
  "processingMillis": 0
}

Changing Mappings and Settings of Indexes

Updating Repository Index Configuration

Nuxeo comes with a default mapping that sets the locale for full-text and declares some fields as being date or numeric.

For fields that are not explicitly defined in the mapping, Elasticsearch will try to guess the type the first time it indexes the field. If the field is empty it will be treated as a String field. This is why most of the time you need to explicitly set the mapping for your custom fields that are of type date, numeric or full-text. Also fields that are used to sort and that could be empty need to be defined to prevent an unmapped field error.

The default mapping is located in the ${NUXEO_HOME}/templates/common-base/nxserver/config/elasticsearch-config.xml.nxftl.

To override and tune the default mapping:

Instead of overriding the extension point you can simply override the default mapping or settings JSON files:

  1. Create a custom template like myapp with a nuxeo.defaults file that contains:

     myapp.target=.

  2. In this custom template create a file named nxserver/config/elasticsearch-doc-mapping.json to override the mapping. You can create a file named nxserver/config/elasticsearch-doc-settings.json to override the settings.
    Important: You must add your custom mapping to the existing one, you cannot just set your custom mapping in the file, Nuxeo does not merge your mapping with the default one. So, you must duplicate the original file at ${NUXEO_HOME}/templates/common-base/nxserver/config/elasticsearch-doc-mapping.json to myapp/nxserver/config/elasticsearch-doc-mapping.json , and modify the copy.

  1. Update the nuxeo.conf to use your custom template.

     nuxeo.templates=default,myapp

  2. Restart and re-index the entire repository (see previous section). A re-indexing is needed to apply the new settings and mapping.

For mapping customization examples, see the page Configuring the Elasticsearch Mapping.

Updating the Audit Logs Index Configuration

Here the index is a primary storage and you cannot rebuild it. So we need a tool that will extract the _source of documents from one index and submit it to a new index that have been setup with the new configuration.

  1. Update the mappings or settings configuration by overriding the {NUXEO_HOME}/templates/common-base/nxserver/config/elasticsearch-audit-index-config.xml(follow the same procedure as the section above for the repository index)
  2. Use a new name for the audit.elasticsearch.indexName (like nuxeo-audit2)
  3. Start the Nuxeo Platform.
    The new index is created with the new mapping.
  4. Stop the Nuxeo Platform

  5. Copy the audit logs entries in the new index using the _reindex endpoint. Here we copy nuxeo-audit to nuxeo-audit2.

     curl -X POST http://localhost:9200/_reindex -H 'Content-Type: application/json' -d '{
 "source": {
 "index": "nuxeo-audit"
 },
 "dest": {
 "index": "nuxeo-audit2"
 }
 }'

Configuration for Multi Repositories

You need to define an index for each repository. This is done by adding an elasticSearchIndex contribution.

  1. Create a custom template as described in the above section "Changing the mapping of the index".

  2. Add a second elasticSearchIndex contribution:

     <elasticSearchIndex name="nuxeo-repo2" type="doc" repository="repo2"> ....

    Where name is the Elasticsearch index name and repository the repository name.

Investigating and Reporting Problems

Activate Traces

To understand why a document is not present in search results or not indexed, you can activate a debug trace.

Open at the lib/log4j2.xml file and uncomment the ELASTIC section:

<!-- Elasticsearch logging -->
<File name="ELASTIC" fileName="${sys:nuxeo.log.dir}/elastic.log" append="false">
  <PatternLayout pattern="%d{ISO8601} %-5p [%t] [%c] %m%n" />
</File>

<Logger name="org.nuxeo.elasticsearch" level="trace" additivity="false">
  <AppenderRef ref="ELASTIC" />
</Logger>

The elastic.log file will contain all the requests done by the Nuxeo Platform to Elasticsearch including the curl command ready to be copy/past/debug in a term.

Reporting Settings and Mapping

It is also important to report the current settings and mapping of an Elasticsearch index (here called nuxeo)

curl localhost:9200/nuxeo/_settings?pretty > /tmp/nuxeo-settings.json
curl localhost:9200/nuxeo/_mapping?pretty > /tmp/nuxeo-mapping.json
# misc info and stats on Elasticsearch
curl localhost:9200 > /tmp/es-info.txt
curl localhost:9200/_cluster/stats?pretty >> /tmp/es-info.txt
curl localhost:9200/_nodes/stats?pretty >> /tmp/es-info.txt
curl localhost:9200/_cat/health?v >> /tmp/es-info.txt
curl localhost:9200/_cat/nodes?v >> /tmp/es-info.txt
curl localhost:9200/_cat/indices?v >> /tmp/es-info.txt

Testing an Analyzer

To test the full-text analyzer:

curl -s -X GET "localhost:9200/nuxeo/_analyze" -H 'Content-Type: application/json' -d' {
  "analyzer" : "fulltext",
  "text" : "This is a text for testing, file_name/1-foos-BAR.jpg"
}'

To test an analyzer derived from the mapping:

curl -s -X GET "localhost:9200/nuxeo/_analyze" -H 'Content-Type: application/json' -d' {
  "field" : "ecm:path.children",
  "text" : "workspaces/main folder/sub-folder"
}'

Viewing Indexed Terms for Document Field

This can be done using tool like Luke to analyze at the Lucene index level. It is also possible to use aggregate on fields that are not text or text with fielddata option:

# view indexed tokens for dc:title.fulltext of document 3d50118c-7472-4e99-9cc9-321deb4fe053
curl -XGET 'localhost:9200/nuxeo/doc/_search?pretty' -H 'Content-Type: application/json' -d'{
 "query" : {"ids" : { "values" : ["3d50118c-7472-4e99-9cc9-321deb4fe053"] }},
 "aggs": {"my_aggs": {"terms": {"field": "dc:title", "order" : { "_count" : "desc" }, "size": 1000}}}}'

You may need to change the size parameter to get more or less indexed terms.

Explain and Profile Elasticsearch Queries

When trace level logs are actived, Elasticsearch curl command will be present in the elastic.log log file. Getting more details on what is happening during the query execution, can either be done using explain or profile. Those two approaches will help to understand the mapping and the field scoring, it can also gives inputs about unmapped fields for example.

Comparing the Elasticsearch Index with the Database Content

You can use the esync tool to compare both content and pinpoint discrepancies.

This tool is a read-only standalone tool, it requires both access to the database and Elasticsearch (using transport client on port 9300).

Other Elasticsearch Documentation
Other Related Documentation