Upgrade from LTS 2016 to LTS 2017

From LTS 2016 to LTS 2017

Installation and Configuration

Reindex the full repository following Rebuilding the repository index page, or using Nuxeo Dev Tools Extension. See NXP-21279.

Requirements

• MySQL >= 5.6.4 is required. See NXP-21338

• Elasticsearch >= 5.6.x is required.

Parameters to Update

New Parameters

Nuxeo Studio Application Definition

Make sure your Nuxeo Studio project depends on the Nuxeo JSF UI target package. See the pages How to Update the Version of Your Target Platform in Studio and Specific Upgrade Instructions for more details.

Data

Elasticsearch

Elasticsearch 5.6.x is required. Follow those necessary steps to upgrade:

1. Upgrade Elasticsearch version
2. Update your custom settings and mapping to follow new Elasticsearch rules
3. Re-index the repository to apply the new settings and mapping
4. Update your custom code that queries Elasticsearch directly

Upgrade Elasticsearch Version

If your indices have been created with LTS 2016 they are in Elasticsearch 2.x format and can be read by Elasticsearch 5.6, in this case follow the full cluster restart upgrade procedure.

If your indices have been created before LTS 2016 they are in Elasticsearch 1.x format and Elasticsearch 5.x will not start, in this case an index need to be migrated to the new Elasticsearch 5.x format:

• The repository index named nuxeo by default doesn't need this migration because the repository will be re-indexed in the next step, so once this index has been backed up you can delete it.
• The sequence index named nuxeo-uidgen cannot be migrated because the _source field is disabled, Nuxeo will take care to re-create this index at startup, so once this index has been backed up you can delete it.
• The audit index named nuxeo-audit need to be migrated. Follow the reindex upgrade procedure.
• In case the current nuxeo-audit index contains a dueDate field (Workflow usage), the format has changed from long to Date. This means the mapping in the target index should have the dueDate field type set to Date before reindexing. This could be done by adding the following to the nuxeo mapping or your custom one if needed:

    "extended": {
      "properties": {
        "dueDate": {
          "type": "date"
        }
      }
    },
  

  If not migrated the field will generate errors like the following:

  2018-09-18 10:39:40,100 ERROR [AuditLogWriterPool-00,in:168636,inCheckpoint:168617,out:0,lastRead:1537259979915,lastTimer:1537259979752,wm:201491740013363201,loop:1285532,record] [org.nuxeo.elasticsearch.audit.ESAuditBackend] Unable to index audit entry 16 :MapperParsingException[failed to parse [extended.dueDate]]; nested: NumberFormatException[For input string: "2018-09-19T10:39:39.486+02:00"];
  

  Please follow the audit purge documentation for further details on how to proceed.

Please refer to Elasticsearch documentation for more information on upgrading your Elasticsearch cluster.

Update Your Custom Elasticsearch Settings and Mapping

If you have a custom configuration for Elasticsearch it needs to be adapted to follow the new 5.6 rules. It is probably easier to start from the default Nuxeo 9.3 configuration and migrate your necessary change instead of modifying your existing contribution directly.

Since Nuxeo 9.3, you can configure the settings and mapping by overriding JSON files in your custom template:

• The default settings is located in templates/common-base/nxserver/config/elasticsearch-doc-settings.json.nxftl. The important changes are that settings no longer supports an analyzer alias (the fulltext alias need to be removed).

• The default mapping is located in templates/common-base/nxserver/config/elasticsearch-doc-mapping.json. The important changes are:

  • Type string must be rewritten to keyword (in place of not_analyzed string) or text (the analyzed string version)
  • not_analyzed is deprecated type it should be replaced by keyword
  • Type multi_field does not exists anymore, it must be rewritten as type keyword or text
  • _all is disabled. A custom all_field is used instead. By default all string fields are copied to this field, to not index a field this requires to add a mapping without the "copy_to": "all_field" instruction.
  • Text field used for sorting or aggregating must be of type keyword or text with fielddata=true property
  • Text field used with NXQL LIKE requires a text field. If the field is also used for sorting it must be fielddata. For instance dc:title:

      "dc:title": {
        "type": "text",
        "copy_to": "all_field",
        "fielddata": true,
        "fields": {
          "fulltext": {
            "type": "text",
            "analyzer": "fulltext"
          }
        }
      },
    

  • Highlight must be done on text field with a fulltext analyzer.

Re-Index the Repository

The new mapping and setting for the repository index must be applied, this means that the entire repository must be re-indexed.

When using the JSF UI this can be done from the Admin > Elasticsearch > Admin page.

Or this can be done via REST:

curl -v -X POST 'http://localhost:8080/nuxeo/site/automation/Elasticsearch.Index' -u Administrator:Administrator -H 'content-type: application/json+nxrequest' -d '{"params":{},"context":{}}'

Update Your Custom Code That Query Elasticsearch Directly

Any custom native queries done using the passthrough or code need to be reviewed to follow Elasticsearch 5 query format, for instance:

• filtered query must be replaced with bool query
• not filter should be replaced with a bool query
• Aggregate size=0 to get all buckets is not anymore supported the size must be explicitly defined (> 0)
• There is no more default _all field. query_string and simple_query_string must use the new custom all_field field.

Keeping Old Tags

The tag implementation has changed in 9.3. If you want to keep your old tags, add the following contribution:

  <require>org.nuxeo.ecm.platform.tag.service.migrator</require>
  <extension target="org.nuxeo.runtime.migration.MigrationService" point="configuration">
    <migration id="tag-storage">
      <defaultState>relations</defaultState>
    </migration>
  </extension>

If you want to migrate tags to the new storage model, follow the Tag migrations steps.

Tag Migration

To migrate tags to the new storage model:

1. Follow the step from section Keeping old tags.

2. In the Nuxeo Platform's JSF UI, go to Admin > System Information > Migration, click the button Migrate tags from relations to facets and wait until migration is completed.

3. Remove the contribution added at step 1.

Directories

MongoDB Directories

Directories are now stored in MongoDB when enabling the mongodb template. You don't need to install the addon Nuxeo MongoDB Audit and Directories anymore, which is only needed for Audit now. See NXP-21582.

Migrating your directories to MongoDB

To migrate your custom directories to MongoDB, enable the mongodb template and use the extension point org.nuxeo.ecm.directory.GenericDirectory instead of org.nuxeo.ecm.directory.sql.SQLDirectoryFactory. See NXP-19175.

Using MongoDB with SQL directories

If you want to keep your custom SQL directories, contribute them to the extension point org.nuxeo.ecm.directory.sql.SQLDirectoryFactory.

Code Changes

Automation Scripting

Backward Compatibility was broken for 9.1 (NXP-21508): parameters are no longer extracted from the context of the params object when calling the scripting operation. You have to explicitly set each parameter in the operation definition. Follow NXP-22190 for resolution.

Started from 9.2: you can set the parameter nuxeo.automation.scripting.inline-context-in-params to true in order to get back compatibility with scripts created before 9.1.

Code Behavior Changes

• Setting a property value to null now always sets it to the default value if a default value is configured for the schema field. See NXP-19466.

• Expression Language artifact has been renamed from org.nuxeo.ecm.platform:nuxeo-platform-el to org.nuxeo.ecm.core:nuxeo-core-el. See NXP-21713.

• When using the feature RepositoryElasticSearchFeature, you need to add a test Maven dependency on org.nuxeo.ecm.automation:nuxeo-automation-server:

    <dependency>
      <groupId>org.nuxeo.ecm.automation</groupId>
      <artifactId>nuxeo-automation-server</artifactId>
      <scope>test</scope>
    <dependency>
  

  See NXP-21186.

• Class org.nuxeo.ecm.core.blob.BlobManager.BlobInfo moved to org.nuxeo.ecm.core.blob.BlobInfo. See NXP-22217.

• In nuxeoctl, the register-trial command reads parameters: ./nuxectl register-trial [<firstname> <lastname> <email> <company> <project>]. See NXP-22550.

• During OAuth2 challenge (as identity provider), parameters state and redirect_uri are now mandatory for security reason. See NXP-22183 and NXP-22186.

• A new statusCode field has been added to NuxeoException to specify which HTTP code should be returned in case the exception is thrown, default to 500. See NXP-21776.

• New org.nuxeo.ecm.platform.query.api.PageProvider#getResultsCountLimit method to access the limit of number of results beyond which the page provider may not be able to compute the result count. See NXP-23202.

• Setting the value of a complex property now overwrites any previous values, and does not do a partial update anymore. See NXP-19261.

• NuxeoGroupImpl class has been moved to org.nuxeo.ecm.platform.usermanager package. The code relying on it must be updated. See NXP-20619.

Operation Changes

• Operations Services.CreateUser and Services.QueryUsers are now part of the base platform. They have been renamed into User.CreateOrUpdate and User.Query.
  Aliases with old names have been kept for compatibility. See NXP-21962

• Tag.Suggestion operation no longer requires JSF; You have to pass the target document using the new parameter document. See NXP-21717.

• Document.AddPermission operation: The existence of the input user name or group name is now checked before adding a permission. See NXP-21559.

• Operations Document.CopySchema, Document.AddFacet and Document.RemoveFacet have a new parameter save to force a session save or not. See NXP-22912.

Deprecated APIs

Nuxeoctl register-trial

The command nuxeoctl register-trial has been deprecated. See NXP-23122.

Automatic Versioning

We introduced a new automatic versioning system in 9.1. We deprecated versioningRules extension point. You now have to use new extension points, see documentation.

Nuxeo provides a backward compatibility mechanism, but it could be not accurate.

Previous automatic versioning mechanisms were removed or upgraded to new system:

• File manager and bulk edit don't automatically version documents since 9.1, you could put back this behavior by contributing a new rule and use source parameter to restrict versioning to this functionalities (for instance use: fileimporter-{name-of-your-importer} or bulkEdit)
• Nuxeo Drive is not responsible of automatic versioning anymore. The last contributor rule is now a default policy, and delay rule is introduced by Nuxeo Drive and rely on source parameter mechanism.

It's the case for:

• nuxeo.drive.force.versioning
• versioningDelay and versioningOption used in DefaultFileSystemItemFactory.
  Note that VersioningFileSystemItemFactory is deprecated.

Fields file:filename and common:size Removed

To fix exports, manually edit the ZIP file (prior to 9.1) by removing <common:size> and <file:filename> elements from the files document.xml. See NXP-21025.

FileManager Can Now Prevent Overwriting Existing Document

Added:

• FileManager#createFolder(CoreSession documentManager, String fullname, String path, boolean overwrite)
• FileManagerService#defaultCreateFolder(CoreSession documentManager, String fullname, String path, boolean overwrite)
• FileManagerService#defaultCreateFolder(CoreSession documentManager, String fullname, String path, String containerTypeName, boolean checkAllowedSubTypes, boolean overwrite)

Deprecated:

• FileManager#createFolder(CoreSession documentManager, String fullname, String path)
• FileManagerService#defaultCreateFolder(CoreSession documentManager, String fullname, String path)
• FileManagerService#defaultCreateFolder(CoreSession documentManager, String fullname, String path, String containerTypeName, boolean checkAllowedSubTypes)

See NXP-21853.

BlobDispatcher Can Now Dispatch Depending on Document's XPath

Added:

• BlobDispatcher#getBlobProvider(Document doc, Blob blob, String xpath)

Deprecated:

• BlobDispatcher#getBlobProvider(Document doc, Blob blob)

See NXP-21891.

Scoped DocumentModel Context Data

Any usage of ScopedMap or ScopeType is deprecated and will be removed in the future.

• Instead of ScopedMap, use Map<String, Serializable>.
• Instead of ScopeType in getContextData and putContextData, use equivalent methods without a scope.

See NXP-21570.

LoginScreenHelper Mobile Methods

Removed:

• LoginScreenHelper#NUXEO_PROTOCOL
• LoginScreenHelper#getURLForMobileApplication(HttpServletRequest request)
• LoginScreenHelper#getURLForMobileApplication(String baseURL, String requestedURL)

Use org.nuxeo.ecm.platform.web.common.MobileBannerHelper instead. See NXP-21679.

Oauth2Client

• Operations Services.CreateUser and Services.QueryUsers are now part of the base platform. They have been renamed into User.CreateOrUpdate and User.Query.
  Aliases with old names have been kept for compatibility. See NXP-21962

• Constructor OAuth2Client(String name, String id, String secret) is deprecated. See NXP-22183.

WebException

WebException (and most subclasses) has been deprecated, NuxeoException (and subclasses) are now the exceptions to be thrown. See NXP-21776.

Group Entity-type (REST)

The groupname and grouplabel properties at the root are deprecated. id property must be used instead of groupname. Both are now part of the properties field with all others group schema properties. See NXP-22542.

Addons

Nuxeo Drive - Server APIs Changes

Added:

• FileSystemItemManager#createFolder(String parentId, String name, Principal principal, boolean overwrite)
• FileSystemItemManager#createFile(String parentId, Blob blob, Principal principal, boolean overwrite)
• FolderItem#createFolder(String parentId, String name, Principal principal, boolean overwrite)
• FolderItem#createFile(String parentId, Blob blob, Principal principal, boolean overwrite)
• the overwrite parameter to the NuxeoDrive.CreateFolder operation
• the overwrite parameter to the NuxeoDrive.CreateFile operation

Deprecated:

• FileSystemItemManager#createFolder(String parentId, String name, Principal principal)
• FileSystemItemManager#createFile(String parentId, Blob blob, Principal principal)
• FolderItem#createFolder(String parentId, String name, Principal principal)
• FolderItem#createFile(String parentId, Blob blob, Principal principal)

See NXP-21854.

Nuxeo Drive - Synchronization Root Behavior

Synchronization root is now reset with a copied document. In LTS 2016 and earlier, copying a synchronization root Document resulted in creating a new synchronization root.

To re-activate the old behavior:

<extension target="org.nuxeo.runtime.ConfigurationService" point="configuration">
    <property name="org.nuxeo.drive.resetSyncRootsOnCopy">false</property>
</extension>

See NXP-21676.

Segment.io

Nuxeo Segment.io plugin is now aligned on com.github.segmentio:analytics:1.0.7, thus it has received a big update and breaks backward compatibility. See NXP-22462 and NXP-22212.

Nuxeo Mqueue - Deprecated

The nuxeo-mqueues addon is replaced by nuxeo-stream. See NXP-23854.

Complementary Information

• Upgrade notes for LTS 2017
• Release notes for LTS 2017