Quality Assurance and Continuous Integration

From Martin Fowler - Continuous Integration (an introduction)

Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly.

See http://en.wikipedia.org/wiki/Continuous_Integration

It's as important to follow quality processes on development as to maintain this quality among time. Nuxeo is involved in such practices that will guarantee or reinforce its products quality.

Watch the related courses on Hyland University

Expert Session on the Nuxeo Software Factory

Rules and Means

Nuxeo products and tools are continuously built over time, at each change and against multiple environments. Nuxeo QA team sets and maintain a QA environment applying CI rules and so providing to developers means to check their code quality and being warned in case of any problem.

Maintain a code repository
Nuxeo sources repositories on Mercurial and Git are under continuous integration.
This includes the Nuxeo Platform, the documentation, the tools and the plugins.
Builds are automated
This is done by Jenkins on Nuxeo QA Unit, Functional and Integration tests.
Every commit on mainline is integrated
When code is committed, target project is built, as all projects depending on it. The full chain is verified, from build to deployment.
Mainlines on Nuxeo EP and addons are the main branches in development: 5.1 and 5.2 (resp. 1.4 and 1.5 for associated subtrees). For projects under SubVersion, that means the trunk and, if exists, 5.1 branch.
Everyone can see the results of the latest build
Jenkins plugins ensure to warn potential responsible(s) of build fail by mail and jabber, so they can react quickly.
Moreover, every build fail is sent on ECM QA mailing list.
Make it easy to get the latest deliverables
Nightly builds are done. Produced artifacts are published on our Maven repositories maven.nuxeo.org. Currently managed with Nexus, our repositories store all released artifacts and recent snapshots.
Keep the build fast
Continuous Integration is done on multiple servers, more or less powerful, using slaves in order to distribute the load.
Thanks to Maven and to Nuxeo modularity, each module is built separately and as a consequence, quickly.
Test in a clone of the production environment
We have two integration levels: unit and functional.
First level checks code compilation and runs Unit tests. A lot of Unit tests simulate target environments (with mock objects). Dependent projects/modules are then added to the CI chain.
Second level runs packaging tools and automated deployment against multiple environments (we aim at covering JVM versions, SQL backends, OS, browsers, performance, ...). Finally we use Selenium tests to check functional integrity. This also indirectly provides a continuous integration on our tools (packaging, convenient scripts...).

Quality Directives for Nuxeo Developers

These practices apply on every script, project or module. They should be strictly followed.

Code must be under continuous integration Except for prototype and spike solutions (sandbox projects or temporary branches), all projects must be under CI. If not, ask for it to the QA team, providing the informations mentioned in the following Jenkins part.

Automate the build Think about QA tools that will have to test the project without any human intervention. Provide Maven, Ant or, in the worse case, Shell autonomous configuration.

Make your build self-testing Think "test-driven development". Simply building a project/module and running its Unit tests should be a valuable measurement of the code stability. Unit tests code coverage often needs to be increased.

Commit every day Smaller are the commits, lower is the risk of conflicting changes and easier is the bug analysis.

Stay tuned Be aware of CI builds, particularly failed builds.

Log on http://qa.nuxeo.org/ and check your profile's informations, especially your Jabber address. Jenkins will then be able to contact you via Jabber/XMPP/GTalk when you are suspected of having broken something.
Subscribe to ECM QA mailing list. Use mail filters to quickly catch and fix problems. Jenkins will send you a mail if it detects one of your commits between succeed and failed tests.
If you're used to, RSS feeds are also available.
Check regularly your projects health on our QA sites. Inform QA team if you notice any issue.
Always consider a build failed as an emergency.

Maven Parent POM

Maven Parent POM file gives a lot of useful information. Take care to fill in you project's pom.xml file:

Main tags.

<name>Nuxeo ECM Projects</name>
<description>Nuxeo ECM Platform and related components</description>
<organization>
  <name>Nuxeo</name>
  <url>http://www.nuxeo.com/</url>
</organization>

<licenses>
  <license>
    <name>GNU LESSER GENERAL PUBLIC LICENSE, Version 2.1</name>
    <url>http://www.gnu.org/licenses/lgpl-2.1.html</url>
  </license>
</licenses>

<mailingLists>
  <mailingList>
    <name>Nuxeo ECM list</name>
    <subscribe>http://lists.nuxeo.com/mailman/listinfo/ECM</subscribe>
    <unsubscribe>http://lists.nuxeo.com/mailman/listinfo/ECM</unsubscribe>
    <archive>http://lists.nuxeo.com/pipermail/ecm/</archive>
  </mailingList>
</mailingLists>

<issueManagement>
  <system>jira</system>
  <url>http://jira.nuxeo.org/browse/NXP</url>
</issueManagement>

<ciManagement>
  <system>Jenkins</system>
  <url>http://qa.nuxeo.org/jenkins/</url>
  <notifiers>
    <notifier>
      <type>mail</type>
      <configuration>

<address>[email protected]</address>
      </configuration>
    </notifier>
  </notifiers>
</ciManagement>

SCM tag.

<scm>
  <connection>scm:git:git://github.com/nuxeo/nuxeo-sample-project.git</connection>
  <developerConnection>scm:git:ssh://[email protected]:nuxeo/nuxeo-sample-project.git</developerConnection>
  <url>https://github.com/nuxeo/nuxeo-sample-project</url>
</scm>

Developers tag (there's no rule for tags within the "developer" tag, feel free to add useful information such as "role", "url", "organization" or "module").

<developers>
  <developer>
    <id>Nuxeo</id>
    <name>Nuxeo Developers Team</name>
    <email>[email protected]</email>
    <timezone>+1</timezone>
  </developer>
</developers>

You also have to add section in the project's parent POM in order to make your project fully autonomous.

<repositories>
  <repository>
    <id>public</id>
    <url>http://maven.nuxeo.org/nexus/content/groups/public</url>
    <releases>
      <enabled>true</enabled>
    </releases>
    <snapshots>
      <enabled>false</enabled>
    </snapshots>
  </repository>
  <repository>
    <id>public-snapshot</id>
    <url>http://maven.nuxeo.org/nexus/content/groups/public-snapshot</url>
    <releases>
      <enabled>false</enabled>
    </releases>
    <snapshots>
      <enabled>true</enabled>
    </snapshots>
  </repository>
</repositories>

All these tags are intelligently inherited so that, if your project's Maven parent is org.nuxeo:nuxeo-ecm or one of its children, you don't have to repeat informations such as "organization", "licenses", "mailingLists", "issueManagement". Also, when working on a project with sub-modules, it's only necessary to set "scm" on the parent POM.

Jenkins Job Configuration

See page Jenkins Job Configuration.

Release Process

Overview

The release process is managed and tested by multiple tools:

Jenkins Continuous Integration jobs which generates a release candidate (aka date-based release) every night from the development branch.
Python scripts.
Integration scripts.
Integration and Functional tests with Selenium, WebDriver, Funkload... against all the supported environments (OS, DB, JVM...).
NSIS scripts for Windows packaging.
dpkg scripts for Debian packaging.

When a release is wanted, all the continuous integration chain must be satisfied, nightly builds are manually tested to complete automated tests and the release candidate is "promoted" to a final release (code tags are pushed to GitHub, artifacts are uploaded to the Maven repositories and packages are published on Nuxeo web sites).

Continuous Integration Coverage

The current coverage includes those configurations:

Nuxeo Platform build and unit testing.
Nuxeo CAP distribution integration and functional testing.
Nuxeo addons and their Nuxeo Packages (Unit, Integration and Functional tests).
Java Oracle and OpenJDK 6, 7, 8 (depending on Nuxeo Platform version) and a few other Java providers.
Tomcat application server.
VCS backend on supported databases (H2, PostgreSQL, Oracle, MySQL, MSSQL).
Nuxeo scripts.
Linux Ubuntu (Debian), Mac OS X, Windows.

Help Testing

Download a Release

We recommend to download and test the latest FastTrack release from http://www.nuxeo.com/en/downloads or http://community.nuxeo.com/static/releases/.

You can download nightly candidate releases from http://qa.nuxeo.org/jenkins/view/Depl/, the snapshot APT repository or http://community.nuxeo.com/static/snapshots/.

You can also build the source code or download the latest SNAPSHOT.

Feedback

Test and report feedback on http://answers.nuxeo.com/. In case of bugs confirmed, an issue will be filled in https://jira.nuxeo.com/.