From Martin Fowler - Continuous Integration (an introduction)
Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly.
See http://en.wikipedia.org/wiki/Continuous_Integration
It's as important to follow quality processes on development as to maintain this quality among time. Nuxeo is involved in such practices that will guarantee or reinforce its products quality.
Rules and means
Nuxeo products and tools are continuously built over time, at each change and against multiple environments. Nuxeo QA team sets and maintain a QA environment applying CI rules and so providing to developers means to check their code quality and being warned in case of any problem.
- Maintain a code repository
Nuxeo sources repositories on mercurial and git are under continuous integration.
This includes Nuxeo EP, Nuxeo addons, Nuxeo RCP, Nuxeo WebEngine, Nuxeo Books, tools and plugins and of course all our customers' projects. - Builds are automated
This is done by Hudson on Nuxeo QA Unit, Functional and Integration tests. - Every commit on mainline is integrated
When code is committed, target project is built, as all projects depending on it. The full chain is verified, from build to deployment.
Mainlines on Nuxeo EP and addons are the main branches in development: 5.1 and 5.2 (resp. 1.4 and 1.5 for associated subtrees). For projects under SubVersion, that means the trunk and, if exists, 5.1 branch. - Everyone can see the results of the latest build
Hudson plugins ensure to warn potential responsible(s) of build fail by mail and jabber, so they can react quickly.
Moreover, every build fail is sent on ECM QA mailing list. - Make it easy to get the latest deliverables
Nightly builds are done. Produced artifacts are published on our Maven repositories maven.nuxeo.org. Currently managed with Nexus, our repositories store all released artifacts and recent snapshots. - Keep the build fast
Continuous Integration is done on multiple servers, more or less powerful, using slaves in order to distribute the load.
Thanks to Maven and to Nuxeo modularity, each module is built separately and as a consequence, quickly. - Test in a clone of the production environment
We have two integration levels: unit and functional.
First level checks code compilation and runs Unit tests. A lot of Unit tests simulate target environments (with mock objects). Dependent projects/modules are then added to the CI chain.
Second level runs packaging tools and automated deployment against multiple environments (we aim at covering JVM versions, SQL backends, OS, browsers, performance, ...). Finally we use Selenium tests to check functional integrity. This also indirectly provides a continuous integration on our tools (packaging, convenient scripts...).
Quality directives for Nuxeo developers
These practices apply on every script, project or module. They should be strictly followed.
- Code must be under continuous integration.
Except for prototype and spike solutions (sandbox projects or temporary branches), all projects must be under CI. If not, ask for it to the QA team, providing the informations mentioned in the following Hudson part. - Automate the build
Think about QA tools that will have to test the project without any human intervention. Provide Maven, Ant or, in the worse case, Shell autonomous configuration. - Make your build self-testing
Think "test-driven development". Simply building a project/module and running its Unit tests should be a valuable measurement of the code stability. Unit tests code coverage often needs to be increased. - Commit every day
Smaller are the commits, lower is the risk of conflicting changes and easier is the bug analysis. - Stay tuned
Be aware of CI builds, particularly failed builds.
Log on http://qa.nuxeo.org/hudson/ and check your profile's informations, especially your jabber address. Hudson will then be able to contact you via Jabber when you are suspected of having broken something.
Subscribe to ECM QA mailing list. Use mail filters to quickly catch and fix problems. Hudson will send you a mail if it detects one of your commits between succeed and failed tests.
If you're used to, RSS feeds are also available.
Check regularly your projects health on our QA sites. Inform QA team if you notice any issue.
Always consider a build failed as an emergency.
Maven parent POM
Maven Parent POM file gives a lot of useful information. Take care to fill in you project's pom.xml file:
- main tags
- "scm" tag
- "developers" tag (there's no rule for tags within the "developer" tag, feel free to add useful information such as "role", "url", "organization" or "module")
- You also have to add <repositories> section in the project's parent POM in order to make your project fully autonomous.
All these tags are intelligently inherited so that, if you're project's Maven parent is nuxeo-ecm or one of its children, you don't have to repeat informations such as "organization", "licenses", "mailingLists", "issueManagement". Also, when working on a project with sub-modules, it's only necessary to set "scm" on the parent POM.
Hudson configuration
Adding a project under continuous integration in Hudson requires:
- SCM URL
- Build command
In case of Maven, it's the goals to run (usually, it will be "clean install"). Consider using Maven "Profiles" to manage different behaviors like development versus production environment.
In case of Ant, you may need to provide some parameters on the command line (equivalent to what can be set by a human user in a build.properties).
In case of Shell (avoid it as much as possible), it's a simple command with working default values in case of required parameters. If needed, some environment constants may be set.
- Notification target(s): eg, the team mailing list.
Release process
Overview
Release process is managed and tested by multiple tools:
- Hudson continuous integration which generates a candidate release every night based on snapshots.
- Selenium tests
- Funkload tests
- IZPack for jar packaging
- NSIS Setup Script for Windows packaging
- dpkg scripts for Debian packaging
When a release is wanted, all continuous integration chain must be satisfied, nightly builds are manually tested to complete automated tests and candidate release is "promoted" to public release (code is tagged, artifacts are uploaded to the maven repository and packages are published on Nuxeo web site.
Continuous integration coverage
For now, continuous integration covers those configurations:
- Nuxeo DM.
- Linux Ubuntu (Debian).
- Sun Java 5, Sun Java 6.
- JBoss application server.
- VCS backend on H2.
- VCS backend on PostgreSQL.
Not automatically tested at integration level (full deployment and tests): - Nuxeo EP.
- Nuxeo-shell and scripts.
- JCR backend with various databases: H2, Derby, PostgreSQL, MySQL, Oracle, ...
- VCS backend with various databases: Derby, MySQL, Oracle, ...
- Various application servers: Jetty, GlassFish (GF3) and Tomcat.
- Various Windows OS.
- Various Java providers.
Help testing release candidates
You can download nightly candidate releases from http://qa.nuxeo.org/hudson/view/NX Deployments/, test and send feedback on our ECM mailing list or, in case of bugs confirmed, in our Issue Tracker (Jira).