Nuxeo Core Developer Guide

Isolated Testing

Updated: October 22, 2018 Page Information Edit on GitHub

Here are solutions for testing your developments before merging them on the master branch.

"On-Demand Test and Push" Jobs

Developers have their own dedicated job for building a branch, testing it and eventually automatically merging it if succeed.

  1. Push your changes to the GitHub Nuxeo repo(s) of your choice under a branch named "feature-NXP-xxxx-description" (see Nuxeo common usage and best practices about branch naming).
  2. Create a new Jenkins job and choose the "Nuxeo On-Demand test and push" template
  3. Name it ondemand-testandpush-USERNAME[-suffix] (for instance: "ondemand-testandpush-jcarsique").
  4. Configure the parameters and save. The job is ready for use.
  5. Click on Build with Parameters.
  6. Fill the build parameters. See the help tooltip for details.

Configuration Details

You can view the resulting configuration by adding /config.xml to the job URL. Here are a few details about the job configuration.

Old Builds (Log Rotation Strategy)

  • Days to keep builds: 21
  • Max # of builds to keep: 100
  • Days to keep artifacts: (empty)
  • Max # of builds to keep with artifacts: 1

Restrict Where This Project Can Be Run

In order to make your job use a slave created on the cloud, on demand, set the label to ondemand.

Run the Build Against MultiDB

In order to run your build against one or several DB you can use profile and slave parameters.

You need to use the customdb profile to init build environment, and then desired profile among : pgsql, mssql, oracle10g, oracle11g, oracle12g, mysql, mongodb...

You also need to use a slave owning the configuration for desired DB, for example MULTIDB_LINUX.

Execute Shell

Shell script

#!/bin/bash -xe
START=$(date +%s)
unset DOCKER_HOST

if [ ! -z $JDK_PATH ]; then
  export JAVA_HOME=$JDK_PATH
  export PATH=$JDK_PATH/bin:$PATH
fi

if [ "$CLEAN" = true ] || [ ! -e .git ]; then
  rm -rf * .??*
  git clone [email protected]:nuxeo/nuxeo.git .
fi
git checkout $BRANCH 2>/dev/null || git checkout $PARENT_BRANCH

. scripts/gitfunctions.sh
if [ "$(type -t shr)" != "function" ]; then
  echo ERROR: the current job is not compliant with this version of gitfunctions.sh
  exit 1
fi

if [ "$CLEAN" = false ]; then
  gitfa fetch --all
  gitfa checkout $PARENT_BRANCH
  gitfa pull --rebase
  ! gitfa -q diff --name-status --diff-filter=U | grep -B1 -e "^U"
fi

# switch on feature $BRANCH if exists, else falls back on $PARENT_BRANCH
./clone.py $BRANCH -f $PARENT_BRANCH
gitfa rebase origin/$PARENT_BRANCH
! gitfa -q diff --name-status --diff-filter=U | grep -B1 -e "^U"
if [ "$MERGE_BEFORE_BUILD" = true ]; then
  shr -a "git checkout $PARENT_BRANCH; git pull --rebase; git merge --no-ff $BRANCH --log"
  ! gitfa -q diff --name-status --diff-filter=U | grep -B1 -e "^U"
fi

rm -rf $WORKSPACE/.repository/org/nuxeo/
echo "Init $(($(date +%s) - $START))" > $WORKSPACE/.ci-metrics
echo "$(date +%s)" >$WORKSPACE/.ci-metrics-mavenstart

Invoke Top-Level Maven Targets

Maven call

Goals: $MVN_TARGETS -Paddons,distrib $PROFILES $MVN_ADDITIONAL_OPTS
JVM options: -Xms1024m -Xmx4096m

Use private Maven repository: Maven will store artifacts in $WORKSPACE/.repository instead of ~/.m2/repository.

Execute Shell

Shell script

#!/bin/bash  -xe
echo "Maven $(($(date +%s) - $(cat $WORKSPACE/.ci-metrics-mavenstart)))" >> $WORKSPACE/.ci-metrics
rm $WORKSPACE/.ci-metrics-mavenstart
START=$(date +%s)

. scripts/gitfunctions.sh
(! (shopt -s globstar && grep -hle ".*ERROR.*" nuxeo-distribution/**/log/server.log nuxeo-distribution/**/log/console.log nuxeo-distribution/**/log/nuxeo-error.log))
if [ "$MERGE_BEFORE_BUILD" != true ]; then
  shr -a "git checkout $PARENT_BRANCH; git pull --rebase; git merge --no-ff $BRANCH --log"
  ! gitfa -q diff --name-status --diff-filter=U | grep -B1 -e "^U"
fi

ROOT="$(basename $WORKSPACE)"

function pr_or_push() {
  echo "[$1]"
  local repository=$1
  if [ "$repository" = "." ] || [ "$repository" = "$ROOT" ]; then
    repository="nuxeo"
  fi
  if [ -z $(git rev-list -1 origin/$PARENT_BRANCH..) ] ; then
    echo "NOOP"
  elif [ "$DO_PR" = true ]; then
    JSON="{
      \"title\": \"$BRANCH\",
      \"head\": \"$BRANCH\",
      \"base\": \"$PARENT_BRANCH\",
      \"body\": \"PR created from $BUILD_URL\"
    }"
    curl -n -XPOST "https://api.github.com/repos/nuxeo/$repository/pulls" -d "$JSON"
  else
    git push origin $PARENT_BRANCH
  fi
  echo
}

if [ "$PUSH_IF_SUCCEED" = true ]; then
  if [ "$MERGE_BEFORE_BUILD" = true ]; then
    echo ****** Pull in case another commit had been pushed during the build
    gitfa pull --rebase
    ! gitfa -q diff --name-status --diff-filter=U | grep -B1 -e "^U"
  fi

  if $CREATE_PR; then
    echo ***** SUCCEED! CREATING PULL REQUEST *****
    DO_PR=true
  else
    echo ***** SUCCEED! PUSHING MERGE *****
    DO_PR=false
  fi

  shr -a 'pr_or_push $dir'
  cd ..
fi

echo "Finalize $(($(date +%s) - $START))" >> $WORKSPACE/.ci-metrics

Post Build Actions

  1. Post build task
    if [ -f .ci-metrics-mavenstart ]; then
    echo "Maven $(($(date +%s) - $(cat .ci-metrics-mavenstart)))" >> .ci-metrics
    rm .ci-metrics-mavenstart
    fi
    
  2. Scan for compiler warnings Use Maven and javac parsers on the console. Use javac parser on nuxeo-distribution/**/log/*.log files.
  3. Archive the artifacts **/target/failsafe-reports/*, **/target/*.png, **/target/screenshot*.html, **/target/*.json, **/target/results/result-*.html, **/*.log, nuxeo-distribution/**/log/*, **/nxserver/config/distribution.properties, .ci-metrics
  4. Publish JUnit results report **/target/failsafe-reports/*.xml, **/target/surefire-reports/*.xml, **/target/nxtools-reports/*.xml
  5. Set build description $BRANCH
  6. Configure the Jenkins text finder to browse nuxeo-distribution/**/log/server.log looking for .*ERROR.* and setting the build as unstable if found.
  7. Send to you an email using the Editable Email Notification.
  8. Send to you a Jabber notification.
  9. Trigger parameterized build on other projects
    Project: /System/ondemand-testandpush-metrics
    Parameters:
    JOB=$JOB_NAME
    BUILD=$BUILD_NUMBER
    

How to Edit the Template

Template plugin documentation: https://go.cloudbees.com/docs/cloudbees-documentation/cje-user-guide/index.html#template

Here is the recommended/easier method for such a change:

  1. instantiate a WIP job:

  2. "detach" the job:

    • $ ssh [email protected]
      cd .jenkins/jobs/TestAndPush/jobs/feature-NXBT-2318
      cp config.xml config.xml.bak
      vi config.xml
      
    • Using VIM to search and remove the XML section used for the templating:

      vi config.xml
      /cloudbees-template
      vatd
      :wq
      
      $ diff --side-by-side --suppress-common-lines config.xml.bak config.xml
          <com.cloudbees.hudson.plugins.modeling.impl.jobTemplate.J <
            <instance>                          <
              <model>TestAndPush/template_ondemand</model>          <
              <values class="tree-map">                  <
                <entry>                          <
                  <string>defaultSlave</string>              <
                  <string>ondemand</string>                  <
                </entry>                          <
                <entry>                          <
                  <string>jdk</string>                  <
                  <string>java-8-openjdk</string>              <
                </entry>                          <
                <entry>                          <
                  <string>maven</string>                  <
                  <string>maven-3</string>                  <
                </entry>                          <
                <entry>                          <
                  <string>name</string>                  <
                  <string>feature-NXBT-2318</string>              <
                </entry>                          <
                <entry>                          <
                  <string>notification</string>              <
                  <string>[email protected]</string>          <
                </entry>                          <
              </values>                          <
           </instance>                          <
         </com.cloudbees.hudson.plugins.modeling.impl.jobTemplate. <
      
    • https://qa.nuxeo.org/jenkins/job/TestAndPush/job/feature-NXBT-2318/reload
  3. configure the detached job with the UI as wanted to resolve the ticket:

  4. Use diff again to get the related XML changes:

     [email protected]:~/.jenkins/jobs/TestAndPush/jobs/feature-NXBT-2318$ diff --side-by-side --suppress-common-lines config.xml.bak config.xml
    
  5. Apply the XML changes on the template:

  6. Request an administrator for approval of the new script:

  7. Reattach the job, regenerate, check if the ticket is properly resolved then cleanup:

2 months ago manonlumeau NXDOC-1650 fix about integrating changes, add mention on multiple attempts
3 years ago Kevin Leturc 14
3 years ago Manon Lumeau 13
4 years ago Solen Guitter 12
4 years ago Julien Carsique 11
4 years ago Julien Carsique 10
5 years ago Thomas Roger 9
5 years ago Thomas Roger 8
6 years ago Julien Carsique 7
6 years ago Julien Carsique 6
6 years ago Julien Carsique 5
6 years ago Julien Carsique 4
6 years ago Julien Carsique 3
6 years ago Julien Carsique 2
6 years ago Julien Carsique 1
History: Created by Julien Carsique

We'd love to hear your thoughts!

All fields required