Handshake failure with Maven

I have experienced an annoying issue while building a projet.
I got a javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure message while downloading resources at https://publicsuffix.org/list//effective_tld_names.dat.

I followed instructions from this forum and installed JCE librairies in my JDK’s security directory (I am using Oracle JDK 8). Unfortunately, Maven kept on failing with this host. I finally upgraded to a more recent version of Maven (from 3.2.2 to 3.6.0) and it worked.

I hope this will help someone if the case occurs once again.

Shared Responsibilities in Jenkins Pipelines for Docker Images

This article explains how to implement the pipeline described in this article (and summed up in the diagram below) for Docker images with Jenkins (I used version 2.145 for my tests).

Pipeline stages are defined in separate Git repositories

We assume Jenkins runs on nodes. The pipeline can be adapted to run in containers. However, running tests on Docker images might be harder. Solutions like Docker in Docker, and most of all, Docker outside of Docker (share the host’s Docker daemon inside a container to create siblings) may work. However, running tests of Docker containers is easier when the Jenkins agent runs directly on a node. And Docker outside of Docker is not considered as a safe practice.

Let’s see what we need for Jenkins:

  • A Jenkins master, with the Pipeline plug-in.
  • Jenkins agents that run on (virtual) machines.
  • At least one agent node with a Docker daemon installed. It will be used to build images and run tests.
  • The same agent node should have access to a Clair instance (through Clair-Scanner) or to a Dagda install.

Here is the Jenkins topology with Clair.
In this diagram, the Clair server would be managed by the security team/role. Not every project should have its own instance of Clair, that would make no sense. It is something that should be common.

Jenkins topology with a specialized node agent (Docker + access to Clair)

The topology would be almost the same with Dadga.
To integrate Dadga in a CI process, you have two options. Dagda being a Python script, either you create a web service to invoke during the build process (that will execute the Python script), or you deploy its database on a separate VM, and you execute Dadga as a container on the Jenkins slave. This container should connect to the remote database. That would be the best option to have quick scans. For the notice, Dadga uses MongoDB instead of PostgreSQL.

It is also important to secure the Docker daemon on the node.
As I do not want to spend too much time on it, here are some links:

Pipeline

With Jenkins, the convention is to define pipelines in a file named Jenkinsfile.
To make ours simple, we provide an all-in-one package. Projects only have to provide parameters and their own tests for their Docker image. The following sample gives an idea about what such a project repository would look like…

image/
- Dockerfile
- ...
tests/
- execute-tests.sh
- ...
Jenkinsfile
readme.md

The project repository contains resources to build the image, resources to test it, and a Jenkinsfile, that describes the build process for Jenkins. Here is the content of this file.

allInOne(
    imageName: 'my-image-name',
    imageVersion: '1.0.0'
)

allInOne is not something Jenkins recognizes by default.
In fact, it refers to a Jenkins shared library we need to define. There will be 3 of them: one for the composite pipeline (« allInOne »), one for the security team, and one for the governance team. Each shared library is defined in its own Git repository, each one having its own permissions. This way, the security team can be sure only it can access/commit about security stuff. Same thing for the Software governance team. And they can all have read access to the shared library that agregates the various stages.

Shared library for the composite pipeline

This is the shared library to use for the « allInOne » tag.
Jenkins pipeline allows to define three things in a shared library: helpers, a pipeline or a step. What would have been better would have been defining stages. But this is not possible. Our allInOne library will thus declare a pipeline with parameters. Here is the main code:

// In vars/allInOne.groovy (shared library that defines the generic pipeline)
def call(Map config) { 
 
	node { 
		def timeStamp = Calendar.getInstance().getTime().format('YYYYMMdd-hhmmss', TimeZone.getTimeZone('Europe/Paris'))
		def buildId = "${config.imageVersion}-${timeStamp}"

		stage('Checkout') { 
			echo "Checking out the sources..." 
			checkout scm 
		}

		stage('Build Image') {
			// Enforce the shape of the repository and assume the Dockerfile is always under image/
			sh 'docker build -t "${config.imageName}:${buildId}" image/'
		}

		stage('Project tests') {
			def scriptFileContent = libraryResource( 'com/linagora/execute-project-tests.sh' )
			sh scriptFileContent
		}

		stage('Security checks') {
			echo "Checking security..."
			securityInspection( "${config.imageName}", "${buildId}" )
		}

		stage('Software Governance') {
			echo "Handling Software checks..."
			softwareCheck( "${config.imageName}", "${buildId}" )
		}

		stage('Promotion') {
			echo "Promoting the local image to a trusted repository..."
			def scriptFileContent = libraryResource( 'com/linagora/promote-image.sh' )
			sh scriptFileContent
		}
	}
}

All the stages are sequential. If one fails, everything stops. The securityInspection and softwareCheck steps are defined farther as global shared libraries. The promote-image.sh and execute-project-tests.sh scripts are provided as resources of the shared library. You can find the whole project online, on Github.

Notice the build stage is very basic.
In this example, we do not handle build parameters. This part can easily be adapted.

Project tests

How can a project specify tests for its Docker image?
Well, this is not complicated. I already faced the issue with a previous project. The idea is the following:

  1. Have a script that launches tests.
  2. This script instantiates the image in detached mode, with a shared volume.
  3. We then execute another script inside the container, by using docker exec.
  4. This second script is in charge of verifying assertions inside the container. Example: verify some process is running, verify a given file was created, etc. When an assertion fails, a message is written inside a file, located in the shared volume. When the container terminates, this file will remain available on the host system.
  5. Once the script has completed, the first script can verify the content of the error file. If there are errors, it can just fail the build.
  6. It is also possible to verify some assertions about the container from the outside (e.g. try to ping or reach a given port).

Here is another repository that illustrates this strategy.
It was used to test RPM and Debian packages in Docker containers. The script that launches the tests and verifies the result is here. Scripts that verify assertions in the containers are located in this directory.

This kind of tests works for functional tests, but it can also be applied for security checks (e.g. permissions on files) or governance ones. The difference here is that these tests lie in the same Git repository than the Dockerfile.

To run them, the allInOne pipeline must verify some script test exists and start it. A naming convention is enough.

Security checks

Security checks are maintained by the security team/role as a shared library, hosted in its own Git repository. Security checks can be verified with a script (e.g. permissions, etc). But for Docker images, there also exist solutions that inspect images and detect leaks.

Clair, from CoreOS, is one of them.
This project uses a database of known vulnerabilities, and scans images. It then provides a dashboard, indicating which CVE were found and for which images.

Dadga is another solution.
Although it is more recent, it provides additional features than Clair. It works the same way, but also uses ClamAV to search for virus, trojans and so on. What takes time for both Dadga and Clair is to update the database of known vulnerabilities. This is why the database cannot be deployed on the fly during the build process. It must preexist to pipeline executions.

Both solutions can also be used to regularly control Docker images. But since the focus of these articles is about delivery pipelines, you will have to dig this part yourself. What is interesting is to integrate these tools in the delivery pipeline, as a validation step. To keep the example simple, I only include Clair. Clair is great, but using it as a command-line is not easy. The best option is to use Clair Scanner as a complement.

And here is our shared library

// In vars/securityInspection.groovy (shared library for the security role)
def call(String imageName, String buildId) { 

	// We assume clair-scanner is available in the path
	def host = sh(returnStdout: true, script: 'hostname -i').trim()
	clair-scanner -c <CLAIR_SERVER_URL> --ip ${host} --t High ${imageName}:${buildId}
}

Here, Clair will scan the given image. If vulnerabilities are found with a severity higher or equal to high, then clair-scanner will return a non-zero exit code and thus fail the build.

If you use Dadgda instead of Clair, you simply run a Python script. The installation is a little bit different, but the pipeline step would remain simple. You can also add custom scripts to perform additional verifications (just add new steps in the pipeline).

Software Governance

Software governance can be managed in the same way that previous stages.
Since it depends on the organization itself, I have no generic tool to suggest. Instead, I assume there is some REST end-point somewhere that can be contacted during the build. The goal is to extract information and send them to a remote web service that will store them, and optionally trigger an alert or a build failure in case of exotic findings.

So, here is an example of associated shared library

// In vars/softwareCheck.groovy (shared library for the Software Governance role)
def call(String imageName, String buildId) { 

	def scriptFileContent = libraryResource( 'com/linagora/analyze-dockerfile.sh' )
	sh scriptFileContent
	sh 'echo "imageName: ${imageName}" >> /tmp/gov.results.txt'
	sh 'echo "imageVersion: ${buildId}" >> /tmp/gov.results.txt'
	sh 'curl --data-binary "@/tmp/gov.results.txt" -X POST...'
	sh 'rm -rf /tmp/gov.results.txt'
}

Pipeline for existing Docker images

The initial pipeline includes a stage to build an image.
However, a project team may want to use an existing image. Community images benefit from various feedbacks and contributors. How to guarantee a project can safely use it inside an organization with its own guidelines?

Well, our generic pipeline, with project tests, security checks and governance, perfectly fits such a use case. The guidelines should be enforced in the automated pipeline. The only difference is that we do not build an image, we use an existing one from the outside. So, let’s adapt our allInOne shared library to cover such a scenario.

// In vars/allInOne.groovy (shared library that defines the generic pipeline, upgraded to support existing images)
def call(Map config) { 

	node { 
		def timeStamp = Calendar.getInstance().getTime().format('YYYYMMdd-hhmmss', TimeZone.getTimeZone('Europe/Paris'))
		def buildId = "${config.imageVersion}-${timeStamp}"

		// Alway checkout the sources, as they may include tests
		stage('Checkout') { 
			echo "Checking out the sources..." 
			checkout scm 
		}

		if (config.existing == true) {
			stage('Docker pull') {
				def buildId = "${config.imageVersion}"
				sh 'docker pull "${config.imageName}:${buildId}"'
			}
		}

		if (config.existing != true) {
			stage('Build Image') {
				// Enforce the shape of the repository and assume it is always under image/
				sh 'docker build -t "${config.imageName}:${buildId}" image/'
			}
		}

		stage('Project tests') {
			def scriptFileContent = libraryResource( 'com/linagora/execute-project-tests.sh' )
			sh scriptFileContent
		}

		stage('Security checks') {
			echo "Checking security..."
			securityInspection( "${config.imageName}", "${buildId}" )
		}

		stage('Software Governance') {
			echo "Handling Software checks..."
			softwareCheck( "${config.imageName}", "${buildId}" )
		}

		stage('Promotion') {
			echo "Promoting the local image to a trusted repository..."
			def scriptFileContent = libraryResource( 'com/linagora/promote-image.sh' )
			sh scriptFileContent
		}
	} 
}

If we use an existing image we simply pull it. Otherwise, we build it. The other parts of the pipeline are the same.
A project using an existing image would then declare its Jenkinsfile as…

allInOne(
    imageName: 'my-image-name',
    imageVersion: '1.0.0',
    existing: true
)

Integration in Jenkins

The most simple solution is to define our 3 shared libraries as global shared libraries. Besides, the shared libraries above all expose global variables, which avoids using import declarations in our Jenkinsfiles. To do so, go into Jenkins administration, system configuration and find the pipeline section. The shared library will be loaded on the fly from a git repository, in every job. It is never cached. For security reasons, the (Jenkins) user that pulls the repository should only have read access.

Here is a screenshot that shows how a shared library is defined in Jenkins.

Jenkins Administration

You can add as many ones as necessary. This article has defined 3 shared libraries, so you would add 3 into Jenkins. It is possible to set versions to shared libraries, but I think it is not necessary for global shared libraries. If stages had to differ between projects, you would define different composite pipelines. And the stages are managed on the fly, in the same fashion than 12 factors.

Perimeter

Two questions can arise once we are here.

  • Where does this pipeline lead?
  • How to prevent a project from by-passing these checks?

The answer to the first question influences the second one.
In my opinion, this pipeline promotes the built image into a trusted Docker registry, that can then be used in Kubernetes environments. You cannot test this image in a K8s cluster before (or if you can, then it must be a sandbox cluster). Once this is clear, the second answer becomes obvious. A project cannot by-pass this pipeline because otherwise, it cannot import its Docker images in the trusted registry. The allInOne shared-library is the only part that can access the trusted registry. It cannot be done from anywhere else: the credentials are kept secret and a single user (Jenkins) should have the write permissions to push an image in the trusted registry. All the other users have read-only access.

Summary

This article has shown how to use Jenkins shared libraries to build a validation process for Docker images, process that is under different responsibilities. I skipped some details to not lost readers, but I believe the main aspects are all explained here.

The next article is about Helm packages: how to verify quality criteria (e.g. linting), test them and so on. All of this with a Jenkins pipeline and the same distribution of responsibilities.

Shared Responsibilities in Pipelines for Docker and Kubernetes

DevOps, and in fact DevSecOps, tend to be the norm today.
Or let’s say it tend to be an ideal today. Many organizations are far from it, even if more and more are trying to adopt this way of work.

This kind of approach better fits small structures. With few means, it is not unusual for people to do everything themselves. Big structures, in particular among the GAFAM, have split their teams in small units (pizza teams are one example) that behave more or less like small structures. The issue with big organizations, is that they face additional challenges, mainly keeping a certain coherence in the information system, containing entropy and preventing anarchy. This is about having a common security policy, using identified languages, libraries and frameworks, and knowing what is used and where. In one word: governance.

This may seem like obsolete debates from a technical point of view. But from an organizational point of view, this is essential. Big structures often have support contracts. They need to know what runs or not. Especially when there can be controls from their contractors (think about Oracle or IBM as an example). They also need to know what kinds of technologies run. If a single person in the company masters this or that framework, how will the project be maintained? Knowing what runs allows to identify the required skills. Structures with many projects also need to know how projects interact with each others. This can impact the understanding when a crisis arises (what parts of the system depend on what). Last but not least, security is a crucial aspect, that goes beyond the sole projects. Having common guidelines (and checks) is imperative, in particular when things go faster and faster, which became possible by automating delivery processes.

One of the key to be agile, goes through continuous integration, and now continuous deployments. The more things are automated, the faster things can get delivered. The governance I described above should find a place in these delivery pipelines. In fact, such a pipeline can be cut into responsibility domains. Each domain is associated with a goal, a responsibility and a role that is in charge. In small organizations, all the roles may be hold by the same team. In big ones, they could be hold by different teams or councils. The key is that they need to work together.

This does not minor the benefits of DevSecOps approaches.
The more polyvalent a team is, the faster it can deliver its project. And if you only have excellent developers, you can have several DevOps teams and expect them to cope on their own. But not every organization is Google or Facebook. Some projects might need help, and keeping a set of common procedures is healthy. You will keep on having councils or pools of experts, even if your teams are DevSecOps. The only requirement is that the project teams are aware of these procedures (regular communication, training sessions) and all of these verifications should be part of automated delivery pipelines.

Responsibility Domains

I have listed 3 responsibility domains for automated pipelines:

  • Project is the first one, where the project team can integrate its own checks and tests. It generally includes functional tests. This is the usual part in continuous integration.
  • Security is the obvious one. It may imply verifications that developers may not think about.
  • Software Governance is the last domain. It may include used programs (do we have a support contract?), version checks, notify some cartography API, etc.

Different stages in the build pipeline cover various concerns

The goal is that each domain is a common, reusable set of steps, defined in its own (Git) repository. Only the security team/role could modify the security stage. Only the Software governance team/role could modify its stage. And only the project team should be able to specify its tests. Nobody should be able to block another domain from upgrading its part. If the security team needs to add a new verification, it can commit it anytime. The global build process should include this addition as soon as possible. The composition remains, but the components can evolve independently.

Every stage of the pipeline is controlled from a different Git repository

This article is the first from a series about implementing such a pipeline for Kubernetes. Since the projects I am involved in all consist in providing solutions as Kubernetes packages, I focus on Docker images and Helm packages. The Docker part asks no question. Helm provides a convenient way to deliver ready-to-use packages for Kubernetes. Assuming a team has to provide a solution for clients, this is in my opinion the best option if one wants to support it efficiently.

Assumptions

To make things simple, I consider we have one deliverable per Git repository. That is to say:

  • Project sources, whose build result is stored somewhere (Maven repo, NPM…).
  • A git repo per Docker image used in the final application.
  • A git repo for the Helm package that deploy everything (such a package can depend on other packages, and reference several Docker images).

A project that develops its own sources would thus have at least 3 Git repositories.
We could mix some of them, and complexify our pipeline. Again, I avoided it to keep things easy to understand.

There are also two approaches in CI/CD tools: build on nodes, and build in containers. There are also many solutions, including Jenkins, GitLab CI, Travis CI, etc. Given my context, I started with Jenkins with build performed on nodes. I might add other options later. The global idea is very similar for all, only the implementation varies.

Testing Eclipse’s User Workflows: from OOMPH to Subversive, m2e and WTP

Goals

Few months ago, I worked on automating the tests of user workflows that involve Eclipse tooling. The client organization has more than a hundred of developers and they all use common frameworks based on JEE. They all use the same tools, from source version control to m2e and WTP. Eclipse being their IDE since quite a long time, they decided, some years ago, to automate the installation of Eclipse with preconfigured tools and predefined preferences. They did create their own solution. When OOMPH was released and became Eclipse’s official installer, they quickly dropped their project and adopted OOMPH.

From an OOMPH’s point of view, this organization has its own catalog and custom setup tasks. Unlike what the installer usually shows, there is only one distribution. Everything works behind a proxy. Non-composite p2 repositories are proxyfied by Nexus. All the composite p2 repositories (such as official Eclips’s ones) are mirrored by using Eclipse in command-line. The installer shows a single product, but in different versions (e.g. Neon, Oxygen…). It also provides several projects: several JDKs, several versions of Tomcat, several versions of Maven, several Eclipse tools, etc. We can really say this organization uses all the features OOMPH provides.

Here is a global overview of what is shown to users.

First screen in Eclipse's installer
Second screen in Eclipse's installer

So, this organization is mostly a group of Eclipse users. Their developments are quite limited. Their focus is about delivering valid Eclipse distributions to their members and verify everything work correctly in their environment. Given this context, my job was to automate things: update sites creation (easy with Tycho), prepare the installer for the internal environement and automate tests that launch the installer, make a real installation, start the newly installed Eclipse, make it execute several actions a real developer would do and verify everything works correctly inside this (restrained / controlled) environment.

Let’s take a look at the various parts.

Automating the creation of Custom Installers

This part is not very complicated.
I created a project on GitHub that shows how it works. Basically, we have a Maven module that invokes ANT. The ANT script downloads the official installer binaries from Eclipse.org. It verifies the checksum, unwrap their content, update the eclipse-inst.ini file, adds predefined preferences (related to the proxy) and rebuilds a package for users. To prevent downloading binaries everytime, we use a local cache (as a directory). If a binary already exists, we verify its checksum against the value provided by Eclipse.org. If it succeeds, it means our cache is valid against Eclipse repositories. Otherwise, it may indicate the cache is invalid and that a newer version was released. In such a situation, we indicate the user he (or she) should retry and/or delete the cache before giving it another try.

Since all of this a Maven project, it is possible to deploy these installers on a Maven repository.

Automating OOMPH tests with SWT Bot

OOMPH is a SWT application.
So, testing it automatically immediately made sense thanks to SWT Bot. Testing with SWT Bot implies deploying it in the tested application. Fortunately, OOMPH is also a RCP application. It means we can install things with p2. That was the first thing to do. And since I enjoy the Maven + ANT combo, I wrote an ANT script for this (inspired from the one available on Eclipse’s wiki – but much more simple). I also made the tasks reusable so that it can also deploy the bundle with the tests to run.

The next step was writing a SWT Bot test and run it against the installer.
The first test was very basic. The real job was launching it. When one wants to run SWT Bot tests, it launches a custom application that itself launches Eclipse. Unfortunately, the usual org.eclipse.swtbot.eclipse.junit.headless.swtbottestapplication application did not work. There are checks in it about the workbench. And even if OOMPH is a RCP and has SWT widgets, it does not have any workbench. This is why I created a custom application I embedded with my SWT Bot test. Once there, everything was ready.

1 – I have a bundle with SWT Bots tests. With a feature. With an update site (that can remain local, no need to deploy it anywhere).
2 – I have an ANT script that can install SWT Bot and my test bundle in OOMPH.
3 – I have an ANT script that can launch my custom SWT Bot application and executes my tests in OOMPH.

It works. The skeleton for the project is available on Github.
Otherwise, the shape and the Maven and ANT settings are the same. I only simplified the tests executed for OOMPH (they would not be meaningful for this article). The main test we wrote deploys Eclipse, but also downloads and unzip specific versions of Maven and Tomcat. Obviously, the catalog is made in such a way that installing these components also updates the preferences so that m2e and WTP can use them.

Notice there are settings in the ANT script that delete user directories (OOMPH puts some resources and information in cache). To make tests reliable, it is better to delete them. This can be annoying if you have other Eclipse installations on your machine. In the end, such tests aim at being executed on a separate infrastructure, e.g. in continuous integration.

Configuring Eclipse for SWT Bot

Once the tests for the installer have run, we have a new Eclipse installation.
And we have other tests to run in it. Just like what we did for OOMPH, we have to install SWT Bot in it. The p2 director will help us once again.

Notice we make this step separated from the execution of the tests themselves.
Testing OOMPH is quite easy. But the tests written for the IDE are much more complicated and we need to be able to re-run them. So, the configuration of the new Eclipse installation is apart from the tests execution.

Writing and Running Tests for Eclipse

In the same manner than for OOMPH, we have a custom plug-in that contains our tests for Eclipse. There is also a feature. and the (same) local update site. This plug-in is deployed along with SWT Bot. Launching the test is almost the same thing than for OOMPH, except there is a workbench here. We can rely on the usual SWT Bot application for Eclipse.

What is more unusual is the kind of test we run here.
I will give you an example. We have a test that…

1. … waits for OOMPH to intialize the workspace (based on the projects selected during the setup – this step is common to all our tests).
2. … opens the SVN perspective
3. … declares a new repository
4. … checks out the last revision
5. … lets m2eclipse import the projects (it is a multi-module project and m2e uses a custom settings.xml)
6. … applies a Maven profile on it
7. … waits for m2eclipse to download (many) artifacts from the organization’s Nexus
8. … waits for the compilation to complete
9. … verifies there is no error on the project
10. … deploys it on the Tomcat server that was installed by OOMPH (through WTP – Run as > Run on Server )
11. … waits for it to be deployed
12. … connects to the new web application (using a POST request)
13. … verifies the content of the page is valid.

This test takes about 5 minutes to run. It implies Eclipse tools, pre-packaged ones too, but also environment solutions (Nexus, SVN server, etc). Unlike what SWT Bot tests usually do, we make integration tests with an environment that is hardly reproductable. It is not just more complex, it must also acknowledge some situations like timeouts or slowlyness. And as usual, there may be glitches in the user interface. As an example, projects resources that are managed by SVN have revision numbers and commit’s author names as a suffix. So, you cannot search resources by full label (hence the TestUtils.findPartialLabel methods). Another example is that when one expands nodes in the SVN hierarchy, it may take some time for the child resources to by retrieved. Etc.

But what was the most complicated was developing these tests.

Iterative Development of these Tests

Usually, SWT Bot tests are developed and tested from the developer’s workspace: right click on the test class, Run as > SWT Bot test. It opens a new workbench and the test runs. That was not possible here. The Eclipse into which the tests must run is supposed to have been configured by OOMPH. You cannot compile the Maven project if you do not have the right settings.xml. You cannot deploy on Tomcat if it has not been declared in the server preferences. And you cannot set these preferences in the test itself because it is part of its job to verify OOMPH did it correctly! Again, it is not unit testing but integration testing. You cannot break the chain.

This is why each test is defined in its own Maven profile.
To run scenario 1, we execute…

mvn clean verify -P scenario1

We also added a profile that recompiles the SWT Bot tests and upgrade the plug-in in the Eclipse installation (the p2 directory can install and uninstall units at once). Therefore, if I modified a test, I can recompile, redeploy and run it by typing in…

mvn clean verify -P recompile-ide-tests -P scenario1

This is far from being perfect, but it made the developement much less painful than going through the full chain on every change.
I wished I could have duplicated preferences from the current workspace when I run tests from Eclipse (even if it is clear other problems would have arisen). We had 4 important scenarios, and each one is managed separately, in the code and in the Maven configuration.

Conclusion

Let’s start with the personal feedback.
I must confess this project was challenging, despite a solid experience with Maven, Tycho and SWT Bot. The OOMPH part was not that hard (I only had to dig in SWT Bot and platform’s code). Testing the IDE itself, with all the involved components and the environment, was more complicated.

Now, the real question is: what is worth the effort?
The answer is globally yes. The good parts are these tests can be run in a continuous integration workflow. That was the idea at the beginning. Even if it is not done (yet), that could (should) be a next step. I have spent quite some time to make these tests robust. I must have run them about a thousand times, if not more. And still sometimes, one can fail due to an environment glitch. This is also why we adopted the profile-per-scenario approach, to ease the construction of a build matrix and be able to validate scenarios separately and/or in parallel. It is also obvious that these tests run faster than by hand. An experienced tester spends about two hours to verify these scenarios manually. A novice will spend a day. Running the automated tests takes at most 30 minutes, provided you can read the docs and execute 5 succeeding Maven commands. And these tests can be declined over several user environments. So, this is globally positive.

Now, there are few drawbacks. We did not go to the continuous integration. For the moment, releases will keep on being managed on-demand / on-schedule (so few times a year). In addition, everything that was done was for Linux systems. There would be minor adaptations to test the process on Windows (mainly, do not launch the same installer). We also found out minor differences between Eclipse versions. SWT Bot intensively uses labels. However, there are labels and buttons that have changed, as an example, between Neon and Oxygen. So, our tests do not work on every Eclipse version. The problem would remain if we tested by hand. Eventually, and unlike what it seems when you read them, the written tests remain complex to maintain. So, short and mid-term benefits might be counter-balanced by a new degree of complexity (easy to use, not so easy to upgrade). Tests by hand take time but remain understandable and manageable by many persons. Writting or updating SWT Bot tests require people to be well-trained and patient (did I mention I run IDE tests at least a thousand times?). Besides, having automated tests does not prevent from tracking tests on TestLink. So, manual tests remain documented and maintained. In fact, not all the tests have been automated, only the main and most painful ones.

Anyway, as usual, progress is made up of several steps. This work was one of them. I hope those facing the same issues will find help in this article and in the associated code samples.