Using Graylog for Centralized Logs in K8s platforms and Permissions Management

This article explains how to centralize logs from a Kubernetes cluster and manage permissions and partitionning of project logs thanks to Graylog (instead of ELK). The fact is that Graylog allows to build a multi-tenant platform to manage logs. Let’s take a look at this.

Reminders about logging in Kubernetes

As it is stated in Kubernetes documentation, there are 3 options to centralize logs in Kubernetes environements.

The first one is about letting applications directly output their traces in other systems (e.g. databases). This approach always works, even outside Docker. However, it requires more work than other solutions. Not all the applications have the right log appenders. It can also become complex with heteregenous Software (consider something less trivial than N-tier applications). Eventually, log appenders must be implemented carefully: they should indeed handle network failures without impacting or blocking the application that use them, while using as less resources as possible. So, althouth it is a possible option, it is not the first choice in general.

Applications output their logs directly in a central store

The second solution is specific to Kubernetes: it consists in having a side-car container that embeds a logging agent. This agent consumes the logs of the application it completes and sends them to a store (e.g. a database or a queue). This approach is better because any application can output logs to a file (that can be consumed by the agent) and also because the application and the agent have their own resources (they run in the same POD, but in different containers). Side-car containers also gives the possibility to any project to collect logs without depending on the K8s infrastructure and its configuration. However, if all the projets of an organization use this approach, then half of the running containers will be collecting agents. Even though log agents can use few resources (depending on the retained solution), this is a waste of resources. Besides, it represents additional work for the project (more YAML manifests, more Docker images, more stuff to upgrade, a potential log store to administrate…). A global log collector would be better.

Collector in side-car containers

That’s the third option: centralized logging. Rather than having the projects dealing with the collect of logs, the infrastructure could set it up directly. The idea is that each K8s minion would have a single log agent and would collect the logs of all the containers that run on the node. This is possible because all the logs of the containers (no matter if they were started by Kubernetes or by using the Docker command) are put into the same file. What kubectl log does, is reading the Docker logs, filtering the entries by POD / container, and displaying them. This approach is the best one in terms of performances. What is difficult is managing permissions: how to guarantee a given team will only access its own logs. Not all the organizations need it. Small ones, in particular, have few projects and can restrict access to the logging platform, rather than doing it IN the platform. Anyway, beyond performances, centralized logging makes this feature available to all the projects directly. They do not have to deal with logs exploitation and can focus on the applicative part.

Centralized logging

Centralized Logging in K8s

Centralized logging in K8s consists in having a daemon set for a logging agent, that dispatches Docker logs in one or several stores. The most famous solution is ELK (Elastic Search, Logstash and Kibana). Logstash is considered to be greedy in resources, and many alternative exist (FileBeat, Fluentd, Fluent Bit…). The daemon agent collects the logs and sends them to Elastic Search. Dashboards are managed in Kibana.

Things become less convenient when it comes to partition data and dashboards. Elastic Search has the notion of index, and indexes can be associated with permissions. So, there is no trouble here. But Kibana, in its current version, does not support anything equivalent. All the dashboards can be accessed by anyone. Even though you manage to define permissions in Elastic Search, a user would see all the dashboards in Kibana, even though many could be empty (due to invalid permissions on the ES indexes). Some suggest to use NGinx as a front-end for Kibana to manage authentication and permissions. It seems to be what Red Hat did in Openshift (as it offers user permissions with ELK). What I present here is an alternative to ELK, that both scales and manage user permissions, and fully open source. This relies on Graylog.

Here is what Graylog web sites says: « Graylog is a leading centralized log management solution built to open standards for capturing, storing, and enabling real-time analysis of terabytes of machine data. We deliver a better user experience by making analysis ridiculously fast, efficient, cost-effective, and flexible. »

I heard about this solution while working on another topic with a client who attended a conference few weeks ago. And indeed, Graylog is the solution used by OVH’s commercial solution of « Log as a Service » (in its data platform products). This article explains how to configure it. It is assumed you already have a Kubernetes installation (otherwise, you can use Minikube). To make things convenient, I document how to run things locally.

Architecture

Graylog is a Java server that uses Elastic Search to store log entries.
It also relies on MongoDB, to store metadata (Graylog users, permissions, dashboards, etc).

HA architecture for Graylog

What is important is that only Graylog interacts with the logging agents. There is no Kibana to install. Graylog manages the storage in Elastic Search, the dashboards and user permissions. Elastic Search should not be accessed directly. Graylog provides a web console and a REST API. So, everything feasible in the console can be done with a REST client.

Deploying Graylog, MongoDB and Elastic Search

Obviously, a production-grade deployment would require a highly-available cluster, for both ES, MongoDB and Graylog. But for this article, a local installation is enough. A docker-compose file was written to start everything. As ES requires specific configuration of the host, here is the sequence to start it:

sudo sysctl -w vm.max_map_count=262144
docker-compose -f compose-graylog.yml up

You can then log into Graylog’s web console at http://localhost:9000 with admin/admin as credentials. Those who want to create a highly available installation can take a look on Graylog’s web site.

Deploying the Collecting Agent in K8s

As discussed before, there are many options to collect logs.
I chose Fluent Bit, which was developed by the same team than Fluentd, but it is more performant and has a very low footprint. There are also less plug-ins than Fluentd, but those available are enough.

What we need to is get Docker logs, find for each entry to which POD the container is associated, enrich the log entry with K8s metadata and forward it to our store. Indeed, Docker logs are not aware of Kubernetes metadata. We therefore use a Fluent Bit plug-in to get K8s meta-data. I saved on Github all the configuration to create the logging agent. It gets logs entries, adds Kubernetes metadata and then filters or transforms entries before sending them to our store.

The message format we use is GELF (which a normalized JSON message supported by many log platforms). Notice there is a GELF plug-in for Fluent Bit. However, I encountered issues with it. As it is not documented (but available in the code), I guess it is not considered as mature yet. Instead, I used the HTTP output plug-in and built a GELF message by hand. Here is what it looks like before it is sent to Graylog.

{
"short_message":"2019/01/13 17:27:34 Metric client health check failed...",
"_stream":"stdout",
"_timestamp":"2019-01-13T17:27:34.567260271Z",
"_k8s_pod_name":"kubernetes-dashboard-6f4cfc5d87-xrz5k",
"_k8s_namespace_name":"test1",
"_k8s_pod_id":"af8d3a86-fe23-11e8-b7f0-080027482556",
"_k8s_labels":{},
"host":"minikube",
"_k8s_container_name":"kubernetes-dashboard",
"_docker_id":"6964c18a267280f0bbd452b531f7b17fcb214f1de14e88cd9befdc6cb192784f",
"version":"1.1"
}

Eventually, we need a service account to access the K8s API.
Indeed, to resolve to which POD a container is associated, the fluent-bit-k8s-metadata plug-in needs to query the K8s API. So, it requires an access for this.

You can find the files in this Git repository. The service account and daemon set are quite usual. What really matters is the configmap file. It contains all the configuration for Fluent Bit: we read Docker logs (inputs), add K8s metadata, build a GELF message (filters) and sends it to Graylog (output). Take a look at the Fluent Bit documentation for additionnal information.

Configuring Graylog

There many notions and features in Graylog.
Only few of them are necessary to manage user permissions from a K8s cluster. First, we consider every project lives in its own K8s namespace. If there are several versions of the project in the same cluster (e.g. dev, pre-prod, prod) or if they live in different clusters does not matter. What is important is to identify a routing property in the GELF message. So, when Fluent Bit sends a GELF message, we know we have a property (or a set of properties) that indicate(s) to which project (and which environment) it is associated with. In the configmap stored on Github, we consider it is the _k8s_namespace property.

Now, we can focus on Graylog concepts.
We need…

An input

An input is a listener to receive GELF messages.
You can create one by using the System > Inputs menu. In this example, we create a global one for GELF HTTP (port 12201). There are many options in the creation dialog, including the use of SSL certificates to secure the connection.

Screenshot of the inputs management in Graylog

Indices

Graylog indices are abstractions of Elastic indexes. They designate where log entries will be stored. You can associate sharding properties (logical partition of the data), retention delay, replica number (how many instances for every shard) and other stuff to a given index. Every projet should have its own index: this allows to separate logs from different projects. Use the System > Indices to manage them.

Indices in Graylog's web console

A project in production will have its own index, with a bigger retention delay and several replicas, while a developement one will have shorter retention and a single replica (it is not a big issue if these logs are lost).

Streams

A stream is a routing rule. They can be defined in the Streams menu. When a (GELF) message is received by the input, it tries to match it against a stream. If a match is found, the message is redirected into a given index.

Creating a stream in Graylog

When you create a stream for a project, make sure to check the Remove matches from ‘All messages’ stream option. This way, the log entry will only be present in a single stream. Otherwise, it will be present in both the specific stream and the default (global) one.

The stream needs a single rule, with an exact match on the K8s namespace (in our example).
Again, this information is contained in the GELF message. Notice that the field is _k8s_namespace in the GELF message, but Graylog only displays k8s_namespace in the proposals. The initial underscore is in fact present, even if not displayed.

Creating a rule for a stream

Do not forget to start the stream once it is complete.

Graylog streams

Dashboards

Graylog’s web console allows to build and display dashboards.
Make sure to restrict a dashboard to a given stream (and thus index). Like for the stream, there should be a dashboard per namespace. Using the K8s namespace as a prefix is a good option.

Dashboards are defined directly in Graylog

Graylog provides several widgets…
Take a look at the documentation for further details.

Sample dashboard in Graylog

Roles

Graylog allows to define roles.
A role is a simple name, coupled to permissions (roles are a group of permissions). You can thus allow a given role to access (read) or modify (write) streams and dashboards. For a project, we need read permissions on the stream, and write permissions on the dashboard. This way, users with this role will be able to view dashboards with their data, and potentially modifying them if they want.

Roles and users can be managed in the System > Authentication menu.

Managing access to streams

Managing access to dashboards

The list of roles

Users

Apart the global administrators, all the users should be attached to roles.
These roles will define which projects they can access. You can consider them as groups. When a user logs in, Graylog’s web console displays the right things, based on their permissions.

Creating a user in Graylog

There are two predefined roles: admin and viewer.
Any user must have one of these two roles. He (or she) may have other ones as well. When a user logs in, and that he is not an administrator, then he only has access to what his roles covers.

The user only sees the stream for his stream

Clicking the stream allows to search for log entries.

Log entries associated with the stream

Notice that there are many authentication mechanisms available in Graylog, including LDAP.

Summary

Graylog uses MongoDB to store metadata (stream, dashboards, roles, etc) and Elastic Search to store log entries. We define an input in Graylog to receive GELF messages on a HTTP(S) end-point. These messages are sent by Fluent Bit in the cluster.

When such a message is received, the k8s_namespace_name property is verified against all the streams.
When one matches this namespace, the message is redirected in a specific Graylog index (which is an abstraction of ES indexes). Only the corresponding streams and dashboards will be able to show this entry.

How permissions and isolation are managed in Graylog

Eventually, only the users with the right role will be able to read data from a given stream, and access and manage dashboards associated with it. Logs are not mixed amongst projects. Isolation is guaranteed and permissions are managed trough Graylog.

In short : 1 project in an environment = 1 K8s namespace = 1 Graylog index = 1 Graylog stream = 1 Graylog role = 1 Graylog dashboard. This makes things pretty simple. You can obviously make more complex, if you want…

Testing Graylog

You can send sample requests to Graylog’s API.

# Found on Graylog's web site
curl -X POST -H 'Content-Type: application/json' -d '{ "version": "1.1", "host": "example.org", "short_message": "A short message", "level": 5, "_some_info": "foo" }' 'http://192.168.1.18:12201/gelf'

This one is a little more complex.

# Home made
curl -X POST -H 'Content-Type: application/json' -d '{"short_message":"2019/01/13 17:27:34 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.","_stream":"stdout","_timestamp":"2019-01-13T17:27:34.567260271Z","_k8s_pod_name":"kubernetes-dashboard-6f4cfc5d87-xrz5k","_k8s_namespace_name":"test1","_k8s_pod_id":"af8d3a86-fe23-11e8-b7f0-080027482556","_k8s_labels":{},"host":"minikube","_k8s_container_name":"kubernetes-dashboard","_docker_id":"6964c18a267280f0bbd452b531f7b17fcb214f1de14e88cd9befdc6cb192784f","version":"1.1"}' http://localhost:12201/gelf

Feel free to invent other ones…

Automating stuff

Every features of Graylog’s web console is available in the REST API.
It means everything could be automated. Every time a namespace is created in K8s, all the Graylog stuff could be created directly. Project users could directly access their logs and edit their dashboards.

Check Graylog’s web site for more details about he API. When you run Graylog, you can also access the Swagger definition of the API at http://localhost:9000/api/api-browser/

Going further

The resources in this article use Graylog 2.5.
The next major version (3.x) brings new features and improvements, in particular for dashboards. There should be a new feature that allows to create dashboards associated with several streams at the same time (which is not possible in version 2.5, a dashboard being associated with a single stream – and so a single index). That would allow to have transverse teams, with dashboards that span across several projects.

See https://www.graylog.org/products/latestversion for more details.

Hints for Tests

If you do local tests with the provided compose, you can purge the logs by stopping the compose stack and deleting the ES container (docker rm graylogdec2018_elasticsearch_1). Then restart the stack.

If you remove the MongoDB container, make sure to reindex the ES indexes.
Or delete the Elastic container too.

Handshake failure with Maven

I have experienced an annoying issue while building a projet.
I got a javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure message while downloading resources at https://publicsuffix.org/list//effective_tld_names.dat.

I followed instructions from this forum and installed JCE librairies in my JDK’s security directory (I am using Oracle JDK 8). Unfortunately, Maven kept on failing with this host. I finally upgraded to a more recent version of Maven (from 3.2.2 to 3.6.0) and it worked.

I hope this will help someone if the case occurs once again.

Shared Responsibilities in Jenkins Pipelines for Docker Images

This article explains how to implement the pipeline described in this article (and summed up in the diagram below) for Docker images with Jenkins (I used version 2.145 for my tests).

Pipeline stages are defined in separate Git repositories

We assume Jenkins runs on nodes. The pipeline can be adapted to run in containers. However, running tests on Docker images might be harder. Solutions like Docker in Docker, and most of all, Docker outside of Docker (share the host’s Docker daemon inside a container to create siblings) may work. However, running tests of Docker containers is easier when the Jenkins agent runs directly on a node. And Docker outside of Docker is not considered as a safe practice.

Let’s see what we need for Jenkins:

  • A Jenkins master, with the Pipeline plug-in.
  • Jenkins agents that run on (virtual) machines.
  • At least one agent node with a Docker daemon installed. It will be used to build images and run tests.
  • The same agent node should have access to a Clair instance (through Clair-Scanner) or to a Dagda install.

Here is the Jenkins topology with Clair.
In this diagram, the Clair server would be managed by the security team/role. Not every project should have its own instance of Clair, that would make no sense. It is something that should be common.

Jenkins topology with a specialized node agent (Docker + access to Clair)

The topology would be almost the same with Dadga.
To integrate Dadga in a CI process, you have two options. Dagda being a Python script, either you create a web service to invoke during the build process (that will execute the Python script), or you deploy its database on a separate VM, and you execute Dadga as a container on the Jenkins slave. This container should connect to the remote database. That would be the best option to have quick scans. For the notice, Dadga uses MongoDB instead of PostgreSQL.

It is also important to secure the Docker daemon on the node.
As I do not want to spend too much time on it, here are some links:

Pipeline

With Jenkins, the convention is to define pipelines in a file named Jenkinsfile.
To make ours simple, we provide an all-in-one package. Projects only have to provide parameters and their own tests for their Docker image. The following sample gives an idea about what such a project repository would look like…

image/
- Dockerfile
- ...
tests/
- execute-tests.sh
- ...
Jenkinsfile
readme.md

The project repository contains resources to build the image, resources to test it, and a Jenkinsfile, that describes the build process for Jenkins. Here is the content of this file.

allInOne(
    imageName: 'my-image-name',
    imageVersion: '1.0.0'
)

allInOne is not something Jenkins recognizes by default.
In fact, it refers to a Jenkins shared library we need to define. There will be 3 of them: one for the composite pipeline (« allInOne »), one for the security team, and one for the governance team. Each shared library is defined in its own Git repository, each one having its own permissions. This way, the security team can be sure only it can access/commit about security stuff. Same thing for the Software governance team. And they can all have read access to the shared library that agregates the various stages.

Shared library for the composite pipeline

This is the shared library to use for the « allInOne » tag.
Jenkins pipeline allows to define three things in a shared library: helpers, a pipeline or a step. What would have been better would have been defining stages. But this is not possible. Our allInOne library will thus declare a pipeline with parameters. Here is the main code:

// In vars/allInOne.groovy (shared library that defines the generic pipeline)
def call(Map config) { 
 
	node { 
		def timeStamp = Calendar.getInstance().getTime().format('YYYYMMdd-hhmmss', TimeZone.getTimeZone('Europe/Paris'))
		def buildId = "${config.imageVersion}-${timeStamp}"

		stage('Checkout') { 
			echo "Checking out the sources..." 
			checkout scm 
		}

		stage('Build Image') {
			// Enforce the shape of the repository and assume the Dockerfile is always under image/
			sh 'docker build -t "${config.imageName}:${buildId}" image/'
		}

		stage('Project tests') {
			def scriptFileContent = libraryResource( 'com/linagora/execute-project-tests.sh' )
			sh scriptFileContent
		}

		stage('Security checks') {
			echo "Checking security..."
			securityInspection( "${config.imageName}", "${buildId}" )
		}

		stage('Software Governance') {
			echo "Handling Software checks..."
			softwareCheck( "${config.imageName}", "${buildId}" )
		}

		stage('Promotion') {
			echo "Promoting the local image to a trusted repository..."
			def scriptFileContent = libraryResource( 'com/linagora/promote-image.sh' )
			sh scriptFileContent
		}
	}
}

All the stages are sequential. If one fails, everything stops. The securityInspection and softwareCheck steps are defined farther as global shared libraries. The promote-image.sh and execute-project-tests.sh scripts are provided as resources of the shared library. You can find the whole project online, on Github.

Notice the build stage is very basic.
In this example, we do not handle build parameters. This part can easily be adapted.

Project tests

How can a project specify tests for its Docker image?
Well, this is not complicated. I already faced the issue with a previous project. The idea is the following:

  1. Have a script that launches tests.
  2. This script instantiates the image in detached mode, with a shared volume.
  3. We then execute another script inside the container, by using docker exec.
  4. This second script is in charge of verifying assertions inside the container. Example: verify some process is running, verify a given file was created, etc. When an assertion fails, a message is written inside a file, located in the shared volume. When the container terminates, this file will remain available on the host system.
  5. Once the script has completed, the first script can verify the content of the error file. If there are errors, it can just fail the build.
  6. It is also possible to verify some assertions about the container from the outside (e.g. try to ping or reach a given port).

Here is another repository that illustrates this strategy.
It was used to test RPM and Debian packages in Docker containers. The script that launches the tests and verifies the result is here. Scripts that verify assertions in the containers are located in this directory.

This kind of tests works for functional tests, but it can also be applied for security checks (e.g. permissions on files) or governance ones. The difference here is that these tests lie in the same Git repository than the Dockerfile.

To run them, the allInOne pipeline must verify some script test exists and start it. A naming convention is enough.

Security checks

Security checks are maintained by the security team/role as a shared library, hosted in its own Git repository. Security checks can be verified with a script (e.g. permissions, etc). But for Docker images, there also exist solutions that inspect images and detect leaks.

Clair, from CoreOS, is one of them.
This project uses a database of known vulnerabilities, and scans images. It then provides a dashboard, indicating which CVE were found and for which images.

Dadga is another solution.
Although it is more recent, it provides additional features than Clair. It works the same way, but also uses ClamAV to search for virus, trojans and so on. What takes time for both Dadga and Clair is to update the database of known vulnerabilities. This is why the database cannot be deployed on the fly during the build process. It must preexist to pipeline executions.

Both solutions can also be used to regularly control Docker images. But since the focus of these articles is about delivery pipelines, you will have to dig this part yourself. What is interesting is to integrate these tools in the delivery pipeline, as a validation step. To keep the example simple, I only include Clair. Clair is great, but using it as a command-line is not easy. The best option is to use Clair Scanner as a complement.

And here is our shared library

// In vars/securityInspection.groovy (shared library for the security role)
def call(String imageName, String buildId) { 

	// We assume clair-scanner is available in the path
	def host = sh(returnStdout: true, script: 'hostname -i').trim()
	clair-scanner -c <CLAIR_SERVER_URL> --ip ${host} --t High ${imageName}:${buildId}
}

Here, Clair will scan the given image. If vulnerabilities are found with a severity higher or equal to high, then clair-scanner will return a non-zero exit code and thus fail the build.

If you use Dadgda instead of Clair, you simply run a Python script. The installation is a little bit different, but the pipeline step would remain simple. You can also add custom scripts to perform additional verifications (just add new steps in the pipeline).

Software Governance

Software governance can be managed in the same way that previous stages.
Since it depends on the organization itself, I have no generic tool to suggest. Instead, I assume there is some REST end-point somewhere that can be contacted during the build. The goal is to extract information and send them to a remote web service that will store them, and optionally trigger an alert or a build failure in case of exotic findings.

So, here is an example of associated shared library

// In vars/softwareCheck.groovy (shared library for the Software Governance role)
def call(String imageName, String buildId) { 

	def scriptFileContent = libraryResource( 'com/linagora/analyze-dockerfile.sh' )
	sh scriptFileContent
	sh 'echo "imageName: ${imageName}" >> /tmp/gov.results.txt'
	sh 'echo "imageVersion: ${buildId}" >> /tmp/gov.results.txt'
	sh 'curl --data-binary "@/tmp/gov.results.txt" -X POST...'
	sh 'rm -rf /tmp/gov.results.txt'
}

Pipeline for existing Docker images

The initial pipeline includes a stage to build an image.
However, a project team may want to use an existing image. Community images benefit from various feedbacks and contributors. How to guarantee a project can safely use it inside an organization with its own guidelines?

Well, our generic pipeline, with project tests, security checks and governance, perfectly fits such a use case. The guidelines should be enforced in the automated pipeline. The only difference is that we do not build an image, we use an existing one from the outside. So, let’s adapt our allInOne shared library to cover such a scenario.

// In vars/allInOne.groovy (shared library that defines the generic pipeline, upgraded to support existing images)
def call(Map config) { 

	node { 
		def timeStamp = Calendar.getInstance().getTime().format('YYYYMMdd-hhmmss', TimeZone.getTimeZone('Europe/Paris'))
		def buildId = "${config.imageVersion}-${timeStamp}"

		// Alway checkout the sources, as they may include tests
		stage('Checkout') { 
			echo "Checking out the sources..." 
			checkout scm 
		}

		if (config.existing == true) {
			stage('Docker pull') {
				def buildId = "${config.imageVersion}"
				sh 'docker pull "${config.imageName}:${buildId}"'
			}
		}

		if (config.existing != true) {
			stage('Build Image') {
				// Enforce the shape of the repository and assume it is always under image/
				sh 'docker build -t "${config.imageName}:${buildId}" image/'
			}
		}

		stage('Project tests') {
			def scriptFileContent = libraryResource( 'com/linagora/execute-project-tests.sh' )
			sh scriptFileContent
		}

		stage('Security checks') {
			echo "Checking security..."
			securityInspection( "${config.imageName}", "${buildId}" )
		}

		stage('Software Governance') {
			echo "Handling Software checks..."
			softwareCheck( "${config.imageName}", "${buildId}" )
		}

		stage('Promotion') {
			echo "Promoting the local image to a trusted repository..."
			def scriptFileContent = libraryResource( 'com/linagora/promote-image.sh' )
			sh scriptFileContent
		}
	} 
}

If we use an existing image we simply pull it. Otherwise, we build it. The other parts of the pipeline are the same.
A project using an existing image would then declare its Jenkinsfile as…

allInOne(
    imageName: 'my-image-name',
    imageVersion: '1.0.0',
    existing: true
)

Integration in Jenkins

The most simple solution is to define our 3 shared libraries as global shared libraries. Besides, the shared libraries above all expose global variables, which avoids using import declarations in our Jenkinsfiles. To do so, go into Jenkins administration, system configuration and find the pipeline section. The shared library will be loaded on the fly from a git repository, in every job. It is never cached. For security reasons, the (Jenkins) user that pulls the repository should only have read access.

Here is a screenshot that shows how a shared library is defined in Jenkins.

Jenkins Administration

You can add as many ones as necessary. This article has defined 3 shared libraries, so you would add 3 into Jenkins. It is possible to set versions to shared libraries, but I think it is not necessary for global shared libraries. If stages had to differ between projects, you would define different composite pipelines. And the stages are managed on the fly, in the same fashion than 12 factors.

Perimeter

Two questions can arise once we are here.

  • Where does this pipeline lead?
  • How to prevent a project from by-passing these checks?

The answer to the first question influences the second one.
In my opinion, this pipeline promotes the built image into a trusted Docker registry, that can then be used in Kubernetes environments. You cannot test this image in a K8s cluster before (or if you can, then it must be a sandbox cluster). Once this is clear, the second answer becomes obvious. A project cannot by-pass this pipeline because otherwise, it cannot import its Docker images in the trusted registry. The allInOne shared-library is the only part that can access the trusted registry. It cannot be done from anywhere else: the credentials are kept secret and a single user (Jenkins) should have the write permissions to push an image in the trusted registry. All the other users have read-only access.

Summary

This article has shown how to use Jenkins shared libraries to build a validation process for Docker images, process that is under different responsibilities. I skipped some details to not lost readers, but I believe the main aspects are all explained here.

The next article is about Helm packages: how to verify quality criteria (e.g. linting), test them and so on. All of this with a Jenkins pipeline and the same distribution of responsibilities.

Shared Responsibilities in Pipelines for Docker and Kubernetes

DevOps, and in fact DevSecOps, tend to be the norm today.
Or let’s say it tend to be an ideal today. Many organizations are far from it, even if more and more are trying to adopt this way of work.

This kind of approach better fits small structures. With few means, it is not unusual for people to do everything themselves. Big structures, in particular among the GAFAM, have split their teams in small units (pizza teams are one example) that behave more or less like small structures. The issue with big organizations, is that they face additional challenges, mainly keeping a certain coherence in the information system, containing entropy and preventing anarchy. This is about having a common security policy, using identified languages, libraries and frameworks, and knowing what is used and where. In one word: governance.

This may seem like obsolete debates from a technical point of view. But from an organizational point of view, this is essential. Big structures often have support contracts. They need to know what runs or not. Especially when there can be controls from their contractors (think about Oracle or IBM as an example). They also need to know what kinds of technologies run. If a single person in the company masters this or that framework, how will the project be maintained? Knowing what runs allows to identify the required skills. Structures with many projects also need to know how projects interact with each others. This can impact the understanding when a crisis arises (what parts of the system depend on what). Last but not least, security is a crucial aspect, that goes beyond the sole projects. Having common guidelines (and checks) is imperative, in particular when things go faster and faster, which became possible by automating delivery processes.

One of the key to be agile, goes through continuous integration, and now continuous deployments. The more things are automated, the faster things can get delivered. The governance I described above should find a place in these delivery pipelines. In fact, such a pipeline can be cut into responsibility domains. Each domain is associated with a goal, a responsibility and a role that is in charge. In small organizations, all the roles may be hold by the same team. In big ones, they could be hold by different teams or councils. The key is that they need to work together.

This does not minor the benefits of DevSecOps approaches.
The more polyvalent a team is, the faster it can deliver its project. And if you only have excellent developers, you can have several DevOps teams and expect them to cope on their own. But not every organization is Google or Facebook. Some projects might need help, and keeping a set of common procedures is healthy. You will keep on having councils or pools of experts, even if your teams are DevSecOps. The only requirement is that the project teams are aware of these procedures (regular communication, training sessions) and all of these verifications should be part of automated delivery pipelines.

Responsibility Domains

I have listed 3 responsibility domains for automated pipelines:

  • Project is the first one, where the project team can integrate its own checks and tests. It generally includes functional tests. This is the usual part in continuous integration.
  • Security is the obvious one. It may imply verifications that developers may not think about.
  • Software Governance is the last domain. It may include used programs (do we have a support contract?), version checks, notify some cartography API, etc.

Different stages in the build pipeline cover various concerns

The goal is that each domain is a common, reusable set of steps, defined in its own (Git) repository. Only the security team/role could modify the security stage. Only the Software governance team/role could modify its stage. And only the project team should be able to specify its tests. Nobody should be able to block another domain from upgrading its part. If the security team needs to add a new verification, it can commit it anytime. The global build process should include this addition as soon as possible. The composition remains, but the components can evolve independently.

Every stage of the pipeline is controlled from a different Git repository

This article is the first from a series about implementing such a pipeline for Kubernetes. Since the projects I am involved in all consist in providing solutions as Kubernetes packages, I focus on Docker images and Helm packages. The Docker part asks no question. Helm provides a convenient way to deliver ready-to-use packages for Kubernetes. Assuming a team has to provide a solution for clients, this is in my opinion the best option if one wants to support it efficiently.

Assumptions

To make things simple, I consider we have one deliverable per Git repository. That is to say:

  • Project sources, whose build result is stored somewhere (Maven repo, NPM…).
  • A git repo per Docker image used in the final application.
  • A git repo for the Helm package that deploy everything (such a package can depend on other packages, and reference several Docker images).

A project that develops its own sources would thus have at least 3 Git repositories.
We could mix some of them, and complexify our pipeline. Again, I avoided it to keep things easy to understand.

There are also two approaches in CI/CD tools: build on nodes, and build in containers. There are also many solutions, including Jenkins, GitLab CI, Travis CI, etc. Given my context, I started with Jenkins with build performed on nodes. I might add other options later. The global idea is very similar for all, only the implementation varies.