This chapter will cover all aspects of Jenkins Pipeline, from running pipeline jobs to writing your own pipeline code, and even extending Pipeline.
This chapter is intended to be used by Jenkins users of all skill levels, but beginners may need to refer to some sections of "Using Jenkins" to understand some topics covered in this chapter.
If you are not yet familiar with basic Jenkins terminology and features, start with Getting Started with Jenkins.
Jenkins Pipeline is a suite of plugins which supports implementing and integrating continuous delivery pipelines into Jenkins. Pipeline provides an extensible set of tools for modeling simple-to-complex delivery pipelines "as code" via the Pipeline DSL. [1]
Typically, this "Pipeline as Code" would be written to a Jenkinsfile and
checked into a project’s source control repository, for example:
pipeline {
agent { // <1>
label ''
}
stages{
stage('Build') { // <2>
steps { // <3>
sh 'make' // <4>
}
}
stage('Test'){
steps {
sh 'make check'
junit 'reports/**/*.xml' // <5>
}
}
stage('Deploy') {
steps {
sh 'make publish'
}
}
}
}| 1 | agent indicates that Jenkins should allocate an executor and workspace for
this part of the Pipeline. |
| 2 | stage describes a stage of this Pipeline. |
| 3 | steps describes the steps to be run in this stage |
| 4 | sh executes the given shell command |
| 5 | junit is a Pipeline step provided by the
JUnit plugin
for aggregating test reports. |
Jenkins is, fundamentally, an automation engine which supports a number of automation patterns. Pipeline adds a powerful set of automation tools onto Jenkins, supporting use cases that span from simple continuous integration to comprehensive continuous delivery pipelines. By modeling a series of related tasks, users can take advantage of the many features of Pipeline:
Code: Pipelines are implemented in code and typically checked into source control, giving teams the ability to edit, review, and iterate upon their delivery pipeline.
Durable: Pipelines can survive both planned and unplanned restarts of the Jenkins master.
Pausable: Pipelines can optionally stop and wait for human input or approval before continuing the Pipeline run.
Versatile: Pipelines support complex real-world continuous delivery requirements, including the ability to fork/join, loop, and perform work in parallel.
Extensible: The Pipeline plugin supports custom extensions to its DSL [1] and multiple options for integration with other plugins.
While Jenkins has always allowed rudimentary forms of chaining Freestyle Jobs together to perform sequential tasks, [2] Pipeline makes this concept a first-class citizen in Jenkins.
Building on the core Jenkins value of extensibility, Pipeline is also extensible both by users with Pipeline Shared Libraries and by plugin developers. [3]
The flowchart below is an example of one continuous delivery scenario easily modeled in Jenkins Pipeline:
When Pipeline was first conceived it was natural to begin with Groovy as the foundation. Jenkins already had an embedded Groovy scripting engine with a console to interact with all parts of Jenkins and Groovy is a great language for creating a custom Domain-Specific Language (DSL). The Job-DSL, plugin for Jenkins was also written in Groovy. This plugin lets you automate the creation and editing of any job in Jenkins, including Pipelines.
Pipeline initially introduced key pipeline-specific concepts such as node,
stage, parallel, and extension points to allow plugins to add other steps
to the DSL but, otherwise didn’t restrict the use of Groovy. This original syntax
for creating Pipelines is now referred to as "Scripted Pipeline" and includes
full programmatic control to allow scripting continuous delivery pipelines in Groovy.
This givesPipeline creators tremendous flexibility in defining a pipeline and allows it to
be extended via Shared Libraries or plugins.
| Due to the need to serialize all variables for durability some Groovy idioms are not fully supported yet. See JENKINS-27421 and JENKINS-26481 for more information. |
Writing Pipelines with Scripted Pipeline syntax, however, does require at least some proficiency with Groovy. Requiring all team members that touch the application’s Pipeline to understand Groovy limits the ability for full-participation in code-review, audits, and editing of the Pipeline as part of the application code. Enter "Declarative Pipeline" syntax.
Declarative Pipeline syntax was created to extend Pipeline to users of all experience levels and complement Scripted Pipeline syntax. As the name implies, it is intended to enable declarative programming [4] for defining Pipelines as opposed to the imperative programming [5] provided by Scripted Pipeline. While it is still a DSL written on top of Groovy, Declarative Pipeline is a limited to a pre-defined structure that is much more specific to continuous delivery. This allows all stakeholders to help create, edit, review, and audit the application’s Pipeline.
Scripted Pipeline and Declarative Pipeline both use the same underlying Pipeline execution engine and both are fully-supported. You can use whichever you prefer in any of your Pipelines and even combine them when needed. All examples in this handbook will show both a Declarative Pipeline version and Scripted Pipeline version for your reference.
A single task; fundamentally steps tell Jenkins what to do. For example,
to execute the shell command make use the sh step: sh 'make'.
When a plugin extends the Pipeline DSL, that typically means the plugin has
implemented a new step.
Most work a Pipeline performs is done in the context of one or more
declared node steps. Confining the work inside of a node step does two
things:
Schedules the steps contained within the block to run by adding an item to the Jenkins queue. As soon as an executor is free on a node, the steps will run.
Creates a workspace (a directory specific to that particular Pipeline) where work can be done on files checked out from source control.
| Depending on your Jenkins configuration, some workspaces may not get automatically cleaned up after a period of inactivity. See tickets and discussion linked from JENKINS-2111 for more information. |
stage is a step for defining a conceptually distinct subset of the
entire Pipeline, for example: "Build", "Test", and "Deploy", which is used by many
plugins to visualize or present Jenkins Pipeline status/progress.
[6]