Understand pipeline stages

Pipelines enable you to automate the steps in your deployment process. Your process might include several logical groups of jobs that you want to run. In this unit, you'll learn about pipeline stages and how to use them to add quality control processes to your Bicep deployments.

What are pipeline stages?

Stages help you to divide your pipeline into multiple logical blocks. Each stage can contain one or more jobs. Jobs contain an ordered list of steps that should be completed, like running command-line scripts.

You can use stages in your pipeline to mark a separation of concerns. For example, when you work with Bicep code, validating the code is a separate concern from deploying your Bicep file. When you use an automated pipeline, building and testing your code are often called continuous integration (CI). Deploying code in an automated pipeline is often called continuous deployment (CD).

During CI stages, you check the validity of the changes that have been made to your code. CI stages provide quality assurance. You can run them without affecting your live production environment.

In many programming languages, code needs to be built before someone can run it. When a Bicep file is deployed, it's converted, or transpiled, from Bicep to JSON. The tooling performs this process automatically. In most situations, you don't need to manually build Bicep code to JSON templates in your pipeline. We still use the term continuous integration when we talk about Bicep code, though, because the other parts of CI still apply, such as validating your code.

After your CI stages run successfully, you should have increased your confidence that the changes you made will also deploy successfully. During CD stages, you deploy your code to each of your environments. You usually start with test and other nonproduction environments and then continue to production environments. In this module, you'll deploy to a single environment. In a future module, you'll learn how to extend your deployment pipeline to deploy to multiple environments, such as nonproduction and production environments.

Stages run in a sequence. You can control how and when each stage runs. For example, you can configure your CD stages to run only after your CI stages run successfully. Or you might have multiple CI stages that need to run in sequence, for example, to build your code and then test it. You might also include a rollback stage that runs only if previous deployment stages fail.

Shifting left

By using stages, you can verify the quality of your code before you deploy it. Using these stages is sometimes called shifting left.

Consider a timeline of the activities that you perform when you write code. The timeline starts with the planning and design phases. It then moves to the building and testing phases. Finally, you deploy and support your solution. The following diagram shows these stages on a timeline.

It's a well-understood rule in software development that the earlier in the process you find an error (the closer to the left side of the timeline), the easier, faster, and cheaper it is to fix. The later in the process that you catch an error, the harder it is to fix.

So the goal is to shift the discovery of problems toward the left side of the diagram. Throughout this module, you'll see how you can add more validation and testing to your pipeline as it progresses.

You can even add validation well before your deployment begins. When you work with tools like Azure DevOps, pull requests typically represent changes that someone on your team wants to make to the code on your main branch. It's helpful to create another pipeline that automatically runs your CI steps during the review process for the pull request. This technique helps validate that the code still works, even with the proposed changes. If the validation succeeds, you have some confidence that the change won't cause problems when it's merged to your main branch. If the check fails, you know that there's more work to do before the pull request is ready to merge.

Define a pipeline stage

Every pipeline contains at least one stage. If your pipeline has only one stage, you don't need to explicitly define it. Azure Pipelines automatically defines it for you. When you have multiple stages in a pipeline, you need to define each one. Stages run in a sequence that you specify.

Imagine that you've built a Bicep file that you need to deploy twice: once to infrastructure in the United States and once to infrastructure in Europe. Before you deploy the file, you validate your Bicep code. Here's an illustration of a multistage pipeline that defines this process:

Notice that this example has three stages. The Validate stage is similar to a CI stage. The DeployUS and DeployEurope stages run next. Each deploys the code to one of the environments.

Here's how the stages are defined in a pipeline YAML file:

stages:

- stage: Test
  jobs:
  - job: Test

- stage: DeployUS
  jobs: 
  - job: DeployUS

- stage: DeployEurope
  jobs: 
  - job: DeployEurope

Control the sequence of stages

By default, the stages run in the order that you define. A stage runs only if the previous stage succeeds. You can add dependencies between the stages to change the order.

Continuing the previous example, imagine that you want to run both of your deployments in parallel, like this:

You can specify the dependencies between stages by using the dependsOn keyword:

stages:

- stage: Test
  jobs:
  - job: Test

- stage: DeployUS
  dependsOn: Test
  jobs: 
  - job: DeployUS

- stage: DeployEurope
  dependsOn: Test
  jobs: 
  - job: DeployEurope

When you use the dependsOn keyword, Azure Pipelines waits for the dependent stage to finish successfully before it starts the next stage. If Azure Pipelines detects that all the dependencies for multiple stages are satisfied, it can run those stages in parallel.

You might want to run a stage when a previous stage fails. For example, here's a different pipeline. If the deployment fails, a stage called Rollback runs immediately afterward:

You can use the condition keyword to specify a condition that should be met before a stage runs:

stages:

- stage: Test
  jobs:
  - job: Test

- stage: Deploy
  dependsOn: Test
  jobs: 
  - job: Deploy

- stage: Rollback
  condition: failed('Deploy')
  jobs: 
  - job: Rollback

In the preceding example, when everything goes well, Azure Pipelines runs the Validate stage first, and then it runs the Deploy stage. It skips the Rollback stage. However, if the Deploy stage fails, Azure Pipelines runs the Rollback stage. You'll learn more about rollback later in this module.

Every job runs on a new agent. That means that every job starts from a clean environment, so in every job you typically need to check out the source code as your first step.

Bicep deployment stages

A typical Bicep deployment pipeline contains several stages. As the pipeline moves through the stages, the goal is to become increasingly confident that the later stages will succeed. Here are the common stages for a Bicep deployment pipeline:

1. Lint: Use the Bicep linter to verify that the Bicep file is well formed and doesn't contain any obvious errors.
2. Validate: Use the Azure Resource Manager preflight validation process to check for problems that might occur during deployment.
3. Preview: Use the what-if command to validate a list of changes that will be applied against your Azure environment. Have a human manually review the what-if results and approve the pipeline before it proceeds.
4. Deploy: Submit your deployment to Resource Manager and wait for it to finish.
5. Smoke Test: Run basic post-deployment checks against some of the important resources that you deployed. These reviews are called infrastructure smoke tests.

Your organization might have a different sequence of stages, or you might need to integrate your Bicep deployments into a pipeline that deploys other components. After you understand how the stages work, you can design a pipeline to suit your needs.

Throughout this module, you'll learn more about the stages listed here, and you'll progressively build a pipeline that includes each stage. You'll also learn:

• How pipelines stop the deployment process if anything unexpected happens during any of the previous stages.
• How to configure your pipeline to pause until you manually verify what happened in a previous stage.