The deployment paradigm of development, testing, production, and disaster recovery from the pre-cloud, pre-microservices days still lingers in my mind. In those days, buying, configuring, and maintaining infrastructure was expensive and complex, so you were unlikely to get the data center ops team to sign off on creating more environments and deployment flexibilities. By today’s standards, deployment frequencies were low, so agile development teams found ways to make do with the available infrastructure.
Today, devops organizations use CI/CD (continuous integration and continuous delivery) to automate delivery, Infrastructure as Code to configure infrastructure, Kubernetes to containerize environments, continuous testing to improve quality, and AIops to centralize monitoring and observability alerts. These practices make it possible to increase deployment frequency while minimizing security, performance, and reliability risks from the changes.
But when you need to lower risk and control releases based on what’s changing and who gets access to the changes, you need additional techniques to configure and manage the application or deployment.
Release pipelines require flexible controls and deployment options
One way to create controlled deployments is to use feature flags to toggle, A/B test, and segment access to features by user. Devops team can use feature flags to validate reengineered features when moving apps to the cloud or refactoring monolithic apps into microservices.
Feature flags are useful for development teams and product managers who want to control who sees what capabilities in an app or microservice. But sometimes, devops or IT operations teams need flexibility in managing deployments and controlling multiple versions of an application running in production. Some examples:
- Validating apps and microservices when patching or upgrading critical system components such as operating systems, databases, or middleware
- Testing apps and services when deploying to multiple clouds
- Monitoring user experiences after releasing new versions of machine learning models
There are several different deployment strategies for microservices and cloud-native applications. In the early days of developing apps for the cloud, we employed blue-green deployments where there were two production environments: a green live one and a blue one on standby. The deployment went to the blue environment, and then load balancers controlled the cutover, effectively making the blue environment with the latest deployment live and the new green environment the backup.
Blue-green deployments were effective in minimizing downtime during deployments and provided the option for rollbacks if necessary. But they were inefficient because they required dedicated infrastructure for the blue environment.
What are canary deployments?
The evolution of blue-green deployments includes canary deployments or releases. I talked with Rohan Gupta, senior product manager at Harness, on how he defines canary releases. He says, “A canary release is a deployment strategy that releases an application or service incrementally to a subset of users and reduces the scope, impact, and risk of deploying new software artifacts to production. All infrastructure in a target environment is updated in small phases (for example, in increments of 2%, 25%, 75%, 100% of traffic load), and because of this control, a canary release is the lowest risk-prone approach compared to all other deployment strategies.”
There are several ways to implement canary deployments with virtualization and cloud technologies, and Spinnaker is one open source tool supporting this capability. The generalized approach has a production environment, a smaller baseline production replica, and a canary environment for new deployments. Intelligent routing controls traffic to these environments dynamically, and rules determine if and when a new canary deployment meets acceptance criteria.
Canary deployments are one approach to achieve continuous deployment where CI/CD pipelines also trigger deployments to production environments. Continuous deployment requires a solid software development process, vigorous observability practices, reliable CI/CD pipelines, robust test automation capabilities, comprehensive AIops monitoring, and intelligent production deployment methodologies, which is where canary deployments fit in.
How devops teams can use canary deployments
John Kodumal, CTO and cofounder of LaunchDarkly, shares how devops teams use feature flagging with canary deployments. He says, “A canary release is when you make new software features available to a limited selection of users ahead of a wider release. Its purpose is to provide a better software experience for users with less overall risk. Canary releases are useful in providing a broad sanity check of a release and would ideally work in conjunction with feature flags, which hit at the application layer and can isolate individual changes.”
Gupta says one main reason to use canary deployments is to compare the results of multiple production releases. He says, “Canary deployments allow organizations to test in production with real users and use cases and compare different service versions side by side. It is cheaper than a blue-green deployment because it does not require two production environments. Also, it is fast and safe to trigger a rollback to a previous version of an application.”
Gupta also acknowledges that canary releases often require architecting a solution and advanced scripting. He says, “Drawbacks to canary deployments involve testing in production and the expertise needed to implement this deployment pattern. Scripting a canary release is complex because manual verification or testing takes time, and the required monitoring and instrumentation for testing in production may involve additional effort. A good CD solution should provide templates and automation to solve all of these challenges.”
Canary deployments may be overkill if you’re developing apps and microservices with lower reliability and scalability requirements. But for devops teams seeking continuous deployment for high volume, mission-critical customer experiences and dependent microservices, then canary deployments can be a game-changing devops practice.
Margaret Francis, president and COO at Armory, agrees. She says, “Being able to run canary releases is particularly important at scale. When your business depends on your service being available, performant, and predictable, then progressive delivery of code to production ensures your services are performing and that you can ship innovation at the same time. It’s not as critical when you have a team of two and a release schedule of once a month. But companies that depend on their digital services, and organizations with hundreds of developers performing daily and hourly releases should prioritize a canary-type release process. These organizations can’t afford to sacrifice the speed of innovation with service stability—or vice versa.”
Target continuous deployment when speed, scale, and reliability matter
As I noted earlier, continuous deployment is a high bar and requires devops teams to have several mature testing, automating delivery, and monitoring practices. For many businesses, applications, and services, continuous delivery may be sufficient, and it may be too costly or technically complex to pursue continuous deployment capabilities.
But when developing large-scale apps and microservices, and if the goal is to release frequently, then enabling canary releases is a key strategy for deploying high-performing, reliable releases.