Combining Feature Flags with Deployment Strategies to Optimize the Release Process - Part I

Introduction

The modern software development and deployment cycle has become very complex but also very efficient. Often referred to collectively as DevOps, this approach aims to help teams iterate rapidly and accurately on their products to meet business needs. It emphasizes the need for fast and secure software deployment.

Today, many teams have devised their deployment strategies based on well-established DevOps best practices. This article will introduce some typical software product deployment strategies, then discuss when, why, and how feature flags should be incorporated into the deployment process. At the end, I will provide examples of how to optimize the release process by combining feature management with deployment strategies. Below is the table of contents for this article:

• Typical Deployment Strategies (Frequency)
• DORA Metrics for Deployment Strategies - The Key Metrics to Measure Deployment Performance
• Methods to Derisk the Product Release
• Why Introduce Feature Flags - Derisking Feature Release & Innovating Faster
• When to Introduce Feature Flags
• Examples of How to Combining Feature Flags with Deployment Strategies to Optimize the Release Process

Note: This practice article will be divided into two separate pages. The chapter on "How to Introduce Feature Flags" and "Examples of Combining Feature Flags with Deployment Strategies" can be found on the next page.

I hope that after reading this article, those skeptical of feature flags may change their minds, and those unfamiliar with them will learn how to use them effectively.

Typical Deployment Strategies (Frequency)

Deployment strategies in the world of software development can significantly vary depending on the size of the organization, the nature of the project, and the team's agility. Two common approaches stand out: scheduled deployments and multiple deployments without a fixed schedule. Each strategy has its unique advantages and scenarios where it's most beneficial.

Scheduled Deployment

Scheduled deployment, often adopted by larger organizations, involves setting specific dates and times for when new features or updates are released into production. This approach typically sees deployments occurring on a bi-weekly basis, although the frequency can vary based on the company's needs and the complexity of the projects.

Key Benefits and Scenarios:

• Predictability and Stability: Perfect for complex, large-scale projects requiring thorough testing and coordination. Scheduling deployments in advance allows for better coordination with stakeholders, including product/project manager, marketing, customer support, and sales teams, to prepare for feature launches.
• Regulatory Compliance: Suitable for industries with stringent regulatory requirements, providing the needed timeframe for compliance checks.

Multiple Deployments Without a Fixed Schedule

This strategy is more commonly seen in smaller teams or within larger organizations for sub-projects. Deployments can occur multiple times per day or week, depending on the team's progress and the urgency of the updates. The lack of a fixed schedule allows for greater flexibility and faster iteration.

Key Benefits and Scenarios:

• Flexibility and Rapid Iteration: Best for projects needing fast adjustments to user feedback or market changes.
• Continuous Improvement: Ideal for parts of larger projects or small projects that can be independently developed and deployed, enhancing team efficiency and product quality through frequent updates.

When a larger project can be broken down into smaller, independent parts, deploying these parts without a fixed schedule can increase overall team efficiency. Next, we will explore the DORA Metrics for Deployment Strategies to better understand how to measure the performance of these deployment strategies.

DORA to Measure Deployment Performance

The DevOps Research and Assessment (DORA) metrics are widely used to measure the performance of software development and deployment. These metrics help teams understand how well their deployment strategies are working and where they can improve. The four key metrics are:

• Deployment Frequency: How often a team deploys code to production.
• Lead Time for Changes: The time it takes for a code change to be deployed to production.
• Mean Time to Recover (MTTR): The time it takes to recover from a failure in production.
• Change Failure Rate: The percentage of deployments causing a failure in production.

In DORA's report, it shows that high-performing teams deploy more frequently, have shorter lead times, and recover from failures faster. But in the recent year reports, it's invesgating more about why the deployment frequency can't be descreased for may big organizations.

The most important aspect is that we can use these metrics to evaluate the effectiveness of our deployment strategies and make improvements accordingly. These metrics are crucial for measuring whether the hypothesis 'Combining Feature Flags with Deployment Strategies to Optimize the Release Process' holds true.

Methods to Derisk Product Deployment

In DORA metric, it measure the speed and stability of the deployment. But how to make the deployment more stable? You should pay attention to many steps in deployment pipeline, such as code scanning, testing, monitoring, deployment, release amd so on. In this article, we focus on the last steps: deployment and release. Here are some traditional methods to derisk the product deployment:

Image below comes from ByteByteGo System Design Alliance

• Blue-green deployment, which reduces downtime and risk by running two identical production environments.
• Canary deployment/release, which reduces risk by rolling out a new feature or update to a small subset of users before rolling it out to the entire user base.
• Progressive deployment/release, which reduces risk by releasing a new feature or update to a small subset of users, then gradually increasing the percentage of users who can access it.
• Targeted Experience deployment/release, which reduces risk by releasing a new feature or update to a small subset of users, without making it visible to the entire user base. In comparison with Canary Deployment and Rolling Deployment, it focuses more on trialing and approving new ideas and features, rather than mitigating risks from coding bugs.
• A/B testing, which reduces risk by comparing two versions of a feature or update to see which performs better.
• And so on.

There're several ways to implement the methods above, such as control the traffic by load balancer, or use feature flags to control the traffic. Compare to load balancer solution, feature flags keep you control the release in the code level, and it's more flexible and easy to manage.

Note: Feature flags are not always the best option, and in certain scenarios, they may not be suitable at all. For example, when upgrading your backend service from .NET 6 to .NET 8, you cannot use feature flags to manage traffic. Instead, you should use a load balancer to direct traffic between two different service hosts, containers, or VMs. But you can still write your feature flag in your load balancer service :D.

Why Introduce Feature Flags - hint from DORA metric

Using feature flags can make your deployment strategy Safer and Faster, as evidenced by DORA metrics.

Innovate Faster

Deploy more frequently, means users can get new features faster.

Using feature flags allows for early release of new features without full completion, avoiding the complexities of long-term branch management. Developers can merge changes into the main branch regularly while limiting feature access to certain users. This strategy also streamlines major codebase refactorings by updating new features alongside, reducing the risk of difficult merge conflicts later on. You can deploy your application which contains the new feature, but the feature is not visible to the users. It's a good way to innovate faster.

Have shorter lead times, means the time to market is shorter.

With feature flag, time-to-production becomes shorter. A feature flag can be maintained in an off state for everyone except for the development team and the Quality Assurance (QA) team. This setup allows these teams to thoroughly test and adjust the feature in preparation for a wider release. Once the feature is considered ready for production use, it can be activated for all users without the need for a new software deployment.

Using feature flags can enhance the implementation of the Trunk-based development method which leads deploying more frequently and Have shorter lead times. Deploy your code whenever you want, and release it when you're ready.

Derisk Feature Release

Decrease the change failure rate, means the feature flags can help to decrease the risk of the new feature release.

Feature flags provide a powerful mechanism for reducing the change failure rate by enabling more controlled, safer feature releases and facilitating rapid response to any issues that arise. As we mentioned in the chapter of "Methods to Derisk Product Deployment", feature flags can be used to implement Canary deployment, Progressive deployment, A/B testing, and so on. Beside deployment methods, good feature flag tools can also help in Performance Monitoring and enhance Environment Consistency. It helps you to discover the issues in the early stage, and fix it before the feature is visible to the all users.

Recover from failures faster, means the feature flags can help to recover from the failure faster.

Feature flags can be used to quickly disable a feature that is causing issues, without the need for a new software deployment. This capability allows teams to respond to issues in real-time, reducing the time it takes to recover from failures. It's a good way to "decrease the MTTR". The open source feature flags tools like FeatBit supports the integration with the monitoring tools, so you can:

1. Getting the alert when the feature is causing issues. Many OBS tools can figure out the issue by the feature flag name, even the postion of the code hidden behind the feature flag.
2. Setting auto trigger to disable the feature flag when the alert is triggered. FeatBit support two ways: Flag Trigger and Rest API.
3. Address the issue and deploy the fix immediately or upon your return to the office the next day. A bug caused by one person won't disrupt the entire team's sleep.

Ref: Feature Flags: What They Are and How to Use Them by JetBrains Team

Figure below shows the ROI of Feature Management

When to Introduce Feature Flags

You can introduce feature flags into your deployment strategies in various ways, with the core spirit being to de-risk and innovate faster. Here are some hints that may inspire you:

• Introduce feature flags when you aim to minimize the blast radius of bugs, issues, or poor user experiences. By leveraging feature flags, you can roll out features in a controlled manner, even without a dedicated operations team or extensive operations expertise.
• When you are about to release a new feature or update that has a high risk of failure.
• Utilize feature flags to innovate faster and manage risk more effectively, especially in large teams with numerous projects, sub-projects, and libraries. Implementing feature flags ensures that an issue with a single feature does not necessitate rolling back other features, allowing for a more stable and continuous delivery process.
• Deploy feature flags to enhance your DORA (DevOps Research and Assessment) metrics performance. This approach enables faster and more frequent deployments with fewer problems, aligning with the goals of deploying quickly and reliably.
• Product or Project Managers should employ release strategies using feature flags based on customer feedback, their roadmap, and marketing requirements. The role of the Product or Project Manager is crucial in determining when and how to implement these strategies to best support the product's success.
• DevOps teams or Development teams may use feature flags for infrastructure migration strategies. This is particularly useful when migrating infrastructure components such as IP addresses or databases. Feature flags allow for smooth, step-by-step migrations with the option for immediate reversion, minimizing potential risks, panic, and costs.
• Implement feature flags when the user experience of a new feature is critical to your reputation and business. Careful rollout of features is necessary to maintain and enhance user satisfaction.
• Use feature flags for A/B testing to determine which strategies are more effective or user-friendly, thus supporting business growth and customer satisfaction.

In conclusion, companies should consider mandating the use of feature flag management for any new feature introduction. This is especially relevant for organizations with large teams and multiple ongoing projects, as it ensures that the release of one feature does not adversely affect others, fostering a culture of innovation and safety.

Examples of How to Combining Feature Flags with Deployment Strategies to Optimize the Release Process

Introducing feature flags into your codebase is not solely the responsibility of developers; it also involves the product manager, project manager, and DevOps team. There are many considerations to keep in mind when implementing feature flags, especially to optimize the release process in alignment with your deployment strategy.

I am writing a series of practical articles titled "How to Get Your Team On Board With Using Feature Flags". Stay tuned!

With love and peace. ღ( ´･ᴗ･` )