Mastering Feature Flags: Life Cycle

This article is part of a series; if you got here first, it might be worth it to check some of the previous ones first:

• The basics
• Types of feature flags
• Life cycle (you are here)
• Testing feature flags
• Risk mitigation
• Performance and Resilience

Feature flags, or feature toggles, are pivotal in contemporary software development. This article provides an overview of the typical life cycle of a feature flag, from its inception to eventual decommissioning. Similarly to the software development life cycle, the feature flag life cycle may vary depending on the business domain and system criticality. This article presents a generalized life cycle without making assumptions about specific business requirements.

The life cycle at a glance

Creation

The first step entails documenting the flag’s objectives, stakeholders, and expected results. A detailed testing plan should be developed as well. The flag is then “created” in the team’s chosen flag management system and added to the codebase, usually in its default “off” state.

Risk assessment is required to evaluate the impact of the feature flag on the system and business and any potential dependencies to other features and flags. KPIs should be set to measure the flag objectives. Especially in regulated industries, it is vital to consider how feature flags interact with legal requirements.

Naming convention

A standardized naming convention for feature flags is required to maintain clarity and consistency across development teams. It should eliminate ambiguity and simplify identifying each flag’s purpose, scope, and ownership, reducing the risk of misconfiguration or misuse. Moreover, a well-defined naming structure streamlines documentation and monitoring processes, enhancing overall project manageability.

Depending on the company’s scale and size, there are many ways to approach this. This section outlines a somewhat generic approach to naming that should be distilled based on one’s reality and needs.

• Names should be split into categories like “global” and “team” flags.
• Include prefix for the type of flag:
• Release: RLS
• Experimentation: EXP
• Operational: OPS
• Permissions: PRM

1. Flags owned by a specific team → prefix-productCode-teamCode-flagName:
2. RLS-ShoppingCart-CartTeam-ShoppingCartV2.1
3. EXP-ShoppingCart-DataScienceTeam-FloatingShoppingCart
4. OPS-ShoppingCart-DataScienceTeam-IncreasedLogLevelForFloatingShoppingCart
5. PRM-ShoppingCart-DataScienceTeam-EnabledFloatingShoppingCart
6. Global flags → prefix-globalPrefix-domain-flagName: OPS-Global-FeatureFlags-InvalidateAllCachedFlags

Testing

A flag undergoes validation in controlled environments. Its various states are verified against the testing plan, and interactions/dependencies to other flags are also tested, where applicable. There is an upcoming article that is going to do a deep dive in flag testing.

Deployment

Before deployment, the feature flag is configured to its default state in the flag management system. The application code, which includes the feature flag, is then deployed. Verification should be performed to ensure the application functions correctly with the flag in its default “off” state — turning the flag off would be the first action during an incident response.

Activation

There should be no change to the flag value without a “green light.” Necessary approvals for activation should be obtained following a process similar to the one required to deploy code to the production environment. The rollout strategy should be determined (gradual, percentile, canary, etc.). Subsequently, the flag is turned on for the desired user group.

After the necessary approvals for activation are secured, and an audit log is maintained — the rollout strategy, whether gradual or percentile-based, is established. The flag is activated for its intended audience.

Any change to the value of a flag should be considered a rollout if it affects the system’s end users. In the case of feature flags, there are a few different rollback considerations:

• Return to a previously known good state of the flag in the flag management system.
• Redeployment of an older known good state of the codebase.

Changes in feature flags on production should be treated as a production deployment.

Monitoring

Key performance indicators (KPIs) are closely monitored to assess the flag’s impact on the system and the user experience. Measuring the performance of feature flags involves considering both the effect of the flag on the design and the effectiveness of the flag in achieving its intended goals.

There will be a more detailed chapter on monitoring in a follow-up article.

Deactivation (optional)

The flag may be deactivated for a variety of reasons:

• Adverse impact on revenue
• System instability
• Flag reaching its end-of-life

Clean up

Once the flag is stable, it is removed from the codebase. The associated feature either becomes a permanent part of the system or is removed. The flag remains in the management system until full user adoption.

Decommission

The flag is officially retired! It should be removed from the flags management system, and all related documentation should be updated. Both the planned and actual decommissioning dates are noted.

The flag’s name should remain unusable for some time to ensure that rollback to the previous application version would not require the re-creation of the flag.

Post-decommission, the flag’s data and audit log should be archived for compliance or historical analysis.

The process around the life cycle

Managing the life cycle of a feature flag is not just a technical endeavor but also involves strategic planning, cost considerations, and stakeholder engagement.

Documenting feature flags

Good documentation helps ensure that everyone on the team understands what the flag does, when it should be used, who can approve activation/deactivation, and aids in troubleshooting.

At the very least, a feature flag should have the following:

• Name — According to the approved naming convention.
• Status — Active/Inactive, Environments availability: Production, Staging, Testing, etc.
• Creation date — Date of creation in the feature flag management system.
• Expected to decommission date — Date when all code and feature flag management system references would be removed.
• Description — Detailed information about what the flag does, what features or settings it controls — links to relevant documentation, disaster recovery plan, etc.
• Type of the flag — Experimentation, Operational, Permissions, Release.
• Scope of the flag — Long/Short lived, User/System based.
• Change log — Record of changes to the flag (audit log exists in feature flags management system).
• Owner of the flag — Stakeholders responsible for the creation of the flag.
• Values descriptions — What are the possible values of the flag and their expected behavior?
• Dependencies — What features, teams, and domains are affected by changes to the value? Who should be notified before and after modifications to the flag?
• Fallback behavior — What is the expected default behavior in a feature flag management system outage?

Maintenance

Ongoing maintenance is a cornerstone of effective feature flag management. Regular audits help identify obsolete flags, minimize technical debt, and optimize system performance. As part of the maintenance, an awareness of the total cost of ownership of the system should be kept up to date.

Cost Analysis

A comprehensive understanding of the cost structure is essential for sustainable feature flag management. This encompasses development, operational overhead, and the complexities introduced by an increasing number of active flags. Keeping the number of active flags in the system low should be one of the priorities. Three flags with two values each would give eight possible configuration permutations. Taking a more extensive system as an example, where there could easily be 100 flags with an average of 4 values per flag, the number of possible permutations is exponentially growing to 1.6069380442589902755419620923412e+60 permutations.

Versioning

Maintaining consistency across different software versions is crucial when a feature flag is involved. Proper versioning ensures that the flag behaves as expected, irrespective of the software release it interacts with. Versioning is also an aspect to consider when defining a rollback strategy.

Stakeholder Involvement

The life cycle of a feature flag is a collaborative process that requires input from various stakeholders. Their involvement is critical in defining the flag’s objectives, approving its activation, monitoring its impact, and ensuring alignment with organizational goals. For example, there shouldn’t be a case where the data analyst team is unaware that an experiment is stopped.

Additional categorization

Splitting the flags based on other properties could be beneficial for tracking and documenting purposes. While these categories could vary wildly based on business niche and development process, here are a few flags to consider: time-based, geolocation, device-specific, dependency, visibility level, environment, etc.

Variations based on flag type

Release Flags

1. Creation: Often short-lived, these flags may require less documentation than other types.
2. Testing: Primarily tested in staging to ensure the new feature doesn’t break existing functionalities.
3. Monitoring: Less emphasis on user behavior and more on system stability.
4. Decommission: Usually decommissioned quickly once the feature is stable.

Experimentation Flags

1. Creation: Requires detailed documentation outlining the A/B test hypotheses and metrics.
2. Testing: Rigorous testing in both staging and production to ensure that data collection is accurate.
3. Monitoring: Heavy focus on analytics and user behavior metrics.
4. Decommission: This may remain longer in the code for future experiments.
5. Customer support: the CX team should be aware of variations and be able to support the customers if any issues arise.

Operational Flags

1. Creation: Documentation focuses on system parameters and flag activation/deactivation conditions.
2. Testing: Often tested under simulated operational conditions.
3. Monitoring: Emphasis on system performance metrics.
4. Decommission: May remain in the system long-term for operational flexibility.

Permission Flags

1. Creation: Documentation includes user roles/segments and permissions — analysis on dependencies and if user segments overlap.
2. Testing: Requires testing for different user roles and segments.
3. Monitoring: Focus on user access logs and potential security issues.
4. Decommission: It is often long-lived and may only be decommissioned if the feature it controls becomes universally accessible. The customer support team should also be aware of variations and be able to support the customers if any issues have arisen.

Long-lived vs. Short-lived Flags

• Long-lived Flags: These often go through a more rigorous documentation and approval process and may require periodic review.
• Short-lived Flags: These may skip some life cycle stages like long-term monitoring and may be decommissioned more quickly.

User-based vs. System-based Flags

• User-based Flags: More focus on monitoring user behavior and user-specific metrics. For a flag to be user-based, it has to use at least one piece of user context to operate.
• System-based Flags: Emphasis on system performance and stability metrics. System-based flags have a hard requirement not to use any user context.

Summary

Effective management of feature flags requires a disciplined approach through each stage of their life cycle, from creation to decommissioning. Adherence to testing plans, securing necessary approvals, and vigilant monitoring are crucial. Understanding this life cycle helps teams mitigate risks while maximizing the benefits of feature flags, such as agile development and dynamic feature control.

Glossary

• Feature Flag Management System: A tool for managing feature flags’ creation, deployment, and monitoring.
• Default State: The initial state of a feature flag, usually set to ‘off.’
• Rollout Strategy: The method used for gradually releasing a new feature to users.
• Audit Log: A record of changes, often used for compliance and monitoring.
• Key Performance Indicators (KPIs): Metrics used to evaluate the success of a feature or experiment.

This article is part of a series, if you enjoyed it you should check out some of the other parts as well:

• The basics
• Types of feature flags
• Life cycle (you are here)
• Testing feature flags
• Risk mitigation
• Performance and Resilience

Want to learn more about DraftKings’ global Engineering team and culture? Check out our Engineer Spotlights and current openings!