Featureflow

Got something big to release? Jittery knees? Not sure how the system or the public will react? Gradual rollouts can be your friend and they can be easier than you'd expect.

Risk Reduction

If there is one thing that can kill a company's reputation, that's a poor release, either functionally or worse with a killer defect. The bigger the user base the greater the reaction. If you've worked on enterprise scale releases you'll know how even a subtle change to a colour or piece of functionality can cause an immediate flurry of calls to your poor old call centre.

The solution to this is to employ a Canary release approach. The concept is simple: new features are released to a small subset of users before rolling out to the entire customer base. This gives you the ability to gauge many things first:

Customer reactions
Server load and issues
The effect on customer behaviour such as conversion rates of visitors to purchasers

Often this approach is associated with a blue-green deployment which involves a rolling deployment of a new service – using a load balancer to redirect traffic. So let's discuss the possible approaches to getting this done.

Basic Blue-Green Deployment

Managing a gradual release in a basic blue-green fashion often involves a reverse proxy or an API manager in front of your services. Let's break down the options:

API Manager

An API manager often offers a lot of additional functionality and comes at the expense of cost and complexity. Tools such as Apigee can run into the tens of thousands a month quite quickly and add yet another network layer on top. They can however manage migrating gradually from one service to another really well. Apigee will use rules to direct traffic to one IP address or another. It's a broad stroke that helps migrate users to new physical servers – of course all features in the newly released service will be available to those customers.

Reverse Proxy

If you don't have the need for an API Manager then a simple reverse proxy is more than capable of managing a quick switchover. Tools such as nginx and HAProxy are all very capable, even managing sessions as they roll over if you must still use sessions. At Featureflow we use HAProxy combined with Rancher. They are both free, very solid and dependable and offer a seamless rollover experience.

We also don't have to keep double resources permanently. New containers are spun up, checked – HAProxy is switched to the new backends and the previous containers are held in reserve until we agree to roll forward – at which point the old containers are disposed. A rollback is a case of reinstating the old container. That's great but not very gradual. So how do we make that gradual?

Blue-Green Dark Release

In doing this you have done something wonderful, you have successfully separated the code release from the feature rollout

Dark blue-green release diagram showing deployment with features toggled off — A dark blue-green release – switch to the new service but with features toggled off

The solution is to dim the lights and perform a dark blue-green release. How it works is this: you deploy your new server/service with the new features turned off. You should still migrate the new service using a reverse proxy switch over but when you get there the customer should see no change (or very little).

In doing this you have done something wonderful – you have successfully separated the code release from the feature rollout. What this gives you is a number of benefits:

You have mitigated as much risk as possible by doing as much as possible up front

You have given yourself more precise control over the subsequent feature rollout

So now you have switched over, nothing has happened. Can we go to the pub yet? Well nearly.

Moving the control of the feature rollout to the service itself gives you ultimate control. It's not just a gradual rollout from one API or one service to another, you can now enable just a single feature, or a combination of features within that one service. You may wish to try a combination of variants and target specific user types for each to assess the impact.

Enabling a single feature for a distinct subset of users — Now you can enable a specific feature within your new service for a specific subset of users

Using Featureflow for Dark Releases

Using Featureflow you can do this simply and safely by having your developer wrap your new feature in a line of code and then controlling it in our control panel. No API Managers required – in addition you can run dozens of scenarios. Doing that with an API manager alone you might end up spinning up a new service for each A/B test and each combination of gradual rollout test you wanted to try – you could end up with dozens of servers or disparate rules.

So now you can gradually turn on 'feature A' in isolation – using Featureflow you can simply drag a slider to increase the rollout rate to, say 10% – combine that with a role of 'beta users' or just a list of specific user logins – your production testers perhaps.

Complex rollout scenario with multiple targeting rules — A rather complex rollout scenario – you get the idea

Additional Benefits of Feature Controls

The business team can turn on / ramp up the feature any time after the release without the engineering team getting involved
You can ramp down or "virtual rollback" an individual feature that may be misbehaving without having to rollback the entire service
You can manage not just API migrations, but internal behaviour migrations – want to add an extra item to the 'gold' package? It doesn't change the API but you can roll that out gradually too

Summary

There are a few components to mitigating risk and providing a seamless upgrade experience to your end users:

1. A reverse proxy or API manager

Depending on the complexity of your solution and budget. A reverse proxy is perfectly suitable if managed well. Other gateways such as Netflix Zuul are highly worthy of managing switching to new code deployments and managing service migrations.

2. Dark Blue-Green release

Gives you ultimate safety and control. Switching seamlessly to a new software release without the new features enabled allows the development team to deploy at any time prior to actually releasing the features. It ensures that as much work has been done up front as possible.

3. Feature controls

Adding feature controls to your code allows you to manage a gradual or 'canary' rollout at the individual feature level. This allows you to not just gradually rollout the whole codebase or endpoint but target new features individually to specific subsets of users (such as prod testers, beta users or '10% of all users') and at different times (tonight, next week when the marketing team are ready) without having to manage multiple deployments or involve engineering teams.

Reducing risk with dark blue-green releases