Turn expensive features down before the service falls over

The database is busy and the site is slowing down. Someone asks which features to switch off. If that list only exists in a senior engineer's head, the person on call may not have it.

Login and checkout are often fine on their own. They get into trouble because they share capacity with work that can wait: personalised recommendations, live search suggestions, a "customers also viewed" query, a CSV export that scans a large table. On a quiet day that extra work is fine. During a spike, login and checkout are the paths that have to keep working.

Adding capacity works if you can wait for the new machines. A kill switch works when a feature is misbehaving. A spike is a third case: the features are doing what they should, and together they cost too much. Write down the order for turning them down, so you can turn them back up when the load drops.

One flag, a few levels

Several separate on/off flags means on-call has to remember which combination is safe. One multivariate flag, service-level, can name the levels you are willing to run:

• full: everything on.
• reduced: drop live suggestions, serve cached recommendations, queue exports instead of running them inline.
• essential: login, browse, cart and checkout. Other features show a short "back shortly" message.

Each optional feature can branch on the variant name. You decide what can be dropped while reviewing the code, when there is time to argue about it:

import Featureflow from 'featureflow-node-sdk';

const featureflow = new Featureflow.Client({
  apiKey: process.env.FEATUREFLOW_SERVER_KEY,
  // If flag config can't be fetched, stay on full service
  withFeatures: [new Featureflow.Feature('service-level', 'full').build()],
});

function serviceLevel(user) {
  const level = featureflow.evaluate('service-level', user);
  if (level.is('essential')) return 'essential';
  if (level.is('reduced')) return 'reduced';
  return 'full';
}

async function productPage(req) {
  const level = serviceLevel(req.user.id);
  return {
    product: await getProduct(req.params.id),            // always
    recommendations: level === 'full'
      ? await liveRecommendations(req.user.id)
      : level === 'reduced' ? cachedRecommendations() : [],
  };
}

In that example, reduced always means the same thing, and changing it means a deploy. To let on-call change the mix, put a JSON object on the reduced variant in the dashboard and read it with jsonValue(). The Node SDK has had that since 0.7.0.

// On the "reduced" variant in the dashboard:
// { "liveSuggestions": false, "recommendations": "simple" }
const preset = featureflow.evaluate('service-level', user).jsonValue() ?? {
  liveSuggestions: true,
  recommendations: 'live',
};

if (preset.liveSuggestions) startLiveSuggestions(user);

const recommendations = preset.recommendations === 'simple'
  ? cachedRecommendations()
  : preset.recommendations === 'off'
    ? []
    : await liveRecommendations(user);

On-call can turn live suggestions off, or set recommendations to simple, without shipping code. jsonValue() is undefined when that variant has no JSON, including the full failover, so the default object is the path you get in that case.

Because the level is a targeting rule, you can apply it to part of the traffic. Free-tier users can sit on reduced while paying customers stay on full, or one region can drop a level while the others stay where they are.

Write the levels into the runbook

Put the levels in the incident runbook next to the alert that should trigger them, and flip the flag in staging once so the path is familiar. Register it with full as the failover, so a network blip does not drop the service to essential by accident. In Featureflow, changes to the flag are written to the audit log, so the review afterwards can see when the level changed and when it went back. The docs cover multivariate flags, JSON values on a variant, and pre-registering features.

Choose the features you would drop first and put them behind this flag. The next spike is a change to service-level, which you can reverse when the database recovers.

#FeatureFlags#SRE#IncidentResponse#Resilience#ContinuousDelivery