Object Storage Migration with Feature Flags: Cutover and Rollback Boundaries

Use feature flags to control which application reads reach a new object store, while a separate migration workflow copies and verifies the data. Keep write ownership explicit throughout the rollout. Switching reads back is a useful recovery action only while the original store can still serve the correct data.

Changing an S3 endpoint can look like a small configuration update. In a document portal, that change affects customer downloads, export jobs, signed links, permissions, and files created while the migration is running. A successful copy job does not prove those workflows work through the new backend.

This guide uses a hypothetical document portal to show a staged migration design. The application starts with immutable, completed documents; its existing catalog remains authoritative for ownership, active versions, and deletion state. The rollout controls reads first and treats the eventual transfer of write ownership as a separate decision.

AI-generated illustration. A conceptual migration workflow, not a product screenshot or a measured customer result.

Define what the first rollout is allowed to change

Start with one customer action: downloading a completed document through the application backend. Restrict the initial scope to objects written under unique, immutable keys. A revised document gets a new key, and the catalog selects its active version.

The first flag changes where eligible downloads read their bytes. Uploads, overwrites, deletes, and the catalog's business rules keep their existing owners. A deleted document remains unavailable even if a physical copy is still present in either bucket.

Write a short migration contract before sending production traffic:

The source copy, target verification record, and catalog entry need to refer to the same immutable object. An object count or a matching filename alone is insufficient. Validate content integrity with an appropriate checksum, as well as the metadata and response behavior the application needs. Amazon S3's object integrity documentation describes checksum validation; confirm the equivalent operations supported by the target and the copy tool you actually use.

For mutable keys, build a version-aware consistency design before applying this pattern. A copy verified yesterday cannot establish that today's overwritten object is ready.

Put the flag at the application read boundary

Deploy both storage adapters behind the application's existing download boundary. In this document portal, RustFS can serve as the target object store for migrated document files: the candidate adapter fetches their bytes, while the existing catalog still controls ownership, active versions, and deletion state. RustFS is an open-source, self-hostable object store with an S3-compatible API, giving the target adapter a familiar bucket-and-object interface for document reads.

RustFS's on-demand migration walkthrough shows how it can populate a target bucket by pulling missing objects from an external source when they are read, or by running a background backfill. For this guide's verified-read policy, use backfill to prepare the target copies, verify their contents, and then mark those exact objects as ready in the catalog before directing customer reads to them. That separates the two jobs: RustFS provides the destination and a way to bring objects into it; FeatBit controls which eligible tenants may read the verified copies. The pull mechanism does not continuously track later source changes or send target writes back to the source, which is why the immutable-key scope and explicit write-cutover plan still matter.

Authorize the request and resolve the document in the catalog before selecting a backend. Keep credentials, endpoint selection, and tenant context on the server. A browser parameter must never determine the storage endpoint or grant access to another tenant's object.

Create a Boolean flag such as documents-read-from-target in FeatBit:

• false selects the source read path; use this for the OFF variation and the application's initial fallback.
• true permits the target read path for an assigned tenant, subject to object readiness.
• An internal targeting rule admits test tenants first. A later rule admits a percentage of eligible production tenants.

Follow the SDK connection guide to initialize a server SDK and supply trusted context. Evaluate the flag once at the request boundary, then pass the result to the read planner. A background job should likewise use one decision for its defined unit of work, rather than changing storage policy halfway through a download or batch.

AI-generated illustration. The flag selects a read policy; application code enforces object readiness, and a separate worker copies and verifies objects. Writes still go to the source.

The following server-side TypeScript uses the official FeatBit Node.js SDK. Install it in the backend:

npm install @featbit/node-server-sdk

Set FEATBIT_SDK_KEY, FEATBIT_STREAMING_URI, and FEATBIT_EVENTS_URI from your environment's SDK configuration. Load them into the server process before importing the module below. Keep the environment secret on the server.

Initialize one SDK client per server process

Create a shared featbit.ts module. The SDK synchronizes flag configuration and evaluates against the data it has received; creating a client for every download would create unnecessary connections.

// featbit.ts
import { FbClientBuilder } from "@featbit/node-server-sdk";

function requiredEnv(name: string): string {
  const value = process.env[name];
  if (!value) throw new Error(`[FeatBit] Missing ${name}`);
  return value;
}

export const fbClient = new FbClientBuilder()
  .sdkKey(requiredEnv("FEATBIT_SDK_KEY"))
  .streamingUri(requiredEnv("FEATBIT_STREAMING_URI"))
  .eventsUri(requiredEnv("FEATBIT_EVENTS_URI"))
  .startWaitTime(2_000)
  .logLevel("warn")
  .build();

// Share this initialization attempt across requests.
export const flagsReady = fbClient.waitForInitialization().catch(
  (error: unknown) => {
    console.error("[storage-migration] FeatBit initialization failed", error);
  },
);

Missing configuration fails at startup. If the configured SDK cannot obtain initial flag data within this example's two-second wait, the error is logged and the evaluation below can return its false default. A later connection outage can leave the SDK evaluating cached configuration, so an outage does not itself guarantee that every request switches back to the source. Reuse the client for the process lifetime and call fbClient.close() during graceful shutdown after draining work.

Evaluate the flag once for an authorized download

The backend first authenticates the caller, authorizes the document, and loads its catalog entry. Pass that trusted user key and document to planDocumentRead once per download. In FeatBit, set the rollout's Dispatch by attribute to tenantId; the SDK user still identifies the caller, while that custom attribute groups colleagues into the same tenant allocation.

// document-storage.ts
import { UserBuilder } from "@featbit/node-server-sdk";
import { fbClient, flagsReady } from "./featbit.js";

const flagKey = "documents-read-from-target";

type AuthorizedDocument = {
  tenantId: string;
  objectKey: string; // Immutable key resolved after permission/state checks.
  targetVerified: boolean; // Verification belongs to this exact object.
};

export async function planDocumentRead(
  document: AuthorizedDocument,
  authenticatedUserKey: string,
  migrationRevision: string,
) {
  const user = new UserBuilder(authenticatedUserKey)
    .custom("tenantId", document.tenantId)
    .build();

  await flagsReady;

  let candidateAssigned = false;
  try {
    candidateAssigned = await fbClient.boolVariation(flagKey, user, false);
  } catch (error) {
    console.error("[storage-migration] Flag evaluation failed", {
      flagKey,
      tenantId: document.tenantId,
      error,
    });
  }

  const backend =
    candidateAssigned && document.targetVerified
      ? "target"
      : "source";

  return {
    flagKey,
    tenantId: document.tenantId,
    objectKey: document.objectKey,
    backend,
    candidateAssigned,
    migrationRevision,
    reason: !candidateAssigned
      ? "source-policy"
      : document.targetVerified
        ? "verified-target"
        : "target-not-ready",
  };
}

FbClientBuilder, UserBuilder, waitForInitialization(), and boolVariation() are SDK APIs. The catalog's targetVerified field, migration revision, and returned read plan belong to the application. The SDK selects the tenant's policy; it does not verify or copy the object.

Await the plan, then use its backend to execute the read: target uses the adapter configured for RustFS, and source uses the existing object store. Catch and log adapter errors with request, tenant, migration revision, and selected-backend context, then preserve the application's error contract. Track completion of the response body as well as the initial storage response: a download can fail after its headers arrive.

For a React or browser client, keep this storage decision in the backend and return the authorized download response or signed URL. The frontend consumes that result; it does not need a second flag evaluation to choose a storage endpoint.

For large documents, stream the body using the application's existing limits and cancellation handling. Preserve required headers, range reads, and content disposition. If the application normally issues presigned URLs, make the routing decision before signing; an already issued URL continues to address its original endpoint until it expires. CDN entries and in-flight transfers also need their own recovery handling.

Keep tenant assignment stable and object readiness separate

For a shared document portal, assign the storage policy by tenant so colleagues do not receive different rollout policies simply because they have different user accounts. FeatBit's percentage rollouts support dispatching by a chosen user attribute. Supply a trusted tenantId attribute and use it consistently across requests and workers.

Keep eligibility rules, variation ordering, and the dispatch attribute stable while increasing exposure. Record those settings so a recovery can restore the intended policy. Use targeting rules to exclude unsupported tenants and operations before percentage allocation.

Tenant assignment answers whether a tenant may use the candidate path. Object readiness answers whether a particular immutable object can use it. A tenant assigned to the candidate can still read a recently created, unverified document from the source. Log both facts; otherwise a nominal 10% tenant rollout may appear healthy while almost every download still uses the original backend.

A percentage of tenants also does not equal the same percentage of requests or bytes. One tenant may account for most downloads. Review tenant count, traffic volume, object sizes, and target capacity together when choosing the next stage.

Expand from a working download to a working business flow

Begin with a copied test set and internal tenants. Include representative file sizes, Unicode keys, metadata, partial downloads, and permission cases. Exercise export workers and other callers that bypass the web page.

Use a sequence such as the one below. The percentages are illustrative allocations of eligible tenants, not safety guarantees or a fixed schedule.

Before each increase, the release owner reviews evidence and records a continue, hold, or return-to-source decision. Choose an observation window that includes enough relevant downloads and the workloads that matter, such as a scheduled export. A quiet five minutes does not validate a nightly job.

AI-generated illustration. Read exposure expands after evidence review. Write cutover has a separate readiness gate; the illustrated stages are a proposed workflow.

Before canary traffic starts, agree on the conditions that stop expansion:

Set thresholds against the application's service objectives and representative source behavior. Keep correctness failures separate from availability averages. Compare like workloads; a target serving only small cached files cannot establish performance for large exports.

Record the flag key, evaluated variation, assignment unit, actual backend, routing reason, and migration revision with the relevant telemetry. Avoid logging document contents, credentials, or signed URLs. FeatBit controls exposure; the application, copy workers, and monitoring system produce the storage and business signals. The progressive rollout patterns guide provides more context for evidence-based expansion.

Rehearse returning reads to the source

While the source still owns writes and retains the required objects, disabling candidate reads can return subsequent application reads to the source. Verify propagation across service instances, including SDK instances using cached configuration. A flag update does not cancel an active transfer, replace an issued signed URL, or flush a CDN.

Rehearse recovery before inviting production tenants:

1. Record the current rules, dispatch attribute, variation allocation, and migration revision.
2. Disable candidate reads and confirm new requests select the source on every relevant service.
3. Download known objects, recently uploaded documents, and representative large files through the normal customer path.
4. Check that deleted documents remain unavailable and tenant permissions still hold.
5. Verify download completion, error rate, and response time after the change.
6. Keep copy workers and cleanup jobs under their own explicit controls; switching reads does not stop or reverse them.

Keep the SDK fallback aligned with this phase. During the read migration, false is appropriate because the source remains valid. Before changing write ownership or retiring the source, revise that assumption deliberately and deploy the corresponding application behavior.

An application may also retry a failed target read against the source when both represent the same authorized, active immutable object and no response bytes have been sent. Treat this as an explicit recovery policy with its own metrics. An automatic retry on every error can hide a broken candidate or amplify an outage. In particular, do not interpret an access-denied response as proof that a key is missing: Amazon S3 can return 403 for a missing object when the caller lacks bucket-listing permission, as documented in GetObject permissions.

Treat write cutover as a new consistency boundary

Once the target accepts authoritative writes that the source does not receive, returning reads to the source may hide new documents or expose stale state. A feature flag cannot reconstruct those missing writes.

Before transferring write ownership, choose and rehearse the actual data protocol. Depending on the application, that may involve a controlled write pause and final reconciliation, or a durable change log with idempotent replay. If you use dual writes, specify which side owns truth, how partial failures are retried, how ordering and deletes are preserved, and what proves convergence. Two successful SDK calls are not a cross-store transaction.

Fence the old writers before enabling the new owner. Account for service instances with stale configuration, scheduled jobs, upload URLs already issued to clients, incomplete multipart uploads, and external integrations. A rollout flag can coordinate intended behavior, but it is not a distributed lock that prevents stale writers from accepting data.

Document the new recovery boundary explicitly: either the source remains current through a tested synchronization mechanism, or recovery requires reconciliation before reads can return there. Also validate listings, overwrites, deletes, version behavior, and other operations beyond the initial immutable-download scope before declaring the whole storage migration complete.

Finish the migration and retire its controls deliberately

Keep an owner for the migration flag, the readiness records, both storage adapters, and source retention. Record when recovery to the source stops being supported and what takes its place. Confirm that all application and background callers have moved before removing the old credentials and data according to the retention plan.

Once the target is the permanent path, follow the feature flag cleanup workflow: deploy code that no longer evaluates the temporary migration flag, verify production behavior, then archive the flag. Preserve the rollout decisions and data reconciliation evidence with the migration record.

To implement the release-control side, start with a server SDK and a tenant-level targeting rule. Teams that want to operate that control plane in their own infrastructure can evaluate self-hosted FeatBit. The first milestone is concrete: one verified document, one included tenant, one excluded tenant, and a demonstrated return to the source before expanding.

Image disclosure: The cover and both illustrations in this article were generated with AI. They explain the proposed application and data workflows; they are not product screenshots or evidence of measured performance.