A VTGate fleet can present thousands of MySQL tablets through a MySQL-compatible endpoint, but it cannot hide a poor shard key, make cross-shard work free, or turn an untested traffic switch into a safe migration. The abstraction succeeds only when the application’s transaction boundaries match the data placement model.

Situation

MySQL often scales farther on one primary than an early architecture forecast suggests. Better indexes, bounded queries, read replicas, connection control, caching, table partitioning, and larger instances can delay horizontal partitioning. That is useful: sharding adds a distributed routing layer, more failure domains, and permanent constraints on transactions and schema design.

Eventually, some workloads do need more write throughput, storage, maintenance isolation, or tenant-level blast-radius control than one MySQL replication group can provide. Manual application sharding can meet that requirement, but it spreads placement logic across connection pools, services, migrations, background jobs, and operational tooling. Every reshard then becomes an application release.

Vitess centralizes much of that machinery. Applications connect to VTGate. A VSchema describes how logical tables map to keyspaces and shards. VTGate plans and routes queries to VTTablet processes, and each VTTablet mediates access to a MySQL instance. VReplication moves data for workflows such as resharding and table migration. A topology service stores control-plane metadata used by the Vitess components.

This is a mature architecture, not an infinitely scalable MySQL server. It changes which constraints the organization must manage.

The Problem

The first mistake is to choose Vitess because a database is “large.” Size is not a failure mode. A platform needs evidence that a specific resource or operation has exceeded an acceptable envelope: primary write CPU, buffer-pool miss cost, redo or binlog throughput, recovery time, schema-change duration, backup time, storage headroom, noisy-neighbor isolation, or inability to meet a maintenance window.

The second mistake is to shard before understanding access paths. A lexical split such as customers A–M and N–Z usually produces skew and makes placement depend on mutable business data. A hash-like vindex distributes rows more evenly but generally sacrifices range locality of the original key. A tenant-based key can preserve transaction locality but allows one large tenant to dominate a shard. No key simultaneously guarantees uniform load, local joins, ordered range scans, and easy tenant movement.

The third mistake is to treat the routing tier as transparent. VTGate can route a query directly when the predicate resolves through a unique vindex. Without that routing value, it may scatter the query to multiple shards. Fan-out increases total work, tail latency, memory pressure, and the number of partial failures a request can encounter. A query that is acceptable against four shards may become an incident against hundreds.

The decision question is therefore: can the system choose a stable shard key that localizes its dominant transactions, and can it operate the exceptions without allowing scatter, distributed commits, or resharding workflows to consume the fleet?

Design the Contract Before Deploying Vitess

The core architecture separates query serving from topology and workflow management, but every layer needs an explicit availability contract.

flowchart TD
    APP["Application services"] --> LB["Regional load balancer"]
    LB --> G1["VTGate — cell A"]
    LB --> G2["VTGate — cell B"]
    VS["VSchema and topology metadata"] -.-> G1
    VS -.-> G2
    G1 --> T1["VTTablet — shard 10"]
    G1 --> T2["VTTablet — shard 20"]
    G2 --> T1
    G2 --> T2
    T1 --> M1["MySQL primary and replicas"]
    T2 --> M2["MySQL primary and replicas"]
    VR["VReplication workflow"] --> T1
    VR --> T2
    OBS["Metrics and query logs"] --> G1
    OBS --> G2
    OBS --> T1
    OBS --> T2

Start with a query-to-placement matrix

Inventory the operations that define correctness and load, not every SQL statement ever issued. For each operation, record:

OperationRouting value availableExpected shard countTransaction boundaryFailure behavior
Read tenant profileYesOneRead onlyRetry on another VTGate
Create order and linesYesOne when co-locatedOne local transactionRoll back locally
Update two unrelated tenantsTwo valuesTwoAvoid or use explicit distributed policyAmbiguous result must be resolved
Global aggregateNoAllAnalyticalMove to an analytical system or bound fan-out
Lookup by emailIndirectOne after lookupRead path plus lookup maintenanceVerify lookup consistency

This table is the architecture. If a critical operation has no routing value, either redesign the data model, introduce a lookup vindex, accept bounded scatter, or keep that workload outside the sharded keyspace. Lookup vindexes add a maintained lookup table and write-path work. Owned and unowned variants assign different maintenance responsibility; consistent_lookup* variants add locking and transaction semantics to protect lookup consistency under concurrent writes.

Vitess keyspaces are logical database namespaces. A primary vindex maps one or more row columns to a keyspace ID, and each shard owns a keyspace-ID range. Secondary vindexes support additional routing paths. The mapping must be stable. Changing a primary vindex is a data-placement migration, not an index rebuild.

Co-locate rows that must commit together. If orders, order lines, and payment state normally share a tenant or account boundary, carry that routing key through each table and request. Enforce the invariant in schema review and application APIs. A global identifier may still be useful, but uniqueness and placement are separate concerns.

Budget scatter as fleet-wide work

Scatter-gather is not automatically wrong. Low-frequency administrative queries over a small shard count may be acceptable. The risk comes from multiplying per-shard work by concurrency and shard count.

Define controls before production:

  • a maximum expected shard fan-out by query class;
  • execution and row limits that fail closed before a proxy accumulates unbounded results;
  • workload isolation for administrative or batch traffic;
  • per-VTGate memory and concurrency budgets;
  • query-plan and route-type telemetry;
  • an analytical path for unbounded aggregates, wide joins, and scans.

Monitor total shard queries, not only application QPS. One request scattered to 128 shards creates at least 128 shard-targeted operations and can amplify a modest retry storm into a fleet event.

Make distributed transactions exceptional

Vitess supports multi-shard transactions. In MULTI mode, atomicity is not guaranteed if a later shard commit fails after an earlier one succeeds. In TWOPC mode, Vitess uses a metadata manager and prepare phase to provide atomic commit across participants. The Vitess 24.0 guarantee requires semi-synchronous replication to be enabled; it does not add cross-shard isolation, so readers can observe a fractured state while participants commit at different times.

Two-phase commit does not mean “if one shard crashes, every already committed shard is rolled back.” A committed participant cannot generally be undone by protocol magic. Vitess records metadata and normally resolves prepared participants toward one decision, but its documentation exposes critical failure counters for cases where atomicity has failed. Abandoned transactions can require DTID-level inspection, manual repair, and Conclude after the data is reconciled. That adds durable metadata, locks, failure recovery, operational inspection, and longer tail latency.

Use two-phase commit only for a small, measured set of invariants that cannot be modeled within one shard or implemented as an idempotent workflow. Configure and monitor --twopc-abandon-age; expose Unresolved, CommitUnresolved, and the critical failure counters; document the DTID inspection, repair, and Conclude path; and test coordinator and participant failures at each phase. Application transaction deadlines remain a separate control. If cross-shard transactions are common, the shard key is probably misaligned with the domain.

Treat topology as a control plane with cached consumers

Vitess uses a global topology and per-cell topology data. VTGate and VTTablet consume topology information, while management operations update it. A topology-service outage is serious, but it should not be described as an automatic global database outage. Serving components cache information and use resilient watches; the effect depends on which topology data is unavailable, whether serving state must change, and which cell is affected.

Separate the tests:

  • can existing single-shard reads and writes continue with the local topology unavailable;
  • can a new VTGate or VTTablet start;
  • can a shard primary fail over;
  • can a VSchema or serving-graph change propagate;
  • can an operator run resharding or emergency commands;
  • does one cell’s topology failure remain cell-scoped.

Run the topology implementation as its own consensus system with supported durability, backup, quorum, latency, and upgrade procedures. Do not prescribe local NVMe or a fixed availability-zone layout without validating the chosen implementation’s support model and correlated-failure assumptions.

Reshard in observable, reversible phases

Vitess resharding is a workflow, not an instantaneous map update. VReplication copies source data and then streams ongoing changes. Operators validate the destination, switch traffic, observe the result, and only later complete the workflow and remove old resources.

A safe production plan has gates:

  1. Confirm source health, destination capacity, schema compatibility, and replication prerequisites.
  2. Start the workflow and monitor copy progress, stream errors, lag, and source overhead.
  3. Validate row counts and domain-specific invariants; counts alone do not prove equivalence.
  4. Use SwitchTraffic for a bounded tablet type first when the rollout calls for staged reads; by default the command can switch all tablet types, so the runbook must pass the intended scope explicitly.
  5. Before switching primary traffic, verify the allowed lag and timeout. The workflow makes sources read-only, waits for targets to catch up, and can pause or fail writes while the cutover completes.
  6. Keep reverse VReplication enabled, as it is by default, and exercise ReverseTraffic as the post-switch rollback path.
  7. Hold at the new state while monitoring application and database signals.
  8. Run Complete only after the rollback window closes and backups are verified; its default cleanup removes old source shards from topology and drops source tables.

Traffic-switch semantics depend on the Vitess version and workflow. In Vitess 24.0, Cancel is available only before any traffic has switched. After switching, ReverseTraffic is the rollback mechanism while the reverse stream and source resources remain. Complete is the destructive boundary that removes that ordinary rollback path. The runbook must use the exact version’s commands, preserve reverse replication, and name completion—not write switching—as the final cleanup decision.

In Practice

Slack’s engineering account documents why it adopted Vitess and how it approached migration. The useful lesson is not a claim that a particular Slack outage was caused by VTGate; the publicly documented article does not establish that story. The defensible pattern is that a large existing application treated Vitess adoption as a staged datastore migration, not as a proxy installation.

That distinction changes the work. Compatibility has to be measured against the application’s real SQL, transaction patterns, schema conventions, and operational tools. Traffic movement needs incremental validation and a rollback design. The organization must learn to operate both the source and destination during the migration window.

The documented Vitess architecture produces a practical readiness sequence:

  1. Prove the current bottleneck. Capture resource saturation, maintenance risk, and growth projections. Compare sharding with vertical growth, archival, workload separation, and managed alternatives.
  2. Replay representative SQL. Classify direct, lookup, scatter, unsupported, and multi-shard operations. Include background jobs and incident tooling, not only API traffic.
  3. Select and test vindexes. Measure distribution with current data and plausible future tenants. Inspect the largest keys and hottest write keys separately from average distribution.
  4. Build a shadow keyspace. Exercise schema deployment, backups, point-in-time recovery, tablet failover, topology loss, and VTGate scaling before carrying production writes.
  5. Move one bounded domain. Use VReplication workflows and explicit validation gates. Keep ownership and rollback criteria clear.
  6. Re-measure after every shard split. More shards change fan-out cost, connection counts, buffer efficiency, operational cardinality, and topology churn.

The proof of readiness is an evidence packet: query classification, vindex distribution, load test, failure-injection results, restore result, migration reconciliation, and rollback timing. Vendor adoption lists are not evidence that a specific data model is ready.

Observe routing and data planes together

A useful dashboard joins application, VTGate, VTTablet, MySQL, VReplication, and topology signals:

  • query rate and latency by keyspace, tablet type, route type, and error code;
  • shard fan-out distribution and rows returned;
  • VTGate memory, CPU, connection pressure, retries, and rejected queries;
  • VTTablet query latency, pool saturation, health state, and MySQL availability;
  • per-shard QPS, write rate, storage, replication lag, and hot-key concentration;
  • VReplication copy state, stream lag, errors, and validation results;
  • topology watch failures, stale state, quorum health, and cell scope;
  • unresolved distributed transactions and recovery age.

Page on violated service objectives, not component existence. A VTGate restart behind a healthy load balancer may be noise. Rising scatter fan-out combined with proxy memory and tablet-pool saturation is an architectural incident.

Where It Breaks

Failure modeUnsafe assumptionConsequenceControl
Hot shard or hot keyHashing guarantees balanced loadOne shard sets fleet latency and capacityDistribution tests, hot-key metrics, tenant isolation strategy
Missing routing predicateVTGate makes every query localScatter multiplies downstream workQuery policy, fan-out limits, analytical offload
Cross-shard transaction growthTwo-phase commit behaves like local MySQLLock duration, recovery work, and tail latency increaseCo-location, explicit transaction mode, unresolved-transaction runbook
VTGate fleet overloadStateless means capacity is unlimitedRouting becomes the shared bottleneckIndependent replicas, load shedding, memory and concurrency budgets
Topology impairmentEvery loss either has no effect or stops everythingStartup, failover, management, or serving changes fail differentlyCell-scoped drills, cache behavior tests, quorum operations
VSchema errorMetadata changes are harmlessQueries misroute or begin scatteringVersioned review, staged rollout, route-plan comparison
Reshard validation gapCopy completion proves correctnessMissing or inconsistent domain dataChecksums plus business-invariant reconciliation
Premature workflow cleanupTraffic switch is instantly irreversible or risk-freeRollback path disappears before confidence existsDefined hold period and last-reversible-point runbook
MySQL failure inside a shardVitess eliminates database HA workOne shard loses writes or exceeds RPOPer-shard replication, backup, failover, and restore testing
Fleet-wide schema changeOnline DDL removes all riskLong-running migrations contend across many shardsConcurrency waves, throttling, pause criteria, verification

Vitess is a poor fit when the workload depends on frequent unbounded cross-tenant joins, global serializable transactions, ad hoc analytics over the serving fleet, or a team that cannot staff the topology, workflow, routing, and MySQL operational layers. Sharding can increase the ceiling, but it also increases the number of systems that must be correct during recovery.

What to Do Next

  • Problem: The team is proposing Vitess without a measured single-primary limit. Solution: Define the saturated resource and compare lower-complexity remedies. Proof: Produce capacity and maintenance evidence under representative load. Action: Do not approve sharding from database size alone.
  • Problem: The shard key is selected from entity names rather than transaction boundaries. Solution: Build the query-to-placement matrix and test candidate vindexes against real skew. Proof: Show the percentage of critical operations routed to one shard and the largest-key distribution. Action: Make routing-key presence part of API and schema review.
  • Problem: Scatter and distributed commits are treated as transparent compatibility features. Solution: Give them explicit budgets, policies, and failure runbooks. Proof: Load-test fan-out and inject failure into every two-phase-commit stage. Action: Block production migration until exceptional paths are observable and bounded.
  • Problem: Resharding success is defined as a completed command. Solution: Gate copy, validation, read switching, write switching, rollback, and cleanup separately. Proof: Reconcile domain invariants and execute a timed rollback rehearsal. Action: Preserve the reverse path until the agreed confidence window closes.

Sources