Kafka Ordering and Version Gates: Keeping Order Status Consistent

What will you learn?

Stop a late Kafka event from rewinding a customer's order. Run the same deliveries through an unsafe consumer and a guarded consumer, and see why a stable order key, a version check, and recovery for missing events each solve a different part of the problem.

Java 21 · Spring Boot 3 · Apache Kafka · event-driven commerce

The customer incident

A shopper pays for a laptop. Payment succeeds, checkout saves the confirmed order, and the shopper sees the confirmation page. Later, the parcel is dispatched and the shopper receives a shipment notification.

They open My Orders: the laptop says Shipped. After a refresh, it changes back to Paid — preparing your order. Has the shipment been cancelled? Is the parcel still coming? Support sees conflicting information and cannot confidently explain the order's progress.

The payment did not fail, and the parcel did not stop moving. The error is in the separate database used by the tracking screen. This database is an order-status projection: a read model updated from events. A delayed notification of an already successful payment overwrote a newer shipping status there.

This example models a separate tracking read model. It does not claim that every checkout uses this architecture. The lab does not charge a card or dispatch a parcel; those operations need their own payment, fulfillment, and idempotency checks.

This is an illustrative incident, not a claim about an employer or a measured production outage.

Two paths: checkout confirmation and tracking updates

Checkout path: payment succeeds → the order service records the confirmed order → checkout returns confirmation to the shopper.

Tracking path: the order service publishes events → a background Kafka consumer processes them → the consumer updates the My Orders database.

The confirmation page uses the checkout result in this example. It does not wait for this separate tracking consumer. That is why checkout can succeed while tracking updates are delayed or incorrect.

The word delayed refers to processing the payment event, not to taking the customer's payment.

These are example steps, not measured timings. The business completed its actions in the right order; the tracking consumer processed their notifications in the wrong order.

Terms used in this lesson

• Event: a message describing something that already happened, such as OrderPaid.
• Consumer: the application that reads and processes those messages.
• Order version: a sequence number assigned by the service that owns the order. Here, v1 is Created, v2 is Paid, and v3 is Shipped.
• Kafka offset: a record's position within one topic partition. It is not the order's business version.
• Version gate: a check that accepts only the next expected order version, while safely handling older or missing versions.
• Reconciliation: checking the authoritative order or event history to resolve a missing or conflicting update.

Follow the failure before looking at the fix

1. The order owner records CREATED v1 → PAID v2 → SHIPPED v3 in business order. A version is the order's sequence number, not its Kafka offset.
2. The tracking consumer temporarily fails to process PAID v2 and forwards that event to a retry topic. Processing on the main topic continues, so the consumer later encounters SHIPPED v3 before the payment-event retry succeeds.
3. An unsafe consumer blindly writes SHIPPED v3 to My Orders.
4. The retry later delivers PAID v2. The unsafe consumer writes PAID v2, rewinding the screen even though the package is still shipped.

Kafka has not shuffled offsets in a partition. The application allowed a retry path to overtake the original processing sequence. Non-blocking retry topics can break processing order; see the Spring Kafka retry explanation.

Try it: open the side-by-side simulation, choose Delayed payment event, and press Next event. Predict the shopper's status before each step. Both consumers receive exactly the same events; only their decision rules differ. Checkout has already succeeded when the lab starts.

What the solution changes for the customer

The protected consumer saves SHIPPED v3 in a waiting queue because its tracking database still needs PAID v2. My Orders keeps its last validated tracking status and indicates that updates are being checked. This does not reverse the successful checkout confirmation.

When the PAID v2 event arrives, the consumer applies it. A recovery worker then checks the waiting SHIPPED v3 event and applies it because payment is now recorded in this read model. A later replay of PAID v2 cannot move the tracking screen backwards.

The trade-off: a temporarily stale but explainable status instead of an immediate, contradictory update. A missing event must trigger reconciliation and monitoring; waiting forever is not a solution.

Why choosing an order key is necessary, but not sufficient

Another way to create the same symptom is inconsistent keys: one producer keys OrderPaid by customerId, while another keys OrderShipped by shipmentId. Events for the same order can then land in different partitions. Kafka preserves order inside one partition, not across a topic or between topics. Different consumers may also process records concurrently or observe events after retry/replay. Even using a consistent key does not magically impose a business-wide ordering across independent producers; the state machine must enforce valid transitions.

The naïve approach

kafkaTemplate.send("order-events", UUID.randomUUID().toString(), event);

A random key can scatter events for one order across partitions. Separately, a consumer that blindly overwrites status can apply an older event after a newer one. In this example, that corrupts the tracking view. If downstream applications use the same view to trigger business actions, the consequences can extend beyond an incorrect screen.

Design: order-scoped ordering + state-machine guard

1. Establish a canonical aggregate ID (orderId) used as the Kafka message key for every event belonging to that order in the same topic.
2. Make the service that owns order state the single logical authority for producing ordered state transitions, ideally via a transactional outbox written with the order database transaction.
3. Configure topic partitions and consumer group so each partition is assigned to only one active consumer in that group. Within each consumer, process records for a partition serially, or use key-aware execution that preserves per-key order.
4. Carry eventId, orderId, version, eventType, occurredAt, and schemaVersion in the envelope. occurredAt is observability metadata, not an ordering guarantee.
5. In the consumer, persist the last applied version and verify the business transition atomically. Ignore duplicate events, defer or reconcile future-version gaps, and reject impossible transitions (CREATED → SHIPPED). Do not automatically discard a missing-version event: it might be delayed.
6. Use retries and a DLQ for poison messages, but recognize that skipping a failed record may allow later records to be processed; maintain per-order reconciliation and alerting.
7. Track lag, gap age, transition rejections, duplicate counts and outbox backlog. Test with consumer crashes, rebalance, retries, replay and partition-count changes.

Translate the guard into a decision

A business-state guard is separate from a sequence check: SHIPPED v3 → PAID v4 is still invalid in this simplified lifecycle even though 4 follows 3. Cancellation, refunds, and partial shipments need explicit states in a real model.

For one order, use this simplified path:

Order owner + transactional outbox (issues v1, v2, v3)
                     ↓ key = orderId
Kafka main topic ───────────→ consumer
        └─ failed v2 → retry ─────┘
                               ↓
               atomic version + transition check
                 ↙                         ↘
      durable waiting queue          My Orders projection
                 └─ recover missing event → recheck

This is an illustrative design. It does not describe a deployed client system. The browser lab's waiting queue is in memory; production must persist held events before acknowledging them.

Java 21 / Spring Boot 3: keyed publishing

record OrderEvent(UUID eventId, UUID orderId, long version,
                  String type, Instant occurredAt, int schemaVersion) {}

// Use this from an outbox relay in production; do not treat this send as
// atomic with an unrelated database transaction.
public CompletableFuture<SendResult<String, OrderEvent>> publish(OrderEvent event) {
    return kafkaTemplate.send("order-events", event.orderId().toString(), event);
}

Important: KafkaTemplate.send does not atomically commit the order row and Kafka message. A transactional outbox closes that dual-write gap, with at-least-once relay delivery and idempotent consumers.

Database guard (PostgreSQL illustrative)

CREATE TABLE order_projection (
 order_id uuid PRIMARY KEY,
 version bigint NOT NULL,
 status text NOT NULL CHECK (status IN ('CREATED','PAID','SHIPPED'))
);
-- The consumer must execute version and transition checks in ONE transaction.
UPDATE order_projection
SET status='PAID', version=2
WHERE order_id = :order_id AND version=1 AND status='CREATED';
-- Require exactly one updated row; otherwise classify duplicate, gap or conflict.

For SHIPPED, require the previous persisted state to be PAID, not merely that the incoming event says it is shipped. Real deployments should define cancellation, refund and partial-shipment transitions explicitly.

What ordering guarantees do not mean

• Kafka partition order is append order, not wall-clock event-time order.
• Consistent keys do not guarantee order across different topics, independent producer races, or after repartitioning.
• Idempotent Kafka producers reduce duplicate broker writes but do not replace application idempotency.
• A retry topic can reorder delivery relative to the original topic; version gates still matter.
• A consumer group ensures a partition has one active owner in the group, but an asynchronous worker pool can reintroduce application-level reordering.

Trade-offs and scaling

Hot orders can overload one partition; use many partitions and stable order keys, monitor skew, and never arbitrarily shard a single order's events if strict per-order sequencing is required. Increasing partition count changes the key-to-partition mapping for future messages, so treat repartitioning as a controlled migration. More consumer replicas than partitions cannot increase the active partition parallelism in a group. For high-throughput workloads, use a partition-aware processing model and backpressure.

Production validation checklist

• Publish CREATED v1, PAID v2, SHIPPED v3 and verify expected state.
• Inject SHIPPED v3 before PAID v2: v3 is held; no shipment state is exposed.
• Replay PAID v2 twice: only one state transition applies.
• Simulate producer retry, relay crash, consumer rebalance, poison message and DLQ replay.
• Alert on missing sequence numbers and reconciliation latency.
• Use tracing to correlate order ID, event ID, partition, offset and version without leaking sensitive payment details.

Interview Q&A

Q: Does Kafka guarantee ordering? Yes, records are read in offset order within a partition. Not globally across partitions.

Q: Is orderId as the key sufficient? No. It supports partition affinity, but independent producers and business state still require an authoritative sequence and guarded transitions.

Q: Can we use event timestamps to sort? Not as the source of truth: clocks skew and events can be delayed. Use a monotonic aggregate version issued by the state owner.

Q: What if v3 arrives before v2? Hold and reconcile; never apply an invalid transition just to keep the consumer moving.

Q: Does exactly-once solve it? Kafka transactions can coordinate Kafka read-process-write flows under specific conditions, but they do not make external databases or business transitions magically exactly-once.

Interactive practice

1. Delayed payment event: step through the shipment event, delayed payment event, waiting-event recovery, and old-event replay. Compare the shopper's two screens at each step. The actual payment already succeeded.
2. Duplicate delivery: replay PAID v2 and compare the number of projection writes. More deliveries should not mean more business effects.
3. Invalid transition: deliver PAID v4 after SHIPPED v3. Explain why the protected consumer rejects it despite the consecutive version.
4. Healthy order: compare normal delivery to the failure scenarios. Use Previous to inspect an earlier decision or Auto-play to watch the sequence.

Every counter and event is simulated. This lab demonstrates projection decisions, not a running Kafka cluster, payment gateway, or fulfillment service.

When the missing event never arrives

Persist the gap, its first-seen time, and the held event. After a defined timeout, alert and reconcile with the authoritative order service or replay a verified event from the event history. Do not manufacture PAID v2 or mark a package shipped because a timer expired. Decide whether to pause a partition or isolate that order; pausing protects order but can delay unrelated records, while an order-scoped durable queue needs more coordination.

Commit the projection update and any deduplication record in one database transaction, and acknowledge Kafka only after that durable work succeeds. Recovery workers must use the same atomic checks as the main consumer. External effects such as sending an email need their own idempotency or an outbox; a projection version check alone does not make those effects exactly-once.

References and next lesson

• Apache Kafka design: partitions and delivery semantics
• Spring Kafka: non-blocking retries and their ordering trade-off
• Transactional outbox: keep database state and event intent together
• Kafka retry topics and DLQ: recover failed processing

Conclusion

Partition by the order aggregate, preserve per-partition processing, and enforce versioned business transitions. Ordering is a data-contract and state-machine problem, not a Kafka checkbox.