DLQ Revive — Case study · Mohammed Saifulhuq

The problem

When a Kafka consumer can't process a message, the message lands in a dead-letter topic. Getting it back out usually means writing a one-off script: a consumer to find the message, some code to fix the payload, and a producer to send it back. When I asked engineers on Reddit how they handle this, the answer was consistent: everyone writes those scripts, nobody enjoys it, and the scripts run untested against production data.

DLQ Revive replaces the script with a self-hosted tool. You connect it to a cluster, browse the dead-letter topic, fix messages with a declarative transform, preview the result side by side, and replay them to a target topic. Every action is audited.

Constraints

Moving bytes between topics is easy. Doing it next to production without making things worse is the actual problem. Four rules shaped the design:

Don't disturb live consumers. Browsing a dead-letter topic must never trigger a rebalance or move a committed offset.
Replays must be safe to repeat. A double-click, a retry or a pod restart mid-replay must not deliver the same message twice.
User-written transforms must not run code. The transform box is a text field on a server that can reach your Kafka cluster.
Memory stays bounded, however large the backlog is.

Architecture

Angular dashboard

Connect
Browse
Transform
Audit

Spring Boot API

DLQReader
JsonataEngine
IdempotencyGuard
RedriveEngine
AuditLogger

DLQ topic

assign() + seek()
no commits

Target topic

acks=all
idempotent producer

PostgreSQL

redrive_log
audit_trail
transform_templates

The API is the only component that touches Kafka; the browser never holds broker credentials.

An Angular dashboard talks to a Spring Boot API over REST, with Server-Sent Events streaming replay progress back message by message. The API reads from Kafka with a short-lived, read-only consumer, transforms with JSONata, produces to the target topic, and records every replay and user action in PostgreSQL.

Key decisions

01Read with assign() + seek(), never subscribe()

Subscribing makes a consumer a member of a consumer group, and membership changes trigger rebalances. Instead, the reader assigns itself one partition and seeks straight to the requested offset, so it never joins a group. It also never commits offsets: browsing is strictly read-only, and the consumer is closed in a finally block after every page.

02Idempotency in PostgreSQL, not Kafka transactions

Every source message is identified by (topic, partition, offset). Before producing, the replay engine checks a redrive_log table and skips anything already replayed, and it records the message before sending it. A UNIQUE(topic, partition, offset) constraint is the real guarantee; the lookup is only a fast path. Two concurrent sessions can't both replay the same message.

if (idempotencyGuard.alreadyRedriven(topic, partition, offset)) {
    skipped++;
    continue;
}
// Record first: a restart after this line can't produce the message again
idempotencyGuard.recordRedrive(topic, partition, offset, user);
producer.send(new ProducerRecord<>(targetTopic, key, transformed)).get();

This survives restarts without coordinating a transaction across a database and a Kafka cluster.

03JSONata only for transforms

Message fixes are written as JSONata expressions. JSONata is declarative: it can reshape JSON, but it has no access to the file system, network, environment or processes. Groovy or JavaScript would have been more flexible, and would also have turned the transform box into remote code execution on a server that can reach the Kafka cluster.

	JSONata	Groovy / JS `eval()`
System, file or network access	None	Full
Turing-complete	No	Yes
Remote code execution risk	None	Critical

04Bound everything, then send synchronously

Pages are capped at 100 messages, a read gives up after three empty polls, and a replay session handles at most 100 messages. The producer runs with acks=all and idempotence enabled, and each send waits for the broker's acknowledgement before the next one starts.

Testing

The reader is tested against a real broker using @EmbeddedKafka: the test publishes messages, reads a page back, and checks that the consumer is closed. The idempotency guard has a test asserting that recording the same (topic, partition, offset) twice is rejected by the database. The replay engine, the JSONata engine, the Kafka connector and each REST controller have their own tests. GitHub Actions runs the backend suite on Java 17 for every push and pull request.

What I'd change next

Make "replayed" a state, not just a row. Write the log entry as PENDING, flip it to SENT once the broker acknowledges, and let failed entries be retried. That keeps the no-duplicates guarantee without stranding a message whose send failed.
Fold the check and the insert into one statement. INSERT … ON CONFLICT DO NOTHING reports whether the row was claimed. That removes a round trip and turns a concurrent duplicate into a clean "skipped" instead of a failure. It's the same pattern I proposed for request-level idempotency in Apache Fineract.
Run streaming replays on a bounded executor. Each streaming replay currently starts its own thread; a managed pool would cap how many can run at once.