The readonly projector design did not start from publication.

It started from a SQL sink.

This is the story of how that happened. For the current architecture shape, see Verified Projector Architecture.

The first goal was straightforward: store record deltas from committed transactions into SQL so operators could inspect state history like a blockchain explorer.

Given a transaction identity, the explorer should answer:

  • what command came in;
  • which transaction sequence committed it;
  • which records changed;
  • which events were produced;
  • which result code was committed;
  • which schema version interpreted the command.

Given a record revision, it should also answer the reverse question:

  • which transaction produced this revision?
  • what was the input?
  • what else changed in the same transaction?

This was not meant to be a new source of truth. It was a derived view over committed history. But once the design worked, it revealed a cleaner architecture than the one we started with.

The first shape: SQL as a record-delta explorer

The SQL sink needed ordered transaction deltas.

It could not be fed by ad hoc application events, because the explorer must explain committed state, not just service-side notifications. It also could not rebuild missing historical rows from a final snapshot, because a snapshot only tells you what the world looks like now. It does not tell you which transaction changed which field.

So the readonly projector model took shape:

canonical txlog
      |
      v
readonly apply
      |
      v
deterministic materialization
      |
      v
projected transaction stream
      |
      v
SQL record-delta projector

The leader and standby remain responsible for canonical state change. They ingest commands, execute the business runtime, commit ordered results, replicate those results, and maintain the state that defines truth.

The readonly projector does something different. It consumes committed transactions, re-materializes deterministic deltas and events, and writes plugin-owned derived output.

That boundary matters.

The projector can be slow. It can rebuild. It can fail. It can own a SQL schema, an index, a search store, or a query service. None of that should roll back canonical state.

The stream is not just a callback

The useful part was not merely "call a plugin after a transaction."

The useful part was introducing a host-owned materialized stream between canonical apply and projector delivery.

That stream carries the shape a projector actually needs:

  • tx_seq;
  • command identity;
  • committed status;
  • deterministic record deltas;
  • deterministic events;
  • schema identity;
  • a transaction hash.

The stream is local to a readonly node. It is not canonical state and it is not the source of truth for runtime recovery. But it is a verification boundary for projectors.

Before a projected transaction is appended, the readonly node replays or materializes the canonical transaction with the loaded runtime schema, computes the deterministic output evidence, and compares it with the canonical transaction hash. If that comparison fails, the stream does not advance.

That is the important move.

A projector does not receive "whatever the leader said to publish." It receives output that passed readonly replay/materialization.

The old publication path

Before this architecture emerged, publication lived closer to the canonical server path.

The leader had several responsibilities:

ingest command
      |
      v
execute business logic
      |
      v
commit canonical result
      |
      +--> replication
      +--> publication

That shape is natural at first. The leader already sees the committed result. It already knows which events the transaction produced. So publishing from the leader feels direct.

But it also grows the leader's responsibility.

The leader becomes responsible for command ingress, execution, replication, and external publication. Publication failure, queue pressure, transport behavior, sink status, delivery tracking, and recovery all sit near the canonical state machine.

During chaos testing, publication-related bugs made this coupling more visible.

The exact bug was less important than the architectural signal: publication behaved like an external effect, but it was living too close to the canonical commit path.

Publication is just another effect

Once the SQL projector existed, the question became obvious:

What is publication, really?

For a trading engine, a trade event is an effect. A balance update event is an effect. A user-position update, market-data update, database write, WebSocket notification, and risk projection are all effects derived from committed state transitions.

Publication is not special because it uses Kafka or another transport.

It is special only because downstream systems care about it.

But the SQL explorer also cares. Market data cares. Account queries care. Risk services care. A database persister cares. Each one needs an ordered, consistent, replayable view of the same committed history.

That makes publication a projector problem.

canonical state change
      |
      v
verified projected stream
      |
      +--> SQL explorer
      +--> market data projector
      +--> account/query projector
      +--> risk projector
      +--> publication projector

The canonical server should decide truth. Projectors should own external effects.

Verified publication

Moving publication to a readonly projector is not just a cleanup.

It changes the trust model.

The publication projector is no longer a passive follower of the leader's publication path. It consumes the verified projected stream. That means the readonly node has replayed or materialized the canonical transaction, computed deterministic output evidence, and compared it with the leader's committed hash before publication sees the transaction.

If the projected output does not match the canonical evidence, the projector refuses to publish.

That gives the system a stronger property:

published event = effect derived from verified canonical execution

This is why the earlier work on speculative execution matters.

Speculative execution makes the input and output of a transaction explicit before it crosses the durable boundary. The committed result can carry the evidence needed for replay, audit, and verification. A readonly projector can then independently re-materialize the same transaction and compare the result.

Verified publication comes from that shape.

What gets simpler

Once publication moves out of the canonical leader path, the server gets smaller in the place that matters most.

The canonical server can focus on:

  • command ingress;
  • deterministic execution;
  • ordered commit;
  • canonical state;
  • replication;
  • recovery.

Publication-specific concerns move to the projector:

  • sink frontier;
  • transport retry;
  • idempotent event keys;
  • bounded delivery queues;
  • active publisher lease;
  • publication lag;
  • downstream visibility.

That reduces the leader's failure surface.

It also makes publication recovery more honest. The projector owns its durable publication frontier. Host status may observe it, but the projector's sink/frontier is the recovery truth for publication. The projected stream tail is verification progress, not publication success.

This distinction prevents a common mistake: treating "the transaction was materialized" as "the event was published." Those are different frontiers.

What gets faster

The obvious benefit is reliability.

The less obvious benefit is performance.

If the leader is responsible for too many external effects, the hot path starts carrying work that belongs elsewhere. Even when publication is asynchronous, the leader still needs queues, backpressure policy, delivery status, shutdown behavior, retry accounting, and fault handling around something that is not canonical state.

Moving that work to a readonly projector gives the leader a narrower job.

The critical path becomes cleaner:

command -> deterministic execution -> canonical commit -> replication

Effects can scale beside it:

verified projected stream -> effect projectors

This does not make projectors free. They still need bounded queues, durable frontiers, idempotent writes, gap policy, rebuild tools, and clear operator status.

But their complexity is placed behind the right boundary.

The architecture lesson

The surprising part of this evolution is that it started from an audit/query need.

We wanted a SQL record-delta explorer. To make it correct, we needed a materialized projected stream. To make that stream trustworthy, we needed deterministic verification. Once that existed, publication no longer looked like a special leader responsibility. It looked like another external effect.

That is a good sign.

A good architecture does not only make the current feature work. It makes adjacent responsibilities fall into place.

The leader and standby own canonical state change.

Readonly projectors own external effects.

The projected stream is the bridge: derived from canonical history, verified by deterministic replay/materialization, and delivered to plugins that own their own sinks and frontiers.

The result is cleaner, safer, and potentially faster.

Beautiful architecture is not only reasonable and tidy.

Sometimes the cleanest architecture is also the fastest one.