ethlambda: what we learned running a 1024-validator devnet

Last week we ran a devnet with 1024 validators, split across 8 aggregation sub-networks. That layout keeps the network scalable, since each aggregator combines 128 signatures per interval, well inside budget. Each of the 32 ethlambda nodes is in its own server, with 32 validators each, twice as many as our previous runs.

The network stayed up and kept finalizing for the whole run, with less margin than we want. The proposer turned out to be the bottleneck, with high block build times, which matches what we saw in earlier runs and wrote about in our last post.

The numbers

One block in nine got reorged, left outside the chain, and finality ran 24 slots behind the head.

With 4s slots, and 0.8s intervals, the signature path holds up: 97% of signatures arrive within their interval, 95% of committee aggregations arrive on time, and committee aggregation peaks at 0.42s.

Block production misses its window. 5% of blocks arrive on time, and 59% arrive during signature propagation, the interval in which validators should be voting on them. Block building averages 2.5s and recursive aggregation 1.3s, both longer than the 0.8s interval each one has to finish in.

Late blocks explain the reorgs

Every reorg we saw traces back to a proposer publishing late.

Validators vote at a fixed point in the slot, for the head they can see at that moment. A block that has not arrived yet is not the head, so the votes go to its parent instead. The next proposer builds on the parent, because that is where the weight is, and orphans the late block no matter how valid it was. The proposer broke no rule and the network dropped nothing: it finished the block after the votes that decided its fate.

That makes the reorg rate a direct function of how late a block is. A block that finishes building after its interval has a 3% chance of being reorged, and the chance climbs with the delay: a full slot late, and the block gets reorged 80% of the time. With an average build time of 2.5s and only 5% of blocks arriving on time, a lot of proposals sit somewhere on that curve. The 59% of blocks that arrive during signature propagation survive most of the time, since they land before the next proposer freezes their view. The remaining 36% are where most of our reorgs come from, and about a quarter of them arrive late enough that the next proposer builds past them.

Slow recursive aggregation explains the finality lag

Finality suffers for a related reason. Fewer blocks land on chain, so validators have fewer places to pack attestations into and votes take longer to accumulate. In the lagging slots, the attestations that justify a block, the first step to finality, land 2 to 4 slots after the proposal. We expect that much: fork-choice needs at least one slot to confirm the vote, and 4 slots is the worst case for a block to become a safe target under synchrony.

The slots after that are where we lose the time. Later blocks keep including more signatures for the same attestation data, and around 11 slots pass before a block carries enough votes to be justified. Two rounds of that account for most of the 24-slot finality lag, so transactions take almost two minutes to finalize. That beats today's beacon chain and still falls short of what we want.

The reason justification takes 11 slots instead of 2 or 3 is that recursive aggregation isn't keeping up, and the 95% on-time figure for committee aggregations hides that. Committee aggregation is only the first round. It gives us one payload per subnet, covering 128 signatures out of 1024, and it finishes inside its interval with room to spare. Turning those 8 payloads into one that covers the whole validator set takes recursive rounds on top, and that work costs 1.3s on average, longer than the interval the committee aggregates arrived in. Aggregators run it as a blocking step against their other duties, so the recursed result is not ready by the time the proposer needs it.

The proposer builds with what it has. Blocks end up carrying attestations with no recursive aggregation rounds applied, a single committee's worth of votes each, so each block carries a fraction of the weight it could. Votes that were all produced on time still take several slots to accumulate into a justifying majority, one committee at a time.

Where the time goes

Block building averages 2.5s, and multi-message aggregation, the aggregation of aggregated payloads for different votes, accounts for almost all of it. We skip recursive aggregation of same-message payloads, keeping only the best aggregated payload for each vote, during block building, since that slows the build down further, so combining payloads from different attestations dominates the cost.

One way out is to make multi-message aggregation faster outright. That buys enough headroom to also recursively aggregate same-message payloads, which packs more attestations into every block and shortens the justification path.

Another is to stop doing all the work at proposal time. Multi-message aggregation splits in two, payloads we know in advance, like finality votes targeting blocks from previous slots, and payloads that arrive right before proposal, like the current slot's votes and the proposer signature. We can aggregate the first group ahead of time. On the API side that means a streaming interface: add proofs one by one and collect them into a block proof at the end, or recursively aggregate multi-message payloads as they arrive.

There is also a smaller win available, that we wrote about before. Part of block building is spent turning the proposer signature into an aggregatable payload. Moving that signature out of the block proof and into its own field drops one payload from the aggregation set, and at 1.3 KB per signature against the 220 KB of a block proof, block size is practically unaffected.

Aggregator throughput helps too. Aggregators run recursive aggregation as a blocking step today; pipelining it against their other duties, along with the leanVM performance work, should bring the 1.3s average down.

The part that works

Committee aggregation peaked at 0.42s, and we kick off aggregation up to 0.6s before its interval starts, so the result lands with room to spare. If the cost stays linear, doubling the validators per subnet takes that peak to 0.84s, longer than the interval itself. It still lands on time, but only because of the head start. Signatures should keep arriving on time at 2048 validators with the subnet layout unchanged.

What's next

Across these devnets ethlambda stayed stable and scaled, and we now know which component to spend our time on. We're running interop devnets to shake out interop problems, and starting network simulations to prepare for the aggregation improvements coming with the new leanVM over binary fields. Most of the numbers above should look different once that lands.

In parallel we're contributing to the discussion on where the post-quantum Ethereum effort goes next.

Follow us

Join us on Telegram for daily development updates and follow us on X for announcements and our weekly community calls. Visit ethlambda.xyz for more information and useful links.