A technical post-mortem of the Fogo Testnet outage of August 13th.
Summary
On August 13th, 2025 at 05:40 UTC, Fogo Testnet experienced a complete halt at slot 287501008. The incident resulted in a period of 14 hours and 20 minutes of outage. The Douro Labs team operating the validators identified, resolved the issue, and restarted the network by 20:02 UTC. The incident was caused by networking infrastructure instability at the time of a zone transition between consensus zones in Tokyo (TYO) and London (LON).
Timeline
August 12th, 20:19 UTC
Teraswitch, the infrastructure provider, detected the loss of subsea capacity between EWR to LON. Telemetry shows higher latency and packet loss, but network is still operational.August 13th, 05:40 UTC
Testnet halts at slot 287501008 during the zone transition between Tokyo and London.August 13th, 06:12 UTC
Team announces the incident.August 13th, 08:25 UTC
Team restarts the network at slot 287500278.August 13th, 09:25 UTC
Testnet halts at slot 2875901008 during a zone transition between London and Newark. Half of the validators in Newark did not sync to the network after the restart and the zone failed to form majority.August 13th, 15:53 UTC
Team restarts the network at slot 287500278 again.August 13th, 18:54 UTC
Testnet halts at slot 287771008 during a zone transition between Tokyo and London. The connection between Tokyo and London is still unstable at this point and it caused another unrecoverable fork.August 13th, 20:02 UTC
Team restarts the network at slot 287770278, and shortly after the restart, disables the Tokyo zone so consensus alternates between London and Newark.August 13th, 20:45 UTC
Incident is deemed resolved after a success zone transition.
Root Cause Analysis
During a zone transition, the last leader of the current zone must broadcast the last block in the current zone epoch before the first validator in the next zone can begin proposing blocks. If blocks don’t arrive from the previous zone, the next zone validator will assume the slots are empty/skipped and start proposing its own blocks.
If the next zone is completely unavailable during a zone transition, the zone mechanism will prevent it from becoming active and move to the next available zone. However, if the connection between the current and next zones is degraded, blocks might arrive late and cause the validators in the next zone to fork the network because now they observe two branches: the original blocks from the previous zone that arrived late, and the branch with skipped slots it started voting on.
Fogo consensus, just like Solana, uses switch proofs to resolve forks based on their vote weights. However, the stake weight is completely changed during a zone transition, and delayed blocks crossing the epoch/zone boundary will cause 100% of the new stake (i.e. the validator set in the upcoming/next zone) to fork and vote on an alternative branch of the chain indefinitely since a switch proof is impossible.
Resolution
Once the degraded networking conditions were identified, the team decided to remove the Tokyo zone from the consensus rotation. The transition between Tokyo and London is particularly sensitive because it has the highest latency of all. The Tokyo zone is to be restablished once the infrastructure is deemed recovered.
The team is working with the networking infrastructure provider to have the Tokyo-London link repaired and improved, and making sure the fallback routes have the capacity and stability necessary. The team is also continuing the ongoing work on zone transition strategies.
