In the last blog post, we marked Albatross’s one year anniversary with an in-depth walkthrough of Nimiq’s Albatross tech stack, which makes the Proof-of-Stake consensus mechanism one of a kind.
This post recounts the intense week around Nimiq’s Albatross Proof-of-Stake launch. It dives into the unexpected chain stall days later, the tense overnight debugging, and the eventual recovery after fixing a validator communication issue. Read the full story to relive the launch week from the inside and see how the team and protocol were tested and proved themselves under critical conditions.
Nimiq.FM - The Soundtrack of the Nimiq Blockchain
To mark Albatross’s first anniversary, we’re adding a banner on nimiq.com that redirects to nimiq.fm. It will stay up until our next announcement! Nimiq FM plays a live musical interpretation of the block production, with each producer's signature shaping a different sound. Block by block, the chain forms a continuous piece of music. Explore the nimiq.fm page and enjoy the reading!
A Week of Launch
The launch on November 19, 2024, was scheduled to coincide with a Nimiq hackathon. We chose block 3’456’000 as the genesis of Nimiq Proof-of-Stake because it coincides with the Nimiq team being together in person. It meant that everyone who had worked on Albatross could be together, in the same place, sharing the same focus. Plus, it made coordination easier and made the moment feel shared.
The days leading up to the launch were busy in practically every way. People double-checked configurations and ensured all monitoring tools were ready. Nobody was dramatic about it, but everyone was clearly aware of the importance of what was coming. You could see it in the way people leaned over each other’s screens to confirm details.
Because we were already gathered for the hackathon, there was a natural sense of working side by side. Some teams were polishing last-minute tooling, while others were reviewing the launch steps. It wasn’t chaotic, but it wasn’t relaxed either, just a group of people preparing for something they had been building toward for a long time.
November 19th - The First Block
Tuesday morning brought an unexpected bit of sun to the rainy season. We met early at the workspace; people brought cake and treats. Enough to mark the moment.
Sören had prepared a simple monitor to track validator readiness, and it quickly became the screen everyone kept glancing at. One by one, nodes turned green.
When block 3’456’000 was reached, Albatross went live, followed by ear-to-ear smiles, a couple of cheers, and the sound of someone opening a bottle of sparkling wine.
We shared cake, took pictures, and let the moment sink in. Most of the technical work had been handled in the days before, so the morning was mostly about watching things run as they were supposed to and feeling the relief of seeing it finally happen. Any minor technicalities were addressed on the days after the launch.
November 24th - The Temporary Stall
On Sunday, the 24th of November, we gathered for a small event to celebrate the biggest achievement in Nimiq’s history! We had a bonfire, ordered pizzas, had beers, and everyone was in a really good mood.
Mid-conversation, we received some alerts of a pause in the chain. At first, we thought there was a joke, but the moment we realized it was real, everyone moved in a heartbeat. From this moment, it took only a matter of seconds for everyone to get back to the cars and their computers. It was as stressful as it was reassuring to see the entire team run to their places. All their work was put in jeopardy, and in the blink of an eye, they were all at their desks.
In the middle of the night, there was a power outage. We had a generator, so the lights were back on within minutes. However, this seemed like a paranormal joke added to the stress.
November 25th - The Permanent Resolution
Devs worked like crazy and pulled an all-nighter. Those who couldn’t help with the protocol stayed around, feeling a bit helpless while the core team dug into the issue.
The “Validator Song” was a small program that turned Nimiq Proof-of-Stake blocks into sound. Maxi and Alberto had created it a few days earlier as a fun addition for launch day. It played an audio representation of each block in the chain. They decided to keep it running on the night of the stall, hoping that if new blocks appeared, we would hear the distinct, not exactly melodic, sound of the chain coming back to life.
The workspace was super quiet. When we have hackathons, we take the opportunity to have multiple meetings a day and have conversations, so the place is hectic. Not this time. There was a dramatic silence and worry in the air.
Dinner came, and people looked defeated. After that, there was the cake. Inês thought about bringing some sweetness to the team, as everybody has been working so hard. No pictures from before devouring the cakes, sadly.
Everyone lined up for cake, we called “the stall cake”!
Suddenly, from nowhere, while some were still lining up and others were already eating, we heard the Validator Song in the background.
Basti promptly said, “That’s not funny, guys”.
But it was not only funny but also true!
The chain resumed its regular operation!
What Happened?
It’s easier to follow the story with a bit of context on the issue.
A few days after launch, the chain hit an unexpected pause. Until that moment, all macro blocks had been produced smoothly, but one macro block didn’t finalize. This one stalled because, unexpectedly, validators no longer shared the same view of the round they were in.
From a technical POV, the validator peer information, specifically the mapping between validator IDs and peer IDs, had become inconsistent. As a result, validators could not reliably reach each other, ended up in different rounds and states, and were unable to exchange messages reliably, something that shouldn't happen in normal operation.
A multitude of measures were taken to address the issue, including actions to alleviate miscommunication between validators. These steps helped restore a consistent view of the network, allowing validators to realign and resume normal block production gradually. Once the conditions were met, the blockchain "healed itself".
The fix worked, but it didn’t take effect immediately. Validators still needed time to naturally “walk” into the same round and align again. Because they had diverged so far, this reconciliation was slow.
Total duration: roughly 25 hours from stall to recovery.
Shoutout to Nimiq Validators
Throughout the stall and recovery, community validators and node operators were right there with us. Many stayed up late and kept a close eye on their nodes and the network while we rolled out fixes and improvements. Your patience, quick reactions, and commitment to the protocol made a real difference in how smoothly the chain recovered. To everyone who helped monitor, test, and stick with us during those tense hours: thank you. 🧡
Looking Back
The Nimiq Proof-of-Stake preparation and launch were demanding. The entire team prepared thoroughly for it, and the community was with us since day one, something we don’t take for granted.
We are grateful to everyone who joined us on this lengthy and technical ride. The community, friends, and family.
Pura Vida,
Team Nimiq