This week, the Solana mainnet was hit by a wave of intermittent service that led to the network being turned off.
Solana has experienced a phenomenal surge of growth in recent months, largely due to its prominence as a favoured crypto asset for trading non-fungible tokens (NFTs).
In recent weeks, SOL has seized claim to the title of the world’s sixth-largest digital asset and seen impressive price action driving a new all-time high of $214.96.
But it seems the honeymoon period of jaw-dropping growth might be coming to an abrupt end after the Solana network descended into instability following its intermittent service announcement.
Price action responded sharply with investors clearly spooked. SOL’s value plummeted 17% when the network went offline, tumbling from $174.93 to $144.15.
Solana mainnet-beta is experiencing intermittent instability. This began approximately 45 minutes ago, and engineers are investigating the issue.
— Solana Status (@SolanaStatus) September 14, 2021
The last Solana block validated before the network went down occurred at 8am on September 14.
By mid-day, the CEO of Solana Labs – Anatoly Yakovenko – was forced to tweet a statement offering an explanation for the situation in an effort to reassure concerned followers.
Bots during a raydium ido are flooding the network at 300k txs per second. The queues that forwards txs to block producers grew in size to a point that caused excessive forking. The fix to prioritize messages in this queue was already in the works but wasn’t out yet.
— toly 🇺🇸 (@aeyakovenko) September 14, 2021
Yakovenko appeared to place the blame on the network being absolutely overwhelmed by transactions generated by trading bots during the initial DEX offering (IDO) for Raydium.
Raydium is a much-anticipated decentralised exchange (DEX) aiming to deliver an automated market maker (AMM) for the Solana ecosystem to facilitate trading of crypto assets using liquidity pools.
Further details were revealed throughout the afternoon, with the issue ultimately being pinned down to GPU validators running out of available random-access memory to handle the transactional volume.
2/ This forking led to excessive memory consumption, causing some nodes to go offline. Engineers across the ecosystem attempted to stabilize the network, but were unsuccessful.
— Solana Status (@SolanaStatus) September 14, 2021
This highlights a network under the strain of its own success. Simply put, the network’s usage has exponentially outgrown its validator’s capacity and, when stress-tested, it fails. This is a fate Cardano (ADA) is keen to avoid through a steady growth strategy.
Nick Spanos, co-founder of Zap Protocol, was quick to explain the impact of the outage in comments given to Forbes.
“The outage of Solana’s network will have undoubtedly played a part in the decline of SOL’s price,” he commented.
“But it can bounce back provided the network is restored swiftly and investors remain confident in the project long term and don’t get too unnerved by the incident.
“They need to get it running fast, and if they do not, and people do not know how long it is going to be, then that is a problem for perception.”
There is an urgent need for more validators with bigger capacity across the Solana network and enhanced decentralisation to avoid network failure.
Coin Rivet can reveal that the Solana network is significantly more centralised than it appears at first glance, with the revelation that the network is now completely dependent on one Amazon Web Services server to stay online.
The data centre kill-switch
Back in March, a downtime alert from the Hetzner Falkenstein data centre revealed the stress limits of the network – at the time the centre was responsible for 15% of active SOL staking on the validator network.
There was a brief spike of serious concern in the Solana validator community about the potential impact this could have, with many validators left wondering exactly how much actively staked SOL the network needed to remain functional.
If 33% of $SOL's network goes down, @solana can't confirm transactions. This data center hosts 15% of Solana's active network stake. 2 datacenters account for 45% of validators.
Alerted last week: https://t.co/aVR2bi3bBx pic.twitter.com/GZjLdaMRqa
— Mira Christanto (@asiahodl) February 28, 2021
The Solana Foundation clarified that if more than 33% of the network’s validator nodes went offline, the Solana network would be unable to continue validating transactions.
As a proof of stake (PoS) network, the security and stability of the platform hangs on maintaining a large and decentralised web of validator nodes.
A key consideration of any PoS blockchain is also the need to incentivise more validators to participate in staking on the network in order to ensure it is capable of handling growth in the number of transactions.
But is appears this has been a challenge for Solana in the face of a sustained period of serious growth, and serious question marks hang over the extent to which the network – which has a total transaction volume approaching 30bn – is truly decentralised enough to guarantee service.
Coin Rivet can also reveal data from Block Logic‘s validator app, which highlights that just one Amazon data centre server (known by its assigned ASN) accounts for 37.45% of active Solana stake – with 142,877,155 SOL currently staked (worth more than $22bn).
In effect this gives the AWS server – known as ‘Amazon-02‘ – a kill-switch stake in the Solana network.
Dimitris Karakostas, an expert in proof-of-stake blockchain at the University of Edinburgh, was quick to explain the ramifications.
“Solana’s protocol guarantees that transactions are processed, as long as 67% of the total stake is active,” he explained.
“If some nodes control more than 33% of stake and they, or the servers on which they are hosted, go offline, the Solana network would halt, until some of them go online again to reach the 67% threshold.”
(14/09/21)
Therefore if the Amazon-02 server goes offline – for maintenance or as the result of a cyber attack – the Solana network will be unable to process transactions until it’s able to migrate validators, and the network will fail.
Some of the biggest validators staking through the AWS ASN are private (likely to be institutional finance), and Binance – both with in excess of 6,000,000 actively staked SOL (around $1bn in value).
Alongside the AWS are the Hetzner online data servers which are linked to the Hetzner Falkenstein project. With 68,459,414 SOL actively staked this ASN represents 17.94% of actively staked SOL.
Together, the two data centres account for more than 55% of the actively staked SOL on the network.
With a majority stake handled by just two ASNs – decentralisation is seriously under threat, which could likely drive future intermittences and disruption if the network is unable to increase the number of validators operating.
Why does Solana lack validators?
In order to understand why the network has become so centrally dependent, it’s important to understand the factors at play that put off new validators joining the network.
Fundamentally the main allegation comes from the cost of entry.
There is an initial hardware investment requirement for setting up a node which runs a few thousand dollars. There is then the need to pay the associated costs such as electricity for the operation.
On top of that Solana charge validator nodes a daily fee of 1.1 SOL (around $170 as of today), meaning there is a prerequisite requirement for validators to either have a significant stack of SOL ready to use or to push that fee onto investors staking their SOL through the node.
Couple that with the 8% APY offered to validators and, suddenly, hundreds of thousands of SOL are needed as start-up capital to break even.
In effect, this leads some to say that validator nodes are too expensive for regular people to operate and can only be run by those with deep pockets.
This enhances network centralisation, and creates a crypto project model in which validators can only recoup their investment by extracting value from smaller investors staking their SOL – not exactly Satoshi’s dream.
A developer that wishes to remain unnamed admits the barriers to entry are high but explained that the Solana designers specifically imposed those validator requirements for the reason of maintaining the advertised 50k transactions per second (TPS).
Enhanced centralisation of the network for the time being enables better controls to maintain low latency, and is also necessary to manage throughput as the network continues to avoid imposing a sharding layer.
More crypto news and information
If you want to find out more information about Bitcoin or cryptocurrencies in general, then use the search box at the top of this page. Here’s an article to get you started.
As with any investment, it pays to do some homework before you part with your money. The prices of cryptocurrencies are volatile and go up and down quickly. This page is not recommending a particular currency or whether you should invest or not.
Disclaimer: The views and opinions expressed by the author should not be considered as financial advice. We do not give advice on financial products.