In the last development update, we discussed the Swish team’s involvement in developing the THORChain network, a high-performance liquidity and payment protocol for decentralized exchange use cases. As you may recall, we started work on the protocol some months ago, culminating in the deployment of the Genesis testnet.

With a running testnet, it was time to perform some serious performance and stability testing, while in parallel implementing additional features and building the ASGARDEX exchange on top of the network.

Let’s first discuss our testing procedures and results, before getting into the current development goals.

Throughput Expectations

THORChain uses a pBFT-based consensus algorithm between its elected validator nodes. pBFT is an efficient consensus protocol, which precedes blockchain technology but is being rediscovered for scenarios in which almost instant block finality and high transaction throughput are desired. As explained in the last update, we are building on Tendermint and the Cosmos SDK. The former is an efficient implementation of pBFT for blockchains.

In case you are interested in the low-level details of this adaption of pBFT, this recent paper explains the protocol quite well.

The latter adds blockchain-specific details on top of pure consensus, such as support for tokens and staking.

In order to prepare performance testing with sensible expectations, one must first calculate what is realistically possible. According to some now-outdated specifications, Tendermint can handle up to 10,000 transactions per second (tps).

However, before we could get excited and set this number as our target, we had to consider two issues:

• Not all transactions are equal. Tendermint can be used for many things, and transactions can be very small payments without any additional data, or they can be very heavy transactions, shifting large amounts of data around.
• Tendermint is just a consensus framework, rather than a full blockchain. It is typically used as a component to build blockchains. Pure Tendermint transactions are more comparable with distributed database transactions than with blockchain transactions. Adding the additional layers required in a blockchain also adds some overhead. We are using the Cosmos SDK, which builds on top of Tendermint. We also add our own layers, such as built-in liquidity, on top of Cosmos SDK, some which are quite IO-heavy.

Keeping this in mind, we realized 10,000 tps might be an unrealistic target, at least on our first testnet. We took a look at other projects that used Tendermint at the consensus level to see what they had achieved. BigchainDB, for example, has invested a lot of time in optimizing transaction throughput and report peak performance of around 1,000 tps.

After some research, we established 700-800 tps as our performance target for the Genesis testnet. Eventually, the target for THORChain will be much higher, particularly once the Yggdrasil sharding protocol has been deployed. For now, the above number seems like a realistic target given our research into benchmarks from other teams currently building on Cosmos and Tendermint.

Experimental Setup

The Genesis testnet was initially launched with four validator nodes deployed on cloud-hosted virtual machines. The four validator nodes are hosted on Amazon Web Services.

It seemed trivial to launch a small number of nodes and run a testnet for these purposes, but in order to make such a network truly useful, some careful planning was required. The team invested some time in automating and documenting the deployment process. This made it easy to redeploy whenever bugs would be fixed or new features implemented. Logging and scripts to extract and analyze the logs locally were an important part of this process.

If you are interested in the details of the setup or would like to replicate it, you can look at the code and documentation on GitHub.

This initial setup is, of course, artificial, in that all nodes are equal and have an excellent connection with the other nodes. However, it served as a good starting point to scope out the theoretical limits of the blockchain in terms of performance and it let us evaluate whether the algorithms we implemented were functioning correctly.

The Genesis testnet serves three purposes:

• Provide an environment to test our algorithms. Some of these tests are manual, but the team also has a suite of automated tests that can be run any time a change has been applied.
• Constitute the basis for performance benchmarking by providing measurable results in terms of transaction throughput and latency in a best-case scenario.
• Allow the team to run stress tests to see how the blockchain behaves under heavy load by tweaking the parameters of our spamming scripts.

To address performance benchmarking and stress testing, we have developed a bot capable of “spamming” the testnet. Spamming a network is not as easy as it sounds, as there are some practicalities that need to be taken into account. First of all, there need to be enough accounts available to create a high enough volume of transactions. The Cosmos SDK light client used currently only allows one transaction per account in each block. To work around this limitation, the team has written a script that creates a configurable number of accounts that can be used by the bot to emit transactions. Secondly, we needed to make sure that all accounts had enough testnet rune, THORChain’s native token, to be able to move them around between accounts. In early testing, our bots ran out of tokens and transactions failed because of this.

Once these issues had been resolved, though, we deployed two bots that continuously sent out transactions at a configurable rate to the testnet. Detailed logs recorded the results, but an overview of the network’s stats can also be viewed on our block explorer.

Initial Results

Once the test environment was set up, the bots started spamming the THORChain Genesis testnet. Initially, we quickly reached 900 tps with a block finality of 750 ms. However, before we were able to crack open the champagne over these very positive performance results, we hit a stability problem. The blockchain was running well, producing 351,609 blocks, before all nodes lost connection to the network and failed to reconnect, eventually freezing the network. Reducing the load, we managed to get the blockchain fairly stable at around 100 tps. However the nodes still occasionally froze and needed a restart. The fact that the issue was still present at lower loads led us to believe that the problem was not due to stress on individual nodes, but represented an underlying stability issue in the Tendermint protocol. At 900 tps we simply ran into the problem earlier, meaning that nodes lost connections and needed to be restarted more frequently. Nodes still occasionally freeze until now.

The THORChain team has narrowed the issue down to a few lines of code and is in contact with the Tendermint team to fix the issue and to ensure better testnet stability.

You can see our open issue on the Tendermint GitHub page if you are interested in the low-level details.

We believe we are close to the solution and are encouraged by our earlier results with high transaction throughput. We have recently upgraded to the latest release of the Cosmos SDK and we continue to investigate log output to narrow down any further issues and ensure longer-term blockchain stability before we turn our focus to further performance optimizations to hit our 700-800 TPS target.

The Road to Helheim

The next big milestone in the THORChain project is Helheim. Therefore, in parallel to working on testnet performance and stability, the team is currently busy implementing a number of new features for the next testnet release.

Helheim is all about making decentralized trading a reality. To this end, we are currently working on two main components:

- On-chain trading support. We have already talked about continuous liquidity pools (CLPs) as a means of providing on-chain liquidity. CLPs are already a feature of the Genesis testnet. It is now time to enable trading through CLPs on THORChain, which will ensure that even pairs with low trading volume will have enough liquidity to enable fast and cost-effective decentralized trading. We are also implementing on-chain order books and limit orders.

- Blockchain bridges. In order to perform decentralized trading with other blockchains, the THORChain interoperability features need to be implemented. These consist of a number of bridges between external chains and their THORChain TokenChain representations. Bridges use THORChain’s interoperability protocol Bifröst. The research team has just released an update of the Bifröst whitepaper, and we are finalizing the technical specification, as work on implementing bridges to Ethereum is just about to start. We will discuss the Bifröst protocol in much more detail in the next development update.

- ASGARDEX. In order to demonstrate THORChain’s use for implementing decentralized exchanges (DEX), we are doing just that: implementing a DEX. ASGARDEX is a user interface built on top of THORChain’s trading facilities, aimed at rivaling centralized exchanges in terms of usability, performance, and liquidity, but with the added security and the freedom of decentralized exchanges. ASGARDEX is a proof of concept, intended to be forked by the community. Anyone will be able to implement their own flavor of DEX on top of THORChain. Development on ASGARDEX is well on the way and we are expecting to release an alpha version very soon.

What’s Up Next?

As discussed above, exciting times are ahead of us for the THORChain project. Work on improving the stability of the testnet is one big priority that is already progressing well. Building out the on-chain trading features with ASGARDEX is progressing in parallel. Building Bifröst bridges, to Ethereum initially, is the next big step to the Helheim milestone. Work is just about to start in this area and we will provide detailed updates on our progress in the next development update.

This update was brought to you by Stefan Beyer from the Blockchain team at Swish.

We build cutting-edge blockchain-based solutions for inspiring projects. Have an idea we can bring to life? Let’s talk.