Release testing

Release testing

Summary

Document a set of missing tests that would be nice to have as part of the existing automated test suite.

Motivation

Increase confidence in a release by enhancing the automated test suite to cover as many of the scenarios end users are going to end up running into as possible. Ideally, all of these tests would run in every commit or in a regular cadence to preempt issues cropping up just before a release is cut but for now even getting them in shape for release testing would be beneficial.

Regression testing

node

• update the node on old data, run all node tests and expect them to pass
• new version with stale data joins a network of older versions, can catch up syncing (can be done in isolation, floonet, or mainnet)

Enhancing the node tests to enable testing various upgrade scenarios is tracked at the grin issue tracker here.

stratum

• latest node version X can run with latest grin-miner versions Y and Y-1 and vice versa, ie. Y can run with X and X-1 (need to define how support between node and grin-miner versions is going to look like)

wallet

• update the wallet on old data, run all wallet tests and expect them to pass
• send transactions between wallet versions (test with all supported methods: file, http, keybase)
• latest node version X can run with latest wallet versions Y and Y-1 and vice versa, ie. Y can run with X and X-1 (need to define how support between node and wallets versions is going to look like)

Enhancing the wallet tests to enable testing various upgrade scenarios is tracked at the grin-wallet issue tracker here.

Stress testing

The following questions are meant to result in a suite of stress tests that reach Grin to its limits. Stress-testing is useful for identifying bottlenecks in the current system and performance monitoring across releases.

• how many transactions per second can the network handle?
• how many connections can a grin node handle?
• how many workers can the stratum server handle?
• ...

In order to run such tests, an investement needs to be made in infrastructure (probably cannot do stress-testing in any of the hosted CI services for free or at all).