Add Echidna Tests #16

naszam · 2021-04-30T21:45:28Z

Description:

Add Echidna Tests for DssVest

https://app.clubhouse.io/scdomaincommunityteam/story/11462/add-echidna-tests-for-dss-vest

COB Team Name or Author(s): @naszam

Contribution Checklist

brianmcmichael · 2021-05-01T21:13:00Z

Thanks, can you add some additional docs to the readme with general setup for this?

naszam · 2021-05-03T13:38:40Z

I've added the fuzz docs as well as GitHub Actions Fuzz CI and badge 🙂

kmbarry1 · 2021-05-05T01:38:56Z

src/fuzz/DssVestEchidnaTest.sol

+    }
+
+    function test_init_params(uint256 _amt, uint256 _bgn, uint256 _tau, uint256 _clf, uint256 _pmt, address _mgr) public {
+        _amt = 1 * WAD + _amt % uint128(-1);


this can exceed uint128(-1), which I assume will produce a violation since init will revert

I just set _amt and _pmt min seed value to zero

this should be done.

kmbarry1 · 2021-05-05T01:39:23Z

src/fuzz/DssVestEchidnaTest.sol

+    }
+
+    function test_vest(uint256 _amt, uint256 _bgn, uint256 _tau, uint256 _clf, uint256 _pmt, address _mgr, uint256 _tick) public {
+        _amt = 1 * WAD + _amt % uint128(-1);


same comment as above

kmbarry1 · 2021-05-05T01:45:23Z

src/fuzz/DssVestEchidnaTest.sol

+        _amt = 1 * WAD + _amt % uint128(-1);
+        _bgn = block.timestamp + _bgn % vest.MAX_VEST_PERIOD();
+        _tau = 1 + _tau % vest.MAX_VEST_PERIOD();
+        _clf = 1 + _clf % _tau;


Why not allow _clf ==0? Seems like a valid case.

kmbarry1 · 2021-05-05T01:45:37Z

src/fuzz/DssVestEchidnaTest.sol

+        _amt = 1 * WAD + _amt % uint128(-1);
+        _bgn = block.timestamp + _bgn % vest.MAX_VEST_PERIOD();
+        _tau = 1 + _tau % vest.MAX_VEST_PERIOD();
+        _clf = 1 + _clf % _tau;


same comment as above

kmbarry1 · 2021-05-05T01:53:26Z

src/fuzz/DssVestEchidnaTest.sol

+        _tau = 1 + _tau % vest.MAX_VEST_PERIOD();
+        _clf = 1 + _clf % _tau;
+        _pmt = 1 * WAD + _pmt % _amt;
+        _tick = block.timestamp + _tick % uint128(-1);


Okay I need some education on echidna...how does the value of _tick affect the behavior of the the vest call? I guess what I'm expecting is some sort of special instruction that creates a relation between _tick and block.timestamp before the call to vest, similar to hevm.warp, but I'm not seeing it.

I've added _tick as a seed to simulate time for time dependant scenarios.

I've just removed _tick as echidna should increase block.timestamp when fuzzing.
Also fixed the check in test_vest with block.timestamp >= fin.
Let me know if it's ok.

kmbarry1 · 2021-05-05T17:33:56Z

src/fuzz/DssVestEchidnaTest.sol

+    }
+
+    function test_init_ids(uint256 _amt, uint256 _bgn, uint256 _tau, uint256 _clf, uint256 _pmt, address _mgr) public {
+        _amt = 0 + _amt % uint128(-1);


I kind of think _amt = 0 is an uninteresting case--I'd recommend enforcing a reasonable minimum like you were before, just in a way that won't exceed the uint128(-1) limit. For example, here's one option:

_amt = _amt % uint128(-1); if (_amt < WAD) _amt *= WAD;

That will never cause a uint128 overflow since WAD * (WAD - 1) < 2^128 - 1.

Oh yeah...we want to avoid _amt == 0...so we need something like:

_amt = _amt % uint128(-1); if (_amt < WAD) _amt = (1 + _amt) * WAD;

kmbarry1 · 2021-05-05T22:06:55Z

src/fuzz/DssVestEchidnaTest.sol

+        _pmt = _pmt % _amt;
+        uint256 prevId = vest.ids();
+        uint256 id = vest.init(address(this), _amt, _bgn, _tau, _clf, _pmt, _mgr);
+        assert(vest.ids() == add(prevId, id));


This won't hold in general. Suggest replacing with:

assert(vest.ids() == add(prevId, 1)); assert(vest.ids() == id);

kmbarry1 · 2021-05-05T22:09:04Z

src/fuzz/DssVestEchidnaTest.sol

+        require((z = x - y) <= x);
+    }
+
+    function test_init_ids(uint256 _amt, uint256 _bgn, uint256 _tau, uint256 _clf, uint256 _pmt, address _mgr) public {


If we're bothering to pass _mgr as a fuzz parameter, we should at least verify that it's being set correctly in the award (same for the other values).

this should be done in test_init_params

Actually I'd suggest to combine test_init_ids and test_init_params into a single test named test_init. And it's probably unnecessary to pass _mgr in test_vest; alternatively, test_vest could be modified to randomly call vest on a previously-initialized award.

Removed the _mgr seed and replace with echidna_mgr

…ackets

README.md

.github/workflows/fuzz.yml

brianmcmichael

Fuzzing!

brianmcmichael · 2021-05-12T22:32:22Z

src/fuzz/DssVestEchidnaTest.sol

+        assert(z == x);
+    }
+
+    function test_init(uint256 _tot, uint256 _bgn, uint256 _tau, uint256 _clf, uint256 _end) public {


might also be worthwhile to have a test that hevm warps to some random point during or after init and check that payouts and accrued and unpaid values all line up.

yeah, I think @kmbarry1 will make some dapp fuzz tests as well leveraging on hevm.warp to fuzz time-dependant logic and compare the two approaches.

yeah I was kind of waiting for all the churn to settle down in the code itself

kmbarry1 · 2021-05-13T01:26:56Z

src/fuzz/DssVestEchidnaTest.sol

+    }
+
+    function test_vest(uint256 id) internal {
+        vest.vest(id);


So, the issue here is that the award will almost never exist if id is a random uint256. You'll need to mod it into the appropriate range:
id = 1 + id % vest.ids();
to effectively test already-created vests.

test_vest and test_yank are called inside test_init with an existing valid id.
I can fuzz them independently leveraging on the preserved state option eventually

I think we shouldn't fuzz the ids cause they're checked in all the functions on dss-vest, if the award exists, so declaring both test_vest and test_yank internal allows to check for asserts on test_init seeds.

I think this approach is good enough. The only downside is every award gets yanked before any more are created, so you're never testing with multiple awards in existence. However, given how the contract is written, there's very little chance of bugs that only surface when multiple awards exist.

kmbarry1 · 2021-05-13T01:27:08Z

src/fuzz/DssVestEchidnaTest.sol

+    }
+
+    function test_yank(uint256 id, uint256 end) internal {
+        ( , , , uint48 _fin, , , ) = vest.awards(id);


You'll have a similar issue here as above.

In this case test_yank, that is called inside test_init, will use an existing valid id plus an _end seed from test_init.

kmbarry1

LGTM

Add echidna tests for dss-vest

b8b0c46

naszam added 4 commits May 3, 2021 14:21

Add fuzz docs to readme

76263a2

Merge branch 'master' into fuzz

17eb123

Add fuzz CI + update echidna config + cleanup

249bc10

Add CI fuzz badge + local fuzz settings to readme

ce48f5a

naszam added 3 commits May 3, 2021 14:53

Cleanup readme

9b5ac48

Update .gitignore

ccdc6ad

Update readme

c758324

kmbarry1 reviewed May 5, 2021

View reviewed changes

naszam added 2 commits May 5, 2021 13:23

Allow _clf equal zero

859bd83

Set _amt and _pmt min seed value to zero

a7067de

kmbarry1 reviewed May 5, 2021

View reviewed changes

naszam added 3 commits May 5, 2021 20:40

Fix _amt seed value with check for WAD

15b99cd

Remove _tick + fix fin check + cleanup

a2a1c3f

Cleanup

987d9e9

kmbarry1 reviewed May 5, 2021

View reviewed changes

naszam added 7 commits May 6, 2021 00:05

Fix _amt seed + assert ids

8bd10a3

Fuse test_init_ids and test_init_params into test_init + add curly br…

b410712

…ackets

Update readme

f6c75c3

Remove _mgr seed

1e38053

Merge branch 'master' into fuzz

265cfc2

Remove _pmt seed + fix _bgn seed range + cleanup

0bc19c0

Cleanup _bgn range

92ed7bc

naszam added 2 commits May 12, 2021 19:39

Add corpusDir echidna config opt

6178810

Update readme

f09c6fb

brianmcmichael reviewed May 12, 2021

View reviewed changes

README.md Outdated Show resolved Hide resolved

naszam added 2 commits May 12, 2021 20:52

Cleanup

05110a0

Update readme with docker pull

2264418

brianmcmichael reviewed May 12, 2021

View reviewed changes

README.md Outdated Show resolved Hide resolved

brianmcmichael reviewed May 12, 2021

View reviewed changes

README.md Outdated Show resolved Hide resolved

Cleanup readme

ed9846b

brianmcmichael reviewed May 12, 2021

View reviewed changes

README.md Outdated Show resolved Hide resolved

brianmcmichael reviewed May 12, 2021

View reviewed changes

.github/workflows/fuzz.yml Outdated Show resolved Hide resolved

naszam added 8 commits May 12, 2021 21:50

Duplicate echidna config file for local and ci

0c23bcb

Cleanup readme

cc7595f

Bump CI echidna to v1.7.1

b313c1e

Cleanup coverage in echidna.config.yml

14c0f50

Stick with echidna v1.6.0 for CI

7bc78af

Update readme echidna to v1.7.0

c518a3c

Bump CI echidna to v1.7.0

4cf1724

Update readme nix install from master

1ba313b

brianmcmichael approved these changes May 12, 2021

View reviewed changes

brianmcmichael reviewed May 12, 2021

View reviewed changes

kmbarry1 reviewed May 13, 2021

View reviewed changes

naszam added 6 commits May 13, 2021 21:41

Bump CI echidna to v1.7.1 with fix

11c4382

Optimise echidna config opts

0549117

Cleanup

0380b12

Add corpus folder to .gitignore

f586a92

Update maxTimeDelay to 1 year for CI echidna config

7417bfa

Fix test_yank execution order

2d2cc02

kmbarry1 approved these changes May 14, 2021

View reviewed changes

brianmcmichael merged commit 9b8b37b into makerdao:master May 14, 2021

Add Echidna Tests #16

Add Echidna Tests #16

Conversation

naszam commented Apr 30, 2021

Description:

Contribution Checklist

brianmcmichael commented May 1, 2021

naszam commented May 3, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

naszam May 5, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

naszam May 5, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

naszam May 5, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

naszam May 5, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

naszam May 5, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

naszam May 5, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

naszam May 5, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brianmcmichael left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

naszam May 12, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

naszam May 13, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

naszam May 13, 2021 • edited Loading

Choose a reason for hiding this comment

kmbarry1 left a comment

Choose a reason for hiding this comment

naszam May 5, 2021 •

edited

Loading

naszam May 5, 2021 •

edited

Loading

naszam May 5, 2021 •

edited

Loading

naszam May 5, 2021 •

edited

Loading

naszam May 5, 2021 •

edited

Loading

naszam May 5, 2021 •

edited

Loading

naszam May 5, 2021 •

edited

Loading

naszam May 12, 2021 •

edited

Loading

naszam May 13, 2021 •

edited

Loading

naszam May 13, 2021 •

edited

Loading