tally in sql; start generative testing

[dckc]

a 2nd, possibly 3rd implementation for #35 :

plan, status:

• download transaction info from index service
• save to .json file
• load ballot info into sqlite3 db
• load transacitons into db
• load registered voters into db
• tally in sql

postponed as #105:

• upload transactions to rchain
• tally in rholang

[dckc]

Example session:

$ node tally.js ../web/ballotexample.json kc-strip.madmode.com:7070 --save tx-by-addr.json
downloading transactions from 15 choices listed in the ballot...
downloading transactions from 19 voters...

$ python3 load_votes.py ../web/ballotexample.json tx-by-addr.json m1.db
INFO:__main__:in: <module 'json' from '/usr/lib/python3.6/json/__init__.py'> out: m1.db
INFO:__main__:(re-)created table: tx
INFO:__main__:(re-)created table: choice
INFO:__main__:inserted 15 records into choice
INFO:__main__:inserted 260 records into tx

$ sqlite3 m1.db --header --column <tally.sql 
qid         sentiment   qty       
----------  ----------  ----------
Board: DaD  no          3         
Board: DaD  abstain     2         
Board: DaD  yes         2         
Board: DoD  no          3         
Board: DoD  abstain     2         
Board: DoD  yes         2         
Board: RR   yes         3         
Board: RR   abstain     2         
Board: RR   no          2         
Board: WEC  no          3         
Board: WEC  yes         3         
Board: WEC  abstain     2         
Member Swa  abstain     2         
Member Swa  no          2         
Member Swa  yes         2  

[dckc]

The SQL tally now excludes unregistered voters. I fixed a pretty significant bug, too.

I think it doesn't handle conflicting votes with the same timestamp.

I'm inclined to take the suggestion from @leithaus to do generative testing; I've heard good things about hypothesis. Not sure whether to put that in this PR or another.

(base) ~/projects/rv2020/src/cli
21:38 connolly@jambox$ node tally.js ../web/ballotexample.json kc-strip.madmode.com:7070 --save tx-by-addr.json
downloading transactions from 15 choices listed in the ballot...
downloading transactions from 20 voters...
(base) ~/projects/rv2020/src/cli
21:44 connolly@jambox$ python load_votes.py tx-by-addr.json m1.db 
INFO:__main__:looking up rho:id:1ri71weozwuoanef9zit5p7ooafkmkzhkwo6phgaourcknbmi6ke7t
INFO:__main__:looking up rho:id:5rcmyxwu8r7yywjz4qqg4ij3pox3d96joeky1gczdpf3fkaujejdxr
INFO:__main__:in: <module 'json' from '/home/connolly/opt/miniconda3/lib/python3.7/json/__init__.py'> out: m1.db
INFO:__main__:(re-)created table: tx
INFO:__main__:(re-)created table: choice
INFO:__main__:(re-)created table: voter
INFO:__main__:inserted 15 records into choice
INFO:__main__:inserted 111 records into voter
INFO:__main__:inserted 291 records into tx
(base) ~/projects/rv2020/src/cli
21:45 connolly@jambox$ sqlite3 -header -column m1.db <tally.sql
qid         prop         addr                                                  
----------  -----------  ------------------------------------------------------
Board: DaD  abstainAddr  11112nT2XooHcCVQLEAsEJhQm6boCS5B7XQ1DBmw6ex3xveiCWRWAx
Board: DaD  noAddr       11112Cwtg2Bs4WUAYrXhL9xZXXSXr9Gn62Cty39RhUaBnqjrKkqwAZ
Board: DaD  yesAddr      1111TnFUN7eZBWXp3QQACQRRxpcS5uH5Bpf67vikWhA5e3F6ikAmU 
Board: DoD  abstainAddr  11113Y89LxqCmjDK9PUDi1dfsEcjAHbBW7mQ3Zw2yqqiwSUibaTkq 
Board: DoD  noAddr       1111JoeZHDYXqyAgo89VaidQnp7W7M9pvdkFUJTqEBU7SHKx6WF2z 
Board: DoD  yesAddr      1111rbdV9Lsw6DyMSq8ySXDacX7pRUxmVGoYho9gGtfZcQYFdAN42 
Board: RR   abstainAddr  11112CgGiNg3DdMDsYz7UikeSxh7CfFdEbDYzmoJLfS4vx3uZjm55V
Board: RR   noAddr       1111swBFUPVRwR4ugkDBCvrLwPeR1621B1cHQf3cAkNxt3Zad2eac 
Board: RR   yesAddr      1111krbAKSbyGA9vfa7w4K2pKAxZZn6qjaVEduDLWotDZ8HLt2aXR 
Board: WEC  abstainAddr  1111pKehMgsPBAiqzCSkSekXP4aUXMjY5DvtSXcz72ATP7Pm3RK9o 
Board: WEC  noAddr       11112aoa6NLYomYZro566XZVGEXyCDqeqDcp8Pzg81Ckuws6SexC99
Board: WEC  yesAddr      11112gUFvJR6JBDYJURETaWUBpEDa1EyjgRHFncEfQ4hGECnciPnhw
Member Swa  abstainAddr  111184Ab7raMAoVy6fX8JuoPFB5PggfrEWfzXE4WMzTKioFwmQMsa 
Member Swa  noAddr       11112uGayGEi57D44Drq3V4iw5WWyfXbcVvsDangRTE7TaR3J4U4FD
Member Swa  yesAddr      11112i8bYVDYcm4MSbY3d1As28uY151xoMS7AyiTvZ2YmNJ8Nw13v9
qid         fromAddr                                                toAddr                                                  prop         amount      timestamp    
----------  ------------------------------------------------------  ------------------------------------------------------  -----------  ----------  -------------
Board: DaD  11112PNUTSEf1Gujfj4u415KQLy8S3NTWYW6QxdjDAK5Hj4RbHrMti  11112nT2XooHcCVQLEAsEJhQm6boCS5B7XQ1DBmw6ex3xveiCWRWAx  abstainAddr  1           1601733861644
Board: DoD  11112PNUTSEf1Gujfj4u415KQLy8S3NTWYW6QxdjDAK5Hj4RbHrMti  1111rbdV9Lsw6DyMSq8ySXDacX7pRUxmVGoYho9gGtfZcQYFdAN42   yesAddr      1           1601733861644
Board: RR   11112PNUTSEf1Gujfj4u415KQLy8S3NTWYW6QxdjDAK5Hj4RbHrMti  11112CgGiNg3DdMDsYz7UikeSxh7CfFdEbDYzmoJLfS4vx3uZjm55V  abstainAddr  1           1601733861644
Board: WEC  11112PNUTSEf1Gujfj4u415KQLy8S3NTWYW6QxdjDAK5Hj4RbHrMti  1111pKehMgsPBAiqzCSkSekXP4aUXMjY5DvtSXcz72ATP7Pm3RK9o   abstainAddr  1           1601733861644
Member Swa  11112PNUTSEf1Gujfj4u415KQLy8S3NTWYW6QxdjDAK5Hj4RbHrMti  11112i8bYVDYcm4MSbY3d1As28uY151xoMS7AyiTvZ2YmNJ8Nw13v9  yesAddr      1           1601733861644
qid         sentiment   qty       
----------  ----------  ----------
Board: DaD  abstain     1         
Board: DoD  yes         1         
Board: RR   abstain     1         
Board: WEC  abstain     1         
Member Swa  yes         1

[dckc]

generative testing seems to be working...

,gentest.txt

[dckc]

For the AGM, running it works like this:

; node tally.js ../web/ballotexample.json status.rchain.coop --save agmtx.json
; python3 load_votes.py agmtx.json agm.db
; sqlite3 -header -column agm.db <tally.sql

b26860f