bcoin wallet pagination test: docker container and init script #167

pinheadmz · 2019-01-10T01:02:55Z

After checking out this branch you'll need to rebuild the bcoin container because it pulls from a different repo (my PR on Braydon's PR): https://github.com/pinheadmz/bcoin/tree/wallet-pagination-zip

docker-compose build bcoin
docker-compose up bcoin

...then run bPanel and switch to _docker client to access the changes.

The init script:

Mines 144 * 3 blocks to primary/default to activate SegWit on the regtest chain
Creates 10 wallets (randomly segwit or not) with 3 accounts each
Disperses all the funds from the initial mining evenly to all accounts
Randomly creates transactions between the accounts: Sending acct, number of outputs, receiving accts, send amount. You can see these settings in the first few lines of the init script
Mines batches of txs into blocks

Also - and this part is especially cool :-)
Even though the script mines hundreds of blocks very quickly on launch, the timestamps of these blocks are manipulated so they "appear" to be 10 minutes apart. I did this because the pagination API includes parameters for time, so I thought it'd be nice to be able to sort the txs by the "virtual" time they were mined.

bucko13 · 2019-01-15T17:27:38Z

scripts/pagination-test-wallets.js

+  const feeRate = network.minRelay * 10; // for some reason bc segwit??!!
+  const wdb = wallet.wdb;
+
+  const numInitBlocks = 144 * 3; //. to activate segwit


are we still waiting for this to be removed in bcoin? It's up as a PR right?

No PR yet, just a head-scratching issue, I could spend more time figuring it out but I don't know if that's a priority since we can just bump the fee:
bcoin-org/bcoin#666

I was talking about the activation rather than the fee: bcoin-org/bcoin#580

Oh I see! Well there ya go, still a PR. It's ok though just means there's more blocks mined at startup, more funds to disperse... You could bump numTxBlocks and numTxPerBlock if you want to spread that dough around and mke more txs to paginate...

bucko13 · 2019-01-15T17:28:36Z

scripts/pagination-test-wallets.js

+  const mineRegtestBlockToPast = async function(coinbaseAddr) {
+    const entry = await chain.getEntry(node.chain.tip.hash);
+    const job = await miner.createJob(entry, coinbaseAddr);
+    job.attempt.time = virtualNow;


very cool. Did not know about this.

yeah! I was playing with this testing actual time-based CSV transactions for the atomic swaps app/guide. Wrote a PR to test CSV-by-time for bcoin:
bcoin-org/bcoin#613

etc/regtest.bcoin.env

pinheadmz · 2019-01-18T00:19:20Z

bonus commit: fixing consensus.COINBASE_MATURITY bug in funded-dummy-wallets script

pinheadmz · 2019-01-18T00:21:04Z

This branch is OK to merge now if review is cool. Won't break anything!

bucko13 · 2019-01-18T02:26:34Z

docker-compose.yml

-    #     repo: https://github.com/bcoin-org/bcoin.git
-    #     branch: master
-    #     rebuild: 0
+    build:


I think we should should the Dockerfile-bcoin pull from the experimental branch and then we can build from the bpanel/bcoin:dev image from docker hub (I'm not actually sure you can or at least are supposed to have both image and build. I think this will cause it to pull an image from dockerhub uselessly and then build from the local image). The advantage of using it from docker hub is that the image comes prebuilt, all the bcrypto stuff, the downloading and installing is done on their servers. Then when you run docker-compose up bcoin if you use image instead of this build section, it just downloads the image and then starts bcoin. This might mean we have to fix experimental though before we can merge this :)

etc/regtest.bcoin.env

bucko13 · 2019-01-18T02:27:43Z

scripts/funded-dummy-wallets.js

@@ -1,4 +1,4 @@
-const { protocol: consensus } = require('bcoin');
+const { protocol: { consensus } } = require('bcoin');


😬 whoops!

bucko13 · 2019-01-18T04:01:43Z

scripts/pagination-test-wallets.js

+  logger.info('Creating a big mess!...');
+  for (let b = 0; b < numTxBlocks; b++) {
+    for (let t = 0; t < numTxPerBlock; t++) {
+      // TO


old comment?

Oh, nvm. I see the FROM below. Maybe a little more detail. I thought this was a half written "TODO"

bucko13 · 2019-01-18T04:02:23Z

scripts/pagination-test-wallets.js

+  await mineRegtestBlockToPast(minerReceive);
+
+  logger.info('Creating a big mess!...');
+  for (let b = 0; b < numTxBlocks; b++) {


I'm probably missing something really obvious, but what is numTxBlocks for?

const numInitBlocks = 144 * 3; // blocks mined to primary/default to activate segwit and start chain const numTxBlocks = 10; // how many blocks to fill with transactions once wallets are all set up and initial mined funds are distributed const numTxPerBlock = 10; // how many txs to generate randomly between accounts before mining a block const maxOutputsPerTx = 4; // each random tx has ranomd # of outputs const minSend = 50000; // tx amounts are also random const maxSend = 100000;

scripts/pagination-test-wallets.js

pinheadmz · 2019-01-18T07:24:07Z

Ok I think everything's cleaned up - so we can't really test until this is merged and dockerhub updates though?

bucko13 · 2019-01-18T15:57:56Z

We can test locally by uncommenting and building locally, but you won’t be able to pull the updated image from docker hub until it gets merged in.

pinheadmz · 2019-01-18T17:57:17Z

comments added to explain params

tynes · 2019-01-23T17:53:38Z

I really don't like the fact that we need to comment in/comment out parts of the config. Its a major source of problems and is just bad practice. Without some crazy sed and/or grep commands, its not possible to automate building the container locally and pulling from dockerhub without inspecting the files and writing custom scripts. Developers shouldn't have to manually edit the file and run a single command, there should just be 2 commands

pinheadmz · 2019-01-23T17:55:41Z

Well I think we can only have one init script -- maybe just remove funded-dummy-wallets altogether? This new script is more verbose anyway, maybe it should just replace?

tynes · 2019-01-23T17:55:42Z

Running into some issues. I commented in the build section of the docker-compose.yml file for bcoin and built it locally, then i ran docker-compose up bcoin, watched it generate a ton of transactions, and then it broke

pinheadmz · 2019-01-23T18:01:24Z

Huh, are you sure it rebuilt? Might need to wipe out docker container on disk already?
I blow away everything when I run it:

 docker stop $(docker ps -a -q); docker rm $(docker ps -a -q); docker volume rm `docker volume ls -q`; docker-compose up bcoin

pinheadmz · 2019-01-23T21:07:29Z

Addressed your comments: pagination test script is run at init and no messages suggesting users comment stuff out. Added a rough try/catch to prevent wallet already exist errors. Should be able to docker-compose up bcoin multiple times no prob

* clientFactory: update client conf handling (#97) * index: loading animation (#98) * bugfix: works over tls (#100) * bugfix: works over tls * bugfix: account for when port is undefined * build-plugins: Loading, docker, and plugin-config tweaks (#102) * Loading and docker tweaks * Plugins can be overridden with cli or env configs * Plugins config survives space padding * footer: update stylings, add attribution, include node pool info (#105) * feature: expose react-router to plugins (#106) * router: send supported services with client info (#107) * check for existence of client * separate config check for multisig * fix linting error * put services in object * server: Socket Manager (#103) * test scaffolding for server side tests * better logging on new socket for legacy system * SocketManager basic functionality * passing tests for subscriptions and for multiple connections * add dispatch support * add support for multiple client types and unsubscribe * migrate from socketHandler to socketManager and cleanup * package-lock * docker: hsd update (#108) * update package versions and cleanup * fixes for docker environment bugs * some cleanup * webpack: add support for user specified api keys in bpanel config directory (#112) * add support for user specified api keys in bpanel config directory * nit: declaration in function scope * store: switch network (#109) * move reducer composer into own file * update node info on network switch * working reset state * persist currentClient * fix currentClient check * update bpanel-util version * add warnings about client info chain * add all clients to socketManager * Add STATE_REFRESHED support for better signaling and constants cleanup * updated packages * routes: Client config endpoints (#113) * finished createConfig functionality with tests * update for force support * update GET endpoint for statusCheck and full configs scrubbed of keys and tokens * send which nodes are failing for GET * add delete endpoint * add PUT route * update API for more efficient network checking, and being more explicit with reporting errors (no more nested try/catch/if/else. Also updated status api to 'health' * bugfix: pull in resolves lost in rebase (#115) * server: Cleanup and remove deprecated endpoints (#114) * cleanup deprecated endpoints * remove unecessary code for default client. also moves out some config helpers into the helpers file * Modularize api (#116) * cleanup deprecated endpoints * remove unecessary code for default client. also moves out some config helpers into the helpers file * modularize api endpoints for clients * fix bug with clients POST handler * remove need for closures in endpoint handlers * cleanup * bugfix: remove bcoinuri (#119) * deps: bump bpanel-ui (#118) * server: api filters - blacklisting (#117) * cleanup deprecated endpoints * remove unnecessary code for default client. also moves out some config helpers into the helpers file * modularize api endpoints for clients * fix bug with clients POST handler * add basic blacklist support * add consistent error response * remove need for closures in endpoint handlers * flexible blacklisting with RegExp * cleanup delete handler * error catching cleanup * update custom catchall error handler * move blacklisting to top of middleware stream * deps: bump bmultisig (#121) * Backend plugins (#120) * add backend plugin endpoints * set clientsMap on request object * build-plugins: add check for correct versions of npm and node (#123) * add check for correct versions of npm and node * add warning about path and fix typo * socketmanager: socket proxy (#122) * add backend support for client side node via wsproxy * set the socket port in the app store so other parts of the app can retrieve it * set socket port from environment * Fixes (#126) * truncate table header overflow * lint fixes * update network check behavior for plugin installation * update network status check * deps: remove bad dep (#127) * SocketManager: Disconnect sockets (#124) * testing subscription handling * add support for unsubscribing on disconnect * update package-lock * add more clarifying comments and cleanup * update version of bcrypto * update nav-actions crypto algo * themeGenerator: add selectable and hover row CSS (#125) * vendor: add license (#131) * vendor: add license * bugfix * webapp: viewport tag for mobile (#133) * package.json: stability (#135) * update karma * stabalize package versions * stabalize versions * server: client configs refactor (#132) * update handling of client errors * update version of bcrypto * update nav-actions crypto algo * sanitize empty strings from client options * label styles in theme config * add package constants * optimize dll webpack build * support for node-host or host config options * add support for checking if config's chain matches with node * expose pkg file * add support to reset clients without restarting the server * add client and make bsert available via webpack * cleanup api * cleanup * add reducer to remove client config * clients updates for health/status check * update client reducer replaces whole previous client state * add new styles for Paper component * log health status message * package: update dependencies (#136) * Panel: Loadable fix (#138) * fix lint command * remove react loadable * server: add process.title (#142) * configs: ignore argv (#140) * client configs: ignore argv * server: join client prefix path * api: remove add, update, and delete config endpoints (#139) * add, update, and delete config endpoints removed from core * add handler for health check * update backend plugin api to support before _and_ after core middleware * support custom log levels * index.html: add base to html template (#145) * theme: add modal styles (#143) * basic modal styles * update styling * update paper styling * bump bpanel-ui * fixes for sidebar ordering and client health (#146) * update client and services handling * keep pathName if passed * fix empty socket bug and bump bpanel-ui * fix startup server bug * cleanup * bugfix: remove extra div (#147) * theme: remove z-index (#149) * clients: support for startup with no clients (#152) * refactor for try/catch around new connection handling * return 404 for no default client found * update server to allow for no available client configs * catch situations on server where no default config or client * handle situation on app mount where there is no default config * reset state when clearing current client * update deps * Preset 0 (#153) * upgrade from preset-3 to preset-0 * remove carrot * theme: fixes for bwallet polish (#155) * fixes to default modal, requires update for bpanel-ui * version bumps * Docker: fix builds (#157) * Docker: fix builds * remove extra whitespace * bugfix: git parse, handshake deps * bugfix: allow for nginx to route properly * bpanel: expose tls port * add details to README * add gitkeep for local plugins folder * package: bump -utils to 0.1.7 (#162) adds support for isSpv check in client. Needed for recent blocks update. * remove lookbehind (#160) * remove lookbehind * support spaces in agent string * server: Consolidate loggers (#161) * deps: remove winston plus other deps * logger: convert to blgr from winston * fix tests * working blgr w/ bpanel configs and cleanup * package.json: update dependencies (#163) * preinstall: start with settings and connection-manager plugins (#164) * start with settings and connection-manager plugins * remove bui * scripts: typo (#166) * server: multisig updates (#168) * github bmultisig and multisig serve default false * default to multisig false * fix host config bug * add clientsDir listener after endpoints are setup * docs: self signed cert helper (#172) * store: Persist state for whitelisted plugins (#174) * track state of rehydration to fix persistence with clientsHydrated * bugfix: old browserslist (#176) * bugfix: old browserslist * deps: update * update packages (#178) * update packages * update bledger * fix so that localhost is properly used as fallback for client factory (#181) * bcoin wallet pagination test: docker container and init script (#167) * delink from local datadir * use braydonf wallet-pag branch * warning in readme * switch to pinheadmz wall-pag branch * reenable local datadir * new init script wallet acct setup * working init script * revert readme * fix mining bug * randomize acct segwit bool * switch to bcoin-expirimental * fix dummy-wallets coinbase maturity * comments * workaround restarts * make pagination script default init * Fix "account not found" error (#182) * fix docker image (#185) * Patches (#186) * simpler script, update readme, and update deps * connect socket on app mount

pinheadmz added 9 commits January 7, 2019 13:20

delink from local datadir

4e1ae4f

use braydonf wallet-pag branch

7a390e5

warning in readme

a3df5e7

switch to pinheadmz wall-pag branch

e2ab6eb

reenable local datadir

9a0fbe6

new init script wallet acct setup

f98ec12

working init script

c793802

revert readme

57f2ae8

fix mining bug

07b6fa0

bucko13 reviewed Jan 15, 2019

View reviewed changes

etc/regtest.bcoin.env Show resolved Hide resolved

pinheadmz added 3 commits January 15, 2019 13:55

randomize acct segwit bool

a51e063

switch to bcoin-expirimental

d87af83

fix dummy-wallets coinbase maturity

0944130

pinheadmz changed the title ~~🚫 bcoin wallet pagination test: docker container and init script~~ bcoin wallet pagination test: docker container and init script Jan 18, 2019

bucko13 reviewed Jan 18, 2019

View reviewed changes

etc/regtest.bcoin.env Outdated Show resolved Hide resolved

bucko13 reviewed Jan 18, 2019

View reviewed changes

small changes

6f6a6e3

bucko13 reviewed Jan 18, 2019

View reviewed changes

clean up

7e78c27

pinheadmz force-pushed the pagtest branch from 604c13a to 7e78c27 Compare January 18, 2019 07:21

nits

cfb71c1

comments

754954e

pinheadmz requested a review from tynes January 18, 2019 23:12

pinheadmz added 2 commits January 23, 2019 13:03

workaround restarts

802dcb7

make pagination script default init

d26a7f4

tynes approved these changes Jan 23, 2019

View reviewed changes

tynes merged commit 5bbc36e into development Jan 23, 2019

tynes deleted the pagtest branch January 23, 2019 23:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bcoin wallet pagination test: docker container and init script #167

bcoin wallet pagination test: docker container and init script #167

pinheadmz commented Jan 10, 2019 •

edited

Loading

bucko13 Jan 15, 2019

pinheadmz Jan 15, 2019

bucko13 Jan 15, 2019

pinheadmz Jan 15, 2019

bucko13 Jan 15, 2019

pinheadmz Jan 15, 2019

pinheadmz commented Jan 18, 2019

pinheadmz commented Jan 18, 2019

bucko13 Jan 18, 2019

bucko13 Jan 18, 2019

bucko13 Jan 18, 2019

bucko13 Jan 18, 2019

bucko13 Jan 18, 2019

pinheadmz Jan 18, 2019

pinheadmz commented Jan 18, 2019

bucko13 commented Jan 18, 2019

pinheadmz commented Jan 18, 2019

tynes commented Jan 23, 2019

pinheadmz commented Jan 23, 2019

tynes commented Jan 23, 2019

pinheadmz commented Jan 23, 2019 •

edited

Loading

pinheadmz commented Jan 23, 2019

		@@ -1,4 +1,4 @@
		const { protocol: consensus } = require('bcoin');
		const { protocol: { consensus } } = require('bcoin');

bcoin wallet pagination test: docker container and init script #167

bcoin wallet pagination test: docker container and init script #167

Conversation

pinheadmz commented Jan 10, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pinheadmz commented Jan 18, 2019

pinheadmz commented Jan 18, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pinheadmz commented Jan 18, 2019

bucko13 commented Jan 18, 2019

pinheadmz commented Jan 18, 2019

tynes commented Jan 23, 2019

pinheadmz commented Jan 23, 2019

tynes commented Jan 23, 2019

pinheadmz commented Jan 23, 2019 • edited Loading

pinheadmz commented Jan 23, 2019

pinheadmz commented Jan 10, 2019 •

edited

Loading

pinheadmz commented Jan 23, 2019 •

edited

Loading