Output json result to file during simulation #1206

station384 · 2018-01-23T03:58:50Z

See issue #1199

JSON file is implemented in genetic_backtester

…ng-simulation

This is to improve reliability of passing information to backtested scripts.

* Fix SRSI calculation * Added test case for SRSI. Switched RSI and SRSI output to float and adjusted display accordingly * Oops, disabled all other tests by mistake * Fix test assertion

This fixes the error `/usr/bin/env: ‘node\r’: No such file or directory`

…ng-simulation

…tion

DeviaVir

LGTM, will allow space for others to comment as well.

…ult-to-file-during-simulation

firepol · 2018-01-23T14:25:25Z

@station384 thanks for this. Since you worked on saving information to a JSON file, MAYBE can you consider this? #1131

Or comment here or in my post some draft / hints so if you have no time at least I can try to implement my feature request? Sharing knowledge always helps.

Also, is the produced JSON in this commit in the .gitignore? If not please add it, else everytime you add a simulation it'll add more files you'll have to be careful not to include in future commits...

Thanks

station384 · 2018-01-23T14:44:06Z

I'll take a look at #1131.
It may be the weekend before I can get to really checking it out. Work calling, I can no longer procrastinate.....

station384 · 2018-01-23T15:50:54Z

Found an error in the new vsBuyHold calculation. correcting shortly.

Missed return result of vsBuyHold in the Sim command JSON results. vsBuyHold needs to be a percentage represented by whole numbers. i.e. 99.5 instead of 0.995.

changed json file name to new format sim_<selector>_<marketExchangePair>_<generation>.json if selector or marketExcahngePair contains ‘_’ it will be stripped this allows for easier parsing of the file name. Final additions or I’ll just keep tweaking it.

station384 · 2018-01-23T21:52:30Z

issue #1210 addressed in last update.

Bug fixes only now, or I will keep tweaking it and it will never get merged :)

DeviaVir · 2018-01-24T10:39:53Z

scripts/auto_backtester/backtester.js

 let StripAnsi = require('strip-ansi');

 let VERSION = 'Zenbot 4.04 Backtester v0.2';

-let PARALLEL_LIMIT = require('os').cpus().length;
+let PARALLEL_LIMIT = 3//require('os').cpus().length;


why not make this depending on the cpus?

If I have to take a guess this was introduced for debug purposes and then forgotten to undo

correct. it wasn't supposed to be checked in. Still figuring out git. totally breaks my source control flow ( used to SVN )

@station384 since you are new to git, I assume you are also a bit new to github, like me. Just sharing some info that I figured out also in one of my first pull requests... what I did is to merge changes from the upstream branch to my fork, commit one change and finally do a pull request comparing my fork with the official repo/branch. As I noted here, the pull request included lot of commits I didn't do. Somehow this is a weird behavior of github.

To avoid that (next time, now too late since it was merged already), at best you keep working on your branch, but just a few minutes before making the pull request you create a new local branch based on the upstream one (I try to call such branches with the name of what they do, or issue id... but the name doesn't matter). Then you cherry pick your changes or merge and make just one commit, and push your branch new branch containing just that one commit.

Then, when you do the pull request and choose the branch you just pushed in your repo, github will compare that branch with the actual one, where you want your changes to go. Like this, github picks only your changes and nothing else added in the meantime by other developers, and keeps the commit message clean (only your commit message).

I had the same issue, if you see your commit now it seems you did so many changes, but this is because github included lot of stuff really unrelated to your change...

@DeviaVir I think that this should be documented, if not in the readme, there should be a readme for contributors with guidelines to follow.

I see that there is a code of conduct, maybe it makes sense to create a DEVELOPMENT.md file or something similar, containing also such hints as the one I wrote above, useful also for people new to github (this issue isn't really documented in github)...

@firepol usually one would add a CONTRIBUTING.md but I don't think you'll find basic git usage in that document, usually. Would not be a bad place for it though.

The reason we don't have a CONTRIBUTING.md is because it's really a lot of writing and setting up general rules, which I simply haven't had the time yet to do. When I do have a bit of spare time I prefer to review PR's and play around with strats/scripts/ideas vs. writing up documentation, if you know what I mean.

If anyone reading this feels inclined to jump in and write this document, the guidelines are available here:
https://help.github.com/articles/setting-guidelines-for-repository-contributors/

I sort of figured out the branching part, but wasn't sure, for every change I want to push I was making branches off of my fork of unstable, the largest problem I've had is when I've committed something, and published. in SVN I would just check out from a prior check in and recheck in. I assume GIT is similar.

I also get messed up because in svn, I would just check in on, and then there is a separate commit where I can cherry pick the files I commit (I usually have 4 or more mods going at once in the same branch in SVN).
My SVN process seems to be what the branching in git does.

Git has more planning ahead of what you want to commit, where as SVN happens later.

Sooner or later I'll catch on, I've been using SVN for the last decade so old habits die hard, and the stuff I learned on git from about 2 years ago has to be relearned :/

But thanks for the tips, confirms a couple of things, and lay off the batch commit of files like I have been doing and start doing them very specifically as you described.

and "the pull request included lot of commits I didn't do. Somehow this is a weird behavior of github" this was bit alarming, to the point of me redoing branches and such.
I was reading up on it and it appears you can do something with rebase to eliminate them, essentially moving your point forward, but I've yet to figure it out.

@station384 for your last bit, if you have no changes in your fork's unstable branch you can do:

git reset --hard upstream/unstable git push --force

Then any time you branch off of your fork's unstable it won't have all the changes, as the head has been moved forward.
Unfortunately, you'll need to do this every time upstream (hey, that's me) updates.

TYVM that is exactly what I was looking for.
It has been driving me nuts.

Now to make that command into a git alias :)

micheledicosmo

comment deleted

micheledicosmo · 2018-01-24T10:29:09Z

scripts/auto_backtester/backtester.js

 let StripAnsi = require('strip-ansi');

 let VERSION = 'Zenbot 4.04 Backtester v0.2';

-let PARALLEL_LIMIT = require('os').cpus().length;
+let PARALLEL_LIMIT = 3//require('os').cpus().length;


now matches genetic_backtester.

station384 and others added 14 commits January 21, 2018 12:26

Merge branch 'DeviaVir/unstable' into unstable

6735c93

Add trend_bollinger

3d8b8c6

Add wavetrend strategy

5110ab6

Add StdDev

2eec559

Add Momentum Strategy

804d855

Merge branch 'DeviaVir/unstable' into Output-json-result-to-file-duri…

dadf3ec

…ng-simulation

Save options to file if backtester_generation has value

89a5685

This is to improve reliability of passing information to backtested scripts.

Fixed StochRSI calculations (#1196)

f4ace06

* Fix SRSI calculation * Added test case for SRSI. Switched RSI and SRSI output to float and adjusted display accordingly * Oops, disabled all other tests by mistake * Fix test assertion

Fix cexio update-products.sh (#1194)

688a4b2

This fixes the error `/usr/bin/env: ‘node\r’: No such file or directory`

Merge branch 'DeviaVir/unstable' into Output-json-result-to-file-duri…

c44a077

…ng-simulation

Merge branch 'DeviaVir/unstable' into unstable

ae7cc52

Merge branch 'unstable' into Output-json-result-to-file-during-simula…

718c944

…tion

Sim result passed as file instead of being scraped

0d361f6

Cleanup code and comments

28efeba

DeviaVir approved these changes Jan 23, 2018

View reviewed changes

Merge remote-tracking branch 'DeviaVir/unstable' into Output-json-res…

55a5964

…ult-to-file-during-simulation

Ignore the sim outputs

a8099ed

Correct mistake in vsBuyHold

7d8c41a

Missed return result of vsBuyHold in the Sim command JSON results. vsBuyHold needs to be a percentage represented by whole numbers. i.e. 99.5 instead of 0.995.

station384 mentioned this pull request Jan 23, 2018

backtester.js: let rawParams = jsonRegexp.exec(output2)[1]; throws TypeError: Cannot read property '1' of null #1210

Closed

DeviaVir reviewed Jan 24, 2018

View reviewed changes

micheledicosmo reviewed Jan 24, 2018

View reviewed changes

Remove debug code

5cf8906

now matches genetic_backtester.

DeviaVir approved these changes Jan 24, 2018

View reviewed changes

DeviaVir merged commit c27526d into DeviaVir:unstable Jan 24, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Output json result to file during simulation #1206

Output json result to file during simulation #1206

station384 commented Jan 23, 2018

DeviaVir left a comment

firepol commented Jan 23, 2018 •

edited

Loading

station384 commented Jan 23, 2018

station384 commented Jan 23, 2018

station384 commented Jan 23, 2018 •

edited

Loading

DeviaVir Jan 24, 2018

micheledicosmo Jan 24, 2018

station384 Jan 24, 2018

firepol Jan 24, 2018 •

edited

Loading

DeviaVir Jan 24, 2018

station384 Jan 24, 2018

DeviaVir Jan 24, 2018

station384 Jan 24, 2018

micheledicosmo left a comment •

edited

Loading

micheledicosmo Jan 24, 2018

Output json result to file during simulation #1206

Output json result to file during simulation #1206

Conversation

station384 commented Jan 23, 2018

DeviaVir left a comment

Choose a reason for hiding this comment

firepol commented Jan 23, 2018 • edited Loading

station384 commented Jan 23, 2018

station384 commented Jan 23, 2018

station384 commented Jan 23, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

firepol Jan 24, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

micheledicosmo left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

firepol commented Jan 23, 2018 •

edited

Loading

station384 commented Jan 23, 2018 •

edited

Loading

firepol Jan 24, 2018 •

edited

Loading

micheledicosmo left a comment •

edited

Loading