Some benchmarks. First two plots are fine. Something's up with the classifier performances.

Pdr profit

Tdr profit
(lost this one :( )

Classifier performance

Model response

Some benchmarks. Right after commit e475ee2, where I cleaned up separation of HIGH and LOW classifier models. Performance plots look a bit more sane.

• Classifier accuracy is >79%. If this is true, it's amazing

Settings: my_ppss.yaml.txt

• stop loss % = take profit % = classifier threshold % = 0.2%

Plots to ignore: pdr/tdr profit vs prob_up, when prob_up < 0.5. (So far it's only plotting probs_up_UP)

2024-06-23 12:27:21,788 sim_engine INFO Iter #5000/5000 ut=1719138000000 dt=2024-06-23_10:20 ║ prob_up=0.038 pdr_profit=3.77 up + -26.43 down = -22.66 OCEAN (cumul 2407.88 OCEAN) ║ Acc=4498/5000 =  78.62% [ 77.6%,  79.7%] prcsn=0.562 recall=0.463 f1=0.503 loss=0.591 ║ tdr_profit=$  0.00 (cumul $104.35)

Pdr profit

Tdr profit

Classifier performance

Model response

Some benchmarks, run right after previous benchmark.

• Identical my_psss.yaml file
• Except running on main branch (vs this PR). Main branch has stop loss / take profit but has a single model classifying up/down.

Pdr profit

Tdr profit

Model perf

Model response surface

Some benchmarks. Right after commit a03244e

• "Improve predictoor logic: only stake when models are confident enough"
• I've also created prob_up_MERGED, for use in predictoor bot staking, and the plots. Update plots to use that value. Therefore the plots on pdr/tdr profit vs prob_up are now sane.

Settings: same ppss as before: my_ppss.yaml.txt

• stop loss % = take profit % = classifier threshold % = 0.2%
• predictoor_ss.sim_only.others_accuracy = 0.50001

Pdr profit

Tdr profit

Classifier performance

Model response

Benchmarks, with setup like above, but for more coins. Focus on trading results.

BTC (like above)

ETH

BNB

XRP

ADA

DOGE

SOL

LTC

TRX

DOT

New experiments, with longer runs

Four different experiments

1. "reframe2" x "easy"
2. "reframe2" x "real"
3. "reframe3" x "easy"
4. "reframe3" x "real"

Where

• reframe2 = refactored code
• reframe3 = code before refactor
• easy = predictoor_ss.others_accuracy=0.50001, trader_ss.sim_only.fee_percent=0
• real = others_accuracy=0.504 (50.4% like actual accuracy), fee_percent=0.001 (0.1% like Binance spot)

Parameters that are static across runs

• Just BTC/USDT in, and out
• On 50K iterations (173 days, ending on jul 15, 2024). Vs typical benchmark of 5000 iterations.
• All thresholds are 0.2%
• max_n_train: 1000, autoregressive_n: 2
• approach: ClassifLinearRidge, weight_recent: 10x_5x, balance_classes: None, calibrate_probs: CalibratedClassifierCV_Sigmoid, train_every_n_epochs: 1

Experiment 1: "reframe2" x "easy"

All params: my_ppss.yaml.txt

Final log line: Iter #50000/50000 ut=1721001600000 dt=2024-07-15_00:00 ║pdr_profit=67.41 OCEAN (cumul 9006.65 OCEAN) ║ tdr_profit=$ -2.22 (cumul $634.74)

Predictoor profit vs time

Trader profit vs time

Experiment 2: "reframe2" x "real"

All params: my_ppss.yaml.txt

Final log line: Iter #50000/50000 ut=1721001600000 dt=2024-07-15_00:00 ║pdr_profit=66.24 OCEAN (cumul 6177.61 OCEAN) ║ tdr_profit=$ -4.43 (cumul $-11332.49)

Predictoor profit vs time

Trader profit vs time

Experiment 3: "reframe3" x "easy"

All params:
my_ppss.yaml.txt

Final log line: sim_engine INFO Iter #50000/50000 ut=1721001600000 dt=2024-07-15_00:00 ║ prob_up=0.731 pdr_profit=33.68 up + -12.43 down = 21.25 OCEAN (cumul 9901.44 OCEAN) ║ Acc=33315/50000 = 69.47% [ 69.1%, 69.9%] prcsn=0.528 recall=0.576 f1=0.528 loss=0.820 ║ tdr_profit=$ -2.31 (cumul $874.00

Predictoor profit vs time

Trader profit vs time

Experiment 4: "reframe3" x "real"

All params:
my_ppss.yaml.txt

Final log line: sim_engine INFO Iter #50000/50000 ut=1721001600000 dt=2024-07-15_00:00 ║ prob_up=0.731 pdr_profit=33.15 up + -12.43 down = 20.72 OCEAN (cumul 3682.33 OCEAN) ║ Acc=33315/50000 = 69.47% [ 69.1%, 69.9%] prcsn=0.528 recall=0.576 f1=0.528 loss=0.820 ║ tdr_profit=$ -4.61 (cumul $-17374.36)

Predictoor profit vs time

Trader profit vs time

Analysis / Discussion

• Zooming in on 5K-iteration (17.8 days) chunk of performance vs time, it might tell us "good" or "bad" behavior, depending on which 5K-iteration chunk. Therefore 5K-sims are not enough to draw much conclusion; we need 50K (178 days).
• In "real" settings, both reframe2 & reframe3 made $ predicting, and lost $ trading. reframe2 did slightly better than reframe3.
• Q: are these results better than if just 1 model (status quo)? Need to run those experiments too...