Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Futures order position tracking & FTX scaled collateral calculation #868

Merged
merged 49 commits into from
Feb 28, 2022

Conversation

gloriousCode
Copy link
Collaborator

@gloriousCode gloriousCode commented Jan 7, 2022

PR Description

This PR is based on the following branch, minus any backtester changes here.

As getting futures support working for the backtester requires many changes, I thought I'd extract some reviewable parts of my work. This PR does the following:

  • Adds a new futures position controller, responsible for tracking futures positions across all exchanges
  • Adds the ability to track futures positions across multiple orders (see "things to consider" below)
    • Tracks orders from position opening to closure, allowing for positions to be amended or having the position direction swapped
  • Calculates PNL based on orders in a position
  • Calculates collateral for FTX
  • Adds new GRPC command getfuturesposition (note: mocking my sweet PNL is banned)
gctcli> go run . --timeout=5m getfuturesposition --exchange="ftx" --asset="futures" --pair="BTC-1231"
{
 "totalOrders": 22,
 "totalRealisedPNL": "0.07871883",
 "totalPNL": "0.07871883",
 "positions": [
  {
   "status": "CLOSED",
   "realisedPNL": "0.11357933",
   "openingDate": "2021-10-31 23:38:26 +0000",
   "closingDate": "2021-11-01 02:25:43 +0000"
  },
  {
   "status": "CLOSED",
   "realisedPNL": "-0.00130672",
   "openingDate": "2021-12-02 01:44:41 +0000",
   "closingDate": "2021-12-02 01:45:32 +0000"
  },
  {
   "status": "CLOSED",
   "realisedPNL": "-0.01402298",
   "openingDate": "2021-12-06 01:25:32 +0000",
   "closingDate": "2021-12-06 01:28:35 +0000"
  },
  {
   "status": "CLOSED",
   "realisedPNL": "-0.02453374",
   "openingDate": "2021-12-06 02:18:02 +0000",
   "closingDate": "2021-12-06 02:20:41 +0000"
  },
  {
   "status": "CLOSED",
   "realisedPNL": "-0.02332539",
   "openingDate": "2021-12-06 04:13:28 +0000",
   "closingDate": "2021-12-06 04:17:35 +0000"
  },
  {
   "status": "CLOSED",
   "realisedPNL": "-0.01530603",
   "openingDate": "2021-12-06 04:29:48 +0000",
   "closingDate": "2021-12-06 04:31:09 +0000"
  },
  {
   "status": "CLOSED",
   "realisedPNL": "0.04363436",
   "openingDate": "2021-12-20 22:54:29 +0000",
   "closingDate": "2021-12-21 02:37:12 +0000"
  }
 ]
}
  • Adds new GRPC command getcollateral
gctcli> go run . getcollateral ftx futures false true
{
 "totalCollateral": "1337.5468421889777368103900077",
 "currencyBreakdown": [
  {
   "currency": "USD",
   "scaledCollateral": "1333.404998391084"
  },
  {
   "currency": "USDT",
   "scaledCollateral": "0.000000009508303729750508",
   "scaledToCurrency": "USD"
  },
  {
   "currency": "BTC",
   "scaledCollateral": "4.141841689076325",
   "scaledToCurrency": "USD"
  },
  {
   "currency": "AUD",
   "scaledCollateral": "0.0000000020176057803824997",
   "scaledToCurrency": "USD"
  },
  {
   "currency": "BNB",
   "scaledCollateral": "0.000002097291502300257",
   "scaledToCurrency": "USD"
  }
 ]
}

These features have been implemented for FTX as that's what I'm currently targeting for backtesting futures work

Things to consider

  • Futures orders are much more complicated to track how much money is being made. Long running and complicated futures positions won't be calculated properly and will be a future PR.
    • At the moment, some of the PNL calculation is done based on opening price, however, complex orders which append to a short or long after a position is already open have different prices to calculate PNL from. This is not yet happening and is solely on the opening price.
  • For PNL calculation to work accurately, you must track from your opening position! Tracking midway while you have positions open will cause problems as there will be no endpoint/an inaccurate endpoint. If this happens to you, use the optional paramter overwrite when requesting position data via GRPC. It will clear any existing data for the exchange+currencypair+asset

Type of change

  • New feature (non-breaking change which adds functionality)

How has this been tested

I've added tests to all new functions as well as added some additional coverage where it came up.

  • go test ./... -race
  • golangci-lint run
  • Test coverage of all new additions

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation and regenerated documentation via the documentation tool
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally and on Github Actions/AppVeyor with my changes
  • Any dependent changes have been merged and published in downstream modules

@gloriousCode gloriousCode added the review me This pull request is ready for review label Jan 7, 2022
@gloriousCode gloriousCode requested a review from a team January 7, 2022 02:28
@gloriousCode gloriousCode self-assigned this Jan 7, 2022
@codecov
Copy link

codecov bot commented Jan 9, 2022

Codecov Report

Merging #868 (f692f1c) into master (5c8113b) will increase coverage by 0.32%.
The diff coverage is 50.82%.

❗ Current head f692f1c differs from pull request most recent head bd73beb. Consider uploading reports for the commit bd73beb to get more accurate results

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #868      +/-   ##
==========================================
+ Coverage   44.67%   45.00%   +0.32%     
==========================================
  Files         313      314       +1     
  Lines       79926    81214    +1288     
==========================================
+ Hits        35707    36549     +842     
- Misses      38762    39122     +360     
- Partials     5457     5543      +86     
Impacted Files Coverage Δ
engine/portfolio_manager.go 43.40% <0.00%> (-1.74%) ⬇️
exchanges/account/account.go 89.28% <ø> (-0.19%) ⬇️
exchanges/alphapoint/alphapoint_wrapper.go 26.81% <0.00%> (-0.13%) ⬇️
exchanges/binance/binance_wrapper.go 42.22% <0.00%> (-0.07%) ⬇️
exchanges/bitfinex/bitfinex_wrapper.go 45.81% <0.00%> (-0.07%) ⬇️
exchanges/bithumb/bithumb_wrapper.go 42.08% <0.00%> (-0.33%) ⬇️
exchanges/bitmex/bitmex_wrapper.go 40.75% <0.00%> (ø)
exchanges/bitstamp/bitstamp_wrapper.go 47.00% <0.00%> (-0.08%) ⬇️
exchanges/bittrex/bittrex_wrapper.go 27.67% <0.00%> (-0.05%) ⬇️
exchanges/btcmarkets/btcmarkets_wrapper.go 34.61% <0.00%> (-0.11%) ⬇️
... and 46 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5c8113b...bd73beb. Read the comment docs.

Copy link
Collaborator

@shazbert shazbert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is quite a loaded PR, appreciate you pulled it out from the main work you are doing. Looks really good so far, Still need to do more tests after 3 coffees. 🚀

cmd/documentation/engine_templates/order_manager.tmpl Outdated Show resolved Hide resolved
cmd/documentation/exchanges_templates/orders.tmpl Outdated Show resolved Hide resolved
cmd/gctcli/commands.go Outdated Show resolved Hide resolved
cmd/gctcli/commands.go Outdated Show resolved Hide resolved
cmd/gctcli/commands.go Show resolved Hide resolved
exchanges/order/futures.go Outdated Show resolved Hide resolved
exchanges/order/futures.go Outdated Show resolved Hide resolved
exchanges/order/futures.go Show resolved Hide resolved
exchanges/order/futures_test.go Outdated Show resolved Hide resolved
exchanges/order/futures_types.go Outdated Show resolved Hide resolved
@gloriousCode gloriousCode added reconstructing Based on PR feedback, this is currently being reworked and is not to be merged and removed review me This pull request is ready for review labels Jan 10, 2022
@gloriousCode gloriousCode added review me This pull request is ready for review and removed reconstructing Based on PR feedback, this is currently being reworked and is not to be merged labels Jan 11, 2022
Copy link
Collaborator

@shazbert shazbert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing all the change requests.

Position tracking testing:

go run ./... --timeout=5m getfuturesposition --exchange="ftx" --asset="futures" --pair="BTC-perp"
{
 "totalOrders": 4,
 "totalRealisedPNL": "-0.2770445625",
 "positions": [
  {
   "status": "CLOSED",
   "unrealisedPNL": "0",
   "realisedPNL": "-0.1481740275",
   "openingDate": "2021-10-25 23:05:28 +0000",
   "closingDate": "2021-10-25 23:06:48 +0000"
  },
  {
   "status": "CLOSED",
   "unrealisedPNL": "0",
   "realisedPNL": "-0.128870535",
   "openingDate": "2021-10-25 23:20:30 +0000",
   "closingDate": "2021-10-25 23:20:37 +0000"
  }
 ]
}

Can confirm position return calc from front end.
image

Would be good in future to see a percentage increase or decrease as well.

Collateral calculation testing:

attempt 1 offline:

go run ./... --timeout=5m getcollateral --exchange="ftx" --asset="spot" --i --c
{
 "totalCollateral": "1.07902616",
 "currencyBreakdown": [
  {
   "currency": "USD",
   "scaledCollateral": "1.07902616"
  }
 ]
}

another attempt same account:

go run ./... --timeout=5m getcollateral --exchange="ftx" --asset="spot" --i --c
{
 "totalCollateral": "4007.3620206",
 "currencyBreakdown": [
  {
   "currency": "BTC",
   "scaledCollateral": "0",
   "scaledToCurrency": "USD"
  },
  {
   "currency": "OKB",
   "scaledCollateral": "0",
   "scaledToCurrency": "USD"
  },
  {
   "currency": "BNB",
   "scaledCollateral": "0",
   "scaledToCurrency": "USD"
  },
  {
   "currency": "USD",
   "scaledCollateral": "4007.3620206"
  },
  {
   "currency": "XRP",
   "scaledCollateral": "0",
   "scaledToCurrency": "USD"
  },
  {
   "currency": "SRM",
   "scaledCollateral": "0",
   "scaledToCurrency": "USD"
  },
  {
   "currency": "FTT",
   "scaledCollateral": "0",
   "scaledToCurrency": "USD"
  }
 ]
}

Don't know why it takes a couple requests to populate it all just yet.

Also: When we designate a sub-account via config we cannot utilize getcollateral for main or other sub-accounts, which for now is not a big issue.

exchanges/coinbasepro/coinbasepro.go Show resolved Hide resolved
exchanges/ftx/ftx_types.go Outdated Show resolved Hide resolved
exchanges/ftx/ftx_types.go Outdated Show resolved Hide resolved
exchanges/ftx/ftx_wrapper.go Outdated Show resolved Hide resolved
@shazbert shazbert added reconstructing Based on PR feedback, this is currently being reworked and is not to be merged and removed review me This pull request is ready for review labels Jan 11, 2022
@gloriousCode
Copy link
Collaborator Author

Don't know why it takes a couple requests to populate it all just yet.

From our discussions, it may have been a result of you using a subaccount, but this isn't something I have been able to replicate. If it comes up again, please provide steps to recreate. The RPCServer now returns a subaccount name on both GetCollateral and GetFuturesPosition results if it is set/specified

 go run . --timeout=5m getcollateral ftx futures false true
{
 "subAccount": "helloMoto",
 "totalCollateral": "0"
}

or

 go run . --timeout=5m getcollateral --exchange="ftx" --asset="futures" --subaccount="helloMoto" --i --c
{
 "subAccount": "helloMoto",
 "totalCollateral": "0"
}

@gloriousCode gloriousCode added review me This pull request is ready for review and removed reconstructing Based on PR feedback, this is currently being reworked and is not to be merged labels Jan 13, 2022
Copy link
Collaborator

@shazbert shazbert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for those changes, just one thing I found when getting collateral. And a pedantic suggestion. 👨‍🎤

exchanges/ftx/ftx.go Outdated Show resolved Hide resolved
exchanges/ftx/ftx_wrapper.go Outdated Show resolved Hide resolved
@gloriousCode gloriousCode added the review me This pull request is ready for review label Feb 23, 2022
@gloriousCode
Copy link
Collaborator Author

Okay, I think I've done it this time 😄 I swear! I've made it more clear and the valuations are more accurate

{
 "collateral_currency": "USD",
 "maintenance_collateral": "629.3638590701802 USD",
 "available_collateral": "625.6802840701802 USD",
 "used_collateral": "3.7975 USD",
 "used_breakdown": {
  "used_in_futures": "3.7975 USD"
 },
 "currency_breakdown": [
  {
   "currency": "USD",
   "total_funds": "623.3059901 USD",
   "available_for_use_as_collateral": "623.3061901 USD",
   "approx_fair_market_value": "1 USD",
   "weighting": "1",
   "collateral_contribution": "623.3061901 USD",
   "scaled_to_currency": "USD"
  },
  {
   "currency": "FTT",
   "excluded_from_collateral": true,
   "total_funds": "0.14464336 FTT",
   "available_for_use_as_collateral": "0.02373389 FTT",
   "approx_fair_market_value": "41.371 USD",
   "weighting": "0",
   "collateral_contribution": "0 USD",
   "scaled_to_currency": "USD",
   "used_breakdown": {
    "locked_in_stakes": "5.00214568337 USD"
   }
  },
  {
   "currency": "ETH",
   "total_funds": "0.0009998 ETH",
   "available_for_use_as_collateral": "0.0009998 ETH",
   "approx_fair_market_value": "2622.77754812 USD",
   "weighting": "0.95",
   "collateral_contribution": "2.4911403429798572 USD",
   "scaled_to_currency": "USD"
  },
  {
   "currency": "BTC",
   "total_funds": "0.0001 BTC",
   "available_for_use_as_collateral": "0.0001 BTC",
   "approx_fair_market_value": "37748.24233026 USD",
   "weighting": "0.975",
   "collateral_contribution": "3.68045362720035 USD",
   "scaled_to_currency": "USD"
  }
 ],
 "position_breakdown": [
  {
   "currency": "BTC-0325",
   "size": "0.0001",
   "open_order_size": "0",
   "position_size": "0.0001",
   "mark_price": "37975 USD",
   "required_margin": "1",
   "total_collateral_used": "3.7975 USD"
  }
 ]
}

It also works with SRM_LOCKED:

 {
   "currency": "SRM_LOCKED",
   "excluded_from_collateral": true,
   "total_funds": "48.1644741 SRM_LOCKED",
   "available_for_use_as_collateral": "0.00292357 SRM_LOCKED",
   "approx_fair_market_value": "1.927 USD",
   "weighting": "0",
   "collateral_contribution": "0 USD",
   "scaled_to_currency": "USD",
   "used_breakdown": {
    "locked_in_stakes": "92.80730787131 USD"
   }
  },

Copy link
Collaborator

@shazbert shazbert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Collateral looks good and the added lovely display currency is a nice touch! Good stuff! tACK.

Copy link
Collaborator

@thrasher- thrasher- left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can see that futures UPNL and online collateral calculations are matching up with the front-end 🎉 We're still missing the "additional collateral used" breackdown field per spot margin borrowed item, but that's something I have a ticket with FTX open ATM.

image

The locked_in_stakes value shows the serum amount in USD, but should be SRM amount * current SRM market price

Copy link
Collaborator

@shazbert shazbert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Recent changes look good and the collateral with the front end matches for me. Online compared to offline is a $1 difference in avail collateral 👍.

Copy link
Collaborator

@thrasher- thrasher- left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for making those additional changes! Collateral matches FTX front-end and also same with the futures PNL on an account with staking, margin borrowing and free assets 🎉 Only have some minor nits left

var overwrite bool
if c.IsSet("overwrite") {
overwrite = c.Bool("overwrite")
} else if c.Args().Get(2) != "" {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
} else if c.Args().Get(2) != "" {
} else if c.Args().Get(8) != "" {

var verbose bool
if c.IsSet("verbose") {
verbose = c.Bool("verbose")
} else if c.Args().Get(6) != "" {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean 7?

}

var subAccount string
if c.IsSet("subaccount") {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Subaccount is last in the ArgsUsage blurb, but second last here

Copy link
Collaborator

@thrasher- thrasher- left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tACK, nice work!

Copy link
Collaborator

@shazbert shazbert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is pretty much a tACK; just one thing I noticed:

Offline calculations have approx_fair_market_value as 0 for USD.

{
   "currency": "USD",
   "total_funds": "4018.00442214 USD",
   "available_for_use_as_collateral": "0.03206414 USD",
   "approx_fair_market_value": "0 USD",
   "weighting": "1",
   "collateral_contribution": "0.03206414 USD",
   "scaled_to_currency": "USD",
   "funds_in_use": "4017.972358 USD"
  },

Copy link
Collaborator

@shazbert shazbert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ChangesACK.

@thrasher- thrasher- merged commit 86d3724 into thrasher-corp:master Feb 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
review me This pull request is ready for review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants