Should we change non-existing prices from -1 to None?

[mattsta]

I was looking around more of the internals, and I found where unset prices are defaulted to -1 which has always caused problems in my code everywhere (because with spreads, negative prices are valid and it's annoying to check if -1 is an error or a valid credit price!)

How does everybody else think the change would impact their current systems if instead of reporting unknown prices as -1 (during after hours, unpopulated quote tickers, live bids or asks with no volume) we instead moved to having bid=None, ask=None (instead of bid=-1, ask=-1)?

I guess we could also make it a runtime option to have a "default price" setting somewhere so you could switch between -1, None, or even nan if you wanted to (or your own "UnknownPrice" class).

We'd make a new major version number for the breaking change (though, if we make it configurable no breaking change needed), but I think it would reduce the number of math bugs I've run across over the years from accidentally averaging a -1 bid into a valid ask for example.

If anybody wants to test their own code on the change locally, the line to update is here and it should turn -1 unknown prices into None everywhere:

ib_async/ib_async/wrapper.py

Line 870 in 38cf54a

price = -1.0

Update: so that's not where the value is set. It's set in a couple places, but I've found out how to make all the NaN values go away and turn into something else. I already added the fix to my own cli project and it's working as expected (only took two hours of refactoring all my accidental math against -1s everywhere (mostly in quote displays on startup as Ticker values load in async over a couple seconds)).

Current idea is just a parameter to IB() as IB(defaultEmptyValue=None) which will allow people to change the default behavior of IB(defaultEmptyValue=float('nan')).

In other news, I found some un-parsed tick types for extra streaming data I've enabled for my own testing (last trade timestamp, stock average volumes over 3, 5, 10 minutes). If those keep working for me I'll add those soon along with additional error handlers so we notice missing tick types sooner.

[geepeem]

Hi Matt, thanks for bringing this issue to our attention - I wasn't aware that -1 was the default for unset prices! I usually see 'nan' when using ticker.marketPrice() and the market is closed...

As someone who trades spreads and regularly deals with negative prices, I agree we should move away from -1. Your idea of setting a parameter at startup is a good one, and will allow flexibility for everyone.

Keep us posted please!

[mattsta]

Thanks for the feedback!

Here's an old screenshot I had showing the original price=-1 effect in some places (expand to make it less fuzzy at the original resolution).

You see IBKR returns -1 for any "non-tradable" instruments like index values which are just numbers you can't actually trade.

---

After the update, for display purposes, I manually set the None prices to nan (along with size=X instead of size=0) so it's easier to see actually missing prices (why still nan on the display though? just because it's easier since nan operates with float formatting. as a placeholder you can still use a=ask if ask is not None else nan; b=f"{a:,.2f}" where python would just throw exceptions if you try to float-format None as f"{None:,.2f}"):

---

ib_insync had always set every value in a quote Ticker to nan as a default before receiving any API data, but then IBKR sets -1 as the price for "empty value" along with size == 0. So, as users, we have to check everywhere is nan? or (is -1? AND quantity == 0?), but of course we forget all the checks in half the places (or we forget to use the isBidReallyAPrice(ticker) validation wrappers every time we want to read a price).

It's much easier to have the library itself say if API price == -1 and size == 0 then set price = None so we can just use if ask is not None: everywhere (then it becomes important to always use if ask is not None and bid is not None: in places though because, again with spreads, $0.00 prices can be valid for orders so if not ask: would cause failures if a spread ask price is actually 0.00).

[geepeem]

Thanks for clarifying: yeah it's a necessary evil to always check for valid prices everywhere to avoid potentially costly runtime issues/crashes. I rely on a little helper method that checks if ticker.hasBidAsk() == True before returning the bid/ask values, otherwise return 'None's, which are then handled by the method asking for price data. So it kinda works as you describe above, but probably not very efficient. In my case, I'm not constantly checking hundreds of tickers, only a handful, so it's not really an issue (yet!).

[gnzsnz]

thanks for bringing this for discussion.

I agree that -1 is not a good default. Personally I vote for NaN as I mainly do calculations with pandas or numpy. Another advantage of NaN is it's portability across different languages. For example a NaN can be serialized to JSON(true NaN is not valid JSON, but is a commonly used extension, python json defaults to allow NaN) and receive it in javascript or any other language.

https://stackoverflow.com/questions/6601812/sending-nan-in-json

[mattsta]

All good points.

The IBKR -1 pattern is really just because the IBKR API can only send integers as price/size updates, so they use price=-1 size=0 to mean "no data here" (but also ib_insync uses nan as a different value for "no data received from API yet" so it makes everything double confusing, then the IBKR API also has its "unset integer" and "unset double" values as INT_MAX and FLOAT_MAX in various places too...).

The only problems with NaN are equality and propagation. It makes us do things like if price == price: to see if it is NaN or not (or use a wrapper) plus it can also corrupt all math silently until sometimes it's too late if we forget a single check somewhere.

For my own work, I prefer direct errors of "no, here's an exception because you tried to do None + 3" instead of "nan + 3" working silently and generating bad results after multiple steps.

I think having it as a configurable default should make everybody happy so people can use:

• legacy -1 behavior remains as default
• letting users set nan or None
• or, perhaps even better, users could define custom NoPrice() and NoSize() objects too.

In [1]: float("nan") > 0
Out[1]: False

In [2]: float("nan") < 0
Out[2]: False

In [3]: float("nan") < float("inf")
Out[3]: False

In [4]: float("nan") > float("inf")
Out[4]: False

In [5]: float("nan") == float("nan")
Out[5]: False

In [6]: float("nan") + 100
Out[6]: nan

In [7]: float("nan") * 100
Out[7]: nan

also in more updates I've found:

• two edge cases where our wrapper incorrectly "deletes" orders from the client cache when the orders are still live — I ended up owning way too much NVDA I didn't expect as it was crashing yesterday because the library was treating API warnings from order modifications as entire order cancellation events incorrectly. This is fixed and will be updated soon.
• also going to make timezone (for trade event log entries) an optional user parameter because I like my datetime objects in US/Eastern market time and not UTC, but it will still be datetime.timezone.utc by default and users can specify a different ptz.timezone("US/Eastern") or anything else they want.

[mattsta]

Added some attempts at the change here (along with other updates) https://github.com/ib-api-reloaded/ib_async/tree/next — feel free to look / review / test / feedback.

All the details are in the expanded commit messages for now. It's on a temporary branch so we can still change anything before it hits a wider audience.

I've been running this most of this week and everything has worked nicely so far.