ib.connect Timeout Error on apiStart (Error: Cannot find market rule for conid XXXXXXXXX at exchange) #82
Comments
|
Some other related TWS logs: these happened in huge chunks, i only copy a snippet.. |
|
That does sound odd. Generally, "no security definition found" is what IBKR says after a contract id has expired and you just can't look it up anymore. If the logs are showing error messages of You can also try running an updated gateway version and also configure it with at least 4096 MB RAM under Settings: https://investors.interactivebrokers.com/en/index.php?f=16454 — it's technically just a headless TWS, but TWS has never worked great for me anyway. |
|
Thanks for the quick response.
|
|
abit more isolation..
|
oh, I see what you're saying. If it works with NONE, then it must mean somehow your IBKR historical data can't be loaded (expired contracts again?) — but the original problem was also using NONE above too? There are bugs in TWS/Gateway where sometimes if it gets bad connection attempts or data loading attempts, the entire gateway just stops responding until a full exit/restart (but, if you restart and the next connect attempt generates another bad session, it would continue breaking i guess). Yes, the So one good usage is requesting "load ALL data but not EXECUTIONS": fetchFields=ib_async.StartupFetchALL & ~ib_async.StartupFetch.EXECUTIONS,If you wanted to figure out which fields are breaking for you, you could run through different combinations... StartupFetchALL =
StartupFetch.POSITIONS
| StartupFetch.ORDERS_OPEN
| StartupFetch.ORDERS_COMPLETE
| StartupFetch.ACCOUNT_UPDATES
| StartupFetch.SUB_ACCOUNT_UPDATES
| StartupFetch.EXECUTIONS The logic for how all those fields are read is here: Lines 2030 to 2070 in 38cf54a |
|
interesting, thats something new and actually insightful... so i identified its executions that's the problem.. at least it's properly caught.. but still without executions data im not sure whats the potential implications.. |
|
follow up
so i fetched without executions and tried to call it separately.. I'm still somewhat guessing this is nothing to do with ib_async/ib_insync my question then is more of how can we handle this more gracefully? regarding memory, i have 6000mb set for memory with 64gb ram im currently exploring to see if i can filter the execution request.. |
|
If IBKR decides to not reply to a request, the APIs can just hang indefinitely like that sometimes (it's why the regular calls have timeouts attached in different places because the client needs to timeout if the server never returns). You can also try checking the gateway API log for the exact request sent? Here's an example of a request for all executions with an empty reply (I currently have no executions). In this case, the request id for the command is the 22:57:48:068 (sync 22:57:47:507) <- 7-3-11327-0-------
22:57:48:071 (sync 22:57:47:510) -> ---�55-1-11327-Just run You can also try starting your client with more protocol detail output in API debug mode with If the request is going out exactly as above (just with a different request id) but no reply ever happens, then you're probably right your data is broken somehow inside IBKR's database for your account. Also, IBKR only saves about 1 to 2 days of request history per account, so if your problem does clear up every 2 days, maybe you are hitting limits in their system somewhere (doesn't explain the contract id problems though). and I guess a final test (just to prove it is their data problem and not a problem with our API implementation) could be trying to fetch execution history using their official library in either the Latest or Beta python versions from https://interactivebrokers.github.io/ |
|
So i tried to interpret the API logs and did some conid search on it API Log I notice that it seems to be related to yesterday's conid not today's.. then i tried the ExecutionFilter by time and slowly shift the time forward..
i somehow narrowed it to a particular trade..
so if i set the filter time to 2:02:00 i will hit it working backwards, thats the first Bag/COMB order.. since ExecutionFilter time is 'onwards'.. i cant fetch anything earlier..
i also tried adding side and exchange filter into it, which manage to pull earlier than 2:02:00 I'm not sure how all these relates.. just throwing more data points.. more insights would be appreciated.. |
|
some quick update.. I did secType="OPT" on the ExecutionFilter and it avoided the COMBO as i wished.. i think its an indirect effect of how ib_async doing all these passive fetching/requesting of data.. im not sure how from ib_async perspective this could be better handled.. one small little thing not sure if its a concern, there's no exception raised on the execution timeout.. |
Interesting! If only automatic execution fetching causes problems, you can just disable it with the Maybe we could add an extra parameter to enable custom
There is the extra option for |
|
i kinda need executions to better track commission/fees yea i missed the raiseSyncErrors.. not entire familiar with the entire ib_async codebase.. just my thought |
|
The timeout on reqExecutions has been around at least a year now and I hit it frequently. From the old ib_insync group Re: Random timeout while connecting File ~/.local/lib/python3.11/site-packages/ib_insync/ib.py:1782, in IB.connectAsync(self, host, port, clientId, timeout, readonly, account) The reqExecutions call is raising the timeout. But I don’t know why or how to fix it. Re: Random timeout while connecting from ibapi.client import * class TestApp(EClient, EWrapper): app = TestApp() I worked around it by using fetchFields=StartupFetchNONE, then making any needed calls myself and failing if reqExecutions fails. I don't remember if restarting the gateway fixes it, but I think it does. |
Oh — this may help: the startup loading only fetches the recent history, but you always get live updates of new execution status changes. You can watch these events for various state changes when orders get filled:
Exactly. Especially with PnL calculations on bags/spreads where the events only report PnL per leg, but you have to look at the full execution history for full spread PnL or run the math individually.
One of the design goals of the library is to always synchronize user positions and account details, so I think it's safe to assume the connection can't be trusted until
Thanks a lot for finding those details. I've also seen places where the gateway will accept a request then just never reply, but it all lives in their java blob app thing so we can't figure it out. A different example of the gateway not replying I can reproduce: I tried adding support for the new Event Contracts instrument and I can't get them to preview or trade (the gateway receives the request with the contract id in the regular format, no reply ever comes back), but I can get quote Tickers for them. The only place the official IBKR API docs mention Event Contracts is using their newer public web API endpoint not using the gateway, so if IBKR starts only supporting new features on a different API, we will have to either add adapters/workarounds or eventually move support to the different API too. The web API already supports more features the TWS API can't do like:
But it also appears their web API isn't designed with any real time feedback mechanism (no streaming websocket account updates, but it does have websocket quote streaming), so you have to just refresh the We could make a new package to support the web API with good defaults and automatic/transparent state management like the current library, but it would be a full time job to build it out with the same level of automation and completeness (if anybody wants to sponsor building it for a year). |
Hello, I hope you have resolved your issue. I was facing the same issue with the connection fail error but maybe for a different reason. Looks like the problem you were/are facing is due to performance of an unsupported operation that resulted in locking up your session via clientId 777. To bypass the lock, maybe try changing your clientId and basically start a new session. I use "clientId=random.randint(1, 10000)" to get a new clientId. Hopefully this helps. gl |
|
I've also run into this What is really worrisome is that this is a long-time regression apparently introduced alongside the ability to specify all or today's execution reports and it doesn't seem like IBKR is able or willing to address it. You can no longer download TWS versions that predate the introduction of this bug. |
|
Update: I was able to get When I first discovered this I thought it was some sort of timezone issue related to daylight savings/UTC offsets changing and that when I would encourage others to report this issue to IBKR support as it's existed for years and who knows when the workaround will break—being able to get your executions seems pretty important! 😅 |
Recently I'm encountering some weird connectivity errors that I've never seen before in past 2-3yrs running my bot.
I highly suspect its TWS side issue but I can't really figure out.
Some background:
Some added tests I did to give more context:
At this point i already kinda figure out its some corrupted data somehow on my username when connecting.
So I did some own investigating and try to narrow the code to replicate the issue in jupytor notebook:
Here's the errors:
The above error doesnt help much its just timeout error.
I further dug into the ib_insync/ib_async code and see the issue happening here in client.py > connectAsync():
await asyncio.wait_for(ib.client.apiStart, timeout)I even contacted the IBKR technical support but they're not in anyway helping, giving me random excuses like ib_insync not compatible and their TWS API had recent changelogs which are totally not true.
So i retrieved the TWS log in diagnostic detailed logging, this chunk always happen whenever i do a connect:
From what I have so far (above) it seems like something went missing in my account somehow and causing subsequent connect to timeout.
my current workaround if this ever happens is to keep switching the username every other day..
subscribing to marketdata on both username.. which is not the most ideal..
I'm just kinda documenting it here to see if anyone also encounter anything similar..
Or anyone have any idea what else can I try resolve this
or even anyone who understand whats going on could help explain and advise anything..
or is there something i can do on ib_async side to prevent triggering the dummy error..
any help would be deeply appreciated. thanks.
The text was updated successfully, but these errors were encountered: