IBC: Detected dialog entitled: API client needs write access action confirmation #10
Comments
|
You need to ensure that the 'Read-Only API' setting in the Gateway's API configuration is not checked. |
|
Is there any way to automate that? |
|
No. It's a setting that only needs to be made once. One way to do it is to set up the Gateway on a local machine, login to the target account and and make sure all the settings you need are properly configured. Then locate the Gateway's configuration file. To do this, look in the launcher.log file in the Jts directory, and the last line shows the user directory. In that directory you'll find a file called ibg.xml, and that contains all the settings for your gateway configuration (and you'll find the readOnlyApi setting in there). You'll then need to make arrangements to get this file into your Docker image after installing Gateway and before running it. Another way is to copy the entire Gateway installation into the Docker image, rather than installing Gateway from scratch each time. |
|
@rlktradewright Thanks for the pointers! I'll update this thread with what I end up doing |
|
Based on my reading of ConfigureTwsApiPortTask.java it would not be too hard to implement a class which does this. Would a PR for this be welcome? Although now that I consider it, there are quite a number of options and perhaps rather than implementing a class to configure each one of them, copying in a configuration file would be more scalable / maintainable. |
|
I was able to get the right information from the launcher.log file. If anybody is curious, here is what I ended up with. #!/bin/bash
### entrypoint.sh
set -euxo pipefail
IBC_INI=$IBC_PATH/config.ini
LOG_PATH=$IBC_PATH/logs
# don't override the defaults
JAVA_PATH=""
FIXUSERID=""
FIXPASSWORD=""
cat > $IBC_INI <<EOF
LogToConsole=yes
FIX=no
IbLoginId=$TWSUSERID
IbPassword=$TWSPASSWORD
PasswordEncrypted=no
FIXLoginId=
FIXPassword=
FIXPasswordEncrypted=yes
TradingMode=$TRADING_MODE
IbDir=
StoreSettingsOnServer=no
MinimizeMainWindow=no
ExistingSessionDetectedAction=primary
AcceptIncomingConnectionAction=accept
ShowAllTrades=no
OverrideTwsApiPort=
ReadOnlyLogin=no
AcceptNonBrokerageAccountWarning=yes
IbAutoClosedown=no
ClosedownAt=
AllowBlindTrading=yes
DismissPasswordExpiryWarning=yes
DismissNSEComplianceNotice=yes
SaveTwsSettingsAt=
CommandServerPort=
ControlFrom=
BindAddress=
CommandPrompt=
SuppressInfoMessages=yes
LogComponents=never
EOF
APP=GATEWAY
TWS_CONFIG_PATH="$TWS_PATH"
export TWS_MAJOR_VRSN
export IBC_INI
export TRADING_MODE
export IBC_PATH
export TWS_PATH
export TWS_CONFIG_PATH
export LOG_PATH
export TWSUSERID
export TWSPASSWORD
export FIXUSERID
export FIXPASSWORD
export JAVA_PATH
export APP
if [[ "$(echo ${APP} | tr '[:lower:]' '[:upper:]')" = "GATEWAY" ]]; then
gw_flag=-g
fi
# a poor man's logrotate
if [ -e "$TWS_PATH/launcher.log" ]
then
cat $TWS_PATH/launcher.log >> $TWS_PATH/launcher.log.old
rm $TWS_PATH/launcher.log
fi
# bootstrap a gateway instance to find out where the user directory is
timeout --preserve-status -k 60s 60s \
xvfb-run -a "${IBC_PATH}/scripts/ibcstart.sh" \
"${TWS_MAJOR_VRSN}" ${gw_flag} \
"--tws-path=${TWS_PATH}" \
"--tws-settings-path=${TWS_CONFIG_PATH}" \
"--ibc-path=${IBC_PATH}" \
"--ibc-ini=${IBC_INI}" \
"--java-path=${JAVA_PATH}" &
PID=$!
sleep 1
tail --pid=$PID -n+1 -f $TWS_PATH/launcher.log | \
grep --line-buffered 'ibgateway.log' | \
while read needle
do
perl -pe "s/.*'(.*)\/ibgateway.log'.*/\\1/" <<< $needle > /tmp/jts_user_dir
kill $PID
done
e=$(wait $PID; echo $?)
# if it timed out then throw
if [ $e -eq 124 ] || [ $e -eq 137 ]
then
exit $e
fi
user_dir=$(cat /tmp/jts_user_dir)
# copy our hand-crafted settings file into the user directory
cp $TWS_PATH/ibg.xml $user_dir/
# do it all over again this time with hand-crafted settings
xvfb-run -a "${IBC_PATH}/scripts/ibcstart.sh" \
"${TWS_MAJOR_VRSN}" ${gw_flag} \
"--tws-path=${TWS_PATH}" \
"--tws-settings-path=${TWS_CONFIG_PATH}" \
"--ibc-path=${IBC_PATH}" \
"--ibc-ini=${IBC_INI}" \
"--java-path=${JAVA_PATH}"### Dockerfile
FROM debian:stable-slim
RUN apt-get update && apt-get install -y \
curl \
unzip \
xvfb \
libxtst6 \
libxrender1 \
libxi6 \
procps
RUN curl -o install-ibgateway.sh "https://download2.interactivebrokers.com/installers/ibgateway/stable-standalone/ibgateway-stable-standalone-linux-x64.sh"
RUN echo 'n' | sh install-ibgateway.sh
RUN curl -L -o ibc.zip "https://github.com/IbcAlpha/IBC/releases/download/3.6.0/IBCLinux-3.6.0.zip"
RUN unzip ibc.zip -d /opt/ibc && chmod +x /opt/ibc/*.sh /opt/ibc/scripts/*.sh
ENV IBC_PATH /opt/ibc
ENV TWS_MAJOR_VRSN 963
ENV TWS_PATH /root/Jts
ENV TZ America/Los_Angeles
COPY entrypoint.sh .
COPY jts.ini $TWS_PATH/
COPY ibg.xml $TWS_PATH/
CMD ["./entrypoint.sh"] |
|
Yes, you hit the nail on the head, and this note is really just to amplify what you said. It would actually be straightforward to automate setting ReadOnlyApi to false, but I've always rejected this suggestion so far. My summary reason for this is: 'IBC is not intended to be a tool for configuring TWS/Gateway'. The longer version of this is as follows: TWS and Gateway have a very large number of configurable options (if you browse through the TWS or Gateway settings files, you'll see what I mean!). Of course the number of options for TWS is much greater than for Gateway, because it includes things like layouts of ticker pages etc, but the number for Gateway is still significant. The defaults are mostly sensible, so that if you want to run TWS or Gateway in the same machine as your API applications, and you don't want any remote access to the API (ie no Trusted IP addresses), and you're happy with the default port numbers, etc etc, then you may be able to get away with running them straight out of the box with no configuration. But for many (if not most?) API users this may not be the case, and therefore they need to explicitly configure TWS/Gateway to meet their needs - and in general this only needs to be done once (I haven't touched my live TWS config for years, but I fiddle with the paper-trading TWS/Gateway quite a bit). The really annoying thing is that when IB introduced the Read-Only API setting (which in itself is not a bad idea, though I would prefer to have this option on a per-Trusted IP Address basis), they set it to True by default for both TWS and Gateway: but this is a stupid default for Gateway, because a read-only Gateway is practically useless for most people. Consequently every Gateway user is forced to do at least one configuration action on Gateway, ie set Read-Only API to false, though it only ever needs to be done once provided the settings file ibg.xml is always accessible every time Gateway starts up. For most users, this simply isn't an issue. Now, if I allow IBC to automate setting Read-Only API to false, that will only help those users who want to install Gateway from scratch with a clean default config every time they run it, and I suspect there are few such users. And it wouldn't surprise me at all to find that users who start out like that then find that the other defaults don't meet their needs, so they will then have to find a means of configuring the Gateway appropriately and then carrying the ibg.xml settings file forward each time they start it. What I want to avoid is people then suggesting that IBC could be enhanced to set all the other options like Trusted IP Addresses: no, 'IBC is not intended to be a tool for configuring TWS/Gateway'. |
|
Thanks for the insights @rlktradewright. I understand why you want to keep a separation between having IBC control TWS and configure it. I think that adding a couple helper functions to IBC may go a long way here - specifically, either being able to have TWS show the username it generates or having IBC copy in a user-specified configuration file. |
|
@charles-cooper thanks very much for those suggestions. I know that the problems of getting a containerised IBC up and running has caused quite a bit of difficulty for those who have attempted it, but it certainly seems that getting either the Gateway/TWS configuration file or the whole Gateway/TWS installation into the target container is the key move. But I've never before considered whether, and how, IBC itself might be able to assist in that operation (without going as far as doing the configuration itself, which is a road I really don't want to go down). As always, there are several obvious possibilities (that I haven't thought through yet in the few minutes since seeing your post), and probably several others that won't occur to me, so suggestions are welcome, or even experimental PRs. I think it would be best to raise a new issue specifically to address this topic, so I'll do that within the next few minutes. |
|
Unfortunately it appears that the name of the directory is no longer appearing in The internet says something about IB encrypting its logs due to GDPR requirements... I'll look for another way to grab the directory for the |
|
I was able to get it to work with inotify, as follows: #!/bin/bash
set -Eeuxo pipefail
IBC_INI=$IBC_PATH/config.ini
LOG_PATH=$IBC_PATH/logs
#JAVA_PATH=""
#FIXUSERID=""
#FIXPASSWORD=""
APP=GATEWAY
TWS_CONFIG_PATH="$TWS_PATH"
#export TWS_MAJOR_VRSN
#export IBC_INI
#export IBC_PATH
#export TWS_PATH
#export TWS_CONFIG_PATH
#export LOG_PATH
#export FIXUSERID
#export FIXPASSWORD
#export JAVA_PATH
#export APP
export TRADING_MODE
export TWSUSERID
export TWSPASSWORD
export TWSPORT
bash ibc_ini.template.sh > $IBC_INI
if [[ "$(echo ${APP} | tr '[:lower:]' '[:upper:]')" = "GATEWAY" ]]; then
gw_flag=-g
fi
IBC_VRSN=$(cat $IBC_PATH/version)
export IBC_VRSN
# a poor man's logrotate
if [ -e "$TWS_PATH/launcher.log" ]
then
cat $TWS_PATH/launcher.log >> $TWS_PATH/launcher.log.old
rm $TWS_PATH/launcher.log
fi
# bootstrap a gateway instance to find out where the user directory is
timeout --preserve-status -k 60s 60s \
xvfb-run -a "${IBC_PATH}/scripts/ibcstart.sh" \
"${TWS_MAJOR_VRSN}" ${gw_flag} \
"--tws-path=${TWS_PATH}" \
"--tws-settings-path=${TWS_CONFIG_PATH}" \
"--ibc-path=${IBC_PATH}" \
"--ibc-ini=${IBC_INI}" &
XVFB_PID=$!
# Split out into another process so we can kill it cleanly
# from within the while loop
inotifywait --monitor --recursive -o /tmp/inotify.log ~/Jts/ &
INOTIFY_PID=$!
tail -n+1 --pid=$INOTIFY_PID --retry -f /tmp/inotify.log | \
grep --line-buffered -P '.* MODIFY ibgateway.*ibenc' | \
while read needle
do
cut -f1 -d' ' <<< $needle | tee /tmp/jts_user_dir
kill $INOTIFY_PID
break
done \
|| echo Break # It's late at night, no idea how to do this more cleanly.
kill $XVFB_PID
e=$(wait $XVFB_PID; echo $?)
# if it timed out then throw
if [ $e -eq 124 ] || [ $e -eq 137 ]
then
echo Needle not found, exiting. >&2
exit $e
fi
user_dir=$(cat /tmp/jts_user_dir)
# copy our hand-crafted settings file into the user directory
cp $TWS_PATH/ibg.xml $user_dir/
# do it all over again this time with hand-crafted settings
xvfb-run -a "${IBC_PATH}/scripts/ibcstart.sh" \
"${TWS_MAJOR_VRSN}" ${gw_flag} \
"--tws-path=${TWS_PATH}" \
"--tws-settings-path=${TWS_CONFIG_PATH}" \
"--ibc-path=${IBC_PATH}" \
"--ibc-ini=${IBC_INI}"This needed addition of diff --git a/ibcontroller/Dockerfile b/ibcontroller/Dockerfile
index 7c7f6c3..9dfd614 100644
--- a/ibcontroller/Dockerfile
+++ b/ibcontroller/Dockerfile
@@ -2,6 +2,7 @@ FROM debian:stable-slim
RUN apt-get update && apt-get install -y \
curl \
+ inotify-tools \
unzip \
xvfb \
libxtst6 \EDIT 05/08/2019: #!/bin/bash
set -exuo pipefail
cat <<EOF
LogToConsole=yes
FIX=no
IbLoginId=$TWSUSERID
IbPassword=$TWSPASSWORD
PasswordEncrypted=no
FIXLoginId=
FIXPassword=
FIXPasswordEncrypted=yes
TradingMode=$TRADING_MODE
IbDir=
StoreSettingsOnServer=no
MinimizeMainWindow=no
ExistingSessionDetectedAction=secondary
AcceptIncomingConnectionAction=accept
ShowAllTrades=no
OverrideTwsApiPort=$TWSPORT
ReadOnlyLogin=no
AcceptNonBrokerageAccountWarning=yes
IbAutoClosedown=no
ClosedownAt=
AllowBlindTrading=yes
DismissPasswordExpiryWarning=yes
DismissNSEComplianceNotice=yes
SaveTwsSettingsAt=
CommandServerPort=
ControlFrom=
BindAddress=
CommandPrompt=
SuppressInfoMessages=yes
LogComponents=never
EOF |
|
Thanks for this bombshell! What are IBKR thinking? What use is a logfile that we can't look at? It doesn't even contain anything particularly personal, except the account number (which also appears in the api logs, and they're not encrypted). And it's entirely local to your own machine, not publicly accessible. I don't believe GDPR is of the slightest relevance here, but, as they say, 'I am not a lawyer'. I frequently look in the TWS and Gateway log files and they've provided masses of information in the past that have helped me to understand various things about how TWS works. Now I'm going to have one arm tied behind my back... Anyway, thanks for your update, I'll try to find the time to understand it some time soon! |
|
Well, it looks like IBKR have changed their minds about this in the last 20 minutes. I tried loading up the latest TWS 20 minutes ago and it did indeed use an encrypted logfile. But I did it again just now, and it has reverted back to non-encrypted logfiles! So perhaps they deployed this 'feature' in error, or they've realised the stupidity of what they had done and have fixed it. I'll check again later to see if it's still ok. |
|
Well, that was using the demo account. I've just tried restarting my paper account TWS (using TWS 972) and that is still creating an encrypted log file. So perhaps they're in the process of rolling back this silly update, but haven't got to paper accounts yet... |
|
The closest thing I could find to an explanation was this forum post: https://www.elitetrader.com/et/threads/ib-front-ends-buttontrader-rip.323699/. |
When I try to place an order via API it fails with the following error:
The IBC logs say
More info:
I am running IBC inside docker in headless mode. IB Gateway version is 963, IBC version is 3.6.0. It's possible I made mistakes in porting over from IBController - but I didn't find any mention of this particular message in the source code, groups or issues, so I figure I should report it.
Full dockerfile:
Full jts.ini
Full entrypoint.sh:
Full log
The text was updated successfully, but these errors were encountered: