[not an issue with ibeam] Account seems to be limited after a few days of running ('Gateway session active but not authenticated') #19

beanhedge · 2021-04-25T19:00:24Z

First of all, thank you so much for this project, it's been a great help.

I just wanted to report that my account seems to have been limited after running the container 24/7 over 4 days. (This isn't a problem with ibeam because manually running the client has the same issue. I just wanted to put it here to warn others.)

It's been running fine the past few days with:

2021-04-23 04:40:46,760|I| Gateway session found but not authenticated, authenticating...
2021-04-23 04:40:52,701|I| Login succeeded
2021-04-23 04:40:53,809|E| Gateway session active but not authenticated
2021-04-23 04:41:46,761|I| Gateway running and authenticated
2021-04-23 04:42:46,777|I| Gateway running and authenticated
2021-04-23 04:43:46,766|I| Gateway running and authenticated
2021-04-23 04:44:46,764|I| Gateway running and authenticated

but today it started doing

2021-04-25 18:16:55,886|E| Gateway session active but not authenticated
2021-04-25 18:17:46,775|I| Gateway session found but not authenticated, authenticating...
2021-04-25 18:17:53,064|I| Login succeeded
2021-04-25 18:17:54,168|E| Gateway session active but not authenticated
2021-04-25 18:18:46,784|I| Gateway session found but not authenticated, authenticating...
2021-04-25 18:18:53,107|I| Login succeeded
2021-04-25 18:18:54,212|E| Gateway session active but not authenticated
2021-04-25 18:19:46,765|I| Gateway session found but not authenticated, authenticating...
2021-04-25 18:19:52,941|I| Login succeeded
2021-04-25 18:19:54,047|E| Gateway session active but not authenticated
2021-04-25 18:20:46,804|I| Gateway session found but not authenticated, authenticating...
2021-04-25 18:20:53,058|I| Login succeeded

I started the portal manually on my computer to check what was going on. Login through the web page still works, I still get "Client login succeeds". https://localhost:5000/v1/portal/sso/validate and https://localhost:5000/v1/portal/tickle, https://localhost:5000/v1/portal/portfolio/accounts still return 200. However, ticker shows "authenticated: false".
https://localhost:5000/v1/portal/iserver/marketdata/history and https://localhost:5000/v1/portal/iserver/accounts
gives

401
{"error":"not authenticated","statusCode":401}

The text was updated successfully, but these errors were encountered:

Voyz · 2021-04-26T05:45:00Z

Hey @beanhedge welcome to IBeam! 😊

Thanks for reporting your issues. Unfortunately, like you correctly observe this doesn't seem to be IBeam failing but Gateway (or IBKR server) that's not handling authentication well. IBeam checks for that 'authenticated' flag and attempts to reauthenticate each time it sees it set to False.

To highlight the importance of these issues and possibly accelerate improvements, I'd suggest you submit a ticket with IBKR support highlighting this problem.

If you have any further information on why this issue seems to be appearing - for instance some repeated pattern - let us know. Otherwise the current advice is to just wait it through - it oftentimes resolves itself after some time.

Let me know if I could help you in any way here 👍
Thanks! 👋 Voy

beanhedge · 2021-05-14T00:38:21Z

The account issue resolved itself after a couple days. They did email me back and told me to try:

1. Please open conf.yaml under the root folder.
2. Change the proxyRemoteHost to “https://1.api.ibkr.com”

I didn't try it because my issue was already resolved.

However, today I hit a new issue with the market data history endpoint:
{'error': 'Too many active charts. Cannot create a new one. Please unsubscribe from old subscriptions to allow new ones. ', 'statusCode': 200}

I ran a get request against "https://localhost:5000/v1/api/iserver/marketdata/unsubscribeall" and got {'unsubscribed': True} as well as the conid/unsubscribe endpoint but it didn't help. I emailed them again and will post what they say.

Voyz · 2021-05-15T18:51:28Z

That's useful, I'll see if using that different proxy URL helps at all - thanks a lot for posting this update.

That new error you're running into sounds more like what the error itself says. Maybe try through TWS?

beanhedge · 2021-05-16T18:02:34Z

I heard back from them today, I got:

Thank you for contacting Interactive Brokers.

We are aware of the error happening. We have forwarded our findings to the developers.
Your logs would help us finding a fix faster. Can you submit HAR logs and Gateway logs to [email protected] . You can find how to submit HAR files here: LINK../3512. The Gateway logs are available here: ..\CPWEBGW\logs
Unfortunately, no solution or workaround is available at this time.

If you have further question please contact us: ibkr.com/clientservices

Regards,
Roman P.
IBKR Client Services - Technical Assistance Center, API

So I guess there's nothing we can do.

Greatsamps · 2021-07-18T07:46:14Z

We have just started to play with this project, and are getting the above error straight off the bat. We are NOT getting the error however when running the windows gateway locally.

Just to be clear, the docker container has never been able to successfully establish authentication it would seem.

2021-07-18 07:41:18,948|I| Gateway started with pid: 13
2021-07-18 07:41:25,847|I| No active sessions, logging in...
2021-07-18 07:41:43,845|I| Authentication process succeeded
2021-07-18 07:41:46,899|E| Gateway session active but not authenticated
2021-07-18 07:41:47,397|I| ############ Starting IBeam version 0.3.0 ############
2021-07-18 07:41:47,442|I| Starting maintenance with interval 60 seconds
2021-07-18 07:42:52,596|I| Gateway session found but not authenticated, authenticating...
2021-07-18 07:43:03,647|I| Authentication process succeeded
2021-07-18 07:43:06,705|E| Gateway session active but not authenticated
2021-07-18 07:43:52,988|I| No active sessions, logging in...
2021-07-18 07:44:05,079|I| Authentication process succeeded
2021-07-18 07:44:08,133|E| Gateway session active but not authenticated
2021-07-18 07:44:52,998|I| No active sessions, logging in...
2021-07-18 07:45:04,500|I| Authentication process succeeded
2021-07-18 07:45:07,675|E| Gateway session active but not authenticated
2021-07-18 07:45:52,588|I| Gateway session found but not authenticated, authenticating...

Voyz · 2021-07-18T11:36:34Z

Hey @Greatsamps thanks for your contribution. It's interesting to hear that you notice the Gateway working correctly when initialised without IBeam. Could I ask you to run IBeam with LOG_LEVEL set to debug and pass over the program output produced?

Greatsamps · 2021-07-18T12:41:24Z

Like this:

-e LOG_LEVEL=debug

Made no difference to output..

Voyz · 2021-07-18T16:14:47Z

Sorry for the confusion, running it in debug requires you to specify it as a variable used by the Python 'logging' module, therefore you need to put DEBUG in all caps. See here for more:

https://github.com/Voyz/ibeam/wiki/IBeam-Configuration#debugging

Greatsamps · 2021-07-18T16:38:10Z

Ok got it working, it's IBEAM_LOG_LEVEL that needed to be set.

incase this matters, this is a paper trading account.

output below:

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by io.netty.util.internal.ReflectionUtil (file:/srv/clientportal.gw/build/lib/runtime/netty-common-4.1.15.Final.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of io.netty.util.internal.ReflectionUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
-> mount demo on /demo
Java Version: 11.0.9.1

version: ed4af2592e9dd4a784d5403843bd18292fd441ea Fri, 9 Nov 2018 13:23:18 -0500

This is a Beta release of the Client Portal Gateway
for any issues, please contact [email protected]
and include a copy of your logs

https://www.interactivebrokers.com/api/doc.html

Open https://localhost:5000 to login
App demo is available after you login under: https://localhost:5000/demo/
2021-07-18 16:34:44,828|I| Gateway started with pid: 12
2021-07-18 16:34:44,828|D| HTTPS (unverified) request to: https://localhost:5000
2021-07-18 16:34:44,832|D| Cannot ping Gateway. Retrying for another 20 seconds
2021-07-18 16:34:45,833|D| HTTPS (unverified) request to: https://localhost:5000
2021-07-18 16:34:51,702|D| Gateway connection established
2021-07-18 16:34:51,702|D| HTTPS (unverified) request to: https://localhost:5000/v1/api/tickle
2021-07-18 16:34:51,822|I| No active sessions, logging in...
2021-07-18 16:34:51,823|D| Loading auth webpage at https://localhost:5000/sso/Login?forwardTo=22&RL=1&ip2loc=on
2021-07-18 16:34:59,350|D| Gateway auth webpage loaded
2021-07-18 16:34:59,350|D| Login attempt number 1
2021-07-18 16:35:04,509|D| Submitting the form
2021-07-18 16:35:06,416|D| Webpage displayed "Client login succeeds"
2021-07-18 16:35:08,418|D| Cleaning up the resources. Display: <pyvirtualdisplay.display.Display object at 0x7fb70e5e99d0> | Driver: <selenium.webdriver.chrome.webdriver.WebDriver (session="334adb27932b8065518ee13664eabb0e")>
2021-07-18 16:35:08,472|I| Authentication process succeeded
2021-07-18 16:35:11,475|D| HTTPS (unverified) request to: https://localhost:5000/v1/api/tickle
2021-07-18 16:35:11,525|E| Gateway session active but not authenticated
2021-07-18 16:35:12,027|I| ############ Starting IBeam version 0.3.0 ############
2021-07-18 16:35:12,031|D| {'INPUTS_DIR': '/srv/inputs/', 'OUTPUTS_DIR': '/srv/outputs', 'GATEWAY_DIR': '/srv/clientportal.gw', 'CHROME_DRIVER_PATH': '/srv/chrome_driver/chromedriver', 'GATEWAY_STARTUP': 20, 'GATEWAY_PROCESS_MATCH': 'ibgroup.web.core.clientportal.gw.GatewayStart', 'MAINTENANCE_INTERVAL': 60, 'SPAWN_NEW_PROCESSES': False, 'LOG_LEVEL': 'DEBUG', 'LOG_TO_FILE': True, 'REQUEST_RETRIES': 1, 'REQUEST_TIMEOUT': 15, 'GATEWAY_BASE_URL': 'https://localhost:5000', 'ROUTE_AUTH': '/sso/Login?forwardTo=22&RL=1&ip2loc=on', 'ROUTE_USER': '/v1/api/one/user', 'ROUTE_VALIDATE': '/v1/portal/sso/validate', 'ROUTE_REAUTHENTICATE': '/v1/portal/iserver/reauthenticate?force=true', 'ROUTE_AUTH_STATUS': '/v1/api/iserver/auth/status', 'ROUTE_TICKLE': '/v1/api/tickle', 'USER_NAME_EL_ID': 'user_name', 'PASSWORD_EL_ID': 'password', 'SUBMIT_EL_ID': 'submitForm', 'ERROR_EL_ID': 'ERRORMSG', 'SUCCESS_EL_TEXT': 'Client login succeeds', 'OAUTH_TIMEOUT': 15, 'PAGE_LOAD_TIMEOUT': 15, 'ERROR_SCREENSHOTS': False, 'MAX_FAILED_AUTH': 5, 'MAX_IMMEDIATE_ATTEMPTS': 10, 'TWO_FA_EL_ID': 'twofactbase', 'TWO_FA_INPUT_EL_ID': 'chlginput', 'TWO_FA_HANDLER': None, 'STRICT_TWO_FA_CODE': True}
2021-07-18 16:35:12,078|I| Starting maintenance with interval 60 seconds
2021-07-18 16:36:12,078|D| Maintenance
2021-07-18 16:36:12,080|D| HTTPS (unverified) request to: https://localhost:5000/v1/api/tickle
2021-07-18 16:36:17,238|I| Gateway session found but not authenticated, authenticating...
2021-07-18 16:36:17,239|D| Loading auth webpage at https://localhost:5000/sso/Login?forwardTo=22&RL=1&ip2loc=on
2021-07-18 16:36:19,792|D| Gateway auth webpage loaded
2021-07-18 16:36:19,792|D| Login attempt number 1
2021-07-18 16:36:24,945|D| Submitting the form
2021-07-18 16:36:26,181|D| Webpage displayed "Client login succeeds"
2021-07-18 16:36:28,183|D| Cleaning up the resources. Display: <pyvirtualdisplay.display.Display object at 0x7f0260867e50> | Driver: <selenium.webdriver.chrome.webdriver.WebDriver (session="582f791a294bde309c74704ba93d04de")>
2021-07-18 16:36:28,236|I| Authentication process succeeded
2021-07-18 16:36:31,240|D| HTTPS (unverified) request to: https://localhost:5000/v1/api/tickle
2021-07-18 16:36:31,292|E| Gateway session active but not authenticated

Voyz · 2021-07-18T20:04:29Z

Great, thanks for posting this. Everything seems to be in order, which would point at a similar issue that @beanhedge encountered. It's interesting that you say that running the Gateway manually doesn't cause this issue.

In order to verify if that indeed is the case, could I ask you to do the following in short succession:

Start IBeam and note if this issue persist, then stop IBeam.
Start the Gateway manually without IBeam
Performing a tickle request on it once it's running and write down the response of the tickle request.
Stop the Gateway that's running manually
Start IBeam once again and see if the issue continues to persist.

Thank you!

Greatsamps · 2021-07-20T11:38:10Z

Hi,

Sorry for delay on this. Here is output from step 3:

{"session":"b42059b2586ea798f8219b8ca26aa332","ssoExpires":594650,"collission":false,"userId":65305922,"iserver":{"tickle":true,"authStatus":{"authenticated":false,"competing":false,"connected":true,"message":"","MAC":"08:F1:EA:83:2D:DC"}}}

It appears you are using the beta version of the GW whereas i am using the stable one. Could this be the difference?

Greatsamps · 2021-07-20T12:10:43Z

I have found something else strange. It seems running the GW locally i can't access any of the iserver endpoints. When i call iserver/auth/status i get the following:

{
"authenticated": false,
"competing": false,
"connected": true,
"message": "",
"MAC": "08:F1:EA:83:2D:DC",
"fail": "xxx can't authenticate"
}

This is a paper trading account, do you think that could be the problem, and perhaps this is also why IBeam is complaining?

Voyz · 2021-07-20T14:25:37Z

Thanks for doing this test and for passing over the results 👍

I don't think the paper account should have any influence on it, as I've noticed this error appearing on both paper and live accounts. We tried upgrading to beta version at some point but it didn't seem to solve the issues at hand so we reverted back to the stable from 9 Nov 2018 which I believe is the latest. That looks to me like the same issue as before - there's something failing at IBKR side, and even your vanilla Gateway fails to authenticate.

I'd recommend you get in touch with the IBKR support and try to work with them to figure out why the authentication fails despite providing correct credentials. I'm sorry I cannot be of more help, but this is a known problem that we cannot jump over on IBeam side despite multiple attempts. Feel free to suggest any fixes if you can think of any.

kavindu-athaudha · 2021-08-19T10:34:32Z

Hi Voyz,

Need to start this out by thanking you for the hard work you've done; you've really done a huge addition to everyone trying to use IBKR's REST API.

I faced this issue as well and I just contacted IBKR support. What they said is that it's a known issue and that they're working on it. But by the looks of this thread, this has been going on for months but to no avail.

IBKR support requested me to try the beta version, but this thread has shown me that that won't work as well.
It doesn't seem like Interactive Brokers is taking priority on this significant issue.

Is there any suggestion we, as a community, can do to pressure Interactive Brokers to fixing this?

Voyz · 2021-08-19T16:11:15Z

hey @kavindu-athaudha welcome and thanks for your super kind words! 😊

Thanks for sharing your story about the conversation with IBKR. Indeed, we're around 7 months into these issues appearing regularly and there doesn't seem to be much improvement. I've tested the beta and stable versions and both appear to have this issue. From what I've read, IBKR never seemed to put retail investors and small firms, who probably will be primary users of the Web API, high up on their priority list - somewhat understandably.

Is there any suggestion we, as a community, can do to pressure Interactive Brokers to fixing this?

And to answer your question - I honestly have no idea. I wouldn't want to act aggressively towards them (eg. on social media), and I don't know what methods there are to tackle behemoth companies into focusing on their underserved audience. If you (or anyone reading this) would have suggestions then please share them here.

kavindu-athaudha · 2021-08-19T23:33:55Z

I wouldn't want to act aggressively towards them (eg. on social media)

Yes, I agree on not doing anything aggressive.
My assumption is that they have not prioritized this issue as they probably receive few tickets from support regarding this issue (despite a lot of people facing the issue, not everyone raises tickets with them).

What I was thinking was something along the lines of getting more people (e.g. like the people in this thread) to all raise the issue with Interactive Brokers so they have a clearer picture of how important this issue is and maybe they would prioritize it more.

Voyz · 2021-08-23T08:01:56Z

Okay cool, @kavindu-athaudha and how would you suggest we could organise in such way? I advice all IBeam users to contact IBKR support when running into this issue, do you have any other ideas for how we could encourage them to submit a ticket?

usr97629238 · 2021-08-25T16:40:18Z

It looks that the gateway is subject to some obscure rate limits. Also it looks like that sending malformed requests also provokes de-authentication.
And I'm afraid that submitting tickets would end with the usual boilerplate replies.
So I'd say a better approach would be approaching them nicely (they have an account here https://github.com/InteractiveBrokers) and asking for indepth infos on those rate limits, etc.

kavindu-athaudha · 2021-08-25T21:16:23Z

I advice all IBeam users to contact IBKR support when running into this issue, do you have any other ideas for how we could encourage them to submit a ticket?

@Voyz
Well, I doubt there's more we can do than asking them.
After all, GitHub is a community-driven platform which rely on nothing but the good-will of the community; I would say we would have to rely on that and just ask.

And I'm afraid that submitting tickets would end with the usual boilerplate replies.

@usr97629238
Yes, there is a good possibility of getting boilerplate replies and the issue not reaching the development team.
My thoughts were that there is nothing else we can do.

So I'd say a better approach would be approaching them nicely (they have an account here https://github.com/InteractiveBrokers) and asking for indepth infos on those rate limits, etc.

That is a very good idea, but I just went through their GitHub and unfortunately, it doesn't look as if it's very active. You could probably reach the developers directly but I'm doubtful (and I really hope I'm wrong) that we would get any fruitful replies.

usr97629238 · 2021-08-26T15:23:20Z

By the way, there is a new problem with the gateway, getting a 200 OK empty reply from /iserver/accounts (while /sso/validate and /portal/tickle report being authenticated)

--2021-08-26 18:12:49--  https://ibeam:5000/v1/api/iserver/accounts
HTTP request sent, awaiting response...
  HTTP/1.1 200 OK
  x-response-time: 0ms
  Content-Type: application/json;charset=utf-8
  Expires: Thu, 26 Aug 2021 15:12:49 GMT
  Cache-Control: max-age=0, no-cache, no-store
  Pragma: no-cache
  Date: Thu, 26 Aug 2021 15:12:49 GMT
  Connection: keep-alive
  Server-Timing: cdn-cache; desc=MISS
  Server-Timing: edge; dur=23
  Server-Timing: origin; dur=21
  Vary: Origin
  Transfer-Encoding: chunked
Length: unspecified [application/json]
Saving to: ‘STDOUT’

[<=>                                               ]       0  --.-KB/s
{-                                 [ <=>                                              ]       2  --.-KB/s    in 0s

@kavindu-athaudha

That is a very good idea, but I just went through their GitHub and unfortunately, it doesn't look as if it's very active. You could probably reach the developers directly but I'm doubtful (and I really hope I'm wrong) that we would get any fruitful replies.

They are indeed working somehow on the Gateway, at least https://gdcdyn.interactivebrokers.com/portal.proxy/v1/portal/swagger/swagger gets updates every two weeks.

Voyz · 2021-08-27T10:13:52Z

Thanks for your contributions @usr97629238 @kavindu-athaudha 👏

submitting tickets would end with the usual boilerplate replies

Indeed, this unfortunately all we've got so far.

That accounts endpoint issue you're running into @usr97629238 is something new, I haven't seen that before, although I've seen endpoints randomly returning no data.

Let us know if you've got any luck in reaching IBKR either through support ticket or privately over GitHub like you suggest.

gmara13 · 2021-08-31T15:19:43Z

Been having these active but unauthenticated error cycles pretty frequently. latest one was lasting for the the past 10 hours. i switch proxy as below:

1. Please open conf.yaml under the root folder.
2. Change the proxyRemoteHost to “https://1.api.ibkr.com”

and it immediately worked when reloading the container...FYI

Voyz · 2021-09-03T10:05:51Z

hey @gmara13 thanks for following up on this - can I ask you if you're doing this every time the error appears, or do you mean that switching to it just once solves the issue for you forever?

Voyz · 2021-10-04T14:13:49Z

only switching the proxyRemoteHost config to a different (less busy?) node

Thanks for reporting on these attempts, it's useful to know 👏 Seeing that /logout seemed to work for some users, I've just released a new release candidate docker pull voyz/ibeam:0.4.0-rc1 with that functionality in place. Give it a shot when you find some time, and should it prove unsuccessful I'll look into proxy wrangling in the conf.yaml.

if the gateway restart is needed to reset some local cache, the need for full-blown restart could be prevented programmatically by implementing the abovementioned three responsibilities, taking the load-balancer node as an input dynamically.

Thanks for suggesting this, but I'm not sure I'm following your proposal here - sorry. Do I understand you suggest to call /iserver/auth/ssodh/init, calculate the hash using the logic you pasted and send it to /iserver/auth/ssodh/response? If I understand correctly we'd be reproducing what the Gateway is currently doing, how would this help solve the Gateway session active but not authenticated? Did you try this method?

danielkrizian · 2021-10-04T17:28:30Z

suggest to call /iserver/auth/ssodh/init, calculate the hash using the logic you pasted and send it to /iserver/auth/ssodh/response? If I understand correctly we'd be reproducing what the Gateway is currently doing

my understanding too. however, gateway java app from IBKR might also be doing more than that, like caching some hashes that are time-dependent and keeping sending the same hash (until gateway restart and cache reset) to the load balancer, which in turn can cache it too server-side to identify client gateway sessions.

the above is just a conjecture, but explains my observations when sometimes not even changing nodes (described above) and /logout calls help until the whole gateway (cache/sessionid?) was restarted.

how would this help solve the Gateway session active but not authenticated

having own gateway app that implements the three functions above and accepts load balancer node number as a (dynamic) parameter in a function (instead of reading it from static config file) would be a more lightweight method of resetting the session cache than the full-blown gateway app restart.

buntagonalprism · 2021-10-05T11:02:05Z

only switching the proxyRemoteHost config to a different (less busy?) node

Thanks for reporting on these attempts, it's useful to know 👏 Seeing that /logout seemed to work for some users, I've just released a new release candidate docker pull voyz/ibeam:0.4.0-rc1 with that functionality in place. Give it a shot when you find some time, and should it prove unsuccessful I'll look into proxy wrangling in the conf.yaml.

if the gateway restart is needed to reset some local cache, the need for full-blown restart could be prevented programmatically by implementing the abovementioned three responsibilities, taking the load-balancer node as an input dynamically.

Thanks for suggesting this, but I'm not sure I'm following your proposal here - sorry. Do I understand you suggest to call /iserver/auth/ssodh/init, calculate the hash using the logic you pasted and send it to /iserver/auth/ssodh/response? If I understand correctly we'd be reproducing what the Gateway is currently doing, how would this help solve the Gateway session active but not authenticated? Did you try this method?

Thanks for the rapid update! Will try out the new 0.4.0-rc1 version.

With the current version I've just observed calling /logout manually did not resolve the issue after 12 minutes. But calling logout a second time then resolved the issue within a minute. So hopefully a few repeated logouts will work.

isque03 · 2021-10-15T16:12:20Z

Thanks for setting up this project @Voyz ! Great work.

I've been working with the IBKR api and I see the same issue where, after a while, sso is authenticated, but the brokerage session will not authenticate for several hours. Reauthenticating through the https://localhost:5000/ page does not fix the issue.

Short summary of what worked:

I tried the suggestions from the comment above and so far it has been successful.

Gateway gets in bad state where you can't get an authenticated brokerage session
Trigger /logout endpoint
Next, trigger /reauthenticate
Back in action again!

I did not change the proxyRemoteHost to N.api.ibkr.com. Just left it at the default.

More details below:
From the gateway logs you can it it attempting the ssodh flow as described here: https://www.tradersinsight.news/ibkr-quant-news/tutorial-web-api-connect-to-brokerage-session/

Note the requests with 200 response code (,200) resulted in successful brokerage session authentication.
After a while those, that POST starts getting a 503 response. This is where the gateway fails to authenticate.

-> request: /v1/api/iserver/auth/ssodh/init
-> POST https://api.ibkr.com/v1/api/iserver/auth/ssodh/init,200|541ms
-> request: /v1/api/iserver/auth/ssodh/init
-> POST https://api.ibkr.com/v1/api/iserver/auth/ssodh/init,200|30ms
-> request: /v1/api/iserver/auth/ssodh/init
-> POST https://api.ibkr.com/v1/api/iserver/auth/ssodh/init,503|10041ms
-> request: /v1/api/iserver/auth/ssodh/init
-> POST https://api.ibkr.com/v1/api/iserver/auth/ssodh/init,503|10047ms

I then did a POST to the gateway using https://localhost:5000/v1/portal/iserver/reauthenticate

You can see the gateway attempts the ssodh authentication flow, in response to this post, but it's continuing to fail due to the 503 response from api.ibkr.com

15:42:00.388 INFO vert.x-eventloop-thread-0 BaseServiceProxy : -> POST https://api.ibkr.com/v1/api/iserver/auth/ssodh/init,503|10102ms
15:42:00.389 WARN vert.x-eventloop-thread-0 BaseServiceProxy : failed /v1/api/iserver/auth/ssodh/init | reason 503
15:42:00.389 DEBUG vert.x-eventloop-thread-0 ClientPortalService : iserver init : null
15:42:00.389 ERROR vert.x-eventloop-thread-0 ClientPortalService : iserver init failed
io.vertx.core.impl.NoStackTraceThrowable: 503
15:42:00.390 DEBUG vert.x-eventloop-thread-0 BaseServiceProxy : forward event ON_SSODH_FAILED, payload none
15:42:00.390 ERROR vert.x-eventloop-thread-0 GatewayHttpProxy : ssodh failed, retry with /iserver/reauthenticate
15:42:10.090 INFO vert.x-eventloop-thread-0 BaseServiceProxy : -> POST https://api.ibkr.com/v1/api/iserver/auth/ssodh/init,503|10095ms
15:42:10.090 WARN vert.x-eventloop-thread-0 BaseServiceProxy : failed /v1/api/iserver/auth/ssodh/init | reason 503
15:42:10.090 DEBUG vert.x-eventloop-thread-0 ClientPortalService : iserver init : null
15:42:10.091 ERROR vert.x-eventloop-thread-0 ClientPortalService : iserver init failed
io.vertx.core.impl.NoStackTraceThrowable: 503
15:42:10.092 DEBUG vert.x-eventloop-thread-0 BaseServiceProxy : forward event ON_SSODH_FAILED, payload none
15:42:10.092 ERROR vert.x-eventloop-thread-0 GatewayHttpProxy : ssodh failed, retry with /iserver/reauthenticate

Given the thread above I next did a logout when it was in this state of continuous 503 errors.

curl -X POST https://localhost:5000/v1/portal/logout -H 'Content-Type: application/json' -d {}

After that I did a reauthenticate:

curl -X POST https://localhost:5000/v1/portal/iserver/reauthenticate

I can see in the gateway logs it then completes an SSO reauthentication, then also attempts the brokerage session reauthentication with the ssodh endpoints.

This time, ssodh/init returns a 200 response again, and the gateway is able to setup a new brokerage session.

-> POST https://api.ibkr.com/v1/api/iserver/auth/ssodh/init,200|169ms

I'm not sure if this is a 100% reliable workaround since I've not tried it that many times, but at least there was some success.

Voyz · 2021-10-19T07:29:51Z

@danielkrizian thanks for outlining your idea in more detail - that's very helpful and I understand it now 👍 At the moment I won't have time to investigate this idea further, but if you're successful in putting together some code that does such reauthentication then we could look at adding it into IBeam

Voyz · 2021-10-19T07:39:42Z

@isque03 thanks for bringing up your observations and outlining them in so much detail. Just to double check - which IBeam image are you using? I guess that adding a call to /reauthenticate between logging out and restarting won't add a lot of overhead so we could just try putting it in and see if it helps - what do you recon?

isque03 · 2021-10-19T14:13:05Z

@isque03 thanks for bringing up your observations and outlining them in so much detail. Just to double check - which IBeam image are you using? I guess that adding a call to /reauthenticate between logging out and restarting won't add a lot of overhead so we could just try putting it in and see if it helps - what do you recon?

Hi @Voyz . I'm actually not using iBeam. I discovered this project after I already went down the rabbit hole of writing something similar in nodejs for my own project.

Short answer: Yes, adding the reauthenticate after /logout seems fine. Note. It takes 20-40 seconds for the gateway to complete reauthentication.

Long answer:
Here's what I'm doing:
Every X minutes (where X is 5 right now):

Check SSO status. Look for USER_NAME in response from GET /v1/api/sso/validate/. If this fails re-trigger sso login.
If SSO is ok, next post('/v1/api/iserver/auth/status'). If 'authenticated:true' all is good.
If we are not authenticated with a brokerage session: post('/v1/api/logout', {}), then post('/v1/api/iserver/reauthenticate').
(Note the POST call to logout wants an empty json object as payload {} and Content-Type of application/json or it fails.)

The "logout + reauthenticate" (not including SSO login) takes the gateway server between 25-40 seconds to complete (just looking through the logs and spot checking a few events).

I also trigger the same 3 steps before any new trading event occurs so I can be reasonably sure that I'll be properly authenticated before attempting any orders.

FWIW I don't restart the java gateway very often. The only state is seems to have are cookies that it holds after an SSO login. If you re-trigger an SSO login it will get new cookies So you might be able to save some time by just keeping it running.

So far so good with that pattern.

There is one more authenticate issue that I still see cropping up (Warning: perhaps a topic for a different issue/thread)

The other thing that occurs is this:

curl -s http://localhost:5000/v1/api/iserver/accounts
will just return {} even though the above says your are authenticated. The /order endpoints will either return a 503, or will return a response saying that you need to invoke /portfolio/accounts first. Invoking /portfolio/accounts, won't help, since it doesn't have a read/write brokerage session.

Gateway logs show this:

GET /v1/api/tickle,200|61ms
02:43:34.032 DEBUG vert.x-eventloop-thread-0 ClientPortalService : tickle > {"session":"asdasdadasd","ssoExpires":469642,"collission":false,"userId":123456,"iserver":{"authStatus":{"authenticated":true,"competing":true,"connected":true,"message":"Another session with the same user name has disconnected your session.","MAC":"00:00:00:00:00:00"}}}

The only workaround I have found so far is to manually log into IBKR on the mobile app (or I suppose web/desktop). It will say "another session is in use, disconnect?" I click "yes" to disconnect, then I explicitly click the logout in the mobile app. After this I'm able to get a properly working brokerage session from the logout/reauthenticate flow.

My suspicion is that it's the call for brokerage session ssodh/init that the gateway makes.
See this document again: https://www.tradersinsight.news/ibkr-quant-news/tutorial-web-api-connect-to-brokerage-session/

Specifically the compete param in here:

`SSODH Init Request

POST https://api.ibkr.com/v1/api/iserver/auth/ssodh/init

The body, of type x-www-form-urlencoded, must have the following parameters:

      machineId (String)       8 char alphanumberic string e.g. CCCCCC01-99
      mac (String)                   six 2-char alphanumerical pairs separated by a hyphen
      **compete (Boolean)      whether or not the session should compete, usually set to false**
      locale (String)                set to “en_US
      username (String)        set to dash “-“`

The compete param here recommends setting it to false. What I think this does is trigger any other brokerage session to be invalidated, and allow a new one to be created. Setting compete "true" I beleive means that it will not kick out the other session. (Even though that seems sort of backwards from what you'd expect)
The gateway passes complete:true. (From gateway logs)
post={"username":"XXXX","machineId":"12349","compete":true}

Programmatic workarounds here? I think the only choices are:

File a ticket with IBKR and see if they'll add the ability to control the compete param in the gateway
Have iBeam actually implement the ssodh/init /complete flows to explicitly trigger brokerage session creatation (this would replace the call to /reauthenticate). I have not proven this compete true|false theory, so I wouldn't recommend going down this path yet.

Voyz · 2021-10-19T14:40:28Z

Fantastic reply @isque03 thank you for writing it all up, that's a lot of very useful information 🙏 I understand that using your flow we could save ourselves from having to restart the Gateway after logging out. I'll put together a release where we do that instead of th restart and publish it when I find a moment.

(Note the POST call to logout wants an empty json object as payload {} and Content-Type of application/json or it fails.)

How are you calling it? Through the Gateway? Or directly to IBKR servers? I'm using /logout through a GET request without any issues.

The 503 Service Unavailable issue has cropped up in a few places before, so along with your suggestion I've created a separate issue to discuss it: #38

isque03 · 2021-10-19T20:24:42Z

(Note the POST call to logout wants an empty json object as payload {} and Content-Type of application/json or it fails.)

How are you calling it? Through the Gateway? Or directly to IBKR servers? I'm using /logout through a GET request without any issues.

The endpoint documentation says it's a POST. I never tried it as a GET. Yes, I call the gateway, not direct to IBKR.
https://www.interactivebrokers.com/api/doc.html#tag/Session/paths/~1logout/post

Here's how I call it from curl.

curl -v -X POST https://localhost:5000/v1/api/logout -H 'Content-Type: application/json' -d '{}'

Interesting that GET works too. (I see the same response and behavior). Looks like the gateway is intercepting /v1/api/logout and proxying it on to api.ibkr.com. I'm not sure if there's a behavior difference between the two or not.

Though GET requests really shouldn't be changing state, so POST is probably safer. A GET request might be cached by a proxy somewhere between the gateway and IBKR which would mean the logout actually never happened. You just got a cached "success" response from some proxy server along the way.

JackD111 · 2021-10-20T11:47:51Z

@isque03 After reading this and also having had problems with the iserver/accounts endpoint returning "{}" I was thinking of trying to use mitmproxy as a transparent proxy. If fiddeled a bit with the run.sh of the gateway to append https.proxyHost to force the mitmpoxy but that didn't work so running mitmproxy in transparent mode with iptable redirects should work (in theory). That would allow us to capture all the traffic of the gateway for analyzing and maybe figure out how to do the login ourselfs. More importantly, it would allow use to modify the ssodh/init payload to set the compete to false instead of the default true

isque03 · 2021-10-20T13:25:20Z

@JackD111 that's a great idea. See last comment in #38 . But the proxy would help with the "I don't have a session token" problem. It's also a much easier quick way to test if changing the compete property from true to false even works. I'll try some testing tonight to see if the compete=false even works.

isque03 · 2021-10-21T22:06:31Z

I was able to get the gateway to send a compete=false instead of true when creating the brokerage session. But it doesn't look like this helps solve the problem of forcing a new brokerage session to "win" and kick out any previous session.

I setup two gateway clients. Logged in with one (Gateway A) and made sure that /v1/api/iserver/accounts returned a non empty response. Then I tried logging in using compete:false. The original Gateway A remained logged in.

The second gateway (Gateway B) could not get a brokerage session. The /v1/api/iserver/accounts endpoint would just return {}. After about 30 seconds the /v1/api/tickle endpoint started reporting:
authenticated:true,competing:true, message:"Another session with the same user name has disconnected your session"

I also did a few rounds of calling /v1/api/logout and /v1/api/reauthenticate. No difference.

Gateway A consistently reported
{"authenticated":true,"competing":false}
There really wasn't any noticeable behavior change on gateway A after gateway B tried to authenticate.

Went back to original settings with compete=true

This time Gateway A was logged out after about a minute and its SSO session was invalid. Endpoints returned 401.
Gateway B still couldn't get a brokerage session. It returned the same response to /v1/api/tickle. That is
authenticated:true
competing:true
message:"Another session with the same user name has disconnected your session"

I was able to regain an SSO and brokerage session on Gateway A by doing another SSO login again.

FYI on the Proxy:
Like @JackD111 said. The gateway doesn't respect the usual java -DhttpProxyHost= -DhttpsProxyHost= -DhttpsProxyPort -DhttpProxy port system properties.

It looks like the gateway uses a library valled Vertx. That library advertises a separate set of system properties for specifying the proxy server. But those didn't work either. Possibly the version of Vertx used by the gateway is too old and didn't support an http proxy at the time. There was also a bug reported in 2017. Maybe that's what I was running into, not sure: eclipse-vertx/vert.x#1944

https://docs.mitmproxy.org/stable/howto-transparent/
Transparent proxying works, but in my case I had to use a separate machine as the gateway. I was running this in a debian linux environment on a chromebook and ran into this:
mitmproxy/mitmproxy#4149

DanielHTpg · 2021-10-22T09:17:16Z

@isque03 Interesting findings. I've also tried mitmproxy yesterday and manged to get it working. Here is a rough outline of what I did but steps might be missing. I've executed all the commands inside the ibeam container to make it easier and manged to get it working that way in transparent mode. Looking at the traffic it also seems the gateway isn't using the full auth flow as descirbed in https://www.tradersinsight.news/ibkr-quant-news/tutorial-web-api-connect-to-brokerage-session/. I didn't see the compete param used anywhere in the auth requests.

The ibeam container also has to be started with --cap-add=NET_ADMIN to be able to modify the iptables for the traffic redirection to mitmproxy.

These were the commands to set the whole thing up:

-- Use "docker exec -u 0 -it <ibeam container name> /bin/bash" to login as root on ibeam container
apt-get update
apt-get install sudo
apt-get install procps
apt-get install iptables

-- Setup mitmproxy with a seperate user. Needed for iptables traffic rediect so we can exclude redirecting traffic from the proxy to the outside

useradd --create-home mitmproxyuser
sudo -su mitmproxyuser
curl https://snapshots.mitmproxy.org/7.0.4/mitmproxy-7.0.4-linux.tar.gz >> mitmproxy-7.0.4-linux.tar.gz
tar -xf mitmproxy-7.0.4-linux.tar.gz

-- Run mitmproxy once so the certificates are generated
./mitmproxy --mode transparent --showhost

-- ctrl+c to exit and switch back to root user

-- register ssl cert with system
mkdir /usr/local/share/ca-certificates/extra
cp /home/mitmproxyuser/.mitmproxy/mitmproxy-ca-cert.cer /usr/local/share/ca-certificates/extra/mitmproxy-ca-cert.crt
update-ca-certificates

-- Java uses its own certificate store ignoring the "normal" certificates so we need to explicitly add the certificates
$JAVA_HOME/bin/keytool -import -noprompt -trustcacerts -alias mitmproxy -file /home/mitmproxyuser/.mitmproxy/mitmproxy-ca-cert.pem -keystore /usr/lib/jvm/java-11-openjdk-amd64/lib/security/cacerts -storepass changeit

-- Needed as descripbed in mitmproxy documentation
sysctl -w net.ipv4.ip_forward=1
sysctl -w net.ipv6.conf.all.forwarding=1
sysctl -w net.ipv4.conf.all.send_redirects=0

-- Need to switch to legacy rules since I didn't know how to use the new rules system
update-alternatives --set iptables /usr/sbin/iptables-legacy

-- Final iptables rule to redirect all traffic to 443 to mitmproxy but exclude outgoing traffic from mitmproxyuser
iptables -t nat -A OUTPUT -p tcp -m owner ! --uid-owner mitmproxyuser --dport 443 -j REDIRECT --to-port 8080

-- switch back to mitmproxyuser and start mitmproxy, all traffic from gateway should now be picked up
sudo -su mitmproxyuser
./mitmproxy --mode transparent --showhost

andrey-ned · 2021-10-29T15:53:04Z

@Voyz I want to say Thank You for Your code.
But i have a problem with my paper account.
I try to run Ibeam in different ways python and docker, on local machine and server.
Always, I get this but not authenticated
App demo is available after you login under: https://localhost:5000/demo/
2021-10-29 15:50:03,238|I| No active sessions, logging in...
2021-10-29 15:50:16,650|I| Authentication process succeeded
2021-10-29 15:50:19,806|E| Gateway session active but not authenticated
2021-10-29 15:50:19,807|I| Logging out and restarting the Gateway
2021-10-29 15:50:19,924|I| Gateway logout successful
2021-10-29 15:50:20,931|I| Gateway shutdown successful

Can you help me?
I am from Ukraine, maybe is it reason ?

Voyz · 2021-11-03T10:41:24Z

Hey @andrey-ned thanks for the kind words and for using IBeam 😊 The issue you're encountering is the same as one everyone else is experiencing here. We're waiting for IBKR to solve it on their end, while some work is being done to mitigate its impact. You can read about it in this very issue, however there isn't any direct fix I can suggest right now. Stay tuned for more updates in the future 👍

andrey-ned · 2021-11-03T10:50:56Z

Hi, @Voyz, so It's official problem on IBRK side ?
And where can I check progress of fixing this problem ?

Voyz · 2021-11-03T10:57:25Z

@andrey-ned We don't know how official it is, but it is almost certainly a problem with Gateway and IBKR servers.

Voyz · 2021-12-17T09:35:10Z

voyz/ibeam:0.4.0-rc3 was just released with the alternative reauthentication logic as proposed here: #40 (comment)

Let us know how the 0.4.0 versions are working for you - has it helped with the repeated Gateway session active but not authenticated?

jeffj0110 · 2022-01-05T20:18:19Z

Not using ibeam (need to check it out). I ran into this issue after using the REST client portal API with the local gateway running in my Windows desktop.

I changed proxyRemoteHost: to "https://1.api.ibkr.com" and it did resolve the lost authentication issue for me.

Thanks everyone for your comments.

Voyz · 2022-03-23T13:41:56Z

To anyone involved - seeing I haven't heard back regarding this issue I'm going to be looking at merging the v0.4.0 into the master and releasing it as latest. Let me know if you'd want to bring something up before we do that. Many thanks to everyone involved in testing and figuring out these various solutions 👍

Voyz · 2022-04-07T12:44:20Z

Closing this as v0.4.0 just got released which should fix this issue. Feel free to open a new issue if you'd like to continue the discussion 👍 Thank you for participating!

beanhedge added the enhancement New feature or request label Apr 25, 2021

Voyz added bug Something isn't working and removed enhancement New feature or request labels Apr 26, 2021

Voyz self-assigned this Apr 26, 2021

Voyz mentioned this issue Jul 25, 2021

Connected but not authenticated #26

Closed

gomisha mentioned this issue Sep 12, 2021

Can't authenticate - US and Canadian live and paper trading accounts #34

Closed

Wenuka mentioned this issue Sep 13, 2021

Constantly getting "Gateway session active but not authenticated" #7

Closed

Voyz mentioned this issue Oct 19, 2021

ARM64 docker build #31

Merged

Voyz changed the title ~~[not an issue with ibeam] Account seems to be limited after a few days of running~~ [not an issue with ibeam] Account seems to be limited after a few days of running ('Gateway session active but not authenticated') Oct 19, 2021

Voyz mentioned this issue Oct 19, 2021

503 Service Unavailable (IBKR/Gateway issue) #38

Closed

Voyz mentioned this issue Jan 31, 2022

Being kicked out of IBKR server daily. #53

Closed

Voyz closed this as completed Apr 7, 2022

elderanakain mentioned this issue May 17, 2023

Timeout error in 0.4.4 version #140

Closed

eswark18 mentioned this issue Jun 21, 2023

Another IB login page change? #145

Closed

vlpraveen mentioned this issue Jul 11, 2023

v0.5.0 - development #147

Closed

dennisrosenbaum mentioned this issue Aug 9, 2023

SSL: WRONG_VERSION_NUMBER error #157

Closed

[not an issue with ibeam] Account seems to be limited after a few days of running ('Gateway session active but not authenticated') #19

[not an issue with ibeam] Account seems to be limited after a few days of running ('Gateway session active but not authenticated') #19

Comments

beanhedge commented Apr 25, 2021 • edited Loading

Voyz commented Apr 26, 2021

beanhedge commented May 14, 2021 • edited Loading

Voyz commented May 15, 2021

beanhedge commented May 16, 2021 • edited Loading

Greatsamps commented Jul 18, 2021

Voyz commented Jul 18, 2021

Greatsamps commented Jul 18, 2021

Voyz commented Jul 18, 2021

Greatsamps commented Jul 18, 2021

Voyz commented Jul 18, 2021

Greatsamps commented Jul 20, 2021

Greatsamps commented Jul 20, 2021

Voyz commented Jul 20, 2021

kavindu-athaudha commented Aug 19, 2021

Voyz commented Aug 19, 2021

kavindu-athaudha commented Aug 19, 2021

Voyz commented Aug 23, 2021

usr97629238 commented Aug 25, 2021

kavindu-athaudha commented Aug 25, 2021

usr97629238 commented Aug 26, 2021 • edited Loading

Voyz commented Aug 27, 2021

gmara13 commented Aug 31, 2021

Voyz commented Sep 3, 2021

Voyz commented Oct 4, 2021

danielkrizian commented Oct 4, 2021 • edited Loading

buntagonalprism commented Oct 5, 2021 • edited Loading

isque03 commented Oct 15, 2021

Voyz commented Oct 19, 2021

Voyz commented Oct 19, 2021

isque03 commented Oct 19, 2021

Voyz commented Oct 19, 2021

isque03 commented Oct 19, 2021

JackD111 commented Oct 20, 2021

isque03 commented Oct 20, 2021

isque03 commented Oct 21, 2021

DanielHTpg commented Oct 22, 2021

andrey-ned commented Oct 29, 2021

Voyz commented Nov 3, 2021 • edited Loading

andrey-ned commented Nov 3, 2021 • edited Loading

Voyz commented Nov 3, 2021

Voyz commented Dec 17, 2021

jeffj0110 commented Jan 5, 2022

Voyz commented Mar 23, 2022

Voyz commented Apr 7, 2022

beanhedge commented Apr 25, 2021 •

edited

Loading

beanhedge commented May 14, 2021 •

edited

Loading

beanhedge commented May 16, 2021 •

edited

Loading

usr97629238 commented Aug 26, 2021 •

edited

Loading

danielkrizian commented Oct 4, 2021 •

edited

Loading

buntagonalprism commented Oct 5, 2021 •

edited

Loading

Voyz commented Nov 3, 2021 •

edited

Loading

andrey-ned commented Nov 3, 2021 •

edited

Loading