Constantly getting "Gateway session active but not authenticated"

[AndreaOrru]

Describe the bug
After logging in and authenticating through 2FA via the IBKR mobile app, the login succeeds, but the session immediately appears as non-authenticated:

2020-12-12 05:22:11,726|I| Gateway started with pid: 12
2020-12-12 05:22:13,200|I| No active sessions, logging in...
2020-12-12 05:22:53,237|I| Login succeeded
2020-12-12 05:22:54,451|E| Gateway session active but not authenticated
2020-12-12 05:22:54,731|I| Starting maintenance with interval 60 seconds
2020-12-12 05:23:54,943|I| Gateway session found but not authenticated, authenticating...
2020-12-12 05:24:11,539|I| Login succeeded
2020-12-12 05:24:12,753|E| Gateway session active but not authenticated
2020-12-12 05:24:56,693|I| Gateway session found but not authenticated, authenticating...
2020-12-12 05:25:18,914|I| Login succeeded
2020-12-12 05:25:20,123|E| Gateway session active but not authenticated

And it constantly asks me to login again.
I seem to be able to make some requests anyway: i.e. /portfolio/accounts works just fine.
But /portfolio/{accountId}/positions/0 doesn't work.

To Reproduce
Steps to reproduce the behavior:

1. Start IBeam through Docker providing authentication details in an env file
2. Complete 2FA through the mobile app
3. Observe "Login succeded" in the log, rejoice temporarily
4. Despare at the sight of "Gateway session active but not authenticated"

Expected behavior
After login succeeds, the gateway session should be and stay authenticated.

Environment
IBeam version: 0.2.0
Python version: 3.7.7
OS: Linux
(running on the default Docker image)

[AndreaOrru]

Nevermind. I had to explicitly log out from my web session :)
Thanks for your work!

[Voyz]

Hey thanks for reporting and following up on this issue! 👋
I'm glad you managed to find a solution so quickly 😊 To be quite honest - I noticed similar pattern appearing sometimes, despite not being logged in anywhere else than using IBeam. It seems to stabilise after a few minutes and then the problem goes away - still, I'd rather this didn't happen at all. Please let me know if this problem repeats and we'll try to look into it further. Thanks! Voy

[Proksima]

Hmm, I started playing with the timeouts (because I was running out of time to do 2FA) and now I'm getting this issue fairly consistently. Here is a snippet of the logs

2021-03-02 17:44:31,392|I| Gateway started with pid: 12
2021-03-02 17:44:32,319|I| No active sessions, logging in...
2021-03-02 17:44:49,811|I| Login succeeded
2021-03-02 17:44:50,903|E| Gateway session active but not authenticated
2021-03-02 17:44:51,322|I| Starting maintenance with interval 60 seconds
2021-03-02 17:45:51,635|I| Gateway session found but not authenticated, authenticating...
2021-03-02 17:46:11,505|E| Error encountered during authentication
Traceback (most recent call last):
  File "/srv/ibeam/src/gateway_client.py", line 176, in authenticate_gateway
    WebDriverWait(driver, _OAUTH_TIMEOUT).until(success_present)
  File "/opt/venv/lib/python3.7/site-packages/selenium/webdriver/support/wait.py", line 80, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: 


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/srv/ibeam/src/gateway_client.py", line 183, in authenticate_gateway
    raise RuntimeError('Error encountered during authentication') from e
RuntimeError: Error encountered during authentication
2021-03-02 17:46:11,516|I| Login failed
2021-03-02 17:46:51,796|I| Gateway session found but not authenticated, authenticating...

Tried restoring the constants to their original values with no luck. This is on 0.3.0-rc2 (even though I didn't try to get 2FA through IBeam to work yet).

Any idea of how to make the issue less likely to happen?

Edit: reverted to latest with same issue.

[Voyz]

Hey @Proksima thanks for reporting this issue! 😊 Let's try to figure out why are you running into these errors that often.

Firstly though - unless I'm mistaken the log you pasted is from the older version, probably latest ie. 0.2.1. Can you confirm that this also happens on 0.3.0-rc2 and possibly paste the log from that version here too? Especially if this is happening repeatedly I'd appreciate if you could let IBeam run for a while and run into a number of these errors. Furthermore, you can try turning on more verbose logging by setting IBEAM_LOG_LEVEL to DEBUG - this could tell us more about the cause too.

I made some improvements to the authentication and ensuring that the gateway is running before attempting to communicate with it on the recent release candidates - I'd like to verify that you're running into these issues despite these improvements.

Also - do I understand correctly you manually complete 2FA with your phone when IBeam is trying to authenticate? If so, I'm assuming the timeout happens after you complete 2FA, not before it, right?

Lastly, you wrote "I started playing with the timeouts (because I was running out of time to do 2FA) and now I'm getting this issue fairly consistently." - does that mean you weren't running into this issue that often before you started changing the timeouts?

[Proksima]

I will come around and retry probably tomorrow or this weekend, but I will answer what I can in the meantime.

"Firstly though - unless I'm mistaken the log you pasted is from the older version, probably latest ie. 0.2.1. Can you confirm that this also happens on 0.3.0-rc2 and possibly paste the log from that version here too?"

I was under the impression my tests were on 0.3.0-rc2, but now you have me doubting; I will make sure that I was not using an older image by mistake.

"Furthermore, you can try turning on more verbose logging by setting IBEAM_LOG_LEVEL to DEBUG - this could tell us more about the cause too."

Will follow up on this next time I have a chance to work on it.

"Also - do I understand correctly you manually complete 2FA with your phone when IBeam is trying to authenticate? If so, I'm assuming the timeout happens after you complete 2FA, not before it, right?"

The first 2FA prompt makes IBeam goes to "Logged in but not authenticated." On the next authentication attempts I do not get a 2FA prompt.

"Lastly, you wrote "I started playing with the timeouts (because I was running out of time to do 2FA) and now I'm getting this issue fairly consistently." - does that mean you weren't running into this issue that often before you started changing the timeouts?"

It used to work flawlessly all the time. I mentioned this, but at the same time, I think its a coincidence. Otherwise, it would imply that this would have a lasting impact on my authentication which does not make sense to me... but I could be wrong. I think low key I played with them because I thought I was missing the 2FA and then it just made the problem more obvious. I doubt it actually contributed...

[Voyz]

Great, thanks for filling me in on more details!

The first 2FA prompt makes IBeam goes to "Logged in but not authenticated." On the next authentication attempts I do not get a 2FA prompt.

This seems to indicate something indeed started going wrong afterwards. Could it be maybe you got locked out due to failed 2FA attempts? Can you still login to TWS using the same credentials?

I'm thinking the 0.3.0-rc2 logs should be a bit more verbose, especially when running in DEBUG logging - we should find more clues there. Looking forward to seeing these 👍

[gmcmicken]

I run into this too if I'm logged in somewhere else but also just seen to have it happen every once in a while. I usually stop iBeam and log in manually, go to trade tab hit the take over button, then log out.

[Proksima]

Okay, fairly sure I'm on 0.3.0-rc2 as per my ansible definition: ibeam.txt

I am now getting a different behavior: 2FA is asked at every authentication trials. Didn't work on the first trial but worked on the second (more consistent with what I read on this thread).

Here are my logs for this trial: ibeam.log.txt

Since it now works and I can't guarantee it will stop working on 0.3.0-rc2 (still unsure what I was using before because I was not specifying the version for the docker_container in my configuration; could have been either since both were available). I think this ticket can be closed?

I am leaving DEBUG on (there are very little debug logging anyway) and if it every does that weird behavior again I'll make sure to re-open and attach the full log.

Thank you for the pointers!

[Voyz]

Thanks for leaving some feedback in that regard @gmcmicken! For sure having the same credentials used will cause authentication errors along these lines. I'm curious to see whether the localhost:5000 login page displays some 'take over the credentials' button when this happens, that we could potentially click at that stage - may do some more research into that.

Okay, fairly sure I'm on 0.3.0-rc2 as per my ansible definition
still unsure what I was using before because I was not specifying the version for the docker_container in my configuration; could have been either since both were available

Yupp, that sounds like previously you were running 0.2.1 @Proksima - thanks for all the extra info! Just to clarify why that happened: when the version is unspecified Docker will default to pulling the one tagged latest and -rc* versions don't override the latest tag as they are pre-releases. Your new logs also reflect that and I can confirm you're running 0.3.0-rc2 - glad to have this sorted :) Your case actually gave me an idea I should have thought of ages ago - I'll include the IBeam version in the startup log.

I am now getting a different behavior: 2FA is asked at every authentication trials. Didn't work on the first trial but worked on the second (more consistent with what I read on this thread).
I am leaving DEBUG on (there are very little debug logging anyway) and if it every does that weird behavior again I'll make sure to re-open and attach the full log.

Right, thanks for reporting that. Frankly, IBeam also sometimes fails to authenticate for me on the first one or two attempts - I just kind of waved it off as this seems to be internal IB stuff figuring out the permissions and it seems to resolve itself after a while. Definitely a good idea to have that behaviour accounted for in the code that uses IB. My system will attempt to execute its requests on a repeated basis until all of them succeed - although this can also be potentially risky if care isn't put into preventing double requests. Nevertheless - if you can think of some ways we could improve the authentication or are running into some other unexplained issues, do let me know. Happy we managed to get you up and running again!

Thanks! 👋 Voy

[Wenuka]

I run into this too if I'm logged in somewhere else but also just seen to have it happen every once in a while. I usually stop iBeam and log in manually, go to trade tab hit the take over button, then log out.

can you please elaborate on exactly how do you do this? Are you doing it from the IBKR Trader Workstation (where we can place trades) or in AccountManagement (where we only see the portfolio)? And how exactly to do it.

I have two paper-trading accounts and in one, there is no problem and in the other one it has this issue of repetitive Gateway running and authenticated.

To be exact, it was running well, then this happens,

2021-09-12 21:56:59,833|I| Gateway running and authenticated
2021-09-12 21:57:59,838|I| Gateway running and authenticated
2021-09-12 21:58:59,822|I| Gateway running and authenticated
2021-09-12 21:59:59,835|I| Gateway running and authenticated
2021-09-12 22:00:59,929|I| Gateway session found but not authenticated, authenticating...
2021-09-12 22:01:11,685|I| Authentication process succeeded
2021-09-12 22:01:14,773|E| Gateway session active but not authenticated
2021-09-12 22:01:59,845|I| Gateway session found but not authenticated, authenticating...
2021-09-12 22:02:11,545|I| Authentication process succeeded
2021-09-12 22:02:14,618|E| Gateway session active but not authenticated
2021-09-12 22:02:59,844|I| Gateway session found but not authenticated, authenticating...
2021-09-12 22:03:11,497|I| Authentication process succeeded
2021-09-12 22:03:14,569|E| Gateway session active but not authenticated
2021-09-12 22:03:59,849|I| Gateway session found but not authenticated, authenticating...

and it has been the same since.

Thanks in advance.

[Wenuka]

Hi,

Sorry for the trouble, but the error was resolved by changing the proxyRemoteHost to “https://1.api.ibkr.com” in conf.yaml as mentioned in issue 19.

[MislavSag]

I am receiving the same message, after I made successful authentication. But I am not using 2FA. Is it possible that I am getting this error because I don't have 2FA active?
I wanted to make it work without 2FA, and than try with 2FA latter.

[Voyz]

@MislavSag We haven't seen any relationship between this error message and the 2FA. As far as we can tell now this seems to be a periodic issue with IBKR

[Wenuka]

@Voyz since this seems like a common issue, if possible can you improve iBeam to automatically detect continuous Gateway session active but not authenticated and toggle the proxyRemoteHost between “https://api.ibkr.com”, “https://1.api.ibkr.com” and “https://2.api.ibkr.com”? (I have personally tried these three URLs, maybe there are many)

As of now, I am facing this problem, so I continuously check the status just before the market open and if the gateway is not authenticated I change the proxyRemoteHost value and rerun the docker container. Then it works like a charm.

[Voyz]

That's a great idea @Wenuka 👏 Definitely worth exploring. I'm currently unavailable to implement such a fix, so feel free to propose a PR, but otherwise I'll look into it once I find some time

[popiol]

Maybe a simpler solution would be to automatically log out when the issue is identified

https://localhost:5000/v1/api/logout

That seems to work for me

[Voyz]

@popiol thanks for leaving this suggestion, I'll try to implement it first and see if it helps with the problem. There's been another report that /logout helps here - #19 (comment)

[Voyz]

@popiol @Wenuka @AndreaOrru @Proksima @gmcmicken new release candidate image docker pull voyz/ibeam:0.4.0-rc1 has just been released with the automatic logout and restart upon experiencing Gateway session active but not authenticated. It has been reported that logging out should help with this error. If we see no such results, I'll look into wrangling the IBKR proxies in the conf.yaml Give it a shot when you find some time 👍

[popiol]

Looks good for me:

2021-10-06 05:23:28,688|I| Gateway session found but not authenticated, authenticating...
2021-10-06 05:23:41,648|I| Authentication process succeeded
2021-10-06 05:23:44,787|I| Gateway running and authenticated

[Wenuka]

Hi,
The above solution fixed the original problem and now I am always getting Gateway running and authenticated.
However, when I tried to place an order (using https://localhost:{port_num}/v1/api/iserver/account/{accountId}/orders API) I am getting a 503 response as below,

{'_content': b'', '_content_consumed': True, '_next': None, 'status_code': 503, 'headers': {'x-response-time': '10000ms', 'X-Cnection': 'close', 'Expires': 'Mon, 18 Oct 2021 11:32:58 GMT', 'Cache-Control': 'max-age=0, no-cache, no-store', 'Pragma': 'no-cache', 'Date': 'Mon, 18 Oct 2021 11:32:58 GMT', 'Connection': 'close', 'Server-Timing': 'cdn-cache; desc=MISS, edge; dur=4, origin; dur=10019', 'Vary': 'Origin', 'Transfer-Encoding': 'chunked'}, 'raw': <urllib3.response.HTTPResponse object at 0x7f15b168d520>, 'url': 'https://localhost:5000/v1/api/iserver/account/XXXXXX/orders', 'encoding': None, 'history': [], 'reason': 'Service Unavailable', 'cookies': <RequestsCookieJar[]>, 'elapsed': datetime.timedelta(seconds=10, microseconds=68901), 'request': <PreparedRequest [POST]>, 'connection': <requests.adapters.HTTPAdapter object at 0x7f15b166aeb0>}

Then I killed the docker session and re-run using the changed proxyRemoteHost value and it worked as expected.

P.s:
By the way, in IBKR documents, they have mentioned that we need to manually log in every week. Can this problem relate to that constraint or does IBeam somehow manage to overcome that constraint?

Thank you in advance.

[Voyz]

Thanks for reporting back @Wenuka 👍 It sounds like we may want to resort to proxy wrangling. Due to workload I won't have time to look into it in the few upcoming weeks, so feel free to propose a PR handling this functionality in the meantime.

As for manually logging in every week - could you link up the documentation you're referring to? I think due to all the various errors I never had the Gateway continuously authenticated for as long as a week, so it's hard for me to tell if that could cause any issues, but thanks for highlighting it as a potential problem.

[DanielHTpg]

@Voyz Thanks for releasing voyz/ibeam:0.4.0-rc1. I've been running ibeam now for two months and always when the gateway was stuck in Gateway session active but not authenticated I've stopped ibeam and changed the "proxyRemoteHost" to one of the other API endpoints (1.api...). That usually solved the issue. My paper trading ibeam was just stuck again in this loop and after pulling in the new rc1 image it did the whole logout and started running smootly again. Heres the log of the whole thing

2021-10-20 11:30:05,409|I| No active sessions, logging in...
2021-10-20 11:30:19,731|I| Authentication process succeeded
2021-10-20 11:30:22,796|E| Gateway session active but not authenticated
2021-10-20 11:30:22,796|I| Logging out and restarting the Gateway
2021-10-20 11:30:22,842|I| Gateway logout successful
2021-10-20 11:30:23,850|I| Gateway shutdown successful
2021-10-20 11:30:23,935|I| Starting maintenance with interval 15 seconds
2021-10-20 11:30:38,930|I| Gateway not found, starting new one...
2021-10-20 11:30:38,930|I| Note that the Gateway log below may display "Open https://localhost:5000 to login" - ignore this command.
running
 runtime path : root:dist/ibgroup.web.core.iblink.router.clientportal.gw.jar:build/lib/runtime/*
 verticle     :
2021-10-20 11:30:39,938|I| Gateway started with pid: 123
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by io.netty.util.internal.ReflectionUtil (file:/srv/clientportal.gw/build/lib/runtime/netty-common-4.1.15.Final.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of io.netty.util.internal.ReflectionUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Java Version: 11.0.12
****************************************************
version: ed4af2592e9dd4a784d5403843bd18292fd441ea Fri, 9 Nov 2018 13:23:18 -0500
****************************************************
This is a Beta release of the Client Portal Gateway
for any issues, please contact [email protected]
and include a copy of your logs
****************************************************
https://www.interactivebrokers.com/api/doc.html
****************************************************
Open http://localhost:5000 to login
App demo is available after you login under: http://localhost:5000/demo/
2021-10-20 11:30:43,094|I| No active sessions, logging in...
Execution of job "GatewayClient._maintenance (trigger: interval[0:00:15], next run at: 2021-10-20 11:30:53 UTC)" skipped: maximum number of running instances reached (1)
2021-10-20 11:30:55,013|I| Authentication process succeeded
2021-10-20 11:30:58,058|I| Gateway running and authenticated
2021-10-20 11:31:09,013|I| Gateway running and authenticated
2021-10-20 11:31:23,967|I| Gateway running and authenticated

One small thing maybe would be to stop the maintance when doing the logout and restart it once the gateway has been restarted.

[Voyz]

Thanks for reporting @DanielHTpg very glad to hear the new fixes are working for you 👍 Also good suggestion, I'll look into stopping maintenance during restart, thanks!

[jonbattista]

Running into these intermittent connection issues as well.

Hitting https://localhost:5000/v1/api/logout to refresh everything looks to solve this issue.

However even with voyz/ibeam:0.4.0-rc1 I am still getting a 503 after some time.

IBeam still thinks the Gateway is running and authenticated, but the API isnt returning any data:

 curl -k https://localhost:5000/v1/portal/iserver/auth/status
{"authenticated":true,"competing":false,"connected":true,"message":"","MAC":"XX:XX:XX:XX:XX","fail":""}%

curl -k https://localhost:5000/v1/api/iserver/accounts
{}%

I dont understand how Interactive Brokers can offer such a poorly designed and implemented API.

How are you supposed to automate trading if you have to re-auth every hour/day and the API is broken?

[Voyz]

sorry to hear @jonbattista have a look at #38 - this would be the issue you're encountering

[10ne1]

For what it is worth: I also get the Gateway session active but not authenticated message if I am logged in to the website or the Android mobile app. If I explicitly logout / de-authenticate from the website / app then ibeam will correctly authenticate and start working. If I do not login to the website or app, ibeam wil continue to work.

[Voyz]

@10ne1 if I understand it correctly what you're seeing is expected. You can use one login credentials only for either the Client Portal Web API or for Web/Mobile app.

The issue others are describing here is that even when not logged in elsewhere the CP Gateway will start returning this error message.

[10ne1]

@Voyz Thanks, I'll have to do more observations to figure out if all my current error message are due to being logged into web/mobile or if I also receive errors without being logged in. I have upgraded to voyz/ibeam:0.4.0-rc1 and I do not recall receiving errors since the upgrade if I log out from web/mobile.

What does happen though is sometimes I get logged out after a few hours, then ibeam does the auth dance again as expected in rc1.

I do however note that I need to completely log out from web/mobile: any lingering web/mobile session, no matter how old, even in read-only mode can cause the error to appear.

[Voyz]

voyz/ibeam:0.4.0-rc3 was just released with the alternative reauthentication logic as proposed here: #40 (comment)

Let us know how the 0.4.0 versions are working for you - has it helped with the repeated Gateway session active but not authenticated?

[murtazayusuf]

I am not having an issue with IBeam but there has been a weird problem lately. If I run get requests related to account such as https://localhost:5000/v1/api/portfolio/{accountId}/ledger or https://localhost:5000//v1/api/portfolio/accounts then the response is fine but when getting market data like https://20.65.51.14:5000/v1/api/iserver/marketdata/history?conid=461318866 the response is blank. But when running the gateway manually on my local machine I get response 200 with the data. Is this happening with with anyone else. I have a hunch that this could be the case because the market is close right now.

sorry I had to write this here because there are not many forums to discuss this.

[Voyz]

@murtazayusuf you may be experiencing a similar issue as the one described in #38. If you feel that this is a different problem then I'd suggest you open a new issue, as this one is mainly related to authentication problems, rather than empty responses.

If you aren't sure if it's due to market closed or not, I'd suggest to run both IBeam and the Gateway tests one after another during opening hours and verify this assumption then.

[murtazayusuf]

Thanks @Voyz .

[Voyz]

To anyone involved - seeing I haven't heard back regarding this issue I'm going to be looking at merging the v0.4.0 into the master and releasing it as latest. Let me know if you'd want to bring something up before we do that. Many thanks to everyone involved in testing and figuring out these various solutions 👍

[Voyz]

Closing this as v0.4.0 just got released which should fix this issue. Feel free to open a new issue if you'd like to continue the discussion 👍 Thank you for participating!

[mujtabaanwer]

Hello
We are still facing this issue.
I am not logged into IBKR using any account. However, I keep receiving this error:
I have consulted some experience developers over last 4 months but no one is able to help
Is it possible for you to provide the steps to be able to get rid of this error and make things work?

2023-01-31 16:13:13,534|I| No active sessions, logging in...
2023-01-31 16:13:25,995|I| Authentication process succeeded
2023-01-31 16:13:29,330|E| Gateway session active but not authenticated
2023-01-31 16:13:29,331|I| Logging out and restarting the Gateway
2023-01-31 16:13:29,618|I| Gateway logout successful
2023-01-31 16:13:29,933|I| Gateway session found but not authenticated, authenticating...
2023-01-31 16:13:41,206|I| Authentication process succeeded
2023-01-31 16:13:44,545|E| Gateway session active but not authenticated
2023-01-31 16:13:44,545|I| Logging out and restarting the Gateway
2023-01-31 16:13:44,843|I| Gateway logout successful
2023-01-31 16:13:45,285|I| Gateway session found but not authenticated, authenticating...
2023-01-31 16:13:56,392|I| Authentication process succeeded
2023-01-31 16:13:59,710|E| Gateway session active but not authenticated
2023-01-31 16:13:59,710|I| Logging out and restarting the Gateway
2023-01-31 16:13:59,991|I| Gateway logout successful
2023-01-31 16:14:00,325|I| Gateway session found but not authenticated, authenticating...

[Voyz]

@mujtabaanwer I wish I could help you here, but we haven't been able to identify the cause of this issue appearing. The best I can suggest is talking to IBKR support and letting them know that your sessions' authentication is not working and see if they can help.