503 Service Unavailable (IBKR/Gateway issue)

[Voyz]

Describe the bug
IBKR/Gateway sometimes runs into returning a 503 Service Unavailable error. In this issue we try to understand what can be done to overcome it.

Environment
IBeam-independent, it is the Gateway's issue

Additional context
So far it has been observed when calling:

• /iserver/account/{accountId}/orders

---

In #7 (comment) @Wenuka outlines:

when I tried to place an order (using https://localhost:{port_num}/v1/api/iserver/account/{accountId}/orders API) I am getting a 503 response as below,

{'_content': b'', '_content_consumed': True, '_next': None, 'status_code': 503, 'headers': {'x-response-time': '10000ms', 'X-Cnection': 'close', 'Expires': 'Mon, 18 Oct 2021 11:32:58 GMT', 'Cache-Control': 'max-age=0, no-cache, no-store', 'Pragma': 'no-cache', 'Date': 'Mon, 18 Oct 2021 11:32:58 GMT', 'Connection': 'close', 'Server-Timing': 'cdn-cache; desc=MISS, edge; dur=4, origin; dur=10019', 'Vary': 'Origin', 'Transfer-Encoding': 'chunked'}, 'raw': <urllib3.response.HTTPResponse object at 0x7f15b168d520>, 'url': 'https://localhost:5000/v1/api/iserver/account/XXXXXX/orders', 'encoding': None, 'history': [], 'reason': 'Service Unavailable', 'cookies': <RequestsCookieJar[]>, 'elapsed': datetime.timedelta(seconds=10, microseconds=68901), 'request': <PreparedRequest [POST]>, 'connection': <requests.adapters.HTTPAdapter object at 0x7f15b166aeb0>}

Then I killed the docker session and re-run using the changed proxyRemoteHost value and it worked as expected.

---

In #19 (comment) @isque03 outlines:

Gateway logs show this:

GET /v1/api/tickle,200|61ms
02:43:34.032 DEBUG vert.x-eventloop-thread-0 ClientPortalService : tickle > {"session":"asdasdadasd","ssoExpires":469642,"collission":false,"userId":123456,"iserver":{"authStatus":{"authenticated":true,"competing":true,"connected":true,"message":"Another session with the same user name has disconnected your session.","MAC":"00:00:00:00:00:00"}}}

The only workaround I have found so far is to manually log into IBKR on the mobile app (or I suppose web/desktop). It will say "another session is in use, disconnect?" I click "yes" to disconnect, then I explicitly click the logout in the mobile app. After this I'm able to get a properly working brokerage session from the logout/reauthenticate flow.

My suspicion is that it's the call for brokerage session ssodh/init that the gateway makes.
See this document again: https://www.tradersinsight.news/ibkr-quant-news/tutorial-web-api-connect-to-brokerage-session/

Specifically the compete param in here:

`SSODH Init Request

POST https://api.ibkr.com/v1/api/iserver/auth/ssodh/init

The body, of type x-www-form-urlencoded, must have the following parameters:

      machineId (String)       8 char alphanumberic string e.g. CCCCCC01-99
      mac (String)                   six 2-char alphanumerical pairs separated by a hyphen
      **compete (Boolean)      whether or not the session should compete, usually set to false**
      locale (String)                set to “en_US
      username (String)        set to dash “-“`

The compete param here recommends setting it to false. What I think this does is trigger any other brokerage session to be invalidated, and allow a new one to be created. Setting compete "true" I beleive means that it will not kick out the other session. (Even though that seems sort of backwards from what you'd expect)
The gateway passes complete:true. (From gateway logs)
post={"username":"XXXX","machineId":"12349","compete":true}

Programmatic workarounds here? I think the only choices are:

1. File a ticket with IBKR and see if they'll add the ability to control the compete param in the gateway
2. Have iBeam actually implement the ssodh/init /complete flows to explicitly trigger brokerage session creatation (this would replace the call to /reauthenticate). I have not proven this compete true|false theory, so I wouldn't recommend going down this path yet.

[Voyz]

@isque03 brought this discussion over from #19. Thanks for being very detailed once again, great information.

I have not proven this compete true|false theory, so I wouldn't recommend going down this path yet.

Have you considered doing it? Would it be as simple as adding a HTTP request to your existing flow to verify this theory?

[isque03]

@isque03 brought this discussion over from #19. Thanks for being very detailed once again, great information.

I have not proven this compete true|false theory, so I wouldn't recommend going down this path yet.

Have you considered doing it? Would it be as simple as adding a HTTP request to your existing flow to verify this theory?

Giving it a try tonight.

The document here isn't quite correct:
https://www.tradersinsight.news/ibkr-quant-news/tutorial-web-api-connect-to-brokerage-session/

• First it's only taking Content-Type: application/json
• username has to be the actual username, not '-'.
• After those changes I did get a challenge hash.

The response is a bit tougher:
Challenge and Response

The challenge is needed to calculate the challenge response, or R whereby seed = challenge, verifier = st.

To calculate R, ensure both the challenge and session token calculated during initial authentication are in hex form. Then, concatenate the challenge and session tokens, with the challenge being in front.

The problem here is I'm not sure we have access to the "session token" referred to in the document. I can see in the gateway logs it prints the session token out as "K=". It also prints K=XXX for the final challenge response, not the session token. So, that's a bit confusing.

Using test values from the log. This will calculate the correct response value given the challenge, plus the session token.

const crypto = require('crypto'); const hash = crypto.createHash('sha1'); const seed = process.argv[2]; const verifier = process.argv[3]; hash.update(Buffer.from(seed + verifier, 'hex')); const result = hash.digest('hex'); console.log(result);

Almost there.. Just need to figure out how to get the session token! I noticed the /v1/api/tickle endpoint returns a property called session in its response. But this doesn't seem to be the correct value. It never matches K= in the logs and when I use that to calculate the response I just get a 10 second delay, then a 503 response.

[isque03]

Just an update here. I wasn't able to get the session token. But it looks like it's available during the initial SSO process. This is the SSO process that is done through the browser.
See #19 for details. Short summary that I followed a suggestion from @DanielHTpg and setup a proxy to intercept the brokerage session initialization call. It turns out that compete=false was actually worse.

I have found that it is stable if I only login in one place. Still testing with paper account, so I basically just don't log into that account except through the gateway. IBKR allows creating additional login accounts, so for live trading it's probably best to create a new login account that is dedicated to iBeam / gateway logins.

[Voyz]

Thanks for the updates on this @isque03 👏 I haven't got anything to comment on the process at this stage, but I appreciate you sharing the results of your attempts. Good observations on logins 👍

[DanielHTpg]

I've been using voyz/ibeam:0.4.0-rc1 with docker-compose now for two weeks. I've also added cronjobs to stop ibeam outside of trading hours and restart it before trading opens for the region I trade in. Since those two weeks I didn't have any troubles with getting errors on the trading endpoints. I request order data frequently and before the newest rc1 and docker-compose changes if had to change proxyRemoteHost atleast once a week. Previously I didn't use docker-compose and had the ibeam-container running indefinitely.

Since I don't persist the docker logs in any way, I loose all logs, since docker-compose cleans up the containers daily. So I can't check if the logout happend at some point before. What I also think helps alot now, is that each day ibeam and the IB gateway get a complete clean new container without any previous caches or whatnot. This might be the main reason, that I've not had the issue anymore.

I can also only reccomend everyone to implement incremental retry logic for api calls in their trading bots. I've set the ibeam maintance interval to 15s so if the gateway connections gets lost or anything it will be re-established asap. Here's a small outtake of my logs showing how the incremental retries gives ibeam some time to login again and after a few retries everything will continue again:

[alex1999Tra]

I am using voyz/ibeam:0.4.0-rc1. I observed that the gateway is running and authenticated. But the /iserver/accounts returns an empty account with 200 status. And iserver/marketdata/snapshot returns 500. I am not sure if this is also related to the issue. Any suggestions?

Ibeam log
2021-11-11 17:27:31,747|I| Gateway running and authenticated 2021-11-11 17:28:31,751|I| Gateway running and authenticated 2021-11-11 17:29:31,721|I| Gateway running and authenticated 2021-11-11 17:30:31,715|I| Gateway running and authenticated

IBKR Gateway log
11-11-2021 17:28:02 UTC: DEBUG : _make_request : https://localhost:5000 "GET /v1/api/iserver/accounts HTTP/1.1" 200 None 11-11-2021 17:28:02 UTC: DEBUG : _make_request : https://localhost:5000 "GET /v1/api/iserver/marketdata/snapshot?conids=199169591&fields=31%2C55%2C70%2C71%2C74 HTTP/1.1" 500 None 11-11-2021 17:28:02 UTC: ERROR : invest : There are errors when investing 500 Server Error: Internal Server Error for url: https://localhost:5000/v1/api/iserver/marketdata/snapshot?conids=199169591&fields=31%2C55%2C70%2C71%2C74

[Voyz]

@DanielHTpg thanks for suggesting cleaning out the container daily - that's an interesting observation. Anyone else here tried this method and can comment on its results?

Also, that suggestion to keep the requests repeating is good to point out - thanks for sharing it and that output from your trading system.

But the /iserver/accounts returns an empty account with 200 status

@alex1999Tra that's new, I don't think anyone encountered this yet, we've seen 503s on that endpoint so far. I'd suggest getting in touch with IBKR support and trying to see what they say about it.

[DanielHTpg]

@Voyz The issue with no accounts being returned was one of the biggest problems I had after switching the proxyRemoteHost. Sometimes it only went away the next day. Since using docker-compose I haven't this issue.

[eswark18]

any update on this issue?
I'm facing response 503 while placing orders. I tried rate limiter to limit the API calls but still no use.

[Voyz]

@eswark18 I'd suggest highlighting it with IBKR, as this is an internal server error reported back to IBeam

[kashyappatel7]

I'm seeing the same issue. When I place an order then call iserver/reply, I'm getting:

'Response code 503 (Service Unavailable)'

[Voyz]

I'm going to close this issue due to inactivity. Thanks for your contribution and please feel free to reopen if you'd like to continue the discussion 👍

[Voyz]

This helpful comment by @demansou could help address 503 error: #147 (comment)

[gikeberyfinn]

Hello, Voyz. Thank you for this excellent service. However, I am encountering a 503 error again.

I tried modifying conf.yaml:
Changing
proxyRemoteHost: "https://api.ibkr.com/v1/api"
to
proxyRemoteHost: "https://5.api.ibkr.com/v1/api"

I also attempted logging in via the mobile version with the same API credentials and then forcefully logging out from the mobile version. Unfortunately, nothing helped—I still receive a 503 error when trying to place an order. However, I have no issues retrieving my portfolio positions.

What can I do now?
I am using docker pull voyz/ibeam:latest.

[Voyz]

@gikeberyfinn thanks for your kind words and sorry you're running into 503. Other than reading other users' suggestions, I'd say getting in touch with IBKR support and talking to them about it would be the best choice, as this is unrelated to IBeam itself

[gikeberyfinn]

@Voyz I asked them, and they requested some logs. I don’t understand where to find them. Do I need to enable some kind of debug mode? Could you please advise?

"we would need you to generate .har logs for us.

• Reproduce the steps where you experienced the issues using a browser to generate relevant log entries."