Performance Impovement for REST Endpoints #60
Comments
|
hey @mmarihart thanks for this superb suggestion 👏 I tried implementing your code, and indeed it is a very easy drop in replacement. Great to see that 👍 However, there is one problem - in my tests not only that there is no improvement, but in some cases it's slower than the original method. I'm using https://github.com/Voyz/ibind/blob/master/examples/rest_05_marketdata_history.py for tests. See below: Original Session: The differences in time seem arbitrary, there's no clear improvement. On top of the changes you proposed, ChatGPT suggested I do the following: self.session = requests.Session()
adapter = HTTPAdapter(pool_connections=10, pool_maxsize=10, pool_block=True)
self.session.mount("https://", adapter)
self.session.mount("http://", adapter)And attach headers: {
"Connection": "close" # or "keep-alive", neither helped
}Could I ask you to put together a minimal example script that would show how the change actually introduce improvements? |
|
Hmm, you are getting a lot better speeds than I do on the marketdata history. With the subsecond duration it's not surprising that it does not make a lot of difference. Without session: With session: Here's some other testing code that I am running/using: import time
from datetime import datetime
from functools import partial
from ibind import IbkrClient, snapshot_keys_to_ids, make_order_request, ibind_logs_initialize, QuestionType
start = time.time()
account = 'CHANGEME!!!!'
symbol = 'MNQ'
c = IbkrClient()
c.initialize_brokerage_session()
c.receive_brokerage_accounts()
result = c.security_future_by_symbol(symbols=symbol)
if result:
result = result.data[symbol]
else:
raise Exception("No contracts")
# Find the contract with the largest open interest
# sort them by contract month so we can skip if the OI decreases
if not result:
raise Exception("No contracts")
result.sort(key=lambda x: x['ltd'])
conids = [str(x['conid']) for x in result]
highest_open_interest = 0
highest_oi_contract = None
field_id = snapshot_keys_to_ids(['futures_open_interest'])
c.live_marketdata_snapshot(conids=conids, fields=field_id)
print("Fetching futures open interest for {}".format(symbol))
response = c.live_marketdata_snapshot(conids=conids, fields=field_id).data
for entry in response:
try:
open_interest = entry[field_id[0]]
if isinstance(open_interest, str):
if 'K' in open_interest:
open_interest = float(open_interest.split('K')[0]) * 1000
else:
open_interest = float(open_interest)
if open_interest > highest_open_interest:
highest_open_interest = open_interest
highest_oi_contract = entry['conid']
elif highest_oi_contract:
# we are going to lower contracts now, so we can stop
counter = 100
break
except KeyError:
continue
symbol = str(highest_oi_contract)
c.marketdata_unsubscribe(conids)
# Pre-fetch for market data
c.live_marketdata_snapshot(conids=symbol, fields=snapshot_keys_to_ids(['open', 'high', 'low', 'close', 'volume', 'last_price']))
# Create market long entry order with OCO SL and TP
order_tag = f'my_order1-{datetime.now().strftime("%Y%m%d%H%M%S")}'
order_request_partial = partial(make_order_request, conid=symbol, acct_id=account, quantity=1)
parent = order_request_partial(side='BUY', order_type='MARKET', price=0.0, coid=order_tag)
stop_loss = order_request_partial(side='SELL', order_type='STP', price=21100, parent_id=order_tag)
take_profit = order_request_partial(side='SELL', order_type='LMT', price=21400, parent_id=order_tag)
requests = [parent, stop_loss, take_profit]
answers = {
QuestionType.PRICE_PERCENTAGE_CONSTRAINT: True,
QuestionType.ORDER_VALUE_LIMIT: True,
QuestionType.MISSING_MARKET_DATA: True,
QuestionType.STOP_ORDER_RISKS: True,
'You are attempting to close a position that has open orders.': True,
}
orders = c.place_order(requests, answers, account).data
# Wait a bit
time.sleep(2)
# Pre-flight for orders
c.live_orders(account_id=account, force=True).data
print(c.live_orders(account_id=account, filters=['Submitted', 'PreSubmitted']).data)
print(c.get(f'/portfolio2/{account}/position/{symbol}').data)
# Update SL
stop_loss['price'] = 21101
for order in orders:
if order.get('parent_order_id', None):
# This is the SL order
result = c.modify_order(order_id=order['order_id'], account_id=account,
answers=answers, order_request=stop_loss)
print(c.live_orders(account_id=account, filters=['Submitted', 'PreSubmitted']).data)
position_size = c.get(f'/portfolio2/{account}/position/{symbol}').data[0]['position']
order_tag = f'my_order2-{datetime.now().strftime("%Y%m%d%H%M%S")}'
order_request = make_order_request(
conid=symbol,
side='SELL',
quantity=position_size,
order_type='MARKET',
acct_id=account,
coid=order_tag
)
order_request['isClose'] = True
orders = c.place_order(order_request, answers, account).data
print(c.get(f'/portfolio2/{account}/position/{symbol}').data)
time.sleep(0.05)
print(c.live_orders(account_id=account, filters=['Submitted', 'PreSubmitted']).data)
result = c.post('iserver/marketdata/unsubscribe', params={'conid': str(symbol)})
print("Took {} seconds".format(time.time() - start))Results I get for this: |
|
Okay, wow, these are certainly undisputable differences. I'll implement it, thanks for sharing all of that. Are you using OAuth or Gateway/IBeam? I think I'll add it as an optional functionality, as there seem to be cases where non-session implementation has advantages. Switching gears for a second, I notice you call c.get(f'/portfolio2/{account}/position/{symbol}')This is the second time I see you know of some endpoints that aren't documented. How did you figure out that these are available? Do you have a fuller list of or some method of finding out what's available? |
I am using the Client Portal Gateway. Default config, like it is described in the IBKR docs.
I just stupidly tried as I wanted to go around the caching stuff :) There is the portfolio2/{account}/positions endpoint, so why not one for specific symbol? ^^ The isClose parameter I found with curiosity (why is this functionality working in the IBKR web application?) and the browser network tab. |
|
@mmarihart try https://github.com/Voyz/ibind/tree/change-to-requests-session branch or
Yeah, possibly some restriction from IBKR if you overused the service. I think they do mention doing that somewhere in the docs.
Specifying a value in the conf.yaml. If you look through IBeam's issues you should find some info on how to do it. Though I'm not sure if that would help. I'm currently in Europe too for context. Try running this test to see your speeds to US: https://testmy.net/mirror?testServer=sf Mine is:
Haha awesome, I thought this could have been your method, I would have done the same 😅 Honestly, I'm kinda glad, I thought I'm missing on some huge info repository on IBKR somehow. Thanks for clarifying! If you spot these endpoints, do let me know, we should consider implementing them. |
I can confirm that ibind 0.1.12rc1 is faster and so far working for my use cases with the following results:
I debugged why my system was so slow. I am running Windows 10 for those tests as I need Windows for my other trading stuff (-.-). On my linux systems I got good speeds as well with version 0.1.10. So it clearly has to be a Windows problem. Using wireshark, we can see that for every request made by ibind to the Gateway portal on localhost it first tries to connect via IPv6 (::1 as destination) and gets reset packets as a response:
Afterwards it switches to IPv4 (127.0.0.1 as destination) and the connection is successful. This happens for every connection establishment, making it very slow without sessions. With this knowledge I changed one line in the example code we used for these tests and got the same fast speeds as you and on my Linux systems even with version 0.1.10. Therefore, I would propose changing the default value from 'localhost' to '127.0.0.1' at ibind/ibind/client/ibkr_client.py Line 46 in 433c645 |
|
Wow, superb digging @mmarihart 👏👏 I'll absolutely change that localhost to 127.0.0.1, thanks for the suggestion! Also glad to hear that the addition works. I'll merge it to the master branch and publish with the next release. Please try it through ps. no shame in running on Windows, video games and editing software in my case 😉 |
|
Both test codes work and run fast with 0.1.12rc2! Thanks for fixing. |
Hi, since we often need to use preflight requests I noticed that the REST API functionality can take quite a long time to execute some functions.
What I saw was, that for every request a new TCP connection is opened and closed in
ibind/ibind/base/rest_client.py
Line 214 in 587a46e
Using a persistent TCP session, by utilizing requests.Session improves performance massively and does not require a lot of changes.
For the above line the updated version would be:
And then we only need the following extra code in the __init__ function at:
ibind/ibind/base/rest_client.py
Line 76 in 587a46e
For some test code I have (fetching futures contract data, market buy with bracket order, adjusting SL price, closing positions and orders) the run time reduced from ~60 seconds to 13 seconds.
I think that should work without causing any problems.
The text was updated successfully, but these errors were encountered: