-
Notifications
You must be signed in to change notification settings - Fork 27
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
OAuth support #34
OAuth support #34
Conversation
First commit |
Okay, I saw that you started with changing all the methods in mixins. I appreciate the effort but I don't think that will be necessary. If we look at the def send_oauth_request(
request_method: str,
request_url: str,
oauth_token: str | None = None,
live_session_token: str | None = None,
extra_headers: dict[str, str] | None = None,
request_params: dict[str, str] | None = None,
signature_method: str = "HMAC-SHA256",
prepend: str | None = None,
) -> requests.Response:
headers = {
"oauth_consumer_key": consumer_key,
"oauth_nonce": generate_oauth_nonce(),
"oauth_signature_method": signature_method,
"oauth_timestamp": generate_request_timestamp(),
}
if oauth_token:
headers.update({"oauth_token": oauth_token})
if extra_headers:
headers.update(extra_headers)
base_string = generate_base_string(
request_method=request_method,
request_url=request_url,
request_headers=headers,
request_params=request_params,
prepend=prepend,
)
logger.info(
msg={
"message": "generated base string",
"timestamp": time.time(),
"details": {
"base_string": base_string,
"request_method": request_method,
"request_url": request_url,
"request_headers": headers,
"request_params": request_params,
"prepend": prepend,
},
}
)
if signature_method == "HMAC-SHA256":
headers.update(
{
"oauth_signature": generate_hmac_sha_256_signature(
base_string=base_string,
live_session_token=live_session_token,
)
}
)
else:
headers.update(
{
"oauth_signature": generate_rsa_sha_256_signature(
base_string=base_string,
private_signature_key=read_private_key(
signature_key_fp
),
)
}
)
logger.info(
msg={
"message": "generated signature",
"timestamp": time.time(),
"details": {
"signature": headers["oauth_signature"],
"signature_method": signature_method,
},
}
)
response = requests.request(
method=request_method,
url=request_url,
headers={
"Authorization": generate_authorization_header_string(
request_data=headers,
realm=realm,
)
},
params=request_params,
timeout=10,
)
logger.info(
msg={
"message": "sent oauth request",
"timestamp": time.time(),
"details": {
"request_method": request_method,
"request_url": response.request.url,
"request_headers": response.request.headers,
"request_body": response.request.body,
"response_status_code": response.status_code,
"response_error_message": response.text if not response.ok else None,
},
}
)
return response You'll notice that all it essentially does is set some headers on the request. Then there's some logging, but we don't need that, or we could do it elsewhere. Hence, in reality, all we need to do to implement this into the current version of IBind, is to expand our existing request flow to attach these headers. To start, we need to add a empty def get_headers(
request_method: str,
request_url: str,
request_params: dict[str, str] | None = None,
):
return None In the ibind/ibind/base/rest_client.py Line 127 in 91c4687
We can call the ibind/ibind/base/rest_client.py Line 162 in 91c4687
Then in the IbkrClient, we override the def get_headers(
request_method: str,
request_url: str,
request_params: dict[str, str] | None = None,
):
# TODO: read or generate all necessary variables here
headers = {
"oauth_consumer_key": consumer_key,
"oauth_nonce": generate_oauth_nonce(),
"oauth_signature_method": signature_method,
"oauth_timestamp": generate_request_timestamp(),
}
if oauth_token:
headers.update({"oauth_token": oauth_token})
if extra_headers:
headers.update(extra_headers)
base_string = generate_base_string(
request_method=request_method,
request_url=request_url,
request_headers=headers,
request_params=request_params,
prepend=prepend,
)
if signature_method == "HMAC-SHA256":
headers.update(
{
"oauth_signature": generate_hmac_sha_256_signature(
base_string=base_string,
live_session_token=live_session_token,
)
}
)
else:
headers.update(
{
"oauth_signature": generate_rsa_sha_256_signature(
base_string=base_string,
private_signature_key=read_private_key(
signature_key_fp
),
)
}
)
return {
"Authorization": generate_authorization_header_string(
request_data=headers,
realm=realm,
)
} That method does require us to provide the It also requires a Within This way, all extra authentication should happen automatically for the user, and all existing methods will be able to stay the same. We'd just have to set the correct environment variables and specify that we wanna use OAuth when constructing the IbkrClient: ibkr_client = IbkrClient(..., use_oauth=True) Let me know if you can see some roadblocks in this implementation or if you'd like to discuss anything before giving it a crack. Otherwise, let me know if you get stuck anywhere and I'll be happy to help. |
That sounds like a good plan, I'll update the code and commit again for review shortly. |
Voy, I've started to make the changes you suggested, let me know if I understand you code correctly and how they look. The first step gets a live_session_token and access_token if the user sets use_oauth=True.
But the call to post, and then request functions in rest_client.py need checking.
Once authorized, subsequent calls to endpoints will get the headers in the request function, but it needs some work.
|
Hey @hughandersen great job making the progress! Just to start reviewing it, I'm gonna ask you to add .venv to .gitignore file and remove it from GitHub. It's usually not a good idea to include these in a public repo. Any kind of local files like these, node-modules, .idea, etc. Then I'll be able to give it a review, as currently it's 1000s of files that are slowing down the PR review page |
@Voyz Sure, my mistake, I forgot to check that the gitignore file excludes .venv before committing. Do you have a preferred method to remove the folder? |
No problem at all 👍 Just the normal way, remove them from git but keep them locally: https://stackoverflow.com/questions/1143796/remove-a-file-from-a-git-repository-without-deleting-it-from-the-local-filesyste |
@Voyz Done, check and let me know if the folder hasn't been removed successfully. |
@Voyz Can you take a look at the file rest_08_oauth.py in examples?
Is this something to do with the |
Better! Try doing the same with |
To be able to run it you need to add the root directory to PYTHONPATH: https://www.simplilearn.com/tutorials/python-tutorial/python-path You shouldn't need to change:
Try that and let me know if it helps. |
@Voyz Voy the path has been fixed (simple typo) and I'm working on getting live session and access tokens. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@hughandersen superb job on the first iteration of this PR 👏👏 You've followed the suggestions I made very well and did an amazing job resolving the module import / pythonpath issues you faced initially (I hate these too 🙄).
I've left a number of comments for you to address in order to do the next few steps towards getting this merged in, but honestly you're on the right path and are doing really well so far. Thanks!
OAuth session initialization (getting live session and access tokens) is now working. But getting the brokerage session established is not working yet (see |
Code can now get live_session_token,live_session_token_expires_ms data, and get market data using original ibind methods, see file
@Voyz take a look when you get a chance |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good progress @hughandersen 👏👏 I've left a few comments.
Do I understand you can communicate with the IBKR API now correctly using OAuth? Calling various endpoints in the example file works?
I didn't review the oauth_requests.py
and get_session.py
as these seem to be your WIP files - do I understand correctly?
ibind/client/ibkr_client.py
Outdated
super().__init__(url=url, cacert=cacert, timeout=timeout, max_retries=max_retries) | ||
|
||
if self._use_oauth: | ||
self.live_session_token,self.live_session_token_expires_ms=self.req_live_session_token() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great 👏 I understand these can get generated just once per session and don't need to be re-generated?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good question. The live session token times out after 24hours, and I think it would need to be re-created if the session drops out. Maybe also allow the live session token to be re-created in a function?
It took some fiddling to get the order of the attributes correct (super before if self._use_oauth etc.), can you see any room for improvement?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well we do get that live_session_token_expires_ms
value right - that should tell us when it expires?
Or even better - we could observe what does IBKR return back when we try to call its endpoints with an expired live_session_token. Then we catch that error and request a new live session token when it happens. No need to keep track of the expiry date, just handle it once it happens. What do you recon?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, I could check what get returns after the expiry date, but I think there are other bigger bugs to fix right now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure. Can you add a TODO comment in here so that we remember to handle this later? We shouldn't release until we have a nice way to handle it
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok I'll add a comment.
examples/rest_08_oauth_test.py
Outdated
|
||
#%% | ||
# get brokerage session | ||
brokerage_session_response=client.initialize_brokerage_session(publish='true',compete='true') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We need to call this for things to work? If so, how about we do it automatically for the user?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think so, but the IBKR document is unclear, and I needed to call it before calling the other endpoints.
I'll check with them.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh nice initiative! Though I'd say just comment this line out and see if it works. If it does, then leave it out, if it doesn't leave it in 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've emailed IBKR, lets see what they come back with.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@hughandersen what's the status on this? Do we know if we need to call this? What happens if we don't? I see that you already make several requests before calling this one - do they succeed? If so, can we remove it?
@hughandersen hold up, you got OAuth 2.0 with IBKR? Can you share what docs are you using to do this?
Ideally this would be as straightforward as switching a few environment variables and/or constructor parameters. Anyway, because this PR is already quite long, I strongly recommend we tackle OAuth1.0a in this PR and address OAuth2.0 later. |
@Voyz Yes I have access to OAuth2 and I'm in the process of getting it working. I agree that we should finish this PR, and once its working start another one for OAuth2. |
Thanks for clarifying @hughandersen It's just that if OAuth 2.0 is already a viable option, then @janfrederik's argument to name things |
@Voyz I was given access and docs by IBKR because I have a small business account, I don't think OAuth2 is open for individuals so I don't think we should change or ditch the work done on OAuth1. I think you should merge the code when you're confident, and then we can open another PR for OAuth2 with discussions etc. I think localhost, OAuth1 and OAuth2 can work together. |
Yes, we definitely will not ditch OAuth1.0a, this PR is pretty much ready to be merged in. But if OAuth2.0 is indeed already available to use for small business users - something I wasn't aware of and thought is still far off on the roadmap - then OAuth version reflected in the nomenclature would make sense. I suggest we reflect it in the # default to OAuth1aConfig when no oauth_config is specified
ibkr_client = IbkrClient(..., use_oauth=True)
# specify 1.0a
oauth_config = OAuth1aConfig(...)
ibkr_client = IbkrClient(..., use_oauth=True, oauth_config=oauth_config)
# specify 2.0
oauth_config = OAuth2Config(...)
ibkr_client = IbkrClient(..., use_oauth=True, oauth_config=oauth_config) Naturally, we won't implement the Please let me know of any suggestions regarding this, otherwise I'll go ahead and implement the renaming for OAuth1aConfig. |
Looks good as interface for users using one of the OAuth1a/2Config objects. As for the env vars and var.py, I would suggest a mostly distinct set for oauth1a and for oauth2 to minimize confusion. Soemthing like this:
|
Please see the
Note:
The usage is pretty much the same, just change I think this will conclude the development on this PR, so I'll give it a few days for any last minute tests and feedback and merge it in. If you get a chance, please try this recent version out and let us know if it works fine or if you run into any issues. Thanks @janfrederik for bringing the naming into our attention and @hughandersen for reporting the 2.0 availability! 🙌 |
@Voyz That sounds like a good plan. |
@Voyz FYI, |
fix websocket oauth
@Voyz I've done a merge in the oauth branch, can you now merge the oauth branch with the master branch? |
@hughandersen absolutely, I'm planning to do this next week as I'm caught up in other matters and cannot sit down to writing the WiKi yet, which I'd like to follow up with promptly afterwards. Thanks for merging in @SmartJQ's changes 👍 |
Going ahead with a merge 🥳 Many thanks to @hughandersen for writing it, you did a superb job carrying out most of the work on your own based on my suggestions. As you can already see, this seemed to have been a very popular and needed addition to the library, so once again great job leading the change here 🙌 Also, thanks to everyone who's contributed in making it better; I appreciate all the useful comments, suggestions and questions, in particular @thouseef @salsasepp @weklund @janfrederik @SmartJQ thanks for making this addition so great 🙌 There is an issue I've created to track the OAuth 1.0a documentation: I'd appreciate your input there on any topic related. In particular on the items I personally don't have access - related to requesting OAuth access from IBKR. I've marked these tasks with 🙈 emoji, if anyone would like to write one of these and share, let me know in the comment in that issue. Thanks everyone! |
Congratulations on the milestone guys 🎉 |
@Voyz I hope the merge goes well with no issues and I'm pleased the addition is useful to others. |
@Voyz Glad to see it merged. Congratulations @hughandersen for achieving this milestone and many thanks for taking the initiative. Sorry, I couldn't spend more time here .. Once the craziness is over, I shall join the discussions again. |
Hey @hughandersen @thouseef @salsasepp @janfrederik would you mind dropping me an email when you find a minute? You did a great job with those recent discussions and contributions on this PR, and I wondered if I could ask you a couple of questions in private: [email protected] (I'm reaching out here as I cannot find any emails to contact you directly) |
Hey guys, there's some users who are wondering if you can use OAuth API with paper credentials: #61 Has anyone here succeeded in using paper credentials? If so, do you think you could chip in and outline your process? Thanks 👍 |
I have it working. Answered in #61 (comment). |
My login via OAuth1.0 was working well. |
thanks @janfrederik 🙌 @hughandersen understood, strange it stopped for you entirely. IBKR told me that it's pretty exceptional they'd give out OAuth 2.0 access, so consider yourself lucky 👍 How is that going btw, is it working well for you? |
OAuth IBKR code added to repo.
Work to make functions to call IBKR via local host or OAuth needs to be developed.