OAuth support

[hughandersen]

OAuth IBKR code added to repo.
Work to make functions to call IBKR via local host or OAuth needs to be developed.

[hughandersen]

First commit

[Voyz]

Okay, I saw that you started with changing all the methods in mixins. I appreciate the effort but I don't think that will be necessary. If we look at the send_oauth_request:

 def send_oauth_request(
        request_method: str,
        request_url: str,
        oauth_token: str | None = None,
        live_session_token: str | None = None,
        extra_headers: dict[str, str] | None = None,
        request_params: dict[str, str] | None = None,
        signature_method: str = "HMAC-SHA256",
        prepend: str | None = None,
    ) -> requests.Response:
    headers = {
        "oauth_consumer_key": consumer_key,
        "oauth_nonce": generate_oauth_nonce(),
        "oauth_signature_method": signature_method,
        "oauth_timestamp": generate_request_timestamp(),
    }

    if oauth_token:
        headers.update({"oauth_token": oauth_token})
    if extra_headers:
        headers.update(extra_headers)
    base_string = generate_base_string(
        request_method=request_method,
        request_url=request_url,
        request_headers=headers,
        request_params=request_params,
        prepend=prepend,
    )
    logger.info(
        msg={
            "message": "generated base string",
            "timestamp": time.time(),
            "details": {
                "base_string": base_string,
                "request_method": request_method,
                "request_url": request_url,
                "request_headers": headers,
                "request_params": request_params,
                "prepend": prepend,
            },
        }
    )
    if signature_method == "HMAC-SHA256":
        headers.update(
            {
                "oauth_signature": generate_hmac_sha_256_signature(
                    base_string=base_string,
                    live_session_token=live_session_token,
                )
            }
        )
    else:
        headers.update(
            {
                "oauth_signature": generate_rsa_sha_256_signature(
                    base_string=base_string,
                    private_signature_key=read_private_key(
                        signature_key_fp
                    ),
                )
            }
        )
    logger.info(
        msg={
            "message": "generated signature",
            "timestamp": time.time(),
            "details": {
                "signature": headers["oauth_signature"],
                "signature_method": signature_method,
            },
        }
    )
    response = requests.request(
        method=request_method,
        url=request_url,
        headers={
            "Authorization": generate_authorization_header_string(
                request_data=headers,
                realm=realm,
            )
        },
        params=request_params,
        timeout=10,
    )
    logger.info(
        msg={
            "message": "sent oauth request",
            "timestamp": time.time(),
            "details": {
                "request_method": request_method,
                "request_url": response.request.url,
                "request_headers": response.request.headers,
                "request_body": response.request.body,
                "response_status_code": response.status_code,
                "response_error_message": response.text if not response.ok else None,
            },
        }
    )
    return response

You'll notice that all it essentially does is set some headers on the request. Then there's some logging, but we don't need that, or we could do it elsewhere.

Hence, in reality, all we need to do to implement this into the current version of IBind, is to expand our existing request flow to attach these headers.

To start, we need to add a empty get_headers method to RestClient. We will later override it in IbkrClient:

def get_headers(        
        request_method: str,
        request_url: str,
        request_params: dict[str, str] | None = None,
        ):    
    return None

In the RestClient's requestmethod:

ibind/ibind/base/rest_client.py

Line 127 in 91c4687

def request(self, method: str, endpoint: str, attempt: int = 0, log: bool = True, **kwargs) -> Result:

We can call the get_headers and then attach them when making request here:

ibind/ibind/base/rest_client.py

Line 162 in 91c4687

response = requests.request(method, url, verify=self.cacert, timeout=self._timeout, **kwargs)

---

Then in the IbkrClient, we override the get_headers method, and generate the headers similar to how it happens in the send_oauth_request:

def get_headers(        
        request_method: str,
        request_url: str,
        request_params: dict[str, str] | None = None,
        ):    
    
    # TODO: read or generate all necessary variables here

    headers = {
        "oauth_consumer_key": consumer_key,
        "oauth_nonce": generate_oauth_nonce(),
        "oauth_signature_method": signature_method,
        "oauth_timestamp": generate_request_timestamp(),
    }

    if oauth_token:
        headers.update({"oauth_token": oauth_token})
    if extra_headers:
        headers.update(extra_headers)
    base_string = generate_base_string(
        request_method=request_method,
        request_url=request_url,
        request_headers=headers,
        request_params=request_params,
        prepend=prepend,
    )
    if signature_method == "HMAC-SHA256":
        headers.update(
            {
                "oauth_signature": generate_hmac_sha_256_signature(
                    base_string=base_string,
                    live_session_token=live_session_token,
                )
            }
        )
    else:
        headers.update(
            {
                "oauth_signature": generate_rsa_sha_256_signature(
                    base_string=base_string,
                    private_signature_key=read_private_key(
                        signature_key_fp
                    ),
                )
            }
        )
    return {
            "Authorization": generate_authorization_header_string(
                request_data=headers,
                realm=realm,
            )
        }

That method does require us to provide the oauth_token and consumer_key - this seems quite easy, I assume we can just read them from environment variables.

It also requires a live_session_token, which we can generate automatically on the fly. We use the same methods from IBKR, but run them automatically when we call get_headers method.

Within get_headers we can also check fo a new member variable of IbkrClient called self._use_oauth that we pass as a constructor parameter. If it is false, we return None. This would allow us to maintain the existing authentication flow. But if self._use_oauth is True, we go ahead and generate the live_session_token and subsequently the headers.

---

This way, all extra authentication should happen automatically for the user, and all existing methods will be able to stay the same. We'd just have to set the correct environment variables and specify that we wanna use OAuth when constructing the IbkrClient:

ibkr_client = IbkrClient(..., use_oauth=True)

Let me know if you can see some roadblocks in this implementation or if you'd like to discuss anything before giving it a crack. Otherwise, let me know if you get stuck anywhere and I'll be happy to help.

[hughandersen]

That sounds like a good plan, I'll update the code and commit again for review shortly.

[hughandersen]

Voy, I've started to make the changes you suggested, let me know if I understand you code correctly and how they look.

The first step gets a live_session_token and access_token if the user sets use_oauth=True.

self._use_oauth=use_oauth
self.live_session_token,self.access_token=(self.req_live_session_token if self._use_oauth else None,None)]

But the call to post, and then request functions in rest_client.py need checking.

response=self.post(self, path=REQUEST_URL, params = extra_headers,prepend=prepend, log = True)

Once authorized, subsequent calls to endpoints will get the headers in the request function, but it needs some work.

header=self.get_headers(request_method=method,request_url=endpoint,request_params=TBD)

[Voyz]

Hey @hughandersen great job making the progress! Just to start reviewing it, I'm gonna ask you to add .venv to .gitignore file and remove it from GitHub. It's usually not a good idea to include these in a public repo. Any kind of local files like these, node-modules, .idea, etc. Then I'll be able to give it a review, as currently it's 1000s of files that are slowing down the PR review page

[hughandersen]

@Voyz Sure, my mistake, I forgot to check that the gitignore file excludes .venv before committing. Do you have a preferred method to remove the folder?

[Voyz]

No problem at all 👍 Just the normal way, remove them from git but keep them locally:

https://stackoverflow.com/questions/1143796/remove-a-file-from-a-git-repository-without-deleting-it-from-the-local-filesyste
https://stackoverflow.com/questions/936249/how-to-stop-tracking-and-ignore-changes-to-a-file-in-git

[hughandersen]

@Voyz Done, check and let me know if the folder hasn't been removed successfully.

[hughandersen]

@Voyz Can you take a look at the file rest_08_oauth.py in examples?
I usually use the debug function in an interactive cell to find problems, but I get an error

ModuleNotFoundError                       Traceback (most recent call last)
File d:\git_repos\ibind\examples\rest_08_oauth.py:4
      [1](file:///D:/git_repos/ibind/examples/rest_08_oauth.py:1) breakpoint()
      [2](file:///D:/git_repos/ibind/examples/rest_08_oauth.py:2) #%%
----> [4](file:///D:/git_repos/ibind/examples/rest_08_oauth.py:4) from ibind.client.ibkr_client.py import IbkrClient
      [7](file:///D:/git_repos/ibind/examples/rest_08_oauth.py:7) # Construct the client, set use_oauth=False, if working, try creating a live session by setting use_oath=True
      [8](file:///D:/git_repos/ibind/examples/rest_08_oauth.py:8) client = IbkrClient(use_oauth=False)

ModuleNotFoundError: No module named 'ibind.client.ibkr_client.py'; 'ibind.client.ibkr_client' is not a package

Is this something to do with the __init__.py file?
Once this is working I can try the code and fix the bugs.

[Voyz]

Better! Try doing the same with .vscode

[Voyz]

To be able to run it you need to add the root directory to PYTHONPATH:

https://www.simplilearn.com/tutorials/python-tutorial/python-path
https://stackoverflow.com/questions/41471578/visual-studio-code-how-to-add-multiple-paths-to-python-path
https://stackoverflow.com/questions/3402168/permanently-add-a-directory-to-pythonpath

You shouldn't need to change:

from ibind import IbkrClient to from ibind.client.ibkr_client.py import IbkrClient

Try that and let me know if it helps.

[hughandersen]

@Voyz Voy the path has been fixed (simple typo) and I'm working on getting live session and access tokens.
Can you review the code and add any suggestions/improvements?

[Voyz]

@hughandersen superb job on the first iteration of this PR 👏👏 You've followed the suggestions I made very well and did an amazing job resolving the module import / pythonpath issues you faced initially (I hate these too 🙄).

I've left a number of comments for you to address in order to do the next few steps towards getting this merged in, but honestly you're on the right path and are doing really well so far. Thanks!

[hughandersen]

OAuth session initialization (getting live session and access tokens) is now working. But getting the brokerage session established is not working yet (see examples/rest_09_oauth_session.py).
Once brokerage method is working I'll clean up the code.

[hughandersen]

Code can now get live_session_token,live_session_token_expires_ms data, and get market data using original ibind methods, see file rest_08_oauth_test.py.
To do:

• either use a .env file or os environment to store variables
• there is a conflict using **kwargs in line 183 of file rest_client.py

response = requests.request(method,url,headers=header_oauth,params=params,timeout=10)
# response = requests.request(method, url, verify=self.cacert,headers=header_oauth,params=params, timeout=self._timeout, **kwargs)

• endpoints need to be tested more thoroughly

@Voyz take a look when you get a chance

[Voyz]

Good progress @hughandersen 👏👏 I've left a few comments.

Do I understand you can communicate with the IBKR API now correctly using OAuth? Calling various endpoints in the example file works?

I didn't review the oauth_requests.py and get_session.py as these seem to be your WIP files - do I understand correctly?

[Voyz]

Great 👏 I understand these can get generated just once per session and don't need to be re-generated?

[hughandersen]

Good question. The live session token times out after 24hours, and I think it would need to be re-created if the session drops out. Maybe also allow the live session token to be re-created in a function?
It took some fiddling to get the order of the attributes correct (super before if self._use_oauth etc.), can you see any room for improvement?

[Voyz]

Well we do get that live_session_token_expires_ms value right - that should tell us when it expires?

Or even better - we could observe what does IBKR return back when we try to call its endpoints with an expired live_session_token. Then we catch that error and request a new live session token when it happens. No need to keep track of the expiry date, just handle it once it happens. What do you recon?

[hughandersen]

Yes, I could check what get returns after the expiry date, but I think there are other bigger bugs to fix right now.

[Voyz]

Sure. Can you add a TODO comment in here so that we remember to handle this later? We shouldn't release until we have a nice way to handle it

[hughandersen]

Ok I'll add a comment.

[Voyz]

We need to call this for things to work? If so, how about we do it automatically for the user?

[hughandersen]

I think so, but the IBKR document is unclear, and I needed to call it before calling the other endpoints.
I'll check with them.

[Voyz]

Oh nice initiative! Though I'd say just comment this line out and see if it works. If it does, then leave it out, if it doesn't leave it in 👍

[hughandersen]

I've emailed IBKR, lets see what they come back with.

[Voyz]

@hughandersen what's the status on this? Do we know if we need to call this? What happens if we don't? I see that you already make several requests before calling this one - do they succeed? If so, can we remove it?

[Voyz]

@hughandersen hold up, you got OAuth 2.0 with IBKR? Can you share what docs are you using to do this?

suggestions how a user can authenticate using localhost or OAuth1 or OAuth2?

Ideally this would be as straightforward as switching a few environment variables and/or constructor parameters.

---

Anyway, because this PR is already quite long, I strongly recommend we tackle OAuth1.0a in this PR and address OAuth2.0 later.

[hughandersen]

@Voyz Yes I have access to OAuth2 and I'm in the process of getting it working. I agree that we should finish this PR, and once its working start another one for OAuth2.
I still can't get access using OAuth1 and maybe my paper account is damaged. Since others can get access I think you should go ahead and merge the OAuth branch with the Master branch when you're happy with the code.

[Voyz]

Thanks for clarifying @hughandersen It's just that if OAuth 2.0 is already a viable option, then @janfrederik's argument to name things OAuth1a and OAuth2 gains a different light. Can you talk in more depth about it - did IBKR suggest you do it, or did you find a way to do it yourself? Did IBKR give you some documentation on how to set it up, or have you found some docs yourself? Can you share all the resources you're using to implement it? Anything you can add in context of OAuth2.0's viability will be useful here

[hughandersen]

@Voyz I was given access and docs by IBKR because I have a small business account, I don't think OAuth2 is open for individuals so I don't think we should change or ditch the work done on OAuth1. I think you should merge the code when you're confident, and then we can open another PR for OAuth2 with discussions etc. I think localhost, OAuth1 and OAuth2 can work together.

[Voyz]

Yes, we definitely will not ditch OAuth1.0a, this PR is pretty much ready to be merged in. But if OAuth2.0 is indeed already available to use for small business users - something I wasn't aware of and thought is still far off on the roadmap - then OAuth version reflected in the nomenclature would make sense.

I suggest we reflect it in the OAuthConfig object, but keep the other variables the same. Therefore, we'd rename it to OAuth1aConfig and anticipate the OAuth2Config, as per @janfrederik's suggestions. I think keeping use_oauth as a universal variable between the two can work - it would be the oauth_config parameter that would define which version we'd be using, ie:

# default to OAuth1aConfig when no oauth_config is specified
ibkr_client = IbkrClient(..., use_oauth=True)

# specify 1.0a
oauth_config = OAuth1aConfig(...)
ibkr_client = IbkrClient(..., use_oauth=True, oauth_config=oauth_config)

# specify 2.0
oauth_config = OAuth2Config(...)
ibkr_client = IbkrClient(..., use_oauth=True, oauth_config=oauth_config)

Naturally, we won't implement the OAuth2Config yet, but this would be the proposed way we'd use it once we have a chance to implement it.

Please let me know of any suggestions regarding this, otherwise I'll go ahead and implement the renaming for OAuth1aConfig.

[janfrederik]

I suggest we reflect it in the OAuthConfig object, but keep the other variables the same. Therefore, we'd rename it to OAuth1aConfig and anticipate the OAuth2Config, as per @janfrederik's suggestions. I think keeping use_oauth as a universal variable between the two can work - it would be the oauth_config parameter that would define which version we'd be using, ie:

Looks good as interface for users using one of the OAuth1a/2Config objects.

As for the env vars and var.py, I would suggest a mostly distinct set for oauth1a and for oauth2 to minimize confusion. Soemthing like this:

IBIND_USE_OAUTH                   --> IBIND_USE_OAUTH1A
IBIND_OAUTH_REST_URL              --> IBIND_OAUTH1A_REST_URL
IBIND_OAUTH_WS_URL                --> IBIND_OAUTH1A_WS_URL
IBIND_LIVE_SESSION_TOKEN_ENDPOINT --> IBIND_OAUTH1A_LIVE_SESSION_TOKEN_ENDPOINT
IBIND_ACCESS_TOKEN                --> IBIND_OAUTH1A_ACCESS_TOKEN
IBIND_ACCESS_TOKEN_SECRET         --> IBIND_OAUTH1A_ACCESS_TOKEN_SECRET
IBIND_CONSUMER_KEY                --> IBIND_OAUTH1A_CONSUMER_KEY
IBIND_DH_PRIME                    --> IBIND_OAUTH1A_DH_PRIME
IBIND_ENCRYPTION_KEY_FP           --> IBIND_OAUTH1A_ENCRYPTION_KEY_FP
IBIND_SIGNATURE_KEY_FP            --> IBIND_OAUTH1A_SIGNATURE_KEY_FP
IBIND_DH_GENERATOR                --> IBIND_OAUTH1A_DH_GENERATOR
IBIND_REALM                       --> IBIND_OAUTH1A_REALM
IBIND_INIT_OAUTH                  --> IBIND_INIT_OAUTH
IBIND_INIT_BROKERAGE_SESSION      --> IBIND_INIT_BROKERAGE_SESSION
IBIND_MAINTAIN_OAUTH              --> IBIND_MAINTAIN_OAUTH
IBIND_SHUTDOWN_OAUTH              --> IBIND_SHUTDOWN_OAUTH

[Voyz]

Please see the ibind[oauth]==0.1.10-rc17 with the most recent renaming from OAuth to OAuth 1.0a or OAuth1a. This is due to @hughandersen revealing OAuth 2.0 is already available to business users to ensure clarity once we get a chance to implement the 2.0.

• OAuth 1.0a - used in text, comments and documentation
• OAuth1a (and its variations) - used in code

Note:

• Env vars specific to OAuth 1.0a have had OAUTH1A section added, eg.
  • IBIND_LIVE_SESSION_TOKEN_ENDPOINT -> IBIND_OAUTH1A_LIVE_SESSION_TOKEN_ENDPOINT
• Env vars that are common stayed the same, eg: IBIND_USE_OAUTH, IBIND_INIT_OAUTH, etc.
• var.py now has two sections in order to distinguish between OAuth common and OAuth 1.0a specific variables.
• OAuth common variables have been moved to now-abstract OAuthConfig base class:
  • init_oauth
  • init_brokerage_session
  • maintain_oauth
  • shutdown_oauth
• OAuth 1.0a specific variables are now in the OAuth1aConfig concrete class. Note that variables don't carry any indicators that they belong to oauth1a ; they are the same as before, eg OAuth1aConfig(dh_generator='foo')
• ibind.support.oauth module has been removed; OAuth-related code has been moved into ibind.oauth subpackage, with OAuth 1.0a code living within ibind.oauth.oauth1a module (most of the the old oauth.py file)
• rest_08_oauth.py example has been updated to reflect these changes

The usage is pretty much the same, just change OAuthConfig for OAuth1aConfig, or in case of env vars, add OAUTH1A part where needed.

I think this will conclude the development on this PR, so I'll give it a few days for any last minute tests and feedback and merge it in. If you get a chance, please try this recent version out and let us know if it works fine or if you run into any issues. Thanks @janfrederik for bringing the naming into our attention and @hughandersen for reporting the 2.0 availability! 🙌

[hughandersen]

@Voyz That sounds like a good plan.

[salsasepp]

@Voyz FYI, 0.1.10-rc17 has been working just fine here all day. Thank you all for the work you're putting in!

[hughandersen]

@Voyz I've done a merge in the oauth branch, can you now merge the oauth branch with the master branch?

[Voyz]

@hughandersen absolutely, I'm planning to do this next week as I'm caught up in other matters and cannot sit down to writing the WiKi yet, which I'd like to follow up with promptly afterwards. Thanks for merging in @SmartJQ's changes 👍

[Voyz]

Going ahead with a merge 🥳

Many thanks to @hughandersen for writing it, you did a superb job carrying out most of the work on your own based on my suggestions. As you can already see, this seemed to have been a very popular and needed addition to the library, so once again great job leading the change here 🙌

Also, thanks to everyone who's contributed in making it better; I appreciate all the useful comments, suggestions and questions, in particular @thouseef @salsasepp @weklund @janfrederik @SmartJQ thanks for making this addition so great 🙌

There is an issue I've created to track the OAuth 1.0a documentation:

#49

I'd appreciate your input there on any topic related. In particular on the items I personally don't have access - related to requesting OAuth access from IBKR. I've marked these tasks with 🙈 emoji, if anyone would like to write one of these and share, let me know in the comment in that issue.

Thanks everyone!

[message]

Congratulations on the milestone guys 🎉

[hughandersen]

@Voyz I hope the merge goes well with no issues and I'm pleased the addition is useful to others.
I think it was a combined effort, and we worked well together.

[thouseef]

@Voyz Glad to see it merged. Congratulations @hughandersen for achieving this milestone and many thanks for taking the initiative.

Sorry, I couldn't spend more time here .. Once the craziness is over, I shall join the discussions again.

[Voyz]

Hey @hughandersen @thouseef @salsasepp @janfrederik would you mind dropping me an email when you find a minute? You did a great job with those recent discussions and contributions on this PR, and I wondered if I could ask you a couple of questions in private: [email protected]

(I'm reaching out here as I cannot find any emails to contact you directly)

[Voyz]

Hey guys, there's some users who are wondering if you can use OAuth API with paper credentials: #61

Has anyone here succeeded in using paper credentials? If so, do you think you could chip in and outline your process? Thanks 👍

[janfrederik]

Hey guys, there's some users who are wondering if you can use OAuth API with paper credentials: #61

Has anyone here succeeded in using paper credentials? If so, do you think you could chip in and outline your process? Thanks 👍

I have it working. Answered in #61 (comment).

[hughandersen]

My login via OAuth1.0 was working well.
I initially set it up by logging into the API registration page using my Paper account and pw and setting all the keys. It worked fine for a while until I changed the keys, then for no reason it died. However, I was given OAuth2 access, so IBKR support suggested I use that instead.

[Voyz]

thanks @janfrederik 🙌

@hughandersen understood, strange it stopped for you entirely. IBKR told me that it's pretty exceptional they'd give out OAuth 2.0 access, so consider yourself lucky 👍 How is that going btw, is it working well for you?