I am trying to bring the data from GCP Bigquery to Azure data lake using Azure data factory. I was able to setup and was able to bring the data into Azure. But my problem is, the GCP refresh token is keep on expiring.
How to avoid GCP refresh token expiry?
How to generate new refresh token from ADF every time we load the data?
Any help is much appreciated.
I was dealing with the same issue. I Followed the official documentation where it is recommended the tutorial indicated by #Veikko. On this tutorial and during my short research, everyone establish that the refresh_token won't expire. During my test, the refresh_token provided me access during the day. Once the day finished, the access is revoked.
The solution to my issue was to use ServiceAuthentication. The problem is you will require a Self-hosted Integration Runtime. You just need to configure a Service Account on BigQuery. This will provide you and email and a .p12 file that you need to copy on the machine where you installed the integration runtime and done!
My suspect for the reason is that I use multi-factor authentication and VPN to grant access, and it may be the root cause. But it is an open issue for me. Any comment is welcome.
Refresh token should not be expiring. Make sure you are using a refresh token, not an access token. Azure Data Factory will handle internally obtaining and refreshing the access token for you and you will not need to worry about it.
I think that Azure Data Factory BigQuery connector accepts also access token in refresh token -field and I would assume it would behave exactly like in your situation and would not work after access token has expired.
Good documentation on how to authenticate with BigQuery connector can be found here: https://jpda.dev/getting-your-bigquery-refresh-token-for-azure-datafactory-f884ff815a59. In phase Getting our refresh token make sure you use value from key refresh_token, not access_token.
More info on authentication types can be found from Microsoft documentation here: https://learn.microsoft.com/en-us/azure/data-factory/connector-google-bigquery#using-user-authentication
This topic is very confusing given the wide variety of advice especially generalized advice that doesn't apply to Bigquery/GA and ADF! #Veikko would be incorrect in saying it shouldn't expire - Google says differently regarding their tokens.
Refresh token expiration is controlled by the resource and according to Bigquery documentation...
"Access tokens have limited lifetimes. If your application needs access to a Google API beyond the lifetime of a single access token, it can obtain a refresh token. A refresh token allows your application to obtain new access tokens."
I read where this used to support control of the timeout but after Dec 30 2021 it's mandatory at 30 minutes.
Regardless you have to get a refresh token everytime you query Bigquery if you want to use OAuth2.0. I've read a couple of methods of getting the refresh token but are pretty complicated (at least to me). My workaround is same as #lalfab....create hosted IR and store the p12 version of the credentials on the VM referenced by the IR there. It works but of course means maintaining a VM and starting/stopping when needed. I'd love to replace this with OAuth2.0 connection to the Bigquery client id. Would love to find a simple way of doing this. But ADF's bigquery linked server also doesn't support dynamic expression to replace refresh token at connect time so it seems like we need to wait until Microsoft makes the bigquery adapter more intelligent.
Well all of these answers are actually correct.
After a lot of research, I think I found the issue.
The issue is that refresh token DO expire, as long as the project is not published yet. The button to publish it, is under the OAuth consent screen.
Please read: https://developers.google.com/identity/protocols/oauth2#expiration
I havent tested it yet, but have a strong feeling that this is causing the issues for all of us. Hope it helps!
Related
For the past few years I have been using an unattended remote server to process invoices through the XERO API (Oauth 1.0)
Periodically (every financial year) we create a new XERO organisation to keep things tidy and avoid slow down.
I have just come to create a new organisation and associated app but have found that I can only use Oauth 2.0, which I do not have a massive issue with, BUT the fact I have to 'user' authenticate is going to be a real problem as my process is 'unattended' and started via CRON jobs.
Can anyone tell me if there is a way around this? and if not are there any solutions to do this?
Alternatively is there any way I can change one of my existing Oauth 1.0 apps to point to a different organisation (i.e. the new one I have just created)?
It seems a little short sighted not to have considered unattended processes, I cannot be the only person doing this??
Any help or pointers would be greatly appreciated.
Thanks,
Mike.
Yes you are not the only one doing this :) & yes, private apps are essentially deprecated end of 2020 - the move was not taken lightly. Since every API action through Xero's api is on behalf of a user account the team decided to move towards OAuth2.0 (Industry Standard) with a user consent screen.
If you need these long standing api connections on behalf of a user - they will need to initially authenticate that API connection at least a single time to get you an access_token and `refresh_tokenà. Access tokens are valid for 30 minutes, while the refresh token is good for 60 days.. So as long as you refresh > 2 months you can persist that longtail process.
If you don't have the means to build out that initial authentication screen to have your user validate on their own, you can use this CLI tool to get your initial token set to securely store in your remote server. An additional change is that that (or some) process will need to ensure the token is refreshed before use, and has given permissions to connect to a specific user's tenant-id as they might be a part of multiple Xero orgs & that manual consent screen is where a user selects which tenant/org they are giving API permissions to.
CLI to get Xero tokens from the command line
https://github.com/XeroAPI/xoauth
More info here: https://community.xero.com/developer/discussion/109207632#answer110970761
UPDATE
client_credentials aka machine to machine are coming to Xero's OAuth2.0 gateway.
You can read more about it here:
https://developer.xero.com/announcements/custom-integrations-are-coming/
Background
I am able to create Trello cards from Google Apps Script via the
Trello API using the OAuth 1.0 library. The principle is proven/code
works.
I have two distinct Google Apps Scripts projects that need to be able to create Trello cards.
The code in the two different Apps Scripts/Projects is identical - including the same API key/secret.
Only one Apps Script will create a Trello card. This is my problem.
If I reauthorise the other Apps Script, that script will work and the other will give me an API return of "invalid token" and vise-versa. Only one works at a time, but I need both to work.
My thoughts
I think that Trello, via OAuth, see each Apps Script is its own distinct project.
I think that because of this it won't let both apps use the same API key/secret to work with my Trello account. Only one project appears to be able to use the key/secret.
If this is the case I don't know how to make each Apps Script its own project for the Trello API to work for both simultaneously.
Help needed
Does anyone know how to make this work? I need both scripts to be able to create Trello cards. I have a feeling that each apps needs to identify itself uniquely, but I honestly have no idea.
This is really an OAuth logic issue, it's a feature, not a bug. In OAuth, your application exchanges refresh tokens for access tokens. The access tokens only have a limited life span.
When you use a refresh token to generate a new access token, you also get a new unique refresh token and your script stores this for future use, the old refresh token is no longer valid. Similarly, when you re-authorize the application, you get fresh tokens, and any previously generated tokens are rendered invalid.
So when you authorise one script using the same Client ID and Client secret as the other script, you get a new access token and refresh token, and the old credentials, stored by the other script, become invalid.
As a result, the other script can no longer exchange the refresh token it has stored for new access tokens, and it no longer works. Once you re-authorize this copy, the refresh token and access token in the other copy are invalidated in the same way. So you end up going in circles.
You have two options:
Set up a separate OAuth Client (with different Client ID and Client Secret) for each script.
Modify your scripts to use the same storage location for the OAuth Access Token and Secret.
The first approach is going to give you the most reliable consistent results. If you try the second approach, you could still have cases where the scripts run at the exact same time, and one has valid tokens while the other tries to use the now invalid ones. (race conditions).
I am very new to MS Graph and Office 365 and have made good progress. I am an O365 Global Admin for my organisation (a school) and have app development experience. There is a lot of scope for using MS-Access databases in our context for "globally" managing the O365 content. eg contacts, distribution lists and planner tasks. We want to manage these from an on-premises ms-access database or two and with an admin person authenticating the ms-graph activity, ideally.
So, to test, I created a new db and have managed to get it to consume the following endpoint using VBA but with no user authentication for now.
https://graph.microsoft.com/v1.0/groups
However, when I try
https://graph.microsoft.com/v1.0/planner/plans/with my plan id here
I get 401 - Unauthorized: Access is denied due to invalid credentials.
So, clearly my Application registration is wrong or my authentication or both! I have spent hours searching for examples and help and because of the evolving nature of the ecosystem I am finding it pretty hard to work out what I should do now (as opposed to a year or two ago).
The authorisation that generates the access_token that works to allow me access to the groups is:
POST
https://login.microsoftonline.com/{my tenant id here}/oauth2/token
grant_type=client_credentials
client_id={my client id}
client_secret={my url encoded secret} resource=https://graph.microsoft.com
but using that same access_token for the planner tasks throws the 401 error.
My app permissions look like this:
I presume this is because of the difference between the Application and Delegated types but have not fully grasped it all yet. And, I suspect I am using the wrong authentication flow anyway. :-(
So, my questions are:
1. Do my permissions look right?
2. Is my authentication flow correct? Should I be using these instead? ie have I been working from old information?
https://login.microsoftonline.com/{my tenant id here}/oauth2/v2.0/authorize
https://login.microsoftonline.com/{my tenant id here}/oauth2/v2.0/token
As you can tell I have become somewhat confused. If anyone can point me in the right overall direction given what I am attempting that would be so helpful.
Thanks so much,
Murray
1. Do my permissions look right?
Yeah undoubtedly, your azure portal permission seems alright. You need dedicated permission for that also need to grant admin consent which you have done perfectly shown on screen shot.
2. Is my authentication flow correct?
As you are using Client Credentials Grant Flow request format seems alright. But I doubt this flow is suitable for the API you are trying to call. because this API requires dedicated permission.
3. Should I be using these instead?
Since this API need dedicated permission you could use authorization code grant flow.
Follow below steps to get your token using Authorization Code grant flow
Get Authorization Code:
https://login.microsoftonline.com/YourTenant.onmicrosoft.com/oauth2/v2.0/authorize?client_id={ClientId}&response_type=code&redirect_uri={redirectURI}&response_mode=query&scope=https://graph.microsoft.com/.default
Request Token oauth2/V2.0/token with your code:
Request URL: https://login.microsoftonline.com/common/oauth2/V2.0/token Or https://login.microsoftonline.com/YourTenant.onmicrosoft.com/oauth2/V2.0/token
Method: POST
Request Body Format
client_id:Your_Clinet_Id
scope:https://graph.microsoft.com/.default
redirect_uri:Your_Portal_Redirect_URI
grant_type:authorization_code
client_secret:Your_Client_Secret
code: Paste Code Here
Decode Token:
You could decode your token on https://jwt.io/ and make sure you have required permission on your azure portal.
4. Have I been working from old information?
No, Information has no issue so far I have gone through.
Note: For for details implementation of Authorization Code grant flow you could take a look official docs
I’m building a react native app that will interact with APIs that I also write/manage. I have found Auth0 documentation for implementing this flow, but I’m not sure on where/when to save the tokens. I want to be sure I nail this step, because I feel like it has the potential to reduce the security of the flow by a great deal if I don’t do it correctly.
Here is the flow as I understand it (no error handling, only happy-path for sake of brevity):
A user enters the app for the first time, or is not already logged in
They log in using the Auth0 web-login-thingy
I receive a token
I can use the token to authenticate with my API
Questions:
Do I store that token? I don’t want my users to have to log in every time they use the app. If I do store the token, where do I store it?
If I’m not storing it, what do I do? Do I ping an authentication/authorization endpoint with Auth0 every time they open the app and get a new token?
Say I am storing the tokens, if I'm using the ID token for user data, should I be hitting the API again regularly to keep it up to date? Only when the user opens the app again? Not until they trigger a change in the app?
Instead of using the ID token for user data, should I just use that to get the user's ID and ping my database for user data?
I have the basics of this flow, and I'm able to sandbox it, but I want to start applying production-ready app logic to this flow and that's where I'm stuck. I’m a little lost here, so any help is good help.
Thanks!!
Here's a brief answer to your questions when using Auth0:
Yes! you store it, the most secure way to store the token is in your device's local storage, that way it is not kept either in application's state or in a global variable.
2&3. See above, but to add more information, you can configure your tokens to have an expiry length. in theory you would convert this 'expiry time from inception' to a date object, and can do one of two things; you can request a new token using the Refresh Token (that comes with the original) once the expiry has been reached, or force the user to re-log in and re issue a new token at this time (i prefer the latter, prevents people from just renewing their tokens forever as long as they remain logged in)
Use the auth token to request user information after login, this can be stored in app state/global variables/wherever. You then want to use the auth token in the Authorization Header for each API call, along with whatever data you are sending. this ensures that even once someone is INSIDE the application, they need to have a valid token to actually do anything involving data (imagine someone back-dooring into your app and skipping the authorization, or using something like postman to just hammer your API with garbage). it would work something like this: GET userData { Header: auth token } -> GET userProfile (by sending your user ID returned from GET userData) PLUS {Header: auth token }
I can give more in depth examples if you wish, and i apologize if i misunderstood any of the question and gave redundant/incorrect answers
Edit: Resources about using secure storage for keys
Document for when to use in-memory storage Vs persistent storage. The TL;DR is use in-memory if the key is expected to expire before a standard session duration, and persistent for storing a key between sessions
https://hackernoon.com/mobile-api-security-techniques-682a5da4fe10
link to Keychain Services doc
https://developer.apple.com/documentation/security/keychain_services#//apple_ref/doc/uid/TP30000897-CH203-TP1
link to SharedPreferences doc
https://developer.android.com/reference/android/content/SharedPreferences.html
AsyncStorage is a simple, unencrypted, asynchronous, persistent,
key-value storage system that is global to the app. [1]
You could store it in your AsyncStorage, but thats not necessarily a secure location itself (e.g. not encrypted, accessible on rooted devices...). Typically clients will issue access tokens that last anywhere from several hours to a couple days and these will provide their owner access to your API-resources. If there is sensitive data behind your login screen, you're probably better off simply re-doing the auth-flow and invalidate older access tokens on login.
Authorization code Grant : I know the code is short lived token exchanged for the real long-lived access token. I have gone through the Oauth 2.0 but could not find this information so asking here:
What is the life cycle of code?
Is it for only one-time use?
How many times can a code be exchanged to get access token?
What happen to a code after access token is given for that code?
I am using oAuth 2.0 plugin on Kong API gateway. it is keeping the code alive for a particular time and multiple access token can be generated using same code by that time.
Is it the expected behaviour?
Thanks for any advice.
Authorization Code must be short lived and should be one time use to avoid fake use. So to answer your questions
What is the life cycle of code?
when the user authenticate using authorication_code flow, the once authenticated and granted access for scopes, an short lived (say 1 minute) valid code will be created and sent back to the redirect uri.
Is it for only one-time use?
yes it must be one-time use for best security, when access token is requested using authorication_code, then either the request succeeds or failed (due to some validation error or server error), the authorization code must be deleted or marked as used(depending how you wanted to use it)
How many times can a code be exchanged to get access token?
One authorization_code can grant only one access token, since the code will be revoked once an access token is issued.
What happen to a code after access token is given for that code?
Best practice, the code can be deleted
Check out google oauth2.0 documentations for better understanding and see how its used.
https://developers.google.com/identity/protocols/OAuth2WebServer
For Kong issue it seems its a bug in kong and they promised to give fix in 0.9 release.
Check this discussion.
code is short-lived one-time access token.
once a it is exchanged for a access token, it should get marked as invalid.
for Kong issue its better to ask it here