I am developing an API that will be used by users of my customers. Here is what the flow will look like:
User of my cloud based service creates an API key.
User embeds the API key into their own custom applications.
User deploys the application to their own end users.
The application talks to our API.
I am looking for advice on how to secure this API. I see a few issues:
API key has to be embedded into the users application and is therefore vulnerable to being stolen and abused.
Once an API key is compromised, it can easily be disabled, but how will my users update their applications to use a new API key short of having to rebuild the application and redeploy.
Does anyone have any ideas on how to design this?
I may be mistaken but maybe you could have your customers' users talk to your customers' APIs instead. Basically, your customers would keep their secret key on their servers, and not embed them in the clients they give their users, so it couldn't be hijecked (unless their server was compromised of course). Then the users would talk to your API through your customers' APIs.
It would be slower and need more work on the part of your customers, but also safer.
Two solutions that I can see to this, although I'm sure there are more..
Use oauth's RSA signature method, and implement a secure certification exchange of keys using your "cloud based service" as the exchange mechanism (or a public cert provider).
Implement a service that allows clients to "renew" their consumer key/secret automatically, but then secure that mechanism using RSA or some other public key encryption method.
Both of these are not easy, and would require your user's applications to "phone home" in order to update their consumer keys.
In the future I think OAuth 2 will provide at least protocol definitions for things like this, but for now, if you're using OAuth 1.0a, what you want to do doesn't really fit into the spec very well (i.e. you have to design much of it yourself.)
Related
There are primarily two ways to authenticate using Google's reCAPTCHA Enterprise in a non-Google cloud environment (we use AWS). Google recommends using Service Accounts along with their Java client. However, this approach strikes me as less preferable than the second way Google suggests we can can authenticate, namely, using Api Keys. Authenticating via an Api Key is easy, and it’s similar to how we commonly integrate with other 3rd party services, except rather than a username and password that we must secure, we have an Api Key that we must secure. Authenticating using Service Accounts, however, requires that we store a Json file on each environment in which we run reCAPTCHA, create an environment variable (GOOGLE_APPLICATION_CREDENTIALS) to point to that file, and then use Google's provided Java client (which leverages the environment variable/Json file) to request reCAPTCHA resources.
If we opt to leverage Api Key authentication, we can use Google’s REST Api along with our preferred Http Client (Akka-Http). We merely include the Api Key in a header that is encrypted as part of TLS in transit. However, if we opt for the Service Accounts method, we must use Google’s Java client, which not only adds a new dependency to our service, but also requires its own execution context because it uses blocking i/o.
My view is that unless there is something that I’m overlooking--something that the Service Accounts approach provides that an encrypted Api Key does not--then we should just encrypt an Api Key per environment.
Our Site Key is locked down by our domain, and our Api Key would be encrypted in our source code. Obviously, if someone were to gain access to our unencrypted Api Key, they could perform assessments using our account, but not only would it be exceedingly difficult for an attacker to retrieve our Api Key, the scenario simply doesn't strike me as likely. Performing free reCAPTCHA assessments does not strike me as among the things that a sophisticated attacker would be inclined to do. Am I missing something? I suppose my question is why would we go through the trouble of creating a service account, using the (inferior) Java client, storing a Json file on each pod, create an environment variable, etc. What does that provide us that the Api Key option does not? Does it open up some functionality that I'm overlooking?
I've successfully used Api Keys and it seems to work fine. I have not yet attempted to use Service Accounts as it requires a number of things that I'm disinclined to do. But my worry is that I'm neglecting some security vulnerability of the Api Keys.
After poring a bit more over the documentation, it would seem that there are only two reasons why you'd want to explicitly choose API key-based authentication over an Oauth flow with dedicated service accounts:
Your server's execution environment does not support Oauth flows
You're migrating from reCAPTCHA (non-enterprise) to reCAPTCHA Enterprise and want to ease migration
Beyond that, it seems the choice really comes down to considerations like your organization's security posture and approved authentication patterns. The choice does also materially affect things like how the credentials themselves are provisioned & managed, so if your org happens to already have a robust set of policies in place for the creation and maintenance of service accounts, it'd probably behoove you to go that route.
Google does mention sparingly in the docs that their preferred method for authentication for reCAPTCHA Enterprise is via service accounts, but they also don't give a concrete rationale anywhere I saw.
Consider an API that a client accesses directly (machine to machine) and that doesn't require user-specific authentication. The way I understand it, in client_credentials, the client must store a client_id and client_secret that it uses to acquire and refresh tokens. With an API key, the client just stores the key. What makes OAuth more secure in this case? It would appear to me that if the API key is never compromised, no attacker could pose as the intended client. And if the API key is compromised, it is effectively the same as compromising the client_id and client_secret, which an attacker would be able to use to obtain tokens and access the data in the API, posing as the client.
edit: clarified this is a machine-to-machine call
TLDR;
The difference comes down to direct access vs. delegated access.
OAuth allows you to make delegated access. The benefits of delegated access don't change if there is a user involved or not. The same arguments that make the OAuth Authorization code flow attractive for user-to-machine access, apply to the OAuth Client credentials flow for machine-to-machine access.
Ask yourself, do you want the resource server to handle client credentials or not?
On confidential clients for machine-to-machine access, the cost of delegated access vs. direct access may very well outweigh the benefits. That's why so many APIs still use API keys. You'll have to decide that for your individual use case.
Differences
In the OAuth client credentials flow, the client sends an access token to the resource server, which it got beforehand by the authorization server after presenting its client ID and secret. The resource server never sees the client secret. With an API key, the client sends the key with every request.
OAuth adds an additional layer of indirection with the authorization server, such that the credentials themselves never get transmitted to the resource server. This allows the authorization server to give the client only access for a limited amount of time or with limited permissions, without ever needing to change the actual client credentials. It also allows to revoke access tokens without revoking the credentials themselves. For multiple instances of a client this allows you to revoke access for some but not all.
Of course this all comes at the cost of a more complex implementation, and an additional roundtrip from the client to the authorization server.
I won't touch on transmission (URL, header, body, etc.) or format (random string, signed JWT, etc.), since these can be the same for access tokens just as for API keys.
Another, maybe not so obvious, advantage of OAuth is having a clear spec that libraries, documentation and discussions can be based on. With direct access there is no single best practice and different people may understand different things when referring to direct access methods like API keys.
With client credential flow your Client Id and Client Secret are sent to the authorization server to get back an access token. For all subsequent request to the API/resource servers, you pass the access token and not the client credentials themselves. The access token is usually a JWT, which is a set of encoded claims including the token expiry (exp), not before (nbf), token issuer (iss), authorized party (azp), roles, permissions, etc.
This has a number of advantages over a simple API Key approach. e.g.
If the access token (which is included in requests to the API/resource server) is compromised, it's only valid until it expires (which is typically ~1 day for M2M tokens). If an API Key is compromised, it can be used indefinitely or until it's explicitly blocked by the API/resource server.
JWT access tokens are encoded JSON objects that contains a number of fields (a.k.a. claims) that can be used for fine grained authorization e.g. roles, permissions, grant type, authorized party etc. An API Key is generally opaque and is all or nothing when it comes to auth.
You machine tokens can get validated and authorized on the API/resource servers the same way as your user tokens, so you don't end up with multiple auth implementations on the back-end.
OAuth Client Credentials Flow
What is the security difference between API Keys and the client credentials flow of OAuth?
OAuth client credentials flow is not meant to be used by public clients, just between machines.
From auth0.com/docs:
Client Credentials Flow
With machine-to-machine (M2M) applications, such as CLIs, daemons, or services running on your back-end, the system authenticates and authorizes the app rather than a user. For this scenario, typical authentication schemes like username + password or social logins don't make sense. Instead, M2M apps use the Client Credentials Flow (defined in OAuth 2.0 RFC 6749, section 4.4), in which they pass along their Client ID and Client Secret to authenticate themselves and get a token.
So, I am not sure what is your scenario, but I will assume in my reply that you are referring to public clients.
If it is in the public client code, then it is public
The way I understand it, in client_credentials, the client must store a client_id and client_secret that it uses to acquire and refresh tokens.
Yes, it needs to be stored in the client code for the client to be able to obtain the OAuth token.
If you use the client_secret from a web app or mobile app you are making it public, therefore not a secret anymore.
Extracting secrets from public clients
For example, in a web app all it takes to extract the client_secret is to hit F12 in the browser and search for it, thus how much time can this take?
Now, in a mobile app, some may think it's secure because they are compiled into a binary but is almost as easy as it is in the browser, because we have several open-source tools that can help us with this task, like the MobSF framework, and on Linux, you can even achieve this with the strings command. Using the MobSF to perform static binary analysis on the mobile app binary allows for anyone without hacking knowledge to easily extract the client_secret in minutes, just like I show in my article How to Extract an API key from a Mobile App with Static Binary Analysis:
The range of open source tools available for reverse engineering is huge, and we really can't scratch the surface of this topic in this article, but instead, we will focus in using the Mobile Security Framework(MobSF) to demonstrate how to reverse engineer the APK of our mobile app. MobSF is a collection of open-source tools that present their results in an attractive dashboard, but the same tools used under the hood within MobSF and elsewhere can be used individually to achieve the same results.
So, the process of extracting the api-key in my article is the same you will use to extract the client_secret or any other string of your interest in the mobile app binary.
OAuth or API Key?
What makes OAuth more secure in this case? It would appear to me that if the API key is never compromised, no attacker could pose as the intended client. And if the API key is compromised, it is effectively the same as compromising the client_id and client_secret, which an attacker would be able to use to obtain tokens and access the data in the API, posing as the client.
If used from a public client neither are secure, because if read my linked article, you understand by now how easy is to bypass an API Key or extract the client_secret and client_id.
So, if your client is public you should not use the OAuth client credential flow, thus you need to go with the insecure API key approach or you can be more diligent and try to apply defence-in-depth approaches, but this will depend if the API clients are only web apps or mobile apps or both.
If your API clients are only web apps I invite you to read my answer to the question Secure API data from calls out of the app, especially the section dedicated to Defending the API Server.
In the case the API clients are only mobile apps then I recommend you to read this answer I gave to the question How to secure an API REST for mobile app?, especially the sections Securing the API Server and A Possible Better Solution.
On the other hand, if your API clients are both a web app and a mobile app I recommend you to apply the security measures more relevant to you from both answers linked above.
Remember that security is always about adding as many layers of defences as you can afford or it's required by law. Even in the past century, the castles were built with a lot of different security defence layers, thus this is nothing new to the digital era.
Do You Want To Go The Extra Mile?
In any response to a security question I always like to reference the excellent work from the OWASP foundation.
For APIS
OWASP API Security Top 10
The OWASP API Security Project seeks to provide value to software developers and security assessors by underscoring the potential risks in insecure APIs, and illustrating how these risks may be mitigated. In order to facilitate this goal, the OWASP API Security Project will create and maintain a Top 10 API Security Risks document, as well as a documentation portal for best practices when creating or assessing APIs.
For Mobile Apps
OWASP Mobile Security Project - Top 10 risks
The OWASP Mobile Security Project is a centralized resource intended to give developers and security teams the resources they need to build and maintain secure mobile applications. Through the project, our goal is to classify mobile security risks and provide developmental controls to reduce their impact or likelihood of exploitation.
OWASP - Mobile Security Testing Guide:
The Mobile Security Testing Guide (MSTG) is a comprehensive manual for mobile app security development, testing and reverse engineering.
For Web Apps
The Web Security Testing Guide:
The OWASP Web Security Testing Guide includes a "best practice" penetration testing framework which users can implement in their own organizations and a "low level" penetration testing guide that describes techniques for testing most common web application and web service security issues.
When using the OAuth protocol, you need a secret string obtained from the service you want to delegate to. If you are doing this in a web app, you can simply store the secret in your data base or on the file system, but what is the best way to handle it in a mobile app (or a desktop app for that matter)?
Storing the string in the app is obviously not good, as someone could easily find it and abuse it.
Another approach would be to store it on your server, and have the app fetch it on every run, never storing it on the phone. This is almost as bad, because you have to include the URL in the app.
The only workable solution I can come up with is to first obtain the Access Token as normal (preferably using a web view inside the app), and then route all further communication through our server, which would append the secret to the request data and communicate with the provider. Then again, I'm a security noob, so I'd really like to hear some knowledgeable peoples' opinions on this. It doesn't seem to me that most apps are going to these lengths to guarantee security (for example, Facebook Connect seems to assume that you put the secret into a string right in your app).
Another thing: I don't believe the secret is involved in initially requesting the Access Token, so that could be done without involving our own server. Am I correct?
Yes, this is an issue with the OAuth design that we are facing ourselves. We opted to proxy all calls through our own server. OAuth wasn't entirely flushed out in respect of desktop apps. There is no prefect solution to the issue that I've found without changing OAuth.
If you think about it and ask the question why we have secrets, is mostly for provision and disabling apps. If our secret is compromised, then the provider can only really revoke the entire app. Since we have to embed our secret in the desktop app, we are sorta screwed.
The solution is to have a different secret for each desktop app. OAuth doesn't make this concept easy. One way is have the user go and create an secret on their own and enter the key on their own into your desktop app (some facebook apps did something similar for a long time, having the user go and create facebook to setup their custom quizes and crap). It's not a great experience for the user.
I'm working on proposal for a delegation system for OAuth. The concept is that using our own secret key we get from our provider, we could issue our own delegated secret to our own desktop clients (one for each desktop app basically) and then during the auth process we send that key over to the top level provider that calls back to us and re-validates with us. That way we can revoke on own secrets we issue to each desktop client. (Borrowing a lot of how this works from SSL). This entire system would be prefect for value-add webservices as well that pass on calls to a third party webservice.
The process could also be done without delegation verification callbacks if the top level provider provides an API to generate and revoke new delegated secrets. Facebook is doing something similar by allowing facebook apps to allow users to create sub-apps.
There are some talks about the issue online:
http://blog.atebits.com/2009/02/fixing-oauth/
http://groups.google.com/group/twitter-development-talk/browse_thread/thread/629b03475a3d78a1/de1071bf4b820c14#de1071bf4b820c14
Twitter and Yammer's solution is a authentication pin solution:
https://dev.twitter.com/oauth/pin-based
https://www.yammer.com/api_oauth_security_addendum.html
With OAUth 2.0, you can store the secret on the server. Use the server to acquire an access token that you then move to the app and you can make calls from the app to the resource directly.
With OAuth 1.0 (Twitter), the secret is required to make API calls. Proxying calls through the server is the only way to ensure the secret is not compromised.
Both require some mechanism that your server component knows it is your client calling it. This tends to be done on installation and using a platform specific mechanism to get an app id of some kind in the call to your server.
(I am the editor of the OAuth 2.0 spec)
One solution could be to hard code the OAuth secret into the code, but not as a plain string. Obfuscate it in some way - split it into segments, shift characters by an offset, rotate it - do any or all of these things. A cracker can analyse your byte code and find strings, but the obfuscation code might be hard to figure out.
It's not a foolproof solution, but a cheap one.
Depending on the value of the exploit, some genius crackers can go to greater lengths to find your secret code. You need to weigh the factors - cost of previously mentioned server side solution, incentive for crackers to spend more efforts on finding your secret code, and the complexity of the obfuscation you can implement.
Do not store the secret inside the application.
You need to have a server that can be accessed by the application over https (obviously) and you store the secret on it.
When someone want to login via your mobile/desktop application, your application will simply forward the request to the server that will then append the secret and send it to the service provider. Your server can then tell your application if it was successful or not.
Then if you need to get any sensitive information from the service (facebook, google, twitter, etc), the application ask your server and your server will give it to the application only if it is correctly connected.
There is not really any option except storing it on a server. Nothing on the client side is secure.
Note
That said, this will only protect you against malicious client but not client against malicious you and not client against other malicious clients (phising)...
OAuth is a much better protocol in browser than on desktop/mobile.
There is a new extension to the Authorization Code Grant Type called Proof Key for Code Exchange (PKCE). With it, you don't need a client secret.
PKCE (RFC 7636) is a technique to secure public clients that don't use
a client secret.
It is primarily used by native and mobile apps, but the technique can
be applied to any public client as well. It requires additional
support by the authorization server, so it is only supported on
certain providers.
from https://oauth.net/2/pkce/
For more information, you can read the full RFC 7636 or this short introduction.
Here's something to think about. Google offers two methods of OAuth... for web apps, where you register the domain and generate a unique key, and for installed apps where you use the key "anonymous".
Maybe I glossed over something in the reading, but it seems that sharing your webapp's unique key with an installed app is probably more secure than using "anonymous" in the official installed apps method.
With OAuth 2.0 you can simply use the client side flow to obtain an access token and use then this access token to authenticate all further requests. Then you don't need a secret at all.
A nice description of how to implement this can be found here: https://aaronparecki.com/articles/2012/07/29/1/oauth2-simplified#mobile-apps
I don't have a ton of experience with OAuth - but doesn't every request require not only the user's access token, but an application consumer key and secret as well? So, even if somebody steals a mobile device and tries to pull data off of it, they would need an application key and secret as well to be able to actually do anything.
I always thought the intention behind OAuth was so that every Tom, Dick, and Harry that had a mashup didn't have to store your Twitter credentials in the clear. I think it solves that problem pretty well despite it's limitations. Also, it wasn't really designed with the iPhone in mind.
I agree with Felixyz. OAuth whilst better than Basic Auth, still has a long way to go to be a good solution for mobile apps. I've been playing with using OAuth to authenticate a mobile phone app to a Google App Engine app. The fact that you can't reliably manage the consumer secret on the mobile device means that the default is to use the 'anonymous' access.
The Google App Engine OAuth implementation's browser authorization step takes you to a page where it contains text like:
"The site <some-site> is requesting access to your Google Account for the product(s) listed below"
YourApp(yourapp.appspot.com) - not affiliated with Google
etc
It takes <some-site> from the domain/host name used in the callback url that you supply which can be anything on the Android if you use a custom scheme to intercept the callback.
So if you use 'anonymous' access or your consumer secret is compromised, then anyone could write a consumer that fools the user into giving access to your gae app.
The Google OAuth authorization page also does contain lots of warnings which have 3 levels of severity depending on whether you're using 'anonymous', consumer secret, or public keys.
Pretty scary stuff for the average user who isn't technically savvy. I don't expect to have a high signup completion percentage with that kind of stuff in the way.
This blog post clarifies how consumer secret's don't really work with installed apps.
http://hueniverse.com/2009/02/should-twitter-discontinue-their-basic-auth-api/
Here I have answer the secure way to storing your oAuth information in mobile application
https://stackoverflow.com/a/17359809/998483
https://sites.google.com/site/greateindiaclub/mobil-apps/ios/securelystoringoauthkeysiniosapplication
Facebook doesn't implement OAuth strictly speaking (yet), but they have implemented a way for you not to embed your secret in your iPhone app: https://web.archive.org/web/20091223092924/http://wiki.developers.facebook.com/index.php/Session_Proxy
As for OAuth, yeah, the more I think about it, we are a bit stuffed. Maybe this will fix it.
None of these solutions prevent a determined hacker from sniffing packets sent from their mobile device (or emulator) to view the client secret in the http headers.
One solution could be to have a dynamic secret which is made up of a timestamp encrypted with a private 2-way encryption key & algorithm. The service then decrypts the secret and determines if the time stamp is +/- 5 minutes.
In this way, even if the secret is compromised, the hacker will only be able to use it for a maximum of 5 minutes.
I'm also trying to come up with a solution for mobile OAuth authentication, and storing secrets within the application bundle in general.
And a crazy idea just hit me: The simplest idea is to store the secret inside the binary, but obfuscated somehow, or, in other words, you store an encrypted secret. So, that means you've got to store a key to decrypt your secret, which seems to have taken us full circle. However, why not just use a key which is already in the OS, i.e. it's defined by the OS not by your application.
So, to clarify my idea is that you pick a string defined by the OS, it doesn't matter which one. Then encrypt your secret using this string as the key, and store that in your app. Then during runtime, decrypt the variable using the key, which is just an OS constant. Any hacker peeking into your binary will see an encrypted string, but no key.
Will that work?
As others have mentioned, there should be no real issue with storing the secret locally on the device.
On top of that, you can always rely on the UNIX-based security model of Android: only your application can access what you write to the file system. Just write the info to your app's default SharedPreferences object.
In order to obtain the secret, one would have to obtain root access to the Android phone.
I know this question has been asked a lot of times in varying shapes and forms, but I'm still quite unclear about a few things. It's confusing how resources about this topic around the Web refer to "user" without clear context. API keys are issued for users, and so are access tokens, but I don't think API key users, are the same as access token users.
Do you need to have different API keys for different instances of end user clients? For example if I build a mobile app for a third party API, does each instance keep their own API key? I don't think they do, but how do I tie API keys, and access tokens together to say that a certain request comes from this particular instance of an app authorized by a known user? If I were the auth provider, do I have to keep track of each of those?
API keys, and access tokens are usually represented by a pair of public, and shared keys. As a service provider (server side), which one do I use to verify the message I receive, the API key, or the access token? If I understand correctly, the idea is that each request should come with a signature derived from the secret part of the API key so that the server can check that it comes from a trusted client. Now what use do I have for access token secret? I know the access token is used to verify that a system user has authorized the app to carry operations on their behalf, but which part of the message does the access token secret be useful for?
Is a hash generated from a (secured) random number, and a time stamp salt a good API key generation strategy?
Are there (preferably open source, Java-based) frameworks that do most of these?
Let me try to answer as many of your queries as I can.
Apikey vs Access Token usage
First of all, apikeys are not used per user. Apikeys are assigned per
application (of a developer). A developer of a service signs up their application and obtains a
pair of keys.
On the other hand access tokens are issued for each
end-user in context of the usage (exception is Client Credential
grant).
Service providers can identify the application from the
apikey in use.
Service providers can identify the end-users using
access token attributes.
You should have any end-user APIs, that is an API that has end-user resources (data or context) associated, protected by 3 legged Oauth. So access token should be necessary for accessing those resources.
Developer-only resources can be protected by apikey or two-legged Oauth. Here I am referring to Oauth2 standards.
Oauth1 is preferred when there is no HTTPS is supported. This way the shared secret is not sent over unprotected channel. Instead it is used to generate a signature. I strongly suggest Oauth2 over HTTPs and avoid Oauth1 for ease of use. You and your API consumers would find Oauth2 to be much more simpler to implement and work with. Unless you have a specific reason to use Oauth v1
As a service provider you can use Apigee's Edge platform that provides Oauth 1 and 2. It is not opensource. However you can use it for free, until you need some high TPS or higher SLAs.
I have an issue with letting my secret API key be all over the world on potentially thousands of mobile devices. It could easily be compromised and used for malicious purposes by a hacker.
So what are the options for me?
I would guess a private server which has the secret API key and a web service that encapsulates all method calls. So instead of the mobile device having the secret key and does something like:
List<Friends> = service.GetFriends(secretKey);
If my secret API key is compromised and is used for spamming/abuse purposes, I must shut down the use for all my users, leaving my application dead in the sea.
So my idea is that I can use the mobile device unique device ID and do:
List<Friends> = myService.GetFriends(deviceID);
Of course, a malicious hacker could just call my web service with a fake deviceID, but at least I now have control to blacklist deviceID's. It also introduces some potential bandwidth isssue, but that is of a less concern.
A true PKI is probably out of the question, since the targetted device doesn't handle HTTP client certificates in the current version.
Any other good ideas?
You don't want to publish your API key to Facebook or Twitter, even obfuscated within a client.
Your instincts are correct to proxy through a service you control. Limiting access to that service is up to you - how much risk is unauthorized use? Device ID is a pretty good clue to device/user identity, but can be faked.
There are stronger authentication methods you could use (SMS auth, etc) to establish a long-term session or device key, however these are more complex and impose a higher burden on the end user.
TL;DR
Protect your platform API keys. Secure your own API enough to protect your needs.