There are primarily two ways to authenticate using Google's reCAPTCHA Enterprise in a non-Google cloud environment (we use AWS). Google recommends using Service Accounts along with their Java client. However, this approach strikes me as less preferable than the second way Google suggests we can can authenticate, namely, using Api Keys. Authenticating via an Api Key is easy, and it’s similar to how we commonly integrate with other 3rd party services, except rather than a username and password that we must secure, we have an Api Key that we must secure. Authenticating using Service Accounts, however, requires that we store a Json file on each environment in which we run reCAPTCHA, create an environment variable (GOOGLE_APPLICATION_CREDENTIALS) to point to that file, and then use Google's provided Java client (which leverages the environment variable/Json file) to request reCAPTCHA resources.
If we opt to leverage Api Key authentication, we can use Google’s REST Api along with our preferred Http Client (Akka-Http). We merely include the Api Key in a header that is encrypted as part of TLS in transit. However, if we opt for the Service Accounts method, we must use Google’s Java client, which not only adds a new dependency to our service, but also requires its own execution context because it uses blocking i/o.
My view is that unless there is something that I’m overlooking--something that the Service Accounts approach provides that an encrypted Api Key does not--then we should just encrypt an Api Key per environment.
Our Site Key is locked down by our domain, and our Api Key would be encrypted in our source code. Obviously, if someone were to gain access to our unencrypted Api Key, they could perform assessments using our account, but not only would it be exceedingly difficult for an attacker to retrieve our Api Key, the scenario simply doesn't strike me as likely. Performing free reCAPTCHA assessments does not strike me as among the things that a sophisticated attacker would be inclined to do. Am I missing something? I suppose my question is why would we go through the trouble of creating a service account, using the (inferior) Java client, storing a Json file on each pod, create an environment variable, etc. What does that provide us that the Api Key option does not? Does it open up some functionality that I'm overlooking?
I've successfully used Api Keys and it seems to work fine. I have not yet attempted to use Service Accounts as it requires a number of things that I'm disinclined to do. But my worry is that I'm neglecting some security vulnerability of the Api Keys.
After poring a bit more over the documentation, it would seem that there are only two reasons why you'd want to explicitly choose API key-based authentication over an Oauth flow with dedicated service accounts:
Your server's execution environment does not support Oauth flows
You're migrating from reCAPTCHA (non-enterprise) to reCAPTCHA Enterprise and want to ease migration
Beyond that, it seems the choice really comes down to considerations like your organization's security posture and approved authentication patterns. The choice does also materially affect things like how the credentials themselves are provisioned & managed, so if your org happens to already have a robust set of policies in place for the creation and maintenance of service accounts, it'd probably behoove you to go that route.
Google does mention sparingly in the docs that their preferred method for authentication for reCAPTCHA Enterprise is via service accounts, but they also don't give a concrete rationale anywhere I saw.
Related
Consider an API that a client accesses directly (machine to machine) and that doesn't require user-specific authentication. The way I understand it, in client_credentials, the client must store a client_id and client_secret that it uses to acquire and refresh tokens. With an API key, the client just stores the key. What makes OAuth more secure in this case? It would appear to me that if the API key is never compromised, no attacker could pose as the intended client. And if the API key is compromised, it is effectively the same as compromising the client_id and client_secret, which an attacker would be able to use to obtain tokens and access the data in the API, posing as the client.
edit: clarified this is a machine-to-machine call
TLDR;
The difference comes down to direct access vs. delegated access.
OAuth allows you to make delegated access. The benefits of delegated access don't change if there is a user involved or not. The same arguments that make the OAuth Authorization code flow attractive for user-to-machine access, apply to the OAuth Client credentials flow for machine-to-machine access.
Ask yourself, do you want the resource server to handle client credentials or not?
On confidential clients for machine-to-machine access, the cost of delegated access vs. direct access may very well outweigh the benefits. That's why so many APIs still use API keys. You'll have to decide that for your individual use case.
Differences
In the OAuth client credentials flow, the client sends an access token to the resource server, which it got beforehand by the authorization server after presenting its client ID and secret. The resource server never sees the client secret. With an API key, the client sends the key with every request.
OAuth adds an additional layer of indirection with the authorization server, such that the credentials themselves never get transmitted to the resource server. This allows the authorization server to give the client only access for a limited amount of time or with limited permissions, without ever needing to change the actual client credentials. It also allows to revoke access tokens without revoking the credentials themselves. For multiple instances of a client this allows you to revoke access for some but not all.
Of course this all comes at the cost of a more complex implementation, and an additional roundtrip from the client to the authorization server.
I won't touch on transmission (URL, header, body, etc.) or format (random string, signed JWT, etc.), since these can be the same for access tokens just as for API keys.
Another, maybe not so obvious, advantage of OAuth is having a clear spec that libraries, documentation and discussions can be based on. With direct access there is no single best practice and different people may understand different things when referring to direct access methods like API keys.
With client credential flow your Client Id and Client Secret are sent to the authorization server to get back an access token. For all subsequent request to the API/resource servers, you pass the access token and not the client credentials themselves. The access token is usually a JWT, which is a set of encoded claims including the token expiry (exp), not before (nbf), token issuer (iss), authorized party (azp), roles, permissions, etc.
This has a number of advantages over a simple API Key approach. e.g.
If the access token (which is included in requests to the API/resource server) is compromised, it's only valid until it expires (which is typically ~1 day for M2M tokens). If an API Key is compromised, it can be used indefinitely or until it's explicitly blocked by the API/resource server.
JWT access tokens are encoded JSON objects that contains a number of fields (a.k.a. claims) that can be used for fine grained authorization e.g. roles, permissions, grant type, authorized party etc. An API Key is generally opaque and is all or nothing when it comes to auth.
You machine tokens can get validated and authorized on the API/resource servers the same way as your user tokens, so you don't end up with multiple auth implementations on the back-end.
OAuth Client Credentials Flow
What is the security difference between API Keys and the client credentials flow of OAuth?
OAuth client credentials flow is not meant to be used by public clients, just between machines.
From auth0.com/docs:
Client Credentials Flow
With machine-to-machine (M2M) applications, such as CLIs, daemons, or services running on your back-end, the system authenticates and authorizes the app rather than a user. For this scenario, typical authentication schemes like username + password or social logins don't make sense. Instead, M2M apps use the Client Credentials Flow (defined in OAuth 2.0 RFC 6749, section 4.4), in which they pass along their Client ID and Client Secret to authenticate themselves and get a token.
So, I am not sure what is your scenario, but I will assume in my reply that you are referring to public clients.
If it is in the public client code, then it is public
The way I understand it, in client_credentials, the client must store a client_id and client_secret that it uses to acquire and refresh tokens.
Yes, it needs to be stored in the client code for the client to be able to obtain the OAuth token.
If you use the client_secret from a web app or mobile app you are making it public, therefore not a secret anymore.
Extracting secrets from public clients
For example, in a web app all it takes to extract the client_secret is to hit F12 in the browser and search for it, thus how much time can this take?
Now, in a mobile app, some may think it's secure because they are compiled into a binary but is almost as easy as it is in the browser, because we have several open-source tools that can help us with this task, like the MobSF framework, and on Linux, you can even achieve this with the strings command. Using the MobSF to perform static binary analysis on the mobile app binary allows for anyone without hacking knowledge to easily extract the client_secret in minutes, just like I show in my article How to Extract an API key from a Mobile App with Static Binary Analysis:
The range of open source tools available for reverse engineering is huge, and we really can't scratch the surface of this topic in this article, but instead, we will focus in using the Mobile Security Framework(MobSF) to demonstrate how to reverse engineer the APK of our mobile app. MobSF is a collection of open-source tools that present their results in an attractive dashboard, but the same tools used under the hood within MobSF and elsewhere can be used individually to achieve the same results.
So, the process of extracting the api-key in my article is the same you will use to extract the client_secret or any other string of your interest in the mobile app binary.
OAuth or API Key?
What makes OAuth more secure in this case? It would appear to me that if the API key is never compromised, no attacker could pose as the intended client. And if the API key is compromised, it is effectively the same as compromising the client_id and client_secret, which an attacker would be able to use to obtain tokens and access the data in the API, posing as the client.
If used from a public client neither are secure, because if read my linked article, you understand by now how easy is to bypass an API Key or extract the client_secret and client_id.
So, if your client is public you should not use the OAuth client credential flow, thus you need to go with the insecure API key approach or you can be more diligent and try to apply defence-in-depth approaches, but this will depend if the API clients are only web apps or mobile apps or both.
If your API clients are only web apps I invite you to read my answer to the question Secure API data from calls out of the app, especially the section dedicated to Defending the API Server.
In the case the API clients are only mobile apps then I recommend you to read this answer I gave to the question How to secure an API REST for mobile app?, especially the sections Securing the API Server and A Possible Better Solution.
On the other hand, if your API clients are both a web app and a mobile app I recommend you to apply the security measures more relevant to you from both answers linked above.
Remember that security is always about adding as many layers of defences as you can afford or it's required by law. Even in the past century, the castles were built with a lot of different security defence layers, thus this is nothing new to the digital era.
Do You Want To Go The Extra Mile?
In any response to a security question I always like to reference the excellent work from the OWASP foundation.
For APIS
OWASP API Security Top 10
The OWASP API Security Project seeks to provide value to software developers and security assessors by underscoring the potential risks in insecure APIs, and illustrating how these risks may be mitigated. In order to facilitate this goal, the OWASP API Security Project will create and maintain a Top 10 API Security Risks document, as well as a documentation portal for best practices when creating or assessing APIs.
For Mobile Apps
OWASP Mobile Security Project - Top 10 risks
The OWASP Mobile Security Project is a centralized resource intended to give developers and security teams the resources they need to build and maintain secure mobile applications. Through the project, our goal is to classify mobile security risks and provide developmental controls to reduce their impact or likelihood of exploitation.
OWASP - Mobile Security Testing Guide:
The Mobile Security Testing Guide (MSTG) is a comprehensive manual for mobile app security development, testing and reverse engineering.
For Web Apps
The Web Security Testing Guide:
The OWASP Web Security Testing Guide includes a "best practice" penetration testing framework which users can implement in their own organizations and a "low level" penetration testing guide that describes techniques for testing most common web application and web service security issues.
I am using HERE api in both frontend and backend. If I try to put my app_id and app_code into the frontend code, it will be available to anyone seeing my site.
I can try to create a domain whitelist and put my domain in this. But still, if I set the HTTP header "Referer" to my domain, I am able to access the API from any IP.
So, what do I do?
The Difference Between WHO and WHAT is Accessing the API Server
Before I dive into your problem I would like to first clear a misconception about WHO and WHAT is accessing an API server.
To better understand the differences between the WHO and the WHAT are accessing an API server, let’s use this picture:
So replace the mobile app by web app, and keep following my analogy around this picture.
The Intended Communication Channel represents the web app being used as you expected, by a legit user without any malicious intentions, communicating with the API server from the browser, not using Postman or using any other tool to perform a man in the middle(MitM) attack.
The actual channel may represent several different scenarios, like a legit user with malicious intentions that may be using Curl or a tool like Postman to perform the requests, a hacker using a MitM attack tool, like MitmProxy, to understand how the communication between the web app and the API server is being done in order to be able to replay the requests or even automate attacks against the API server. Many other scenarios are possible, but we will not enumerate each one here.
I hope that by now you may already have a clue why the WHO and the WHAT are not the same, but if not it will become clear in a moment.
The WHO is the user of the web app that we can authenticate, authorize and identify in several ways, like using OpenID Connect or OAUTH2 flows.
OAUTH
Generally, OAuth provides to clients a "secure delegated access" to server resources on behalf of a resource owner. It specifies a process for resource owners to authorize third-party access to their server resources without sharing their credentials. Designed specifically to work with Hypertext Transfer Protocol (HTTP), OAuth essentially allows access tokens to be issued to third-party clients by an authorization server, with the approval of the resource owner. The third party then uses the access token to access the protected resources hosted by the resource server.
OpenID Connect
OpenID Connect 1.0 is a simple identity layer on top of the OAuth 2.0 protocol. It allows Clients to verify the identity of the End-User based on the authentication performed by an Authorization Server, as well as to obtain basic profile information about the End-User in an interoperable and REST-like manner.
While user authentication may let the API server know WHO is using the API, it cannot guarantee that the requests have originated from WHAT you expect, the browser were your web app should be running from, with a real user.
Now we need a way to identify WHAT is calling the API server, and here things become more tricky than most developers may think. The WHAT is the thing making the request to the API server. Is it really a genuine instance of the web app, or is a bot, an automated script or an attacker manually poking around with the API server, using a tool like Postman?
For your surprise, you may end up discovering that It can be one of the legit users manipulating manually the requests or an automated script that is trying to gamify and take advantage of the service provided by the web app.
Well, to identify the WHAT, developers tend to resort to an API key that usually is sent in the headers of the web app. Some developers go the extra mile and compute the key at run-time in the web app, inside obfuscated javascript, thus it becomes a runtime secret, that can be reverse engineered by deobusfaction tools, and by inspecting the traffic between the web app and API server with the F12 or MitM tools.
The above write-up was extracted from an article I wrote, entitled WHY DOES YOUR MOBILE APP NEED AN API KEY?. While in the context of a Mobile App, the overall idea is still valid in the context of a web app. You wish you can read the article in full here, that is the first article in a series of articles about API keys.
Your Problem
I can try to create a domain whitelist and put my domain in this. But still, if I set the HTTP header "Referer" to my domain, I am able to access the API from any IP.
So this seems to be related with using the HERE admin interface, and I cannot help you here...
So, what do I do?
I am using HERE API in both frontend and backend.
The frontend MUST always delegate access to third part APIs into a backend that is under the control of the owner of the frontend, this way you don't expose access credentials to access this third part services in your frontend.
So the difference is that now is under your direct control how you will protect against abuse of HERE API access, because you are no longer exposing to the public the HERE api_id and api_code, and access to it must be processed through your backend, where your access secrets are hidden from public pry eyes, and where you can easily monitor and throttle usage, before your bill skyrockets in the HERE API.
If I try to put my app_id and app_code into the frontend code, it will be available to anyone seeing my site.
So to recap, the only credentials you SHOULD expose in your frontend is the ones to access your backend, the usual api-key and Authorization tokens, or whatsoever you want to name them, not the api_id or api_code to access the HERE API. This approach leaves you only with one access to protect, instead of multiple ones.
Defending an API Server
As I already said, but want to reinforce a web app should only communicate with an API server that is under your control and any access to third part APIs services must be done by this same API server you control. This way you limit the attack surface to only one place, where you will employ as many layers of defence as what you are protecting is worth.
For an API serving a web app, you can employ several layers of dense, starting with reCaptcha V3, followed by Web Application Firewall(WAF) and finally if you can afford it a User Behavior Analytics(UBA) solution.
Google reCAPTCHA V3:
reCAPTCHA is a free service that protects your website from spam and abuse. reCAPTCHA uses an advanced risk analysis engine and adaptive challenges to keep automated software from engaging in abusive activities on your site. It does this while letting your valid users pass through with ease.
...helps you detect abusive traffic on your website without any user friction. It returns a score based on the interactions with your website and provides you more flexibility to take appropriate actions.
WAF - Web Application Firewall:
A web application firewall (or WAF) filters, monitors, and blocks HTTP traffic to and from a web application. A WAF is differentiated from a regular firewall in that a WAF is able to filter the content of specific web applications while regular firewalls serve as a safety gate between servers. By inspecting HTTP traffic, it can prevent attacks stemming from web application security flaws, such as SQL injection, cross-site scripting (XSS), file inclusion, and security misconfigurations.
UBA - User Behavior Analytics:
User behavior analytics (UBA) as defined by Gartner is a cybersecurity process about the detection of insider threats, targeted attacks, and financial fraud. UBA solutions look at patterns of human behavior, and then apply algorithms and statistical analysis to detect meaningful anomalies from those patterns—anomalies that indicate potential threats. Instead of tracking devices or security events, UBA tracks a system's users. Big data platforms like Apache Hadoop are increasing UBA functionality by allowing them to analyze petabytes worth of data to detect insider threats and advanced persistent threats.
All these solutions work based on a negative identification model, by other words they try their best to differentiate the bad from the good by identifying what is bad, not what is good, thus they are prone to false positives, despite the advanced technology used by some of them, like machine learning and artificial intelligence.
So you may find yourself more often than not in having to relax how you block the access to the API server in order to not affect the good users. This also means that these solutions require constant monitoring to validate that the false positives are not blocking your legit users and that at the same time they are properly keeping at bay the unauthorized ones.
Summary
Anything that runs on the client side and needs some secret to access an API can be abused in different ways and you must delegate the access to all third part APIs to a backend under your control, so that you reduce the attack surface, and at the same time protect their secrets from public pry eyes.
In the end, the solution to use in order to protect your API server must be chosen in accordance with the value of what you are trying to protect and the legal requirements for that type of data, like the GDPR regulations in Europe.
So using API keys may sound like locking the door of your home and leave the key under the mat, but not using them is liking leaving your car parked with the door closed, but the key in the ignition.
Going the Extra Mile
OWASP Web Top 10 Risks
The OWASP Top 10 is a powerful awareness document for web application security. It represents a broad consensus about the most critical security risks to web applications. Project members include a variety of security experts from around the world who have shared their expertise to produce this list.
We have a package that we share with out customers. In the package, we have a chunk of code that does HTTP Request callouts to our central API Gateway. As of now, our API Gateway is open and accepts requests from everywhere, which is not good. I want to limit access to our users who would be using our software. The only solution I have found is using IAM and providing authorization that would require us to include our Access Keys in the package. Our users can install our package in any environment they want and we have no control over that environment. So I think a viable option is to create a generic user policy with minimal access to allow our users to call our API Gateway. However, putting access key in the code doesn't seem like a good idea. Another option is to provider our customers with access keys but that also has overhead. What is a better alternative that is more secure and easy to maintain?
You can use built-in API Gateway API Key functionality when IAM policies aren't possible.
So long as your clients could be on any infrastructure, versus limited to AWS, the API Gateway service provides a generic API key solution, which allows you to restrict client traffic to your API Gateway by enforcing that client requests include API keys. This API key interface is part of their "API Usage Plan" feature.
This document explains how to use the console to set up an API Gateway to enforce that client traffic bears an API key:
To set up API keys, do the following:
Configure API methods to require an API key.
Create or import an API key for the API in a region.
Your clients can implement a "secret storage" solution, in order to avoid putting their API keys into their source code.
For sure it isn't wise for your clients to store their API Keys plain-text inside their source code. Instead, they could use a secret storage solution, to store the API keys outside of their codebase, but still give their applications access to the secret.
This article describes an example solution for secure secret storage (e.g. secure API key storage) which grants an application access to the application secret without putting the unencrypted secret into the source code. It uses Amazon KMS + Cryptex, but the same principle can be applied with other technologies: http://technologyadvice.github.io/lock-up-your-customer-accounts-give-away-the-key/
A typical recommendation for securing a REST API is to use HTTP Basic Authentication over SSL. My question is, should HTTP Basic Authentication only be used to authenticate the client (ie. the app accessing the API), or can it also be used to authenticate the user (the consumer of the app)?
It seems most APIs have to deal with both, as almost all web services employ some sort of user accounts. Just consider Twitter or Vimeo—there are public resources, and there are private (user specific) resources.
It seems logical that a simple REST API could do both client and user authentication at the same time using using HTTP Basic Authentication (over SSL).
Is this a good design?
By authenticate the client you probably mean the usage of API Key, this mechanism is used to track the concrete application/client. The second thing is that it gives you the possibility to disable the application by disabling the key, for example when client's author removes his account from the service. If you want to make your API public then it is a good idea.
But you need to remember that it gives you no real protection, everybody can download the client and extract that key.
I would not recommend to use Basic Authentication for API authentication. When it comes to authentication then you should consider that the application (client) developer has to implement its side of the authentication, too. Part of that is not only authentication itself but also how to get credentials and even much more than that.
I recommend to make use of an established authentication standard that ships with client libraries for the most popular programming languages. Those libraries make it much more likely that developers are going to adapt your API, because they reduce implementation effort on the client side.
Another important reason for using authentication standards is that they make developers (and others) more confident in the security of your authentication system. Those standards have been audited by experts and their weaknesses and strengths are well known and documented. It is unlikely that you are going to develop a nearly as solid authentication flow unless you are a security expert :-).
The most established standard in this field is OAuth but you can find alternatives by searching for "oauth alternatives".
How does OAuth help you with your problem setting?
In OAuth 2, the application client has to obtain an access token for a user before accessing any protected resource. To get an access token, the application must authenticate itself with its application credentials. Depending on the use-case (e.g. 3rd party, mobile) this is done in different ways that are defined by the OAuth standard.
An access token should not only represent a user but also which operations may be used on what resources (permissions). A user may grant different permissions to different applications so this information must somehow be linked to the token.
How to achieve such a semantic for access tokens however is not part of OAuth - it just defines the flow of how to obtain access tokens. Therefor, the implementation of the access token semantic is usually application specific.
You can implement such token semantic by storing a link between an access tokens and its permissions in your backend when you create the access token. The permissions may either be stored for every user-application combination or just for every application, depending on how fine-granular you want things to be.
Then, each time that an access token is processed by the API, you fetch this information and check whether the user has sufficient permissions to access the resource and to perform the desired operation.
Another option is to put the permission information into the access token and to sign or encrypt the token. When you receive the access token, you verify or decrypt it and use the permissions that are stored in the access token to make your decision. You may want to have a look on Json Web Tokens (JWT) on how to accomplish that.
The benefit of the later solution is better scalability and less effort during backend implementation. The downside of it are potentially larger requests (especially with RSA encryption) and less control over tokens.
I am developing an API that will be used by users of my customers. Here is what the flow will look like:
User of my cloud based service creates an API key.
User embeds the API key into their own custom applications.
User deploys the application to their own end users.
The application talks to our API.
I am looking for advice on how to secure this API. I see a few issues:
API key has to be embedded into the users application and is therefore vulnerable to being stolen and abused.
Once an API key is compromised, it can easily be disabled, but how will my users update their applications to use a new API key short of having to rebuild the application and redeploy.
Does anyone have any ideas on how to design this?
I may be mistaken but maybe you could have your customers' users talk to your customers' APIs instead. Basically, your customers would keep their secret key on their servers, and not embed them in the clients they give their users, so it couldn't be hijecked (unless their server was compromised of course). Then the users would talk to your API through your customers' APIs.
It would be slower and need more work on the part of your customers, but also safer.
Two solutions that I can see to this, although I'm sure there are more..
Use oauth's RSA signature method, and implement a secure certification exchange of keys using your "cloud based service" as the exchange mechanism (or a public cert provider).
Implement a service that allows clients to "renew" their consumer key/secret automatically, but then secure that mechanism using RSA or some other public key encryption method.
Both of these are not easy, and would require your user's applications to "phone home" in order to update their consumer keys.
In the future I think OAuth 2 will provide at least protocol definitions for things like this, but for now, if you're using OAuth 1.0a, what you want to do doesn't really fit into the spec very well (i.e. you have to design much of it yourself.)