My service allow any HTML documents to be converted to PDF using a POST request.
It is mostly used on the backend of my client's server and thus, the API key used for the communication is kept private.
Now, I'm thinking of a way to let my client's visitors be able to call my service on behalf of my client API key, without exposing this secure API Key.
My main issue here is security. If my client add an XHR POST requests that contains the API key, someone can take that API key and use it for their own purpose and abusing my client's account.
I could filter by domain, but this is easily spoofed so it's not possible.
I was wondering if there was a way to call a private service and be identified without risking its identity to be stolen, from the client ('s client) side?
If you're providing this sublet for authenticated users, then it's fairly trivial to give them unique keys (something that hashes their user ID or session against the API key and an initial timestamp, and checks it / logs it / looks for brutes before accessing the API). If you're doing it on the open web, without any kind of user authentication, then rate limiting gets very tricky indeed. Generally you'd want to use a combination of session hashes, IP address, operating system and browser data to create an anonymous profile that gets a temporary key on the frontend. One fairly solid way to do this is to force users through a CAPTCHA before serving a temporary key that allows them a limited number of uses of the permanent key. Any user whose ip/browser/session matches the existing attributes of a known client key is shunted to that one (and gets to skip the CAPTCHA); anyone who doesn't match an existing profile gets the CAPTCHA. That makes you a less attractive target for spoofing. On top of that, you should always rate-limit the entire thing, within a reasonable number of hits per day based on what kind of traffic you expect (or can afford), just so you don't have any surprises. This is the minimal security you'd want if your client's money is on the line every time their API key is used. It will require a simple database to store these "profiles", track usage, check for brutes and maintain the currently valid client keys. Client keys should always be expired regularly - either with a time diff against when they were created, or a regular cron process, or a maximum number of uses, etc.
One other thing I frequently do is rate-limit based on a curve. If I think 5 uses per minute is reasonable, for example, then after 5 uses in a minute from a session, each usage adds a delay of a fraction of a second * the number of uses in the last minute, squared, before the data is served.
The best answer would be to put this all behind a login system and secure that.
Assuming that you are using OAuth kind of system, In that case, make use of Access Token Mechanism that provides access to private API/User's data on behalf of User(Client) without exposing his/her credentials or API Key(Authentication key), also the access token can be expired based on the time/usage.
Example: The access token is generated against a single endpoint that can be the Html Conversion endpoint and will be expired once the action completion.
https://auth0.com/docs/tokens/access-token
And following blog post would be helpful to architect your authentication system
https://templth.wordpress.com/2015/01/05/implementing-authentication-with-tokens-for-restful-applications/
there is no good way to do front-end secure storage but my recommendation is :
is an API that used HMAC signing of requests in combination with OAuth authentication. The API key is actually a signing key. they key does not get transferred. The API key can still get found on the front-end but it becomes useless because you still need the OAuth token to send a valid request.
i know users will have to login in, but you can see this as an advantage because atleast you can log who is using the app by getting information from oauth.
please consider back-end secure storage!
You can use JWT tokens in my opinion. On the basis of username, password or any other info you can generate unique jwt tokens for different users.
Anyone can decipher these jwt tokens but not he unique security token.
If you want to add more more security to tokens, use JWE, encrypted web tokens.
More about these schemes can be found at https://medium.facilelogin.com/jwt-jws-and-jwe-for-not-so-dummies-b63310d201a3
Hashing is a decent option and should be done anyway, but for a fully secure method that wouldn't add too much complexity, you could simply abstract away from the authorization/API key by building your own API to interface with the API. This way you could both limit the kinds of things that can be done with the API key and also completely obscure the API key from the user
I don't think you should always go for user auth or JWT, it just doesn't fit all use cases. The idea of using a Captcha is interesting but also somewhat complex.
If complexity is not an issue I would rather use an infrastructure approach, I'm most familiar with AWS so I'll focus on that. Assuming you can change the host of your front end you can have your site hosted on an S3 bucket, served through a CDN, and create a proxy Lambda function that will hold the logic to call your API and store the API key as an encrypted environment variable. This Lambda you call through an API Gateway that can only be called by a specific IAM role which the S3 bucket also uses. You can also use a Cognito User Pool without authentication.
Going back to a simpler alternative the Captcha approach can be implemented as an attestation provider. I know of two services that do this, Firebase and KOR Connect. Due to Firebase using this approach only for their own resources as of the time of this writing I much rather use KOR Connect as it’s a very simple middleware that basically solves this issue. I won't go into detail about these services as it’s not the main concern of this topic but you can check the documentation their respective links.
Related
For most client applications and UI approaches JWTs seem to be a good way to carry tokens for an OAuth based approach. This allows decoupling of a User/Auth service and the services that are actually being accessed.
Given this architecture
My question is: in the case of public APIs (ie Github or Slack) where a bearer key is generated with specific roles. How is that key authorized in a microservice architecture? Do those types of keys just require that the Auth service gets queried with every request?
Could an API gateway mitigate this? I would like to understand if a solution exists where there is minimal communication between services. Thank you!
Normally, this is solved using scopes. The scopes are permissions given to a user to do certain operations,for example there will be a scope for read a repository, another for update it, another one for delete etc..
These scopes are tied to the token and normally are requested by the user himself or added automatically depending on the user type. And the same as the authentication process, they could be included in the token itself coded as a claim in a jwt or they could be requested or checked by calling an oauth server when one operation is requested.
The advantages of include them in jwt is that there is not need to call an external server every time an operation is requested so there is a lower latency and less bandwith is required, also you remove a point of failure. Obviously if this solution is used the token must be properly signed or even encrypted to avoid possible manipulations.
However it has also drawbacks, and the most dangerous one is that the token cannot be revoked because this information cannot be included in the token and the service that check if the token is valid only can access the data contained in the token itself. Because of this, this kind of tokens are normally issued with a little expiry time so in case of the token is stolen, the validity of it will be very limited
I have been reading about securing REST APIs and have read about oAuth and JWTs. Both are really great approaches, but from what I have understood, they both work after a user is authenticated or in other words "logged in". That is based on user credentials oAuth and JWTs are generated and once the oAuth token or JWT is obtained the user can perform all actions it is authorized for.
But my question is, what about the login and sign up apis? How does one secure them? If somebody reads my javascript files to see my ajax calls, they can easily find out the end points and the parameters passed, and they could hit it multiple times through some REST Client, more severely they could code a program that hits my sign up api say a thousand times, which would be create a thousand spam users, or they could even brute force the login api. So how does one secures them?
I am writing my API in yii2.
The Yii 2.0 framework has a buil-in filter called yii\filters\RateLimiter that implements a rate limiting algorithm based on the leaky bucket algorithm. It will allow you to limit the maximum number of accepted requests in a certain interval of time. As example you may limit both login and signup endpoints to accept at most 100 API calls within a 10 minutes interval of time. When that limit is exceeded a yii\web\TooManyRequestsHttpException exception (429 status code) will be thrown.
You can read more about it in the Yii2 RESTful API related documentation or within this SO post.
I didn't use it myself so far but from what I did read about it in official docs, and I mean this:
Note that RateLimiter requires
$user
to implement the
yii\filters\RateLimitInterface.
RateLimiter will do nothing if
$user
is not set or does not implement
yii\filters\RateLimitInterface.
I guess it was designed to work with logged in users only maybe by using the user related database table, the default one introduced within the advanced template. I'm not sure about it but I know it needs to store the number of allowed requests and the related timestamp to some persistent storage within the saveAllowance method that you'll need to define in the user class. So I think you will have to track your guest users by IP addresses as #LajosArpad did suggest then maybe redesigning your user class to hold their identities so you can enable it.
A quick google search let me to this extension:yii2-ip-ratelimiter to which you may also have a look.
Your URLs will easily be determined. You should have a black list of IP addresses and when an IP address acts suspiciously, just add it to the black list. You define what suspicious is, but if you are not sure, you can start with the following:
Create something like a database table with this schema:
ip_addresses(ip, is_suspicious, login_attempts, register_attempts)
Where is_suspicious means it is blacklisted. login_attemtps and register_attempts should be json values, showing the history of that ip address trying to log in/register. If the last 20 attempts were unsuccessful and were within a minute, then the ip address should be blacklisted. Blacklisted ip addresses should receive a response that they are blacklisted whatever their request was. So if they deny your services or try to hack things, then you deny your services from them.
Secure passwords using sha1, for example. That algorithm is secure-enough and it is quicker than sha256, for instance, which might be an overkill. If your API involves bank accounts or something extremely important like that, important-enough for the bad guys to use server parks to hack it, then force the users to create very long passwords, including numbers, special characters, big and small letters.
For javascript you should use OAuth 2.0 Implicit Grant flow like Google or Facebook.
Login and Signup use 2 basic web page. Don't forget add captcha for them.
For some special client such as mobile app or webServer:
If you sure that your binary file is secure, You can create a custom login API for it. In this API you must try to verify your client.
A simple solution, you can refer:
use an encryption algorithm such as AES or 3DES to encrypt password
from client use a secret key (only client and server knows about it)
use a hash algorithm such as sha256 to hash (username + client time + an other
secret key). Client will send both client time and hash string to
server. Server will reject request if client time is too different
from server or hash string is not correct.
Eg:
api/login?user=user1&password=AES('password',$secret_key1)&time=1449570208&hash=sha256('user1'+'|'+'1449570208'+'|'+$secret_key2)
Note: In any case, server should use captcha to avoid brute force attack, Do not believe in any other filter
About captcha for REST APIs, we can create captcha base on token.
Eg.
For sign up action: you must call 2 api
/getSignupToken : to get image captcha url and a signup token
respectively.
/signup : to post sign up data (include signup token and
captcha typed by user)
For login action: we can require captcha by count failed logins base on username
Folow my api module here for reference. I manager user authentication by access token. When login, i generate token, then access again, client need send token and server will check.
Yii2 Starter Kit lite
Please someone explain me how to use an api key and what is it good for.
I have searched a lot about this and I got different and conflicting answers. One says that an API key is kept secret and its never sent as the part of the communication, while others send it to the client without any encryption. What is the client's signature? How can he generate it and what can do the server with it? Why should monkeying with api keys instead of using the good old username-password pair? Could someone explain me how the communications look between a client (Android device) and the server (php api) in detail.
I'd appreciate any good tutorials, code samples, and explanations for beginners.
The topic of API authentication is a complex one. Below I'm going to do my best to explain one part of the issue: why is an API key better than a username / password?
Here we go.
When building (or working with an API), a common question developer's ask is "Why does this service require an API key instead of my username and password?" It's a great question!
First, let's talk about what API keys typically are.
API keys are usually randomly generated strings of letters and numbers. Furthermore, an API key typically comes in two parts: an ID and a secret. If you're using a web service like Stormpath, for instance, you might have two API keys that look like this:
API_KEY_ID=kzjbOg3iOm4k4MRpBss76yxlZSPJtoOPaqxirsfX
API_KEY_SECRET=A8FnQWM7RpgGjU3sZjOUgMIq5t8mvAhQES9iE30S
You can think of an API key ID as a username. This is a globally unique identifier which allows the API service to find your account.
You can think of an API key secret as a password. This is a password that, when matched up with the correct API key ID, will grant you access to the API service in question.
The main reason you WOULDN'T want to use a username and password to authenticate against an API is that:
Even if the API is served over SSL, there are many exploits available which can compromise your credentials. If you used your username / password to log into API services, and an attacker grabs these credentials, they have access to your account as a whole.
If you use your username / password to authentication against an API, what happens if one of your servers / API clients is compromised? This means you need to reset your username / password and update it for all of the clients which are using it. This can be time consuming, and costly.
By using a username / password, you're usually restricting yourself to a certain type of API usage. By having API key pairs, you're able to separate out API credentials to different levels of access (maybe on key pair can only access certain data, while another can access other types of data).
API key pairs are, in general, a much better idea. In addition to the obvious security benefits, they also serve other purposes:
If an API key pair is leaked, you can usually create / cycle API key pairs without needing to update every single client you own.
You can use API key pairs to provide sub-account functionality for your API.
Hope that helps!
have a look at this
REST authentication and exposing the API key
Why do some API providers require an API key?
And study a lil about Oauth
I have started to design a RESTful API and I'm thinking about how to handle authentication. I want to use some kind of authentication token but I can't use OAuth o similar infrastructures so I have to handle it myself.
One of the requirements for this API is that it must have good performance, enough to handle a high volume of requests before there is the need to scale; my concern is how to make on each request the time needed to verify the token (integrity, expiration, IP Address, etc...) as little as possibile.
I suppose the token should some kind of hash and not an encrypted string containing the user information because the decryption time would be to heavy.
I've read that I could store the tokens in an in-memory hashtable where the key is the token and the value is the user info needed to process the request, but how can I make this work in a clustered environment where there will be an hashtable on each "node"?
Should I put tokens on a DB table an hit the DB every time also Handling manually the retention of expired tickets?
Probably it's not that important for the question but I'm using Spring MVC for the RESTfull API.
Thanks in advance.
I solved my problem by using both an in-memory cache and a db cache. Here is a summary of my solution that may help anyone with the same task.
the user logs in and in that moment a unique key is generated and sent back to the user.
that login token (which is basically a GUID with some processing) is also store in a db table with additional info like exipiration and with the user's info and roles. the same pieces of information are also store in memory (google guava hashtable where the token is the key)
the token must be passed along with every api call in the authorization token as #ipa suggested
the server code checks if the token is in its memory cache the user info are already available otherwise (e.g. the api call is done on another node in the cluster) the token is search in the token db
once the token is found you can check expiration, roles, etc...
This grants a good level of performance and security, the token can be generated with any arbitrary algorithm even a relative slow one since you don't have to recalculate it on every api call. Also this works with a stateless service wich can be scaled up horizontally.
I assume you use https and therefore all the traffic is encrypted. I'd suggest one of the following principles.
Basic Authentication
You can add the credentials in the Authorization header of the request. This credentials are encoded with Base64 (see below). This credentials could be sent on every request and then checked with your DB. To get this faster and less IO intensive you can still use a cache. Once I implemented an API like this without a cache and was able to handle thousands of requests per second.
Authorization: Basic QWxhZGRpbjpvcGVuIHNlc2FtZQ==
Authorization Token
There are different ways to implement your idea with a token. A common one is that every API user has his own token usually called api key which never expires. Another one is that you first have to authorize (Basic Authentication) and then get a token back which expires. This one is then used as api key for a certain time.
Either way you have to decide whether to use a cache or not. I would keep it simple and go for basic authentication and check the db every time. Almost every framework has very good support for this approach because it's simple http. If this causes performance issues (I'd recommend performance tests anyway) try to add the table with your credentials to the JPA cache. If you want to implement something with expiring tokens have a look at Infinispan.
You can store token in Redis. If you are going to store it in DB, make sure you optimise server (if you are managing it) for read operations. I have couple of implementation where folks have used key value store as well. Hashtable is also good idea.
Overview
I'm looking to create a (REST) API for my application. The initial/primary purpose will be for consumption by mobile apps (iPhone, Android, Symbian, etc). I've been looking into different mechanisms for authentication and authorization for web-based APIs (by studying other implementations). I've got my head wrapped around most of the fundamental concepts but am still looking for guidance in a few areas. The last thing I want to do is reinvent the wheel, but I'm not finding any standard solutions that fits my criteria (however my criteria my be misguided so feel free to critique that as well). Additionally, I want the API to be the same for all platforms/applications consuming it.
oAuth
I'll go ahead and throw out my objection to oAuth since I know that will likely be the first solution offered. For mobile applications (or more specifically non-web applications), it just seems wrong to leave the application (to go to a web-browser) for the authentication. Additionally, there is no way (I am aware of) for the browser to return the callback to the application (especially cross-platform). I know a couple of apps that do that, but it just feels wrong and gives a break in the application UX.
Requirements
User enters username/password into application.
Every API call is identified by the calling application.
Overhead is kept to a minimum and the auth aspect is intuitive for developers.
The mechanism is secure for both the end user (their login credentials are not exposed) as well as the developer (their application credentials are not exposed).
If possible, not require https (by no means a hard requirement).
My Current Thoughts on Implementation
An external developer will request an API account. They will receive an apikey and apisecret. Every request will require at minimum three parameters.
apikey - given to developer at regisration
timestamp - doubles as a unique identifier for each message for a given apikey
hash - a hash of the timestamp + the apisecret
The apikey is required to identify the application issuing the request. The timestamp acts similarly to the oauth_nonce and avoids/mitigates replay attacks. The hash ensures that request was actually issued from the owner of the given apikey.
For authenticated requests (ones done on the behalf of a user), I'm still undecided between going with an access_token route or a username and password hash combo. Either way, at some point a username/password combo will be required. So when it does, a hash of several pieces of information (apikey, apisecret, timestamp) + the password would be used. I'd love feedback on this aspect. FYI, they would have to hash the password first, since I don't store the passwords in my system without hashing.
Conclusion
FYI, this isn't a request for how to build/structure the API in general only how to handle the authentication and authorization from solely within an application.
Random Thoughts/Bonus Questions
For APIs that only require an apikey as part of the request, how do you prevent someone other than the apikey owner from being able to see the apikey (since sent in the clear) and make excessive requests to push them over usage limits? Maybe I'm just over thinking this, but shouldn't there be something to authenticate that a request was verified to the apikey owner? In my case, that was the purpose of the apisecret, it is never shown/transmitted without being hashed.
Speaking of hashes, what about md5 vs hmac-sha1? Does it really matter when all of the values are hashed with with sufficiently long data (ie. apisecret)?
I had been previously considering adding a per user/row salt to my users password hash. If I were to do that, how could the application be able to create a matching hash without knowing the salt used?
The way I'm thinking about doing the login part of this in my projects is:
before login the user requests a login_token from the server. These are generated and stored on the server on request, and probably have a limited lifetime.
to login the application calculates the hash of the users password, then hashes the password with the login_token to get a value, they then return both the login_token and the combined hash.
The server checks the login_token is one that it has generated, removing it from its list of valid login_tokens. The server then combines its stored hash of the user's password with the login_token and ensures that it matches the submitted combined token. If it matches you have authenticated your user.
Advantages of this are that you never store the user's password on the server, the password is never passed in the clear, the password hash is only passed in the clear on account creation (though there may be ways around this), and it should be safe from replay attacks as the login_token is removed from the DB on use.
That's a whole lot of questions in one, I guess quite a few people didn't manage to read all the way to the end :)
My experience of web service authentication is that people usually overengineer it, and the problems are only the same as you would encounter on a web page. Possible very simple options would include https for the login step, return a token, require it to be included with future requests. You could also use http basic authentication, and just pass stuff in the header. For added security, rotate/expire the tokens frequently, check the requests are coming from the same IP block (this could get messy though as mobile users move between cells), combine with API key or similar. Alternatively, do the "request key" step of oauth (someone suggested this in a previous answer already and it's a good idea) before authenticating the user, and use that as a required key to generate the access token.
An alternative which I haven't used yet but I've heard a lot about as a device-friendly alternative to oAuth is xAuth. Have a look at it and if you use it then I'd be really interested to hear what your impressions are.
For hashing, sha1 is a bit better but don't get hung up about it - whatever the devices can easily (and quickly in a performance sense) implement is probably fine.
Hope that helps, good luck :)
So what you're after is some kind of server side authentication mechanism that will handle the authentication and authorisation aspects of a mobile application?
Assuming this is the case, then I would approach it as follows (but only 'cos I'm a Java developer so a C# guy would do it differently):
The RESTful authentication and authorisation service
This will work only over HTTPS to prevent eavesdropping.
It will be based on a combination of RESTEasy, Spring Security and CAS (for single sign on across multiple applications).
It will work with both browsers and web-enabled client applications
There will be a web-based account management interface to allow users to edit their details, and admins (for particular applications) to change authorisation levels
The client side security library/application
For each supported platform (e.g.
Symbian, Android, iOS etc) create a
suitable implementation of the
security library in the native
language of the platform (e.g. Java,
ObjectiveC, C etc)
The library
should manage the HTTPS request
formation using the available APIs
for the given platform (e.g. Java
uses URLConnection etc)
Consumers of the general authentication and
authorisation library ('cos that's
all it is) will code to a specific
interface and won't be happy if it
ever changes so make sure it's very
flexible. Follow existing design
choices such as Spring Security.
So now that the view from 30,000ft is complete how do you go about doing it? Well, it's not that hard to create an authentication and authorisation system based on the listed technologies on the server side with a browser client. In combination with HTTPS, the frameworks will provide a secure process based on a shared token (usually presented as a cookie) generated by the authentication process and used whenever the user wishes to do something. This token is presented by the client to the server whenever any request takes place.
In the case of the local mobile application, it seems that you're after a solution that does the following:
Client application has a defined Access Control List (ACL) controlling runtime access to method calls. For example, a given user can read a collection from a method, but their ACL only permits access to objects that have a Q in their name so some data in the collection is quiety pulled by the security interceptor. In Java this is straightforward, you just use the Spring Security annotations on the calling code and implement a suitable ACL response process. In other languages, you're on your own and will probably need to provide boilerplate security code that calls into your security library. If the language supports AOP (Aspect Oriented Programming) then use it to the fullest for this situation.
The security library caches the complete list of authorisations into it's private memory for the current application so that it doesn't have to remain connected. Depending on the length of the login session, this could be a one-off operation that never gets repeated.
Whatever you do, don't try to invent your own security protocol, or use security by obscurity. You'll never be able to write a better algorithm for this than those that are currently available and free. Also, people trust well known algorithms. So if you say that your security library provides authorisation and authentication for local mobile applications using a combination of SSL, HTTPS, SpringSecurity and AES encrypted tokens then you'll immediately have creditibility in the marketplace.
Hope this helps, and good luck with your venture. If you would like more info, let me know - I've written quite a few web applications based on Spring Security, ACLs and the like.
Twitter addressed the external application issue in oAuth by supporting a variant they call xAuth. Unfortunately there's already a plethora of other schemes with this name so it can be confusing to sort out.
The protocol is oAuth, except it skips the request token phase and simply immediately issues an access token pair upon receipt of a username and password. (Starting at step E here.) This initial request and response must be secured - it's sending the username and password in plaintext and receiving back the access token and secret token. Once the access token pair has been configured, whether the initial token exchange was via the oAuth model or the xAuth model is irrelevant to both the client and server for the rest of the session. This has the advantage that you can leverage existing oAuth infrastructure and have very nearly the same implementation for mobile/web/desktop applications. The main disadvantage is that the application is granted access to the client's user name and password, but it appears like your requirements mandate this approach.
In any case, I'd like to agree with your intuition and that of several other answerers here: don't try to build something new from scratch. Security protocols can be easy to start but are always hard to do well, and the more convoluted they become the less likely your third-party developers are to be able to implement against them. Your hypothetical protocol is very similar to o(x)Auth - api_key/api_secret, nonce, sha1 hashing - but instead of being able to use one of the many existing libraries your developers are going to need to roll their own.
Super late to the party but I wanted to throw in some additional points to consider for anyone interested in this issue. I work for a company doing mobile API security solutions (approov) so this whole area is definitely relevant to my interests.
To start with, the most important thing to consider when trying to secure a mobile API is how much it is worth to you. The right solution for a bank is different to the right solution for someone just doing things for fun.
In the proposed solution you mention that a minimum of three parameters will be required:
apikey - given to developer at registration
timestamp - doubles as a unique identifier for each message for a given apikey
hash - a hash of the timestamp + the apisecret
The implication of this is that for some API calls no username/password is required. This can be useful for applications where you don't want to force a login (browsing in online shops for example).
This is a slightly different problem to the one of user authentication and is more like authentication or attestation of the software. There is no user, but you still want to ensure that there is no malicious access to your API. So you use your API secret to sign the traffic and identify the code accessing the API as genuine. The potential problem with this solution is that you then have to give away the secret inside every version of the app. If someone can extract the secret they can use your API, impersonating your software but doing whatever they like.
To counter that threat there are a bunch of things you can do depending on how valuable the data is. Obfuscation is a simple way to make it harder to extract the secret. There are tools that will do that for you, more so for Android, but you still have to have code that generates your hash and a sufficiently skilled individual can always just call the function that does the hashing directly.
Another way to mitigate against excessive use of an API that doesn't require a login is to throttle the traffic and potentially identify and block suspect IP addresses. The amount of effort you want to go to will largely depend upon how valuble your data is.
Beyond that you can easily start getting into the domain of my day job. Anyway, it's another aspect of securing APIs that I think is important and wanted to flag up.