I have started to design a RESTful API and I'm thinking about how to handle authentication. I want to use some kind of authentication token but I can't use OAuth o similar infrastructures so I have to handle it myself.
One of the requirements for this API is that it must have good performance, enough to handle a high volume of requests before there is the need to scale; my concern is how to make on each request the time needed to verify the token (integrity, expiration, IP Address, etc...) as little as possibile.
I suppose the token should some kind of hash and not an encrypted string containing the user information because the decryption time would be to heavy.
I've read that I could store the tokens in an in-memory hashtable where the key is the token and the value is the user info needed to process the request, but how can I make this work in a clustered environment where there will be an hashtable on each "node"?
Should I put tokens on a DB table an hit the DB every time also Handling manually the retention of expired tickets?
Probably it's not that important for the question but I'm using Spring MVC for the RESTfull API.
Thanks in advance.
I solved my problem by using both an in-memory cache and a db cache. Here is a summary of my solution that may help anyone with the same task.
the user logs in and in that moment a unique key is generated and sent back to the user.
that login token (which is basically a GUID with some processing) is also store in a db table with additional info like exipiration and with the user's info and roles. the same pieces of information are also store in memory (google guava hashtable where the token is the key)
the token must be passed along with every api call in the authorization token as #ipa suggested
the server code checks if the token is in its memory cache the user info are already available otherwise (e.g. the api call is done on another node in the cluster) the token is search in the token db
once the token is found you can check expiration, roles, etc...
This grants a good level of performance and security, the token can be generated with any arbitrary algorithm even a relative slow one since you don't have to recalculate it on every api call. Also this works with a stateless service wich can be scaled up horizontally.
I assume you use https and therefore all the traffic is encrypted. I'd suggest one of the following principles.
Basic Authentication
You can add the credentials in the Authorization header of the request. This credentials are encoded with Base64 (see below). This credentials could be sent on every request and then checked with your DB. To get this faster and less IO intensive you can still use a cache. Once I implemented an API like this without a cache and was able to handle thousands of requests per second.
Authorization: Basic QWxhZGRpbjpvcGVuIHNlc2FtZQ==
Authorization Token
There are different ways to implement your idea with a token. A common one is that every API user has his own token usually called api key which never expires. Another one is that you first have to authorize (Basic Authentication) and then get a token back which expires. This one is then used as api key for a certain time.
Either way you have to decide whether to use a cache or not. I would keep it simple and go for basic authentication and check the db every time. Almost every framework has very good support for this approach because it's simple http. If this causes performance issues (I'd recommend performance tests anyway) try to add the table with your credentials to the JPA cache. If you want to implement something with expiring tokens have a look at Infinispan.
You can store token in Redis. If you are going to store it in DB, make sure you optimise server (if you are managing it) for read operations. I have couple of implementation where folks have used key value store as well. Hashtable is also good idea.
Related
For most client applications and UI approaches JWTs seem to be a good way to carry tokens for an OAuth based approach. This allows decoupling of a User/Auth service and the services that are actually being accessed.
Given this architecture
My question is: in the case of public APIs (ie Github or Slack) where a bearer key is generated with specific roles. How is that key authorized in a microservice architecture? Do those types of keys just require that the Auth service gets queried with every request?
Could an API gateway mitigate this? I would like to understand if a solution exists where there is minimal communication between services. Thank you!
Normally, this is solved using scopes. The scopes are permissions given to a user to do certain operations,for example there will be a scope for read a repository, another for update it, another one for delete etc..
These scopes are tied to the token and normally are requested by the user himself or added automatically depending on the user type. And the same as the authentication process, they could be included in the token itself coded as a claim in a jwt or they could be requested or checked by calling an oauth server when one operation is requested.
The advantages of include them in jwt is that there is not need to call an external server every time an operation is requested so there is a lower latency and less bandwith is required, also you remove a point of failure. Obviously if this solution is used the token must be properly signed or even encrypted to avoid possible manipulations.
However it has also drawbacks, and the most dangerous one is that the token cannot be revoked because this information cannot be included in the token and the service that check if the token is valid only can access the data contained in the token itself. Because of this, this kind of tokens are normally issued with a little expiry time so in case of the token is stolen, the validity of it will be very limited
I wrote an asp.net core 3.0 web api where I am using JWT tokens to authenticate a user. Once the user gets token, he/she can use it until it expires.
What I have done is that I have also stored this token in-memory on authentication, to get other minimal details e.g. username, token generated at and "token".
My first question is that is it a good practice? since tokens are stateless and therefore saves server side from the hassle of maintaining it.
My second question is that if it is acceptable to do so, then how do I remove this token information from in-memory once a token expires.
If I am not storing this token in memory, how to extract information like "get a list of all logged-in users".
Yes, it is a good practice to cache the JWT in memory cache like Redis or simple in-memory cache. The newly created tokens are cached in memory with cache eviction time same as token expiration time.
When a request comes in to validate token, its first checked whether it exists in memory cache, if not will be looked in to persistent storage like db.
When the user invalidates token(ie logged out), it should be removed from cache and update the state to invalidated in db.
In a distributed application, its a challenge to maintain the state. For this reason, its better to have separate caching layer backed by redis. In this way, we can maintain the application stateless.
In addition to token expiration time, you may want to add additional check for validation, depends on the content of JWT like (aud claim, signature verification etc).
To retrospect the content of JWT token , you can use tools like below
https://devtoolzone.com/decoder/jwt
Cheers,
Lakshmanan
When you say "in memory", does that mean locally on the client machine or somewhere in the server? I'm going to assume you mean client-side for their use.
I'm currently using JWT myself, so here are my recommendations:
1) Save the tokens in session storage.
2) Just empty the session (or wherever you're storing it).
3) You'll definitely need to store it somewhere if you want to access it. But getting a list of all users sounds like you want the data on the back-end. You can keep track of that on a back-end server, but usually these tokens are handled and persisted into databases. But even on a back-end server, you can just have a array of Client objects to track which ones are logged in (i.e. which ones have unexpired tokens).
The typical practice involves generating two tokens (auth token and refresh token) and then checking them against a database when the user submits a token for authentication.
My service allow any HTML documents to be converted to PDF using a POST request.
It is mostly used on the backend of my client's server and thus, the API key used for the communication is kept private.
Now, I'm thinking of a way to let my client's visitors be able to call my service on behalf of my client API key, without exposing this secure API Key.
My main issue here is security. If my client add an XHR POST requests that contains the API key, someone can take that API key and use it for their own purpose and abusing my client's account.
I could filter by domain, but this is easily spoofed so it's not possible.
I was wondering if there was a way to call a private service and be identified without risking its identity to be stolen, from the client ('s client) side?
If you're providing this sublet for authenticated users, then it's fairly trivial to give them unique keys (something that hashes their user ID or session against the API key and an initial timestamp, and checks it / logs it / looks for brutes before accessing the API). If you're doing it on the open web, without any kind of user authentication, then rate limiting gets very tricky indeed. Generally you'd want to use a combination of session hashes, IP address, operating system and browser data to create an anonymous profile that gets a temporary key on the frontend. One fairly solid way to do this is to force users through a CAPTCHA before serving a temporary key that allows them a limited number of uses of the permanent key. Any user whose ip/browser/session matches the existing attributes of a known client key is shunted to that one (and gets to skip the CAPTCHA); anyone who doesn't match an existing profile gets the CAPTCHA. That makes you a less attractive target for spoofing. On top of that, you should always rate-limit the entire thing, within a reasonable number of hits per day based on what kind of traffic you expect (or can afford), just so you don't have any surprises. This is the minimal security you'd want if your client's money is on the line every time their API key is used. It will require a simple database to store these "profiles", track usage, check for brutes and maintain the currently valid client keys. Client keys should always be expired regularly - either with a time diff against when they were created, or a regular cron process, or a maximum number of uses, etc.
One other thing I frequently do is rate-limit based on a curve. If I think 5 uses per minute is reasonable, for example, then after 5 uses in a minute from a session, each usage adds a delay of a fraction of a second * the number of uses in the last minute, squared, before the data is served.
The best answer would be to put this all behind a login system and secure that.
Assuming that you are using OAuth kind of system, In that case, make use of Access Token Mechanism that provides access to private API/User's data on behalf of User(Client) without exposing his/her credentials or API Key(Authentication key), also the access token can be expired based on the time/usage.
Example: The access token is generated against a single endpoint that can be the Html Conversion endpoint and will be expired once the action completion.
https://auth0.com/docs/tokens/access-token
And following blog post would be helpful to architect your authentication system
https://templth.wordpress.com/2015/01/05/implementing-authentication-with-tokens-for-restful-applications/
there is no good way to do front-end secure storage but my recommendation is :
is an API that used HMAC signing of requests in combination with OAuth authentication. The API key is actually a signing key. they key does not get transferred. The API key can still get found on the front-end but it becomes useless because you still need the OAuth token to send a valid request.
i know users will have to login in, but you can see this as an advantage because atleast you can log who is using the app by getting information from oauth.
please consider back-end secure storage!
You can use JWT tokens in my opinion. On the basis of username, password or any other info you can generate unique jwt tokens for different users.
Anyone can decipher these jwt tokens but not he unique security token.
If you want to add more more security to tokens, use JWE, encrypted web tokens.
More about these schemes can be found at https://medium.facilelogin.com/jwt-jws-and-jwe-for-not-so-dummies-b63310d201a3
Hashing is a decent option and should be done anyway, but for a fully secure method that wouldn't add too much complexity, you could simply abstract away from the authorization/API key by building your own API to interface with the API. This way you could both limit the kinds of things that can be done with the API key and also completely obscure the API key from the user
I don't think you should always go for user auth or JWT, it just doesn't fit all use cases. The idea of using a Captcha is interesting but also somewhat complex.
If complexity is not an issue I would rather use an infrastructure approach, I'm most familiar with AWS so I'll focus on that. Assuming you can change the host of your front end you can have your site hosted on an S3 bucket, served through a CDN, and create a proxy Lambda function that will hold the logic to call your API and store the API key as an encrypted environment variable. This Lambda you call through an API Gateway that can only be called by a specific IAM role which the S3 bucket also uses. You can also use a Cognito User Pool without authentication.
Going back to a simpler alternative the Captcha approach can be implemented as an attestation provider. I know of two services that do this, Firebase and KOR Connect. Due to Firebase using this approach only for their own resources as of the time of this writing I much rather use KOR Connect as it’s a very simple middleware that basically solves this issue. I won't go into detail about these services as it’s not the main concern of this topic but you can check the documentation their respective links.
What is the advantage of using JWTs over sessions in situations like authentication?
Is it used as a standalone approach or is it used in the session?
JWT doesn't have a benefit over using "sessions" per se. JWTs provide a means of maintaining session state on the client instead of doing it on the server.
What people often mean when asking this is "What are the benefits of using JWTs over using Server-side sessions".
With server-side sessions, you will either have to store the session identifier in a database, or else keep it in memory and make sure that the client always hits the same server. Both of these have drawbacks. In the case of the database (or other centralised storage), this becomes a bottleneck and a thing to maintain - essentially an extra query to be done with every request.
With an in-memory solution, you limit your horizontal scaling, and sessions will be affected by network issues (clients roaming between Wifi and mobile data, servers rebooting, etc).
Moving the session to the client means that you remove the dependency on a server-side session, but it imposes its own set of challenges.
Storing the token securely.
Transporting it securely.
JWT sessions can sometimes be hard to invalidate.
Trusting the client's claim.
These issues are shared by JWTs and other client-side session mechanisms alike.
JWT, in particular, addresses the last of these. It may help to understand what a JWT is:
It is a bit of information. For user sessions, you could include the username and the time when the token expires. But it could conceivably be anything, even the session ID or the user's entire profile (please don't do that though).
It has got a secure signature that prevents malicious parties from generating fake tokens (you need access to the server's private key to sign them and you can verify that they were not modified after they were signed).
You send them with every request, just like a cookie or Authorization Header would be sent. In fact, they are commonly sent in the HTTP Authorization header but using a cookie is fine too.
The token is signed and so the server can verify its origin. We will assume that the server trusts its own ability to sign securely (you should use a standard library: don't try to do it yourself, and secure the server properly).
On the issue with securely transporting the token, the answer is commonly to send it via an encrypted channel, usually httpS.
Regarding securely storing the token in the client, you need to ensure that the bad guys can't get to it. This (mostly) means preventing JS from bad web sites from reading the token to send it back to them. This is mitigated using the same strategies used to mitigate other kinds of XSS attacks.
If you have a need to invalidate JWTs, there are definitely ways this can be achieved. Storing a per-user epoch for only users who have requested to have their "other sessions terminated" is a very efficient method that will probably be good enough. If an application needs per-session invalidation, then a session ID can be maintained in the same way and the "killed tokens" table can still be maintained to be much smaller than the full user table (you only need to retain records newer than the longest allowed token lifetime). So the ability to invalidate the token partially negates the benefit of client-side sessions in that you would have to maintain this session killed state. This will more than likely be a much smaller table than the original session state table, so the lookups are still more efficient though.
One other benefit of using JWT tokens is that it is reasonably easy to implement using libraries available in probably every language you can expect to have it. It is also completely divorced from your initial user authentication scheme - if you move to a fingerprint-based system, you do not need to make any changes to the session management scheme.
A more subtle benefit: Because the JWT can carry "information" and this can be accessed by the client, you can now start doing some smart things. For example, remind the user that their session will be expiring a few days before they are logged out, giving them the option to re-authenticate, based on the expiry date in the token. Whatever you can imagine.
So in short: JWTs answers some of the questions and shortcomings of other session techniques.
"Cheaper" authentication because you can eliminate a DB round trip (or at least have a much smaller table to query!), which in turns enable horizontal scalability.
Tamper-proof client-side claims.
While JWTs does not answer the other issues like secure storage or transport, it does not introduce any new security issues.
A lot of negativity exists around JWTs, but if you implement the same security that you would for other types of authentication, you will be fine.
One final note: It is also not Cookies vs Tokens. Cookies is a mechanism for storing and transporting bits of information and can be used to store and transport JWT tokens too.
The short answer is: None.
A longer version is:
I implemented JWTs for session management after reading this recommendation in the GraphQL docs:
If you aren't familiar with any of these authentication mechanisms, we
recommend using express-jwt because it's simple without sacrificing
any future flexibility.
Implementation was indeed simple as it only added a little bit of complexity. After a while however, I (like you) started wondering what the benefits were. It turns out there are very few (or possibly none) for JWT as far as session management goes, as this blog post explains in detail:
Stop using JWT for sessions
I had a similar question choosing between JWT and token + cache for user authentication.
After reading these articles, it's clear to me the benefits JWT promises do not outpace the problems it brings. So token + cache(Redis/Memcached) is the way to go for me.
Auth Headers vs JWT vs Sessions — How to Choose the Right Auth Technique for APIs
Authentication Techniques for APIs
Stop using jwt for sessions
My two cents, which on the way add some contrast to joepie91's famous blog post.
Considering that today's (and tomorrow's) applications are (mostly) cloud native
There's an economic benefit to Stateless JWT Authentication,
which scales as the application scales:
Cloud applications incur cost with every passing second.
This cost is reduced when users no longer have to authenticate "against" a session store.
Detailed below are some factors which add to the cost of an application when not using JWT:
Database Server
Running a session store 24/7 costs money.
You can not get away with local storage / memory based solutions in the world of K8S, as pods are ephemeral.
Sticky sessions will not fare well for the exact same reason.
Storage
Storing data costs money. storing data in a SSD costs even more.
Session related operations need to be resolved quickly, so an optical drive is not an option.
I/O
Some cloud providers charge money for Disc related I/O.
Download
Circa 2022, it is safe to assume that the API and session store are separate server instances.
Some cloud providers charge for downloading information from one instance to another.
Scaling the session store
This affects all aforementioned factors.
Yet another slightly different perspective that may be useful if you are on AWS.
We had implemented PHP5.x session storage on AWS ElastiCache to centralise session storage across multiple servers.
It worked perfected until we moved to PHP7. It was difficult to configure for PHP7 and we were plagued with intermittent issues where it seemed that the session "failed/mismatched/got a bit confused" for a particular user and then they could not log in on that device until the old session expired.
We moved to using DynamoDb to store the session and no more issues. It is marginally slower but only noticeable at login (session storage) stage.
While this was going on, we implemented AWS cognito to replace our authentication and started to use the API-Gateway to deliver content via lambda python functions.
We use the PHP SDK to authenticate with Cognito and then we store the JWT in a cookie but still also use the PHP session to keep our legacy code working.
Now we have two stacks and the best of both worlds: PHP7 does it's bit and gets the main content to the user (very quickly). Then JS takes over and provides additional content using the JWT.
What I think is great about JWT is the fact that it can be passed around between these two stacks and used to authenticate the user in both cases.
Now we are wondering if it is worthwhile taking the plunge and switching over to the new JWT system entirely?
In PHP, we still use our legacy session but we also pass the token to cognito to authenticate it. It's a bit of extra security that is probably not necessary but it gives a warm cozy feeling. Then again, there are costs and maintenance with dynamoDb that could be saved.
In session authentication (or stateful authentication) you can store more data than token. But you have to store the session object somewhere and this makes the authentication centralized. Also, you need to have extra memory to store the sessions and this will give extra work to manage this memory. If your application grows, now you have to implement different designs based on your needs, implementing a session cache service, and storing the session on web application or database.
In Jwt or in general Stateless authentication, you do not store anything. You send the token with the request header. This makes it decentralized authentication. The drawback of this authentication is token revocation. Each token has an expiry time and if your token is stolen, it will be valid till it expires. You have to implement logic to minimize the risk.
I have some general/how-does-it-work-inside questions about WebAPI and OWIN (specifically, the default configuration which is set up when you create new WebAPI project in VS2013 and select Individual user account authentication). I did that, then I registered (using jQuery post) and even logged in (received token which I included in Authorization header, receiving access to protected resource. I just have some more questions about it:
Are my data stored inside authentication token? I know my password isn't, but is token containing encrypted data, or is just a random string? These are the only 2 options that I can think of: either token contains encrypted data (userId, expiration date, etc.) and server app deciphers it and grants me access to resources, or token is a random string and all user data are stored on server (token is used as a key to obtain correct user data entry). If the second theory is right, the token <-> userData lookup must be stored somewhere - is it session, cache or database maybe?
If i wanted to make a RESTful API, what about Roles, etc. (in general - data beyond simple who-are-you identification that I need for every request)? Again: first thing that comes to mind is to store them inside token. But if the data grows large isn't that too much overhead to send with each request (plus headers themselves probably are limited in size)? Second thing is using external OAuth service (like Facebook or Twitter) - if the user authenticates using external token, I can't control what information does it contain. Alternative is to get the data I need from the database each time, but isn't it bad practice? Every single request would need an extra database call to collect user's role and check if he even has access to this particular part of application. I could store it in session, but RESTful API is supposed to be stateless.
Thanks for any help as I'm just starting to dig into OAuth and WebAPI authentication. I know that I can customize everything to work as I want (so use session to store user data, etc.), but I wanted to know what the good practices are and which of them are provided out of the box with default WebAPI project in VS2013 and which need to be implemented manually.
(1) the latter is correct. The server verify token by machine key and can decipher and validate its contents.
(2) You got that correct. Its best to keep the token size min. tbh I am looking to see what others are doing about this. (+1 for the question.)