I came across many APIs that give the user both an API key and a secret. But my question is: what is the difference between both?
In my eyes, one key can be enough. Say I have a key and only I and the server know it. I create a HMAC hash with this key and do an API call. On the server, we create the HMAC hash again and compare it with the sent hash. If it's the same, the call is authenticated.
So why use two keys?
Edit: or is that API key used to lookup the API secret?
You need two separate keys, one that tells them who you are, and the other one that proves you are who you say you are.
The "key" is your user ID, and the "secret" is your password. They just use the "key" and "secret" terms because that's how they've implemented it.
Simple answer, if I understood it correctly...
If you use your API key for encryption, how will the service know who is contacting them? How will they decrypt that message?
You use API key to state who you are, this is what you are sending in plain text.
The SECRET key you do not send to anyone. You simply use it for encryption. Then you send the encrypted message. You do not send the key that was used for encryption, that would defeat the purpose.
One thing that I did not see mentioned here, although it is an extension of Marcus Adams's answer, is that you should not be using a single piece of information to both identify and authenticate a user if there is a possibility of timing attacks, which can use the differences in response times to guess how far a string comparison got.
If you are using a system which uses a "key" to look up the user or credential, that piece of information could be incrementally guessed over time by sending thousands of requests and examining the time that it takes for your database to find (or not find) a record. This is especially true if the "key" is stored in plaintext instead of a one-way hash of the key. You would want to store users's keys in a plaintext or symmetrically-encrypted for if you need to be able to display the key to the user again.
By having a second piece of information, or "secret", you can first look up the user or credential using the "key", which could be vulnerable to a timing attack, then use a timing-safe compare function to check the value of the "secret".
Here is Python's implementation of that function:
https://github.com/python/cpython/blob/cd8295ff758891f21084a6a5ad3403d35dda38f7/Modules/_operator.c#L727
And it is exposed in the hmac lib (and probably others):
https://docs.python.org/3/library/hmac.html#hmac.compare_digest
One thing to note here is that I don't think that this kind of attack will work on values that are hashed or encrypted before lookup, because the values that are being compared change randomly each time a character in the input string changes. I found a good explanation of this here.
Solutions for storing API keys would then be:
Use a separate key and secret, use the key to look up the record, and use a timing-safe compare to check the secret. This allows you to show the user the key and secret to a user again.
Use a separate key and secret, use symmetrical, deterministic encryption on the secret, and do a normal comparison of encrypted secrets. This allows you to show the user the key and secret again, and could save you from having to implement a timing-safe comparison.
Use a separate key and secret, display the secret, hash and store it, then do a normal comparison of the hashed secret. This removes the necessity to use two-way encryption, and has the added benefit of keeping your secret secure if the system is compromised. It has the downside that you cannot show the secret to the user again.
Use a single key, show it to the user once, hash it, then do a normal lookup of the hashed or encrypted key. This uses a single key, but it is not able to be shown to the user again. Has the benefit of keeping keys secure if the system is compromised.
Use a single key, show it to the user once, encrypt it, and do a normal lookup of the encrypted secret. Can be shown to the user again, but at the cost of having keys vulnerable if they system is compromised.
Of these, I think that 3 is the best balance of security and convenience. I have seen this implemented on many websites when getting keys issued.
Also, I invite any actual security experts to critique this answer. I just wanted to get this out there as another discussion point.
There are answers explaining what the secret and (public) key is. It's a public-private key pair that they give confusing names to. But nobody says why the APIs require both, and many APIs only give you one secret! I've also never seen any API's docs explain why they have two keys, so the best I can do is speculate...
It's best to put only your public key in your request and sign the request locally with your private key; sending anything more shouldn't be needed. But some get away with just having the secret in the request. Ok, any good API will use some transport security like TLS (usually over HTTPS). But you're still exposing your private key to the server that way, increasing the risk of them somehow mishandling it (see: GitHub and Twitter's password logging bug recently discovered). And HTTPS is theoretically just as secure, but there are always implementation flaws out there.
But many – actually most it seems – APIs have you send both keys in requests since that's easier than making people do their own signatures; can't have pure cURL examples otherwise! In that case, it's pointless to have them separate. I guess the separate keys are just for in case they change the API later to take advantage of them. Or some have a client library that might do it the more secure way.
Related
I look out many password managers like keeper, 1password, secret-in and I am following secret-in password manager to create my own project and trying to add same features, but got stuck at storing the data of users like his/her secrets, payment secrets in encrypted form. I read encryption model of keeper here but still didn't understand. Where to store a server side encryption key?
I have some data that is symmetrically encrypted with a single key in my database. Rather than hard coding it into my code, I am looking for a safer way to store the encryption key. Where can I safely store it?
The approach here is quite simple.
You only send encrypted data to the server for storage/backup.
The encrypted data received doesn't come with a key.
You need to ensure all encryption and decryption occurs locally on the users device. Thus the user needs to supply the key.
Users aren't good at providing high quality key material, so instead, require the user to provide a password, take that password and pass it through a hash-based key derivation function with parameters that make the function slow (high ops, high mem requirements). An algorithm like pbkdf2 with a strong PRF like HMAC-SHA-2 should be sufficient.
Update:
To answer your specific questions, you need to perform the following steps, you will need to use a cryptographic library that supports key derivation from password and symmetric encryption, like libsodium.
request password from user on first use
run this password through key derivation to derive a key from it: https://libsodium.gitbook.io/doc/key_derivation
execute encryption of user data with key: https://libsodium.gitbook.io/doc/secret-key_cryptography
destroy the key and send data to server for backup
I am working on a project where two parties can each make API requests to each other and are authenticating using a shared secret. The other party wants the shared secret to be the same both ways, but I don't understand how that would work. So I'm either looking for an explanation of how to do this securely, or to find out if this is not something that is possible and that the shared secrets should be different based on the direction of the request.
If the API request were just going in one direction, I would salt the password using a key defined function, and store the hash and salt, then I can authenticate the user by generating and matching the hash on each request. But if I need to use that same shared secret to make requests to the other API, then it seems like I would need to store the password in a way that it can be decrypted, which seems wrong/not possible.
Is there a way of doing this, or should the shared secret be different depending on which direction the request is going?
Your analysis is correct. If both side can be the caller, then both sides need to know the secret (not just be able to verify it).
Having two separate keys (one for each caller, you might as well call them "passwords" in this use case) seems to be a reasonable default setup.
It is always good to think of "parties" and give each their own credentials, especially if more parties will be involved later. Makes it much easier to revoke individual access rights later on, to enforce fine-grained access control, and minimizes the impact of leaked credentials (an attacker can only impersonate that one party, not anyone else in the system).
Having just one may appear slightly easier to manage, but then you become responsible for not leaking the other guy's key (in addition to your own). I would try to avoid that extra burden.
I have been following a couple of articles regarding RESTful web services with WCF and more specifically, how to go about authentication in these. The main article I have been referencing is Aaron Skonnard's RESTful Web Services with WCF 3.5. Another one that specifically deals with HMAC authentication is Itai Goldstiens article which is based on Skonnards article.
I am confused about the "User Key" that is referenced to in both articles. I have a client application that is going to require a user to have both a user name and password.
Does this then mean that the key I use to initialise the
System.Security.Cryptography.HMACMD5 class is simply the users
password?
Given the method used to create the Mac in Itai's article
(shown below), am I right is thinking that key is the users
password and text is the string we are using confirm that the
details are in fact correct?
public static string EncodeText(byte[] key, string text, Encoding encoding)
{
HMACMD5 hmacMD5 = new HMACMD5(key);
byte[] textBytes = encoding.GetBytes(text);
byte[] encodedTextBytes =
hmacMD5.ComputeHash(textBytes);
string encodedText =
Convert.ToBase64String(encodedTextBytes);
return encodedText;
}
In my example, the text parameter would be a combination of request uri, a shared secret and timestamp (which will be available as a request header and used to prevent replay attacks).
Is this form of authentication decent? I've come across another thread here that suggests that the method defined in the articles above is "..a (sic) ugly hack." The author doesn't suggest why, but it is discouraging given that I've spent a few hours reading about this and getting it working. However, it's worth noting that the accepted answer on this question talks about a custom HMAC authorisation scheme so it is possible the ugly hack reference is simply the implementation of it rather than the use of HMAC algorithms themselves.
The diagram below if from the wikipedia article on Message Authentication Code. I feel like this should be a secure way to go, but I just want to make sure I understand it's use correctly and also make sure this isn't simply some dated mechanism that has been surpassed by something much better.
The key can be the user's password, but you absolutely should not do this.
First - the key has an optimal length equal to the size of the output hash, and a user's password will rarely be equal to that.
Second, there will never be enough randomness (entropy to use the technical term) in those bytes to be an adequate key.
Third, although you're preventing replay attacks, you're allowing anyone potentially to sign any kind of request, assuming they can also get hold of the shared secret (is that broadcast by the server at some point or is it derived only on the client and server? If broadcast, a man-in-the-middle attack can easily grab and store that - height of paranoia, yes, but I think you should think about it) unless the user changes their password.
Fourth - stop using HMACMD5 - use HMAC-SHA-256 as a minimum.
This key should at the very least be a series of bytes that are generated from the user's password - typically using something like PBKDF2 - however you should also include something transitory that is session-based and which, ideally, can't be known by an attacker.
That said, a lot of people might tell you that I'm being far too paranoid.
Personally I know I'm not an expert in authentication - it's a very delicate balancing act - so I rely on peer-reviewed and proven technologies. SSL (in this case authentication via client certificates), for example, might have it's weaknesses, but most people use it and if one of my systems gets exploited because of an SSL weakness, it's not going to be my fault. However if an exploit occurs because of some weakness that I wasn't clever enough to identify? I'd kick myself out of the front door.
Indidentally, for my rest services I now use SCRAM for authentication, using SHA512 and 512 bits of random salt for the stretching operation (many people will say that's excessive, but I won't have to change it for a while!), and then use a secure token (signed with an HMAC and encrypted with AES) derived from the authentication and other server-only-known information to persist an authenticated session. The token is stateless in the same way that Asp.Net forms authentication cookies are.
The password exchange works very well indeed, is secure even without SSL (in protecting the password) and has the added advantage of authenticating both client and server. The session persistence can be tuned based on the site and client - the token carries its own expiry and absolute expiry values within it, and these can be tuned easily. By encrypting client ID information into that token as well, it's possible to prevent duplication on to another machine by simply comparing the decrypted values from the client-supplied values. Only thing about that is watching out for IP address information, yes it can be spoofed but, primarily, you have to consider legitimate users on roaming networks.
Being unable to locate a working php/javascript implementation of blowfish, I'm now considering using SHA1 hashing to implement web-based authentication, but the lack of knowledge in this particular field makes me unsure of whether the chosen method is secure enough.
The planned roadmap:
User's password is stored on the server as an MD5 hash.
Server issues a public key (MD5 hash of current time in milliseconds)
Client javascript function takes user password as input, and calculates its MD5 hash
Client then concatenates public key and password hash from above, and calculates SHA1 of the resulting string
Client sends SHA1 hash to the server, where similar calculations are performed with public key and user's password MD5 hash
Server compares the hashes, a match indicates successful authentication.
A mismatch indicates authentication failure, and server issues a new public key, effectively expiring the one already used.
Now, the problematic part is about concatenating two keys before SHA1, could that be prone to some kind of statistical or other attacks?
Is there any specific order in which keys should be concatenated to improve the overall quality (i.e. higher bits being more important to reliability of encryption)?
Thank you in advance.
If you're only using the 'public key' (which isn't actually a public key, it's a nonce, and should really be random, unless you really want it to be usable over a certain timeframe, in which case make sure you use HMAC with a secret key to generate it so an adversary cannot predict the nonce) to prevent replay attacks, and it's a fixed size, then concatenation might not be a problem.
That said, I'm a bit concerned that you might not have a well-thought-out security model. What attack is this trying to prevent, anyway? The user's password hash is unsalted, so a break of your password database will reveal plaintext passwords easily enough anyway, and although having a time-limited nonce will mitigate replay attacks from a passive sniffer, such a passive sniffer could just steal the user's session key anyway. Speaking of which, why not just use the session key as the nonce instead of a timestamp-based system?
But really, why not just use SSL? Cryptography is really hard to get right, and people much smarter than you or I have spent decades reviewing SSL's security to get it right.
Edit: If you're worried about MITM attacks, then nothing short of SSL will save you. Period. Mallory can just replace your super-secure login form with one that sends the password in plaintext to him. Game over. And even a passive attacker can see everything going over the wire - including your session cookie. Once Eve has the session cookie, she just injects it into her browser and is already logged in. Game over.
If you say you can't use SSL, you need to take a very hard look at exactly what you're trying to protect, and what kinds of attacks you will mitigate. You're going to probably need to implement a desktop application of some sort to do the cryptography - if MITMs are going around, then you cannot trust ANY of your HTML or Javascript - Mallory can replace them at will. Of course, your desktop app will need to implement key exchange, encryption and authentication on the data stream, plus authentication of the remote host - which is exactly what SSL does. And you'll probably use pretty much the same algorithms as SSL to do it, if you do it right.
If you decide MITMs aren't in scope, but you want to protect against passive attacks, you'll probably need to implement some serious cryptography in Javascript - we're talking about a Diffie-Hellman exchange to generate a session key that is never sent across the wire (HTML5 Web storage, etc), AES in Javascript to protect the key, etc. And at this point you've basically implemented half of SSL in Javascript, only chances are there are more bugs in it - not least of which is the problem that it's quite hard to get secure random numbers in Javascript.
Basically, you have the choice between:
Not implementing any real cryptographic security (apparently not a choice, since you're implementing all these complex authentication protocols)
Implementing something that looks an awful lot like SSL, only probably not as good
Using SSL.
In short - if security matters, use SSL. If you don't have SSL, get it installed. Every platform that I know of that can run JS can also handle SSL, so there's really no excuse.
bdonlan is absolutely correct. As pointed out, an adversary only needs to replace your Javascript form with evil code, which will be trivial over HTTP. Then it's game over.
I would also suggest looking at moving your passwords to SHA-2 with salts, generated using a suitable cryptographic random number generator (i.e. NOT seeded using the server's clock). Also, perform the hash multiple times. See http://www.jasypt.org/howtoencryptuserpasswords.html sections 2 and 3.
MD5 is broken. Do not use MD5.
Your secure scheme needs to be similar to the following:
Everything happens on SSL. The authentication form, the server-side script that verifies the form, the images, etc. Nothing fancy needs to be done here, because SSL does all the hard work for you. Just a simple HTML form that submits the username/password in "plaintext" is all that is really needed, since SSL will encrypt everything.
User creates new password: you generate a random salt (NOT based off the server time, but from good crypto random source). Hash the salt + the new password many times, and store the salt & resulting hash in your database.
Verify password: your script looks up salt for the user, and hashes the salt + entered password many times. Check for match in database.
The only thing that should be stored in your database is the salt and the hash/digest.
Assuming you have a database of MD5 hashes that you need to support, then the solution might be to add database columns for new SHA-2 hashes & salts. When the user logs in, you check against the MD5 hash as you have been doing. If it works, then follow the steps in "user creates new password" to convert it to SHA-2 & salt, and then delete the old MD5 hash. User won't know what happened.
Anything that really deviates from this is probably going to have some security flaws.
I have been working with APIs and I've always wondered why you have to use a key and a secret?
Why do you need two types of authentication?
When a server receives an API call, it needs to know two things: Who is making the call, and whether or not the call is legitimate.
If you just had one item ("key"), and included it with every call, it would answer both questions. Based on the "key" the server knows who you are, and because only you know the key it proves that the call is actually coming from you. But including the key with every call is bad security practice: if someone can read even one of your messages in transit, your key is compromised, and someone can pretend to be you. So unless you're using HTTPS, this approach doesn't work.
Instead, you can include a digital signature with every call, signed with some "secret" number. (The "secret" number itself is not sent). If an attacker manages to read your message, they won't be able to figure out this "secret" number from the signature. (This is how digital signatures work: they are one-way).
But this doesn't solve the identification question: In the latter case, how does the server know who is making the call? It could try to verify the signature against the "secret" of every single user, but of course this would be very time-consuming.
So, here's what we do: Send both a "key" (that identifies the user), and a signature created using the "secret" number (that proves that the message is legitimate). The server looks up the user based on the key, and then validates the signature using that user's "secret" number.
This is a bit like when you write a check: It has an account number on it (to identify you) and your signature (to prove that you're you). Having just the account number wouldn't prove that you actually wrote the check. Having just the signature without the account number would force the bank to compare your check against all of its signatures for all of its accounts, which would obviously be inefficient.