PKE REST Auth using SHA-1 Hash - api

Question

I'm designing my first RESTful API and am trying to figure out how I'm going to authenticate API calls. I've worked with the Gengo API (dev docs) in the past and had great luck with it, so admittedly, am basing a lot of my auth design on their algorithm described in that link.
To sum their process up, to create a valid/authenticated API call:
Register for an account with them and generate a public/private key set. Then for each API call:
Obtain the UNIX epoch timestamp that the call is being made at.
Calculate the SHA-1 hash of your timestamp "against" your private key.
Make sure that your public key, private key and the calculated hash (above) is present as 3 separate HTTP parameters with every single API call.
At first this was a little confusing to me, but I was able to get authentication working pretty quickly with their API. But I never fully understood why I had to generate this SHA-1 hash, and I had no clue what they were doing on the server-side to actually authenticate my API calls.
Now that I'm writing my own authenticated API, I need to understand these things. So I ask:
What purpose does the timestamp and its derived SHA-1 hash serve? Why is it less secure to just require users send me their public/private keys with each API call?
Is this pubkey + privkey + hashed_timestamp method that Gengo is using a standardized practice for API auth? If so, does it have a name/algorithm? Are there other, equally-secure competitors to it?
I'm confused by the whole HMAC/SHA-1 stuff (see the link above for concrete example). I always thought SHA-1 was a one-way function that turned a string into a unqiue, encoded strinig similar to what MD5 offers. But in that example (see link), it looks like it's passing SHA-1 and the string to some HMAC algorithm. What purpose does this HMAC serve and why does it require 3 arguments (SHA-1, the timestamp and the private key)?
Finally, what do I do with the 3 parameters (pub key, priv key, hashed timestamp) on the server-side to perform authentication? If I was designing a system that only used the pub/priv keys, then I would treat them like a username/password combo and would check the database to see if that combo existed or not. But the hashed timestamp is really throwing me off here.

Answer 1

What purpose does the timestamp and its derived SHA-1 hash serve? Why is it less secure to just require users send me their public/private keys with each API call?
To clear any misunderstanding you seem to have up front, the user should never send the private key over the network. The private key is to stay private. It is a secret shared between you and the user. Reread the Gengo link, you'll see that it is only used as a parameter to the HMAC function. It is up to the user to find a way to secure it, but your API does not need it to verify calls.
The timestamp serves two purposes. First it is a piece of data for which you will get both the plaintext and the HMAC. You will be recomputing the HMAC on your side with the private key of the user. If the HMAC checks, it means that not only the timestamp was not tampered with, but also that only someone knowing the private key could have sent it. It provides integrity and authenticity for that piece of data.
If it was a simple SHA1, a attacker could have intercepted the message, changed the timestamp, and recomputed the hash. By using a keyed hash, you ensure that the sender is who you think he is.
The second purpose for the timestamp is to prevent replay attacks. Even if using a keyed hash, the attacker could have captured an old request and send it again, possibly triggering unwanted actions. If your users hash the time and you test it and reject requests that are unreasonably old, you can prevent such replay attacks.
Is this pubkey + privkey + hashed_timestamp method that Gengo is using a standardized practice for API auth? If so, does it have a name/algorithm? Are there other, equally-secure competitors to it?
Again the privkey is not sent through the pipe. Using HMAC for API authentication is quite common. It is used for Amazon Web Services for example. When used in the Gengo way, the fact that there is seemingly a public/private key pair can be confusing, it is really still symmetric cryptography, and the private key is used as a shared secret.
However I think it is better to include more than just the timestamp in the data that is HMAC'ed. Otherwise an attacker could tamper with other parts of the request. The headers, the HTTP verb, and a hash of the content of the request should be included as well.
Another scheme is to use the private key on client side to sign (encrypt with the private key) a piece of data, so the server only needs to verify it with the public key of the client and needs not know the private key of the client. Embedding a time information is still needed to prevent replays. I do not know much about this scheme, it might be hard to reliably link clients with a given public key in the first place.
What purpose does this
HMAC serve and why does it require 3 arguments (SHA-1, the timestamp
and the private key)?
An HMAC is a keyed hash. Consider the simplest form of message authentication: hash(key + message). It was found that this was not secure (see length extension attack) and a nested structure fixes the vulnerability.
HMAC is a generic name of that structure: hash(k1 + hash(k2 + message)), where k1 and k2 are derived from the actual secret key. So when we do an HMAC we need to pass the name of the actual hash algorithm that will be used (here SHA-1), the message (here, the timestamp), and the secret key.
Finally, what do I do with the 3 parameters (pub key, priv key, hashed
timestamp) on the server-side to perform authentication? If I was
designing a system that only used the pub/priv keys, then I would
treat them like a username/password combo and would check the database
to see if that combo existed or not. But the hashed timestamp is
really throwing me off here.
Hopefully clearer by now. You use the public key as an identifier to retrieve the private key. You take the ts header and recompute the HMAC of it with the private key. If it matches with the hmac header sent, the request is authentic. You check the actual timestamp to see if it's not an old request replayed by some attacker. If all checks, the call can go through. I think it's better to embed all the important information in the HMAC, not just a timestamp though.

Answer 2

You need either public key cryptography or an HMAC, not both.
Let's come back to the timestamp later, and you're confusing authentication with integrity, which we'll also come back to later.
Authentication: in your case this is where the client proves knowledge of some secret to the server. Two common ways to do this are via public key cryptography and using an HMAC.
PKC: before using the service a public/private key pair is generated. The client has the private key; the server has the public key. Important: the private key never leaves the client. In particular, the server does not have access to the private key. To authenticate, the client encrypts some random value N (called a nonce), and sends N and its encrypted form to the server. The server uses the public key to decrypt the encrypted nonce and confirms that it equals the supplied nonce. This proves to the server that the client has the private key.
HMAC: client and server agree a shared secret K beforehand. To authenticate, the client creates a nonce N, computes HMAC(K, N), and sends N and HMAC(K, N) to the server. The server also computes HMAC(K, N) since it knows the shared secret and has received N from the client. If the computed and received HMAC(K, N) values are the same then the server knows that the client has the shared secret K.
The HMAC approach has one significant weakness compared with PKC: if the server is compromised then the attacked gains knowledge of K and can then use that to masquerade as the client.
If using PKC, ideally generate the keypair on the client and send the public key to the server. That way the server never has the private key.
However, unless the communication channel is confidential (e.g. uses SSL/TLS), both approaches have a problem: replay attacks. A passive observer can record the N+encrypted form, or N+HMAC(K,N) and replay them to the server. The server will then think that the observer is a valid client.
Two standard defences are:
Use a time-based nonce.
The server remembers previously-seen nonces, and rejects new requests that use a previously-seen nonce.
That's where the timestamp comes in, and is discussed in more detail here: Should I sign (with a public/private key) a form based authentication (no SSL)?
Integrity: we've proved to the server that we're a valid client, but we haven't provided any protection of the request itself. An attacker could modify our request in flight: we'd authenticate correctly but then execute the attacker's request rather than the client's original request.
To resolve this we need to protect the integrity of the request. We can do this with the same mechanism as above. Rather than just using a nonce (N) or nonce+timestamp, include the entire request in the data been encrypted or hashed. An important consideration here is that encrypted and hashing operate on bytes, not REST requests. You therefore need a reliable way to convert the REST request (i.e. HTTP method, URL, request parameters) into bytes. This is often called "canonicalisation": the client and server must both canonicalise the request in exactly the same way, so that they are encrypting/hashing the same bytes given the request.
This whole process is standardised in things like OAuth, e.g. https://dev.twitter.com/docs/auth/authorizing-request
To answer your specific questions:
The timestamp defends against replay attacks: passive observers can't reply a client's session. The SHA-1 hash is used as a component of the HMAC.
Yes, to a point. But I'd use a fully-fledged implementation of it rather than rolling your own, such as something OAuth-based.
An HMAC is a keyed hash: it's like a standard cryptographic hash (such as SHA-1, except that you also include a shared secret key in the hash. Simply concatenating the key to the data being hashed has cryptographic weaknesses that the HMAC construct avoids. (https://en.wikipedia.org/wiki/HMAC.)
If you're using PKC then you look up the client's public key on the server (based on some client ID, which ist not the client's private key), use that to decrypt the encrypted request, and verify that that request matches the received request. If you're using HMAC then you look up the client's shared secret, canonicalise the request, compute HMAC(K, R) and verify that it matches the received HMAC(K, R). In both cases you must also verify timestamps/nonces to protected against replays.
BUT: rule #1 of crypto: don't roll your own. Use an established mechanism, such as OAuth. You probably also want to use SSL/TLS, which would then also let you use client certificates as a third authentication option. If you used those then you could also rely on SSL/TLS to give you integrity and replay protection. However, implementing SSL/TLS certificate validation correctly seems to fox many developers...