i want to know if using primary id: 1;2;3;... can be used as idempotency toekn on API instead of using long string UUID.
as id is unique i think it can be
is there any risk?
I would recommend always using random idompotency tokens rather than reusing other values you have (like primary keys or sequential numbers).
It makes replaying sequences of commands during a debug session difficult if the IDs used for idempotency might overlap due to test data being reset, or other ways that you want to send the same test data in a request twice in a short period of time and not have the second request ignored.
An attacker may also be able to do a denial of service on your use of the API is they know enough to cause the web service to think that an idempotency token has already been used in a window. If you don't use a random token then you should consider the situations in which someone could disrupt your use of the service by sending their own requests with tokens that you will use (either guessed or just because they aren't generating random tokens themselves). Using either predictable or 'common' tokens in this situation will cause a problem. If the web service does not record tokens for attempts to call the web service without a valid authentication, then this is less of (or not) an issue.
Related
Our main goal for now is optimising the a processing service.
The service has a system-assigned managed identity with accespolicies that allow to get a secret.
This service makes 4 calls to a keyvault. The first one takes a lot longer than the others. I'm scratching my head, because the Managed Identity token takes 91µs to obtain. Application Insights image
I changed the way the tokens were obtained. The program only obtains it once and keeps using that same token for other round trips. I did this by making the CredentialClass AddScoped.
I assume the question is why the first call takes more time. If yes, just throwing in a couple of generic reasons which might contribute:
HTTPS handshake on the first call might take time
The client might need to create a connection pool
In case of Azure Key Vault, the first call does two round-trips AFAIK, first for auth, the second for the real payload
From a lot of articles and commercial API I saw, most people make their APIs idempotent by asking the client to provide a requestId or idempotent-key (e.g. https://www.masteringmodernpayments.com/blog/idempotent-stripe-requests) and basically store the requestId <-> response map in the storage. So if there's a request coming in which already is in this map, the application would just return the stored response.
This is all good to me but my problem is how do I handle the case where the second call coming in while the first call is still in progress?
So here is my questions
I guess the ideal behaviour would be the second call keep waiting until the first call finishes and returns the first call's response? Is this how people doing it?
if yes, how long should the second call wait for the first call to be finished?
if the second call has a wait time limit and the first call still hasn't finished, what should it tell the client? Should it just not return any responses so the client will timeout and retry again?
For wunderlist we use database constraints to make sure that no request id (which is a column in every one of our tables) is ever used twice. Since our database technology (postgres) guarantees that it would be impossible for two records to be inserted that violate this constraint, we only need to react to the potential insertion error properly. Basically, we outsource this detail to our datastore.
I would recommend, no matter how you go about this, to try not to need to coordinate in your application. If you try to know if two things are happening at once then there is a high likelihood that there would be bugs. Instead, there might be a system you already use which can make the guarantees you need.
Now, to specifically address your three questions:
For us, since we use database constraints, the database handles making things queue up and wait. This is why I personally prefer the old SQL databases - not for the SQL or relations, but because they are really good at locking and queuing. We use SQL databases as dumb disconnected tables.
This depends a lot on your system. We try to tune all of our timeouts to around 1s in each system and subsystem. We'd rather fail fast than queue up. You can measure and then look at your 99th percentile for timings and just set that as your timeout if you don't know ahead of time.
We would return a 504 http status (and appropriate response body) to the client. The reason for having a idempotent-key is so the client can retry a request - so we are never worried about timing out and letting them do just that. Again, we'd rather timeout fast and fix the problems than to let things queue up. If things queue up then even after something is fixed one has to wait a while for things to get better.
It's a bit hard to understand if the second call is from the same client with the same request token, or a different client.
Normally in the case of concurrent requests from different clients operating on the same resource, you would also want to implementing a versioning strategy alongside a request token for idempotency.
A typical version strategy in a relational database might be a version column with a trigger that auto increments the number each time a record is updated.
With this in place, all clients must specify their request token as well as the version they are updating (typical the IfMatch header is used for this and the version number is used as the value of the ETag).
On the server side, when it comes time to update the state of the resource, you first check that the version number in the database matches the supplied version in the ETag. If they do, you write the changes and the version increments. Assuming the second request was operating on the same version number as the first, it would then fail with a 412 (or 409 depending on how you interpret HTTP specifications) and the client should not retry.
If you really want to stop the second request immediately while the first request is in progress, you are going down the route of pessimistic locking, which doesn't suit REST API's that well.
In the case where you are actually talking about the client retrying with the same request token because it received a transient network error, it's almost the same case.
Both requests will be running at the same time, the second request will start because the first request still has not finished and has not recorded the request token to the database yet, but whichever one ends up finishing first will succeed and record the request token.
For the other request, it will receive a version conflict (since the first request has incremented the version) at which point it should recheck the request token database table, find it's own token in there and assume that it was a concurrent request that finished before it did and return 200.
It's seems like a lot, but if you want to cover all the weird and wonderful failure modes when your dealing with REST, idempotency and concurrency this is way to deal with it.
For a service REST API, for example, http://service_host/stores/count=30, will return 30 stores extracted from DB.
If someone put count=99999999, the service will spend quite a while to return all the stores. Shall I put a limit on the count parameter from the service side? Or shall I leave it to the client to enforce any limit they need?
Also, is it better to implement the validation of this count parameter (like valid type, positive value etc) on the service side? I tend to make the service strong and safe, but hesitate to put too much checking in it.
Is there any design convention to follow here?
As a general rule, don't rely on clients to behave nicely. Always protect yourself on the server side. Validate types, enforce limits on parameters, check for non-valid input. Otherwise clients can bring your whole system down unintentionally (not to mention malicious attacks).
You should put a limit on the parameter on the service side so that your service is reliable but at the same time you should provide the client with an option to fetch more items with separate requests. This is usually done by accepting 2 parameters - offset and count. The offset is the position of the first item to be returned and the count - the number of items, starting at offset to be returned.
Generally - don't let the client abuse your service. Implement meaningful limits in the service so that it's reliable and let the client do the heavy work (create and send multiple requests). At the same time support those multiple request and document that especially if the service is to be used by a 3rd party developer.
I need one a bit theoretical advice. Here is my situation : I have a search system, which returns a list of found items. But the user is allowed to display only particular amount of items on one page, so when his first request is sent to my WCF service, it gets the whole list, then tests if the list isn't longer then the ammount of items my user is allowed to get and if the list isn't longer, there is no problem and my service returns the whole list, but when it is, then there is problem. I need to let the user choose which page he wants to display, so I let the javascript know that the user should choose page and the "page number dialog" is shown and then user is sending the second request with page number. And based on this request the webservice selects relewant items and sends them back to user. So what I need to do is to store the whole list on the server between first and second request and I 'd appreciate any idehow to store it. I was thinking about session, but I don't know if it is possible to set timeout only to particular sesion (ex. Session["list"]), because the list is used only once and can have thousands of items, so I don't want to keep it on the server to long.
PS. I Can't use standart pagination, the scenario has to be exactly how is described above.
Thanks
This sounds like a classic use-case for memcached. It is a network based key-value store for storing temporary values. Unlike in-memory state, it can be used to share temporary cached values among servers (say you have multiple nodes), and it is a great way to save state across requests (avoiding the latency that would be caused by using cookies, which are transmitted to/from the server on each http request).
The basic approach is to create a unique ID for each request, and associate it with a particular (set of) memcached key for that user's requests. You then save this unique ID in a cookie (or similar mechanism).
A warning, though, the memory is volatile, so can be lost at any point. In practice, this is not frequent, and the memcached algorithm uses a LRU queue. More details http://code.google.com/p/memcached/wiki/NewOverview
http://memcached.org/
Memcached is an in-memory key-value store for small chunks of arbitrary data (strings, objects) from results of database calls, API calls, or page rendering.
I'm not a .net programmer, but there appear to be implementations:
http://code.google.com/p/memcached/wiki/Clients
.Net memcached client
https://sourceforge.net/projects/memcacheddotnet .Net 2.0 memcached
client
http://www.codeplex.com/EnyimMemcached Client developed in .NET 2.0
keeping performance and extensibility in mind. (Supports consistent
hashing.) http://www.codeplex.com/memcachedproviders BeIT Memcached
Client (optimized C# 2.0)
http://code.google.com/p/beitmemcached jehiah
http://jehiah.cz/projects/memcached-win32
I'm not a security expert, so I'm looking for people to poke gaping holes in an authentication scheme I've devised, or point me to a better, existing scheme that fulfills the same goals:
Overview of Problem
I have an interface in which the client maintains session lifecycle (it's an HTTP session on a web server, but it doesn't really matter).
The stateless server provides some services that require the caller to be authenticated (the server has the ability to perform this authentication).
However, it's desirable for the server not to have to authenticate the caller on each invocation, e.g., by passing credentials in each call. (The authentication process can be expensive.)
It's also desirable not to maintain session state on the server. For one thing, it's just asking for a brittle solution to have independent session timeouts on both client and server (the one on the client can't be gotten rid of), and a server timeout seems necessary in order to have a reliable session lifetime on the server (rather than relying on the client to explicitly end the session at an appropriate time). For another thing, the server isn't set up to store this sort of state.
The server has an explicit authenticate method. The problem is then: how does the server verify that, when another method is called, the caller has previously authenticated using the authenticate method, without storing any session state on the server?
Proposed Solution
Here's a scheme I've come up with:
The authenticate method accepts credentials as input parameters. Upon successful authentication, the server returns two things:
A timestamp indicating the time that authentication was performed.
An encrypted version of the tuple of { username, timestamp }, encrypted with a private key
On further method calls, the client passes both of these values back to the server. The server then decrypts the encrypted { username, timestamp } tuple. If the decrypted timestamp matches the unencrypted value that was also sent by the client, the server knows that the client has previously authenticated (as that's the only way to acquire a valid encrypted value). The decrypted username tells the server which user has been authenticated.
The validity period of an encrypted key can be enforced by only allowing timestamps that are within x hours of the current time. This isn't the same as a session timeout, but it limits the window within which a compromised timestamp could be used by a malicious party.
So
I fear that this scheme is naive in a dozen ways. What weaknesses or bad logic do you see?
In case anybody cares (which seems unlikely given the amount of attention this question has gotten!), we ended up implementing a scheme much as described above.
A few of the details vary, though:
The server creates a session token based upon the user name, the session-start timestamp (passed back to the user), and a salt.
The client does not pass this token back to the server. Instead, an MD5 hash is created from the entire request content concatenated with this token.
The MD5 hash is sent to the server along with the timestamp and the the user name (and the request). The server then re-creates the session token and performs the same hashing algorithm. If the MD5 hashes match: valid request.