I’m developing a small game where the player owns droids used to perform some automated actions. The easiest example is giving an order to a droid to send him at a specific position. Basically, the users gives it a position and the droid goes there. I’m already using a lot Azure app function and I’d like to use them to make the droid moves.
On the top of my head, I thought about making one function that would trigger every minute, fetch all the droid that need to move then make them move.
The issue with this approach is that if the game is popular, there could be hundreds of droids and I have to ensure that the function execution time stays below the minute.
I thought about just retrieving all droids that needs to move then for each of them calling a Azure app function via its URL to make it execute for this particular droid. In my head, it would parallelize the execution a bit but I’m not sure I’m correct.
I also have to think about using sql transaction or not in order to be sure not to create deadlocks.
The final question would be « how to handle recurring treatment of potentially large amount of data and ensure that it stays below the minute ? »
Thanks for your advice
Typically, you handle such scenarios with queues. Each order becomes a queue message, and then Azure Function is triggered by it and processes the order. It can and will scale based on the amount of messages in the queue.
If your logic still requires timer-based processing, the timer should be as lean as possible, e.g. send the queue messages to a queue which would do the real work.
Related
I'm struggling to understand how to implement Eventual Consistency with the exposed example of BacklogItems and Tasks from Vaughn Vernon. The statement I've understood so far is (considering the case where he splits BacklogItem and Task into separate aggregate roots):
A BacklogItem can contain one or more tasks. When all remaining hours from a the tasks of a BacklogItem are 0, the status of the BacklogItem should change to "DONE"
I'm aware about the rule that says that you should not update two aggregate roots in the same transaction, and that you should accomplish that with eventual consistency.
Once a Domain Service updates the amount of hours of a Task, a TaskRemainingHoursUpdated event should be published to a DomainEventPublisher which lives in the same thread as the executing code. And here it is where I'm at a loss with the following questions:
I suppose that there should be a subscriber (also living in the same thread I guess) that should react to TaskRemainingHoursUpdated events. At which point in your Desktop/Web application you perform this subscription to the Bus? At the very initialization of your app? In the application code? Is there any reasoning to place domain subscriptors in a specific place?
Should that subscriptor (in the same thread) call a BacklogItem repository and perform the update? (But that would be a violation of the rule of not updating two aggregates in the same transaction since this would happen synchronously, right?).
If you want to achieve eventual consistency to fulfil the previously mentioned rule, do I really need a Message Broker like RabbitMQ even though both BacklogItem and Task live inside the same Bounded Context?
If I use this message broker, should I have a background thread or something that just consumes events from a RabbitMQ queue and then dispatches the event to update the product?
I'd appreciate if someone can shed some clear light over this since it is quite complex to picture in its completeness.
So to start with, you need to recognize that, if the BacklogItem is the authority for whether or not it is "Done", then it needs to have all of the information to compute that for itself.
So somewhere within the BacklogItem is data that is tracking which Tasks it knows about, and the known state of those tasks. In other words, the BacklogItem has a stale copy of information about the task.
That's the "eventually consistent" bit; we're trying to arrange the system so that the cached copy of the data in the BacklogItem boundary includes the new changes to the task state.
That in turn means we need to send a command to the BacklogItem advising it of the changes to the task.
From the point of view of the backlog item, we don't really care where the command comes from. We could, for example, make it a manual process "After you complete the task, click this button here to inform the backlog item".
But for the sanity of our users, we're more likely to arrange an event handler to be running: when you see the output from the task, forward it to the corresponding backlog item.
At which point in your Desktop/Web application you perform this subscription to the Bus? At the very initialization of your app?
That seems pretty reasonable.
Should that subscriptor (in the same thread) call a BacklogItem repository and perform the update? (But that would be a violation of the rule of not updating two aggregates in the same transaction since this would happen synchronously, right?).
Same thread and same transaction are not necessarily coincident. It can all be coordinated in the same thread; but it probably makes more sense to let the consequences happen in the background. At their core, events and commands are just messages - write the message, put it into an inbox, and let the next thread worry about processing.
If you want to achieve eventual consistency to fulfil the previously mentioned rule, do I really need a Message Broker like RabbitMQ even though both BacklogItem and Task live inside the same Bounded Context?
No; the mechanics of the plumbing matter not at all.
What is the best way to achieve DB consistency in microservice-based systems?
At the GOTO in Berlin, Martin Fowler was talking about microservices and one "rule" he mentioned was to keep "per-service" databases, which means that services cannot directly connect to a DB "owned" by another service.
This is super-nice and elegant but in practice it becomes a bit tricky. Suppose that you have a few services:
a frontend
an order-management service
a loyalty-program service
Now, a customer make a purchase on your frontend, which will call the order management service, which will save everything in the DB -- no problem. At this point, there will also be a call to the loyalty-program service so that it credits / debits points from your account.
Now, when everything is on the same DB / DB server it all becomes easy since you can run everything in one transaction: if the loyalty program service fails to write to the DB we can roll the whole thing back.
When we do DB operations throughout multiple services this isn't possible, as we don't rely on one connection / take advantage of running a single transaction.
What are the best patterns to keep things consistent and live a happy life?
I'm quite eager to hear your suggestions!..and thanks in advance!
This is super-nice and elegant but in practice it becomes a bit tricky
What it means "in practice" is that you need to design your microservices in such a way that the necessary business consistency is fulfilled when following the rule:
that services cannot directly connect to a DB "owned" by another service.
In other words - don't make any assumptions about their responsibilities and change the boundaries as needed until you can find a way to make that work.
Now, to your question:
What are the best patterns to keep things consistent and live a happy life?
For things that don't require immediate consistency, and updating loyalty points seems to fall in that category, you could use a reliable pub/sub pattern to dispatch events from one microservice to be processed by others. The reliable bit is that you'd want good retries, rollback, and idempotence (or transactionality) for the event processing stuff.
If you're running on .NET some examples of infrastructure that support this kind of reliability include NServiceBus and MassTransit. Full disclosure - I'm the founder of NServiceBus.
Update: Following comments regarding concerns about the loyalty points: "if balance updates are processed with delay, a customer may actually be able to order more items than they have points for".
Many people struggle with these kinds of requirements for strong consistency. The thing is that these kinds of scenarios can usually be dealt with by introducing additional rules, like if a user ends up with negative loyalty points notify them. If T goes by without the loyalty points being sorted out, notify the user that they will be charged M based on some conversion rate. This policy should be visible to customers when they use points to purchase stuff.
I don’t usually deal with microservices, and this might not be a good way of doing things, but here’s an idea:
To restate the problem, the system consists of three independent-but-communicating parts: the frontend, the order-management backend, and the loyalty-program backend. The frontend wants to make sure some state is saved in both the order-management backend and the loyalty-program backend.
One possible solution would be to implement some type of two-phase commit:
First, the frontend places a record in its own database with all the data. Call this the frontend record.
The frontend asks the order-management backend for a transaction ID, and passes it whatever data it would need to complete the action. The order-management backend stores this data in a staging area, associating with it a fresh transaction ID and returning that to the frontend.
The order-management transaction ID is stored as part of the frontend record.
The frontend asks the loyalty-program backend for a transaction ID, and passes it whatever data it would need to complete the action. The loyalty-program backend stores this data in a staging area, associating with it a fresh transaction ID and returning that to the frontend.
The loyalty-program transaction ID is stored as part of the frontend record.
The frontend tells the order-management backend to finalize the transaction associated with the transaction ID the frontend stored.
The frontend tells the loyalty-program backend to finalize the transaction associated with the transaction ID the frontend stored.
The frontend deletes its frontend record.
If this is implemented, the changes will not necessarily be atomic, but it will be eventually consistent. Let’s think of the places it could fail:
If it fails in the first step, no data will change.
If it fails in the second, third, fourth, or fifth, when the system comes back online it can scan through all frontend records, looking for records without an associated transaction ID (of either type). If it comes across any such record, it can replay beginning at step 2. (If there is a failure in step 3 or 5, there will be some abandoned records left in the backends, but it is never moved out of the staging area so it is OK.)
If it fails in the sixth, seventh, or eighth step, when the system comes back online it can look for all frontend records with both transaction IDs filled in. It can then query the backends to see the state of these transactions—committed or uncommitted. Depending on which have been committed, it can resume from the appropriate step.
I agree with what #Udi Dahan said. Just want to add to his answer.
I think you need to persist the request to the loyalty program so that if it fails it can be done at some other point. There are various ways to word/do this.
1) Make the loyalty program API failure recoverable. That is to say it can persist requests so that they do not get lost and can be recovered (re-executed) at some later point.
2) Execute the loyalty program requests asynchronously. That is to say, persist the request somewhere first then allow the service to read it from this persisted store. Only remove from the persisted store when successfully executed.
3) Do what Udi said, and place it on a good queue (pub/sub pattern to be exact). This usually requires that the subscriber do one of two things... either persist the request before removing from the queue (goto 1) --OR-- first borrow the request from the queue, then after successfully processing the request, have the request removed from the queue (this is my preference).
All three accomplish the same thing. They move the request to a persisted place where it can be worked on till successful completion. The request is never lost, and retried if necessary till a satisfactory state is reached.
I like to use the example of a relay race. Each service or piece of code must take hold and ownership of the request before allowing the previous piece of code to let go of it. Once it's handed off, the current owner must not lose the request till it gets processed or handed off to some other piece of code.
Even for distributed transactions you can get into "transaction in doubt status" if one of the participants crashes in the midst of the transaction. If you design the services as idempotent operation then life becomes a bit easier. One can write programs to fulfill business conditions without XA. Pat Helland has written excellent paper on this called "Life Beyond XA". Basically the approach is to make as minimum assumptions about remote entities as possible. He also illustrated an approach called Open Nested Transactions (http://www.cidrdb.org/cidr2013/Papers/CIDR13_Paper142.pdf) to model business processes. In this specific case, Purchase transaction would be top level flow and loyalty and order management will be next level flows. The trick is to crate granular services as idempotent services with compensation logic. So if any thing fails anywhere in the flow, individual services can compensate for it. So e.g. if order fails for some reason, loyalty can deduct the accrued point for that purchase.
Other approach is to model using eventual consistency using CALM or CRDTs. I've written a blog to highlight using CALM in real life - http://shripad-agashe.github.io/2015/08/Art-Of-Disorderly-Programming May be it will help you.
I'm working on an application that will process data submitted by the user, and compare with past logged data. I don't need to return or respond to the post straight away, just need to process it. This "processing" involves logging the response (in this case a score from 1 to 10) that's submitted by the user every day, then comparing it against the previous scores they submitted. Then if something found, do something (not sure yet, maybe email).
Though I'm worried about the effectiveness of doing this and how it could affect the site's performance. I'd like to keep it server side so the script for calculating isn't exposed. The site is only dealing with 500-1500 responses (users) per day, so it isn't a massive amount, but just interested to know if this route of processing will work. The server the site will be hosted on won't be anything special, probably a small(/est) AWS instance.
Also, will be using Node.js and SQL/PSQL database.
It depends on how do you implement this processing algorithm and how heavy on resources this algorithm is.
If your task is completely syncronous its obviously going to block any incoming requests for your application until its finished.
You can make this "processing-application" as a seperate node process and communicate with it only what you need.
If this is a heavy task and you worry about performance its a good idea to make it a seperate node process so it does not impact the serving of the users.
I recoment to google for "node js asynchronous" to better understand the subject.
Is there a way to have a pub/sub queue in RabbitMq in which any of the subscribers could vote and all give a thumbs up (or more importantly a thumbs down) before processing continues?
I am not sure what to call this so It is very hard to research.
I am trying to make subscribers that can be added and have the ability to veto a process without knowing about them up front.
{edit below}
I am essentially trying to build a distributed set of services that could filter in very specific use cases as they are discovered. I am trying to do this so I do not have to down my service and version every time one of these new use cases is discovered.
Here is a contrived example but it gets the point across:
Lets say I want to calculate if a number is Prime.
I would like to have a service that has general easy rules, (is factor of 2? is factor of 3?)
But lets say that we are getting into very large numbers, and I find a new algorithm that is faster for finding specific cases.
I would like to be able to add this service, have it subscribe to the feed and be able to trigger "NotPrime" and have all other subscribers abandon their work as a result of the veto.
In a monolithic world I would look at some sort of plug in framework and have that implement the filters. That seems like mixing strategies in a bad way if were to do the same within a micro service.
Tamper data
There is terrible thing called Tamper Data. It receives all POST'ing data from FLASH to PHP and give ability for user to change values.
Imagine that in flash game (written in ActionScript 3) are score points and time. After match completed score and time variables are sending to PHP and inserting to database.
But user can easy change values with Tamper Data after match completed. So changed values will be inserted to database.
My idea seems that won't work
I had idea to update data in database on every change? I mean If player get +10 score points I need instant to write It to database. But how about time? I need update my table in database every milisecond? Is that protection solution at all? If user can change POST data he can change It everytime also last time when game completed.
So how to avoid 3rd party software like Tamper Data?
Tokens. I've read article about Tokens, there is talking about how to create random string as token and compare It with database, but It's not detailed and I don't have idea how to realise It. Is that good idea? If yes, maybe someone how to realise It practically?
According to me is better way to send both parameter and value in encrypted format like score=12 send like c2NvcmU9MTI= which is base64
function encrypt($str)
{
$s = strtr(base64_encode(mcrypt_encrypt(MCRYPT_RIJNDAEL_256, md5(SALTKEY), serialize($str), MCRYPT_MODE_CBC, md5(md5(SALTKEY)))), '+/=', '-_,');
return $s;
}
function decrypt($str)
{
$s = unserialize(rtrim(mcrypt_decrypt(MCRYPT_RIJNDAEL_256, md5(SALTKEY), base64_decode(strtr($str, '-_,', '+/=')), MCRYPT_MODE_CBC, md5(md5(SALTKEY))), "\0"));
return $s;
}
In general, there is no way to protect the content generated in Flash and sent to server.
Even if you encrypt the data with a secret key, both the key and the encryption algorithm are contained in the swf file and can be decompiled. It is a bit more harder than simply faking the data so it is kind of usable solution but it will not always help.
To have full security, you need to run all game simulation on the server. For example, if player jumped and catched a coin, Flash does not send "score +10" to the server. Instead, it sends player coordinates and speed, and server does the check: where is the coin, where is the player, what is player's speed and can the player get the coin or not.
If you cannot run the full simulation on the server, you can do a partial check by sending data to server at some intervals.
First, never send a "final" score or any other score. It is very easy to fake. Instead, send an event every time the player does something that changes his score.
For example, every time player catches a coin, you send this event to the server. You may not track player coordinates or coin coordinates, but you know that the level contains only 10 coins. So player cannot catch more than 10 coins anyway. Also, player can't catch coins too fast because you know the minimum distance between coins and the maximum player speed.
You should not write the data to database each time you receive it. Instead you need to keep each player's data in memory and change it there. You can use a noSQL database for that, for example Redis.
First, cheaters will always cheat. There's really no easy solution (or difficult one) to completely prevent it. There are lots of articles on the great lengths developers have gone to discourage cheating, yet it is still rampant in nearly every game with any popularity.
That said, here are a few suggestions to hopefully discourage cheating:
Encrypt your data. This is not unbeatable, but will discourage many lazy hackers since they can't just tamper with plain http traffic, they first have to find your encryption keys. Check out as3corelib for AS3 encryption.
Obfuscate your SWFs. There are a few tools out there to do this for you. Again, this isn't unbeatable, but it is an easy way to make it harder for cheaters to find your encryption keys.
Move all your timing logic to the server. Instead of your client telling the server about time, tell the server about actions like "GAME_STARTED" and "SCORED_POINTS". The server then tracks the user's time and calculates the final score. The important thing here is that the client does not tell the server anything related to time, but merely the action taken and the server uses its own time.
If you can establish any rules about maximum possible performance (for example 10 points per second) you can detect some types of cheating on the server. For example, if you receive SCORED_POINTS=100 but the maximum is 10, you have a cheater. Or, if you receive SCORED_POINTS=10, then SCORE_POINTS=10 a few milliseconds later, and again a few milliseconds later, you probably have a cheater. Be careful with this, and know that it's a back and forth battle. Cheaters will always come up with clever ways to get around your detection logic, and you don't want your detection logic to be so strict that you accidentally reject an honest player (perhaps a really skilled player who is out-performing what you initially thought possible).
When you detect a cheater, "honey pot" them. Don't tell them they are cheating, as this will only encourage them to find ways to avoid detection.