I am trying to understand the concepts of corda. I understood that for a transaction to pass a validity consensus, parties should check all the transactions of the particular assest till the issuance transaction.
Described Here.
But how is this actually implemented in code? Do we have to write the logic ourselves or it is handeled by flow framework itself?
It have two main concepts:
The transaction is accepted by the contracts of every input and output state.
This you can visualize in following way. Suppose you have StateA which is handled by ContractA and StateB which is handled by ContractB.
Now suppose you creates a transaction with CommandA and it have both the states i.e. StateA and StateB in your transaction.
Than it is mandatory that you have CommandA in both the contracts and inside that command you validate the transaction and it's state. Therefore making sure that every contract of all the input and output state accepts the transaction.
The transaction has all the required signatures
This is something which is handled automatically in almost all scenarios. You can use collectSignature subflow and than use verifyRequiredSignatures to make sure that transaction have all the required signatories. You can find more on this at : CollectSignaturesFlow and Flow cookbook
Walking the transaction chain is handled automatically when a node or validating notary verifies a transaction.
Related
If B is transferring bitcoin to C then when will be C receive the bitcoin? Is that after all the miners add that transaction into their local ledger? (I think it is time-consuming and what will happen if some miners are not adding?)
Now, if some miners are finding transaction invalid, while some find it valid so what will be the scenario in that case? When will be C receive the bitcoin?
The blockchain is a public ledger, so B isn't sending the transaction to just C, B is announcing it to everyone in the public network that i am sending bitcoin to C. So everyone in the network can see this transaction and now has to wait to be included in a block.
Miners create blocks of transactions, in every 10 minutes a new block is included into the ledger.and they have to create them in such a way that the rest of the network will accept them. One of the requirements is that the transactions in the block are all valid transactions. So yes, miners will validate that B has send bitcoins to C before they add the transaction to a block.
As all the full nodes in the bitcoin blockchain knows the currect state of bitcoins hold by B, they can easily verify the validity of the transaction.
Note: Each node in the blockchain either accepts or rejects a transaction.
Here's details you may want to have a look
https://en.bitcoin.it/wiki/Protocol_rules#.22tx.22_messages
when will be C receive the bitcoin? Is that after all the minors add
that transaction into their local ledger?
Each minor is trying to mine a block. When someone mines a block, the block is distributed along the network. So, every miner get that block. In a block, there would be some transactions.
However, another block could be mined at the same time. A miner who receives more than one block at the same time will continue mining along a branch. So, after some time, one of the branches will win.
Because of that, a transaction is valid when there are 100 more new blocks (I'm not sure if they are 100 or less) on top of that block.
what will happen if some minors are not adding?
After some time, all the minors will reach consensus about the branch and all of them will have the same blocks.
if some minors are finding transaction invalid, while some find it
valid so what will be the scenario in that case?
When a miner receives a block, verifies if all transactions are valid. If not, it sends a message to the rest of the miners to tell them that the block is invalid.
The result of a transaction (valid or invalid) is the same to all the members of the network.
I have been reading about blockchain and ethereum, but I cannot seem to get my head around a couple of concepts.
First, where in the blockchain is a newly created transaction stored ? So if the blockchain has been on going for some time, and lets say we are on block X right now. If I deploy my contract today, and it gets executed, will my transaction details reside on every block after block X or only on block X + 1 ? And will my transaction details be the only details on that block, or will that block contain every transaction that happened within that time period ? Again, do all of the prior blocks transactions get written to subsequent blocks as well ? What happens if more than 1 transaction gets executed from the same contract, is just written as 2 different blocks or within the same block ?
Second, when designing a contract I have seen that it is usually restricted to two parties that enter into it, and for other people to use the contract a new instance of the contract has to be created, is this understanding correct ? Or should 1 contract be designed in a way that everyone uses it and only one instance of it is ever created ?
where in the blockchain is a newly created transaction stored ?
In the blocks that each node stores.
will my transaction details reside on every block after block X or
only on block X + 1 ?
Each transaction only resides in a block. Each block is related with the previous block, so your transaction is throughout all the blockchain.
And will my transaction details be the only details on that block, or
will that block contain every transaction that happened within that
time period ?
It dependes on the implementation of Blockchain. For example, Bitcoin blocks store all the transactions that have been sent throughout 10 minutes, because each block is mined every 10 minutes (more or less).
Second
Blockchain is a distributed system, where all the members are at the same level. So, they get the consensus about what they are going to do, i.e. all the members have to agree about the functions of their Blockchain.
For each Blockchain, you can have more than one contract. But I'm going to explain it more simply. The Smart Contract is the code that is installed on all the nodes of the Blockchain, and every request is executed against it. So, every node/member must have the same.
What is the best way to achieve DB consistency in microservice-based systems?
At the GOTO in Berlin, Martin Fowler was talking about microservices and one "rule" he mentioned was to keep "per-service" databases, which means that services cannot directly connect to a DB "owned" by another service.
This is super-nice and elegant but in practice it becomes a bit tricky. Suppose that you have a few services:
a frontend
an order-management service
a loyalty-program service
Now, a customer make a purchase on your frontend, which will call the order management service, which will save everything in the DB -- no problem. At this point, there will also be a call to the loyalty-program service so that it credits / debits points from your account.
Now, when everything is on the same DB / DB server it all becomes easy since you can run everything in one transaction: if the loyalty program service fails to write to the DB we can roll the whole thing back.
When we do DB operations throughout multiple services this isn't possible, as we don't rely on one connection / take advantage of running a single transaction.
What are the best patterns to keep things consistent and live a happy life?
I'm quite eager to hear your suggestions!..and thanks in advance!
This is super-nice and elegant but in practice it becomes a bit tricky
What it means "in practice" is that you need to design your microservices in such a way that the necessary business consistency is fulfilled when following the rule:
that services cannot directly connect to a DB "owned" by another service.
In other words - don't make any assumptions about their responsibilities and change the boundaries as needed until you can find a way to make that work.
Now, to your question:
What are the best patterns to keep things consistent and live a happy life?
For things that don't require immediate consistency, and updating loyalty points seems to fall in that category, you could use a reliable pub/sub pattern to dispatch events from one microservice to be processed by others. The reliable bit is that you'd want good retries, rollback, and idempotence (or transactionality) for the event processing stuff.
If you're running on .NET some examples of infrastructure that support this kind of reliability include NServiceBus and MassTransit. Full disclosure - I'm the founder of NServiceBus.
Update: Following comments regarding concerns about the loyalty points: "if balance updates are processed with delay, a customer may actually be able to order more items than they have points for".
Many people struggle with these kinds of requirements for strong consistency. The thing is that these kinds of scenarios can usually be dealt with by introducing additional rules, like if a user ends up with negative loyalty points notify them. If T goes by without the loyalty points being sorted out, notify the user that they will be charged M based on some conversion rate. This policy should be visible to customers when they use points to purchase stuff.
I don’t usually deal with microservices, and this might not be a good way of doing things, but here’s an idea:
To restate the problem, the system consists of three independent-but-communicating parts: the frontend, the order-management backend, and the loyalty-program backend. The frontend wants to make sure some state is saved in both the order-management backend and the loyalty-program backend.
One possible solution would be to implement some type of two-phase commit:
First, the frontend places a record in its own database with all the data. Call this the frontend record.
The frontend asks the order-management backend for a transaction ID, and passes it whatever data it would need to complete the action. The order-management backend stores this data in a staging area, associating with it a fresh transaction ID and returning that to the frontend.
The order-management transaction ID is stored as part of the frontend record.
The frontend asks the loyalty-program backend for a transaction ID, and passes it whatever data it would need to complete the action. The loyalty-program backend stores this data in a staging area, associating with it a fresh transaction ID and returning that to the frontend.
The loyalty-program transaction ID is stored as part of the frontend record.
The frontend tells the order-management backend to finalize the transaction associated with the transaction ID the frontend stored.
The frontend tells the loyalty-program backend to finalize the transaction associated with the transaction ID the frontend stored.
The frontend deletes its frontend record.
If this is implemented, the changes will not necessarily be atomic, but it will be eventually consistent. Let’s think of the places it could fail:
If it fails in the first step, no data will change.
If it fails in the second, third, fourth, or fifth, when the system comes back online it can scan through all frontend records, looking for records without an associated transaction ID (of either type). If it comes across any such record, it can replay beginning at step 2. (If there is a failure in step 3 or 5, there will be some abandoned records left in the backends, but it is never moved out of the staging area so it is OK.)
If it fails in the sixth, seventh, or eighth step, when the system comes back online it can look for all frontend records with both transaction IDs filled in. It can then query the backends to see the state of these transactions—committed or uncommitted. Depending on which have been committed, it can resume from the appropriate step.
I agree with what #Udi Dahan said. Just want to add to his answer.
I think you need to persist the request to the loyalty program so that if it fails it can be done at some other point. There are various ways to word/do this.
1) Make the loyalty program API failure recoverable. That is to say it can persist requests so that they do not get lost and can be recovered (re-executed) at some later point.
2) Execute the loyalty program requests asynchronously. That is to say, persist the request somewhere first then allow the service to read it from this persisted store. Only remove from the persisted store when successfully executed.
3) Do what Udi said, and place it on a good queue (pub/sub pattern to be exact). This usually requires that the subscriber do one of two things... either persist the request before removing from the queue (goto 1) --OR-- first borrow the request from the queue, then after successfully processing the request, have the request removed from the queue (this is my preference).
All three accomplish the same thing. They move the request to a persisted place where it can be worked on till successful completion. The request is never lost, and retried if necessary till a satisfactory state is reached.
I like to use the example of a relay race. Each service or piece of code must take hold and ownership of the request before allowing the previous piece of code to let go of it. Once it's handed off, the current owner must not lose the request till it gets processed or handed off to some other piece of code.
Even for distributed transactions you can get into "transaction in doubt status" if one of the participants crashes in the midst of the transaction. If you design the services as idempotent operation then life becomes a bit easier. One can write programs to fulfill business conditions without XA. Pat Helland has written excellent paper on this called "Life Beyond XA". Basically the approach is to make as minimum assumptions about remote entities as possible. He also illustrated an approach called Open Nested Transactions (http://www.cidrdb.org/cidr2013/Papers/CIDR13_Paper142.pdf) to model business processes. In this specific case, Purchase transaction would be top level flow and loyalty and order management will be next level flows. The trick is to crate granular services as idempotent services with compensation logic. So if any thing fails anywhere in the flow, individual services can compensate for it. So e.g. if order fails for some reason, loyalty can deduct the accrued point for that purchase.
Other approach is to model using eventual consistency using CALM or CRDTs. I've written a blog to highlight using CALM in real life - http://shripad-agashe.github.io/2015/08/Art-Of-Disorderly-Programming May be it will help you.
I have an NServiceBus handler that creates a new sql connection and new sql command.
However, the command that is executed is not being committed to the database until after the whole process is finished.
It's like there is a hidden sql transaction in the handler itself.
I moved my code into a custom console application without nservicebus and the sql command executed and saved immediately. Unlike in nservicebus where it doesn't save until the end of the handler.
Indeed every handler is wrapped in a transaction, the default transaction guarantee is relying on DTC. That is intentional :)
If you disable it then you might get duplicate messages or lose some data, so that must be done carefully. You can disable transactions using endpoint configuration API instead of using options in connection string.
Here you can find more information about configuration and available guarantees http://docs.particular.net/nservicebus/transports/transactions.
Unit of work
Messages should be processed as a single unit of work. Either everything succeeds or fails.
If you want to have multiple units of work executed then
create multiple endpoints
or send multiple messages
This also has the benefit that these can potentially be processed in parallel.
Please note, that creating multiple handlers WILL NOT have this effect. All handlers on the same endpoint will be part of the same unit of work thus transaction.
Immediate dispatch
If you really want to send a specific message when the sending of the message must not be part of the unit of work then you can immediately send it like this:
using (new TransactionScope(TransactionScopeOption.Suppress))
{
var myMessage = new MyMessage();
bus.Send(myMessage);
}
This is valid for V5, for other versions its best to look at the documentation:
http://docs.particular.net/nservicebus/messaging/send-a-message#dispatching-a-message-immediately
Enlist=false
This is a workaround that MUST NOT be used to circumvent a specific transactional configuration as is explained very well by Tomasz.
This can result in data corruption because the same messsage can be processed multiple times in case of error recovery while then the same database action will be performed again.
Found the solution.
In my connection string I had to add Enlist=False
As mentioned by #wlabaj Setting Enlist=False will indeed make sure that a transaction opened in the handler will be different from transaction used by the transport to receive/send messages.
It is however important to note that it changes the message processing semantics. By default, when DTC is used, receive/send and any transactional operations inside a handler will be commited/rolled-back atomically. With Enlist=False it's not the case so it's possible that there will be more than one handler transaction being committed for the same message. Consider following scenario as a sample case when it can happen:
message is received (transport transaction gets started)
message is successfully processed inside the handler (handler transaction committed successfully)
transport transaction fails and message is moved back to the input queue
message is received second time
message is successfully processed inside the handler
...
The behavior with Enlist-False setting is something that might a be desirable behavior in your case. That being said I think it's worth clarifying what are the consequences in terms of message processing semantics.
It's my understanding we have essentially 2 kinds of exceptions when using NServiceBus.
Environmental : Meaning any required component is not currently available. Usually resulting in a full rollback of the transaction. This is the description I see behind the rollback within NServiceBus Documentation (Including putting the message back on the bus - which sounds fantastic). How do I do this?
Validation : A message is being processed that cannot succeed because of business logic, rules, etc. Where in I want to rollback all database interaction but there's no value in keeping the command in the queue. In which case I just want to roll back the NHibernate section of the transaction - not the MSMQ portion. How do I do this? Typically I would perform validation before any single message is processed but when you have multiple messages bound together into a single transaction and you want to roll them all back this isn't possible via pre-validation.
My assumption is either the answer is insanely obvious and I've overlooked it or what I'm trying to do isn't possible (in regards to the Validation exception).
NSB takes care of getting the message out of the way by moving it to an error queue(v2.5). In v3 this functionality is enhanced and will give you more options to handle faults(DB, custom, etc.). The error queue is configured in your app.config.
In my experience, it's easiest (and probably also more appropriate) to ensure that messages have a very high probability that they can succeed when they participate in a distributed transaction.
Therefore, most validation logic should already have been carried out when you dispatch the command message, and rollback is reserved for the truly exceptional case.
If your client cannot perform the validation, maybe you should insert a validation service in front of your current service. This validation service could route invalid command messages somewhere else before they reach the real service.
Thank you for your answers. I believe the answer lies somewhere between the two.
We are unfortunately unable to implement a validation service but we've simply added better upfront validation to the message processing logic.
Unfortunately until we get to v3 we are currently unable to use the Error Queue as we are utilizing the message response functionality to alert integrators of issues with their messages. And throwing an unhandled error prevents any responses from being generated.