I'm looking for an out of the box solution to get a server-side only transaction around some SOAP-requests. I'm explicitly not looking for a distributed transaction or WS-* standard, because they change the interface at the client side and I need it to be server side only.
So, for the client:
webservice.StartTransaction(); // Or included in DoSomething(...)
webservice.DoSomething(Whatever);
webservice.DoSomethingElse();
webservice.CommitTransaction(); // Or included in DoSomethingElse()
At the server, a Transaction must be started at webservice.StartTransaction() and committed at webservice.CommitTransaction. If there is no commit in some limited timeframe, the transaction should rollback.
While your requirements are not entirely clear, you may want to consider leveraging Web Service Coordination (WS-Coordination) along with Web Service Atomic Transaction (WS-AtomicTransaction).
The WS-Coordination specification describes an extensible framework for providing protocols that coordinate the actions of distributed applications. Such coordination protocols are used to support a number of applications, including those that need to reach consistent agreement on the outcome of distributed activities.
http://docs.oasis-open.org/ws-tx/wstx-wsat-1.2-spec-os/wstx-wsat-1.2-spec-os.html
The WS-AtomicTransaction specification provides the definition of the Atomic Transaction coordination type that is to be used with the extensible coordination framework described in WS-Coordination. This specification defines three specific agreement coordination protocols for the Atomic Transaction coordination type: completion, volatile two-phase commit, and durable two-phase commit. Developers can use any or all of these protocols when building applications that require consistent agreement on the outcome of short-lived distributed activities that have the all-or-nothing property.
https://www.oasis-open.org/committees/download.php/35161/wstx-wscoor-1%202-spec-os.pdf
A Microsoft-based example:
http://msdn.microsoft.com/en-us/library/ms752261(v=vs.110).aspx
Related
I am new to message queues and was wondering if people can explain usecase(s) for using an exclusive queue in RabbitMQ.
From the docs:
An exclusive queue can only be used (consumed from, purged, deleted, etc) by its declaring connection.
Exclusive queues are deleted when their declaring connection is closed or gone. They therefore are only suitable for client-specific transient state
Exclusive queues are a type of temporary queue, and as such they:
"...can be a reasonable choice for workloads with transient clients, for example, temporary WebSocket connections in user interfaces, mobile applications and devices that are expected to go offline or use switch identities. Such clients usually have inherently transient state that should be replaced when the client reconnects."
See notes on durability for more context.
Exclusive queues, as you note, have an added restriction. An exclusive queue:
"... can only be used (consumed from, purged, deleted, etc) by its declaring connection."
This makes it (potentially) suitable for use as queue contained within a single application, where the application will create and then process its own work queue items.
The following points are just my opinion, reflecting on the above notes:
I think it may be a relatively rare RabbitMQ use case, compared to "publish-and-subscribe" use cases such as topic exchanges and similar implementations which communicate across different nodes and connections.
Also, I would expect that much of the time, the core functionality of this type of queue could be provided by a language's built-in data structures (such as Java's queue implementations). But if you want a mature out-of-the-box queueing solution for your application to use internally, then maybe it can be a suitable option.
I often read about establishing API contracts, but I'm not sure why it's usually recommended to be consumer driven?
Can someone explain what are the advantages/disadvantages of API contracts being consumer driven as opposed to provider driven?
I'm Yousaf, A developer advocate here at Pact https://pact.io/ and Pactflow - https://pactflow.io/
We have an open forum about contract testing in our Pact Foundation Slack, you can join over at https://slack.pact.io
We've long been advocates of consumer-driven contract testing but heard the pains it caused providers pains, so we recently delivered a new feature to allow for provider or consumer driven workflows.
I will use the terms CDCT (Consumer Driven) / BDCT (Bi-Directional) Contract testing.
When contract testing with Pact or traditional CDCT, you need to write and maintain a separate set of (Pact) tests that are responsible for ensuring systems are compatible. The tests on the consumer side produce a consumer contract containing the example scenarios which must be supported for the consumer to work, which are then replayed against an actual running provider in a record and replay style interaction. If the Provider responds correctly, the contract is valid.
With BDCT, the key difference is that a Provider uploads its own provider contract advertising its full capability which is statically compared to the expectations in the consumer contract - the consumer contract is never replayed against the provider code base. This creates a much simpler and decoupled workflow. See the trade-offs for more.
BDCT is not intended to replace Pact/CDCT testing, but to provide an alternative in the cases where Pact/CDCT may not be best suited.
You can see a table of use cases here which should hopefully help describe some of the various scenarios
https://docs.pactflow.io/docs/bi-directional-contract-testing#comparison-to-pact
Consumer Driven Contract Testing
Advantages
Suitable for Architectures such as Backend For Frontend (BFF) where the API Design of the Provider has to be driven by what is required by the Consumer
Multiple Consumers to a single Provider where the overall API that Provider needs to satisfy is a combination of minimum requirements (Consumer Driven Contract) for each Consumer
Disadvantages
Requires sequential style of development as mentioned above (consumer before provider)
Good understanding for writing tests and basic mocking concepts (while I consider this a good thing, the barrier for entry is an issue sometimes)
API Design may have Consumer Bias (The data shared by Provider may usually be designed to be directly presentable with little processing, as along as you do not intend to re-purpose the APIs for other needs this is not so much of a problem)
Tools - Pact
Provider Driven Contract
Advantages
Suitable for scenarios such as Public facing APIs where the Provider needs to publish its API Contract to Consumers and make sure it is adhering to the Contract it is publishing.
Provider need not be aware of who its consumers are, it has only one contract to adhere to, which is its own that it has defined and shared
Disadvantages
Again this is also a sequential development style, just that Provider is built first
API Design is biased towards Provider (API Design may not keep consumer interests in mind requiring additional processing)
Tools - Spring Cloud Contract in "producer contract testing approach" (Spring Cloud Contract also supports Consumer Driven Contract Testing)
There are some more alternatives too.
Pactflow and BDCT require both Consumers and Providers to publish their respective contracts - #YOU5AF has already explained this in his answer
Specmatic and Contract Driven Development leverage API Specifications such as a OpenAPI as Executable Contracts thereby allowing independent parallel development of both Consumer and Provider (I am lead developer and CTO at Specmatic)
The choice of approach should completely be based on the workflow that you intend to adopt.
One advantage of Consumer Driven contract testing is the fact that the consumers control what they need, providers must satisfy the needs of its consumers.
This reduces the risks of breaking changes and eliminates coupling.
The Provider can not change things that break any of its consumers, and if it does, it must provide backwards compatibility for those consumers else contract testing on the side of the provider will fail and the change will not even make it to production.
Provider driven contract testing gives the control to the provider, and consumers must update when there is a change. This introduces tight coupling between the provider and its consumers.
I want to create a "Notifications Microservice" that will handle different type of notifications (Google Chat, Email, etc).
For this task, we will create a microservice that contains the logic on how to process these messages, and we'll be using Rabbit MQ to manage the queue.
Now, the question that I have, is if it is possible (or if it is a bad practice) to expose two endpoints in the microservice like this:
registerNotification('channel', $data)
processNotification(Rabbit Message)
So I only have to implement the communication with RabbitMQ in one service, and other services will just register messages using this same service instead of directly talking to RabbitMQ.
This way for each channel I could validate in the service that I have everything that I need before enqueuing a message.
Is this a good approach?
I'd suggest splitting your question into two separate ones. As usual, it depends ... there's pros and cons to either one. Below my points without claiming completeness. Assessing those really depends on your specific needs in the end.
1) Is it a good practice to use a Notification / Event Gateway in front of a message queue (here RabbitMQ)?
Pros:
enforce strong guarantees on message structure / correctness
provide advanced authentication / authorization mechanisms if required
provide convenience if languages in your stack lack first-class client support
abstract away / encapsulate technology choices & deployment considerations from services (publishers)
eliminate routing logic for messages from individual services (though, using available routing topologies in RabbitMQ, it's hard to see any added value here)
Cons:
availability becomes a critical concern for your gateway, e.g. assuming you can guarantee an uptime of four nines per service, you are already down to three nines for the composed system by adding this dependency
added operational complexity
added latency
An alternative consideration here might be to use a library to achieve some of the pros above. Though, this approach also comes with its own cons.
2) Is it a good practice to run both message publishers and consumers in one service?
Pros:
quick (shortcut?)
initially less deployed instances (until you have to scale up)
Cons:
operational requirements for producers and consumers (workers!) are typically very different
harder (and more expensive) to scale the system adequately and fine grained
(performance) metrics become difficult to interpret
consumers might impact producer latency negatively as everything is competing for the same resources
loss of flexibility on the consumer side (quick, low risk deployments)
harder to guarantee availability of producers
I hope this helps to better evaluate your architecture based on your own needs / priorities.
Consider a company with a large set of functionality to be exposed via web services. A subset of this functionality is used for building up some very complex and computation intensive scenarios, and requires a session to be maintained during this iterative build-up. Each scenario targets one single base structure, representing, say a single customer. That is, a scenario is a series of heavy operations on a single customer structure. The operations can be grouped by which area they target, but basically all operations in the same scenario roots in the same customer structure.
The following decision is given from the outside, and cannot be altered: an already made custom session handler must be used, which basically operates on a session given a simple GUID-token to be send to/from the client. Therefore, from a technical perspective the session need not to be limited to a single service, but can live across multiple services.
Besides the stateful operations, there is also a number of stateless operations.
Given the above decision about the custom session handler, the question is now: how should all these operations be organized? What organization is most elegant?
Some possibilities:
All stateful operations are gathered in one single stateful service, while all stateless services are grouped in an arbitrary set of services, possibly by which area they target. Possible problem: the single stateful service can become very large.
Both stateful and stateless operations are grouped into smaller services, but stateful and stateless operations are still separated so that no service contains both stateful and stateless operations. Possibly, all session estabilshment and finalization can be put in a separate thin dedicated service, say SessionService. With this a approach we have no huge single stateful service. But is the organization elegant? Why force a strict separation of the stateful and stateless operations at all?
Group all operations by their target, ignoring their statefulness. This gives a number of services with mixed stateful and stateless operations. The former can take the session GUID token as input argument, and a service behavior can take care of automatically handle the session estabilshment given some appropriate naming convention for the session key/token.
Similar to above, a separate dedicated service can take care of session establishment and finalization.
Something else, not mentioned above?
Which organizaion is the most elegant?
I have implemented what is basically your third option. We have 20+ operations, some of which check the request for a SessionID (GUID), some of which do not (like Ping() and Login(DeviceID)). All the session handling is "Custom" in the sense that we're not using WCF persistant sessions; rather, we've created a Login() function that takes a GUID, ID, Password from client requests for authentication.
On Login(), we validate the DeviceID, UserID and Pwd against the DB, creating a row on the Session table containing StartTime (session only good for <8 hrs) and DeviceID. We then send back to the client a SessionID (GUID) that he/she uses in all subsequent connections, whether uploading or downloading.
As far as organization is concerned, the subs and methods are organized by the device type (iOS, PC, Android) and the type of operation, just to keep the apples from the oranges. But each function that's Session-related always authenticates the request, validating the inbound SessionID. That may seem wasteful, checking each session with each request (again and again), but because we're using BasicHTTPBinding, we're forced to use a stateless model.
What is the most sensible approach to integrate/interact NServiceBus Sagas with REST APIs?
The scenario is as follows,
We have a load balanced REST API. Depending on the load we can add more nodes.
REST API is a wrapper around a DomainServices API. This means the API can be consumed directly.
We would like to use Sagas for workflow and implement NServiceBus Distributor to scale-out.
Question is, if we use the REST API from Sagas, the actual processing happens in the API farm. This in a way defeats the purpose of implementing distributor pattern.
On the other hand, using DomainServives API directly from Sagas, allows processing locally within worker nodes. With this approach we will have to maintain API assemblies in multiple locations but the throughput could be higher.
I am trying to understand the best approach. Personally, I’d prefer to consume the API (if readily available) but this could introduce chattiness to the system and could take longer to complete as compared to to in-process.
A typical sequence could be similar to publishing an online advertisement,
Advertiser submits a new advertisement request via a web application.
Web application invokes the relevant API endpoint and sends a command
message.
Command message initiates a new publish advertisement Saga
instance.
Saga sends a command to validate caller permissions (in
process/out of process API call)
Saga sends a command to validate the
advertisement data (in process/out of process API call)
Saga sends a
command to the fraud service (third party service)
Once the content and fraud verifications are successful,
Saga sends a command to the billing system.
Saga invokes an API call to save add details. (in
process/out of process API call)
And this goes on until the advertisement is expired, there are a number of retry and failure condition paths.
After a number of design iterations we came up with the following guidelines,
Treat REST API layer as the integration platform.
Assume API endpoints are capable of abstracting fairly complex micro work-flows. Micro work-flows are operations that executes in a single burst (not interruptible) and completes with-in a short time span (<1 second).
Assume API farm is capable of serving many concurrent requests and can be easily scaled-out.
Favor synchronous invocations over asynchronous message based invocations when the target operation is fairly straightforward.
When asynchronous processing is required use a single message handler and invoke API from the handlers. This will delegate work to the API farm. This will also eliminate the need for a distributor and extra hardware resources.
Avoid Saga’s unless if the business work-flow contains multiple transactions, compensation logic and resumes. Tests reveals Sagas do not perform well under load.
Avoid consuming DomainServices directly from a message handler. This till do the work locally and also introduces a deployment hassle by distributing business logic.
Happy to hear out thoughts.
You are right on with identifying that you will need Sagas to manage workflow. I'm willing to bet that your Domain hooks up to a common database. If that is true then it will be faster to use your Domain directly and remove the serialization/network overhead. You will also lose the ability to easily manage the transactions at the database level.
Assuming your are directly calling your Domain, the performance becomes a question of how the Domain performs. You may take steps to optimize the database, drive down distributed transaction costs, sharding the data, etc. You may end up using the Distributor to have multiple Saga processing nodes, but it sounds like you have some more testing to do once a design is chosen.
Generically speaking, we use REST APIs to model the commands as resources(via POST) to allow interaction with NSB from clients who don't have direct access to messaging. This is a potential solution to get things onto NSB from your web app.