I'll start my question by providing some context about what we're doing and the problems we're facing.
We are currently building a SaaS (hosted on Amazon AWS) that consists of several microservices that sit behind an API gateway (we're using Kong).
The gateway handles authentication (through consumers with API keys) and exposes the APIs of these microservices that I mentioned, all of which are stateless (there are no sessions, cookies or similar).
Each service is deployed using ECS services (one or more docker containers per service running on one or more EC2 machines) and load balanced using the Amazon Application Load Balancer (ALB).
All tenants (clients) share the same environment, that is, the very same machines and resources. Given our business model, we expect to have few but "big" tenants (at first).
Most of the requests to these services translate in heavy resource usage (CPU mainly) for the duration of the request. The time needed to serve one request is in the range of 2-10 seconds (and not ms like traditional "web-like" applications). This means we serve relatively few requests per minute where each one of them take a while to process (background or batch processing is not an option).
Right now, we don't have a strategy to limit or throttle the amount of requests that a tenant can make on a given period of time. Taken into account the last two considerations from above, it's easy to see this is a problem, since it's almost trivial for a tenant to make more requests than we can handle, causing a degradation on the quality of service (even for other tenants because of the shared resources approach).
We're thinking of strategies to limit/throttle or in general prepare the system to "isolate" tenants, so one tenant can not degrade the performance for others by making more requests than we can handle:
Rate limiting: Define a maximum requests/m that a tenant can make. If more requests arrive, drop them. Kong even has a plugin for it. Sadly, we use a "pay-per-request" pricing model and business do not allow us to use this strategy because we want to serve as many requests as possible in order to get paid for them. If excess requests take more time for a tenant that's fine.
Tenant isolation: Create an isolated environment for each tenant. This one has been discarded too, as it makes maintenance harder and leads to lower resource usage and higher costs.
Auto-scaling: Bring up more machines to absorb bursts. In our experience, Amazon ECS is not very fast at doing this and by the time these new machines are ready it's possibly too late.
Request "throttling": Using algorithms like Leaky Bucket or Token Bucket at the API gateway level to ensure that requests hit the services at a rate we know we can handle.
Right now, we're inclined to take option 4. We want to implement the request throttling (traffic shaping) in such a way that all requests made within a previously agreed rate with the tenant (enforced by contract) would be passed along to the services without delay. Since we know in advance how many requests per minute each tenant is gonna be making (estimated at least) we can size our infrastructure accordingly (plus a safety margin).
If a burst arrives, the excess requests would be queued (up to a limit) and then released at a fixed rate (using the leaky bucket or similar algorithm). This would ensure that a tenant can not impact the performance of other tenants, since requests will hit the services at a predefined rate. Ideally, the allowed request rate would be "dynamic" in such a way that a tenant can use some of the "requests per minute" of other tenants that are not using them (within safety limits). I believe this is called the "Dynamic Rate Leaky Bucket" algorithm. The goal is to maximize resource usage.
My questions are:
Is the proposed strategy a viable one? Do you know of any other viable strategies for this use case?
Is there an open-source, commercial or SaaS service that can provide this traffic shaping capabilities? As far as I know Kong or Tyk do not support anything like this, so... Is there any other API gateway that does?
In case Kong does not support this, How hard it is to implement something like what I've described as a plugin? We have to take into account that it would need some shared state (using Redis for example) as we're using multiple Kong instances (for load balancing and high availability).
Thank you very much,
Mikel.
Managing request queue on Gateway side is indeed tricky thing, and probably the main reason why it is not implemented in this Gateways, is that it is really hard to do right. You need to handle all the distributed system cases, and in addition, it hard makes it "safe", because "slow" clients quickly consume machine resources.
Such pattern usually offloaded to client libraries, so when client hits rate limit status code, it uses smth like exponential backoff technique to retry requests. It is way easier to scale and implement.
Can't say for Kong, but Tyk, in this case, provides two basic numbers you can control, quota - maximum number of requests client can make in given period of time, and rate limits - safety protection. You can set rate limit 1) per "policy", e.g for group of consumers (for example if you have multiple tiers of your service, with different allowed usage/rate limits), 2) per individual key 3) Globally for API (works together with key rate limits). So for example, you can set some moderate client rate limits, and cap total limit with global API setting.
If you want fully dynamic scheme, and re-calculate limits based on cluster load, it should be possible. You will need to write and run this scheduler somewhere, from time to time it will perform re-calculation, based on current total usage (which Tyk calculate for you, and you get it from Redis) and will talk with Tyk API, by iterating through all keys (or policies) and dynamically updating their rate limits.
Hope it make sense :)
I have read articles about the differences between SOAP and REST as a web service communication protocol, but I think that the biggest advantages for REST over SOAP are:
REST is more dynamic, no need to create and update UDDI(Universal Description, Discovery, and Integration).
REST is not restricted to only XML format. RESTful web services can send plain text/JSON/XML.
But SOAP is more standardized (E.g.: security).
So, am I correct in these points?
Unfortunately, there are a lot of misinformation and misconceptions around REST. Not only your question and the answer by #cmd reflect those, but most of the questions and answers related to the subject on Stack Overflow.
SOAP and REST can't be compared directly, since the first is a protocol (or at least tries to be) and the second is an architectural style. This is probably one of the sources of confusion around it, since people tend to call REST any HTTP API that isn't SOAP.
Pushing things a little and trying to establish a comparison, the main difference between SOAP and REST is the degree of coupling between client and server implementations. A SOAP client works like a custom desktop application, tightly coupled to the server. There's a rigid contract between client and server, and everything is expected to break if either side changes anything. You need constant updates following any change, but it's easier to ascertain if the contract is being followed.
A REST client is more like a browser. It's a generic client that knows how to use a protocol and standardized methods, and an application has to fit inside that. You don't violate the protocol standards by creating extra methods, you leverage on the standard methods and create the actions with them on your media type. If done right, there's less coupling, and changes can be dealt with more gracefully. A client is supposed to enter a REST service with zero knowledge of the API, except for the entry point and the media type. In SOAP, the client needs previous knowledge on everything it will be using, or it won't even begin the interaction. Additionally, a REST client can be extended by code-on-demand supplied by the server itself, the classical example being JavaScript code used to drive the interaction with another service on the client-side.
I think these are the crucial points to understand what REST is about, and how it differs from SOAP:
REST is protocol independent. It's not coupled to HTTP. Pretty much like you can follow an ftp link on a website, a REST application can use any protocol for which there is a standardized URI scheme.
REST is not a mapping of CRUD to HTTP methods. Read this answer for a detailed explanation on that.
REST is as standardized as the parts you're using. Security and authentication in HTTP are standardized, so that's what you use when doing REST over HTTP.
REST is not REST without hypermedia and HATEOAS. This means that a client only knows the entry point URI and the resources are supposed to return links the client should follow. Those fancy documentation generators that give URI patterns for everything you can do in a REST API miss the point completely. They are not only documenting something that's supposed to be following the standard, but when you do that, you're coupling the client to one particular moment in the evolution of the API, and any changes on the API have to be documented and applied, or it will break.
REST is the architectural style of the web itself. When you enter Stack Overflow, you know what a User, a Question and an Answer are, you know the media types, and the website provides you with the links to them. A REST API has to do the same. If we designed the web the way people think REST should be done, instead of having a home page with links to Questions and Answers, we'd have a static documentation explaining that in order to view a question, you have to take the URI stackoverflow.com/questions/<id>, replace id with the Question.id and paste that on your browser. That's nonsense, but that's what many people think REST is.
This last point can't be emphasized enough. If your clients are building URIs from templates in documentation and not getting links in the resource representations, that's not REST. Roy Fielding, the author of REST, made it clear on this blog post: REST APIs must be hypertext-driven.
With the above in mind, you'll realize that while REST might not be restricted to XML, to do it correctly with any other format you'll have to design and standardize some format for your links. Hyperlinks are standard in XML, but not in JSON. There are draft standards for JSON, like HAL.
Finally, REST isn't for everyone, and a proof of that is how most people solve their problems very well with the HTTP APIs they mistakenly called REST and never venture beyond that. REST is hard to do sometimes, especially in the beginning, but it pays over time with easier evolution on the server side, and client's resilience to changes. If you need something done quickly and easily, don't bother about getting REST right. It's probably not what you're looking for. If you need something that will have to stay online for years or even decades, then REST is for you.
REST vs SOAP is not the right question to ask.
REST, unlike SOAP is not a protocol.
REST is an architectural style and a design for network-based software architectures.
REST concepts are referred to as resources. A representation of a resource must be stateless. It is represented via some media type. Some examples of media types include XML, JSON, and RDF. Resources are manipulated by components. Components request and manipulate resources via a standard uniform interface. In the case of HTTP, this interface consists of standard HTTP ops e.g. GET, PUT, POST, DELETE.
#Abdulaziz's question does illuminate the fact that REST and HTTP are often used in tandem. This is primarily due to the simplicity of HTTP and its very natural mapping to RESTful principles.
Fundamental REST Principles
Client-Server Communication
Client-server architectures have a very distinct separation of concerns. All applications built in the RESTful style must also be client-server in principle.
Stateless
Each client request to the server requires that its state be fully represented. The server must be able to completely understand the client request without using any server context or server session state. It follows that all state must be kept on the client.
Cacheable
Cache constraints may be used, thus enabling response data to be marked as cacheable or not-cacheable. Any data marked as cacheable may be reused as the response to the same subsequent request.
Uniform Interface
All components must interact through a single uniform interface. Because all component interaction occurs via this interface, interaction with different services is very simple. The interface is the same! This also means that implementation changes can be made in isolation. Such changes, will not affect fundamental component interaction because the uniform interface is always unchanged. One disadvantage is that you are stuck with the interface. If an optimization could be provided to a specific service by changing the interface, you are out of luck as REST prohibits this. On the bright side, however, REST is optimized for the web, hence incredible popularity of REST over HTTP!
The above concepts represent defining characteristics of REST and differentiate the REST architecture from other architectures like web services. It is useful to note that a REST service is a web service, but a web service is not necessarily a REST service.
See this blog post on REST Design Principles for more details on REST and the above stated bullets.
EDIT: update content based on comments
SOAP (Simple Object Access Protocol) and REST (Representation State Transfer) both are beautiful in their way. So I am not comparing them. Instead, I am trying to depict the picture, when I preferred to use REST and when SOAP.
What is payload?
When data is sent over the Internet, each unit transmitted includes both header information and the actual data being sent. The header identifies the source and destination of the packet, while the actual data is referred to as the payload. In general, the payload is the data that is carried on behalf of an application and the data received by the destination system.
Now, for example, I have to send a Telegram and we all know that the cost of the telegram will depend on some words.
So tell me among below mentioned these two messages, which one is cheaper to send?
<name>Arin</name>
or
"name": "Arin"
I know your answer will be the second one although both representing the same message second one is cheaper regarding cost.
So I am trying to say that, sending data over the network in JSON format is cheaper than sending it in XML format regarding payload.
Here is the first benefit or advantages of REST over SOAP. SOAP only support XML, but REST supports different format like text, JSON, XML, etc. And we already know, if we use Json then definitely we will be in better place regarding payload.
Now, SOAP supports the only XML, but it also has its advantages.
Really! How?
SOAP relies on XML in three ways
Envelope – that defines what is in the message and how to process it.
A set of encoding rules for data types, and finally the layout of the procedure calls and responses gathered.
This envelope is sent via a transport (HTTP/HTTPS), and an RPC (Remote Procedure Call) is executed, and the envelope is returned with information in an XML formatted document.
The important point is that one of the advantages of SOAP is the use of the “generic” transport but REST uses HTTP/HTTPS. SOAP can use almost any transport to send the request but REST cannot. So here we got an advantage of using SOAP.
As I already mentioned in above paragraph “REST uses HTTP/HTTPS”, so go a bit deeper on these words.
When we are talking about REST over HTTP, all security measures applied HTTP are inherited, and this is known as transport level security and it secures messages only while it is inside the wire but once you delivered it on the other side you don’t know how many stages it will have to go through before reaching the real point where the data will be processed. And of course, all those stages could use something different than HTTP.So Rest is not safer completely, right?
But SOAP supports SSL just like REST additionally it also supports WS-Security which adds some enterprise security features. WS-Security offers protection from the creation of the message to it’s consumption. So for transport level security whatever loophole we found that can be prevented using WS-Security.
Apart from that, as REST is limited by it's HTTP protocol so it’s transaction support is neither ACID compliant nor can provide two-phase commit across distributed transnational resources.
But SOAP has comprehensive support for both ACID based transaction management for short-lived transactions and compensation based transaction management for long-running transactions. It also supports two-phase commit across distributed resources.
I am not drawing any conclusion, but I will prefer SOAP-based web service while security, transaction, etc. are the main concerns.
Here is the "The Java EE 6 Tutorial" where they have said A RESTful design may be appropriate when the following conditions are met. Have a look.
Hope you enjoyed reading my answer.
REST(REpresentational State Transfer)
REpresentational State of an Object is Transferred is REST i.e. we don't send Object, we send state of Object.
REST is an architectural style. It doesn’t define so many standards like SOAP. REST is for exposing Public APIs(i.e. Facebook API, Google Maps API) over the internet to handle CRUD operations on data. REST is focused on accessing named resources through a single consistent interface.
SOAP(Simple Object Access Protocol)
SOAP brings its own protocol and focuses on exposing pieces of application logic (not data) as services. SOAP exposes operations. SOAP is focused on accessing named operations, each operation implement some business logic. Though SOAP is commonly referred to as web services this is misnomer. SOAP has a very little if anything to do with the Web. REST provides true Web services based on URIs and HTTP.
Why REST?
Since REST uses standard HTTP it is much simpler in just about ever way.
REST is easier to implement, requires less bandwidth and resources.
REST permits many different data formats where as SOAP only permits XML.
REST allows better support for browser clients due to its support for JSON.
REST has better performance and scalability. REST reads can be cached, SOAP based reads cannot be cached.
If security is not a major concern and we have limited resources. Or we want to create an API that will be easily used by other developers publicly then we should go with REST.
If we need Stateless CRUD operations then go with REST.
REST is commonly used in social media, web chat, mobile services and Public APIs like Google Maps.
RESTful service return various MediaTypes for the same resource, depending on the request header parameter "Accept" as application/xml or application/json for POST and /user/1234.json or GET /user/1234.xml for GET.
REST services are meant to be called by the client-side application and not the end user directly.
ST in REST comes from State Transfer. You transfer the state around instead of having the server store it, this makes REST services scalable.
Why SOAP?
SOAP is not very easy to implement and requires more bandwidth and resources.
SOAP message request is processed slower as compared to REST and it does not use web caching mechanism.
WS-Security: While SOAP supports SSL (just like REST) it also supports WS-Security which adds some enterprise security features.
WS-AtomicTransaction: Need ACID Transactions over a service, you’re going to need SOAP.
WS-ReliableMessaging: If your application needs Asynchronous processing and a guaranteed level of reliability and security. Rest doesn’t have a standard messaging system and expects clients to deal with communication failures by retrying.
If the security is a major concern and the resources are not limited then we should use SOAP web services. Like if we are creating a web service for payment gateways, financial and telecommunication related work then we should go with SOAP as here high security is needed.
source1
source2
IMHO you can't compare SOAP and REST where those are two different things.
SOAP is a protocol and REST is a software architectural pattern. There is a lot of misconception in the internet for SOAP vs REST.
SOAP defines XML based message format that web service-enabled applications use to communicate each other over the internet. In order to do that the applications need prior knowledge of the message contract, datatypes, etc..
REST represents the state(as resources) of a server from an URL.It is stateless and clients should not have prior knowledge to interact with server beyond the understanding of hypermedia.
First of all: officially, the correct question would be web services + WSDL + SOAP vs REST.
Because, although the web service, is used in the loose sense, when using the HTTP protocol to transfer data instead of web pages, officially it is a very specific form of that idea. According to the definition, REST is not "web service".
In practice however, everyone ignores that, so let's ignore it too
There are already technical answers, so I'll try to provide some intuition.
Let's say you want to call a function in a remote computer, implemented in some other programming language (this is often called remote procedure call/RPC). Assume that function can be found at a specific URL, provided by the person who wrote it. You have to (somehow) send it a message, and get some response. So, there are two main questions to consider.
what is the format of the message you should send
how should the message be carried back and forth
For the first question, the official definition is WSDL. This is an XML file which describes, in detailed and strict format, what are the parameters, what are their types, names, default values, the name of the function to be called, etc. An example WSDL here shows that the file is human-readable (but not easily).
For the second question, there are various answers. However, the only one used in practice is SOAP. Its main idea is: wrap the previous XML (the actual message) into yet another XML (containing encoding info and other helpful stuff), and send it over HTTP. The POST method of the HTTP is used to send the message, since there is always a body.
The main idea of this whole approach is that you map a URL to a function, that is, to an action. So, if you have a list of customers in some server, and you want to view/update/delete one, you must have 3 URLS:
myapp/read-customer and in the body of the message, pass the id of the customer to be read.
myapp/update-customer and in the body, pass the id of the customer, as well as the new data
myapp/delete-customer and the id in the body
The REST approach sees things differently. A URL should not represent an action, but a thing (called resource in the REST lingo). Since the HTTP protocol (which we are already using) supports verbs, use those verbs to specify what actions to perform on the thing.
So, with the REST approach, customer number 12 would be found on URL myapp/customers/12. To view the customer data, you hit the URL with a GET request. To delete it, the same URL, with a DELETE verb. To update it, again, the same URL with a POST verb, and the new content in the request body.
For more details about the requirements that a service has to fulfil to be considered truly RESTful, see the Richardson maturity model. The article gives examples, and, more importantly, explains why a (so-called) SOAP service, is a level-0 REST service (although, level-0 means low compliance to this model, it's not offensive, and it is still useful in many cases).
Among many others already covered in the many answers, I would highlight that SOAP enables to define a contract, the WSDL, which define the operations supported, complex types, etc.
SOAP is oriented to operations, but REST is oriented at resources.
Personally I would select SOAP for complex interfaces between internal enterprise applications, and REST for public, simpler, stateless interfaces with the outside world.
Addition for:
++ A mistake that’s often made when approaching REST is to think of it as “web services with URLs”—to think of REST as another remote procedure call (RPC) mechanism, like SOAP, but invoked through plain HTTP URLs and without SOAP’s hefty XML namespaces.
++ On the contrary, REST has little to do with RPC. Whereas RPC is service oriented and focused on actions and verbs, REST is resource oriented, emphasizing the things and nouns that comprise an application.
A lot of these answers entirely forgot to mention hypermedia controls (HATEOAS) which is completely fundamental to REST. A few others touched on it, but didn't really explain it so well.
This article should explain the difference between the concepts, without getting into the weeds on specific SOAP features.
REST API
RESTful APIs are the most famous type of API. REST stands REpresentational State Transfer.
REST APIs are APIs that follow standardized principles, properties, and constraints.
You can access resources in the REST API using HTTP verbs.
REST APIs operate on a simple request/response system. You can send a request using these HTTP methods:
GET
POST
PUT
PATCH
DELETE
TRACE
OPTIONS
CONNECT
HEAD
Here are the most common HTTP verbs
GET (read existing data)
POST (create a new response or data)
PATCH (update the data)
DELETE (delete the data)
The client can make requests using HTTP verbs followed by the endpoint.
The endpoint (or route) is the URL you request for. The path determines the resource you’re requesting.
When you send a request to an endpoint, it responds with the relevant data, generally formatted as JSON, XML, plain text, images, HTML, and more.
REST APIs can also be designed with many different endpoints that return different types of data. Accessing multiple endpoints with a REST API requires various API calls.
An actual RESTful API follows the following five constraints:
Client-Server Architecture
The client requests the data from the server with no third-party interpretation.
Statelessness
Statelessness means that every HTTP request happens in complete isolation. Each request contains the information necessary to service the request. The server never relies on information from previous requests. There’s no state.
Cacheability
Responses can be explicitly or implicitly defined as cacheable or non-cacheable to improve scalability and performance. For example, enabling the cache of GET requests can improve the response times of requests for resource data.
Layering
Different layers of the API architecture should work together, creating a scalable system that is easy to update or adjust.
Uniform Interface
Communication between the client and the server must be done in a standardized language that is independent of both. This improves scalability and flexibility.
REST APIs are a good fit for projects that need to be
Flexible
Scalable
Fast
SOAP API
SOAP is a necessary protocol that helped introduce the widespread use of APIs.
SOAP is the acronym for Simple Object Access Protocol.
SOAP is a standardized protocol that relies on XML to make requests and receive responses.
Even though SOAP is based on XML, the SOAP protocol is still in wide usage.
SOAP APIs make data available as a service and are typically used when performing transactions involving multiple API calls or applications where security is the primary consideration.
SOAP was initially developed for Microsoft in 1998 to provide a standard mechanism for integrating services on the internet regardless of the operating system, object model, or programming language.
The “S” in SOAP stands for Simple, and for a good reason — SOAP can be used with less complexity as it requires less coding in the app layer for transactions, security, and other functions.
SOAP has three primary characteristics:
Extensibility of SOAP API
SOAP allows for extensions that introduce more robust features, such as Windows Server Security, Addressing, and more.
Neutrality of SOAP API
SOAP is capable of operating over a wide range of protocols, like UDP, JMS, SMTP, TCP, and HTTP.can operate.
Independence of SOAP API
SOAP API responses are purely based on XML. Therefore SOAP APIs are platform and language independent.
Developers continue to debate the pros and cons of using SOAP and REST. The best one for your project will be the one that aligns with your needs.
SOAP APIs remain a top choice for corporate entities and government organizations that prioritize security, even though REST has largely dominated web applications.
SOAP is more secure than REST as it uses WS-Security for transmission along with Secure Socket Layer
SOAP also has more excellent transactional reliability, which is another reason why SOAP historically has been favored by the banking industry and other large entities.
What is REST
REST stands for representational state transfer, it's actually an architectural style for creating Web API which treats everything(data or functionality) as recourse.
It expects; exposing resources through URI and responding in multiple formats and representational transfer of state of the resources in stateless manner. Here I am talking about two things:
Stateless manner: Provided by HTTP.
Representational transfer of state: For example if we are adding an employee. .
into our system, it's in POST state of HTTP, after this it would be in GET state of HTTP, PUT and DELETE likewise.
REST can use SOAP web services because it is a concept and can use any protocol like HTTP, SOAP.SOAP uses services interfaces to expose the business logic. REST uses URI to expose business logic.
REST is not REST without HATEOAS. This means that a client only knows the entry point URI and the resources are supposed to return links the client should follow. Those fancy documentation generators that give URI patterns for everything you can do in a REST API miss the point completely. They are not only documenting something that's supposed to be following the standard, but when you do that, you're coupling the client to one particular moment in the evolution of the API, and any changes on the API have to be documented and applied, or it will break.
HATEOAS, an abbreviation for Hypermedia As The Engine Of Application State, is a constraint of the REST application architecture that distinguishes it from most other network application architectures. The principle is that a client interacts with a network application entirely through hypermedia provided dynamically by application servers. A REST client needs no prior knowledge about how to interact with any particular application or server beyond a generic understanding of hypermedia. By contrast, in some service-oriented architectures (SOA), clients and servers interact through a fixed interface shared through documentation or an interface description language (IDL).
Reference 1
Reference 2
Although SOAP and REST share similarities over the HTTP protocol, SOAP is a more rigid set of messaging patterns than REST. The rules in SOAP are relevant because we can’t achieve any degree of standardization without them. REST needs no processing as an architecture style and is inherently more versatile. In the spirit of information exchange, both SOAP and REST depend on well-established laws that everybody has decided to abide by.
The choice of SOAP vs. REST is dependent on the programming language you are using the environment you are using and the specifications.
To answer this question it’s useful to understand the evolution of the architecture of distributed applications from simple layered architectures, to object & service based, to resources based, & nowadays we even have event based architectures. Most large systems use a combination of styles.
The first distributed applications had layered architectures. I'll assume everyone here knows what layers are. These structures are neatly organized, and can be stacks or cyclical structures. Effort is made to maintain a unidirectional data flow.
Object-based architectures evolved out of layered architectures and follow a much looser model. Here, each component is an object (often called a distributed object). The objects interact with one another using a mechanism similar to remote procedure calls - when a client binds to a distributed object it loads an implementation of the objects interface into its address space. The RPC stub can marshal a request & receive a response. Likewise the objects interface on the server is an RPC style stub. The structure of these object based systems is not as neatly organized, it looks more like an object graph.
The interface of a distributed object conceals its implementation. As with layered components, if the interface is clearly defined the internal implementation can be altered - even replaced entirely.
Object-based architectures provide the basis for encapsulating services. A service is provided by a self-contained entity, though internally it can make use of other services. Gradually object-based architectures evolved into service-oriented architectures (SOAs).
With SOA, a distributed application is composed of services. These services can be provided across administrative domains - they may be available across the web (i.e. a storage service offered by a cloud provider).
As web services became popular, and more applications started using them, service composition (combining services to form new ones) became more important. One of the problems with SOA was that integrating different services could become extremely complicated.
While SOAP is a protocol, its use implies a service oriented architecture. SOAP attempted to provide a standard for services whereby they would be composable and easily integrated.
Resource-based architectures were a different approach to solving the integration problems of SOA. The idea is to treat the distributed system as a giant collection of resources that are individually managed by components.
This led to the development of RESTful architectures. One thing that characterizes RESTful services is stateless execution. This is different than SOA where the server maintains the state.
So… how do service-specific interfaces, as provided by service-oriented architectures (including those that use SOAP) compare with resource-based architecture like REST?
While REST is simple, it does not provide a simple interface for complex communication schemes. For example, if you are required to use transactions REST is not appropriate, it is better to keep the complex state encapsulated on the server than have the client manage the transaction. But there are many scenarios where the orthogonal use of resources in RESTful architectures greatly simplifies integration of services in what would otherwise mean an explosion of service interfaces. Another tradeoff is resource-based architectures put more complexity on the client & increase traffic over the network while service-based increase the complexity of the server & tax its memory & CPU resources.
Some people have also mentioned common HTTP services or other services that do not satisfy the requirements of RESTful architecture or SOAP. These too can be categorized as either service-based or resource-based. These have the advantage of being simpler to implement. You'd only use such an approach if you knew your service will never need to be integrated across administrative domains since this makes no attempt at fixing the integration issues that arise.
These sorts of HTTP-based services, especially Pseudo-RESTful services are still the most common types. Implementing SOAP is complicated and should only be used if you really need it - i.e. you need a service that's easily integrated across domains and you want it to have a service-interface. There are still cases where this is needed. A true RESTful service is also difficult to implement, though not as difficult as SOAP.
I need to implement a very secured Web Service using WCF. I have read a lot of documents about security in WCF concerning authorization, authentication, message encryption. The web service will use https, Windows Authentication for access to the WS, SQL Server Membership/Role Provider for user authentication and authorization on WS operations and finally message encryption.
I read in one of documents that it is good to consider security on each layer indenpendently, i.e. Transport Layer security must be thought without considering Message Layer. Therefore, using SSL through https in combination with message encryption (using public/private key encryption and signature) would be a good practice, since https concerns Transport Layer and message encryption concerns Message Layer.
But a friend told me that [https + message encryption] is too much; https is sufficient.
What do you think?
Thanks.
If you have SSL then you still need to encrypt your messages if you don't really trust the server which stores them (it could have its files stolen), so this is all good practice.
There comes a point where you have a weakest link problem.
What is your weakest link?
Example: I spend $100,000,000 defending an airport from terrorists, so they go after a train station instead. Money and effort both wasted.
Ask yourself what the threat model is and design your security for that. TLS is a bare minimum for any Internet-based communications, but it doesn't matter if somebody can install a keystroke logger.
As you certainly understand, the role of Transport-Level Security is to secure the transmission of the message, whereas Message-Level Security is about securing the message itself.
It all depends on the attack vectors (or more generally the purpose) you're considering.
In both cases, the security models involved can have to purposes: protection against eavesdropping (relying on encryption) and integrity protection (ultimately relying on signatures, since based on public-key cryptography in most cases).
TLS with server-certificate only will provide you with the security of the transport, and the client will know that the communication really comes from the server it expects (if configured properly, of course). In addition, if you use client-certificate, this will also guarantee the server that the communication comes from a client that has the private key for this client certificate.
However, when the data is no longer in transit, you rely on the security of the machine where it's used and stored. You might no longer be able to assert with certainty where the data came from, for example.
Message-level security doesn't rely on how the communication was made. Message-level signature allows you to know where the messages came from at a later date, independently of how they've been transferred. This can be useful for audit purposes. Message-level encryption would also reduce the risks of someone getting hold of the data if it's stored somewhere where some data could be taken (e.g. some intranet storage systems).
Basically, if the private key used to decrypt the messages has the same protection as the private key used for SSL authentication, and if the messages are not stored for longer time than the connection, in that case it is certainly overkill.
OTOH, if you've got different servers, or if the key is stored e.g. using hardware security of sorts, or is only made available by user input, then it is good advice to secure the messages themselves as well. Application level security also makes sense for auditing purposes and against configuration mistakes, although personally I think signing the data (integrity protection) is more important in this respect.
Of course, the question can also become: if you're already using a web-service that uses SOAP/WSDL, why not use XML encrypt/sign? It's not that hard to configure. Note that it does certainly take more processor time and memory. Oh, one warning: don't even try it if the other side does not know what they are doing - you'll spend ages explaining it and even then you run into trouble if you want to change a single parameter later on.
Final hint: use standards and standardized software or you'll certainly run into crap. Spend some time getting getting to know how things work, and make sure you don't accept ill formatted messages when you call verify (e.g. XML signing the wrong node or accepting MD5 and such things).