I have an AWS api that proxies lamba functions. I currently use different endpoints with separate lambda functions:
api.com/getData --> getData
api.com/addData --> addData
api.com/signUp --> signUp
The process to manage all the endpoints and functions becomes cumbersome. Is there any disadvantage when I use a single endpoint to one lambda function which decides what to do based on the query string?
api.com/exec&func=getData --> exec --> if(params.func === 'getData') { ... }
It's perfectly valid to map multiple methods to a single lambda function and many people are using this methodology today as opposed to creating an api gateway resource and lambda function for each discrete method.
You might consider proxying all requests to a single function. Take a look at the following documentation on creating an API Gateway => Lambda proxy integration:
http://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-set-up-simple-proxy.html
Their example is great here. A request like the following:
POST /testStage/hello/world?name=me HTTP/1.1
Host: gy415nuibc.execute-api.us-east-1.amazonaws.com
Content-Type: application/json
headerName: headerValue
{
"a": 1
}
Will wind up sending the following event data to your AWS Lambda function:
{
"message": "Hello me!",
"input": {
"resource": "/{proxy+}",
"path": "/hello/world",
"httpMethod": "POST",
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate",
"cache-control": "no-cache",
"CloudFront-Forwarded-Proto": "https",
"CloudFront-Is-Desktop-Viewer": "true",
"CloudFront-Is-Mobile-Viewer": "false",
"CloudFront-Is-SmartTV-Viewer": "false",
"CloudFront-Is-Tablet-Viewer": "false",
"CloudFront-Viewer-Country": "US",
"Content-Type": "application/json",
"headerName": "headerValue",
"Host": "gy415nuibc.execute-api.us-east-1.amazonaws.com",
"Postman-Token": "9f583ef0-ed83-4a38-aef3-eb9ce3f7a57f",
"User-Agent": "PostmanRuntime/2.4.5",
"Via": "1.1 d98420743a69852491bbdea73f7680bd.cloudfront.net (CloudFront)",
"X-Amz-Cf-Id": "pn-PWIJc6thYnZm5P0NMgOUglL1DYtl0gdeJky8tqsg8iS_sgsKD1A==",
"X-Forwarded-For": "54.240.196.186, 54.182.214.83",
"X-Forwarded-Port": "443",
"X-Forwarded-Proto": "https"
},
"queryStringParameters": {
"name": "me"
},
"pathParameters": {
"proxy": "hello/world"
},
"stageVariables": {
"stageVariableName": "stageVariableValue"
},
"requestContext": {
"accountId": "12345678912",
"resourceId": "roq9wj",
"stage": "testStage",
"requestId": "deef4878-7910-11e6-8f14-25afc3e9ae33",
"identity": {
"cognitoIdentityPoolId": null,
"accountId": null,
"cognitoIdentityId": null,
"caller": null,
"apiKey": null,
"sourceIp": "192.168.196.186",
"cognitoAuthenticationType": null,
"cognitoAuthenticationProvider": null,
"userArn": null,
"userAgent": "PostmanRuntime/2.4.5",
"user": null
},
"resourcePath": "/{proxy+}",
"httpMethod": "POST",
"apiId": "gy415nuibc"
},
"body": "{\r\n\t\"a\": 1\r\n}",
"isBase64Encoded": false
}
}
Now you have access to all headers, url params, body etc. and you could use that to handle requests differently in a single Lambda function (basically implementing your own routing).
As an opinion I see some advantages and disadvantages to this approach. Many of them depend on your specific use case:
Deployment: if each lambda function is discrete then you can deploy them independently, which might reduce the risk from code changes (microservices strategy). Conversely you may find that needing to deploy functions separately adds complexity and is burdensome.
Self Description: API Gateway's interface makes it extremely intuitive to see the layout of your RESTful endpoints -- the nouns and verbs are all visible at a glance. Implementing your own routing could come at the expense of this visibility.
Lambda sizing and limits: If you proxy all -- then you'll wind up needing to choose an instance size, timeout etc. that will accommodate all of your RESTful endpoints. If you create discrete functions then you can more carefully choose the memory footprint, timeout, deadletter behavior etc. that best meets the needs of the specific invocation.
I would have commented to just add a couple of points to Dave Maple's great answer but I don't have enough reputation points yet so I'll add the comments here.
I started to head down the path of multiple endpoints pointing to one Lambda function that could treat each endpoint different by accessing the 'resource' property of the Event. After trying it I have now separated them into separate functions for the reasons that Dave suggested plus:
I find it easier to go through logs and monitors when the functions are separated.
One nuance that as a beginner I didn't pick up on at first is that you can have one code base and deploy the exact same code as multiple Lambda functions. This allows you to have the benefits of function separation and the benefits of a consolidated approach in your code base.
You can use the AWS CLI to automate tasks across the multiple functions to reduce/eliminate the downside of managing separate functions. For example, I have a script that updates 10 functions with the same code.
i've been building 5~6 microservices with Lambda-API Gateway, and been through several try & failure and success.
in short, from my experiences, it's better to delegate all the API calls to lambda with just one APIGateway wildcard mapping, such as
/api/{proxy+} -> Lambda
if you ever used any frameworks like grape you know that when making APIs, features like
"middleware"
"global exception handling"
"cascade routing"
"parameter validation"
are really crucial.
as your API grows, it's almost impossible to manage all the routes with API Gateway mapping, nor API Gateway support non of those feature also.
further more, it's not really practically to break lambda for each endpoints for development or deployment.
from your example,
api.com/getData --> getData
api.com/addData --> addData
api.com/signUp --> signUp
imagine you have data ORM, User authentication logic, common view file (such as data.erb).. then how you gonna share that?
you might can break like,
api/auth/{+proxy} -> AuthServiceLambda
api/data/{+proxy} -> DataServiceLambda
but not like "per endpoint". you might can lookup concept of microservice and best practice about how you can split the service
for those web framework like features, checkout this we just built web framework for lambda since i needed this at my company.
A similar scenario is adressed in the official AWS blogpost named Best practices for organizing larger serverless applications.
The general recommendation is to split "monolithic lambdas" into separate lambdas and move the routing to the API Gateway.
This is what the blog writes about the "monolithic lambda" approach:
This approach is generally unnecessary, and it’s often better to take
advantage of the native routing functionality available in API
Gateway.
...
API Gateway is also capable of validating parameters, reducing the
need for checking parameters with custom code. It can also provide
protection against unauthorized access, and a range of other features
more suited to be handled at the service level.
Going from this:
To this
The responsibility of mapping API requests to Lambda in AWS is handled through a Gateway's API specification.
Mapping of URL paths and HTTP methods as well as data validation SHOULD be left up to the Gateway. There is also the question of permissions and API scope; you'll not be able to leverage API scopes and IAM permission levels in a normal way.
In terms of coding, to replicate this mechanism inside of a Lambda handler is an anti-pattern. Going down that route one will soon end up with something that looks like the routing for a node express server, not a Lambda function.
After having set up 50+ Lambdas behind API Gateway I can say that
function handlers should be kept as dump as possible, allowing them to be reusable independent from the context from which they're being invoked.
As far as I know, AWS allows only one handler per Lambda function. That’s why I have created a little "routing" mechanism with Java Generics (for stronger type checks at compile time). In the following example you can call multiple methods and pass different object types to the Lambda and back via one Lambda handler:
Lambda class with handler:
public class GenericLambda implements RequestHandler<LambdaRequest<?>, LambdaResponse<?>> {
#Override
public LambdaResponse<?> handleRequest(LambdaRequest<?> lambdaRequest, Context context) {
switch (lambdaRequest.getMethod()) {
case WARMUP:
context.getLogger().log("Warmup");
LambdaResponse<String> lambdaResponseWarmup = new LambdaResponse<String>();
lambdaResponseWarmup.setResponseStatus(LambdaResponse.ResponseStatus.IN_PROGRESS);
return lambdaResponseWarmup;
case CREATE:
User user = (User)lambdaRequest.getData();
context.getLogger().log("insert user with name: " + user.getName()); //insert user in db
LambdaResponse<String> lambdaResponseCreate = new LambdaResponse<String>();
lambdaResponseCreate.setResponseStatus(LambdaResponse.ResponseStatus.COMPLETE);
return lambdaResponseCreate;
case READ:
context.getLogger().log("read user with id: " + (Integer)lambdaRequest.getData());
user = new User(); //create user object for test, instead of read from db
user.setName("name");
LambdaResponse<User> lambdaResponseRead = new LambdaResponse<User>();
lambdaResponseRead.setData(user);
lambdaResponseRead.setResponseStatus(LambdaResponse.ResponseStatus.COMPLETE);
return lambdaResponseRead;
default:
LambdaResponse<String> lambdaResponseIgnore = new LambdaResponse<String>();
lambdaResponseIgnore.setResponseStatus(LambdaResponse.ResponseStatus.IGNORED);
return lambdaResponseIgnore;
}
}
}
LambdaRequest class:
public class LambdaRequest<T> {
private Method method;
private T data;
private int languageID;
public static enum Method {
WARMUP, CREATE, READ, UPDATE, DELETE
}
public LambdaRequest(){
}
public Method getMethod() {
return method;
}
public void setMethod(Method create) {
this.method = create;
}
public T getData() {
return data;
}
public void setData(T data) {
this.data = data;
}
public int getLanguageID() {
return languageID;
}
public void setLanguageID(int languageID) {
this.languageID = languageID;
}
}
LambdaResponse class:
public class LambdaResponse<T> {
private ResponseStatus responseStatus;
private T data;
private String errorMessage;
public LambdaResponse(){
}
public static enum ResponseStatus {
IGNORED, IN_PROGRESS, COMPLETE, ERROR, COMPLETE_DUPLICATE
}
public ResponseStatus getResponseStatus() {
return responseStatus;
}
public void setResponseStatus(ResponseStatus responseStatus) {
this.responseStatus = responseStatus;
}
public T getData() {
return data;
}
public void setData(T data) {
this.data = data;
}
public String getErrorMessage() {
return errorMessage;
}
public void setErrorMessage(String errorMessage) {
this.errorMessage = errorMessage;
}
}
Example POJO User class:
public class User {
private String name;
public User() {
}
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
}
JUnit test method:
#Test
public void GenericLambda() {
GenericLambda handler = new GenericLambda();
Context ctx = createContext();
//test WARMUP
LambdaRequest<String> lambdaRequestWarmup = new LambdaRequest<String>();
lambdaRequestWarmup.setMethod(LambdaRequest.Method.WARMUP);
LambdaResponse<String> lambdaResponseWarmup = (LambdaResponse<String>) handler.handleRequest(lambdaRequestWarmup, ctx);
//test READ user
LambdaRequest<Integer> lambdaRequestRead = new LambdaRequest<Integer>();
lambdaRequestRead.setData(1); //db id
lambdaRequestRead.setMethod(LambdaRequest.Method.READ);
LambdaResponse<User> lambdaResponseRead = (LambdaResponse<User>) handler.handleRequest(lambdaRequestRead, ctx);
}
ps.: if you have deserialisation problems (LinkedTreeMap cannot be cast to ...) in you Lambda function (because uf the Generics/Gson), use the following statement:
YourObject yourObject = (YourObject)convertLambdaRequestData2Object(lambdaRequest, YourObject.class);
Method:
private <T> Object convertLambdaRequestData2Object(LambdaRequest<?> lambdaRequest, Class<T> clazz) {
Gson gson = new Gson();
String json = gson.toJson(lambdaRequest.getData());
return gson.fromJson(json, clazz);
}
The way I see, choosing single vs multiple API is a function of following considerations:
Security: I think this is the biggest challenge of having a single API structure. It may be possible to have different security profile for different parts of the requirement
Think microservice model from business perspective:
The whole purpose of any API should be serving some requests, hence it must be well understood and easy to use. So related APIs should be combined. For example, if you have a mobile client and it requires 10 things to be pulled in and out from DB, it makes sense to have 10 endpoints into a single API.
But this should be within reason and should be seen in context of overall solution design. For example, if you design a payroll product, you may think to have separate modules for leave management and user details management. Even if they are often used by a single client, they should still be different API, because their business meaning is different.
Reusability: Applies to both code and functionality reusability. Code reusability is a easier problem to solve, ie build common modules for shared requirements and build them as libraries.
Functionality reusability is harder to solve. In my mind, most of the cases can be solved by redesigning the way endpoints/functions are laid out, because if you need duplication of functionality that means your initial design is not detailed enough.
Just found a link in another SO post which summarizes better
I want to write a JSON API.
My problem is, that sometimes I want to query for an ID, sometimes for a String.
One option would be to add a querystring, for example:
example.com/user/RandomName
example.com/user/1234556778898?id=true
and use it like:
api.get('user/:input', function(req, res) {
if(req.query.id) {
User.find({ '_id': req.params.input }, cb);
} else {
User.find({ 'name': req.params.input }, cb);
}
};
But this seems like bad practice to me, since it leads to a bunch of conditional expressions.
Are there more elegant ways?
I would suggest handling two endpoints. One for getting ALL the users and one for getting a SPECIFC user by ID.
example.com/users
example.com/users/:id
The second endpoint can be used to find a specific user by id.
The first endpoint can be used to find all users, but filters can be applied to this endpoint.
For example: example.com/users?name=RandomName
By doing this, you can very easily create a query in your Node service based on the parameters that are in the URL.
api.get('/users', function(req, res) {
// generate the query object based on URL parameters
var queryObject = {};
for (var key in req.query) {
queryObject[key] = req.query[key];
}
// find the users with the filter applied.
User.find(queryObject, cb);
};
By constructing your endpoints this way, you are following a RESTful API standard which will make it very easy for others to understand your code and your API. In addition, you are constructing an adaptable API as you can now filter your users by any field by adding the field as a parameter to the URL.
See this response for more information on when to use path parameters vs URL parameters.
I'd like to build an JAXRS method, allows to specify a lumps of operations to perform on my resources.
For example, I have got a resource, Book. On this resource I deploy these methods: create, delete and update. If only, I set these method on Book resource, when my client needs to perform a lot of updates over a lot of book resources, he is going to have to send a lot of requests for each Book resource.
I'd like to deploy a JAXRS operation offers this functionality. For example: a batch method that receives which operation must to perform.
However, I've no idea how to do this.
I'm using JAXRS-2.0.
Thanks for all.
This isn't a jaxrs-specific question by any means but more of a design question. The JAXRS implementation of this is relatively straight-forward.
From what you're describing, I would create a batch POST endpoint that has a specific data structure for the job. You accept a JSON blob (serialized form of this data structure) instructing the endpoint what to do. Once the endpoint receives the data, a thread is spawned off in the background to "do work" and an identifier of "the job" is returned (assuming this is a long-running task possibly). If you do return a "job id", you should also have an endpoint to "get status" which will return the current status and presumably some sort of output once the job has completed.
Example data structure you may want to accept as JSON:
{
job_name: "Some job name",
requests: [
{
tasks: ["UPDATE"],
book_id: 122,
data: {
pages: 155,
last_published_date: "2015-09-01"
}
},
{
tasks: ["DELETE"],
book_id: 957
}
]
}
Your actual endpoint in JAXRS may look something like this:
#POST
#Path(value = "batch")
#Produces(MediaType.APPLICATION_JSON)
public String batchRequest(String batchRequest) {
BatchRequest requestObj = null;
if (!StringUtils.isBlank(batchRequest))
{
Gson gson = new Gson();
requestObj = gson.fromJson(batchRequest, BatchRequest.class);
// This is the class that would possibly spawn off a thread
// and return some sort of details about the job
JobDetails jobDetails = JobRunner.run(requestObj);
return gson.toJson(jobDetails);
}
return "error";
}
Hopefully this makes sense and helps out. Please feel free to reply with additional questions and I'll try to help as much as I can!
I'm whondering, wheather there is an easy way to pupolate doctrine entities from request objects. I'm building a RESTful API with fos/rest-bundle, so I dont need forms.
Do you know a good way to do this, in a very easy and short way?
// POST /api/products
public function postProductsAction(Request $request)
{
$product = new Product();
}
In addition, I'm whondering wheather its possible to inject instances of entities directly in the controller with post requests.
// PUT /api/product/1
// I need this functionality for post requests too
public function putProductAction(Product $product)
{
return $product; // { "id" : "1", "name" : "foo" }
}
Greetings,
--marc
What you need is the most common goal of every REST API. And the best way to do this is to use a serializer, in addition to forms (even if you would prefere to not use forms).
I advise you to read for example this tutorial writen by William Durand. It explains every points very well and uses the JMSSerializerBundle to convert entities through the API.
Laravel 4: In the context of consume-your-own-api, my XyzController uses my custom InternalAPiDispatcher class to create a Request object, push it onto a stack (per this consideration), then dispatch the Route:
class InternalApiDispatcher {
// ...
public function dispatch($resource, $method)
{
$this->request = \Request::create($this->apiBaseUrl . '/' . $resource, $method);
$this->addRequestToStack($this->request);
return \Route::dispatch($this->request);
}
To start with, I'm working on a basic GET for a collection, and would like the Response content to be in the format of an Eloquent model, or whatever is ready to be passed to a View (perhaps a repository thingy later on when I get more advanced). It seems inefficient to have the framework create a json response and then I decode it back into something else to display it in a view. What is a simple/efficient/elegant way to direct the Request to return the Response in the format I desire wherever I am in my code?
Also, I've looked at this post a lot, and although I'm handling query string stuff in the BaseContorller (thanks to this answer to my previous question) it all seems to be getting far too convoluted and I feel I'm getting lost in the trees.
EDIT: could the following be relevant (from laravel.com/docs/templates)?
"By specifying the layout property on the controller, the view specified will be created for you and will be the assumed response that should be returned from actions."
Feel free to mark this as OT if you like, but I'm going to suggest that you might want to reconsider your problem in a different light.
If you are "consuming your own API", which is delivered over HTTP, then you should stick to that method of consumption.
For all that it might seem weird, the upside is that you could actually replace that part of your application with some other server altogether. You could run different parts of your app on different boxes, you could rewrite the HTTP part completely, etc, etc. All the benefits of "web scale".
The route you're going down is coupling the publisher and the subscriber. Now, since they are both you, or more accurately your single app, this is not necessarily a bad thing. But if you want the benefits of being able to access your own "stuff" without resorting to HTTP (or at least "HTTP-like") requests, then I wouldn't bother with faking it. You'd be better off defining a different internal non-web Service API, and calling that.
This Service could be the basis of your "web api", and in fact the whole HTTP part could probably be a fairly thin controller layer on top of the core service.
It's not a million miles away from where you are now, but instead of taking something that is meant to output HTTP requests and mangling it, make something that can output objects, and wrap that for HTTP.
Here is how I solved the problem so that there is no json encoding or decoding on an internal request to my API. This solution also demonstrates use of route model binding on the API layer, and use of a repository by the API layer as well. This is all working nicely for me.
Routes:
Route::get('user/{id}/thing', array(
'uses' => 'path\to\Namespace\UserController#thing',
'as' => 'user.thing'));
//...
Route::group(['prefix' => 'api/v1'], function()
{
Route::model('thing', 'Namespace\Thing');
Route::model('user', 'Namespace\User');
Route::get('user/{user}/thing', [
'uses' => 'path\to\api\Namespace\UserController#thing',
'as' => 'api.user.thing']);
//...
Controllers:
UI: UserController#thing
public function thing()
{
$data = $this->dispatcher->dispatch('GET', “api/v1/user/1/thing”)
->getOriginalContent(); // dispatcher also sets config flag...
// use $data in a view;
}
API: UserController#thing
public function thing($user)
{
$rspns = $this->repo->thing($user);
if ($this->isInternalCall()) { // refs config flag
return $rspns;
}
return Response::json([
'error' => false,
'thing' => $rspns->toArray()
], 200);
Repo:
public function thing($user)
{
return $user->thing;
}
Here is how I achieved it in Laravel 5.1. It requires some fundamental changes to the controllers to work.
Instead of outputting response with return response()->make($data), do return $data.
This allows the controller methods to be called from other controllers with App::make('apicontroller')->methodname(). The return will be object/array and not a JSON.
To do processing for the external API, your existing routing stays the same. You probably need a middleware to do some massaging to the response. Here is a basic example that camel cases key names for the JSON.
<?php
namespace App\Http\Middleware;
use Closure;
class ResponseFormer
{
public function handle($request, Closure $next)
{
$response = $next($request);
if($response->headers->get('content-type') == 'application/json')
{
if (is_array($response->original)) {
$response->setContent(camelCaseKeys($response->original));
}
else if (is_object($response->original)) {
//laravel orm returns objects, it is a huge time saver to handle the case here
$response->setContent(camelCaseKeys($response->original->toArray()));
}
}
return $response;
}
}