I want my ZF2 Application to import data from many different REST or SOAP Services, which may use different authentication types and so on.
Now I'm basically looking for a structure / architecture how to implement this, maybe some design patterns or ready to use modules if they exist.
Every information could help. I'm also thankful for API docs or tutorials that you provide.
But my main question is: How should be the structure for this kind of "importer"
My Application:
Based on Zend Skeleton Application
Using Doctrine 2
Trying to use all ZF2 Best Practices I can find
Consists of many modules, entities and complex associations in some cases
Entities that I want to import are already working (crud operations, validation, ...)
Apis that I want to use:
Usually E-Commerce stuff, like products, orders, stock keeping
Magento Api (Thinking of Rest)
Shopware and other important Webshops
Ebay Stores
Amazon (I think is going to be the hardest one)
Must have functionality:
I want the api URLs and authentication data to be configurable in my app with doctrine entities
The "Api" Entity should be associated to my "Shop" Entity. Orders or Products that I import or create directly in my App are also associated to my Shop entities. So every Shop/Ebay-Store/Amazon-Store is a "Shop" in my Application. This is already the part I've done.
For example product import should be done directly from my app frontend, I'm thinking of retrieving the api data first and then import them incremtally / step for step
I don't want fat controllers that transform the data into doctrine entities and save them one by one. This way complex associations would become very hard to maintain.
Need a good approach for data transformation and hydration to doctrine entities. Because the data I retrieve from api will usually not have the same structure as my entities. Maybe an attribute that's a property of the "Product" entity in foreign app is excluded into an associated entity in my own application.
Many modules in my application will have entities that should be importable from these apis, so I need a central component that does the job
How would be the best approach for this? I'm not asking for a complete solution, but ideas that fit these requirements.
The Zend HTTP client and its relatives (like Zend OAuth) provides most of the functionality that you need to implement fetching the data from the services.
You can then persist the response in any number of ways, but a schema-less database like Mongo DB makes saving dynamic data much easier. If you are stuck using a relational DB like MySQL then you can either setup an EAV database, or use dynamically generated tables.
Related
I have a question regarding how to structure my controllers.
Let's assume I have a have a system that has stores and products and stores have products.
so the way to get stores is /api/stores/{id} and the way to get a product is /api/product/{id} or /api/stores/{id}/product/{id}.
How would you structure the Controllers? should I have a controller for products and controller for stores and every route that starts with stores in it should reside in StoresController otherwise in ProductsController?
This is from Microsoft: Api design guidance
In more complex systems, it can be tempting to provide URIs that enable a client to navigate through several levels of relationships, such as /customers/1/orders/99/products. However, this level of complexity can be difficult to maintain and is inflexible if the relationships between resources change in the future. Instead, try to keep URIs relatively simple. Once an application has a reference to a resource, it should be possible to use this reference to find items related to that resource. The preceding query can be replaced with the URI /customers/1/orders to find all the orders for customer 1, and then /orders/99/products to find the products in this order.
So this backend will be consumed by an ad-hoc front end application. But will also be integrated by other systems and we will expose API for them.
When designing the rest I see that there is ONE database table (we call it for table A) that can join many other tables, lets say about 10 to 20 other tables.
Now, my strategy would be to build routes in my backend that will "reason" according to the ad-hoc frontend we have.
So if there is a page in the frontend (let's call this page for page1) that requires to get rows from the table A but also fields from let's say 3 other join tables, then I want to create a route in the backend called maybe "page1" which will return rows from table A and also from the other 3 tables.
This is of course an ordinary way to build a backend. But as it will also be used by other systems then somebody could argue that these systems maybe don't have any need for the route "page1". Their frontend will maybe never build a "page1".
So according to people here, it would better to build the API more agnostically. And instead of creating the route "page1" I should build it according to "hateoas". And if I understand that principle, instead of letting my ad-hoc frontend to request the resource "page1" it would request "pageForTableA". And then, the resource "pageForTableA" should return which are the possible table to be joined.
In this case, for my frontend's page1, I would need to make 4 subsequent request to the server, instead of one like I would like to do if there was a "page1" resource in the backend.
What do you think?
I also see a thirt strategy. I don't know if there is a name for this pattern but it would be this way:
A resource in backend that returns only rows from table A. BUT, the route also takes arguments. And the argument is an array with the name of all the other tables someone want to include.
So if frontend calls:
getTableA(array('tableB', 'tableD', 'tableF'))
Then the resource would include/join the tables B, D and F. In short: API resource let's the frontend decide what it want to get delivered.
Which of these 3 strategies are best do you think? Or there is some more that could be taken in consideration?
You need to architect your API in a way that consumers shouldn't know about how the data is stored in the underlying data store.
Furthermore, if you want to allow consumers to decide which fields you want to project in the response, you could give them using some query string format.
BTW, maybe you should avoid re-inventing the wheel. There's a standard called Open Data (OData) which already defines a lot of things like you already require in your API, and since it has been made by Microsoft, it has deep support on .NET.
Check this tutorial (Create an OData v4 Endpoint Using ASP.NET Web API 2.2) to get more in touch with OData.
I have a Client model. When viewing /clients, I want to return a simple serialization of my clients, with just a few aggregate values (lets say total_unbilled and total_owing). When viewing /clients/1, I want to return a full serialization of the client, including all it's nested tasks and expenses. The back-end has already been configured to do this.
I don't want to return the full serialization of all clients when the user views /clients, as there can be a lot of data under potentially hundreds of clients. I'd like to load that extra information only when needed, when the user views a particular client.
What's the best way to handle this use-case, where models can be serialized in multiple ways, using Ember Data? I know it will cache the initial representation of the client, so if the user visits /clients first, it won't ever try to fetch the full serialization of the client if the user then visits /clients/1. Is there a sensible way to override this? Or would I have to have two different Ember Data models client-side (eg. Client and MiniClient)?
Honestly the easiest is to use two different models, or just don't use Ember Data for the mini clients. It sounds like they won't be used for much more than just an info.
I'd possibly just do POJOs for the mini client, and Ember Data for the full client (since caching would be most useful at that point) Ember without Ember Data
Say I have a relational database with 100+ tables. Each table models some sort of entity (person, address, vehicle, dog, etc etc). Say I also have a restful API and a bunch of people who want to POST data into this database. Many times this data comes in as an XML package or POST data from a web form or something of that nature. Sometimes we need to post to all the tables of the database, sometimes most, sometimes some, sometimes one.
Now requiring our clients to post clumps of multi resource data into a 100+ table persistence via the restful way of
POST /person
POST /email
POST /vehicle
POST /insurance
is insane! So we could have a resource instead that is
POST /auto-record
{ post body of key values for all the tables needed to make an 'auto-record' }
and it would be connected to some sort of business logic that knows to make inserts into the many tables of the database needed. Okay great. But now that I'm thinking about it, does this design abide by the open/closed principle? If we ever needed to update/add/remove to what an 'auto-record' is then we screw up our clients.
How can restful api's deal with resource groupings? Or does it simply not? Are there alternatives?
You can implement more versions of your RESTful API resource /auto-record. For now modify your resource URI to /v1/auto-record. When there will be a feature change request, you will simply provide your customers with a new resource /v2/auto-record. Old functionality will be preserved at /v1/auto-record and new users will have their needed functionality at v2/auto-record.
i am a rails programmer who is on to his 3rd project now (new of course).I am looking for an answer to a general question about Restful architecture. I am sure i am doing something that has a good established answer already.
In restful approach we expose resources but some times this approach feels a little Non user friendly. For example i can expose a product via a show method and then i have another resource called sales that i can expose via product/:id/sales show template to show all sales for a product. But i am taking the user through an extra click here. The ideal will be to show product and all its associated sales on one page itself. But that is a violation of the Restful rule.
I just wanted to ask that are these rules generally broken to make the site user friendly? Being a new comer i dont want to adopt ways that are non ideal so i thought i should ask this question.
Thanks in advance.
Adding in the sales for a particular product would not be breaking any constraints from the RESTful architecture. You have the product ID in the HTTP request so you can just also get the sales for that product. Your separation of concerns should not be affected and you don't need to store a state to get this information. Just extend the model that you return with the view.
It seems like you are more concerned with straying from the convention over configuration that Rails promotes. This extension means that your model will not correlate with only one table in your database, but that is fine. The conventions are meant to reduce the configuration work that you need to do, not restrict your functionality.