Hi everyone I’m new to server-side technologies so maybe this is a bit of a dumb question but after reading dozens of articles and viewing dozens of videos I’m still very confused. This has to do with arquitecture principles of modern apps.
Relational model:
I know that a few years ago the model was to have a database (mostly relational) and a DBMS that enabled the connection between an app and the database.
Question 1: Since we are talking about a relational model some examples of DBMS’s are MySQL or PostgreSQL?
Question 2: What is the process of information exchange? The client-side uses a language like PHP to make a request to the server and then the DBMS transforms the request into SQL and accesses the database? Is the conversion of the PHP into SQL part of DBMS function or another server-side software is needed?
(If someone could provide me summary detailed explanation I would be very thankful)
Non-Relational Models:
Question 2: Nowadays with the rise of NoSQL models does the same concept of DBMS apply? Since these systems allow other querying language other than SQL there should be some piece of software that has this function?
Service Oriented Arquitecture:
Almost every app uses this type of arquitecture. I understand the concept of avoid the creation of too tight software relation between client and server side allowing for multiple use across several platforms. What I don’t understand is what parts constitute a system that is build this way.
Question 3: Does the DBMS provides the API’s that constitute the web services made available?
Web Frameworks:
Last but not least, where do frameworks like Django or Ruby on Rails land on?
Question 4: These are supposed to provide tools to develop everything between the front-end and the database of a SOA system right?
Question 5: I’ve seen a lot of buzz about REST arquitecture. Can you explain me of the querying process happens and what are the software entities involved.
Thank you in advance for any explanation that helps me understating these questions. Please provide some links or any diagrams that you find useful.
EDIT:
I'll tackle your questions individually:
Question 1: Since we are talking about a relational model some examples of DBMS’s are MySQL or PostgreSQL?
Correct. The Database Management System is the suite of software that lets you interact with a particular database technology. The examples you give are correct.
Question 2: What is the process of information exchange? The client-side uses a language like PHP to make a request to the server and then the DBMS transforms the request into SQL and accesses the database? Is the conversion of the PHP into SQL part of DBMS function or another server-side software is needed?
There are many different avenues for this. Typically the API for accessing a database is done via ODBC (Open Database Connectivity). ODBC drivers are available for most (if not all) Relational DB vendors and are all very similar.
A language like PHP could connect to the database via an ODBC connection library (eg http://php.net/manual/en/intro.uodbc.php) which would allow you to send CRUD operations to the DBMS to be executed on the database.
Since most DBMS's use a subset or superset of a SQL standard for querying the database you can either pass this code directly via ODBC or you may use another level of abstraction. A common method is called an ORM (Object Relational Mapper). ORMs (eg SQLAlcmehy for Python: http://www.sqlalchemy.org/) provides an abstraction layer so you're not reliant on writing SQL but instead write queries and database commands in a format more common of your language of choice.
Question 2: Nowadays with the rise of NoSQL models does the same concept of DBMS apply? Since these systems allow other querying language other than SQL there should be some piece of software that has this function?
Same general concept (in that there is a DB Driver that exposes an API that languages can hook into) but there are generally more and differing ways of interacting with DB's now since they have many differing structures. Most NoSQL DBs still have ODBC connectors (eg MongoDB and Hadoop) so the general programming practices still apply whilst connecting with them, but the things you expect the database to do (and their natural query languages) will differ.
This continues to be an evolving space as these technologies evolve.
Question 3: Does the DBMS provides the API’s that constitute the web services made available?
Not sure I'm understanding this question. ODBC and webservices are different. Often webservices sit on top of ODBC if you want to query a database via a web API, but it's one layer of abstraction more than connecting to the DB directly via ODBC.
Last but not least, where do frameworks like Django or Ruby on Rails land on?
Web Frameworks are a way of quickening the development of web applications by trying to stop some of the "reinvent the wheel" things that you commonly do with every web application. They give you the basics, and have a lot of extensions that allow you to implement other common elements of web apps (like a subscription/login system, session management, admin system etc).
These are supposed to provide tools to develop everything between the front-end and the database of a SOA system right?
Both Django and RoR aim to be end-to-end frameworks. They include all the common elements you'll need including an Object Relational Mapper. They don't prescribe which DBMS you have to use, their ORMs can interface with many so that choice is still up to you.
Yes, they are aimed to cover everything from the front end to the DB, including the interaction with and initialization of the structure of the database.
Question 5: I’ve seen a lot of buzz about REST arquitecture. Can you explain me of the querying process happens and what are the software entities involved.
REST stands for Representational State Transfer (quick Wikipedia article: https://en.wikipedia.org/wiki/Representational_state_transfer). In a nutshell the creation of a "RESTful" (REST compliant) web API would mean that you have GET, PUT, POST and DELETE methods to accomplish all your services. REST aligns closely with the HTTP protocol which is why it's been very appropriate for the ever growing concept of Web Apps, it's helped transform the thinking from a Web Page (or set of web pages) into Web Apps.
It's hard to sum up better than the Wikipedia article does, I'd suggest you dive into it.
Hope that's cleared up a few of the questions!
Related
We have a layer based application architecture. It is written in C# and uses the sql objects available in .Net for data access. Some of it is a home built ORM, some is with stored procedures. We have a number of windows services that use this architecture to process data. Scaling and performance have always been issues. A new person on our team is pushing to convert our data access to use rest based data services. This would replace our current data access layer.
I don't think rest is meant for our architecture. I also have concerns about performance. I have to think it will be significantly slower. I don't see how going out of process to effectively a web service and then to the database for CRUD operations is not going to make our performance issues worse. I know rest can lead to performance improvements with caching and further scaling out abilities but that is not being addressed now. It's just a data access replacement with no bells and whistles for now. On top of this the initial implementation will not allow us to use stored procedures. All processing will be table based CRUD operations and any data massaging will be done in the C# code, no set based operations.
I could easily be wrong but I can see a disaster coming and I don't know if I'm right or if I'm a chicken little. Looking for any guidance, advice, case study references on this. Anything that can either help my case or resolve my dread. Thanks.
If all the clients, such as Windows services etc are using stored procedures or direct SQL access, that's tight coupling. Any change in the database schema will mean all clients need to be updated to handle the change. If the schema never changes, it's not an issue. If the schema does change and the people who developed one of the services have left, it's a big issue.
If the database is abstracted away from the clients behind a REST layer, that's loose coupling. Any change in the database schema is irrelevant to the clients. The only thing that needs to change is the data access layer of the REST layer. The client facing endpoints won't change. How those endpoints interact with the database will change.
Essentially, moving from direct SQL to REST is taking a step back from your system design. To paraphrase:
SELECT * FROM ORDERS WHERE NOT PAID
becomes
https://api.com/orders/unpaid
and the returned object is a domain object representing a list of unpaid order objects.
So the clients move away from the implementation (select *) and move towards a domain solution. There is no more tight coupling between clients and database but a loose coupling between the clients and the domain.
Rather than speaking to a database in its own language, the clients now talk the domain language, "get all unpaid orders". They don't care how those orders are stored, or where. They just ask the REST endpoint.
This is all just implementation so far but your understanding of the business will increase as you'll be talking in Domain Driven Design (DDD) terms as the REST endpoints will accept and return domain objects rather than raw SQL.
Is this good for the business? Is it good for developers to understand the business at a domain level? If these answers are "yes", is the cost/benefit ratio of rewriting tons of client code to talk REST positive enough to make the change? Will having a REST/domain interface with the data open up new ways of looking at the data? Will that touch on profits?
The real question is along the lines of, will changing the architecture to be a loosely coupled REST integration that improves understanding of business objects and opens the door to a wider talent pool (potentially more REST coders than SQL gurus?) be worth it in terms of future proofing the business without hitting profits in the short term?
Will thinking of the business in DDD terms be worth the initial hit of moving from SQL to REST? Will that new DDD experience open up new doors in future design?
These are the real questions to ask. Caching, scaling etc are REST implementation issues, only relevant once you've answered the philosophical questions posed above.
Good luck, sounds like an exciting time for you!
I'm looking at the documentation for the Play framework and it appears as though SQL is being written in the framework.
Coming from Rails, I know this as a bad practice. Mainly because developers require the need to switch databases as they scale.
What are the practices in Play to allow for developers to implement conventions and to work with databases without having to hard code SQL?
One of the feature/defects (depending on who you ask) of Play is the there is no ORM. (An ORM is an object-relational mapper/mapping; it is the part of Rails, Django, etc. that writes your SQL for you.)
Pro-ORM: You don't have to write any SQL.
Ease-of-use: Some developers unused to SQL will find this easier.
Code resuse: Tables are usually based on your classes; there is less duplication of code.
Portability: ORMs try smooth over any differences between DMBS vendors.
No-ORM: You get to write your own SQL, and not rely on unseen ORM (black)magic.
Performance: I work for a company which produces high-traffic web applications. With millions of visitors, you need to know exactly what queries you are running, and exactly what indicies you are using. Otherwise, the code that worked so well in dev will crash production.
Flexibility: ORMs often do not have the full range of expression that a domain-specific language like SQL does. Some more complex sub-selects and aggregation queries will be difficult/impossible to write with ORMs.
While you may think "developers require the need to switch databases as they scale", if you scale enough to change your DBMS, changing query syntax will be the least of your scalability issues. Often, the queries themselves will have to be rewritten to use sharding, etc., at which point the ORM is dead.
It is a tradeoff; one that in my experience has often favored no ORM.
See the Anorm page for Play's justification of this decision:
You don’t need another DSL to access relational databases
SQL is already the best DSL for accessing relational databases. We don’t need to invent something new.
...
Play developers will typically write their own SQL (much the same way they will write in other languages, like HTML), use Anorm, and follow common SQL conventions.
If portability is a requirement, use only ANSI SQL (no vendor-specific featues). It is generally well supported.
EDIT: Or if you are really open-minded, you might have a look at NoSQL databases, like Mongo. They are inherently object-based, so object-oriented Ruby/Python/Scala can be used as the API naturally.
In addition to Paul Draper's excellent answer, this post is meant to tell you about what Play developers usually do in practice.
TL;DR: use Slick
Play is less opinionated than Rails and gives the user many more choices for their data storage backend. Many people use Play as a web layer for very complex existing backend systems. Many people use Play with a NoSQL storage backend (e.g. MongoDB). Then there's also people using Play for traditional web-service-with-SQL applications. Generalizing a bit too much, one can recognize two people using Play with relational databases.
"Traditional web developers"
They are used to standard Java technologies or are part of an organization that uses them. The Java Persistence API and its implementations (Hibernate, EclipseLink, etc...) are their ORM. You can do so too. There are also appear to be Scala ORMs, but I'm less familiar with those.
Note that Java/Scala ORMs are still different ORMs in style when compared to Rails' ActiveRecord. Ruby is a dynamic language that allows/promotes loads of monkey patching and method_missing stuff, so there is MyObject.finder_that_I_didnt_necessarily_have_to_write_myself(). This style of ORM is called the active record pattern. This style is impossible to accomplish in pure Java and discouraged in Scala (as it violates type safety), so you have to get used to writing a more traditional style using service layers and data access objects.
"Scala web developers"
Many Scala people think that ORMs are a bad abstraction for database access. They also agree that using raw SQL is a bad idea, and that an abstraction is still in order. Luckily Scala is an expressive compiled language, and people have found a way to abstract database access that does not rely on the object oriented language paradigm as ORMs do, but mostly on the functional language paradigm. This is quite a similar idea to the LINQ query language Microsoft has made for their .NET framework languages. The core idea is that you don't have an ORM to perform query magic, nor write queries in SQL, but write them in Scala itself. The advantages of this approach are twofold:
You get a more fine grained control over what your queries actually execute when compared to ORMs.
Queries are checked for validity by the Scala compiler, so you don't get runtime errors for invalid SQL you wrote yourself. If it is valid Scala, it is translated to a valid SQL statement for you.
Two major libraries exist for accomplishing this. The first is Squeryl. The second is Slick. Slick appears to be the most popular one, and there are some examples floating around the web that show how you are supposed to make it work with Play. Also check out this video that serves as an introduction to Slick and which compares it to the ORM approach.
FWIW, I wrote a short summary of the state of database access in Scala some months ago. I personally found that using Slick in combination with its plain SQL querying capabilities to be quite convenient.
I know there are plenty of threads on when you should/shouldn’t use ORM. But at my level of “expertise” I’m far from qualified to be able to tell whether or not my app would be better off with or without ORM. And I don’t want to wait till I’m proficient enough to figure out whether I should learn ORM or not.
So could you tell me whether or not most of today’s web service apps (and most of web apps in general) are being written using ORM?
PS - I'm learning WCF and Asp.Net Web forms
I guess a lot of webapps (yes, probably most of them) use some orm persistence layer these days, provided they use a relational database. That goes for java (hibernate, jpa), .net, Python, groovy/grails and Rails (ActiveRecord by default).
There are also more and more orm solutions appearing for non-relational data stores.
It certainly depends on the language and application framework you are using, if any.
This spec http://www.w3.org/TR/webdatabase/ says:
This document was on the W3C Recommendation track but specification work has stopped. The specification reached an impasse: all interested implementors have used the same SQL backend (Sqlite), but we need multiple independent implementations to proceed along a standardisation path.
Does this mean that HTML5 database is going away, and for some time we will have a de-facto standard using SQLite, possibly with browser differences? Or has the W3C published a plan of attack for finishing the standard?
According to this article:
[...] we think it is worth explaining our design choices, and why we think IndexedDB is a better solution for the web than Web SQL Database.
In another article, we compare IndexedDB with Web SQL Database, and note that the former provides much syntactic simplicity over the latter. IndexedDB leaves room for a third-party JavaScript library to straddle the underlying primitives with a BTree API, and we look forward to seeing initiatives like BrowserCouch built on top of IndexedDB. Intrepid web developers can even build a SQL API on top of IndexedDB. We’d particularly welcome an implementation of the Web SQL Database API on top of IndexedDB, since we think that this is technically feasible. Starting with a SQL-based API for use with browser primitives wasn’t the right first step, but certainly there’s room for SQL-based APIs on top of IndexedDB.
I'm not personally swayed by the arguments put forth in the article, but it seems clear that (for the time being) Mozilla has decided that Web SQL Database is dead.
Further interesting comments about this article may be found on Hacker News.
My understanding is that this is now called "IndexedDB"
http://www.w3.org/TR/IndexedDB/
Apparently the Firefox team has started implementing this:
http://hacks.mozilla.org/2011/01/indexeddb-in-firefox-4/
I don't know if anyone knows the answer. Mozilla doesn't like the dependence upon SQLite and has decided to go a different way. However, all WebKit based browsers already have it implemented and I don't see them removing it as any websites built to take advantage of the spec would be broken.
This means that at least in certain contexts, mostly within the mobile sphere where most browsers have a webkit implementation, it can still makes sense to use the HTML5 Web SQL spec. I see this as especially true for developers who are looking to create mobile applications using a framework like phonegap.
There are some times where as an application developer you want to provide users with access to data even if they aren't connected to the internet or if the connection is slow and some types of data is just more efficiently stored in a database than in a cookie or JSON cashe. For example, if you have data that has relationships it is much easier and quicker to do a join query to pull the data you need than it is to search a json map.
I don't think the spec is dead, and I actually hope that Mozilla will reverse their stance so that developers can use it to solve problems outside of the mobile webkit world.
Consider the client application that should store its data on remote server. We do not want it to access this data "on-fly", but rather want it to have a copy of this data in local database. So we do not need connection with remote server to use application. Eventually we want to sync local database with remote server. The good example of what I am talking about is Evernote service. This type of applications is very common, for instance, in mobile development, where user is not guaranteed to have permanent Internet connection, bandwidth is limited and traffic can be expensive.
ORM (object relational mapping) solutions generally allows developer to define some intermediate "model" for his business logic data. And then work with it as some object hierarchy in his programming language, having the ability to store it relational database.
Why not to add to an ORM system a feature that will allow automatic synchronization of two databases (client and server), that share the same data model? This would simplify developing the applications of the type I described above. Is there any systems for any platform or language that have this or similar feature implemented?
this links may provide some useful information
http://gwtsandbox.com/?q=node/34
http://www.urielkatz.com/archive/detail/google-gears-orm-v01/
http://www.urielkatz.com/archive/detail/google-gears-orm/
AFAIK, there are no such ORM tools.
It was one of original goals of our team (I'm one of DataObjects.Net developers), but this feature is still not implemented. Likely, we'll start working on database sync this spring, but since almost nothing is done yet, there is no exact deadline for this.
There is at least one Open Source ORM matching your needs, but it is a Delphi ORM.
It is called mORMot, and use JSON in a stateless/RESTless architecture to communicate over GDI messages, named pipes or HTTP/1.1. It is able to connect to any database engine, and embed an optimized SQlite3 engine.
It is a true Client-Server ORM. That is, ORM is not used only for data persistence of objects (like in other implementations), but as part of a global n-Tier, Service Oriented Architecture. This really makes the difference.