I'm looking at the documentation for the Play framework and it appears as though SQL is being written in the framework.
Coming from Rails, I know this as a bad practice. Mainly because developers require the need to switch databases as they scale.
What are the practices in Play to allow for developers to implement conventions and to work with databases without having to hard code SQL?
One of the feature/defects (depending on who you ask) of Play is the there is no ORM. (An ORM is an object-relational mapper/mapping; it is the part of Rails, Django, etc. that writes your SQL for you.)
Pro-ORM: You don't have to write any SQL.
Ease-of-use: Some developers unused to SQL will find this easier.
Code resuse: Tables are usually based on your classes; there is less duplication of code.
Portability: ORMs try smooth over any differences between DMBS vendors.
No-ORM: You get to write your own SQL, and not rely on unseen ORM (black)magic.
Performance: I work for a company which produces high-traffic web applications. With millions of visitors, you need to know exactly what queries you are running, and exactly what indicies you are using. Otherwise, the code that worked so well in dev will crash production.
Flexibility: ORMs often do not have the full range of expression that a domain-specific language like SQL does. Some more complex sub-selects and aggregation queries will be difficult/impossible to write with ORMs.
While you may think "developers require the need to switch databases as they scale", if you scale enough to change your DBMS, changing query syntax will be the least of your scalability issues. Often, the queries themselves will have to be rewritten to use sharding, etc., at which point the ORM is dead.
It is a tradeoff; one that in my experience has often favored no ORM.
See the Anorm page for Play's justification of this decision:
You don’t need another DSL to access relational databases
SQL is already the best DSL for accessing relational databases. We don’t need to invent something new.
...
Play developers will typically write their own SQL (much the same way they will write in other languages, like HTML), use Anorm, and follow common SQL conventions.
If portability is a requirement, use only ANSI SQL (no vendor-specific featues). It is generally well supported.
EDIT: Or if you are really open-minded, you might have a look at NoSQL databases, like Mongo. They are inherently object-based, so object-oriented Ruby/Python/Scala can be used as the API naturally.
In addition to Paul Draper's excellent answer, this post is meant to tell you about what Play developers usually do in practice.
TL;DR: use Slick
Play is less opinionated than Rails and gives the user many more choices for their data storage backend. Many people use Play as a web layer for very complex existing backend systems. Many people use Play with a NoSQL storage backend (e.g. MongoDB). Then there's also people using Play for traditional web-service-with-SQL applications. Generalizing a bit too much, one can recognize two people using Play with relational databases.
"Traditional web developers"
They are used to standard Java technologies or are part of an organization that uses them. The Java Persistence API and its implementations (Hibernate, EclipseLink, etc...) are their ORM. You can do so too. There are also appear to be Scala ORMs, but I'm less familiar with those.
Note that Java/Scala ORMs are still different ORMs in style when compared to Rails' ActiveRecord. Ruby is a dynamic language that allows/promotes loads of monkey patching and method_missing stuff, so there is MyObject.finder_that_I_didnt_necessarily_have_to_write_myself(). This style of ORM is called the active record pattern. This style is impossible to accomplish in pure Java and discouraged in Scala (as it violates type safety), so you have to get used to writing a more traditional style using service layers and data access objects.
"Scala web developers"
Many Scala people think that ORMs are a bad abstraction for database access. They also agree that using raw SQL is a bad idea, and that an abstraction is still in order. Luckily Scala is an expressive compiled language, and people have found a way to abstract database access that does not rely on the object oriented language paradigm as ORMs do, but mostly on the functional language paradigm. This is quite a similar idea to the LINQ query language Microsoft has made for their .NET framework languages. The core idea is that you don't have an ORM to perform query magic, nor write queries in SQL, but write them in Scala itself. The advantages of this approach are twofold:
You get a more fine grained control over what your queries actually execute when compared to ORMs.
Queries are checked for validity by the Scala compiler, so you don't get runtime errors for invalid SQL you wrote yourself. If it is valid Scala, it is translated to a valid SQL statement for you.
Two major libraries exist for accomplishing this. The first is Squeryl. The second is Slick. Slick appears to be the most popular one, and there are some examples floating around the web that show how you are supposed to make it work with Play. Also check out this video that serves as an introduction to Slick and which compares it to the ORM approach.
FWIW, I wrote a short summary of the state of database access in Scala some months ago. I personally found that using Slick in combination with its plain SQL querying capabilities to be quite convenient.
Related
I searched over .... I see many advantages, but it seems that all the advantages comes from a comparison over in-line SQL. I know in-line SQL is bad. But why compare with a bad one to show the other better?
If stored procedures are used (possibly exclusively), it seems none of the advantages still exists. Stored procedures definitely provide performance advantages in terms of security, performance (If a ORM can outrun a stored procedure, then the stored procedure is badly written) and a well written stored procedure is an automatic repository (pattern). Stored procedures can definitely provide better transaction and transaction isolation control.
I really appreciate an answer -- how ORM is better over a well architected application using stored procedures.
--- Thanks for all the answers that I receive so far ... It seems that the advantages still come from comparing using ORM's "dynamically generated SQL" with using "statically written in-line SQL" in the code. Yes, it has advantages. But it is not he question.
The question is better stated as the following:
If you consider having the stored procedures to implement your business logic (SPs can be written very advanced, and also very efficiently), in the Application code (.NET, JAVA), you have a very thin layer wrapper of the stored procedures organized by business need. My question is how ORM out-perform this architecture (Of course a well designed one).
ORM Tools make possible to develop abstraction layer between database and the model in the OO environment. The main advantage of this layer is that the developers who are not familiar with SQL can work with the model.
I have been seeking a good answer myself. Here is what I feel makes the difference:
1) ORM increases the developer productivity - mapping domain class to database is easier.
2) Stored Procs can potentially contain business logic - it is difficult to test these. This is mainly because of lack of tools/mocking framework.
3) ORM frameworks are tested ones which give you features like caching out of the box - no need to reinvent the wheel - and in most applications I've seen which do not use any ORM feature end up writing in-house Data Layer which ORM offers out of the box.
That being said - ORM does add some overhead as well, and it requires the developers to be aware of a new platform - writing efficient mapping comes with practise so there is a learning curve.
In the modern day setup, network bandwidth isn't as precious as rapid development and good quality (well tested) code. I guess this makes ORM well suited for database driven apps.
An ORM is a tool that can be used to build what you call a "well architected system". The idea is that when you are developing in a non-Relational language, there will be an impedance mismatch between the relational operation set provided by SQL/Stored Procedures and the language that you are using to build the rest of your application.
For developers using an object-oriented language (whether it is C++, C#, or Java) there are many considerations when mapping a complex relational schema into a rich Domain Model. It is certainly possible to perform all of this mapping in your own code, but as your interactions in this "no-man's-land" between OO and Relational paradigms grow more complex the more useful an ORM engine and associated tooling can be.
Some considerations as you plan out your mapping layer:
Do you need manage single-table or multi-table inheritance?
Do you want to leverage lazy loading?
Do you want to manually keep classes and tables synchronized or are you planning on using a tool to generate per-table classes (such as with a DataSet)?
Another consideration, especially when working in a team, is that when relational to domain layer mapping is performed by hand, there can be a great deal of variation in the way developers write the mapping. This can lead to inconsistencies, overlapping, and gaps that are difficult to detect. The selection of an ORM (especially a well known / solidly established ORM) can have an enormous (hopefully positive) impact on the solution and the pre-existing community surrounding that ORM will shape how you conceive of the mapping layer (you will find that there are significant cultural differences between Spring.NET and Entity Framework users, for instance).
Does an ORM make a good architecture? No. Are there systems whose architectures would be better off with an ORM? definitely. Are there projects that have been crippled by the unnecessary addition of an ORM? I'm guessing that there are many.
I suggest approaching this question from a different angle, and apply it to the specific application you are working on. Do you have any pain points by using SQL and/or Stored Procedures that an ORM might solve? Do you see any risks or have any concerns over problems that the introduction of an ORM might cause? Only by weighing the answers to these questions will you be able to determine if an ORM is a good fit for any given solution.
I am trying to work out with ORM tool to move over to and have narrowed it down to two candidates.
nHibernate or LLBLGen Pro
Please can you guys give me pros and cons in using both these tools especially if you have experience in both. I am not really interested in any other tools but am wanting some heads up so I can decide which tool to spend time learning....
I already know that one is free and one isn't, I also know that nHibernate might take some learning....
Many thanks, Richard
I have used both. At first I was sold on nHibernate and refused to try anything else even though I knew about other options.
With LLBLGen Pro, I was skeptical, but soon saw the advantages as well. I have not totaly abandoned nHibernate. I will continue to keep int in my "box of tools". I have found LLBLGen useful in some cases especially when interacting with a database that already exists and you have no choice of re-designing it. It takes less than an hour (depending on size of database of course) to generate my LLBLGen Entity Objects from the database, as opposed to having to code all of it manually with nHibernate, AND do the mappings. nHibernate is missing a nice graphical interface to create the mappings. This fact becomes even more important when the database is massive with thousands of tables that you need to potentially access in your application.
Although LLBLGen is more of a Data Access Layer generator (And I am not normally a fan of DAL generators), it has a lot of features a "true ORM" tool would have. In my opinion it has the best of both worlds. Once you start working with it you start to realize that it is very flexible and extendable. One part I like a lot is that it is possible for me to create partial classes for the generated entity objects, where I can code in my business logic, as well as validation.
The code generation is templated so you have full control over the code it generates. With nHibernate I find myself writing some of the same kind of code over and over again. With LLBLGen I can generate it and get to focus on business logic and issues quicker.
For someone who is just starting to use ORM type tools, I really recommend to start with LLBLGen, because nHibernate can be overwhelming. And in the end you will have achieved the same result (More or less).
Edit #1: LLBLGen now also has 100% support for LINQ. (So if you like LINQ to SQL for that reason) further LLBLGen can support many databases, where LINQ to SQL is only for Microsoft SQL Database.
Edit #2:
According to Graviton you can use CodeSmith to do some of the code generating for you for nHibernate. That is really cool, but for a newcomer to ORM I would still recommend LLBLGen. To me that is adding more dependencies where LLBLGen has it all in one package. Also like I said before the learning curve is so much less steep and you will get the same benefits, which will also help you ease in to nHibernate if you ever decide to go there.
The major difference is that LLBLGen is a code generator, while NHibernate is a "true" ORM library.
LLBLGen advantages:
Easy to use model designer. Can import your existing database schema
Fully typed object model and query language
LLBLGen disadvantages:
You need the designer application to change your model
Not free
Can bloat your code because a lot of code is generated
NHibernate advantages:
No designer application needed. Only code
Widely used (based on the most popular Java ORM, Hibernate)
Very powerful for mapping any data model you can imagine
Open source
NHibernate disadvantages:
Hard to learn
Not as strongly typed as one would like (especially queries)
Of course, this is just my personal point of view...
I typed up a fairly long answer before realizing this was a somewhat old question. Oh well. It's still very relevant.
You have narrowed your list to the two best candidates for an ORM in the .NET world. I have limited experience with either, but I've read extensively about the pros and cons of both. They really serve somewhat different needs in different ways.
In the upcoming LLBLGen Pro 3.0, Frans Bouma has talked about adding features to generate NHibernate mappings. So, it's not even necessarily an either/or decision.
If you want to do "class first" design (as opposed to "database first" design), NHibernate is pretty much your best and only option right now (neither LLBLGen Pro nor Entity Framework support this mode, although it sounds like Entity Framework is improving it's support in the next version).
NHibernate and LLBLGen Pro both work hard to work well with legacy databases which you can not change and have to live with. That is their common strength. They both also work with Linq. They both also support some amount of graphical modeling, although LLBLGen Pro is far superior in this regard (ActiveWriter for NHibernate feels like the LinqToSql designer in Visual Studio, but it's not really as feature rich).
LLBLGen Pro has much stronger code generation abilities, but too much code generation can lead to compromised testability and maintainability (one small tweak can cause massive amounts of code to need retesting).
While NHibernate wants to help you work through fairly complex object/relational mapping scenarios like class inheritance, LLBLGen Pro is really just exposing your database as a data layer and business objects in a very quick way.
If you can purchase LLBLGen Pro and have some time, I would try both and see which one better meets your needs. Learning both ORMs is good for your resume in any case.
So, in the end, I would say it's situational. The cost of NHibernate and its lack of serious flaws make a pretty compelling case in the majority of situations.
The new version of LLBLGen Pro (3.0) allows you to generate code for NHibernate, so don't have to choose :). It also allows you to split up your entities into different domains.
I still prefer the LLBLGen pro runtime though, the LINQ interpreter is more complete and it has better change tracking of fields.
Unfortunately there's not many new features in the new LLBLGen Pro 3.0 runtime, as the creator first wanted to focus more on tooling than improving the existing framework.
I've used nHibernate, LLBLGen Pro, a custom data layer from my consulting company, the Enterprise Library, and LINQ. LLBLGen is by far my favorite and it allows writing one business layer that can talk to different types of databases using the same code providing database independence! Another incredible feature is it allows multiple connections to different databases. This is very useful when at a large company and one system is written in Sql Server and the other you have to interface with is in Oracle.
LLBLGen Pro is an amazing product backed up by Frans who is very active and works hard to fix issues. LLBLGen is like PhotoShop, it is an incredible tool and that can do amazing effects in the hands of someone who knows how to use. And like any tool that saves lots of time, it takes a week or two to learn how to use it, but will save months later on your project.
Not only did it speed up the DAL generation side of my app, it is also easy to create queries in the Business layer and send to the presentation layer. It made it easy to create an enterprise class application.
If one really wants to use nHibernate, start with LLBLGen Pro and generate the nHibernate code. If later on your department decides to switch from nHibernate to LINQ, you are covered. Want to switch from Sql Server to Oracle? This is possible and relatively easy with LLBLGen whereas with manually coded nHibernate code, you have to rewrite everything which is almost impossible to cost justify.
Frans was also available and answered some of my questions.
Don't forget one of the greatest plus point of Hibernate: HQL. With HQL, your SQL skill is not wasted. And Hibernate provides very nice, seamless support for native query as well.
If you have some weird, out-of-standard database, it's almost certain that you need your SQL skill at some point, and good luck with LLBL!
For me it boils down to database centric (LLBLGen Pro) vs. domain model centric (NHibernate).
Since I'm a DDD/OO guy, the choice has always been very easy for me, but I do see why LLBLGen Pro is popular.
We use LLBLGen at work, and it's reviled -- namely because we have multiple similar schemas, but you need to have a different DLL/Class library for each schema, meaning that it becomes annoying to write code that can target any schema.
Of course, that's an unusual environment, so it may not apply to you.
I am really torn right now between using O/R mappers or just sticking to traditional data access. For some reason, every time I bring up O/R mappers, fellow developers cringe and speak about performance issues or how they're just bad in general. What am I missing here? I'm looking at LINQ to SQL and Microsoft Entity Framework. Is there any basis to any of these claims? What kind of things do I have to compromise if I want to use an O/R mapper. Thanks.
This will seem like an unrelated answer at first, but: one of my side interests is WWII-era fighter planes. All of the combatant nations (US, Great Britain, Germany, USSR, Japan etc.) built a bunch of different fighters during the war. Some of them used radial engines (P47, Corsair, FW-190, Zero); some used inline liquid-cooled engines (Bf-109, Mustang, Yak-7, Spitfire); and some used two engines instead of one (P38, Do-335). Some used machine guns, some used cannons, and some used both. Some were even made out of plywood, if you can imagine.
In the end, they all went really really fast, and in the hands of a competent, experienced pilot, they would shoot your rookie ass down in a heartbeat. I don't imagine many pilots flew around thinking "oh, that idiot is flying something with a radial engine - I don't have to worry about him at all". Everyone understood that there were many different ways of achieving the ultimate goal, and each approach had its particular advantages and disadvantages, depending on the circumstances.
The debate between ORM and traditional data access is just like this, and it behooves any programmer to become competent with both approaches, and choose the option that is right for the job at hand.
I struggled with this decision for a long time. I think I was hesitant for two primary reasons. First, O/R mappers represented a lack of control over what was happening in a critical part of the app and, second, because so many times I've been disappointed by solutions that are awesome for the 90% case but miserable for the last 10%. Everything works for select * from authors, of course, but when you crank up the complexity and have a high-volume, critical system and your career is on the line, you feel you need to have complete control to tune every query pattern and byte over the wire. Most developers, including me, get frustrated the first time the tool fails us, and we cannot do what we need to do, or our need deviates from the established pattern supported by the tool. I'll probably get flamed for mentioning specific flaws in tools, so I'll leave it at that.
Fortunately, Anderson Imes finally convinced me to try CodeSmith with the netTiers template. (No, I don't work for them.) After more than a year using this, I can't believe we didn't do it sooner. My team uses Visual Studio DB Pro, and on every check-in our continuous integration build drops out a new set of data access layer assemblies. This handles all the common, low risk stuff automatically, yet we can still write custom sprocs for the tricky bits and have them included as methods on the generated classes, and we can customize the templates for the generated code as well. I highly recommend this approach. There may be other tools that allow this level of control as well, and there is a newer CodeSmith template called PLINQO that uses LINQ to SQL under the hood. We haven't that yet examined (haven't needed to), but this overall approach has a lot of merit.
Jerry
O/RM tools designed to perform very well in most situations. It will cache entities for you, it will execute queries in bulks, it has a very low level optimised access to objects which is way faster than manually assigning values to properties, they give you a very easy way to incorporate variations of aspect oriented programming using modern technics like interceptors, it will manage entity state for you and help resolve conflicts and many more.
Now cons of this approach usually lies in lack of understanding of how things work on a very low level. Most classic problem is "SELECT N+1" (link).
I've been working with NHibernate for 2.5 years now, and I'm still discovering something new about it almost on a daily basis...
Good. In most cases.
The productivity benefit of using an ORM, will in most case outweigh the loss of control over how the data is accessed.
There are not that many who would avoid C#, in order to program is MSIL or Assembly, although that would give them more control.
The problem that i see with a lot of OR mappers is that you get bloated domain objects, which are usually highly coupled with the rest of your data access framework. Our developers cringe at that as well :) It's just harder to port these object to another data access technology. If you use L2S, you can take a look at the generated code. It looks like a complete mess. NHibernate is probably one of the best at this. Your entities are completely unaware of your data access layer, if you design them right.
It really depends on the situation.
I went from a company that used a tweaked out ORM to a company that did not use a ORM and wrote SQL queries all the time. When I asked about using an ORM to simplify the code, I got that blank look in the face followed by all the negatives of it:
Its High Bloat
you don't have fine control over your queries and execute unnecessary ones
there is a heavy object to table mapping
its not dry code because you have to repeat your self
on an on
Well, after working there for a few weeks, I had noticed that:
we had several queries that were almost identical, and alot of times if there was a bug, only a handful would get fixed
instead of caching common tables queries, we would end up reading a table multiple times.
We were repeating our selves all over the place
We had several levels of skill level, so some queries were not written the most efficiently.
After I pointed most of this out, they wrote a "DBO" because the didn't want to call it an ORM. They decided to write one from scratch instead of tweaking out one.
Also, alot of the arguments come from ignorance against ORM's I feel. Every ORM that I have seen allows for custom queries, and even following the ORM's conventions, you can write very complex and detailed queries and normally are more human readable. Also, they tend to be very DRY, You give them your schema, and they figure the rest out, down to relationship mapping.
Modern ORM's have a lot of tools to help you out, like migration scripts, multiple DB types accessed to the same objects so you can leverage advantages of both NOSQL and SQL DB's. But you have to pick the right ORM for your project if your going to use one.
I first got into ORM mapping and Data Access Layers from reading Rockford Lhotka's book, C# business objects. He's spent years working on a framework for DAL's. While his framework out of the box is quite bloated and in some cases, overkill, he has some excellent ideas. I highly recommend the book for anyone looking at ORM mappers. I was influenced by his book enough to take away a lot of his ideas and build them into my own framework and code generation.
There is no simple answer to this since each ORM provider will have it's own particular pluses and minuses. Some ORM solutions are more flexible than others. The onus is on the developer to understand these before using one.
However, take LinqToSql - if you are sure you are not going to need to switch away from SQL Server then this solves a lot of the common problems seen in ORM mappers. It allows you to easily add stored procedures (as static methods), so you aren't just limited to generated SQL. It uses deferred execution, so that you can chain queries together efficiently. It uses partial classes to allow you to easily add custom logic to generated classes without needing to worry about what happens when you re-generate them. There is also nothing stopping you using LINQ to create your own, abstracted DAL - it just speeds up the process. The main, thing, though is that it alleviates the tedium and time required to create basic CRUD layer.
But there are downsides, too. There will be a tight coupling between your tables and classes, there will be a slight performance drop, you may occasionally generate queries that are not as efficient as you expected. And you are tied in to SQL Server (though some other ORM technlogies are database agnostic).
As I said, the main thing is to be aware of the pros and cons before pinning your colours to a particular methodology.
I need to collect some information about existing ORM solutions.
Please feel free to write about any programming language.
Can you tell about the best ORM framework you ever use and why is it better then others?
I used NHibernate and Entity Framework.
Current stable version of entity framework is very immature. It is too difficult, or impossible to perform common tasks. Testing your code is also difficult since you cannot really separate your entities from your data access classes. But it perfectly integrates with visual studio ide. Setting up is easy and updating all the models from database takes just a few seconds. Upcoming version of EF (4.0) will solve some of this problems.
NHibernate is quite powerful. It supports plain old clr objects, so you can work with simple entities. Configurations provide great control in great detail. Framework capabilities are satisfying and it has a large and active community and good documentation. Setting up and updating entities is a little difficult since you must use generators that looks up your database and generates entities and xml files. It may be tricky to find a generator or a template that exactly fits your needs. But once you set all things up, you will love it.
I found LINQ to SQL to be a pretty straight forward solution. The first time I used it, I'd say I had a basic ORM working within a few hours, and was creating LINQ queries with it.
Microsoft has an additional ORM (Entity Framework), which I've heard is more complex but may be useful for highly complex distributed applications with multiple data sources etc.
Overall I found LINQ to be an easy and fast to use ORM.
I have been looking at Telerik Open Access for last few months, in genernal this ORM has been a pain to work with, it was advertised as having extensive linq support but in reality many of the linq features you would normally expect dont work server side and are performed on the client. I also had problems using multiple conditions in a where clause, see my last question. Here are a few things that i found
No support for views
Unable to map more than one entity to the same table
Inheritance and Interface support requires you to make changes to you database schema
No visual designer like LINQ to SQL and Entity Framework
If you want to perform an insert any related entities must be fetched first
Rohan
LINQ2SQL was nice, EF makes sense, but is very complex and SQL Server oriented. NHibernate is special and Telerik OpenAccess (fully .NET / Visual Studio) got a broad set of functionality and professional support.
Since I know the product I can comment on Rohan's concerns:
Existing Views can be used and full Views support is coming up
Mapping more than one entity to the same table "works" for class hierarchies, limitation with reverse mapping exists
Inheritance and Interfacer support "do not require" changes to the the database schema, again limitation with reverse mapping exists though
Visual Designer will come, Forward and Reverse Mapping Wizards allow you already to do anything in an easy way
There is a workaround for the insert issue mentioned and it will be fixed generally
Check out the Telerik site to find happy customers and feel free to use the telerik forums and support resources for any question.
-Peter
Im new to OpenAccess ORM and we are using two products. Reporting and OpenAccess.
I think there are some features that people missed.
OpenAccess uses graphical designers while nHibernate still uses handwritten xml files
OpenAccess is not limited to SQl Server as EF4 and similiar frameworks
using it is easier and the forums are pretty helpful.
With ORM there are multiple possibilities, all depends what you want.
As a real ORM mapper I strongly recomment NHibernate and Fluent NH mappings. You need a lot of research to put together a nice architecture, but then nothing stands in your way. With minimal compromises you get real flexibility.
EF6x (core is not prod.-ready IMHO) is called an ORM, but what it generates is more closer to a DAL. There are some thing's you can't do effectively with EF6. Still, this is my favorite tool for a read-model, while I do combine it with NHibernate (where NH I use for a DDD/write model).
Now to performance - its always pro and cons. If you deep deeper into ORM architecture (see my article: avoid ORM bad habits) then you will find intuitively the ways to make it faster. Here's my another article on how to make EF6x 5x faster (at least for read situations): EF6.x 5x faster
I've been a web developer for some time now, and have recently started learning some functional programming. Like others, I've had some significant trouble apply many of these concepts to my professional work. For me, the primary reason for this is I see a conflict between between FP's goal of remaining stateless seems quite at odds with that fact that most web development work I've done has been heavily tied to databases, which are very data-centric.
One thing that made me a much more productive developer on the OOP side of things was the discovery of object-relational mappers like MyGeneration d00dads for .Net, Class::DBI for perl, ActiveRecord for ruby, etc. This allowed me to stay away from writing insert and select statements all day, and to focus on working with the data easily as objects. Of course, I could still write SQL queries when their power was needed, but otherwise it was abstracted nicely behind the scenes.
Now, turning to functional-programming, it seems like with many of the FP web frameworks like Links require writing a lot of boilerplate sql code, as in this example. Weblocks seems a little better, but it seems to use kind of an OOP model for working with data, and still requires code to be manually written for each table in your database as in this example. I suppose you use some code generation to write these mapping functions, but that seems decidedly un-lisp-like.
(Note I have not looked at Weblocks or Links extremely closely, I may just be misunderstanding how they are used).
So the question is, for the database access portions (which I believe are pretty large) of web application, or other development requiring interface with a sql database we seem to be forced down one of the following paths:
Don't Use Functional Programming
Access Data in an annoying, un-abstracted way that involves manually writing a lot of SQL or SQL-like code ala Links
Force our functional Language into a pseudo-OOP paradigm, thus removing some of the elegance and stability of true functional programming.
Clearly, none of these options seem ideal. Has found a way circumvent these issues? Is there really an even an issue here?
Note: I personally am most familiar with LISP on the FP front, so if you want to give any examples and know multiple FP languages, lisp would probably be the preferred language of choice
PS: For Issues specific to other aspects of web development see this question.
Coming at this from the perspective of a database person, I find that front end developers try too hard to find ways to make databases fit their model rather than consider the most effective ways to use database which are not object oriented or functional but relational and using set-theory. I have seen this generally result in poorly performing code. And further it creates code that is difficult to performance tune.
When considering database access there are three main considerations - data integrity (why all business rules should be enforced at the database level not through the user interface), performance, and security. SQL is written to manage the first two considerations more effectively than any front end language. Because it is specifically designed to do that. The task of a database is far different than the task of a user interface. Is it any wonder that the type of code that is most effective in managing the task is conceptually different?
And databases hold information critical to the survival of a company. Is is any wonder that businesses aren't willing to experiment with new methods when their survival is at stake. Heck many businesses are unwilling to even upgrade to new versions of their existing database. So there is in inherent conservatism in database design. And it is deliberately that way.
I wouldn't try to write T-SQL or use database design concepts to create your user-interface, why would you try to use your interface language and design concepts to access my database? Because you think SQL isn't fancy (or new) enough? Or you don't feel comfortable with it? Just because something doesn't fit the model you feel most comfortable with, doesn't mean it is bad or wrong. It means that it is different and probably different for a legitimate reason. You use a different tool for a different task.
First of all, I would not say that CLOS (Common Lisp Object System) is "pseudo-OO". It is first class OO.
Second, I believe that you should use the paradigm that fits your needs.
You cannot statelessly store data, while a function is flow of data and does not really need state.
If you have several needs intermixed, mix your paradigms. Do not restrict yourself to only using the lower right corner of your toolbox.
You should look at the paper "Out of the Tar Pit" by Ben Moseley and Peter Marks, available here: "Out of the Tar Pit" (Feb. 6, 2006)
It is a modern classic which details a programming paradigm/system called Functional-Relational Programming. While not directly relating to databases, it discusses how to isolate interactions with the outside world (databases, for example) from the functional core of a system.
The paper also discusses how to implement a system where the internal state of the application is defined and modified using a relational algebra, which obviously is related to relational databases.
This paper will not give an an exact answer to how to integrate databases and functional programming, but it will help you design a system to minimize the problem.
Functional languages do not have the goal to remain stateless, they have the goal to make management of state explicit. For instance, in Haskell, you can consider the State monad as the heart of "normal" state and the IO monad a representation of state which must exist outside of the program. Both of these monads allow you to (a) explicitly represent stateful actions and (b) build stateful actions by composing them using referentially transparent tools.
You reference a number of ORMs, which, per their name, abstract databases as sets of objects. Truely, this is not what the information in a relational database represents! Per its name, it represents relational data. SQL is an algebra (language) for handling relationships on a relational data set and is actually quite "functional" itself. I bring this up so as to consider that (a) ORMs are not the only way to map database information, (b) that SQL is actually a pretty nice language for some database designs, and (c) that functional languages often have relational algebra mappings which expose the power of SQL in an idiomatic (and in the case of Haskell, typechecked) fashion.
I would say most lisps are a poor man's functional language. It's fully capable of being used according to modern functional practices, but since it doesn't require them the community is less likely to use them. This leads to a mixture of methods which can be highly useful but certainly obscures how pure functional interfaces can still use databases meaningfully.
I don't think the stateless nature of fp languages is a problem with connecting to databases. Lisp is a non-pure functional programming language so it shouldn't have any problem dealing with state. And pure functional programming languages like Haskell have ways of dealing with input and output that can be applied to using databases.
From your question it seems like your main problem lies in finding a good way to abstract away the record-based data you get back from your database into something that is lisp-y (lisp-ish?) without having to write a lot of SQL code. This seems more like a problem with the tooling/libraries than a problem with the language paradigm. If you want to do pure FP maybe lisp isn't the right language for you. Common lisp seems more about integrating good ideas from oo, fp and other paradigms than about pure fp. Maybe you should be using Erlang or Haskell if you want to go the pure FP route.
I do think the 'pseudo-oo' ideas in lisp have their merit too. You might want to try them out. If they don't fit the way you want to work with your data you could try creating a layer on top of Weblocks that allows you to work with your data the way you want. This might be easier than writing everything yourself.
Disclaimer: I'm not a Lisp expert. I'm mostly interested in programming languages and have been playing with Lisp/CLOS, Scheme, Erlang, Python and a bit of Ruby. In daily programming life I'm still forced to use C#.
If your database doesn't destroy information, then you can work with it in a functional manner consistent with "pure functional" programming values by working in functions of the entire database as a value.
If at time T the database states that "Bob likes Suzie", and you had a function likes which accepted a database and a liker, then so long as you can recover the database at time T you have a pure functional program that involves a database. e.g.
# Start: Time T
likes(db, "Bob")
=> "Suzie"
# Change who bob likes
...
likes(db "Bob")
=> "Alice"
# Recover the database from T
db = getDb(T)
likes(db, "Bob")
=> "Suzie"
To do this you can't ever throw away information you might use (which in all practicality means you cannot throw away information), so your storage needs will increase monotonically. But you can start to work with your database as a linear series of discrete values, where subsequent values are related to the prior ones through transactions.
This is the major idea behind Datomic, for example.
Not at all. There are a genre of databases known as 'Functional Databases', of which Mnesia is perhaps the most accessible example. The basic principle is that functional programming is declarative, so it can be optimised. You can implement a join using List Comprehensions on persistent collections and the query optimiser can automagically work out how to implement the disk access.
Mnesia is written in Erlang and there is at least one web framework (Erlyweb) available for that platform. Erlang is inherently parallel with a shared-nothing threading model, so in certain ways it lends itself to scalable architectures.
A database is the perfect way to keep track of state in a stateless API. If you subscribe to REST, then your goal is to write stateless code that interacts with a datastore (or some other backend) that keeps track of state information in a transparent way so that your client doesn't have to.
The idea of an Object-Relational Mapper, where you import a database record as an object and then modify it, is just as applicable and useful to functional programming as it is to object oriented programming. The one caveat is that functional programming does not modify the object in place, but the database API can allow you to modify the record in place. The control flow of your client would look something like this:
Import the record as an object (the database API can lock the record at this point),
Read the object and branch based on its contents as you like,
Package a new object with your desired modifications,
Pass the new object to the appropriate API call which updates the record on the database.
The database will update the record with your changes. Pure functional programming might disallow reassigning variables within the scope of your program, but your database API can still allow in-place updates.
I'm most comfortable with Haskell. The most prominent Haskell web framework (comparable to Rails and Django) is called Yesod. It seems to have a pretty cool, type-safe, multi-backend ORM. Have a look at the Persistance chapter in their book.
Databases and Functional Programming can be fused.
for example:
Clojure is a functional programming language based on relational database theory.
Clojure -> DBMS, Super Foxpro
STM -> Transaction，MVCC
Persistent Collections -> db, table, col
hash-map -> indexed data
Watch -> trigger, log
Spec -> constraint
Core API -> SQL, Built-in function
function -> Stored Procedure
Meta Data -> System Table
Note: In the latest spec2, spec is more like RMDB.
see: spec-alpha2 wiki: Schema-and-select
I advocate: Building a relational data model on top of hash-map to achieve a combination of NoSQL and RMDB advantages. This is actually a reverse implementation of posgtresql.
Duck Typing: If it looks like a duck and quacks like a duck, it must be a duck.
If clojure's data model like a RMDB, clojure's facilities like a RMDB and clojure's data manipulation like a RMDB, clojure must be a RMDB.
Clojure is a functional programming language based on relational database theory
Everything is RMDB
Implement relational data model and programming based on hash-map (NoSQL)