Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 9 years ago.
Improve this question
I've been hearing things about NoSQL and that it may eventually become the replacement for SQL DB storage methods due to the fact that DB interaction is often a bottle neck for speed on the web.
So I just have a few questions:
What exactly is it?
How does it work?
Why would it be better than using a SQL Database? And how much better is it?
Is the technology too new to start implementing yet or is it worth taking a look into?
There is no such thing as NoSQL!
NoSQL is a buzzword.
For decades, when people were talking about databases, they meant relational databases. And when people were talking about relational databases, they meant those you control with Edgar F. Codd's Structured Query Language. Storing data in some other way? Madness! Anything else is just flatfiles.
But in the past few years, people started to question this dogma. People wondered if tables with rows and columns are really the only way to represent data. People started thinking and coding, and came up with many new concepts how data could be organized. And they started to create new database systems designed for these new ways of working with data.
The philosophies of all these databases were different. But one thing all these databases had in common, was that the Structured Query Language was no longer a good fit for using them. So each database replaced SQL with their own query languages. And so the term NoSQL was born, as a label for all database technologies which defy the classic relational database model.
So what do NoSQL databases have in common?
Actually, not much.
You often hear phrases like:
NoSQL is scalable!
NoSQL is for BigData!
NoSQL violates ACID!
NoSQL is a glorified key/value store!
Is that true? Well, some of these statements might be true for some databases commonly called NoSQL, but every single one is also false for at least one other. Actually, the only thing NoSQL databases have in common, is that they are databases which do not use SQL. That's it. The only thing that defines them is what sets them apart from each other.
So what sets NoSQL databases apart?
So we made clear that all those databases commonly referred to as NoSQL are too different to evaluate them together. Each of them needs to be evaluated separately to decide if they are a good fit to solve a specific problem. But where do we begin? Thankfully, NoSQL databases can be grouped into certain categories, which are suitable for different use-cases:
Document-oriented
Examples: MongoDB, CouchDB
Strengths: Heterogenous data, working object-oriented, agile development
Their advantage is that they do not require a consistent data structure. They are useful when your requirements and thus your database layout changes constantly, or when you are dealing with datasets which belong together but still look very differently. When you have a lot of tables with two columns called "key" and "value", then these might be worth looking into.
Graph databases
Examples: Neo4j, GiraffeDB.
Strengths: Data Mining
While most NoSQL databases abandon the concept of managing data relations, these databases embrace it even more than those so-called relational databases.
Their focus is at defining data by its relation to other data. When you have a lot of tables with primary keys which are the primary keys of two other tables (and maybe some data describing the relation between them), then these might be something for you.
Key-Value Stores
Examples: Redis, Cassandra, MemcacheDB
Strengths: Fast lookup of values by known keys
They are very simplistic, but that makes them fast and easy to use. When you have no need for stored procedures, constraints, triggers and all those advanced database features and you just want fast storage and retrieval of your data, then those are for you.
Unfortunately they assume that you know exactly what you are looking for. You need the profile of User157641? No problem, will only take microseconds. But what when you want the names of all users who are aged between 16 and 24, have "waffles" as their favorite food and logged in in the last 24 hours? Tough luck. When you don't have a definite and unique key for a specific result, you can't get it out of your K-V store that easily.
Is SQL obsolete?
Some NoSQL proponents claim that their favorite NoSQL database is the new way of doing things, and SQL is a thing of the past.
Are they right?
No, of course they aren't. While there are problems SQL isn't suitable for, it still got its strengths. Lots of data models are simply best represented as a collection of tables which reference each other. Especially because most database programmers were trained for decades to think of data in a relational way, and trying to press this mindset onto a new technology which wasn't made for it rarely ends well.
NoSQL databases aren't a replacement for SQL - they are an alternative.
Most software ecosystems around the different NoSQL databases aren't as mature yet. While there are advances, you still haven't got supplemental tools which are as mature and powerful as those available for popular SQL databases.
Also, there is much more know-how for SQL around. Generations of computer scientists have spent decades of their careers into research focusing on relational databases, and it shows: The literature written about SQL databases and relational data modelling, both practical and theoretical, could fill multiple libraries full of books. How to build a relational database for your data is a topic so well-researched it's hard to find a corner case where there isn't a generally accepted by-the-book best practice.
Most NoSQL databases, on the other hand, are still in their infancy. We are still figuring out the best way to use them.
What exactly is it?
On one hand, a specific system, but it has also become a generic word for a variety of new data storage backends that do not follow the relational DB model.
How does it work?
Each of the systems labelled with the generic name works differently, but the basic idea is to offer better scalability and performance by using DB models that don't support all the functionality of a generic RDBMS, but still enough functionality to be useful. In a way it's like MySQL, which at one time lacked support for transactions but, exactly because of that, managed to outperform other DB systems. If you could write your app in a way that didn't require transactions, it was great.
Why would it be better than using a SQL Database? And how much better is it?
It would be better when your site needs to scale so massively that the best RDBMS running on the best hardware you can afford and optimized as much as possible simply can't keep up with the load. How much better it is depends on the specific use case (lots of update activity combined with lots of joins is very hard on "traditional" RDBMSs) - could well be a factor of 1000 in extreme cases.
Is the technology too new to start implementing yet or is it worth taking a look into?
Depends mainly on what you're trying to achieve. It's certainly mature enough to use. But few applications really need to scale that massively. For most, a traditional RDBMS is sufficient. However, with internet usage becoming more ubiquitous all the time, it's quite likely that applications that do will become more common (though probably not dominant).
Since someone said that my previous post was off-topic, I'll try to compensate :-) NoSQL is not, and never was, intended to be a replacement for more mainstream SQL databases, but a couple of words are in order to get things in the right perspective.
At the very heart of the NoSQL philosophy lies the consideration that, possibly for commercial and portability reasons, SQL engines tend to disregard the tremendous power of the UNIX operating system and its derivatives.
With a filesystem-based database, you can take immediate advantage of the ever-increasing capabilities and power of the underlying operating system, which have been steadily increasing for many years now in accordance with Moore's law. With this approach, many operating-system commands become automatically also "database operators" (think of "ls" "sort", "find" and the other countless UNIX shell utilities).
With this in mind, and a bit of creativity, you can indeed devise a filesystem-based database that is able to overcome the limitations of many common SQL engines, at least for specific usage patterns, which is the whole point behind NoSQL's philosophy, the way I see it.
I run hundreds of web sites and they all use NoSQL to a greater or lesser extent. In fact, they do not host huge amounts of data, but even if some of them did I could probably think of a creative use of NoSQL and the filesystem to overcome any bottlenecks. Something that would likely be more difficult with traditional SQL "jails". I urge you to google for "unix", "manis" and "shaffer" to understand what I mean.
If I recall correctly, it refers to types of databases that don't necessarily follow the relational form. Document databases come to mind, databases without a specific structure, and which don't use SQL as a specific query language.
It's generally better suited to web applications that rely on performance of the database, and don't need more advanced features of Relation Database Engines. For example, a Key->Value store providing a simple query by id interface might be 10-100x faster than the corresponding SQL server implementation, with a lower developer maintenance cost.
One example is this paper for an OLTP Tuple Store, which sacrificed transactions for single threaded processing (no concurrency problem because no concurrency allowed), and kept all data in memory; achieving 10-100x better performance as compared to a similar RDBMS driven system. Basically, it's moving away from the 'One Size Fits All' view of SQL and database systems.
In practice, NoSQL is a database system which supports fast access to large binary objects (docs, jpgs etc) using a key based access strategy. This is a departure from the traditional SQL access which is only good enough for alphanumeric values. Not only the internal storage and access strategy but also the syntax and limitations on the display format restricts the traditional SQL. BLOB implementations of traditional relational databases too suffer from these restrictions.
Behind the scene it is an indirect admission of the failure of the SQL model to support any form of OLTP or support for new dataformats. "Support" means not just store but full access capabilities - programmatic and querywise using the standard model.
Relational enthusiasts were quick to modify the defnition of NoSQL from Not-SQL to Not-Only-SQL to keep SQL still in the picture! This is not good especially when we see that most Java programs today resort to ORM mapping of the underlying relational model. A new concept must have a clearcut definition. Else it will end up like SOA.
The basis of the NoSQL systems lies in the random key - value pair. But this is not new. Traditional database systems like IMS and IDMS did support hashed ramdom keys (without making use of any index) and they still do. In fact IDMS already has a keyword NONSQL where they support SQL access to their older network database which they termed as NONSQL.
It's like Jacuzzi: both a brand and a generic name. It's not just a specific technology, but rather a specific type of technology, in this case referring to large-scale (often sparse) "databases" like Google's BigTable or CouchDB.
NoSQL the actual program appears to be a relational database implemented in awk using flat files on the backend. Though they profess, "NoSQL essentially has no arbitrary limits, and can work where other products can't. For example there is no limit on data field size, the number of columns, or file size" , I don't think it is the large scale database of the future.
As Joel says, massively scalable databases like BigTable or HBase, are much more interesting. GQL is the query language associated with BigTable and App Engine. It's largely SQL tweaked to avoid features Google considers bottle-necks (like joins). However, I haven't heard this referred to as "NoSQL" before.
NoSQL is a database system which doesn't use string based SQL queries to fetch data.
Instead you build queries using an API they will provide, for example Amazon DynamoDB is a good example of a NoSQL database.
NoSQL databases are better for large applications where scalability is important.
Does NoSQL mean non-relational database?
Yes, NoSQL is different from RDBMS and OLAP. It uses looser consistency models than traditional relational databases.
Consistency models are used in distributed systems like distributed shared memory systems or distributed data store.
How it works internally?
NoSQL database systems are often highly optimized for retrieval and appending operations and often offer little functionality beyond record storage (e.g. key-value stores). The reduced run-time flexibility compared to full SQL systems is compensated by marked gains in scalability and performance for certain data models.
It can work on Structured and Unstructured Data. It uses Collections instead of Tables
How do you query such "database"?
Watch SQL vs NoSQL: Battle of the Backends; it explains it all.
I'm building a web app that requires me to store metadata about files, approximately 15-20 "characteristics" for each file, including some shared ones (i.e. user1 & user2 should have access).
Would you recommend using a relational database for this? or is one of the newer more scalable noSQL databases a better option?
It should be something that scales quickly - and allows us to read and write fast.
Not sure how that would work with a relational DB in terms of performance (say im trying to find all the files that are owned by user1 and shared to user2 that have a certain property - I would essentially have to join 3-4 tables together... which is probably bad for performance?!)
Thanks for your feedback!
I don't think JOINing 3 or 4 tables would cause bad performance. If you are considering open-source relational solutions, I would suggest PostgreSQL, which is the richest SQL implementation currently. But MySQL will work, too, or even SQLite. They all have decent performance.
On the other hand, if the metadata that you need to store will expand in the future, a schema-based database will be a hassle. In that case I would instead suggest a schema-less (aka document-based, NoSQL, etc) database, like the open-source MongoDB. With indexes, it will also have excellent query performance. CouchDB is a richer implementation, but they don't pay as much attention to speed.
I think a relational database is a good fit for this. NoSQL databases typically don't allow easy and flexible querying. That is a strength of good old SQL databases.
Storing documents and some info for them isn't the strength of SQL databases.
I wouldn't choose MySQL, because of its license (or rather that of its data providers), and because you cannot say what Oracle is going to do with it in the future.
You are looking for a NoSQL database that is optimized for storing documents, that is extremely fast, and easy to setup (and use).
One that was written in C++ and not Java, and that non-the-less has bindings for .NET and Java, I assume.
I would say MongoDB would be the ideal choice.
Why not use a vcs such us svn or hg where you can assign attributes to files?
This all depends upon what you want to do with the information.
There has been a lot of talk about contra-revolutionary NoSQL databases like Cassandra, CouchDB, Hypertable, MongoDB, Project Voldemort, BigTable, and so many more. As far as I am concerned, the strongest pros are scalability, performance and simplicity.
I am seriously considering to suggest using some non-relational db for our next project. However, some teams comprise some RDBMS fanatics, so convincing a hard switch might be impossible in some cases just because of emotional reasons. Also, when it comes to complex data models, I personally still believe in the power of RDBMS with their low-level consistance enforcement mechanisms.
Now here comes my question: I was wondering, if someone could seriously consider using both, RDBMS and non-relational DB in a new project: The complex, but not performance critical data model would still be implemented using a relational model and db, while all performance critical, yet, simple models would be implemented with a non-relational db. Furthermore, such a soft paradigm shift would be much easier to sell to some highly emotionalized team members than a hard one.
Would anyone recommend such an approach? Or would you rather recommend a black or white, i.e. relational or non-relational approach? All comments highly welcome!
P.S.: Any idea if such a mix-up works well with Spring and Hibernate/JPA?
Rob Conery recently wrote about his experience building his popular web application TekPub with both MongoDB and MySQL, highlighting the strengths of both:
The high-read stuff (account info, productions and episode info) is perfect for a "right now" kind of thing like MongoDb. The "what happened yesterday" stuff is perfect for a relational system.
At a high-level, Rob breaks their application data into two scopes: runtime data and historical data. The current state of a user's shopping cart, for example, is great for keeping in MongoDB. It's an object blob that is constantly mutating. To keep a historical record of what went in and out of the shopping cart; when it happened; the state of the checkout are great for relational, tabular data in MySQL.
He summarizes with this:
It works perfectly. I could not be happier with our setup. It's incredibly low maintenance, we can back it up like any other solution, and we have the data we need when we need it.
Update May 2016 (6 years later)
This has only become more true in the last 6 years. It's now common to see NoSQL databases powering transactional stores and traditional relational databases powering analytical databases.
I use both. RDBMS is good for complex analysis, reports, and data accessed in different ways by multiple users. NOSQL is great when I'm looking at a data from a single users perspective. A question I ask myself is: "Who is accessing this data?". If the answer is 1 user, I use NOSQL to store it.
Of course, there are other times when it is appropriate, but that's just one example.
As you mentioned, NOSQL is simpler storage, and it could cause you to have to right complex code to maintain data... For example, storing a list of connections.
Non-relational key-value databases are best used for blob storage and caching.
Recently I have read a lot about different NoSQL databases and how they are being effectively deployed by some major websites out there. I'm starting a project in which I think the schema-free nature of a database such as MongoDB would be tremendously useful. Everything I have read though seems to indicate that the main advantage of a NoSQL database is scalability. Is choosing a NoSQL database for the schema-free design just as legitimate a design decision as that of scalability?
Yes, sometimes RDBMS are not the best solution, although there are ways to accomodate user defined fields (see XML Datatype, EAV design pattern, or just have spare generic columns) sometimes a schema free database is a good choice.
However, you need to nail down your requirements before choosing to go with a document database, as you will loose a lot of the power you may be used to with the relational model
eg...
If you would otherwise have multiple tables in your RDBMS database, you will need to research the features MongoDB affords you to accomodate these needs.
If you will need to query the data in specific ways, again you need to research what MongoDB offers you.
I wouldnt think of NoSQL as replacement for RDBMS, rather a slightly different tool that brings its own sets of advantages and disadvantages making it more suitable for some projects than others.
(Both databases may be used in some circumstances. Also if you decide to go down the route of possibly using MongoDB, once you have researched the websites out there and have more specific questions, you can visit Freenode IRC #mongodb channel)
There are a lot of other conditions that I've been hearing about with non-relational systems vs relational. I prefer this terminology over sql/no-sql as I personally think it describes the differences better, and several of the "no-sql" servers have sql add-ons, so anyway.... what sort of concurrency pattern or tranaction isolation is required in your system. One of the purported differences between rel and non-rel dbs is the "consistent-always", "consistent-mostly" or "consistent-eventually". Relation dbs by default usually fall into the "consistent-mostly" category and with some work, and a whole lot of locking and race conditions, ;) can be "consistent-always" so everyone is always looking at the most correct representation of a given piece of data. Most of what I've read/heard about non-rel dbs is that they are mainly "consistent-eventually". By this it means that there may be many instances of our data floating around, so user "A" may see that we have 92 widgets in inventory, whereas user "B" may see 79, and they may not get reconciled until someone actually goes to pull stuff from the warehouse. Another issue is mutability of data, how often does it need to be updated? The particular non-rel db's I've been exposed to have more overhead for updates, some of them having to regenerate the entire dataset to incorporate any updates.
Now mind, I think non-rel/nosql are great tools if they really match your use case. I've got several I'm looking into now for projects I've got. But you've got to look at all the trade offs when making the decision, otherwise it just turns into more resume driven development.
I don't think you should choose NoSQL datastore for its schema free design. Schema free design always existed in RDBMS via XML and some databases have good XML support. It is a lot easier to deal with a database than a NoSQL datastore. Scalability and big data should be the primary drivers to choose a NoSQL datastore otherwise the tradeoff of ACID and SQL is a lot to switch to NoSQL.
the most important things should be noticed to distinguish between No-SQL and SQL
which is :
NO-SQL useful when data base scales in a huge manner like social network
for example :
Stack Overflow: each question has multiple answers and not imaginary an answer without question, so No-SQL will ensure that each question include it's answers
as a result when needing getting answers of a question we can bring all answers without joining.Because join is the most expensive query in related database
thanks alot
what raised this issue that if you have a large server farm and need to manage the distribution of your data and load balancing which is more difficult and harder to implement using RDBMS and requires high IT skills to design, plan and deploy for your solution (and still performance is less).
but if you have only 3 or 4 servers with small project. I don't think you have an issue about it. NoSQL database is usually considered in large server farms not small number of servers
in this post Stack Overflow Architecture i read about something called nosql, i didn't understand what it means, and i tried to search on google but seams that i can't get exactly whats it.
Can anyone explain what nosql means in simple words?
If you've ever worked with a database, you've probably worked with a relational database. Examples would be an Access database, SQL Server, or MySQL. When you think about tables in these kinds of databases, you generally think of a grid, like in Excel. You have to name each column of your database table, and you have to specify whether all the values in that column are integers, strings, etc. Finally, when you want to look up information in that table, you have to use a language called SQL.
A new trend is forming around non-relational databases, that is, databases that do not fall into a neat grid. You don't have to specify which things are integers and strings and booleans, etc. These types of databases are more flexible, but they don't use SQL, because they are not structured that way.
Put simply, that is why they are "NoSQL" databases.
The advantage of using a NoSQL database is that you don't have to know exactly what your data will look like ahead of time. Perhaps you have a Contacts table, but you don't know what kind of information you'll want to store about each contact. In a relational database, you need to make columns like "Name" and "Address". If you find out later on that you need a phone number, you have to add a column for that. There's no need for this kind of planning/structuring in a NoSQL database. There are also potential scaling advantages, but that is a bit controversial, so I won't make any claims there.
Disadvantages of NoSQL databases is really the lack of SQL. SQL is simple and ubiquitous. SQL allows you to slice and dice your data easier to get aggregate results, whereas it's a bit more complicated in NoSQL databases (you'll probably use things like MapReduce, for which there is a bit of a learning curve).
From the NoSQL Homepage
NoSQL is a fast, portable, relational database management system without arbitrary limits, (other than memory and processor speed) that runs under, and interacts with, the UNIX 1 Operating System. It uses the "Operator-Stream Paradigm" described in "Unix Review", March, 1991, page 24, entitled "A 4GL Language". There are a number of "operators" that each perform a unique function on the data. The "stream" is supplied by the UNIX Input/Output redirection mechanism. Therefore each operator processes some data and then passes it along to the next operator via the UNIX pipe function. This is very efficient as UNIX pipes are implemented in memory. NoSQL is compliant with the "Relational Model".
I would also see this answer on Stackoverflow.
Put simply, it means not using a relational database for data storage.
Here's a relevant article: http://www.computerworld.com/s/article/9135086/No_to_SQL_Anti_database_movement_gains_steam_
NoSql is the new database philosophy which talks about all the shortcomings of the relational database design, particularly the problems they have in scaling up for today's demanding web environments.
NoSql is quickly evolving into a movement with new tools, software and formats coming up as alternative to SQL.
RDBMS is as ubiquitous as OOP and while both of these design methodologies solve some problems wonderfully, they don't solve all.
So think of NoSql as the functional programmin of the database world.
Was this simple enough?
NoSQL is the idea that SQL-type databases don't satisfy the demands/requirements of a heavily-used database that requires transactions be reliable and failsafe (or close to it). This ties into the ideas of ACID and CAP, both things worth looking into but not something to lose sleep over unless you run a really popular site that is transaction-heavy (ie Amazon or Ebay). To get a great start on these subjects, I suggest:
http://www.eflorenzano.com/blog/post/my-thoughts-nosql/
and
http://www.julianbrowne.com/article/viewer/brewers-cap-theorem
Something everyone considering a "nosql" approach should consider:
(I shan't risk putting the image into this post as it contains a curse word, and I don't want offensive flags. So clicker beware -- there's an f-word in there. Only click if you have a sense of humor.)
http://browsertoolkit.com/fault-tolerance.png
Found this nice article about no-sql
and this as well:
NoSQL, Yes Search