I was assigned a task to design a database schema using operational model, but unfortunately unable to get any information on what exactly the operational model mean.
I worked extensively on Star and snowflake schemas but management doesn't require these schema designs and stressing me to create a operational model.
If anyone can give some insight on what operational model and how can I get some knowledge would be great help.
Operational Model is a model which would help in daily operations and not just getting statistics about the data.
The management is looking for a daily Transactional Processing Database Systems whereas the snowflake and the star schema are great designs for Analytical Processing Systems and Warehouses but not for daily transactional Processing Systems. They are looking for an OLTP system than an OLAP System.
OLTP (On-line Transaction Processing) is characterized by a large number of short on-line transactions (INSERT, UPDATE, DELETE). The main emphasis for OLTP systems is put on very fast query processing, maintaining data integrity in multi-access environments and an effectiveness measured by a number of transactions per second. In OLTP database there is detailed and current data, and schema used to store transactional databases is the entity model (usually 3NF).
OLAP (On-line Analytical Processing) is characterized by a relatively low volume of transactions. Queries are often very complex and involve aggregations. For OLAP systems a response time is an effectiveness measure. OLAP applications are widely used by Data Mining techniques. In OLAP database there is aggregated, historical data, stored in multi-dimensional schemas (usually star schema).
Related
What are the key differences between OLAP and OLTP databases.
Specifically in terms of implementation (rather than use cases).
OLAP is of course primarily used for reporting while OLTP is used for handling transactions.
I understand that OLAP databases are optimized for read over write, and that OLAP databases contain more denormalised data.
What other characteristics set the two apart?
OLTP:
As the name suggest "Online Transaction Processing", this is used for more transaction needs like "INSERT/SELECT/UPDATE/DELETE".
Low Response Time.
There are the original source of data.
Usually data is stored in 3NF form.
ACID properties are necessarily followed.
OLAP:
As the name suggest "Online Analytical Platform", used for analytical queries and in general are used for complex analytical queries and drawing inferences.
Periodic batch processing jobs are run here.
Typically de-normalized with fewer tables; use of star and/or snowflake schemas.
NOT necessarily follows ACID properties.
There are many difference. You may find tons of answers by googling this question. But some of the characteristics which are derived from practical implementation from my own experiences are:
OLTP is business domain specific system designed to perform specific tasks for example an eCommerce website having a database for handling online order while another OLTP database is being used for back end operation for order processing another OLTP database is for logistics etc. Whereas OLAP systems are designed to look at the information at whole business level by sourcing data from many heterogeneous system.
If I simplified the above example then OLTP is small units of Business Processing system while OLAP system is a large unit of Business Information.
You can refer this link for more clarification.
I just wanted to know does it will make any sense to join sql database with nosql database?
Yes it makes sense, one of the big advantage of NoSQL data storage is that data is not tight to specific schema.
One fundamental difference between SQL and NOSQL dB's is support for transactions.
Imagine you were writing a banking app that keeps account balances. You will not be able to achieve accurate balance values unless you use transactions. This is common in all SQL dbs that support ACID semantics.
However support for transactions is not available in NOSql. Therefore NOSql is not suited for any project that needs transactions.
That said, if the same banking app needs tremendous scale, it can be built such that all non transactional data or data that can tolerate "eventual consistency" can use NOSql and data that needs transaction support can be stored in an SQL db.
The advantage of this design would be the benefit of automatic sharding or splitting of data that NOSql DBs provide that allows them to scale easily. In effect, maintenance needs of the DB can be significantly reduced by choosing for a hybrid model such as this.
I'm new to Analysis Services
My first cube has been deployed and it seems to work.
Dimension tables are ok and fact tables are ok.
My question is very simple : If I add a new record in the related datasource table,
Browsing the cube, I don't see the new record until process again the cube.
In my mind I think if new records are addedd, then cube must reflect the changes.
How to solve this issue? Do I need to reprocess the cube every time a new record is added? This is impossible of course.
You understand that essentially your cube represents a bunch of aggregated measures? That means that when the cube is processed it looks at all the data that is in your fact tables and processes the Measures (according to the dimensions).
The result of this is that you're able to access the data in the cube quickly and efficiently. The downside is as you have mentioned is that when new data is added to the fact table the cube isn't updated.
Typically there will be a daily batch job that will update the cube with the latest fact data, depending on the amount of data you have and the "real-time" requirements this could be done more than once p/day. A lot of people do this out of hours.
If you look closely in BIDS you will notice on the Partitions tab that for each partition it has a Storage Mode which you can define.
I would recommend you read this this article http://sqlblog.com/blogs/jorg_klein/archive/2008/03/27/ssas-molap-rolap-and-holap-storage-types.aspx
Basically, there are a few different modes you can use:
MOLAP (Multi dimensional Online Analytical Processing)
MOLAP is the most used storage type. Its designed to offer maximum query performance to the users. Data AND aggregations are stored in optimized format in the cube. The data inside the cube will refresh only when the cube is processed, so latency is high.
ROLAP (Relational Online Analytical Processing)
ROLAP does not have the high latency disadvantage of MOLAP. With ROLAP, the data and aggregations are stored in relational format. This means that there will be zero latency between the relational source database and the cube.
Disadvantage of this mode is the performance, this type gives the poorest query performance because no objects benefit from multi dimensional storage.
HOLAP (Hybrid Online Analytical Processing)
HOLAP is a storage type between MOLAP and ROLAP. Data will be stored in relational format(ROLAP), so there will also be zero latency with this storage type.
Aggregations, on the other hand, are stored in multi dimensional format(MOLAP) in the cube to give better query performance. SSAS will listen to notifications from the source relational database, when changes are made, SSAS will get a notification and will process the aggregations again.
With this mode it’s possible to offer zero latency to the users but with medium query performance compared to MOLAP and ROLAP.
To get the real-time reporting without having to reprocess your cube you will need to try out ROLAP, but beware, the performance will suffer (depending on the size of your cube and server!).
I'm starting to study SQL Server Analysis Services and I'm working my way through the training book, as well as the Developer Training Kit. In both, I find suggestions that the number of tables used in an OLAP database (ideally, star schema) is greatly reduced from the production OLTP database.
From the training kit:
We followed the data dimensional methodology to architect the data mart schema. From some 200 tables in the operational database, the data mart schema contained about 10 dimension tables and 2 fact tables.
From what I understand, the operational databases are usually (somewhat) normalised and the data mart schemas are heavily denormalised. I also believe that denormalising data usually involves adding more tables, not less.
I can't see how you can go from 200 tables to 12, unless you only need to report on a subset of data. And if you do only need to report on a subset of data, why can't you just use the appropriate tables in the operational database (unless there are significant performance gains to be made by using a denormalised star schema)?
Denormalizing is exactly the opposite of Normalizing a database. In a normalized database everything is spit apart into different tables to support concurrent writes to the data. This also has the side effect of generating any given subset of data exactly once (In an ideal 3rd normal form data structrure). A draw back of normalizing is that reads take a lot longer because of the fact that the data is scattered and we need to join tables to make sense of it again (Joins are pretty expensive operations).
When we denormalize, we are taking the data from multiple tables and merging them in to one table. So now we have repeating data in these tables. The repeating data is useful because we don't have to make joins to any other table to get it anymore. Writing to the data store is normally a bad idea because it would mean alot of writes to change all of the data in a table, whereas it would only take one in a normalized database.
OLTP stands for Online Transactional Processing, notice the word Transactional. Transactions are write operations and the OLTP model is optimiized for this. OLAP stands for Online Analytical Processing, Analysis being the keyword meaning lots of reads.
Going from 200 tables to 12 in an OLTP to OLAP process will suprisingly hold nearly all of the data in the OLTP database plus more. The OLTP is unable to record all of the changes over time, but OLAP specializes in this so you get all of your historical data as well as current data.
The star schema is probably the most common for OLAP data stores, the snowflake schema is also pretty common. You should learn about both and how to properly use them. It's just another great tool in your arsenal.
These two books from IBM will answer your questions much more thouroughly and they are free pdf's.
http://www.redbooks.ibm.com/abstracts/sg247138.html
http://www.redbooks.ibm.com/abstracts/sg242238.html
What is Molap and Rolap and whats the difference between these two ?
MOLAP = Multidimensional Online Analytical Processing
ROLAP = Relational Online Analytical Processing
Essentially with ROLAP, the data is stored in a relational database, whereby with MOLAP, i.e. the traditional OLAP model, this is stored in multidimentional "cubes". Cubes are a multidimensional structure similar to the star schema in an RDBMS, but where the management of the storage is highly optimized to deal with such a structure.
At the risk of adding to the alphabet soup, another model is HOLAP, for Hybrid OLAP, which attempts to provide the best of traditional MOLAP with the benefits of "Relational" in ROLAP.
MOLAP's main advantage is its excellent query performance and fast data retrieval.
Its main disadvantage is that is may be limited in the amount amount of data it can handle. Another disadvantage is the use of proprietary engines.
ROLAP's performance is slower, but it also it is less limited in term of the number of dimensions etc.
These are terms associated with Datawarehousing. ROLAP is relational OLAP.