I’m interested in knowing some different approaches for retrieving data from Analysis Services, to use in either objects in code, or for end-user reporting.
I’ve used two different approaches in the past, one was using ADOMD to pull results and put these into a dataset, the other was using SQL OPENQUERY to a linked SSAS server to get results out as a SQL stored procedure result set. Both of these had advantages and disadvantages.
Over the years I’ve seen various questions along this line, so forgive me for any duplication, but what other methods are there for getting SSAS data into a format where other people’s code could use it?
I’ve considered XML result sets from SSAS over HTTP, then Linq to XML – Anyone have any experience with that?
Ideally I’d like a dataset with typed columns, or objects with properties, but I’m more interested in general approach than code samples. How have you got data from SSAS, apart from SSRS/Other dashboard controls?
I know MS is supposed to support XML/A (XML for Analysis). I am shortly releasing an ajax library to do XML/A requests from web pages.
While I am currently focused on Pentaho's Mondrian, it should work for MS SQLs XML/A too. If you are interested, I am doing a presentation on it on January 13. (see: http://wiki.pentaho.com/display/COM/January+13,+2010+-+Roland+Bouman+-+OLAP+and+Analysis+for+web+applications+using+XMLA) I will be releasing my code by that time too (probably underr a LGPL license)
I would love to get feedback from people that use other XML/A servers, so if you are interestd, it would be great to work together on this.
UPDATE:
the project is now available at http://code.google.com/p/xmla4js/ There is API documentation, code samples, and build scripts. It's LGPL so you're free to use it in your applications, even for commercial purposes. The license does require that you release any modifications to the library itself as LGPL (but this does not affect the application that uses the library)
UPDATE2
The project no resides on github at https://github.com/rpbouman/xmla4js
It works in the browser as well as in nodejs.
I've never used it myself, as we only use ADOMD and Excel to connect to SSAS, but at some point we considered using HTTP and XML. We ended up going the ADOMD route because of a shortened dev schedule, but I guess it's another option that allows for access to SSAS outside of the .Net world.
Here's a link which I found useful when prototyping: Configuring HTTP Access to SQL Server 2008 Analysis Services on Microsoft Windows Server 2008
XMLA is the "high power" approach -- but I'm not aware of a toolkit or library that really exposes the full capabilities of XMLA; I think you would have craft it up yourself. For the projects I've done, that's just way too much work.
Instead, I used ADOMD.NET for retrieving results in code; the CellSet class in particular is fairly rich. For end user analysis (slice and dice), most often I use Excel Pivot Charts (which are fabulous!); sometimes I also use Visio Pivot Diagrams. For fixed reporting, Reporting Services can access SSAS directly, and it even has it's own query builder.
BTW, in case it helps, I have a chapter in my book about integrating SSAS with web sites as a way of offloading SQL Server: Ultra-Fast ASP.NET. My code examples use ADOMD; I also walk through building a simple cube, configuring automatic updates with SSIS, using proactive caching, building simple MDX queries, etc.
Another approach is to use the MSOLAP OLE DB provider. Our code is currently using this method.
Another way (and I would say the simplest one) to get data from SSAS OLAP cubes into .NET is by using LINQ + ADO.NET Entity Framework + SSAS Entity Framework Provider .
(I work for the company that developed SSAS Entity Framework Provider).
Related
We are building new feature sets for one of our financial application. We have our own SQL server database and we will be calling multiple RESTful APIs that return JSON responses. For e.g. some returns news data, some returns stocks info, some returns finance data and our own sql server database has employee data. So, they all come with their own different data format. This new app we are building is going to aggregate all those data, transform it into meaningful display on web like mint.com does.
Web application will display analytical reports based on these data
There will be an option to download reports through various templates
We are completely open in terms of technology stack for our backend and middle-tier. As a first thought NoSQL like mongodb and elasticsearch for search and reporting comes to our mind. There will be a web application build on top of these data (stored or retrieved from API), most likely in Asp.net MVC.
We need your input, specially if you have experience with building similar enterprise solution.
Can you please share your opinions on,
What are some good tech stack you would pick for this app?
How would that scale now and in future when APIs data format changes.
Performance is also important since data will be displayed on web UI.
We have a similar setup to what you are mentioning, using ASP.Net MVC with ElasticSearch (SQL server for relational data, periodically updating ES), aggregating data (XML/JSON) from multiple sources, although with the purpose of improving searching and filtering results instead of reporting. However, I would expect that the scenario you are looking at would also be a suitable match for ElasticSearch, depending on your specific requirements.
1) Since you are already using SQL Server (and I expect are familiar with that), I would suggest combining that with ElasticSearch - the additional mongodb layer seems unnecessary, in terms of maintenance of another technology and development to fit that integration. There is a very good C# library (two actually, ElasticSearch.Net and NEST, used together) that exposes most of the ES functionality.
2) We chose ElasticSearch for its scalability in combination with flexibility and ease-of-use. A challenge you may face could be mapping the documents from C# classes to ElasticSearch documents. In essence, it is incredibly easy to set up, however you do need to do some planning to index data the way you want to search and retrieve it. So if choosing ES as a platform, spend some time with the structure of the documents - by default, dynamic mapping is enabled, so you can pretty much throw any JSON into a document. However, for a production environment, it's better to turn that off and have one or more mappings set up, so they can be queried in a standardized way.
3) Performance is a key factor for us as well, which is why we were looking at Lucene-based engines like Solr and ElasticSearch when doing research, along with NoSQL databases. ElasticSearch outperforms SQL Server by 10 to 1 or better, in most scenarios. Solr vs. ElasticSearch performance depends on scenario, benchmarks and comparisons are around if you Google them. The exception may be if many documents should be retrieved in one query - ES (or Lucene) is not made for that use case, it's best for fast retrieval of fewer results (similar to Google's per page results count) per page. If you need 1000 documents per page/result, a NoSQL database may be a better option.
ElasticSearch is fast to get up and running - install it on a local development box and try it out, you'll get a feel for if it fits.
From my experience, mongodb is the worst choice for reporting, especially for aggregation. It lacks in good aggregation functionality, has some data type conflicts (such as decimals being stored as strings, which you cannot use in it's built in aggregation framework api) and you'll probably will have to maintain map-reduce functions in javascript for most of the scenarios.
If your application's true nature is only reports, and they do not have to be updated in realtime, I would drop off the on-demand RPC calls to external APIs. I would consider copying ahead as much data as possible and storing it under a schema that is the most convenient for you to work with, and synchronising it afterwards under scheduled, predicted intervals.
I wouldn't be in a hurry making assumptions about that data to be available all the time nor it always to be in the format you expect. You also gain optimisation benefits on running your own copy of it, indexed in the way you want, instead of trying to figure which of the RPCs is your bottleneck.
As for your questions:
1) If you don't mind using Python, I would pick Django on top of PostgresSQL database. Django is a fully featured sturdy ORM + Web framework which is excellent for this kind of work. If not, just stick to a relational SQL database. I heard wonders of Cassandra but haven't tried it yet.
2 + 3) As I mentioned before, replicating the data as much as possible for your own good. After everything is "in house" you can cluster it and tweak it freely. Using a distributed cache against heavy client requests is also a good idea (such as REDIS), instead of generating those reports each time on demand.
I've been using Jasper reports and the Jasper Reports Server to integrate into our web app. Jasper accepts many different datasource types including JSON and SQLServer. The core version is free and allows you to product html amd pdf reports of high complexity. The paid version with the server allows you to easily integrate in your web app. The core is Java spring (partially open source) running on tomcat/jboss and you can interact with it using REST web services or the visualize.js library for your web front end. It uses highcharts which can produce some beautiful results and has options for adhoc reporting and dashboards built from many reports.
See demos here: http://www.jaspersoft.com/
This has an assumed stack of your backend db's and data sources, tomcat with Java Spring, web front end HTML/Javascript.
The tool is used by many large enterprises including Amazon so scalibility so shouldn't be an issue.
If the format of your data changes you'll need to change the report. This is xml formatted editted by GUI with WYSIWYG.
Because of performance issue in application we are using in-memory approach for one of my project, in which we are loading all tables in RAM in form of generic collections (using nhibernate).
Issue is that when we were using simple linq to sql approach that time the testing and QA team were easily able to get sql queries using sql profiles for page they were viewing.
but with new approach (in-memory), we are loading all data in collection in one go and then are using linq to get data from that collection, so the testing and QA teams are not able to get the sql queries to verify the business logic and verifying bugs.
Please suggest any solution which can help in this situation, i think its not possible to get sql from linq to object (as all data is already in collection). please suggest any solution/approach/tool which can help me, to generate sql of those linq which is getting run against the collection or any other good solution.
NOTE: i know getting sql out of linq to sql is not possible, i am looking for suggestion which can help my QA and testing team to verify the queries/business logic (like they were doing earlier by capturing sql). like if possible log the linq queries as string which can be further used to be run/analyze.
The only SQL statements that you are running in this situation are the initial SELECT queries to load your data into memory. Once you have done those, you are no longer running "queries", you are instead performing .NET Framework calls.
Given that you have fundamentally changed the architecture of the application, you need to communicate this to the testing and QA teams - they will not be able to "see" what the application is now doing under the hood in the way that they could previously. If this sort of "deep dive" capability is a requirement of your test teams then your architecture is likely to require further modifications.
Is using strongly typed dataset is good.
Currently I am working on a project developed using VB.Net in Visual Studio 2010.
Previously they were using Sql queries directly into SqlCommand of System.Data.SqlClient, but then after i shifted everything to Strongly Typed DataSet and started using TableAdapters every where..
Now i just wanna ask that is this way is good for a project...
Or Should i shift back to old ones using Just SqlCommands
Or Is there any way to make Sql DataBase in a good way because its an ERP and most of the code is for Data Access..
We use strongly typed datasets all the time now.
After shifting to this behaviour it felt really bad to have SQL-querys in code instead of having it done by the table adapter. But there is a bit overhead with datasets so I guess booth ways are good for different solutions.
Its really nice to have intellisence on all fieldnames and if you change a tableadapter so it returns something different you get design-time errors everywhere where you need to change the code to reflect the change, instead of finding out runtime when the customer is running the program.
There are so many win win-things with strongly typed datasets so I'll never go back.
Table adapters .... make a lot of mess with bigger databases, also updating the table structure also causes confusion.
I would recommend to use some auto code generators for the CRUD Operations.
To me your old pattern looks better than switching altogether to table adapters and strongly typed datasets.
If you ever want to move your data across the wire to other platforms (silverlight, web services, wcf services, etc), then using any kind of dataset will box you into a corner.
The way that we have resolved this is to have classes whose list of properties match the database exactly. To move the data in and out of the database, we use reflection to either match stored procedure parameters or generate dynamic SQL statements, depending on the circumstance and platform.
When a database table is changed, the developer making the change is also responsible for updating the class structure and vice-versa.
In order to reduce the amount of hand-coding required, we use the code generation capabilities of CodeSmith to generate classes from the database and create the basic implementations of our standard add/update stored procedures that require field enumeration.
As an added benefit, this approach removes the tight link between the database and business object structure. We are able to use our same data access code and business object classes against SQL Server, Oracle, Sqlite, and SqlServerCE databases. This code is used to create applications in Windows, PocketPC, Web, iPad, and Android apps; all of the mobile apps use local databases specific to the platform, but using the common data access code.
It is a bit more work to setup initially, but it will pay significant dividends in the long run.
After reading many posts on this site, I am considering replacing my accdb be (2007) with SQL Server or SQL Server Express? My fe is currently accdb (accde upon deployment). Is there any advantage to connecting to SQL using .adp as opposed to .accde? I currently link to tables in SQL using ocdb and it seams to work fine.
The only other question I have is, how much effort is it to create my fe forms using Visual Basic in Visual Studio Express? The idea of having a standalone exe makes it tempting. Am I looking at a big learning curve to go from vba to vb? I'm assuming the functionality of my code can be replicated in vb with some changes required.
Just seems like a good time to make this decision, while I'm still in the design stage.
It’s a big leap to go from a full access application to a .Net application with an SQL server backend. I have been going through this for 4 or 5 applications over the past year and what I did was to split the move into 2 parts.
First I migrated the data to SQL server but kept the access front end. I then look closely at the reasons why I would change the front end, for some of the applications it is just not worth the time to rewrite them when the only feature I would be gaining is an .exe extension and not a .mdb one!
Some however it was worth the time and so I have migrated them over but you have to look at what you are gaining, the Access front end / SQL server backend is a very powerful combination
I have worked with MS Access since version One and SQL Server since version 6.5. Access is indeed a relational database, and always has been. I've even seen Data Warehousing done in Access. My answer to RYK would be this: Access is an AWESOME front end for SQL Server for all the reasons that really matter. Access includes Forms, Queries, Reports, Macros and Code Design all in one package. Access has all of these features AND they are extensive. Form design is elegant. Report design is feature rich. Queries and Data Design does everything you want it to. VBA is fully functional. VB.NET/C# will work but .NET development takes longer than Access development. Access even lets you quickly test data scenarios and situations before the data is imported to SQL Server. If anyone knows of an application that has more features than Access all in one app please, please let me know what that is. Access even allows users to perform ad-hoc queries if they know how to. I should add that I am an ASP.NET VB.NET C# Developer and Crystal Reports Designer. I also design Oracle solutions. For over 10 years now if you Have MS Office and SQL Server you can do ANYTHING with data, and do it easier. (Excel Reports, Visio Data Modeling, Data Warehousing, Word Mail Merging, Full featured database apps, you name it) Visual Studio and .NET just add different flavors of icing to an already large and delicious cake.
May I suggest you download and give Visual Studio Lightswitch a go. Really fantastic for replacing/upgrading Access forms apps. You also would not need to worry about what connections to use etc. I find this tool really useful and friendly to replace/replicate access apps and spreadsheets the business was used to use.
The dev tools for SQL in Visual Studio are great.
Is it possible to develop an app for Microsoft SQL Server, and then deploy a MySQL-compatible database instead?
The dev tools for Microsoft SQL Server are really nice (i.e. LINQ support), but a MySQL-compatible column database has better performance for huge datasets.
You would need to use MySql .net connector
http://dev.mysql.com/doc/refman/5.1/en/connector-net.html
With this, you get some of the functionality that you have with SQL server (entity framework, designer, etc) there are still a bunch of things that are not supported, but it's a good start
It's possible, but a number of specifics might tend to prevent a full implementation on SQL Server, particularly with respect to stored procedures.
However, if the intent is to build a scaffolding on VisStudio and finish the development using the MySQL tools, it would work okay. You'll have to learn both SQLs quite thoroughly. The sooner you do that, the less grief there will be in the conversion.
One method for doing this would be to abstract away the database itself. That be done at least a couple of ways; the first way, you could use classes to build the SQL that your application requires, and then just use that; then all that has to happen is that it has to know how to generate the right SQL for the right server. One of the drawbacks of doing it that way, though, is that if you depend on functionality that exists on only one DBMS, you'll have to emulate it in that abstraction layer.
The other method that you could use is to create two versions of your classes that talk to the RDBMS, one for MySQL and one for Microsoft's SQL Server. Use an interface and derive from it in order to do the actual implementation. Of course, you'll want to make sure that the only responsibility of the class is to interact with the database, so if you're doing this for business layer objects, you'll be implementing those sorts of things with two classes: a low-level one for the database API, and a high-level one for actually providing the API that your application is going to consume.
Perhaps not a direct answer to your question, but the dblinq project may be of use to you.
It may be worth a look just to see the MySQL implementations within the project in order to determine what the real differences between SQL Server and MySQL are going to be and how they're going to affect you. The more you can abstract out those differences behind a dependency implementation, the easier it'll be to swap out one implementation for another.
You can write your code for SQL Server and then switch to Devart LinqConnect.
For example, you can create a LINQ to SQL model using Entity Developer (in VS integration mode or in standalone mode), then change the connection to the MySQL-specific one and run Update Database from Model wizard (don't forget to select the Regenerate Storage check box).
As a result, you will obtain a MySQL database, having structure identical to the SQL Server one.