Because of performance issue in application we are using in-memory approach for one of my project, in which we are loading all tables in RAM in form of generic collections (using nhibernate).
Issue is that when we were using simple linq to sql approach that time the testing and QA team were easily able to get sql queries using sql profiles for page they were viewing.
but with new approach (in-memory), we are loading all data in collection in one go and then are using linq to get data from that collection, so the testing and QA teams are not able to get the sql queries to verify the business logic and verifying bugs.
Please suggest any solution which can help in this situation, i think its not possible to get sql from linq to object (as all data is already in collection). please suggest any solution/approach/tool which can help me, to generate sql of those linq which is getting run against the collection or any other good solution.
NOTE: i know getting sql out of linq to sql is not possible, i am looking for suggestion which can help my QA and testing team to verify the queries/business logic (like they were doing earlier by capturing sql). like if possible log the linq queries as string which can be further used to be run/analyze.
The only SQL statements that you are running in this situation are the initial SELECT queries to load your data into memory. Once you have done those, you are no longer running "queries", you are instead performing .NET Framework calls.
Given that you have fundamentally changed the architecture of the application, you need to communicate this to the testing and QA teams - they will not be able to "see" what the application is now doing under the hood in the way that they could previously. If this sort of "deep dive" capability is a requirement of your test teams then your architecture is likely to require further modifications.
Related
I am upgrading a webapp that will be using two different database types. The existing database is a MySQL database, and is tightly integrated with the current systems, and a MongoDB database for the extended functionality. The new functionality will also be relying pretty heavily on the MySQL database for environmental variables such as information on the current user, content, etc.
Although I know I can just assemble the queries independently, it got me thinking of a way that might make the construction of queries much simpler (only for easier legibility while building, once it's finished, converting back to hard coded queries) that would entail an encapsulation object that would contain:
what data is being selected (including functionally derived data)
source (including joined data, I know that join's are not a good idea for non-relational db's, but it would be nice to have the facility just in case, which can be re-written into two queries later for performance times)
where and having conditions (stored as their own object types so they can be processed later, potentially including other select queries that can be interpreted by whatever db is using it)
orders
groupings
limits
This data can then be passed to an interface adapter that can build and execute the query, returning it in an array, or object or whatever is desired.
Although this sounds good, I have no idea if any code like this exists. If so, can anybody point it out to me, if not, are there any resources on similar projects undertaken that might allow me to continue the work and build a basic version?
I know this is a complicated library, but I have been working on this update for the last few days, and constantly switching back and forth has been getting me muddled at times and allowing for mistakes to occur
I would study things like the SQL grammar: http://www.h2database.com/html/grammar.html
Gives you an idea of how queries should be constructed.
You can study existing libraries around LINQ (C#): https://code.google.com/p/linqbridge/
Maybe even check out this link about FQL (Facebook's query language): https://code.google.com/p/mockfacebook/issues/list?q=label:fql
Like you already know, this is a hard problem. It will be a big challenge to make it run efficiently. Maybe consider moving all data from MySQL and Mongo to a third data store that has a copy of all the data and then running queries against that? Replicating all writes to something like Redis or Elastic Search and then write your queries there?
Either way, good luck!
I am working on a project which involves collecting dynamic form data. These forms are user-defined (think surveymonkey) and thus a fixed schema cannot be defined for them. Data in terms of questions/answers would be retrieved for these forms and then stored into the database. Reporting/Searching on this answers (filtering and aggregation) is of utmost importance. There are two approaches which are feasible.
Use a SQL database and store the each field data as a separate row. Reporting/Searching is then done via SQL. My apprehension is that it would result in complicated joins for reporting.
Use a NoSQL database like MongoDB. This seems to be a perfect fit for storing the dynamic data since it is schema-less. However, I am not sure how good its reporting capabilities are.
It seems easier for target users to learn sql than to define map/reduce queries. How easy would it be to build a UI for reporting/searching over mongoDB.
Simple things like - list of users who gave a particular set of answers. How many such users over a period of time etc?
Thanks,
Pulkit
It's already been mentioned in the comments, but I'll re-iterate that you should look at Mongo's map/reduce functionality for reporting and the aggregation framework.
Having done map/reduce in both Couch and Mongo I can say that they are very similar. It's definitely a barrier to entry for a developer that isn't familiar with it, but once you get a few working examples, it's not too bad.
Consider that Mongo can output a map/reduce job to a collection, which I've found to be really useful. This means you can schedule the jobs and run them periodically and output to a place that you can then report on. It's not that hard to create a framework that lets developers write simple Javascript map and reduce functions and then plug them in to be run on a schedule.
The aggregation framework is much easier to understand for a developer coming from SQL. Still a learning curve, but not as bad as map/reduce. It is much more well suited to ad-hoc reporting queries and there is nothing comparable in Couch.
You could maybe make a reporting UI that maps to the aggregation framework, but I wouldn't try to do something similar for map/reduce queries.
I would like to build arbitrary queries to a database, by allowing the user to build queries "on the fly". For every object/table, being able to select its attributes, and then "building" the query (that would translate into a SQL statement) and finally launching it, all through a web interface.
The ticketing system "rt" does that, for example, and another example would be the http://gatherer.wizards.com/Pages/Advanced.aspx webpage.
I'm currently programming in rails but any existing solution that implements this (or something similar) would be welcome.
Just be careful when creating dynamically generated queries like this that will need to be executed via sp_executesql (example: ms sql server), etc..... make sure you cover all of your bases to ensure that your application isnt vulnerable to SQL injection attacks as this type of development will essentially get one in a lot of trouble if its done incorrectly.. I would recommend storing all queries in a table and only reading queries from this table to help isolate the queries that are being ran in your application. Just identify them with a label, and allow the EU to choose the label from a dropdown list control on the frontend.
Good luck and I'm not sure of any software that will help assist
Not quite sure what your use case is here but i would say check out the
Doctrine ORM ( Object Relational Mapper )
**Edit
After reading more and looking at the example. I would only suggest Doctrine for a large website.
Then use Doctrines DQL syntax with some javascript/jquery magic for the forms.
Note that the queries you're referencing aren't arbitrary: they're on a very specific problem domain, on a specific set of sql tables.
That said, if I were you I'd look into how people are building sql queries with javascript. Something like these:
http://code.google.com/p/django-querybuilder/
http://css.dzone.com/articles/sqlike-sql-querying-engine?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+zones%2Fria+(RIA+Zone)
http://thechangelog.com/post/4914956307/rel-arel-ported-to-node-js-with-some-changes
That'll at least get you a good idea of the underlying data structures.
I need to give users the ability to build a simple SQL query against our database. Our application is written in Delphi.
I am assuming only moderate levels of knowledge by the user, but they need the ability to build a simple select statement to be able to query against a couple of tables. If I can make this easy for them, that would be most wondrous.
Does anyone know of a tool or a set of components that I can use to help the users build SQL SELECT statements...
I've used the DevExpress ExpressFilter control to allow users to specify SQL where constraints before.
I've come across FastQueryBuilder http://fast-report.com/en/products/visual-query-builder.html
and
OpenQueryBuilder http://fast-report.com/en/products/free-query-builder.html
(apparently differing versions of the same tool) at Fast-Report.
It looks like the right idea. Anyone have any experience with it?
I've used SimpleQuery for years and am quite happy with it. (http://devtools.korzh.com/eq/vcl/) It's not completely intuitive, but once you get the hang of it, you can present fields to the user from your database(s) and they can combine any series of them with ANDs and ORs to make very complex queries. You get a separate SQL window to save the resulting code. I make The resulting dataset available for printout with PrintDat! or to save as CSV data from the Woll2Woll Infopower grid that I use for display. I occasionally allow for XLS export too with XLSReadWrite. It's all quite automatic at this point. Just take the SimpleQuery example and mold that to your particular needs. You should be able to have something very usable in a day. There is a trial and an example free query builder standalone program that will give you an idea of the final look.
TMS Query Studio? Good value at 75 Euro..
Try EMS Advanced Query Builder.
is a powerful component suite for
Borland® Delphi® and C++ Builder®
intended for visual building SQL
statements for the SELECT, INSERT,
UPDATE and DELETE clauses. It allows
you to build new queries visually
and/or graphically represent the
existing queries in your own
applications. The suite includes
components for working with standard
SQL, MS SQL, InterBase/Firebird,
MySQL, PostgreSQL and many more
dialects. Advanced Query Builder
enables users to make up large and
complicated SQL queries with unions
and subqueries for different servers
without any knowledge of the SQL
syntax.
(source: sqlmanager.net)
(source: sqlmanager.net)
I think the most powerful one is ActiveQueryBuilder, which now included with Delphi IDE (Since 2007 I think)
(source: activequerybuilder.com)
I've written a couple of similar things in Delphi. It's easy enough to allow the user to pick table and column names by querying the metadata and using lists and tree views for display. The difficulty comes when trying to implement things like joins. I've never come up with a good interface for this, and in my experience neither have many of the major data tools players.
You may want to try this free builder:
GSC Visual Query Builder v.0.7
Visual Query Builder v.1.0
and some other in Torry:
http://www.torry.net/pages.php?id=546
I’m interested in knowing some different approaches for retrieving data from Analysis Services, to use in either objects in code, or for end-user reporting.
I’ve used two different approaches in the past, one was using ADOMD to pull results and put these into a dataset, the other was using SQL OPENQUERY to a linked SSAS server to get results out as a SQL stored procedure result set. Both of these had advantages and disadvantages.
Over the years I’ve seen various questions along this line, so forgive me for any duplication, but what other methods are there for getting SSAS data into a format where other people’s code could use it?
I’ve considered XML result sets from SSAS over HTTP, then Linq to XML – Anyone have any experience with that?
Ideally I’d like a dataset with typed columns, or objects with properties, but I’m more interested in general approach than code samples. How have you got data from SSAS, apart from SSRS/Other dashboard controls?
I know MS is supposed to support XML/A (XML for Analysis). I am shortly releasing an ajax library to do XML/A requests from web pages.
While I am currently focused on Pentaho's Mondrian, it should work for MS SQLs XML/A too. If you are interested, I am doing a presentation on it on January 13. (see: http://wiki.pentaho.com/display/COM/January+13,+2010+-+Roland+Bouman+-+OLAP+and+Analysis+for+web+applications+using+XMLA) I will be releasing my code by that time too (probably underr a LGPL license)
I would love to get feedback from people that use other XML/A servers, so if you are interestd, it would be great to work together on this.
UPDATE:
the project is now available at http://code.google.com/p/xmla4js/ There is API documentation, code samples, and build scripts. It's LGPL so you're free to use it in your applications, even for commercial purposes. The license does require that you release any modifications to the library itself as LGPL (but this does not affect the application that uses the library)
UPDATE2
The project no resides on github at https://github.com/rpbouman/xmla4js
It works in the browser as well as in nodejs.
I've never used it myself, as we only use ADOMD and Excel to connect to SSAS, but at some point we considered using HTTP and XML. We ended up going the ADOMD route because of a shortened dev schedule, but I guess it's another option that allows for access to SSAS outside of the .Net world.
Here's a link which I found useful when prototyping: Configuring HTTP Access to SQL Server 2008 Analysis Services on Microsoft Windows Server 2008
XMLA is the "high power" approach -- but I'm not aware of a toolkit or library that really exposes the full capabilities of XMLA; I think you would have craft it up yourself. For the projects I've done, that's just way too much work.
Instead, I used ADOMD.NET for retrieving results in code; the CellSet class in particular is fairly rich. For end user analysis (slice and dice), most often I use Excel Pivot Charts (which are fabulous!); sometimes I also use Visio Pivot Diagrams. For fixed reporting, Reporting Services can access SSAS directly, and it even has it's own query builder.
BTW, in case it helps, I have a chapter in my book about integrating SSAS with web sites as a way of offloading SQL Server: Ultra-Fast ASP.NET. My code examples use ADOMD; I also walk through building a simple cube, configuring automatic updates with SSIS, using proactive caching, building simple MDX queries, etc.
Another approach is to use the MSOLAP OLE DB provider. Our code is currently using this method.
Another way (and I would say the simplest one) to get data from SSAS OLAP cubes into .NET is by using LINQ + ADO.NET Entity Framework + SSAS Entity Framework Provider .
(I work for the company that developed SSAS Entity Framework Provider).