I would like to build arbitrary queries to a database, by allowing the user to build queries "on the fly". For every object/table, being able to select its attributes, and then "building" the query (that would translate into a SQL statement) and finally launching it, all through a web interface.
The ticketing system "rt" does that, for example, and another example would be the http://gatherer.wizards.com/Pages/Advanced.aspx webpage.
I'm currently programming in rails but any existing solution that implements this (or something similar) would be welcome.
Just be careful when creating dynamically generated queries like this that will need to be executed via sp_executesql (example: ms sql server), etc..... make sure you cover all of your bases to ensure that your application isnt vulnerable to SQL injection attacks as this type of development will essentially get one in a lot of trouble if its done incorrectly.. I would recommend storing all queries in a table and only reading queries from this table to help isolate the queries that are being ran in your application. Just identify them with a label, and allow the EU to choose the label from a dropdown list control on the frontend.
Good luck and I'm not sure of any software that will help assist
Not quite sure what your use case is here but i would say check out the
Doctrine ORM ( Object Relational Mapper )
**Edit
After reading more and looking at the example. I would only suggest Doctrine for a large website.
Then use Doctrines DQL syntax with some javascript/jquery magic for the forms.
Note that the queries you're referencing aren't arbitrary: they're on a very specific problem domain, on a specific set of sql tables.
That said, if I were you I'd look into how people are building sql queries with javascript. Something like these:
http://code.google.com/p/django-querybuilder/
http://css.dzone.com/articles/sqlike-sql-querying-engine?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+zones%2Fria+(RIA+Zone)
http://thechangelog.com/post/4914956307/rel-arel-ported-to-node-js-with-some-changes
That'll at least get you a good idea of the underlying data structures.
Related
I am upgrading a webapp that will be using two different database types. The existing database is a MySQL database, and is tightly integrated with the current systems, and a MongoDB database for the extended functionality. The new functionality will also be relying pretty heavily on the MySQL database for environmental variables such as information on the current user, content, etc.
Although I know I can just assemble the queries independently, it got me thinking of a way that might make the construction of queries much simpler (only for easier legibility while building, once it's finished, converting back to hard coded queries) that would entail an encapsulation object that would contain:
what data is being selected (including functionally derived data)
source (including joined data, I know that join's are not a good idea for non-relational db's, but it would be nice to have the facility just in case, which can be re-written into two queries later for performance times)
where and having conditions (stored as their own object types so they can be processed later, potentially including other select queries that can be interpreted by whatever db is using it)
orders
groupings
limits
This data can then be passed to an interface adapter that can build and execute the query, returning it in an array, or object or whatever is desired.
Although this sounds good, I have no idea if any code like this exists. If so, can anybody point it out to me, if not, are there any resources on similar projects undertaken that might allow me to continue the work and build a basic version?
I know this is a complicated library, but I have been working on this update for the last few days, and constantly switching back and forth has been getting me muddled at times and allowing for mistakes to occur
I would study things like the SQL grammar: http://www.h2database.com/html/grammar.html
Gives you an idea of how queries should be constructed.
You can study existing libraries around LINQ (C#): https://code.google.com/p/linqbridge/
Maybe even check out this link about FQL (Facebook's query language): https://code.google.com/p/mockfacebook/issues/list?q=label:fql
Like you already know, this is a hard problem. It will be a big challenge to make it run efficiently. Maybe consider moving all data from MySQL and Mongo to a third data store that has a copy of all the data and then running queries against that? Replicating all writes to something like Redis or Elastic Search and then write your queries there?
Either way, good luck!
Because of performance issue in application we are using in-memory approach for one of my project, in which we are loading all tables in RAM in form of generic collections (using nhibernate).
Issue is that when we were using simple linq to sql approach that time the testing and QA team were easily able to get sql queries using sql profiles for page they were viewing.
but with new approach (in-memory), we are loading all data in collection in one go and then are using linq to get data from that collection, so the testing and QA teams are not able to get the sql queries to verify the business logic and verifying bugs.
Please suggest any solution which can help in this situation, i think its not possible to get sql from linq to object (as all data is already in collection). please suggest any solution/approach/tool which can help me, to generate sql of those linq which is getting run against the collection or any other good solution.
NOTE: i know getting sql out of linq to sql is not possible, i am looking for suggestion which can help my QA and testing team to verify the queries/business logic (like they were doing earlier by capturing sql). like if possible log the linq queries as string which can be further used to be run/analyze.
The only SQL statements that you are running in this situation are the initial SELECT queries to load your data into memory. Once you have done those, you are no longer running "queries", you are instead performing .NET Framework calls.
Given that you have fundamentally changed the architecture of the application, you need to communicate this to the testing and QA teams - they will not be able to "see" what the application is now doing under the hood in the way that they could previously. If this sort of "deep dive" capability is a requirement of your test teams then your architecture is likely to require further modifications.
In HTML+CSS+JS world, http://jsfiddle.net/ is very helpful tool for asking / making example about web development. And I also saw several browser(javascript)-based programming language compilers and REPLs. But I can't find online / web-based test environment for database operations( especially for RDBMS ).
Is there any open/free database service with web-based interfaces for testing queries?
Added: This tool will be good for this situation; If I'm troubling with complex queries, then create a sample table via web interface and ask it on stackoverflow with the 'sample table URL'. Anyone can access to the URL and test their queries on web site. (Yes, queries are running on 'real' database system) And also the query results can be tracked, then we can even make 'ranking' for it :)
Try SQL Fiddle.
You can try your SQL query and execute/test it.
There are free "disposable" database servers like db4free and even MonoQL.
As far as the web-based interfaces and short URLs go, I don't think you'll have much luck.
To manage your data you have to stick to what is provided (usually phpMyAdmin or similar) and there is no short-URL to query mapping. One other caveat of such system is that (without the appropriate user permissions) one user could easily destroy all your test data -- and remember that (relational) database versioning is much more expensive than plain text versioning, so that's pretty much out of the question.
For non-RDBMS, I can think of try.mongodb.org -- but it suffers from the same problems.
Almost forgot, the Stack Exchange Data Explorer, lets you practice T-SQL queries (with permalinks).
PS: As a personal side-note, I think it's a cool idea and I would love to see something like that implemented, perhaps even mashed-up with SchemaBank or similar - that would be just awesome.
You can't really test a query without the right underlying dbms, schemas (or databases), tables, constraints, stored procedures, and permissions, which tend to be highly application specific. (That is, not readily reusable among multiple users.)
Instead, the database world has grown up into database management systems that you can freely download and install locally. Then you can build and populate your own tables, and test your queries however you like.
Most of these come with both a command line interface and some kind of graphical interface. It's not clear to me what a web-based interface would give you that doesn't already exist in one form or another.
I think that, to do what you want, would require commercial licenses for Oracle, DB2, SQL Server, and Sybase. That's a pretty high barrier to entry for a free web site.
Trouble with a web based query analyser is that you'd need to let it 'tunnel' on to your box to run the queries and for many making a development/test box open to the internet is not a possibility.
For a non web based tool you could look at LinqPad http://www.linqpad.net/ - it does Linq & Sql and other stuff too - very handy tool indeed
I need to give users the ability to build a simple SQL query against our database. Our application is written in Delphi.
I am assuming only moderate levels of knowledge by the user, but they need the ability to build a simple select statement to be able to query against a couple of tables. If I can make this easy for them, that would be most wondrous.
Does anyone know of a tool or a set of components that I can use to help the users build SQL SELECT statements...
I've used the DevExpress ExpressFilter control to allow users to specify SQL where constraints before.
I've come across FastQueryBuilder http://fast-report.com/en/products/visual-query-builder.html
and
OpenQueryBuilder http://fast-report.com/en/products/free-query-builder.html
(apparently differing versions of the same tool) at Fast-Report.
It looks like the right idea. Anyone have any experience with it?
I've used SimpleQuery for years and am quite happy with it. (http://devtools.korzh.com/eq/vcl/) It's not completely intuitive, but once you get the hang of it, you can present fields to the user from your database(s) and they can combine any series of them with ANDs and ORs to make very complex queries. You get a separate SQL window to save the resulting code. I make The resulting dataset available for printout with PrintDat! or to save as CSV data from the Woll2Woll Infopower grid that I use for display. I occasionally allow for XLS export too with XLSReadWrite. It's all quite automatic at this point. Just take the SimpleQuery example and mold that to your particular needs. You should be able to have something very usable in a day. There is a trial and an example free query builder standalone program that will give you an idea of the final look.
TMS Query Studio? Good value at 75 Euro..
Try EMS Advanced Query Builder.
is a powerful component suite for
Borland® Delphi® and C++ Builder®
intended for visual building SQL
statements for the SELECT, INSERT,
UPDATE and DELETE clauses. It allows
you to build new queries visually
and/or graphically represent the
existing queries in your own
applications. The suite includes
components for working with standard
SQL, MS SQL, InterBase/Firebird,
MySQL, PostgreSQL and many more
dialects. Advanced Query Builder
enables users to make up large and
complicated SQL queries with unions
and subqueries for different servers
without any knowledge of the SQL
syntax.
(source: sqlmanager.net)
(source: sqlmanager.net)
I think the most powerful one is ActiveQueryBuilder, which now included with Delphi IDE (Since 2007 I think)
(source: activequerybuilder.com)
I've written a couple of similar things in Delphi. It's easy enough to allow the user to pick table and column names by querying the metadata and using lists and tree views for display. The difficulty comes when trying to implement things like joins. I've never come up with a good interface for this, and in my experience neither have many of the major data tools players.
You may want to try this free builder:
GSC Visual Query Builder v.0.7
Visual Query Builder v.1.0
and some other in Torry:
http://www.torry.net/pages.php?id=546
At my work everyone has sql snippets that they use to answer questions. Some are specific to a customer, while some are generic for a given database. I want to consolidate those queries into a library/repository that can be accessed by anyone on the team. The requirements would be:
Accessible
Searchable
Tagable (multiple tags allowed per sql)
Exportable (create a document containing all queries with certain tags)
I'm interested in what has been found to work in other team environments.
You could use a wiki.
You could get started with something as simple as Tiddly wiki.
A wiki is a great approach.
For database specific or project specific snippets it's also very useful to have links to where a similar construct occurs in the code. We use trac's wiki which gives nice integration with out SVN for this.
Rather than pasting SQL snippets, I would consider graduating to an ORM (Object-Relational Mapper) or some other library to make representing and manipulating the data easier. It provides a layer of encapsulation to guard against schema changes and a layer of abstraction so you can think of the data in terms of business logic (ie. a user) rather than a collection of tables (ie. a user table, a password table, an access table...).
In Perl this would be something like DBIx::Class.
Another approach you may want to look at is creating views in your database. 'select * from some_view' can hide quite a bit of SQL. You'll still want to use a wiki to document them, but if its a view you don't have to worry about people keeping outdated copies.