Complex search across grails domain classes - sql

I want to be able to do complex searches on grails objects. This is presently implemented using stored procedures on my site that build up a SQL query and parses all of those results.
Will something like the searchable plugin allow me to simplify this task? My guess is not, since it mostly is doing text based searching. The stored procedures are quite complex, and hard to change. Our users are employees, and the queries have to do with amounts of job experience and who they were working for, what skills they have, etc. The Employee domain object would have things like a list of roles that contain skills. The role would have a start and end date, etc.
A list of example queries:
All users with 5 years of experience in C++
All users who have worked for Stackoverflow, in California
All users who have at least 5 years of C++ experience, at least 2 years of Java experience, have worked for StackOverflow, and are available to work now.

I've never tried the searchable plugin so may be selling it short. Your best bet is probably HQL queries or Hibernate criteria builder. I like HQL for complex queries since it's similar to SQL. For a blog post comparing the use of these technologies from Grails see
http://blog.xebia.com/2008/06/04/querying-associations-in-grails-with-hql-criteria-and-hibernatecriteriabuilder/
For the HQL reference see
http://docs.jboss.org/hibernate/core/3.3/reference/en/html/queryhql.html
For Hibernate criteria see
http://docs.jboss.org/hibernate/core/3.3/reference/en/html/querycriteria.html

You could also have a look at the FilterPane plugin to help you (or the user) to build these queries. There is a GUI that can handle most of such questions, or you could resort to add the FilterPane fields yourself in the HTML and controller if you prefer to have the control over the queries.
FilterPane then translates the fields to Hibernate criteria builder internally.

All of the queries you have listed can be accomplished with the searchable plugin.
I believe you CAN do the queries you have suggested with HQL but Compass/Lucene is really the better tool for that solution

In my experience HQL queries were the only solution for complex queries.
Some queries even made it necessary to use non-Hibernate functions of the underlying DB, e.g. setting dialect = "org.hibernate.dialect.ExtendedMySqlDialect" in DataSource.groovy and then implementing something like this:
package org.hibernate.dialect;
import org.hibernate.Hibernate;
import org.hibernate.dialect.function.*;
public class ExtendedMySqlDialect extends MySQL5InnoDBDialect {
public ExtendedMySqlDialect() {
registerFunction("timeStampAdd", new SQLFunctionTemplate(Hibernate.TIMESTAMP, "TIMESTAMPADD(?1, ?2, ?3)"));
registerFunction("timeStampDiff", new SQLFunctionTemplate(Hibernate.INTEGER, "TIMESTAMPDIFF(?1, ?2, ?3)"));
}
The above functions might also help you when dealing with dates.

Related

Is there an existing piece of software that allows you to (easily) build queries throught a webpage?

I would like to build arbitrary queries to a database, by allowing the user to build queries "on the fly". For every object/table, being able to select its attributes, and then "building" the query (that would translate into a SQL statement) and finally launching it, all through a web interface.
The ticketing system "rt" does that, for example, and another example would be the http://gatherer.wizards.com/Pages/Advanced.aspx webpage.
I'm currently programming in rails but any existing solution that implements this (or something similar) would be welcome.
Just be careful when creating dynamically generated queries like this that will need to be executed via sp_executesql (example: ms sql server), etc..... make sure you cover all of your bases to ensure that your application isnt vulnerable to SQL injection attacks as this type of development will essentially get one in a lot of trouble if its done incorrectly.. I would recommend storing all queries in a table and only reading queries from this table to help isolate the queries that are being ran in your application. Just identify them with a label, and allow the EU to choose the label from a dropdown list control on the frontend.
Good luck and I'm not sure of any software that will help assist
Not quite sure what your use case is here but i would say check out the
Doctrine ORM ( Object Relational Mapper )
**Edit
After reading more and looking at the example. I would only suggest Doctrine for a large website.
Then use Doctrines DQL syntax with some javascript/jquery magic for the forms.
Note that the queries you're referencing aren't arbitrary: they're on a very specific problem domain, on a specific set of sql tables.
That said, if I were you I'd look into how people are building sql queries with javascript. Something like these:
http://code.google.com/p/django-querybuilder/
http://css.dzone.com/articles/sqlike-sql-querying-engine?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+zones%2Fria+(RIA+Zone)
http://thechangelog.com/post/4914956307/rel-arel-ported-to-node-js-with-some-changes
That'll at least get you a good idea of the underlying data structures.

Is there any framework for parsing a SQL-like query into its component parts?

I'm interested in writing a SQL-like query syntax for a CMS I work with. The idea would be that a CMS query could be written in a SQL-ish syntax, and I would convert that to execute through the CMS API.
There would be no field or table selection, so I need some way to get from this:
SELECT WHERE Something = 'something' AND (SomethingElse != 'something' OR AnotherThing == 'something')
Essentially then, I need some way to get the WHERE clauses grouped correctly based on their parentheticals and AND/ORs.
Is there some framework for doing this? Some example of when it's been done? I don't want to re-invent the wheel here, and I know someone else has to have done this in the past.
The answer is yes, there are many frameworks that work in an analog of SQL and convert to SQL. Linq and various Linq translators are a prime example. Knowing exactly which CMS you're working with, and thus which language and platform you're developing in, would be helpful. Some .NET ORMs that support code queries are:
NHibernate - allows use of a SQL-ish language called HQL in strings, or more code-based query construction using expression lists and Linq.
Linq2SQL - On its way out, but for your simpler applications it should be fine. The framework generates DAO classes that map between tables and your domain objects, and you can use coded Linq queries to work with the classes very much like the real tables.
And of course you can use good ol' vanilla ADO.NET with a string SQL query. This has numerous drawbacks, but if you want to have queries in your code, why not make them real SQL? If you wanted to hide your table structure, you could translate table names before submitting queries, so the SQL contained at the web layer (shudder) won't run against your DB.

SQL Query builder in Delphi

I need to give users the ability to build a simple SQL query against our database. Our application is written in Delphi.
I am assuming only moderate levels of knowledge by the user, but they need the ability to build a simple select statement to be able to query against a couple of tables. If I can make this easy for them, that would be most wondrous.
Does anyone know of a tool or a set of components that I can use to help the users build SQL SELECT statements...
I've used the DevExpress ExpressFilter control to allow users to specify SQL where constraints before.
I've come across FastQueryBuilder http://fast-report.com/en/products/visual-query-builder.html
and
OpenQueryBuilder http://fast-report.com/en/products/free-query-builder.html
(apparently differing versions of the same tool) at Fast-Report.
It looks like the right idea. Anyone have any experience with it?
I've used SimpleQuery for years and am quite happy with it. (http://devtools.korzh.com/eq/vcl/) It's not completely intuitive, but once you get the hang of it, you can present fields to the user from your database(s) and they can combine any series of them with ANDs and ORs to make very complex queries. You get a separate SQL window to save the resulting code. I make The resulting dataset available for printout with PrintDat! or to save as CSV data from the Woll2Woll Infopower grid that I use for display. I occasionally allow for XLS export too with XLSReadWrite. It's all quite automatic at this point. Just take the SimpleQuery example and mold that to your particular needs. You should be able to have something very usable in a day. There is a trial and an example free query builder standalone program that will give you an idea of the final look.
TMS Query Studio? Good value at 75 Euro..
Try EMS Advanced Query Builder.
is a powerful component suite for
Borland® Delphi® and C++ Builder®
intended for visual building SQL
statements for the SELECT, INSERT,
UPDATE and DELETE clauses. It allows
you to build new queries visually
and/or graphically represent the
existing queries in your own
applications. The suite includes
components for working with standard
SQL, MS SQL, InterBase/Firebird,
MySQL, PostgreSQL and many more
dialects. Advanced Query Builder
enables users to make up large and
complicated SQL queries with unions
and subqueries for different servers
without any knowledge of the SQL
syntax.
(source: sqlmanager.net)
(source: sqlmanager.net)
I think the most powerful one is ActiveQueryBuilder, which now included with Delphi IDE (Since 2007 I think)
(source: activequerybuilder.com)
I've written a couple of similar things in Delphi. It's easy enough to allow the user to pick table and column names by querying the metadata and using lists and tree views for display. The difficulty comes when trying to implement things like joins. I've never come up with a good interface for this, and in my experience neither have many of the major data tools players.
You may want to try this free builder:
GSC Visual Query Builder v.0.7
Visual Query Builder v.1.0
and some other in Torry:
http://www.torry.net/pages.php?id=546

How do you maintain a library of useful SQL in a team environment?

At my work everyone has sql snippets that they use to answer questions. Some are specific to a customer, while some are generic for a given database. I want to consolidate those queries into a library/repository that can be accessed by anyone on the team. The requirements would be:
Accessible
Searchable
Tagable (multiple tags allowed per sql)
Exportable (create a document containing all queries with certain tags)
I'm interested in what has been found to work in other team environments.
You could use a wiki.
You could get started with something as simple as Tiddly wiki.
A wiki is a great approach.
For database specific or project specific snippets it's also very useful to have links to where a similar construct occurs in the code. We use trac's wiki which gives nice integration with out SVN for this.
Rather than pasting SQL snippets, I would consider graduating to an ORM (Object-Relational Mapper) or some other library to make representing and manipulating the data easier. It provides a layer of encapsulation to guard against schema changes and a layer of abstraction so you can think of the data in terms of business logic (ie. a user) rather than a collection of tables (ie. a user table, a password table, an access table...).
In Perl this would be something like DBIx::Class.
Another approach you may want to look at is creating views in your database. 'select * from some_view' can hide quite a bit of SQL. You'll still want to use a wiki to document them, but if its a view you don't have to worry about people keeping outdated copies.

What to do with queries which don´t have a representation in a domain model?

This is not specific to any language, it´s just about best practices. I am using JPA/Hibernate (but it could be any other ORM solution) and I would like to know how do you guys deal with this situation:
Let´s suppose that you have a query returning something that is not represented by any of your domain classes.
Do you create a specific class to represent that specific query?
Do you return the query in some other kind of object (array, map...)
Some other solutions?
I would like to know about your experiences and best practices.
P.S.
Actually I am creating specific objetcs for specific queries.
We have a situation that sounds similar to yours.
We use separate objects for reporting data that spans several domain objects. Our convention is that these will be backed by a view in the database, so we have come to call them view objects. We generally use them for summarising complex data into a flat format.
I typically write a function that performs a query using SQL and then puts the results into either a list or dictionary (in Java, I'd use either an ArrayList or a HashMap).
If I found myself doing this a lot, I'd probably create a new file to hold all of these queries. Otherwise I'd just make them functions in whatever file they were needed/used.
Since we're talking Java specifically, I would certainly not create a new class in a separate file. However, for queries needed in only one class, you could create a private static inner class with only the function(s) needed to generate the query(s) needed by that class.
The idea of wrapping that up the functionality in some sort of manager is always nice. It allows for better testing, and management therefore of schema changes.
Also allows for easier reuse in the application. NEVER just put the sql in directly!!!. For Hibernate I have found HQL great for just this. In particular , if you can use Named queries. Also be careful of adding an filter values etc use "string append", use parameters (can we say SQL injection ?). Even if the SQL is dynamic in terms of the join or where criteria, have a function in some sort of manager is always best.
#DrPizza
I will be more specific. We have three tables in a database
USER
PROJECT
TASK
USER to TASK 1:n
PROJECT to TASK 1:n
I have a query that returns a list of all projects but showing also some grouped information (all tasks, open tasks, closed tasks). When returned, the query looks like this
PROJECTID: 1
NAME: New Web Site
ALLTASK: 10
OPENTASK: 7
CLOSEDTASK: 3
I don´t have any domain class that could represent this information and I don´t want to create specific methods in Project class (like getAllTasks, getOpenTasks) because each of these methods would trigger a new query.
So the question is:
I create a new class (somenthing like ProjectTasksQuery) just to hold that information?
I return information within array or map?
Something else?
You might feel better after reading about Data Transfer Objects. Some people plain don't like them, but if it feels like a good fit to you, it probably is.