I am using Groovy's Sql object to perform queries on a postgres db. The queries are being executed as follows:
List<Map> results = sql.rows("select * from my_table")
List<Map> result2= sql.rows("select * from my_second_table")
I have a groovy method that performs two queries and then does some processing to loop through the data to make a different dataset, however, on some occasions I recieve a postgres exception "This ResultSet is closed" error.
having searched, I originally thought it might be to do with the issue here: SQLException: This ResultSet is closed (running multiple queries and trying to access the data from the resultsets after the fact) - however, we only seem to get the exception on quite high load - which suggests that it isnt as simple as the first dataset is closed on executing the second query as if this was the case I would expect it to happen consistently.
Can anyone shed any light on how Groovy's Sql object handles these situations or suggest what might be going wrong?
Groovy SQL is kind of a weird cat. Easy to use for simple stuff. If you have more complex scenarios you probably are better off using something else. IMHO
I first suggest doing one query, storing the results into a collection, do the second query and store the results in a collection and then do your operations between two collections rather than result sets. If you data is too large for that, find some way to store the data locally before you start doing your aggregation or whatever.
If you don't like that, you might need to checkout the GDK source code to get a better idea what is done with the Sql.getInstance() related to result sets etc. Then you can sidestep whatever land mine you are inadvertently stepping on.
Perhaps
List<Map> results = sql.rows("select * from my_table")
List<Map> result2= sql.rows("select * from my_second_table")
will not work even in plain Java (as already said in the answer you provided when second call is made on statement all resources dedicated during the previous call have to be released). As mentioned by #Todd W Crone Groovy can optimize resources, e.g. release them dynamically or don't release them depending on certain run.
Actually I've tried with only one query. E.g. I've tried to get ResultSet and then iterate through it, like this (don't mind the names of table and field, query is rather simple; and result is one row that contains one column due to LIMIT 1 clause):
def resultSet = sql.executeQuery("SELECT age FROM person WHERE id = 12345 LIMIT 1")
resultSet.next()
and got This ResultSet is closed error. Seems that Groovy optimizes resources and closes ResultSet immediately. I didn't look into the source code of Groovy SDK. I found that eachRow and other methods with closure-style handlers work fine and don't throw This ResultSet is closed error.
Perhaps, methods with closure-style handlers can help you. For example, look at except from the article where rows() method with closure is used:
String query = 'select id as identifier, name as langName from languages'
def rows = db.rows(query, { meta ->
assert meta.tableName == 'languages'
assert meta.columnCount == 2
// ...
})
Related
I'm using Sybase and had some code that looked like this:
String[] ids = ... an array containing 80-90k strings, which is retrieved from another table and varies.
for (String id : ids) {
// wrap every id with single-quotes
}
String idsAsString = String.join(",", ids);
String query = String.format("select * from someTable where idName in (%s)", idsAsString);
getNamedParameterJDBCTemplate().query(query, resultSetExtractor ->{
// do stuff with results
});
I've timed how long it took to get to the inner body of the resultSetExtractor and it never took longer than 4 seconds.
But to secure the code, I tried going the bind variable route. Thus, that code looked like the following:
String[] ids = ... an array containing 80-90k strings, which is retrieved from another table and varies.
String query = "select * from someTable where idName in (:ids)";
Map<String, Object> params = new HashMap<>();
params.put("ids", Arrays.asList(ids));
getNamedParameterJDBCTemplate().query(query, params, resultSetExtractor ->{
// do stuff with results
});
But doing it this way will take up to 4-5 minutes to finally spew out the following exception:
21-10-2019 14:04:01 DEBUG DefaultConnectionTester:126 - Testing a Connection in response to an Exception:
com.sybase.jdbc4.jdbc.SybSQLException: The token datastream length was not correct. This is an internal protocol error.
I also have other bits of code where I pass in arrays of sizes 1-10 as bind variables and noticed that those queries went from being instantaneous to taking up to 10 seconds.
I'm surprised doing it the bind variable way is at all different, let alone that drastically different. Can someone explain what is going on here? Is it that bind variable does something different underneath the hood as opposed to sending a formatted string through JDBC? Is there another way to secure my code without drastically slowing performance?
You should verify what's actually happening at the database end via a showplan/query plan, but using an 'in' clause will at best usually do one index search for every value in the 'in' clause, therefore 10 values does ten searches, 80k searches does 80k of them and thus massively slower. Oracle actually prohibits putting more than 1000 values in an 'in clause and whilst Sybase is not so restrictive that doesn't mean its a good idea. You risk stack and other issues in your database by putting massive amounts of values in this way I've seen such a query take out a production database instance with a stack failure.
It's much better to create a temporary table, load the 80k values into there and do an inner join between the temporary table and the main table using the column which previously you searched with the in clause.
When you read in a result set in Groovy it comes in a collection of maps.
Seems like you should be able to update values inside those maps and write them back out, but I can't find anything built into groovy to allow me to do so.
I'm considering writing a routine that allows me to write a modified map by iterating over the fields of one of the result objects, taking each key/value pair and using them to create the appropriate update statement, but it could be annoying so I was wondering if anyone else had done this or if it'sa vailable already in groovy.
It seems like just a few lines of code so I'd rather not bring in hibernate for this. I'm just thinking a little "update" method that would allow:
def rows=sql.rows(query)
rows[0].name="newName"
update(sql, rows[0])
to update the first guy's name in the database. Anyone seen/created such a monster, or is something like this already built into Groovy Sql and I'm just missing it?
(I suppose you may have to point out to the update method which field is the key field, but that's doable...)
Using the rows method will actually read out all of the values into a List of GroovyRowResult so it's not really possible to update the data without creating an update method like the one you mention.
It's not really possible to do that in the generic case because your query can contain joins or a column reference that is an aggregate, etc.
If you're selecting from a single table use the Sql.eachRow method however and set the ResultSet to be an updatable one, you can use the underlying ResultSet interface to update as you iterate through:
sql.resultSetConcurrency = ResultSet.CONCUR_UPDATABLE
sql.resultSetType = ResultSet.TYPE_FORWARD_ONLY
sql.eachRow(query) { row ->
row.updateString('name', 'newName')
row.updateRow()
}
Depending on the database/driver you use, you may not be able to create an updatable ResultSet.
I'm building an interface on Flask using SQLAlchemy and part of it is a search API. Essentially a type-ahead input is calling the server with its value (for example an email) and then the server is performing a SQLalchemy query using .like in a filter like below
q = session.query(User).filter(User.email.like('%'+term+'%')).all()
This query isn't really returning anything useful and after the first few characters, nothing at all. But if I perform the same query with the term hardcoded, like so:
q = session.query(User).filter(User.email.like('%mysearchterm%')).all()
It will return results perfectly fine, so there's something with how I'm putting the term into the like() method but I really can't figure out what the issue is. The term is coming in from an ajax POST and the value is there on the server-side, just .like() isn't using it correctly.
By "nothing useful" I mean that the first sets of results coming back have nothing to do with the actual term being inputted, after a term of length higher than 3-4 no results are returned back despite matching items existing in the DB.
Any help greatly appreciated.
Issues been resolved. This query sat inside of a larger query builder function which applied limit and offset to the query later in the function, because the limit and offset were higher than the amount of results back the returned result set was empty.
Can chock this one up to bad human error and lack of sleep.
I have tested and done quite some research online, but still no luck. Did anyone ever encounter this problem ?
Say, I have a doctrine query set up like:
$q = Doctrine_Query::create()
->update('PckFolder')
->set('id_path', "CONCAT(?, RIGHT(id_path, LENGTH(id_path)-?))", array($newPath, $lenOld))
->where("id_path like '$oldPath%'");
// and I print the query out
$qstr = $q->getSqlQuery(array($newPath, $lenOld));
Instead of giving me:
UPDATE pck_folder SET id_path = CONCAT(?, RIGHT(id_path, LENGTH(id_path)-?)) WHERE (id_path like '1/2//%')
Doctrine gave me:
UPDATE pck_folder SET id_path = CONCAT(?, RIGHT(id_path, LENGTH(id_path-?))) WHERE (id_path like '1/2//%')
please note this part RIGHT(id_path, LENGTH(id_path)-?)
(Note: I'm assuming you're using Doctrine 1.2. I haven't used Doctrine 2.0 yet.)
I had not encountered that specific bug before, but I have found numerous problems with the implementation of update() in Doctrine_Query. Essentially anything but the most very straightforward update queries will cause the parser to generate wrong or invalid queries. For example, it can't handle sub-selects within an update.
Try writing a Raw SQL query, or else use a less efficient but fully-functional workaround: Select the records you want to update using Doctrine_Query, then iterate over them and set the field in PHP, then call save() on each one.
By the way, there's a big GOTCHA inherent with use of UPDATE queries and Doctrine that sort of forces you to use that workaround in many cases anyway. That is, if you or your plugins have made use of the nifty Doctrine hook methods within your models but you execute a SQL-level update that affects those records, the hooks will get silently circumvented. Depending on your application, that may wreck your business logic processing.
I'm developing a system (with Rails 2.3.2, Ruby 1.8.7-p72) that has a sizable reporting component. In order to improve performance, I've created a Report model to archive old reports. The idea is that if a matching report already exists for an arbitrary set of conditions then use it, otherwise generate the report and save the results.
Moreover, I'd like to design the Report model in such a way that only the requested attributes have their corresponding SQL queries run. This all stems from the fact that each attribute takes a long time to compute and I'd rather not generate results that won't be used. I.e. I would like to do something like:
def foo
#foo ||= read_attribute(:foo)
if #foo.nil?
#foo = write_attribute(:foo, (expensive SQL query result))
end
#foo
end
The problem I'm experiencing, however, is that results aren't being properly written out to my database and, as a result, the code is constantly reevaluating the SQL query.
Can anyone tell me why write_attribute isn't working? Furthermore, is there a better approach?
Turns out that what I was doing was fine. The real problem was that the object's "id" lookup was being trumped by a piece of code I had elsewhere. I.e. the actual write was occurring to the database, but with the wrong primary key.
Don't you need to call "save" after doing write_attribute?