I have a very expensive subquery that I use for multiple queries in my repository.
I want to find a way to cache that subquery and use it's results in the other native sql queries.
I am using the Doctrine2 DBAL in a symfony2 project.
Native SQL is required due to the complexity of the subquery.
example:
select sum(volume) from ( expensive_subquery );
I have been reviewing Google, and the Doctrine2 documentation but have not found a good solution here.
Other methods are welcome, staying in the confines of the Doctrine2, Symfony2, SQL framework.
You have a lot of options. My first line of action would be to hash the select statement to act as a key and store the results in memcached or something similar. Then in your manager before your run the query. Check to see if the key/cache exists.. If it does,use the cached results.. If the cache didn't exist, create it after the long query runs.. Additionally, you might want to integrate cache expiration.
Related
Usually when I finish a project, I like to run queries in SQL to see if any of my queries from the code are more expensive than they should be (basically, I want to make sure I didn't forget to put an index somewhere). However, my company / the DBAs will not grant me permission to VIEW SERVER STATE, so I can't run the query I need to.
I am using Code-First EF, and am curious if there is any way to add some kind of "tracking" within Entity Framework to see what queries I might be able to optimize / apply indexing to a table somewhere. Any thoughts?
I'm working on a system which will fetch data from a service and put pieces of the response in to a cache and/or into a SQL table.
The cache is needed for consumption by other Java services directly. These services require a more direct connection than the SQL abstraction, so we need to connect directly to the cache.
The table is needed for a JDBC SQL connection to external SQL clients e.g. SQL Workbench, DBeaver, Tableau, 3rd party systems.
My question is how Ignite works regarding caches vs tables. I know it stores its caches as maps similar to other IMDGs. What I guess I don't understand is how that gets turned into a table, or what APIs are available to set/get between the two.
So the question is, how can I take an INSERT from the JDBC/SQL side and query it via the Cache? How can I add() into the Cache and SELECT it from the JDBC/SQL side? If I have a table named "foo", does that also create a cache named "foo"?
Or am I supposed to use one or the other and not bleed between the two? I haven't found many good examples of this, so it seems to be either you use caches or you use tables.
It would be extremely advantageous to have a bridge between the two. We're migrating to Ignite from an H2 implementation where we mushed a Hazelcast cache and H2's SQL together and are hoping Ignite, being built atop H2, has done something similar already.
In particular, I was hoping to use DataStreamers but I'm not finding much in the way of how it relates to the SQL/table side of things.
Ignite cache falls under key-value type of nosql database. You can fire SQL like query from java code to ignite caches as it supports it. For example,
SELECT _KEY, _VAL from "foo".val
Here, foo is your cache name and val is the value part of key-value pair. As this is all NOSQL, relating it to RDBMS SQL is not so much rational, still we can relate all non primary columns in SQL table to the fields of your value object and primary one to the key part.
So, in datastreamer, you can construct collection of key, value objects and stream it. This internally calls nothing but put operation on cache.
To select in SQL fashon, you can fire query like below-
SqlFieldsQuery query = new SqlFieldsQuery(queryString);
FieldsQueryCursor<List<?>> cursor = cache.query(query);
There are multiple ways to do this, SqlFieldsQuery is one of that.
This was answered couple times already, basically you need to refer to Query Entities, Indexed Types or key_type/value_type parameters of CREATE TABLE to make it work. I.e. every entry in cache of correct type will be a row of table and vice versa.
When I troubleshoot a large .NET app which uses only stored procedures, I capture the sql which includes the SP name from SQL Server Profiler and then it's easy to do a global search for the SP in the source files and find the exact line which produced the SQL.
When using Entity Framework, this is not possible due to the dynamic creation of SQL statements. However there are times when I capture some problematic sql statements from production and want to know where in the code they were generated from.
I know one can have EF generate logs and tracing on demand. This probably would be taxing for a busy server and produces too much logs. I read some stuff about using mini profiler but not sure if it fits my needs as I don't have access to the production server. I do however have access to attach SQL Server Profiler to the database server.
My idea is to find a way to have EF attach/inject a unique code to the generated SQL but it doesn't affect the outcome of the SQL. I can then use it to cross reference it to the line of code which injected it into the SQL. The unique code is static which means a unique static code is used for every EF linq statement. Maybe sent as a dummy sql or a comment along with the sql statement.
I know this will add some extra traffic but in my case, it will add extra flexibility and cut a lot of troubleshooting time.
Any ideas of how to do this or any alternatives?
One very simple approach would be to execute something via ExecuteStoreCommand(): Refresh data from stored procedure. I'm not sure if you can "execute" just a comment, but at the very least you should be able to do something like:
ExecuteStoreCommand("DECLARE #MyTag VARCHAR(100) = 'some_unique_id';");
This is very simple, but you would have to find the association in two steps:
Get the SessionID (i.e. SPID) from poorly performing query in SQL Server Profiler
Search the Profiler entries for the prior SQL statement for that same SPID
Another option that might be a little more complicated but would remove that additional step when it comes to making that association is to "intercept" the commands before they get executed and inject a comment with your unique id. Please see the following S.O. Answer for details. You shouldn't need the full extent of what they did, but even if you do, it seems like all of the code (or all the relevant stuff) is there:
Adding a query hint when calling Table-Valued Function
By the way, this situation is a point in favor of using Stored Procedures instead of an ORM. And, what do you expect to be able to do in terms of performance tuning once you do find the offending app code? (another point in favor of using Stored Procedures instead of an ORM ;-).
I am trying to identify slow queries in a large-scale Django 1.3 web application. As it is kind of difficult to match the raw sql query in the slow query log with the specific ORM statement in the code, I wondered if it is possible to add a SQL comment to the query constructed with the ORM, something like..
Object.objects.filter(Q(pub_date__lte=datetime.now)).comment('query no. 123')
Solution found by using .extra() for raw SQL commands on the django-user mailinglist:
Object.objects.filter(Q(pub_date__lte=datetime.now()).extra(where=['1=1 /* query no. 123 */'])
For those reading in 2022 onwards - there is a much better answer these days:
Google's sqlcommenter project has a Django middleware
[A] Django middleware whose purpose is to augment a SQL statement right before execution, with information about the controller and user code to help with later making database optimization decisions, after those statements are examined from the database server’s logs.
We're using NHibernate with Memcache as the second level cache. Occasionally there is a need for more advanced queries or bulk query operations. From the book Nhibernate in Action they recommend the following:
"It’s our view that ORM isn’t suitable for mass-update (or mass-delete) operations. If
you have a use case like this, a different strategy is almost always better: call a stored
procedure in the database, or use direct SQL UPDATE and DELETE statements for that
particular use case."
My concern is that queries against the underlying database do not reflect in the cache (at least until cache expiry) and I was wondering if anyone has come up with any effective strategies for mixing and matching NHibernate with custom SQL statements?
Is there any way of getting say a bulk Update statement (executed with custom sql) to reflect in the second level cache? I am aware of being able to manually evict, but this removes the items from cache and thefore increases hits on the database.
Does the community have any solutions that have been found to be effective in dealing with this problem?
As far as I know there is no method to keep the 2nd level cache up to date with massupdates. But you can partially evict the cache as described in: http://www.nhforge.org/doc/nh/en/index.html#performance-sessioncache.