We use Flask+SQLAlchemy in our web app. We found some slow sql queries in DB monitor. Is there any good way to find/locate the original SQLAlchemy queries according to generated sql? i.e. code mapping between ORM code and SQL code.
e.g. Add any unique token in generated sql comment, so we can find original ORM code by the token.
You can add sql comment with suffix_with method
query = Table.query
query = query.suffix_with('-- %s' % query_token)
Related
Hello for demonstration purposes I trimmed out my actual sql query.
I have a SQL query
SELECT *
FROM dbdev.training.courses
where dbdev is my DEV database table name. When I migrate to TEST env, I want my query to dynamically change to
SELECT *
FROM dbtest.training.courses
I tried using input parameters like {env: p('db_name')} and using in the query as
SELECT * FROM :env.training.courses
or
SELECT * FROM (:env).training.courses
but none of them worked. I don't want my SQL query in properties file.
Can you please suggest a way to write my SQL query dynamically based on environment?
The only alternative way is to deploy separate jars for different environments with different code.
You can set the value of the property to a variable and then use the variable with string interpolation.
Warning: creating dynamic SQL queries using any kind of string manipulation may expose your application to SQL injection security vulnerabilities.
Example:
#['SELECT * FROM $(vars.database default "dbtest").training.courses']
Actually, you can do a completely dynamic or partially dynamic query using the MuleSoft DB connector.
Please see this repo:
https://github.com/TheComputerClassroom/dynamicSQLGETandPATCH
Also, I'm about to post an update that allows joins.
At a high level, this is a "Query Builder" where the code that builds the query is written in DataWeave 2. I'm working on another version that allows joins between entities, too.
If you have questions, feel free to reply.
One way to do it is :
Create a variable before DB Connector:
getTableName - ${env}.training.courses
Write SQL Query :
Select * from $(getTableName);
When trying to do an SQL injection on an Oracle SQL database I have the problem that most of the examples in the tutorials do not work. I already found out that I only can use CASE WHEN a THEN b ELSE c END instead of normal if statements.
The question I have now is how do I get time delay into the injection? Benchmark() and sleep() do not work either.
I already now that the table is named "flag" and the field name I want to read out is named "password".
My only information i get from the database is the time it needed to execute my input (or query since I bypass the input to inject SQL)
I found the following SQL statement on the web at SQL Injection Tutorial
select dbms_pipe.receive_message(('a'),10) from dual;
I am not certain I should be participating in this sort of thing, but since I found it with my first Google Search, I will go ahead and post it.
I tested it and it delayed the result by 10 seconds.
In Django if I want to see the raw sql in debug mode I can check using in django shell:
from django.db import connections
User.objects.all()
print(connections['default'].queries[-1]['sql'])
Similarly can we see the raw response of that sql. In the above case the sql query may return the raw results in csv, tab delimited format. From then django may create the model objects array.
There are better tools to debug SQL queries in Django. The standard tool is to use Django-debug-toolbar, which allows you to view both the SQL query, the result, and the EXPLAIN output, for all queries in a request/response, along with the time needed for each query. The documentation is available at https://django-debug-toolbar.readthedocs.io/en/latest/
you can do:
str(User.objects.all().query)
# or
print(User.objects.all().query)
# or
print(User.objects.all().query.sql_with_params())
I am a SQL Server DBA for a company that sells an ASP.NET MVC3 application that uses LINQ and Entity Framework 4 for all database access. When I find an inefficient query in my SQL Server's plan cache that was generated by LINQ, I would like to be able to find that LINQ statement in the source code so that I can optimize it. What is the best way to find the LINQ that generated a given SQL statement?
For example, is there any way to put an entry in a config file or decorate the code somehow so that the class and method name or the LINQ statement itself are included as comments in the generated SQL?
The commercial tools ORM Profiler, Entity Framework Profiler or Hugati Query Profiler will both give you a stack trace for the methods which generated the SQL. That makes it fairly easy to find the LINQ in code, though it isn't displayed directly.
These tools also have the advantage that they make it easy to find inefficient queries amongst the many other SQL statements executed by the app.
Although it is not a free tool, this may provide the information you need:
http://efprof.com/
There is also a less expensive tool described here, which I have not used, but it looks very promising:
http://huagati.blogspot.com/2010/06/entity-framework-support-in-huagati.html
http://www.huagati.com/L2SProfiler/
I bet Entity Framework Profiler (http://efprof.com/) would help you out. The workflow is very different from what you asked for (which would be pretty cool BTW). It is a good tool, and is worth a look even if it's not your final solution.
Good luck!
If you have access to the ASP.NET code where the LINQ code is you can more or less know which query you are looking for, copy it into a freeware tool called LINQPad and run it directly there to get the generated SQL statements. http://www.linqpad.net/
You need first get the LINQ queries on your .net code, create a connection to your datasource, paste the Linq code in new queries and run them. You will get the SQL Query generated from the LINQ code.
For example:
from e in ETUSERs
where e.LoginName.Contains("a")
orderby e.LoginName
select e
SQL Results Tab:
-- Region Parameters
DECLARE #p0 VarChar(1000) = '%a%'
-- EndRegion
SELECT [t0].[UserID], [t0].[UsrFirstName], [t0].[UsrLastName], [t0].[LoginName], [t0].[Location], [t0].[Password], [t0].[UsrEmail], ...
FROM [ETUSER] AS [t0]
WHERE [t0].[LoginName] LIKE #p0
ORDER BY [t0].[LoginName]
This is probably not exactly what you are looking for, but it is worth knowing about this tool since it is very helpful to quickly test LINQ queries. There you can quickly edit and run to improve the code without recompiling the whole stuff.
I don't think you can modify the generated SQL easily but what you can do is to get the generated SQL before sending the query to the database.
So you can log every query in a separate textfile with timestamp and source code context information. But that means to modify each place in your source where LINQ queries are sent to the database. Maybe there is an extension point somewhere in the DataContext class for simplifying this.
Anyway here is the code to get the corresponding sql query for a LINQ query:
YourDataContext dc = new YourDataContext();
IQueryable<YourEntityClass> query =
from e in dc.YourEntities
where ...
select e;
string command = dc.GetCommand(query).CommandText;
I'd like to "dry-run" Hibernate HQL queries. That is I'd like to know what actual SQL queries Hibernate will execute from given HQL query without actually executing the HQL query against real database.
I have access to hibernate mapping for tables, the HQL query string, the dialect for my database. I have also access to database if that is needed.
Now, how can I find out all the SQL queries Hibernate can generate from my HQL without actually executing the query against any database? Are there any tools for this?
Note, that many SQL queries can be generated from one HQL query and the set of generated SQL queries may differ based on the contents of database.
I am not asking how to log SQL queries while HQL query is executing.
Edit: I don't mind connecting to database to fetch some metadata, I just don't want to execute queries.
Edit: I also know what limits and offsets are applied to query. I also have the actual parameters that will be bind to query.
The short answer is "you can't". The long answer is below.
There are two approaches you can take:
A) Look into HQLQueryPlan class, particularly its getSqlStrings() method. It will not get you the exact SQL because further preprocessing is involved before query is actually executed (parameters are bound, limit / offset are applied, etc...) but it may be close enough to what you want.
The thing to keep in mind here is that you'll need an actual SessionFactory instance in order to construct HQLQueryPlan, which means you won't be able to do so without "connecting to any database". You can, however, use in-memory database (SqlLite and the likes) and have Hibernate auto-create necessary schema for it.
B) Start with ASTQueryTranslatorFactory and descend into AST / ANTLR madness. In theory you may be able to hack together a parser that would work without relying on metadata but I have a hardest time imagining what is it you're trying to do for this to be worth it. Perhaps you can clarify? There has to be a better approach.
Update: for an offline, dry-run of some HQL, using HQLQueryPlan directly is a good approach. If you want to intercept every query in the app, while it's running, and record the SQL, you'll have to use proxies and reflection as described below.
Take a look at this answer for Criteria Queries.
For HQL, it's the same concept - you have to cast to Hibernate implementation classes and/or access private members, so it's not a supported method, but it will work with a the 3.2-3.3 versions of Hibernate. Here is the code to access the query from HQL (query is the object returned by session.createQuery(hql_string):
Field f = AbstractQueryImpl.class.getDeclaredField("session");
f.setAccessible(true);
SessionImpl sessionImpl = (SessionImpl) f.get(query);
Method m = AbstractSessionImpl.class.getDeclaredMethod("getHQLQueryPlan", new Class[] { String.class, boolean.class });
m.setAccessible(true);
HQLQueryPlan plan = (HQLQueryPlan) m.invoke(sessionImpl, new Object[] { query.getQueryString(), Boolean.FALSE });
for (int i = 0; i < plan.getSqlStrings().length; ++i) {
sql += plan.getSqlStrings()[i];
}
I would wrap all of that in a try/catch so you can go on with the query if the logging doesn't work.
It's possible to proxy your session and then proxy your queries so that you can log the sql and the parameters of every query (hql, sql, criteria) before it runs, without the code that builds the query having to do anything (as long as the initial session is retrieved from code you control).