I am querying a postgres db using python/pandas with sqlalchemy. I have an sql query that looks like this:
SELECT table_name
FROM fixed_602fcccd0f189c2434611b14.information_schema."tables" t
WHERE table_type='BASE TABLE'
and table_schema='di_602fccd10f189c2434611be9'
and (
table_name like 'general_journal%'
or table_name like 'output_journal_line%'
or table_name like 'output_journal_tx%'
or table_name like 'flats%'
)
I've tested it in dBeaver and it works perfectly. I am now trying to pass the same query through pandas read_sql as follows:
from_string = pg_db + '.information_schema."tables"'
print(from_string)
pg_query = queries.id_tables.format(from_string,di_id)
The idea is that I construct the query with variables 'pg_db' (string) and 'di_id' (string) as I make a series of queries. The problem is the query returns empty array when done this way. No error is thrown.
I suspected the challenge is the "tables" attribute that when pandas interprets the query eg. strips off the ", but that doesn't actually seem to matter. Any thoughts on how to make this work with pandas?
UPDATE:
I have tried parameterized and met with the same problem. It seems to boil down to the FROM parameter gets passed in with double quotes. I have tried to strip these but it looks like pandas appends them anyways. In principle double quotes should be fine according to postgres docs but that doesn't seem to be the case even when doing the query in dBeaver. If I pass in the query via pandas as it is written at the top of this post, no problem. The challenge is when I try to use variables for the FROM and table_schema parameters, I get syntax errors.
It turns out that the problem disappeared when I removed the parentheses I put around the 'or' statements. I think the message is to pay attention to how you construct the query eg. form and join all the strings and variables before passing them to pandas.
That said I have used parentheses with much more complex queries in pandas and they were not a problem.
I would first suggest that you use a parameterized query for input but in some cases its just easier to use a built in function repr()
s = "SQL sometimes likes \"here\" and %s \"now\" problems"
print(repr(s))
gives
'SQL sometimes likes "here" and %s "now" problems'
Related
I'm trying to speed up a query that uses Contains Near with one that uses regexp_like. The initial Contains Near query takes about 45 minutes to run. Clob Column holds large "documents" and is domain indexed.
Initial query:
SELECT column1
FROM TEST
WHERE CONTAINS(column1,'{NEAR(quick,fox, lazy), 3, FALSE}')>0;
Proposed query:
SELECT column1
FROM TEST
WHERE REGEXP_LIKE(column1, '(\b(quick|fox|lazy)(?:\W+\w+){1,6}?\W(quick|fox|lazy)(?:\W+\w+){1,}?\W(quick|fox|lazy)\b)','i')
I got the original regexp syntax from here:
https://www.regular-expressions.info/near.html.
Problem:
I get the regexp code to work in html https://www.regextester.com, but when I put it in Oracle it doesn't find anything. What is wrong with my syntax? I can't figure it out. Does Oracle handle REGEXP differently?
Alex, you were exactly right. I don't see how to select your answer as correct though.
My problem was apparently that I was using regexp parameters that Oracle doesn't recognize. So, whereas it worked on https://www.regextester.com, it failed to work in Oracle because most of what I used isn't recognized as usable with regexp in Oracle. I really think Oracle should expand their regexp codes it recognized. This was really frustrating.
This code:
cursor.execute('select RLAMBD from ?', OPTable)
print cursor.fetchone().RLAMBD
produces this error:
ProgrammingError: ('42S02', '[42S02] [Oracle][ODBC][Ora]ORA-00903: invalid table name\n (903) (SQLExecDirectW)')
OPTable is an alphanumeric string which I've built from another database query which contains the table name I want to select from.
The following code works just fine within the same script.
sql = 'select RLAMBD from ' + OPTable
cursor.execute(sql)
print cursor.fetchone().RLAMBD
I guess it's not a huge deal to build the sql statements this way, but I just don't understand why it's not accepting the ? parameters. I even have another query in the same script which uses the ? parameterization and works just fine. The parameters for the working query are produced using the raw_input function, though. Is there some subtle difference between the way those two strings might be formatted that's preventing me from getting the query to work? Thank you all.
I'm running python 2.7 and pyodbc 3.0.10.
Parameter placeholders cannot be used to represent object names (e.g., table or column names) or SQL keywords. They are only used to pass data values, e.g., numbers, strings, dates, etc..
I have a simple sql query in orientdb function like this:
select *
from TestExecutionPlanReport
where executionPlan IN :executionPlans
It accepts "executionPlans" as parameter where an array value should be passed.
When I tried just one value like "#59:71", it works and return the wanted output. BUT when I tried to pass in an array value like "[#59:71,#59:214]", is not working. It returns an empty response. It works when I query "select * from SomeClass where field IN [#59:71,#59:214]" (Not in OrientDb function)
I had the same issue when trying to run an SQL query from Scala code. I solved it by passing a set as the parameter instead of a list or an array.
If I got your question right, you are trying to execute the query you've mentioned in a OrientDB server side function (probably written in JavaScript) which takes executionPlan as a parameter.
I also noticed this happening if you use parameterized queries. A parameterized query is something like the following;
var query = "select * from TestExecutionPlanReport where executionPlan IN ?";
return db.query(query, executionPlans);
However, when I used string concatenation to build the query, it works. I'm referring to the following
var query = "select * from TestExecutionPlanReport where executionPlan IN " + executionPlans
If you are using string concatenation I think the issue is with how you pass the parameter. Rather than passing "[#59:71,#59:214]", try passing [#59:71,#59:214] to your function and see if it works.
Usually when you use string concatenation to build queries, you can use the print() function to print the concatenated query on the OrientDB console. However, concatenation based queries could be exploited to do SQL injections, hence discouraged.
I cannot explain why this doesn't work for parameterized queries. I'll do more research on this. Perhaps it's a bug. I'm using OrientDB 2.0.3.
Hope this helps.
The website i worked was recently attempted to be hacked by the following SQL injection script
boys' and 3=8 union
select 1,
concat(0x232425,ifnull(`table_name`,0x30),char(9),ifnull(`table_rows`,0x30), char(9),0x252423),
3,4,5,6,7,8,9
from `information_schema`.`tables`
where table_schema=0x62646B3032 limit 44,1 -- And '8'='8
This injection returned the mysql table name. This was reported by the error reporting system on that website and we managed to fix that part however I am not able to understand what does the above injection mean?
Anyone can explain this?
Penuel
They're using a select from the Information Schema views in mysql server :
http://dev.mysql.com/doc/refman/5.0/en/information-schema.html
They use some clever hacks to rout out simple sql injection prevention techniques.
According to this the MySQL concat()
Returns the string that results from
concatenating the arguments. May have
one or more arguments. If all
arguments are nonbinary strings, the
result is a nonbinary string. If the
arguments include any binary strings,
the result is a binary string. A
numeric argument is converted to its
equivalent binary string form
So 0x232425 is converted to #$% which is simply added to the begining and end of the table_name field. Maybe just to make it easier for them to pull out the Table names later using Regex.
Later on the char(9) is equivalent to a tab as you can see here and is just there to format the output nicer.
The 3,4,5,6,7,8,9 is just there so that the columns match the boys table that they are performing the Union on.
This injection returned the mysql table name.
Do you mean that your website displayed the table name when you gave it this input, or that the query returns that when run from the mysql client? If it showed on your website, then the attacker has the ability to inject much more harmful queries. Check your data.
This question already has answers here:
How to anticipate and escape single quote ' in oracle
(2 answers)
Closed 7 years ago.
I have a database with names in it such as John Doe etc. Unfortunately some of these names contain quotes like Keiran O'Keefe. Now when I try and search for such names as follows:
SELECT * FROM PEOPLE WHERE SURNAME='O'Keefe'
I (understandably) get an error.
How do I prevent this error from occurring. I am using Oracle and PLSQL.
The escape character is ', so you would need to replace the quote with two quotes.
For example,
SELECT * FROM PEOPLE WHERE SURNAME='O'Keefe'
becomes
SELECT * FROM PEOPLE WHERE SURNAME='O''Keefe'
That said, it's probably incorrect to do this yourself. Your language may have a function to escape strings for use in SQL, but an even better option is to use parameters. Usually this works as follows.
Your SQL command would be :
SELECT * FROM PEOPLE WHERE SURNAME=?
Then, when you execute it, you pass in "O'Keefe" as a parameter.
Because the SQL is parsed before the parameter value is set, there's no way for the parameter value to alter the structure of the SQL (and it's even a little faster if you want to run the same statement several times with different parameters).
I should also point out that, while your example just causes an error, you open youself up to a lot of other problems by not escaping strings appropriately. See http://en.wikipedia.org/wiki/SQL_injection for a good starting point or the following classic xkcd comic.
Oracle 10 solution is
SELECT * FROM PEOPLE WHERE SURNAME=q'{O'Keefe}'
Parameterized queries are your friend, as suggested by Matt.
Command = SELECT * FROM PEOPLE WHERE SURNAME=?
They will protect you from headaches involved with
Strings with quotes
Querying using dates
SQL Injection
Use of parameterized SQL has other benefits, it reduces CPU overhead (as well as other resources) in Oracle by reducing the amount of work Oracle requires in order to parse the statement. If you do not use parameters (we call them bind variables in Oracle) then "select * from foo where bar='cat'" and "select * from foo where bar='dog'" are treated as separate statements, where as "select * from foo where bar=:b1" is the same statement, meaning things like syntax, validity of objects that are referenced etc...do not need to be checked again. There are occasional problems that arise when using bind variables which usually manifests itself in not getting the most efficient SQL execution plan but there are workarounds for this and these problems really depend on the predicates you are using, indexing and data skew.
Input filtering is usually done on the language level rather than database layers.
php and .NET both have their respective libraries for escaping sql statements. Check your language, see waht's available.
If your data are trustable, then you can just do a string replace to add another ' infront of the ' to escape it. Usually that is enough if there isn't any risks that the input is malicious.
I suppose a good question is what language are you using?
In PHP you would do: SELECT * FROM PEOPLE WHERE SURNAME='mysql_escape_string(O'Keefe)'
But since you didn't specify the language I will suggest that you look into a escape string function mysql or otherwise in your language.
To deal quotes if you're using Zend Framework here is the code
$db = Zend_Db_Table_Abstract::getDefaultAdapter();
$db->quoteInto('your_query_here = ?','your_value_here');
for example ;
//SELECT * FROM PEOPLE WHERE SURNAME='O'Keefe' will become
SELECT * FROM PEOPLE WHERE SURNAME='\'O\'Keefe\''
Found in under 30s on Google...
Oracle SQL FAQ