I have come across an error several times when working with R, that the RODBC package can't execute an SQL query string, but when I type the exact same string directly to a SQL Server query it works. Note that my strings contained umlauts.
I'm answering this question myself to help others avoid long internet searches if instead it can be simply reduced to this.
Almost always it was just an UNICODE error. Using umlauts or other non-unicode symbols in a R-string with the RODBC package produces this kind of error. So before trying to break it up into sub-queries as suggested by the error statement, check if your string contains only unicode characters.
If not, then the query is really to complex and needs to be split into sub-queries. For this, please refer to the other questions about this topic.
Related
So I've been scouring SO for answer and I've seen some great SQL functions to help try and remove non-ascii characters from my db, but I wanted to post the entire question / process here first to see if maybe upstream on my select from db2 into sql there is a fix.
What I'm doing: Getting data from a db2 database into SQL
Issue: Non-ascii characters causing problems
Process: It's pretty simple. I have a SQL Insert statement to select a bunch of columns from a db2 linkedserver using open query
insert into [table](stuff) select (stuff) From Openquery(SSF400,'select stuff from table')
However, in my SQL db, when editing the landed table, I'm getting weird trailing characters that appear as a space in a sql select statement, but are actually artifacts in SQL Edit mode:
I've tried using a few functions I found here on SO to strip these characters, but after these function(s) I'm leftover with a combination of greek/english characters similar to the below:
I'm thinking there must be a better way for me to do the initial insert other than using openquery so that the junk characters don't come over. I know SQL pretty well, but DB2 not so much...any advice?
Update: There does seem to be a junk character or two in the source system. Discovered using iNavigator. Also, source system is using db2 v7r3m0
Update here is a screenshot of the regexp expression mentioned in the comments used in a query in iNavigator. Although several characters were removed, some do remain. The original column is on the left, the cleansed column is on the right.
Cheers,
MD
I would try REGEXP_REPLACE(stuff,'[^\u0020-\u007E\u0009\u000A\u000D]+','') which will remove everything that is not a character from the 7-bit ASCII set but also removes any 7-bit ASCII control characters apart from Tab, New Line and Carriage Return. It also removes DEL
I have been trying to execute a really long SQL query and get the result into a data frame in R. But the following line throws an error and displays partial query in the console (truncates it)
my_dataframe <- dbGetQuery(conn, my_large_query)
The length of the query is 10564 characters where I use a lot of CTEs.
I have removed the string truncation via options menu in R studio but I wonder if there a character limitation in the dbGetQuery function in R?
Any suggestions?
RDMBS: DB2 (on IBM AS400),
R Package: DBI (library - RJDBC)
You mentioned changing the 'string truncation', but how about warning.length?
options("warning.length"={integer})
I'd also suggest that you test the query out first in whatever GUI is available for your database. When you verify there's no problem with the query, then run it in R.
When dbGetQuery (assuming the DBI package) throws an sql/database error, the actual content of the error won't be shown until after the full text of the query. In other words, if the query text is being truncated in the R error output, the database error code/text coming will not be visible at all. The issue may be something extremely simple like an unmatched bracket or a missing comma.
Please note whether the error message starts with Error in dbGetQuery (R issue) or Error in .verify.JDBC.result (database issue).
I am trying to insert some information in an MS Access database.
In my database I have the following columns and types:
log_order - Autonumber (I need this to keep the order where inserted in the db),
userID - Text,
time - Text,
date_ - text,
message - Text.
My query:
command.CommandText = "INSERT INTO logs(userID, time, date_, message) VALUES ('"+verifiedUser+"', '"+msg_time+"', '"+msg_date+"', '"+msg+"')";
OleDbDataReader reader = command.ExecuteReader();
The error that I get:
System.Data.OleDb.OleDbException: 'Syntax error in INSERT INTO statement.'
I tried several posts but no post helped me. I believe there might be a problem with the autonumber column (log_order). Because of what I remember I don't have to include it in the query.
PS: I know I have to pass the values as parameters.
Thank you in advance
Probably one of your variables (msg?) contains an apostrophe
The way you've written your SQL is a massive security risk. Please immediately look up "parameterized queries" and never, ever, ever write an sql like this again (where you use string concatenation to tack the values into the query). Your code has a proliferation of issues and using parameterized queries will solve all of them; they aren't difficult to write
It seems your data in some of the variables passed in INSERT may be causing this error. Try debugging the value in command.CommandText before executing it.
If any of the variables have a single quote they must be escaped...
Ref: How do I escape a single quote in SQL Server?
Also brush up on SQL Injection Ref: SQL Injection
I totally agree with all that has been said, but to answer your question directly, I am pretty sure you will need to put square brackets around your field names. OleDb tends not to like special characters and could well be having a problem for example with date_ ; sending [date_] instead should get round the issue.
It will not like time either. Same solution
Addendum on SQL Injection
As an aside, in fact calling Access through OleDb is relatively protected from SQL Injection. This is because any attempt to execute multiple instructions in one command fails. (You get an incorrect formatted string error). So whilst you could argue that what you are doing is safe, it is not for other db providers. The sooner you get into good habits, the less likely you will be to introduce a vulnerability in a case where it could be dangerous. If it seems like you are getting a stream of abuse, it is just because everyone here wants to keep the net safe.
The website i worked was recently attempted to be hacked by the following SQL injection script
boys' and 3=8 union
select 1,
concat(0x232425,ifnull(`table_name`,0x30),char(9),ifnull(`table_rows`,0x30), char(9),0x252423),
3,4,5,6,7,8,9
from `information_schema`.`tables`
where table_schema=0x62646B3032 limit 44,1 -- And '8'='8
This injection returned the mysql table name. This was reported by the error reporting system on that website and we managed to fix that part however I am not able to understand what does the above injection mean?
Anyone can explain this?
Penuel
They're using a select from the Information Schema views in mysql server :
http://dev.mysql.com/doc/refman/5.0/en/information-schema.html
They use some clever hacks to rout out simple sql injection prevention techniques.
According to this the MySQL concat()
Returns the string that results from
concatenating the arguments. May have
one or more arguments. If all
arguments are nonbinary strings, the
result is a nonbinary string. If the
arguments include any binary strings,
the result is a binary string. A
numeric argument is converted to its
equivalent binary string form
So 0x232425 is converted to #$% which is simply added to the begining and end of the table_name field. Maybe just to make it easier for them to pull out the Table names later using Regex.
Later on the char(9) is equivalent to a tab as you can see here and is just there to format the output nicer.
The 3,4,5,6,7,8,9 is just there so that the columns match the boys table that they are performing the Union on.
This injection returned the mysql table name.
Do you mean that your website displayed the table name when you gave it this input, or that the query returns that when run from the mysql client? If it showed on your website, then the attacker has the ability to inject much more harmful queries. Check your data.
This question already has answers here:
How to anticipate and escape single quote ' in oracle
(2 answers)
Closed 7 years ago.
I have a database with names in it such as John Doe etc. Unfortunately some of these names contain quotes like Keiran O'Keefe. Now when I try and search for such names as follows:
SELECT * FROM PEOPLE WHERE SURNAME='O'Keefe'
I (understandably) get an error.
How do I prevent this error from occurring. I am using Oracle and PLSQL.
The escape character is ', so you would need to replace the quote with two quotes.
For example,
SELECT * FROM PEOPLE WHERE SURNAME='O'Keefe'
becomes
SELECT * FROM PEOPLE WHERE SURNAME='O''Keefe'
That said, it's probably incorrect to do this yourself. Your language may have a function to escape strings for use in SQL, but an even better option is to use parameters. Usually this works as follows.
Your SQL command would be :
SELECT * FROM PEOPLE WHERE SURNAME=?
Then, when you execute it, you pass in "O'Keefe" as a parameter.
Because the SQL is parsed before the parameter value is set, there's no way for the parameter value to alter the structure of the SQL (and it's even a little faster if you want to run the same statement several times with different parameters).
I should also point out that, while your example just causes an error, you open youself up to a lot of other problems by not escaping strings appropriately. See http://en.wikipedia.org/wiki/SQL_injection for a good starting point or the following classic xkcd comic.
Oracle 10 solution is
SELECT * FROM PEOPLE WHERE SURNAME=q'{O'Keefe}'
Parameterized queries are your friend, as suggested by Matt.
Command = SELECT * FROM PEOPLE WHERE SURNAME=?
They will protect you from headaches involved with
Strings with quotes
Querying using dates
SQL Injection
Use of parameterized SQL has other benefits, it reduces CPU overhead (as well as other resources) in Oracle by reducing the amount of work Oracle requires in order to parse the statement. If you do not use parameters (we call them bind variables in Oracle) then "select * from foo where bar='cat'" and "select * from foo where bar='dog'" are treated as separate statements, where as "select * from foo where bar=:b1" is the same statement, meaning things like syntax, validity of objects that are referenced etc...do not need to be checked again. There are occasional problems that arise when using bind variables which usually manifests itself in not getting the most efficient SQL execution plan but there are workarounds for this and these problems really depend on the predicates you are using, indexing and data skew.
Input filtering is usually done on the language level rather than database layers.
php and .NET both have their respective libraries for escaping sql statements. Check your language, see waht's available.
If your data are trustable, then you can just do a string replace to add another ' infront of the ' to escape it. Usually that is enough if there isn't any risks that the input is malicious.
I suppose a good question is what language are you using?
In PHP you would do: SELECT * FROM PEOPLE WHERE SURNAME='mysql_escape_string(O'Keefe)'
But since you didn't specify the language I will suggest that you look into a escape string function mysql or otherwise in your language.
To deal quotes if you're using Zend Framework here is the code
$db = Zend_Db_Table_Abstract::getDefaultAdapter();
$db->quoteInto('your_query_here = ?','your_value_here');
for example ;
//SELECT * FROM PEOPLE WHERE SURNAME='O'Keefe' will become
SELECT * FROM PEOPLE WHERE SURNAME='\'O\'Keefe\''
Found in under 30s on Google...
Oracle SQL FAQ