dbGetQuery in R truncates really long the SQL query (of length 10564 characters) - sql

I have been trying to execute a really long SQL query and get the result into a data frame in R. But the following line throws an error and displays partial query in the console (truncates it)
my_dataframe <- dbGetQuery(conn, my_large_query)
The length of the query is 10564 characters where I use a lot of CTEs.
I have removed the string truncation via options menu in R studio but I wonder if there a character limitation in the dbGetQuery function in R?
Any suggestions?
RDMBS: DB2 (on IBM AS400),
R Package: DBI (library - RJDBC)

You mentioned changing the 'string truncation', but how about warning.length?
options("warning.length"={integer})
I'd also suggest that you test the query out first in whatever GUI is available for your database. When you verify there's no problem with the query, then run it in R.
When dbGetQuery (assuming the DBI package) throws an sql/database error, the actual content of the error won't be shown until after the full text of the query. In other words, if the query text is being truncated in the R error output, the database error code/text coming will not be visible at all. The issue may be something extremely simple like an unmatched bracket or a missing comma.
Please note whether the error message starts with Error in dbGetQuery (R issue) or Error in .verify.JDBC.result (database issue).

Related

SQLSyntaxErrorException Using LTRIM to trim character 'x' in query

I using TRIM function to trim some characters in query, I using hibernate following is my query.
from ABean s where s.cId in (select ca.id from CBean ca where LTRIM(ca.refNumber,'0') = LTRIM('$ref$','0') and ca.valid = 0)
$ref$ is replace with actual value in query.
I am seeing a different behaviour when I am running with DB2 and When I am running with Mockito test (Using In memory DB).
With DB2 this query is working fine but with Mockito in memory db I am getting java.sql.SQLSyntaxErrorException, Error is something like this.
Syntax error: Encountered "," at line 1, column {column_number_in_actual_query}.
I am not able to make it working with in memory db, Is there anything wrong I am doing?
Thanks.
in IBM DB2, in the SYSIMB schema, LTRIM takes a second argument of characters being trimmed like you have (see here). However, in the SYSFUN schema (and in most other SQL implementations) it only takes one argument and assumes you are trimming whitespace (see here).
Based on the error it looks like the interpreter wasn't expecting a comma, so it's probably trying to use the more standard version of the function and failing when it sees the second argument.
based on the documentation for function references you should be able to replace LTRIM with SYSIBM.LTRIM

Blind SQL Injection Guidance

So I'm doing an exercise for class and I'm having a bit of trouble understanding this particular database that I'm meant to break into blindly.
The database throws the following error with the string:
x'
Error Type:
Microsoft OLE DB Provider for ODBC Drivers (0x80004005)
[Microsoft][ODBC driver for Oracle][Oracle]ORA-01756: quoted string not properly terminated
showing that it is vulnerable.
Similarly, the database concatenates fine with a valid input aka valid'||'input returns the correct webpage for the input.
What confuses me is that the database does not throw an error when the input x' -- is entered, but when the input x'; -- is entered the db throws the following error:
Error Type: Microsoft OLE DB Provider for ODBC Drivers (0x80040E14)
[Microsoft][ODBC driver for Oracle][Oracle]ORA-00911: invalid character
I've also tried URL encoding the input so that it reads x'%3b+-- or x'%3b -- but it returns the same result.
Does anyone have any clue where to step next since it seems that I can't inject a semicolon ; into a query?
Everything's an Edit Below This:
Edit 1: I got to thinking and thought I might be inside of a parenthetical block. I tried the input x'); and it produces the following error:
Error Type:
Microsoft OLE DB Provider for ODBC Drivers (0x80040E14)
[Microsoft][ODBC driver for Oracle][Oracle]ORA-00933: SQL command not properly ended
Edit 2:
Found out that the statement validinput'order by 52-- produced a result, but validinput'order by 53-- produced an error. I concluded the table has 52 columns.
I'm attempting validinput' union select 1 from table_name now, but it feels largely like a huge guessing game. I don't know what any of the table names could possibly be.
Edit 3:
My brain is honestly hurting at this point but I think I'm almost there...
The statement
validnum'+union+select+null,2,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null+from+dual--
returned a legitimate page
Edit 4:
Followed Jeffrey Kemp's suggestion and I got a single table (the output seems to be limited to 1 row). Uh oh. More information though, the version is Oracle 9i, and I know the database name plus user name, and few of the tables through blind luck. The goal of the exercise is to change a value in one of the tables. However, I've run into difficulty getting the column names since the output is limited to 1 row. Any suggestions?

RODBC Error "Some part of your SQL statement is nested too deeply"

I have come across an error several times when working with R, that the RODBC package can't execute an SQL query string, but when I type the exact same string directly to a SQL Server query it works. Note that my strings contained umlauts.
I'm answering this question myself to help others avoid long internet searches if instead it can be simply reduced to this.
Almost always it was just an UNICODE error. Using umlauts or other non-unicode symbols in a R-string with the RODBC package produces this kind of error. So before trying to break it up into sub-queries as suggested by the error statement, check if your string contains only unicode characters.
If not, then the query is really to complex and needs to be split into sub-queries. For this, please refer to the other questions about this topic.

PostgreSQL Driver Appends Hidden Characters On Some SQL Statements

I have an application that I am building (Spring Boot, PostgreSQL); I am executing sql statements from a file sqlStatements.sql against the database. In some instances, it appears that the driver is appending a white-space character to the front of insert statements and when trying to create a function, it does not occur when defining tables though. I used the trim() method on the sql string to no avail. When logging the sql statements, there is no white-space character, but when looking at the PostgreSQL logs, sure enough, it's there. Is this a known issue with the driver? Is there a setting on the database side I need to change? Or do I need to do extract work in Java to avoid this error?
The space disappeared when pasting it here, so I added it; the sql files are all UTF-8.
Application log:
org.postgresql.util.PSQLException: ERROR: syntax error at or near
"insert" Position: 1
PostgreSQL log:
2017-03-04 10:29:26 EST ERROR: syntax error at or near " insert" at
character 1
Any feedback would be appreciated.
the file encoding was different between the various sql files. I had to set the encoding to "UTF-8" instead of "UTF-8 with BOM".
https://apple.stackexchange.com/questions/46586/textedit-removes-byte-order-mark-bom-from-unicode-utf-files-how-to-fix

Get ADODB to parse unicode in where clause, prefix with N

I'm updating one of our older applications to support unicode. It's littered with literal sql queries (SELECT * FROM ...) against the ADODB libraries. Most of this is working OK until I get to the SEEK code. We have some methods that transform user data into a where clause. So it parses a dictionary and produces a string. If the user is looking for something with unicode data I need to append the statement with an N. However, when I add this to the text and pass the command through the recordset.open, ADODB throws an exception :
Failed to Parse SQL
Existing SQL output:
SELECT * FROM ACCOUNT WHERE NAME ='ӐӕӔҔҖӸӽӳ'
this goes through the record set, but fails to retrieve any data.
What I need, but is throwing the exception:
SELECT * FROM ACCOUNT WHERE NAME = N'ӐӕӔҔҖӸӽӳ'
Is there some special magic I need to do to the record set to allow the N through? Or is there another approach I need to take for this to work? (and yes, I realize we SHOULD be parameterizing the queries and that's the long-term fix, but this is spaghetti code).