This question already has an answer here:
DB2 error Improper use of a string column, host variable, constant, or function
(1 answer)
Closed 8 months ago.
I´ve been facing some weird issues when studding SQL raw queries using SqlAlchemy.
sqlstr = 'SELECT "City" from CHICAGO_SCHOOLS;'
with engine.connect() as conn:
result = conn.execute(text(sqlstr))
print (result.all())
The query above returns hundreds of "Chicago" as results. So I just tried to get unique results:
sqlstr = 'SELECT DISTINCT "City" from CHICAGO_SCHOOLS;'
with engine.connect() as conn:
result = conn.execute(text(sqlstr))
print (result.all())
Now, all I got is a weird error :
Exception: SQLNumResultCols failed: [IBM][CLI Driver][DB2/LINUXX8664]
SQL0134N Improper use of a string column, host variable, constant, or
function "City". SQLSTATE=42907
At first I thought it was somehow related to the DISTINCT set quantifier. So I tried the same query with another column.
sqlstr = 'SELECT DISTINCT "School ID" from CHICAGO_SCHOOLS;'
with engine.connect() as conn:
result = conn.execute(text(sqlstr))
print (result.all())
And in this query I got all expected results.
I am not being able to truly understand what is wrong!
The issue was related to the column type. It was a CLOB type and that does not allow use of DISTINCT. Thanks to HoneyBadger
Related
I'm afraid that if a bunch of folks start running my actual code I'll be billed for the queries so my example code is for a fake database.
I've successfully established my connection to BigQuery:
con <- dbConnect(
bigrquery::bigquery(),
project = 'myproject',
dataset = 'dataset',
billing = 'myproject'
)
Then performed a LEFT JOIN using the coalesce function:
dbGetQuery(con,
"SELECT
`myproject.dataset.table_1x`.Pokemon,
coalesce(`myproject.dataset.table_1`.Type_1,`myproject.dataset.table_2`.Type_1) AS Type_1,
coalesce(`myproject.dataset.table_1`.Type_2,`myproject.dataset.table_2`.Type_2) AS Type_2,
`myproject.dataset.table_1`.Total,
`myproject.dataset.table_1`.HP,
`myproject.dataset.table_1`.Attack,
`myproject.dataset.table_1`.Special_Attack,
`myproject.dataset.table_1`.Defense,
`myproject.dataset.table_1`.Special_Defense,
`myproject.dataset.table_1`.Speed,
FROM `myproject.dataset.table_1`
LEFT JOIN `myproject.dataset.table_2`
ON `myproject.dataset.table_1`.Pokemon = `myproject.dataset.table_2`.Pokemon
ORDER BY `myproject.dataset.table_1`.ID;")
The JOIN produced the table I intended and now I'd like to query that table but like...where is it? How do I connect? Can I save it locally so that I can start working my analysis in R? Even if I go to BigQuery, select the Project History tab, select the query I just ran in RStudio, and copy the Job ID for the temporary table, I still get the following error:
Error: Job 'poke-340100.job_y0IBocmd6Cpy-irYtNdLJ-mWS7I0.US' failed
x Syntax error: Unexpected string literal 'poke-340100:US.bquxjob_7c3a7664_17ed44bb4ae' at [2:6] [invalidQuery]
Run `rlang::last_error()` to see where the error occurred.
And if I follow up:
Run `rlang::last_error()` to see where the error occurred.
> rlang::last_error()
<error/rlang_error>
Job 'poke-340100.job_y0IBocmd6Cpy-irYtNdLJ-mWS7I0.US' failed
x Syntax error: Unexpected string literal 'poke-340100:US.bquxjob_7c3a7664_17ed44bb4ae' at [2:6] [invalidQuery]
Backtrace:
1. DBI::dbGetQuery(con, "SELECT *\nFROM 'poke-340100:US.bquxjob_7c3a7664_17ed44bb4ae'\nWHERE Type_1 IS NULL;")
2. DBI::dbGetQuery(con, "SELECT *\nFROM 'poke-340100:US.bquxjob_7c3a7664_17ed44bb4ae'\nWHERE Type_1 IS NULL;")
3. DBI:::.local(conn, statement, ...)
5. bigrquery::dbSendQuery(conn, statement, ...)
6. bigrquery:::BigQueryResult(conn, statement, ...)
7. bigrquery::bq_job_wait(job, quiet = conn#quiet)
Run `rlang::last_trace()` to see the full context.
> rlang::last_trace()
<error/rlang_error>
Job 'poke-340100.job_y0IBocmd6Cpy-irYtNdLJ-mWS7I0.US' failed
x Syntax error: Unexpected string literal 'poke-340100:US.bquxjob_7c3a7664_17ed44bb4ae' at [2:6] [invalidQuery]
Backtrace:
x
1. +-DBI::dbGetQuery(con, "SELECT *\nFROM 'poke-340100:US.bquxjob_7c3a7664_17ed44bb4ae'\nWHERE Type_1 IS NULL;")
2. \-DBI::dbGetQuery(con, "SELECT *\nFROM 'poke-340100:US.bquxjob_7c3a7664_17ed44bb4ae'\nWHERE Type_1 IS NULL;")
3. \-DBI:::.local(conn, statement, ...)
4. +-DBI::dbSendQuery(conn, statement, ...)
5. \-bigrquery::dbSendQuery(conn, statement, ...)
6. \-bigrquery:::BigQueryResult(conn, statement, ...)
7. \-bigrquery::bq_job_wait(job, quiet = conn#quiet)
Can someone please explain? Is it just that I can't query a temporary table with the bigrquery package?
From looking at the documentation here and here, the problem might just be that you did not assign the results anywhere.
local_df = dbGetQuery(...
should take the results from your database query and copy them into local R memory. Take care as there is no check for the size of the results, so it is easy to run out of memory in when doing this.
You have tagged the question with dbplyr, but it looks like you are just using the DBI package. If you want to be writing R and have it translated to SQL, then you can do this using dbplyr. It would look something like this:
con <- dbConnect(...) # your connection details here
remote_tbl1 = tbl(con, from = "table_1")
remote_tbl2 = tbl(con, from = "table_2")
new_remote_tbl = remote_tbl1 %>%
left_join(remote_tbl2, by = "Pokemon", suffix = c("",".y")) %>%
mutate(Type_1 = coalesce(Type_1, Type_1.y),
Type_2 = coalesce(Type_2, Type_2.y)) %>%
select(ID, Pokemon, Type_1, Type_2, ...) %>% # list your return columns
arrange(ID)
When you use this approach, new_remote_tbl can be thought of as a new table in the database which you can query and manipulate further. (It is not actually a table - no data was saved to disc - but you can query it and interact with it as if it were and the database will produce it for you on demand).
There are some limitations of working with a remote table (the biggest is you are limited to commands that dbplyr can translate into SQL). When you want to copy the current remote table into local R memory, use collect:
local_df = remote_df %>%
collect()
I have a table I am trying to call with my usual method
sql <- 'SELECT TOP 10 *
FROM "_SYS_BIC"."data-path.self-service.DOIP/table_name"'
df <- dbGetQuery(jdbcConnection, sql)
and receive the error
Error in .verify.JDBC.result(r, "Unable to retrieve JDBC result set for ", :
Unable to retrieve JDBC result set for SELECT TOP 10 *
FROM "_SYS_BIC"."data-path.self-service.DOIP/table_name" (SAP DBTech JDBC: [2048]: column store error: search table error: [34023] Instantiation of calculation model failed;exception 306106: Undefined variable: $$IP_ExtractionWeekFrom$$. Variable is marked as required but not set in the query)
I've been trying to insert IP_ExtractionWeekFrom into the sql statement with a where clause with no luck
param1 <- 201943
sql <- 'SELECT TOP 10 *
FROM "_SYS_BIC"."ccf-edw.self-service.DOIP/R_CA_B_DemandPlan" where
"$$IP_ExtractionWeek$$" = ?'
SpringVisit <- dbGetQuery(jdbcConnection, sql, param1)
I've tried the term surrounded by the "$$" and without, and both with and without "$$" sourrounded in quotes and not. Usually am met with an "invalid column name" error.
Is this supposed to be called with something other than a where clause?
Consider maintaining your working Tableau query with the integration of parameters in R with properly handling of double quotes for identifiers and single quotes for literals.
Additionally, parameterization is not supported with the old ('PLACEHOLDER'= ('<varname>', <varvalue>)) syntax.
Instead, as explained in How to escape sql injection from HANA placeholder use the PLACEHOLDER."<varname>" => ? syntax.
param1 <- 201943
sql <- "SELECT TOP 10 *
FROM \"_SYS_BIC\".\"ccf-edw.self-service.DOIP/R_CA_B_DemandPlan\"(
PLACEHOLDER.\"$$IP_ExtractionWeekFrom$$\", ?),
PLACEHOLDER.\"$$IP_ExtractionWeekTo$$\",?)
)\"_SYS_BIC\".\"ccf-edw.self-service.DOIP/R_CA_B_DemandPlan\"
WHERE (1 <> 0)"
SpringVisit <- dbGetQuery(jdbcConnection, sql, param1, param1)
Additionally, if your JDBC already connects to the schema_SYS_BIC, use the synonymous qualifier :: as original query in order to reference package and calculation view:
sql <- "SELECT TOP 10 *
FROM \"ccf-edw.self-service.DOIP::R_CA_B_DemandPlan\"(
PLACEHOLDER.\"$$IP_ExtractionWeekFrom$$\", ?),
PLACEHOLDER.\"$$IP_ExtractionWeekTo$$\", ? )
)\"ccf-edw.self-service.DOIP::R_CA_B_DemandPlan\"
WHERE (1 <> 0)"
I'm trying to retrieve some data from a postgresql database using psycogp2, and either exclude a variable number of rows or exclude none.
The code I have so far is:
def db_query(variables):
cursor.execute('SELECT * '
'FROM database.table '
'WHERE id NOT IN (%s)', (variables,))
This does partially work. E.g. If I call:
db_query('593')
It works. The same for any other single value. However, I cannot seem to get it to work when I enter more than one variable, eg:
db_query('593, 595')
I get the error:
psycopg2.DataError: invalid input syntax for integer: "593, 595"
I'm not sure how to enter the query correctly or amend the SQL query. Any help appreciated.
Thanks
Pass a tuple as it is adapted to a record:
query = """
select *
from database.table
where id not in %s
"""
var1 = 593
argument = (var1,)
print(cursor.mogrify(query, (argument,)).decode('utf8'))
#cursor.execute(query, (argument,))
Output:
select *
from database.table
where id not in (593)
I'm trying to write an write this:
Team.last.players.sum("goals")
erb:
SELECT SUM("players"."goals")
FROM "players"
WHERE "players"."team_id" = $1 [["team_id", 2]]
how to rewrite this so that I could use it in a method:
def sql_search
sql = "SELECT SUM \"players\".\"goals\" FROM \"players\" WHERE \"players\".\"team_id\" = $1 [[\"team_id\", #{self.id}"
connection.execute(sql);
end
keep getting this error:
PG::SyntaxError: ERROR: syntax error at or near "."
LINE 1: SELECT SUM "players"."goals" FROM "players" WHERE "players"....
Any ideas would be appreciated
You don't need to add \" in sql statement, just remove them.
def sql_search
sql = "SELECT sum(goals) FROM players WHERE team_id = #{self.id};"
connection.execute(sql);
end
Is there some reason that you want to hard code the SQL query? It's generally bad practice to use string interpolation to insert parameters to SQL queries because of SQL injection attacks. Instead it's recommended to use ActiveRecord's SQL query parameter binding like this:
user_input = 5
Player.where('team_id = ?', user_input).sum(:goals)
Basically what this does is insert the parameter 5 after sanitization. This means you're safe from attacks where a hacker attempts to insert arbitrary SQL into parameter variables attempting to return sensitive data or delete data entirely!
From Java I am doing the following query on DB2:
SELECT * FROM PRV_PRE_ACTIVATION WHERE TRANSACTION_ID = ?
The field TRANSACTION_ID is a VARCHAR of length 32. I set the parameter in the preparedStatement using the setString method.
I get the error:
com.ibm.db2.jcc.am.SqlSyntaxErrorException: DB2 SQL Error: SQLCODE=-270, SQLSTATE=42997, SQLERRMC=63, DRIVER=3.59.81
at com.ibm.db2.jcc.am.dd.a(dd.java:676)
at com.ibm.db2.jcc.am.dd.a(dd.java:60)
at com.ibm.db2.jcc.am.dd.a(dd.java:127)
at com.ibm.db2.jcc.am.bn.c(bn.java:2546)
at com.ibm.db2.jcc.am.bn.d(bn.java:2534)
at com.ibm.db2.jcc.am.bn.a(bn.java:2026)
at com.ibm.db2.jcc.t4.cb.g(cb.java:140)
at com.ibm.db2.jcc.t4.cb.a(cb.java:40)
at com.ibm.db2.jcc.t4.q.a(q.java:32)
at com.ibm.db2.jcc.t4.rb.i(rb.java:135)
at com.ibm.db2.jcc.am.bn.gb(bn.java:1997)
at com.ibm.db2.jcc.am.cn.pc(cn.java:3009)
at com.ibm.db2.jcc.am.cn.b(cn.java:3786)
at com.ibm.db2.jcc.am.cn.bc(cn.java:678)
at com.ibm.db2.jcc.am.cn.executeQuery(cn.java:652)
Where the sqstate means "Capability is not supported by this version of the DB2 application requester, DB2 application server, or the combination of the two." But I don't use any strange functionality.
I have tried using an squ client the query:
SELECT * FROM PRV_PRE_ACTIVATION where transaction_id='A'
And it goes ok.
What is the cause of the problem?
UPDATE: The code where the statement is prepared:
s = con.prepareStatement(sSQL,
ResultSet.TYPE_SCROLL_INSENSITIVE,
ResultSet.CONCUR_UPDATABLE);
Try changing to a specified list of columns in the select list -- my guess is you have a user defined column type (or some other type) which is not supported by your driver. For example, does the statement
SELECT TRANSACTION_ID FROM PRV_PRE_ACTIVATION WHERE TRANSACTION_ID = ?
work? If so then start adding columns in and you will find the problem column.
I've came across this problem lately, and after some searching on web, I've came across this link:
DB2 SQL error: SQLCODE: -270, SQLSTATE: 42997, SQLERRMC: 63
, which specifies this:
A column with a LOB type, distinct type on a LOB type, or
structured type cannot be specified in the select-list of an
insensitive scrollable cursor.
With help from an colleague, we came to this conclusion:
1, Q: When will you get this "SQLCODE=-204, SQLSTATE=42704" exception?
A: When a scrollable PreparedStatement is prepared & executed, yet there are [B|C]LOB fields exist in the select list. e.g.:
String strQuery = "SELECT NUMBER_FIELD, CHAR_FIELD, CLOB_FIELD FROM TABLE_NAME WHERE CONDITION IS TRUE;"
Statement stmt = conn.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE, REsultSet.CONCUR_READ_ONLY);
rs = stmt.executeQuery(strQuery); //and this exception will be thrown here
2, Q: So what's the solution if we want to get rid of it when [B|C]LOB fields are queried?
A: Try to use ResultSet.TYPE_FORWARD_ONLY while creating the query statement.e.g.:
stmt = conn.createStatement(ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY);
Or simply try this one:
stmt = conn.createStatement();
Note that the same rules apply to conn.prepareStatement() too. You may refer to Java API doc for more information.