I am using BIDS to connect to a Progress DB through an ODBC connection:
This query works fine
SELECT
PUB."master"."app-number",
...
PUB."property"."prop-id",
FROM
PUB."master" master JOIN PUB."property" property ON
master."lt-acnt" = property."lt-acnt"
...
LEFT OUTER JOIN PUB."arm" arm ON
master."lt-acnt" = arm."lt-acnt"
WHERE
...
However, I need to add some additional fields from another table. The problem is that I only need the information from the last time these new fields were updated.
I have tried:
SELECT
yt."app-number"
...
yt."disc-adj-tot",
yt."rt-adj-nbr",
yt."base-disc-per"
FROM (
SELECT PUB."master"."app-number",
...
PUB."lt-rt-adj-hdr"."disc-adj-tot",
PUB."lt-rt-adj-hdr"."rt-adj-nbr",
PUB."lt-rt-adj-hdr"."base-disc-per"
FROM PUB."master" master JOIN PUB."property" property ON
master."lt-acnt" = property."lt-acnt"
...
JOIN PUB."lt-rt-adj-hdr" lt_rt_adj_hdr ON
lt_master."lt-acnt" = lt_rt_adj_hdr."lt-acnt") yt
INNER JOIN(
SELECT "app-number",
MAX("rt-adj-nbr") "rt-adj-nbr"
FROM ( PUB."lt-master" lt_master JOIN
PUB."lt-rt-adj-hdr" lt_rt_adj_hdr ON
lt_master."lt-acnt" = lt_rt_adj_hdr."lt-acnt")
GROUP BY "app-number") ss on yt."app-number" = ss."app-number" and
yt."rt-adj-nbr" = ss."rt-adj-nbr"
WHERE ...
This query just hangs and will not return results unless a very simple WHERE clause like "WHERE yt."app-number" = 123456" is used. I am completely stuck.
Has the owner of the Progress DB ever run "update statistics"? The Progress SQL query optimizer needs to have good statistics in order to execute efficiently. Progress applications usually use the 4GL engine rather than SQL so, in many cases, the administrator is not keeping the SQL statistics updated. Which often leads to very poor SQL query performance.
From the 4GL side the admin can use this script to generate a program that will do the job:
/* genUpdateSQL.p
*
* mpro dbName -p util/genUpdateSQL.p -param "tmp/updSQLstats.sql"
*
* sqlexp -user userName -password passWord -db dnName -S servicePort -infile tmp/updSQLstats.sql -outfile tmp/updSQLtats.log
*
*/
output to value( ( if session:parameter <> "" then session:parameter else "updSQLstats.sql" )).
for each _file no-lock where _hidden = no:
put unformatted
"UPDATE TABLE STATISTICS AND INDEX STATISTICS AND ALL COLUMN STATISTICS FOR PUB."
'"' _file._file-name '"' ";"
skip
.
put unformatted "commit work;" skip.
end.
output close.
return.
Or, you could do it if you have sufficient privileges (just plug in your table name for _file._file-name).
Related
I'm afraid that if a bunch of folks start running my actual code I'll be billed for the queries so my example code is for a fake database.
I've successfully established my connection to BigQuery:
con <- dbConnect(
bigrquery::bigquery(),
project = 'myproject',
dataset = 'dataset',
billing = 'myproject'
)
Then performed a LEFT JOIN using the coalesce function:
dbGetQuery(con,
"SELECT
`myproject.dataset.table_1x`.Pokemon,
coalesce(`myproject.dataset.table_1`.Type_1,`myproject.dataset.table_2`.Type_1) AS Type_1,
coalesce(`myproject.dataset.table_1`.Type_2,`myproject.dataset.table_2`.Type_2) AS Type_2,
`myproject.dataset.table_1`.Total,
`myproject.dataset.table_1`.HP,
`myproject.dataset.table_1`.Attack,
`myproject.dataset.table_1`.Special_Attack,
`myproject.dataset.table_1`.Defense,
`myproject.dataset.table_1`.Special_Defense,
`myproject.dataset.table_1`.Speed,
FROM `myproject.dataset.table_1`
LEFT JOIN `myproject.dataset.table_2`
ON `myproject.dataset.table_1`.Pokemon = `myproject.dataset.table_2`.Pokemon
ORDER BY `myproject.dataset.table_1`.ID;")
The JOIN produced the table I intended and now I'd like to query that table but like...where is it? How do I connect? Can I save it locally so that I can start working my analysis in R? Even if I go to BigQuery, select the Project History tab, select the query I just ran in RStudio, and copy the Job ID for the temporary table, I still get the following error:
Error: Job 'poke-340100.job_y0IBocmd6Cpy-irYtNdLJ-mWS7I0.US' failed
x Syntax error: Unexpected string literal 'poke-340100:US.bquxjob_7c3a7664_17ed44bb4ae' at [2:6] [invalidQuery]
Run `rlang::last_error()` to see where the error occurred.
And if I follow up:
Run `rlang::last_error()` to see where the error occurred.
> rlang::last_error()
<error/rlang_error>
Job 'poke-340100.job_y0IBocmd6Cpy-irYtNdLJ-mWS7I0.US' failed
x Syntax error: Unexpected string literal 'poke-340100:US.bquxjob_7c3a7664_17ed44bb4ae' at [2:6] [invalidQuery]
Backtrace:
1. DBI::dbGetQuery(con, "SELECT *\nFROM 'poke-340100:US.bquxjob_7c3a7664_17ed44bb4ae'\nWHERE Type_1 IS NULL;")
2. DBI::dbGetQuery(con, "SELECT *\nFROM 'poke-340100:US.bquxjob_7c3a7664_17ed44bb4ae'\nWHERE Type_1 IS NULL;")
3. DBI:::.local(conn, statement, ...)
5. bigrquery::dbSendQuery(conn, statement, ...)
6. bigrquery:::BigQueryResult(conn, statement, ...)
7. bigrquery::bq_job_wait(job, quiet = conn#quiet)
Run `rlang::last_trace()` to see the full context.
> rlang::last_trace()
<error/rlang_error>
Job 'poke-340100.job_y0IBocmd6Cpy-irYtNdLJ-mWS7I0.US' failed
x Syntax error: Unexpected string literal 'poke-340100:US.bquxjob_7c3a7664_17ed44bb4ae' at [2:6] [invalidQuery]
Backtrace:
x
1. +-DBI::dbGetQuery(con, "SELECT *\nFROM 'poke-340100:US.bquxjob_7c3a7664_17ed44bb4ae'\nWHERE Type_1 IS NULL;")
2. \-DBI::dbGetQuery(con, "SELECT *\nFROM 'poke-340100:US.bquxjob_7c3a7664_17ed44bb4ae'\nWHERE Type_1 IS NULL;")
3. \-DBI:::.local(conn, statement, ...)
4. +-DBI::dbSendQuery(conn, statement, ...)
5. \-bigrquery::dbSendQuery(conn, statement, ...)
6. \-bigrquery:::BigQueryResult(conn, statement, ...)
7. \-bigrquery::bq_job_wait(job, quiet = conn#quiet)
Can someone please explain? Is it just that I can't query a temporary table with the bigrquery package?
From looking at the documentation here and here, the problem might just be that you did not assign the results anywhere.
local_df = dbGetQuery(...
should take the results from your database query and copy them into local R memory. Take care as there is no check for the size of the results, so it is easy to run out of memory in when doing this.
You have tagged the question with dbplyr, but it looks like you are just using the DBI package. If you want to be writing R and have it translated to SQL, then you can do this using dbplyr. It would look something like this:
con <- dbConnect(...) # your connection details here
remote_tbl1 = tbl(con, from = "table_1")
remote_tbl2 = tbl(con, from = "table_2")
new_remote_tbl = remote_tbl1 %>%
left_join(remote_tbl2, by = "Pokemon", suffix = c("",".y")) %>%
mutate(Type_1 = coalesce(Type_1, Type_1.y),
Type_2 = coalesce(Type_2, Type_2.y)) %>%
select(ID, Pokemon, Type_1, Type_2, ...) %>% # list your return columns
arrange(ID)
When you use this approach, new_remote_tbl can be thought of as a new table in the database which you can query and manipulate further. (It is not actually a table - no data was saved to disc - but you can query it and interact with it as if it were and the database will produce it for you on demand).
There are some limitations of working with a remote table (the biggest is you are limited to commands that dbplyr can translate into SQL). When you want to copy the current remote table into local R memory, use collect:
local_df = remote_df %>%
collect()
I have several OR in my SQL statement so I want to save a chuck of it in a cfsavecontent. Here is that part:
<cfsavecontent variable="checkDepartment">
<cfif #wrkDept# EQ #dept[2][1]#>
Department = 'Health' AND
<cfelse>
Department = '#wrkDept#' AND
</cfif>
</cfsavecontent>
But the error I get on the page shows 2 sets of apostrophes around the word Health.
SQL
SELECT COUNT(*) AS numItems
FROM IT_PROJECTS
WHERE
Department = ''Health'' AND
status = 'Cancelled'
Can anyone help me to only get a single apostrophe? Thanks
So this answer seems a lot more complicated than it really is. And without knowing specifically what your query looks like (re:OR conditions), I'm not really sure how to structure it. It can be better. The goal should be to make one single trip to your SQL server with the query that makes the most sense for the data you're trying to get. I'm not sure what you are trying to do with cfsavecontent, but I don't think you need it.
The bulk of my example query (https://trycf.com/gist/4e1f46bfa84a6748aced0f9ee8221c6d/acf2016?theme=monokai) is setup. I chose to go with a cfscript format, because as Redtopia said, I also find it much easier to build a dynamic query in cfscript.
After initial setup, I basically just script out the variables I'll use in my final queryExecute().
// Base query.
qry = "SELECT count(*) AS theCount FROM IT_PROJECTS WHERE 1=1 " ;
// This is our dynamic filter that we build below.
qfilter = {} ;
// Query options.
opts = { "dbtype":"query" } ;
After we have our base, I build up the dynamic part of the query. This is the part that will likely change quite a bit depending on your current needs and setup.
For the first part, I basically replaced your cfif with a ternary evaluation. I'm not sure how your data plays into the evaluation of dept or where that array comes from. But from there I build a basic included statement of the query and set up the queryparam values for it. Then I add a second check that will pick a different set of values for the query (currently based on even/odd seconds). Again, I'm not sure of the intent of your query here, so I just made something dynamic.
//////////// BUILD DYNAMIC FILTER ////////////
qdept = ( wrkDept == dept[2][1] ) ? 'Health' : wrkDept ;
/// This one is an included filter:
qry &= " AND department = :dpt AND status = :sts " ;
qfilter.dpt = {"value":qdept,"cfsqltype":"CFSQLVARCHAR"} ;
qfilter.sts = {"value":"Cancelled","cfsqltype":"CFSQLVARCHAR"} ;
/// Adding Dynamic ORs
// Dynamically set status based on even/odd seconds.
qStatus = ( now().second()%2==0) ? "Cancelled" : "Active" ;
qry &= " OR ( department = :dpt2 AND status = :sts2 ) " ;
qfilter.dpt2 = {value:"IT",cfsqltype:"CFSQLVARCHAR"} ;
qfilter.sts2 = {value:qStatus,cfsqltype:"CFSQLVARCHAR"} ;
This gives us a SQL string that looks like:
SELECT count(*) AS theCount
FROM IT_PROJECTS
WHERE 1=1
AND department = :dpt AND status = :sts
OR
( department = :dpt2 AND status = :sts2 )
With a SQL statement, the placement of AND and OR conditions can greatly impact the results. Use parenthesis to group conditions how you need them.
After we've built the query string, we just have to plug it and our queryparams into the queryExecute().
result = queryExecute( qry , qfilter , opts ) ;
And if we want to output our data, we can go:
writeOutput("There are " & result.theCount & " records." ) ;
Which gives us:
There are 8 records.
Again, I don't know what your main conditions look like. If you can give me an example of a query with a bunch of ORs and ANDs, I'll try to modify this for you.
I am trying to pull data from a SQL database that I have access to. I can connect to the database, see the tables and get the fields associated with a given table, but cannot read a table into an R variable.
I'm working in R Studio, in case this makes a difference.
I have tried using online code snippets (new to R) and these work, except for the dbReadTable() examples. I have used both "Payments" and name="Payments" as the second argument, and both with and without "" quotes.
library(DBI)
con<-(dbConnect(odbc::odbc(), .connection_string="Driver={SQL Server},
Server=example_1234
Database=exampleDB
TrustedConnection=TRUE")
testing123 <- dbListFields(con,"Payments")
testing456 <- dbReadTable(con,"Payments")
I expect a connection to the database which is now named con. This works.
I expect testing123 to contain all the fields in "Payments". This also works.
I expect testing456 to be a data.frame copy of Payments. This produces:
Error: 'SELECT * FROM "Payments"
nanodbc/nanobdc.cpp:1587 42s02 [Microsoft][ODBC SQL SERVER DRIVER][SQL SERVER]Invalid pbject name 'Payments'.
It's slightly different without "Payments" as the argument - simply saying "Object "Payments" not found".
Any help much appreciated.
I suspect that it's because your table is in a different catalog or schema.
Rationale: DBI::dbListFields is doing select * from ... limit 0 (which is not correct syntax for sql server), but odbc::dbListFields is really calling a C++ function connection_sql_columns that is SQL Server specific. It might be permitting you to be a touch sloppy in that it will find the table even if you do not specify the catalog and/or schema. This is why your dbListFields is working. However, DBI::dbReadTable is really doing select * from ... under the hood (and odbc:: is not overriding it), so it is not allowing you to omit the schema (and/or catalog).
First, find the specific table information for your case:
DBI::dbGetQuery(con, "select top 1 table_catalog, table_schema, table_name, column_name from information_schema.columns where table_name='events'")
# table_catalog table_schema table_name column_name
# 1 my_catalog dbo Payments Id
(I'm projecting what you'll find.)
From here, try one of the following until it works:
x <- DBI::dbReadTable(con, DBI::SQL("[Payments]")) # equivalent to the original
x <- DBI::dbReadTable(con, DBI::SQL("[dbo].[Payments]"))
x <- DBI::dbReadTable(con, DBI::SQL("[my_catalog].[dbo].[Payments]"))
My guess is that DBI::dbGetQuery(con, "select top 1 * from Payments") will not work, so for "regular queries" you'll need to use the same hierarchy of catalog.schema.table, such as one of
DBI::dbGetQuery(con, "select top 1 * from dbo.Payments")
DBI::dbGetQuery(con, "select top 1 * from [dbo].[Payments]")
DBI::dbGetQuery(con, "select top 1 * from [my_catalog].[dbo].[Payments]")
(The use of the [ and ] quoted-identifier brackets are often a personal preference, strictly required in only some corner cases.)
Try changing your con argument just slightly:
con <- DBI::dbConnect(odbc::odbc(),
Driver = "SQL Server",
Server = "example_1234",
Database = "exampleDB",
TrustedConnection = TRUE)
# read table to df
testing456 <- dbReadTable(con,"Payments")
# you can also use SQL queries directly, such as:
testing789 <- dbGetQuery(con, statement = "SELECT * FROM Payments WHERE ...")
I have a table that contains a column storing sql functions, column names and similar snippets such as below:
ID | Columsql
1 | c.clientname
2 | CONVERT(VARCHAR(10),c.DOB,103)
The reason for this is to use selected rows to dynamically create results from the main query that match spreadsheet templates. EG Template 1 requires the above client name and DOB.
My Subquery is:
select columnsql from CSVColumns cc
left join Templatecolumns ct on cc.id = ct.CSVColumnId
where ct.TemplateId = 1
order by ct.columnposition
The results of this query are 2 rows of text:
c.clientname
CONVERT(VARCHAR(10),c.DOB,103)
I would wish to pass these into my main statement so it would read initially
Select(
select columnsql from CSVColumns cc
left join Templatecolumns ct on cc.id = ct.CSVColumnId
where ct.TemplateId = 1
order by ct.columnposition
) from Clients c
but perform:
select c.clientname, CONVERT(VARCHAR(10),c.DOB,103) from clients c
to present a results set of client names and DOBs.
So far my attempts at 'injecting' are fruitless. Any suggestions?
You can't do this, at least not directly. What you have to do is, in a stored procedure, build up a varchar/string containing a complete SQL statement; you can execute that string.
declare #convCommand varchar(50);
-- some sql to get 'convert(varchar(10), c.DOB, 103) into #convCommand.
declare #fullSql varchar(1000);
#fullSql = 'select c.clientname, ' + #convCommand + ' from c,ients c;';
exec #fullSql
However, that's not the most efficient way to run it - and when you already know what fragment you need to put into it, why don't you just write the statement?
I think the reason you can't do that is that SQL Injection is a dangerous thing. (If you don't know why please do some research!) Having got a dangerous string into a table - e.g 'c.dob from clients c;drop table clients;'- using the column that contains the data to actually execute code would not be a good thing!
EDIT 1:
The original programmer is likely using a C# function:
string newSql = string.format("select c.clientname, {0} from clients c", "convert...");
Basic format is:
string.format("hhh {0} ggg{1}.....{n}, s0, s1,....sn);
{0} in the first string is replaced by the string at s0; {1} is replaces by tge string at s1, .... {n} by the string at sn.
This is probably a reasonable way to do it, though why is needs all the fragments is a bit opaque. You can't duplicate that in sql, save by doing what I suggest above. (SQL doesn't have anything like the same string.format function.)
I'm using ROracle on a Win7 machine running the following R version:
platform x86_64-w64-mingw32
arch x86_64
os mingw32
system x86_64, mingw32
status
major 3
minor 1.1
year 2014
month 07
day 10
svn rev 66115
language R
version.string R version 3.1.1 (2014-07-10)
nickname Sock it to Me
Eventually, I'm going to move the script to a *nix machine, cron it, and run it with RScript.
I want to do something similar to:
select * from tablename where 'thingy' in ('string1','string2')
This would return two rows with all columns in SQLDeveloper (or Toad, etc).
(Ultimately, I want to pull results from one DB into a single column in a data.frame then use those results to loop through
and pull results from a second db, but I also need to be able to do just this function as well.)
I'm following the documentation for RORacle from here.
I've also looked at this (which didn't get an answer):
Bound parameters in ROracle SELECT statements
When I attempt the query from ROracle, I get two different errors, depending on whether I try a dbGetQuery() or dbSendQuery().
As background, here are the versions, queries and data I'm using:
Driver name: Oracle (OCI)
Driver version: 1.1-11
Client version: 11.2.0.3.0
The connection information is standard:
library(ROracle)
ora <- dbDriver("Oracle")
dbcon <- dbConnect(ora, username = "username", password = "password", dbname = "dbnamefromTNS")
These two queries return the expected results:
rs_send <- dbSendQuery(dbcon, "select * from tablename where columname_A = 'thingy' and rownum <= 1000")
rs_get <- dbGetQuery(dbcon, "select * from tablename where columname_A = 'thingy' and rownum <= 1000")
That is to say, 1000 rows from tablename where 'thingy' exists in columnname_A.
I have a data.frame of one column, with two rows.
my.data = data.frame(RANDOM_STRING = as.character(c('string1', 'string2')))
and str(my.data) returns this:
str(my.data)
'data.frame': 2 obs. of 1 variable:
$ RANDOM_STRING: chr "string1" "string2"
my attempted queries are:
nope <- dbSendQuery(dbcon, "select * from tablename where column_A = 'thingy' and widget_name =:1", data = data.frame(widget_name =my.data$RANDOM_STRING))
which gives me an error of:
Error in .oci.SendQuery(conn, statement, data = data, prefetch = prefetch, :
bind data does not match bind specification
and
not_this_either <- dbGetQuery(dbcon, "select * from tablename where column_A = 'thingy' and widget_name =:1", data = data.frame(widget_name =my.data$RANDOM_STRING))
which gives me an error of:
Error in .oci.GetQuery(conn, statement, data = data, prefetch = prefetch, :
bind data has too many rows
I'm guessing that my problem is in the data=(widget_name=my.data$RANDOM_STRING) part of the queries, but haven't been able to rubber duck my way through it.
Also, I'm very curious as to why I get two separate and different errors depending on whether the queries use the send (and fetch later) format or the get format.
If you like the tidyverse there's a slightly more compact way to achieve the above using purrr
library(ROracle)
library(purrr)
ora <- dbDriver("Oracle")
con <- dbConnect(ora, username = "username", password = "password", dbname = "yourdbnamefromTNSlist")
yourdatalist <- c(12345, 23456, 34567)
output <- map_df(yourdatalist, ~ dbGetQuery(con, "select * from YourTableNameHere where YOURCOLUMNNAME = :d", .x))
Figured it out.
It wasn't a problem with Oracle or ROracle (I'd suspected this) but with my R code.
I stumbled over the answer trying to solve another problem.
This answer about "dynamic strings" was the thing that got me moving towards a solution.
It doesn't fit exactly, but close enough to rubberduck my way to an answer from there.
The trick is to wrap the whole thing in a function and run an ldply on it:
library(ROracle)
ora <- dbDriver("Oracle")
con <- dbConnect(ora, username = "username", password = "password", dbname = "yourdbnamefromTNSlist")
yourdatalist <- c(12345, 23456, 34567)
thisfinallyworks <- function(x) {
dbGetQuery(con, "select * from YourTableNameHere where YOURCOLUMNNAME = :d", data = x)
}
ldply(yourdatalist, thisfinallyworks)
row1 of results where datapoint in YOURCOLUMNNAME = 12345
row2 of results where datapoint in YOURCOLUMNNAME = 23456
row3 of results where datapoint in YOURCOLUMNNAME = 34567
etc