I'm trying to catalog the structure of a MSSQL 2008 R2 database using R/RODBC. I have set up a DSN, connected via R and used the sqlTables() command but this is only getting the 'system databases' info.
library(RODBC)
conn1 <- odbcConnect('my_dsn')
sqlTables(conn1)
However if I do this:
library(RODBC)
conn1 <- odbcConnect('my_dsn')
sqlQuery('USE my_db_1')
sqlTables(conn1)
I get the tables associated with the my_db_1 database. Is there a way to see all of the databases and tables without manually typing in a separate USE statement for each?
There may or may not be a more idiomatic way to do this directly in SQL, but we can piece together a data set of all tables from all databases (a bit more programatically than repeated USE xyz; statements) by getting a list of databases from master..sysdatabases and passing these as the catalog argument to sqlTables - e.g.
library(RODBC)
library(DBI)
##
tcon <- RODBC::odbcConnect(
dsn = "my_dsn",
uid = "my_uid",
pwd = "my_pwd"
)
##
db_list <- RODBC::sqlQuery(
channel = tcon,
query = "SELECT name FROM master..sysdatabases")
##
R> RODBC::sqlTables(
channel = tcon,
catalog = db_list[14, 1]
)
(I can't show any of the output for confidentiality reasons, but it produces the correct results.) Of course, in your case you probably want to do something like
all_metadata <- lapply(db_list$name, function(DB) {
RODBC::sqlTables(
channel = tcon,
catalog = DB
)
})
# or some more efficient variant of data.table::rbindlist...
meta_df <- do.call("rbind", all_metadata)
I have an SQL script which I need to run using R Studio. However, my SQL script has one variable that is defined in my R environment. I am using dbGetQuery; however, I do not know (and I didn't find a solution) how to pass these variables.
library(readr)
library(DBI)
library(odbc)
library(RODBC)
#create conection (fake one here)
con <- odbcConnect(...)
dt = Sys.Date()
df = dbGetQuery(.con, statement = read_file('Query.sql'))
The file 'Query.sql' makes reference to dt. How do I make the file recognize my variable dt?
There are several options, but my preferred is "bound parameters".
If, for instance, your 'Query.sql' looks something like
select ...
from MyTable
where CreatedDateTime > ?
The ? is a place-holder for a binding.
Then you can do
con <- dbConnect(...) # from DBI
df = dbGetQuery(con, statement = read_file('Query.sql'), params = list(dt))
With more parameters, add more ?s and more objects to the list, as in
qry <- "select ... where a > ? and b < ?"
newdat <- dbGetQuery(con, qry, params = list(var1, var2))
If you need a SQL IN clause, it gets a little dicey, since it doesn't bind things precisely like we want.
candidate_values <- c(2020, 1997, 1996, 1901)
qry <- paste("select ... where a > ? and b in (", paste(rep("?", length(candidate_values)), collapse=","), ")")
qry
# [1] "select ... where a > ? and b in ( ?,?,?,? )"
df <- dbGetQuery(con, qry, params = c(list(avar), as.list(candidate_values)))
Trying to solve issue with wrong display of national characters (Polish) in results of query to MS SQL database.
The script is pretty standard
First the definnition connection object
library(DBI)
db.conn <- DBI::dbConnect(odbc::odbc(),
Driver = "SQL Server Native Client 11.0",
Server = "10.0.0.100",
Port = 1433,
Database = "DB",
UID = "user",
PWD = rstudioapi::askForPassword("Database password"),
encoding = "latin1"
)
then SQL statement
db_sql = "
select
*
from test
where active = 'ACTIVE'
order by name_id"
Then execution of SQL
db_query <- dbSendQuery(db.conn, db_sql)
db_data <- dbFetch(db_query)
or
db_data <- dbGetQuery(db.conn, db_sql)
It does not matter whether in connection object definition I use "latin1", "windows-1250" or "utf-8" parameter for encoding parameter the results are always the same
Strings with U+009C or similar
It also does not matter what codepage I select in RStudio Global options.
Problem solved.
First it is necessary to change locale to Polish
Sys.setlocale(category = "LC_ALL", locale = "Polish")
Then set proper encoding in
DBI::dbConnect(odbc::odbc()
...
encoding = "windows-1250"
and voilla, working.
How can I insert a complete R data.frame in one go into a SQL table, with the help of a stored procedure but without using a loop, using R language?
I am using Microsoft Server 2012.
This is my R code:
library(RODBC)
dbhandle <- odbcDriverConnect('driver={SQL Server};server=MySERVERName;database=Testing;trusted_connection=true;')
f= data.frame(
age= c(21,22,23,24),
name= c("fifi", "jojo", "jj", "arbi"), stringsAsFactors = FALSE)
sqlQuery(dbhandle,paste("EXEC BOinsert #age=", f$age, ", #name=", f$name))
This is my SP:
ALTER proc [dbo].[BOinsert]
#age int,
#name nvarchar(50)
as
Insert into Student(Age, Name) values (#age, #name)
I would recommend using RODBCext.
library(RODBCext)
f= data.frame(
age= c(21,22,23,24),
name= c("fifi", "jojo", "jj", "arbi"), stringsAsFactors = FALSE)
sqlExecute(dbhandle,
"EXEC BOinsert #age = ?, #name = ?",
data = f)
Strictly speaking, this is running a for loop over your data and executing the stored procedure once for every row. But I think it is done in C code, and you don't have to worry about making it work. It's one line of code for you. It will also deal with all of the quoting for you so that you don't have to worry about SQL injection attacks.
EDIT:
Just ran a test on my system with a data frame of 1,000 rows. sqlExecute was able to load it all in about 1.52 seconds. It took sqlQuery about 1.76 seconds. So not a huge improvement performance, but at least sqlExecute doesn't look like a loop.
It can be done by MySql using RMySQL
install.packages("RMySQL")
library(RMySQL)
mydb = dbConnect(MySQL(), user='your_user_name', password='your_password', dbname='your_database_name', host='localhost')
dbListTables(mydb)
#create data frame
ds <- data.frame(a=1:10, b= 21:30)
#adding ds data frame to your database
dbWriteTable(mydb, name='db', value=ds)
#then check your database this will be added as table in name of db
Added 02/22/2017:
There is no easy way avoid the loop if you cannot change the stored procedure ("SP") which can only insert one row per call. This means a network roundtrip per stored procedure call (= per data.frame row).
Your alternatives are (all too complicated to show it here in detail):
Bulk insert the data.frame into a temporary or staging table using bcp.exe or the bulk insert statement. Then execute a SQL statement the loops over all rows in the temp/staging table and calls your insert SP. This effectively shifts the loop from the client side (R) to the server side (SQL server) which is much faster since no network roundtrips are required per row to be inserted. For details see: MS-SQL Bulk Insert with RODBC
Still loop over all rows in the R code but create a block of SQL statements that insert many rows at once (e. g. 100 at time). This reduced the network roundtrips by the number of rows you send at once.
Create a wrapper SP that receives the data.frame content as e. g. CSV text string, parses it and calls your insert SP per data row.
[Added 02/23/2017] You could use the package RODBCext which supports prepared statements. This "only" hides the loop but still loops internally at the client side (with one network/server roundtrip per row insert) but saves some time since the SQL statement must not be parsed and "prepared" again (execution planning etc.).
Solutions 2 is the easiest one to implement, but you have to care about the maximum statement length of the ODBC driver (about 1000, 2000, 4000 or 8000 character I guess).
Original answer:
Since you have to use stored procedure to insert each single row you end up with a slow looped solution (which is OK if you do not have to many rows to insert):
library(RODBC)
f <- data.frame(age = c(21,22,23,24), name = c("fifi", "jojo", "jj", "arbi"), stringsAsFactors = FALSE)
dbhandle <- odbcDriverConnect('driver={SQL Server};server=MySERVERName;database=Testing;trusted_connection=true;')
for (i in 1:nrow(f)) {
sqlQuery(dbhandle, paste0("EXEC BOinsert #age=", f[i,]$age, ", #name='", f[i,]$name, "'"))
# print(paste0("EXEC BOinsert #age=", f[i,]$age, ", #name='", f[i,]$name, "'"))
}
odbcClose(dbhandle)
Alternative 2 of my first answer (create a block of SQL statements that insert many rows at once) could be implemented like this:
library(RODBC)
f <- data.frame(age = c(21,22,23,24), name = c("fifi", "jojo", "jj", "arbi"), stringsAsFactors = FALSE)
dbhandle <- odbcDriverConnect('driver={SQL Server};server=MySERVERName;database=Testing;trusted_connection=true;')
block.size.in.rows <- 2 # you should use a much bigger value like 100 (2 is only for demonstration purposes here)
sql <- ''
for (i in 1:nrow(f)) {
# sqlQuery(dbhandle, paste0("EXEC BOinsert #age=", f[i,]$age, ", #name='", f[i,]$name, "'"))
sql <- paste0(sql, "EXEC BOinsert #age=", f[i,]$age, ", #name='", f[i,]$name, "'", "; \n")
if ((i %% block.size.in.rows) == 0) {
cat(paste0("Sending block:\n"))
cat(sql) # debug
sqlQuery(dbhandle, sql)
sql <- ''
}
}
if (nchar(sql) > 0) {
cat(paste0("Sending block:\n"))
cat(sql) # debug
sqlQuery(dbhandle, sql)
}
odbcClose(dbhandle)
It will send multiple SQL statements in a block via one server roundtrip:
Sending block:
EXEC BOinsert #age=21, #name='fifi';
EXEC BOinsert #age=22, #name='jojo';
Sending block:
EXEC BOinsert #age=23, #name='jj';
EXEC BOinsert #age=24, #name='arbi';
This is my first attempt at using R to access data from within MS Access using ODBC.
The following query works:
id <- levels(assetid)[assetid[,1]][12]
qry <- "SELECT DriverName FROM Data WHERE ID = 'idofinterest'"
sqlQuery(con, qry)
However, I would like to know if there is a way to use the variable "id" in the "qry" statement (without using paste)? I have seen some statements on the web with $ and % signs - however I haven't had any success in using them.
Thanks.
Why don't you want to use paste? Anyway, sprintf is an alternative means of string munging.
qry <- sprintf("SELECT DriverName FROM Data WHERE ID = '%s'", id)
sqlQuery(con, qry)
Try fn$ from the gsubfn package :
> library(gsubfn)
> id <- "abc"
> fn$identity("SELECT DriverName FROM Data WHERE ID = '$id'")
[1] "SELECT DriverName FROM Data WHERE ID = 'abc'"