How to access to columns in data frame uploaded in shiny app? - file-io

I am trying to create a shiny app. This app uses a .csv file uploaded by the user. I do not really understand how fileInput() works in the sense of storing the data frame.
I am using this code to upload the file:
data_OBS = reactive({
inFile <- input$file1
if (is.null(inFile))
return(NULL)
read.csv(inFile$datapath, header=T, sep=",")
})
If I understood well, the data frame should be accesible in data_OBS(). If the .csv file I want to upload has a column named for example "CL" in normal R enviorement I should be able to access using data_OBS$CL. However, in shiny data_OBS() is a function. I have tried data_OBS$CL, data_OBS()$CL, data_OBS(CL) but none of them worked. How can access to the data just uploaded?
Thanks in advance,
Best,

You can access it with data_OBS()$CL, but only inside reactive context such as observe, observeEvent, eventReactive.
Here is a minimal example using observeEvent:
ui <- fluidPage(
fileInput("file1", "Choose CSV File"),
textOutput("text"),
actionButton("print", "Print to text output")
)
server <- function(input, output, session){
data_OBS = reactive({
inFile <- input$file1
if (is.null(inFile))
return(NULL)
read.csv(inFile$datapath, header=T, sep=",")
})
observeEvent(input$print, {
req(input$file1) # Doesn't do anything until a file is uploaded
output$text <- renderText(data_OBS()$CL)
})
}
shinyApp(ui, server)

First to check it did not return null. Also, you may force return(read.csv(...))
data_OBS()$CL should work.

Related

How to update the .RDS file with the files user uploads and store it for next time in Shiny (server)?

Basically, there are 50 files which are about 50 MB each and therefore, processed them and made 30k rows processed data as .RDS file to render some visualization.
However, User needs the app to upload recent files and keep updating the visualization.
Is it possible to update the .RDS file with the files user uploads ?
Can the user access this updated .RDS file even next time (session) ?
In the following example,
there is an upload button and render just a file.
Can we store the uploaded files somehow ?
So, that, we can use these uploaded files to update the .RDS file
relevant links:
R Shiny Save to Server
https://shiny.rstudio.com/articles/persistent-data-storage.html
library(shiny)
# Define UI for data upload app ----
ui <- fluidPage(
# App title ----
titlePanel("Uploading Files"),
# Sidebar layout with input and output definitions ----
sidebarLayout(
# Sidebar panel for inputs ----
sidebarPanel(
# Input: Select a file ----
fileInput("files", "Upload", multiple = TRUE, accept = c(".csv"))
),
# In order to update the database if the user clicks on the action button
actionButton("update_database","update Database")
# Main panel for displaying outputs ----
mainPanel(
# Output: Data file ----
dataTableOutput("tbl_out")
)
)
)
# Define server logic to read selected file ----
server <- function(input, output) {
lst1 <- reactive({
validate(need(input$files != "", "select files..."))
##### Here, .RDS file can be accessed and updated for visualiztaion
if (is.null(input$files)) {
return(NULL)
} else {
path_list <- as.list(input$files$datapath)
tbl_list <- lapply(input$files$datapath, read.table, header=TRUE, sep=";")
df <- do.call(rbind, tbl_list)
return(df)
}
})
output$tbl_out <- renderDataTable({
##### Can also access this data for visualizations
lst1()
})
session$onSessionEnded({
observeEvent(input$update_database,{
s3save(appended_data, object = "object_path_of_currentRDS_file",bucket = "s3_bucket")
})
})
}
# Create Shiny app ----
shinyApp(ui, server)
But there was an error:
Warning: Error in private$closedCallbacks$register: callback must be a function
50: stop
49: private$closedCallbacks$register
48: session$onSessionEnded
47: server [/my_folder_file_path.R#157]
Error in private$closedCallbacks$register(sessionEndedCallback) :
callback must be a function
Solution:
replace this code
session$onSessionEnded({
observeEvent(input$update_database,{
s3save(appended_data, object = "object_path_of_currentRDS_file",bucket = "s3_bucket")
})
})
## with this
observeEvent(input$update_database,{
s3saveRDS(appended_data, object = "object_path_of_currentRDS_file",bucket = "s3_bucket")
})

shinyapps.io does not work when my shiny use RODBC to link a SQL database

On my local computer, I use shiny to design a web page to show the analysis result. The data is extracted from the company's SQL database using RODBC to link the database to R. The code is like this:
library(shiny)
library(shinydashboard)
library(DT)
library(RODBC)
library(stringr)
library(dplyr)
DNS <- '***'
uid <- '***'
pwd <- '***'
convertMenuItem <- function(mi,tabName) {
mi$children[[1]]$attribs['data-toggle']="tab"
mi$children[[1]]$attribs['data-value'] = tabName
mi
}
sidebar <- dashboardSidebar(
sidebarMenu(
convertMenuItem(menuItem("Query1",tabName="Query1",icon=icon("table"),
dateRangeInput('Date1','Date Range',start = Sys.Date()-1, end = Sys.Date()-1,
separator=" - ",format="dd/mm/yy"),
textInput('Office1','Office ID','1980'),
submitButton("Submit")), tabName = "Query1"),
convertMenuItem(menuItem("Query2",tabName="Query2",icon=icon("table"),
dateRangeInput('Date2','Date Range',start = Sys.Date()-1, end = Sys.Date()-1,
separator=" - ",format="dd/mm/yy"),
textInput('Office2','Office ID','1980'),
submitButton("Submit")), tabName = "Query2"),
)
)
body <- dashboardBody(
tabItems(
tabItem(tabName="Query1",
helpText('********************************'),
fluidRow(
column(12,DT::dataTableOutput('table1'))
)
),
tabItem(tabName = "Query2",h2("Widgets tab content"))
)
)
dashboardheader <- dashboardHeader(
title = 'LOSS PREVENTION'
)
ui <- dashboardPage(
skin='purple',
dashboardheader,
sidebar,
body
)
server <- function(input, output) {
output$table1 <- DT::renderDataTable({
ch<-odbcConnect(DNS,uid=uid,pwd=pwd)
a <- sqlQuery(ch,paste(' ***'))
odbcClose(ch)
DT::datatable(a,options = list(scrollX=T))
})
}
shinyApp(ui, server)
Then, I have my account on shinyapps.io. And use rsconnect to deploy this programm. And the deployment is successful.
But when I use https://myAccount.shinyapps.io/myshiny/ to access my code. I have the following error:
2018-05-10T00:57:38.473259+00:00 shinyapps[340325]: Warning in RODBC::odbcDriverConnect("DSN=****;UID=****;PWD=****") :
2018-05-10T00:57:38.473262+00:00 shinyapps[340325]: [RODBC] ERROR: state IM002, code 0, message [unixODBC][Driver Manager]Data source name not found, and no default driver specified
But, if there is no RODBC and SQL database involved in my code, the code works fine.
So, the problem is because shinyapps.io cannot access my company's SQL database. How can I deal with it?
The app works on your computer because the Data Source Name (DSN) has been configured there. It is not configured on shinyapps.io. According to this help article you can use for example
odbcDriverConnect('Driver=FreeTDS;TDS_Version=7.0;Server=<server>;Port=<port>;Database=<db>;Uid=<uid>;Pwd=<pw>;Encrypt=yes;TrustServerCertificate=no;Connection Timeout=30;')
A more complete treatment can be found in the documentation.

Automatically import data into SQL Server

I am trying to import data from a .CSV file into SQL Server automatically. For example, if I have a set of .CSV files in a folder, I want the data from those .CSV files to be imported into SQL Server automatically, every time I add a new .CSV file to that folder.
If you want it to be truly automatic, you will need to go beyond SQL. Otherwise, scheduling a load from the csv to your database would require a simple SSIS package. Then you can go and schedule a job using SQL Server Agent to import the data daily, weekly, hourly, etc.
EDITED!! THIS IS FOR PHP
$filenames = glob($_SERVER['DOCUMENT_ROOT'] .'/filespath/*');
foreach($filenames as $filename){ // iterate files
if(!file_exists($filename) || substr($filename,-3) == 'csv') continue;
$file = fopen($filename, "r");
$csv = array();
while (($line = fgetcsv($file)) !== FALSE) {
//$line is an array of the csv elements
$csv[] = $line;
}
fclose($file);
foreach ($csv as $key => $value) {
//put your code here to work with $csv array.
}
}
You can use this code to get what you want. You also need cron to automate your code. You can get it in your host cpanel.

export data from bigquery to cloud storage- php client library - there is one extra empty new line in the cloud storage file

I followed this sample
https://cloud.google.com/bigquery/docs/exporting-data
public function exportDailyRecordsToCloudStorage($date, $tableId)
{
$validTableIds = ['table1', 'table2'];
if (!in_array($tableId, $validTableIds))
{
die("Wrong TableId");
}
$date = date("Ymd", date(strtotime($date)));
$datasetId = $date;
$dataset = $this->bigQuery->dataset($datasetId);
$table = $dataset->table($tableId);
// load the storage object
$storage = $this->storage;
$bucketName = 'mybucket';
$objectName = "daily_records/{$tableId}_" . $date;
$destinationObject = $storage->bucket($bucketName)->object($objectName);
// create the import job
$format = 'NEWLINE_DELIMITED_JSON';
$options = ['jobConfig' => ['destinationFormat' => $format]];
$job = $table->export($destinationObject, $options);
// poll the job until it is complete
$backoff = new ExponentialBackoff(10);
$backoff->execute(function () use ($job) {
print('Waiting for job to complete' . PHP_EOL);
$job->reload();
if (!$job->isComplete()) {
//throw new Exception('Job has not yet completed', 500);
}
});
// check if the job has errors
if (isset($job->info()['status']['errorResult'])) {
$error = $job->info()['status']['errorResult']['message'];
printf('Error running job: %s' . PHP_EOL, $error);
} else {
print('Data exported successfully' . PHP_EOL);
}
I have 37670 rows in my table1, and the cloud storage file has 37671 lines.
And I have 388065 my table2, and the cloud storage file has 388066 lines.
The last line in both cloud storage files is empty line.
Is this a Google BigQuery feature improvement request? or I did something wrong in my codes above?
What you described seems like an unexpected outcome. The output file should generally has the same number of lines as the source table.
Your PHP code looks fine and shouldn't be the cause of the issue.
I'm trying reproduce it but unable to. Could you double-check if the last empty line is somehow added by another tool like a text editor or something? How are you counting the lines of the resulting output.
If you have ruled that out and are sure the newline is indeed added by BigQuery export feature, please consider opening a bug using the BigQuery Issue Tracker as suggested by xuejian and include your job ID so that we can investigate further.

Avoid re-loading datasets within a reactive in shiny

I have a shiny app that requires the input from one of several files. A simplified example would be:
library(shiny)
x <- matrix(rnorm(20), ncol=2)
y <- matrix(rnorm(10), ncol=4)
write.csv(x, 'test_x.csv')
write.csv(y, 'test_y.csv')
runApp(list(ui = fluidPage(
titlePanel("Choose dataset"),
sidebarLayout(
sidebarPanel(
selectInput("data", "Dataset", c("x", "y"), selected="x")
),
mainPanel(
tableOutput('contents')
)
)
)
, server = function(input, output, session){
myData <- reactive({
inFile <- paste0("test_", input$data, ".csv")
data <- read.csv(inFile, header=FALSE)
data
})
output$contents <- renderTable({
myData()
})
}))
In reality, the files I read in are much large, so I would like to avoid reading them in each time input$data changes, if it has already been done once. For example, by making the matrices mat_x and mat_y available within the environment, and then within myData testing:
if (!exists(paste0("mat_", input$data))) {
inFile <- paste0("test_", input$data, ".csv")
data <- read.csv(inFile, header=FALSE)
assign(paste0("mat_", input$data), data)
}
Is there a way to do this, or do I have to create a separate reactive for mat_x and mat_y and using that within myData? I actually have 9 possible input files, but each user may only want to use one or two.
You could do something like
myData <- reactive({
data <- fetch_data(input$data)
data
)}
fetch_data <- function(input) {
if (!exists(paste0("mat_", input))) {
inFile <- paste0("test_", input, ".csv")
data <- read.csv(inFile, header=FALSE)
assign(paste0("mat_", input), data)
} else {
data <- paste0("mat_", input)
}
return (data)
}