I'm currently working on a small project using sql data tools. I'm getting data from excel sheet and write it back to my database table. from the database table I'm generating a report. Time to time I'm updating my excel sheet. but it won't update in my database table. how do I clear the table and rewrite excel sheet data to database table when run the project every time?
post build script would be how and when I would clear or synchronize the data. Then you have to write the truncate or delete and the insert and/or update sections to move data from your excel document to the sql table. I would suspect that the SQL table is a better place to maintain the data though!!! If you maintain in SQL you can always have a query to see what it is in Excel at any given time and you don't have to worry about synching back to SQL.
Related
I have a report that I generate on a weekly basis. I have the code written in SQL and I then pull all the data into excel's data model.
I then create pivot tables and dashboards in excel from that particular data.
The SQL code creates new table of the same name everytime and deletes the older version of the table. There isn't any way for me to just append the new data as the report is run from the very start and not just on the new data.
I wish to automate this process of refreshing my dashboard from the data I produce in SQL. Is there a way to do so?
Currently I create a new table in SQL, import data into the excel's data model and then recreate the dashboard.
I am not even sure if this is possible. Any help would be greatly appreciated!
Solved!
After some digging, I was able to find a feature that Excel's data model supports.
Instead of making a connection directly to a SQL Server Table, you can create a connection by writing a SQL Query.
This way, even if you delete the table for updating it, as far as the name remains the same, Excel's data model would be able to pull data from the table just by you hitting refresh!
Problem:
I need to get data sets from CSV files into SQL Server Express (SSMS v17.6) as efficiently as possible. The data sets update daily into the same CSV files on my local hard drive. Currently using MS Access 2010 (v14.0) as a middleman to aggregate the CSV files into linked tables.
Using the solutions below, the data transfers perfectly into SQL Server and does exactly what I want. But I cannot figure out how to refresh/update/sync the data at the end of each day with the newly added CSV data without having to re-import the entire data set each time.
Solutions:
Upsizing Wizard in MS Access - This works best in transferring all the tables perfectly to SQL Server databases. I cannot figure out how to update the tables though without deleting and repeating the same steps each day. None of the solutions or links that I have tried have panned out.
SQL Server Import/Export Wizard - This works fine also in getting the data over to SSMS one time. But I also cannot figure out how to update/sync this data with the new tables. Another issue is that choosing Microsoft Access as the data source through this method requires a .mdb file. The latest MS Access file formats are .accdb files so I have to save the database in an older .mdb version in order to export it to SQL Server.
Constraints:
I have no loyalty towards MS Access. I really am just looking for the most efficient way to get these CSV files consistently into a format where I can perform SQL queries on them. From all I have read, MS Access seems like the best way to do that.
I also have limited coding knowledge so more advanced VBA/C++ solutions will probably go over my head.
TLDR:
Trying to get several different daily updating local CSV files into a program where I can run SQL queries on them without having to do a full delete and re-import each day. Currently using MS Access 2010 to SQL Server Express (SSMS v17.6) which fulfills my needs, but does not update daily with the new data without re-importing everything.
Thank you!
You can use a staging table strategy to solve this problem.
When it's time to perform the daily update, import all of the data into one or more staging tables. Execute SQL statement to insert rows that exist in the imported data but not in the base data into the base data; similarly, delete rows from the base data that don't exist in the imported data; similarly, update base data rows that have changed values in the imported data.
Use your data dependencies to determine in which order tables should be modified.
I would run all deletes first, then inserts, and finally all updates.
This should be a fun challenge!
EDIT
You said:
I need to get data sets from CSV files into SQL Server Express (SSMS
v17.6) as efficiently as possible.
The most efficient way to put data into SQL Server tables is using SQL Bulk Copy. This can be implemented from the command line, an SSIS job, or through ADO.Net via any .Net language.
You state:
But I cannot figure out how to refresh/update/sync the data at the end
of each day with the newly added CSV data without having to re-import
the entire data set each time.
It seems you have two choices:
Toss the old data and replace it with the new data
Modify the old data so that it comes into alignment with the new data
In order to do number 1 above, you'd simply replace all the existing data with the new data, which you've already said you don't want to do, or at least you don't think you can do this efficiently. In order to do number 2 above, you have to compare the old data with the new data. In order to compare two sets of data, both sets of data have to be accessible wherever the comparison is to take place. So, you could perform the comparison in SQL Server, but the new data will need to be loaded into the database for comparison purposes. You can then purge the staging table after the process completes.
In thinking further about your issue, it seems the underlying issue is:
I really am just looking for the most efficient way to get these CSV
files consistently into a format where I can perform SQL queries on
them.
There exist applications built specifically to allow you to query this type of data.
You may want to have a look at Log Parser Lizard or Splunk. These are great tools for querying and digging into data hidden inside flat data files.
An Append Query is able to incrementally add additional new records to an existing table. However the question is whether your starting point data set (CSV) is just new records or whether that data set includes records already in the table.
This is a classic dilemma that needs to be managed in the Append Query set up.
If the CSV includes prior records - then you have to establish the 'new records' data sub set inside the CSV and append just those. For instance if you have a sequencing field then you can use a > logic from the existing table max. If that is not there then one would need to do a NOT compare of the table data with the csv data to identify which csv records are not already in the table.
You state you seek something 'more efficient' - but in truth there is nothing more efficient than a wholesale delete of all records and write of all records. Most of the time one can't do that - but if you can I would just stick with it.
I have a table with 103 columns and more than 100,000 records. I was told that Oracle Data base has bad data in few of the columns(four columns) and in few of the records(20,000 records approx) and was told to modify data. Also I was given a query where I should modify data(bad data). With the help of query, I exported all the bad data into excel and modified using macros.
How to replace existing bad data in Oracle data base to data which I have in excel(good data). Is it possible to modify data in some part of the database using SQL query? I mean to say 4 out of 103 columns and 20k out of 100k records has to be modified without affecting already existing good data in Oracle database.
I am using SQL developer and Oracle 11g
My query to retrieve bad data
select e.id_number
, e.gender_code
, e.pref_mail_name
, e.prefix
,e.first_name
,e.last_name
,e.spouse_name
,e.spouse_id_number
,e.pref_jnt_mail_name1
,e.pref_jnt_mail_name2
from advance.entity e
where e.person_or_org = 'P'
and ascii(e.spouse_name) <> 32
or ascii(e.spouse_id_number) <> 32"
Note: Moreover I am not changing any primary key or secondary key data. Bad data is in other columns.
You can start by importing all corrected data (which now resides in your Excel file), to a temporary table. See Load Excel data sheet to Oracle database for more details.
The it should be a simple task to write one or more UPDATE statements. If you are new to Oracle syntax, Tech on the Net has some nice examples.
Also, do not forget to backup the original table to a temporary table before you make any changes there. That way, if you mess up the data repair, you can start over.
I have an Excel spreadsheet, with one worksheet, with one column, containing key values I want to use to lookup data in a SQL Server view.
I can import the view data just fine using ADO or ODBC, but cannot figure out how to have the "Microsoft Query" tool add both the Excel Worksheet and the SQL View in a single query, so that I can join the data. I had also hoped VLOOKUP might lookup external data, but did not find anything,
I have looked for hours and am stumped. Certainly this can be done somehow where I can use a key in an Excel file to fetch a single related SQL table row back into Excel? I'm a hard core C# developer but seem to regularly struggle with Excel...
I would import the excel data into a staging table in your SQL Server database.
Of course you can use ADO to get the excel data into SQL Server.
Then you can join the view with your staging table.
hope this helps.
I have a farily simple task of taking an Excel sheet and importing it into a SQL 2005 database table. I need to create an SSIS task for this. The Excel sheet does not have all the columns I need to make the insert directly into the permanent sql table, but I know how I could link out to other tables and get the columns that are missing. So I was wondering how I could import the Excel sheet into a #tempTable (or #VariableTable) and then one in a temp table I could just write my SQL Insert code (using the temptable as well as the other tables that I will link on) in a basic Execute SQL Task. But I am having trouble figuring out how to do this with SSIS. When I drag my excel source and try to link it to a SQL Server Destination the drop down doesn't have an option for temptables.
The SSIS way of doing this would be to use a Merge or Lookup transform. I don't think that you can put things into a temp table like that, but you could have an ExecuteSQL task that creates an actual table that you can then drop at the end of the package. You can then have your package use that.
During the design time you might need to have the table in place to link things up, but it shouldn't need to be there when you actually run the package.
First, you'll need to create the staging table for the excel worksheet. Open SSMS, right click the database, choose tasks, import data. Set the import source as excel. Browse to the file. Set the destination as SQL Server. You can accept the table name or name it as you wish. I suggest naming it something useful. Depending on your understanding of data types and what is in the excel sheet, it may take you a while to get this right. Eventually, you will have a table that will accept the contents of the excel sheet.
Second, create your ssis package by using an excel source and sql serve or oledb destination.
take Execute SQL task in control flow to create the target staging table of excel sheet surce.
in data flowuse excel source and oled db target to staging table create in the first step.
3.In control flow use merge or join statement to your excel targeted staging table with other source table to final target table.
thanks
prav