I have a google doc that I'm using to update a table in a sql table and at the moment I do it manually; copy and paste the data from the google doc. into excel, remove the columns that I don't need then save as a CSV and upload.
I'm wondering if there is a way that I can automate this? The report it's for is daily and the upload is becoming a bit of a bore!
You can also go the python-route by utilising the following python libraries: gspread, oauth2client, pandas and sqlalchemy (source). If you want to automate it, use cron or crontab to run your python-script on a hourly, daily, weekly or monthly basis.
What DBMS do you use?
If you use Microsoft SQL Server, you can use SSIS (SQL Server Integration Services), the easiest way to do this for a novice would be to right-click on the database in SQL Server Management Studio --> Tasks --> Import Data, then follow the wizard to set up the data source (flat file / Excel file etc.) and the data destination (DB table). Once you've finished with the wizard, you can save the package and re-use it later on (or even automate it via SQL Agent job). You can edit the package in Visual Studio (aka SQL Server Data Tools) prior to deployment/automation if needed.
More on SSIS: https://msdn.microsoft.com/en-us/library/ms169917.aspx
Related
Is there a way to export data from a MS Excel file into a SQL Server table automatically? Maybe this is done using a script of some kind.
If it's not possible to be completely automated, perhaps there's a way to do it using minimal user effort. (For example clicking a button or link)
There is a MS Excel spreadsheet where the data keeps having to be manually exported to SQL Server.
I've done this using Excel to Access before, but not too certain on how to do it using SQL Server (MS).
*MS Office 2013 and MS SQL Server 2012.
The other answers are ok. I just want to suggest an additional alternative.
If it is just 1 specific Excel file that is frequently updated, I would consider using VBA. For example, write some VBA code in Excel that uploads changes to the database when the spreadsheet is saved (or the user presses a button).
The problem with using a scheduled job is that Excel is basically a single user application. If someone has the spreadsheet open or is doing something in it when the scheduled job runs or moves the spreadsheet to a different folder, then the job may fail.
This way you also get the updated data in your database in something close to real time instead of waiting on a job to run. This might take more time and effort to set up though than some of the other answers.
You can use SQL Server Agent to run a scheduled job that imports data from an Excel worksheet into a SQL Server table.
The import is relatively straightforward to do using Integration Services, but if you've not used either of these before you might need to do some reading up on it.
You can do the following:
You need to create an SSIS package and then create a job to run the package.
The easiest way to create the SSIS package is with "Import and Export Data" tool of SQL Server. It has a nice step by step wizard.
You set everything it asks you from the source and the destinations. Until you get here, select the "Save SSIS Package":
Then you only have to create the job to run it :)
I want to import CSV files and invoices from SAP R/3 system into a SQL database. The database will be used for reporting purpose only, please tell me what will be the best possible way, which database to use and anything else that will be relevant to me in this context? and I am novice so please help....Thanks:)
If you are routinely importing CSV files then I recommend getting them comma delimited (or whatever delimiter you choose) and going the route of making an SSIS package with a corresponding SQL Agent Job that runs daily to check for the file and run it if it finds it.
Info on SSIS package creation:
http://smallbusiness.chron.com/import-csv-ssis-46849.html
If this is a one time load then I would recommend just using the import export wizard built into SQL Server.
https://msdn.microsoft.com/en-us/library/ms140052.aspx
Pretty easy to use the import export wizard too. Right click the database > tasks > import data. This will launch the wizard and will walk you through the one time import.
Adding Microsoft's official SSIS guide as well:
https://msdn.microsoft.com/en-us/library/ms169917.aspx
I have been given the task to Load all our company's project data from MS Project into SQL Server to be able to create reports and dashboards from the project data.
I know you can export a specific projects data into a access database, but every project must have it's own access database. So my first thoughts was to create some kind of dynamic access SSIS connector and ETL. So that the project managers export their projects to access and the SSIS package takes care of the rest.
Is there a simpler way of doing this?
My company also have a SharePoint infrastructure as well as SAP BusinessObjects infrastructure. Is there a way I can accomplish this by the use of SharePoint or bobj
The simplest but costy way is called MS Project Server - it stores all data in MS SQL database already, has precalculated reporting database and so on. Here is a link http://technet.microsoft.com/en-us/evalcenter/hh973404.aspx. The last version can be either installed locally or hosted by Microsoft.
Another option I see is to use kind of automation around MS Project which will load project plan, extract all interested information and upload to to your database. There is no big magic in this solution.
The third option is to export projects to XML and then use the XML to upload data to SQL Server database. This solution is also doable through Project automation
After defining your database and column structure in SQL Server, just use Project VBA to A) collect the project and task data into an array, B) set a connection string to your database, then C) send it. I have created several applications around this procedure and it works very well.
I have a database on one server that I need to copy to another server. I can do this manually using the Export Data task, which is fine for a one time export, but I would like to speed this up as it is going to be repeated.
The database will always contain the same set of tables, I just need to get a copy of this database with it's tables and their data from one server to another.
I'd like to create some sort of reusable tool that allows you to specify the source and target database servers and then copies this specific database from one to another. Is this possible?
The Export Data task in SQL 2005 and later uses SQL Server Integration Services (SSIS) under the hood. You can save the package you're already using and run it on a schedule or on demand. You can also edit it (once it is saved) using the Business Intelligence Development Studio (BIDS).
At the end of the Export wizard (on the "Save and Run Package" screen), you can tick the "Save SSIS Package" check-box to store the package either within SQL server or on the file system. The file system is probably simpler.
Once you have the package you can execute it from the command line using the dtexec tool, or from a SQL Agent job using an Execute SSIS task.
SSIS is too big a subject to cover in full here - there are decent tutorials within SQL server books online if you need more details - alternatively, as another SO question if you get stuck.
I have a hosting account with servergrid.com. I want to backup my database, they say I have to use Sql Server Integration Service to backup the database and I would need a commercial version of Sql Server management studio.
I have Sql Server 2005 Developer Edition. I have no idea how to do SSIS backup. I tried playing around with the Sql Server Integration Services Project in VS2005 .. but I failed.
Google also seems to have no step-by-step guide for Non-DBA's like me.
Can anybody point me to/write me a Step-by-Step instructions on how to backup database using SSIS? and also if possible how to restore the database again to the host
SSIS is not really a database backup tool.
If you have a fixed number of tables, you can export the data from SQL Server to comma-separated or raw files - i.e. do the data export using SSIS. It is easy to create such package using SSIS project if you have small number of tables. But if you have a lot of tables it is all manual process. Also it could be hard to automate this - every time you add/remove/change a table or a column, you have to regenerate the SSIS package used to export your data.
SSIS does only data export, so you need a separate tool to do metadata backup (table and stored proc definitions). You can export the metadata using Management Studio (simply script database to text file).
One can call these two tools together a poor man backup tool, but if you metadata changes often you'll probably want to use the real SQL backup tool. Not sure why the hosting guys say you have to use SSIS - maybe they don't want to give you backup permissions to ensure DB safety, but SSIS is not the simplest way to do database backup.
You have a local drive on this machine, correct?
Get access to the sql prompt, through their tools or sqlcmd.
BACKUP DATABASE MYDB TO DISK = 'E:\backups\mydb.bak'
Then FTP that file offsite.
Otherwise, forget these creeps that make you use SSIS to perform tasks that can be done with one line of text.