I have looked at a few stackoverflow forum posts but nothing fits (or atleast I dont think so) what I need help with.
I'm looking for general advise, my company has 'tasked' me to look at moving some data from tables stored in our parent companies databases into a database of our own that has all the information we need in one place.
For instance if we want information that related to one thing, we may have to pull data from several different databases. I think I can get my head around the moving of the data and create a sql query to do it, however we're currently using SQL express as our SQL db (the company is more than happy to buy/create a SQL server but as far as we can see SQL express does what we need it too (feel free to correct me)).
I need to look at scheduling the data move for every hour or few hours to keep the data 'up to date' for when reports are generated using the data.
I have looked at a few programs but the as the queries and the database is on a server 2008 r2 system some of the 'programs' don't like it as they were last updated pre 2010 etc. I have also installed SQL management suite 2012 due to SQL server agent but I cant even get that worked (Service is enabled and I have restarted the DB just still nothing within suite).
I'm not looking (however happy to take the help) for a 'Do this and that and that' type reply but more than happy to accept that amount of help but if you guys / gals can point me in the right direction.
Summary:
-Combining data already on databases from our parent company into a table / DB of our own making
-Currently using SQL Express but willing to upgrade to something else that does the job
-Schedule the data moves for every X hours (Windows scheduling?)
-automating the entire thing so don't have to manually do the moves.
Help on any of the points above would be greatly appreciated and I would 'love you long time' for the help.
JB
There are a bunch of limitations for SQL Express. One of them is that SQL Agent is not supported. SSIS like SQL Agent is not supported.
http://msdn.microsoft.com/en-us/library/cc645993.aspx
Do not fret, you can always schedule a job with Windows Scheduler.
http://windows.microsoft.com/en-US/windows/schedule-task#1TC=windows-7
As for moving the data, it is up to you to select a solution.
1 - Write a PowerShell application to perform the Extract, Translate, and Load (ETL).
http://technet.microsoft.com/en-us/library/cc281945(v=sql.105).aspx
2 - Use the SQLCMD to perform logic like calling stored procedures.
http://technet.microsoft.com/en-us/library/ms162773.aspx
3 - Use BCP to dump and load data.
http://craftydba.com/?p=1245
http://craftydba.com/?p=1255
http://craftydba.com/?p=1584
http://craftydba.com/?p=1690
It is funny how youngsters think they need to spend a-lot of $ to create a solution for a business.
However, Microsoft does supply you with a-lot of free tools.
You just have to put them together for a solution.
PS: I remember about 10 years ago I created a custom ETL solution using VBSCRIPT. Unlike power shell, it is installed on old and new programs.
Good luck!
You can create a console application which executes that particular stored procedure which handles your logic. ( http://dotnet.dzone.com/articles/basics-stored-procedures-net )
Of course using SSIS is much easier but it's not available in SQL Server Express Edition.
I think you should have a look at Integartion Services, which is not available for Express Edition. Have a look at this article to get started with SSIS.
Related
I am completely baffled by SQL Server and OPENROWSET permissions.
Our team has an AD Group. This group is included in the DEV server's Windows Administrators local group. This same AD group has SysAdmin privilege on the local installation of SQL Server 2017.
Attempting to run the command:
SELECT *
FROM OPENROWSET('Microsoft.ACE.OLEDB.12.0','Excel 12.0;Database=C:\Work\test.xls;HDR=YES',['sheet1$'])
works for me, but for none of my teammates.
If there is a definitive document on the security requirements for using the OPENROWSET command - I have not found it (and please - don't refer me to learn.microsoft.com - that documentation is not written in any way that I understand).
There are other issues I have found including if I change the name of the sheet in the Excel workbook - the command fails (and yes - I closed the book after making the change).
Finally - some feedback on the use of OPENROWSET - is it generally a good idea? a bad idea? pretty much neutral but be prepared for these kinds of problems?
I hope this question is specific enough to be answered - I have probably spent 20+ hours trying to figure out how to understand how this works so I can make it work and use it consistently.
Thanks!
So honestly troubleshooting security/permissions and errors with SQL Server is probably the most frustrating aspects of my job.
First few questions and thoughts about your dilemma.
Do you really want to be granting your team connected to your db
sysadmin rights? I wouldn't do that period, full-stop.
Will the data be refreshed? If yes, I suggest you ingest this data
into a sql table with a process, perhaps python, ssis, dts package,
powershell, whatever you fancy.
If the data will always be static in that one excel file, I'd suggest perhaps making it act like a linked server for (hopefully) fewer permission issues? Also, it's easier to query that way, from my memory.
In any event, this article (non msdn link) may help? I've done it this way once before and had slightly less of a difficult time, but then again it involves adding a driver (usually) to the sql server. BUT, then I did not have to allow multiple users sysadmin - and I think ANYTHING is better than that.
https://www.sqlshack.com/query-excel-data-using-sql-server-linked-servers/
Sometimes the issue is not with the user running the query, but SQL Server using the account it runs as - to get permissions on the file. This article goes over that aspect as well. I'm not sure that is your issue as you say it works for me but not for thee, but maybe read that portion of the article at least?
I have tried to read up on this topic and I am still a bit unclear how to proceed. This seemed like a fairly basic task but it has been nowhere as simple as I had assumed. I have several SQL queries written and I want to be able to schedule them to run on a certain day each month and then automatically be exported to a .csv file in a selected folder. This will then allow them to be automatically uploaded into a BI and reporting tool that our firm uses (this part I know how to take care of).
I am fairly well versed in the writing of SQL queries, but everything beyond that I am pretty lost on. Right now I am using Microsoft SQL Management Studio 17. I thought that maybe scheduling jobs using the SQL Server Agent would be the solution, but the more I read about that and go down that path, the less I am convinced that it will allow me to export the query results into the .csv file that I need for it to be picked up. It is also important that these results are exported without headers.
Does anyone have any solutions for this? I am happy to answer any follow up questions if I am at all unclear.
You can create a job within the SQL server management studio to handle the whole thing.
http://social.msdn.microsoft.com/Forums/en/transactsql/thread/7d2280cf-3b33-46f7-ba82-4131e8a841c0
I will be honest, I know nothing about SQL Server other than what I have tried to pack into my brain in the last two days.... I have found a couple of scripts on your website that sounded like they would work What are ways to move data older than 'Y' days to an archive/history table in MySQL?. This one in particular really seems like it would fit my needs.
But I want to insert the data into a table or database on another partition of the same server and can't figure out how to change the location.
I have SQL Server 2012 Express, running on a Windows Server 2008 R2 service pack 1. We started the database on 11/21/2013 and we hit the 10 GB limit on 12/30/13. We design crowns & bridges, implants and dentures so we have multiple CT scans per patient that get manipulated in 3D imaging and CAD programs multiple times so it creates a lot of data very quickly.
Questions:
Should I try to use the triggers built in to my PatientNetDB? [OnAfterDeleteDataSets & OnAfterInsertDatasets]
If so how do I change it to make it work like the question from the user I copied above?
We may need to pull data back out of this archive, how in the heck do I do that?
I really appreciate any help you can give me, remember I am a total newb to this stuff and unfortunately will need extremely simple step by step or copy and paste directions/scripts.
Thank you so much!
Linda Saylor
No, don't use triggers for archiving/deleting. Trigger are fired when specific operations occur - INSERT, UPDATE or DELETE - on certain tables, and you cannot control when and how often triggers are fired. Therefore, triggers should be very small and nimble - you should NOT put large and long-running operations into a trigger. A typical trigger might update a row that's been inserted, or it might put a row into a separate table (an Audit or Command table) - but the trigger itself should never much processing.
What you can and should do is to have scheduled tasks - unfortunately, the SQL Server Agent is not available in the Express edition. With SQL Server Agent you could run certain processing operations (T-SQL scripts) at scheduled intervals, e.g. once every night etc.
Since you're using the Express edition, you'll have to find another way to run a task at given times, possibly by writing a small wrapper in your language of choice (C#, VB.NET, whatever), have that scheduled by the Windows scheduler (Scheduled Tasks in your Windows start menu), which would then kick off / execute a T-SQL script to run the cleanup process and archive your data.
I'm in a database class and the teacher wants us to connect through ssh to an oracle database setup on a school server and it's been extremely frustrating. She wants us to turn in an sql file that will create all the necessary table, insert tuples, run certain select commands which I've found to be very hard to get an sql file with everything after i get everything right and I haven't found a way to test the sql file against the server and I don't think I have permission to drop tables anyway. Anyway my question is there a way I can take an sql file with create table and insert commands to convert it to something like an access .mba database or something local i can mess around with? and help would be greatly appreciated didn't find much help on google.
You seem to be confusing terminology a bit; SQL*Plus is a client application, and the database is a shared server resource. You want to create schema objects from an SQL file, I think. But anyway...
There's a very useful online resource for experimenting with bits of SQL in various flavours, SQL Fiddle. Technically not 'offline' of course, but I'm taking that to mean off your school's network, not necessarily completely isolated. You can create tables and run your inserts in the schema panel, and then run queries against that. Make sure you pick the right database product from the drop-down menu so you're using syntax that is valid for your class. You'll see a lot of answers here with links to demonstration fiddles.
That's great for a lot of things but if you want something a bit more robust and scalable, and entirely offline, you can install VirtualBox and get a pre-built developer VM image which gives you a ready-to-go Linux environment with a database installed and running. You can run whatever you want against that, you have SQL*Plus and SQL Developer available, and you can connect to the DB from your host machine if you want to. You can create and test your scripts against that, and in a format that will be closer to what you have to hand in than you'd use with SQL Fiddle.
This is much less work than installing the Oracle software yourself and learning how to create and manage the database, which I'm guessing is a bit more advanced than you need at the moment, based purely on the kinds of thing your question suggests you're dong at the moment. I think you'd learn a lot from the installation and build process, but I'd get comfortable with Oracle first, and maybe practice in a VM first as it's so much easier to trash it and start again when you mess something up.
If I wanted 'something local I can mess around with', I would go for a VM image. Mo posted a walkthrough of the VM setup as a comment to a previous similar answer, which you might find helpful.
"Something local I can mess around with" in terms of Oracle Database is Oracle Database 11g Express Edition. It's free and can be downloaded from oracle.com. You certainly can test sql-files run through sqlplus on Oracle Database XE.
To get the MS Access (GUI) feeling, download SQL Developer. It's free.
Best of luck!
Bjarte
I have a 2 MS SQL 2005 databases,a TEST and DEV database. Now our developer added some extra columns,tables etc in the DEV database.This created differences in the TEST database.is there a script i can write tha can tell me what the changes where in the DEV database between certain dates...i found a couple of tools but they are quite basic and dont really generate change scripts etc. Also tried the change script function in management studio but it seems to be working when the change is first made and not later.
Appreciate your thoughts.
A.
redgate SQL Compare is a good tool to check for differences between databases and even sync them.
However, if you need to do this for free, try this: within in SQL Server Management Studio do the following:
1) script the entire schema of TEST to a file, look here if you don't know how
2) repeat step #1 but for the DEV database
3) diff the files using something like Beyond Compare, should have a 30 trial
I wrote dbscript, and one of its features is to compare two database schemas and create a migration script.
What you need is a way to manage changes to your database schema and then apply them in a controlled and consistent manner. Moreover, you need a single authoritative source for the database schema.
For all this, check out Wizardby:
(source: googlecode.com)