I have a package that is used for DQS cleansing. I have nearly 650,000+ records to clean however after about 350,000 records processed I get a symptom that hangs my project up. For example, I will close my visual studio project/solution but once I try to go back to open my project I get the message "visual studio is waiting for an internal operation" in the lower right hand corner. Once this happens I can't click or scroll anywhere in my project.
I am using SQL 2012 to move data from one table to another but into another database within the SQL 2012 server/instance. In addition, I'm using the DQS client to clean and validate data for last name, state and country. My visual studio is version 2010. I'm running this all on a virtual machine that has 8g of ram and 4 cores. I do have the cumulative service pack installed for SQL 2012.
At this point, I have to kill my VS2010 in task manager but can't seem to work my SSIS project any more. I have to delete all my records in my destination table then I can get into my project once more.
Thanks for any help or ideas,
Michael
DQS Cleansing is a VERY resource intensive task. According to the Data Quality Services Performance Best Practices Guide, even when adhering to hardware recommendations and best practices, DQS cleansing on 1 million rows can take between 2-3.5 hrs.
Also, I agree with Pondlife's comments about running in BIDS vs DTEXEC. BIDS/SSDT is 32bit (limiting memory to 2-3GB) while DTEXEC has a 64bit version which can use way more memory.
Related
I am currently creating a database for my project and was wondering where I should save the table. Should I just save it in documents (D:\Documents) or should I save it in the same folder where my project is saved (D:\Documents\Visual Studio 2015\Projects\ALevelCompUnit3)? I was hoping to develop it at school.
Also, the school only provides Visual Basic 2008; would I have to change the framework to do it at school?
SQL Server Express in fact is a light weight version of the SQL server engine that is stored locally on your file system under the extension .mdf. Within this file you will be saving things like tables, views and stored procedures. You can move this file around similar to other traditional file types and where you store it while your developing your application is up to you. If, and when you application grows you may consider moving it over to a full version of SQL Server and there are tools to quickly migrate over. In regards to your Visual Studio versions - Using 2015 at home and switch over to 2008 at school will be a little painful. There are many features of 2015 that are not available in 2008. You should be able to access the .mdf file but your experience will not be as powerful in terms of capabilities.
On my development box, I always run the latest version of SQL Server. I often deploy databases from my dev box to a live/staging area for review or testing. I've done this many times and it has always been a painful process, but I am realizing that I need to find an easier, more reliable and consistent way of performing this basic operation.
I normally use WebMatrix purely for deployment and it's worked fine, but I've been having problems getting it to work on my server for some reason. Consequently, I am seeking an alternative solution.
Creating a SQL dump file would probably work, but it's not an acceptable solution a database contains images and easily exceeds 2 gigs of data which would take forever.
The Import/Export utility fails due to issues with incomplete schema copies, identity inserts and checks. The solutions offered for these issues has failed to work in my particular case.
The Backup and Restore method also fails due to some strange incompatibilities between SQL Server 2008 and 2012. SQL Server 2008 Management Studio throws exceptions during the restore process of a 2012 database. It's odd that this happens, even though I set the compatibility of the database to version 2008.
I haven't tried the Detaching, copying and reattaching files method, but I haven't bothered trying since it would probably fail for the same reasons the backup and restore method did.
Are there other alternatives out there? Also, why is this so unbelievable hard for a task that is so common and important, especially in this day in age of 2013? Get real Microsoft!
We changed our method of deploying and moving databases between servers, instances and versions by adopting the tools from RedGate. They are expensive, but worth it IMHO.
My team creates scripts for ~everything.
Database Creation, Alter, Inserts, etc, etc.
And we write all scripts that check for the existence of things before trying to create them.
Aka, we can run the scripts over and over and get the same results.
And we deploy to different environments by using SqlCmd.exe.
EDIT
See:
http://odetocode.com/blogs/scott/archive/2008/02/02/versioning-databases-views-stored-procedures-and-the-like.aspx
and
http://odetocode.com/blogs/scott/archive/2008/01/30/three-rules-for-database-work.aspx
=============
If that is "too much" then I agree with the other poster, RedGate is your friend.
Points below aside, have you considered the Database Projects within VS2012? they allow you to script off the tables, sp's, triggers, users etc you want, generate sql cmd scripts, make changes and schema compare and version control your database code, I'd certainly recommend it
"Creating a SQL dump file would probably work, but it's not an
acceptable solution a database contains images and easily exceeds 2
gigs of data which would take forever."
Why is this a problem? where are you transferring the file from and to and over what connection?
"The Backup and Restore method also fails due to some strange
incompatibilities between SQL 2008 and 2012. SQL 2008 Management
Studio throws exceptions during the restore process of a 2012
database. It's odd that this happens, even though I set the
compatibility of the database to version 2008"
This shouldn't be an issue if file is created in 2008 prior to restoring. If you create a new DB in your 2008 instance, then take a backup from that and restore it to a 2012 instance with 2008 compatiblity, then you should be able to use it there, back it up from the 2012 instance and restore to 2008 again afterwards.
I am using only VS2012 Ultimate Server Xplorer Pane to code and execute my job. I used to add a job scheduling to perform some task at specific time.
Now that I realize my executed command implemented is incorrect. I can delete this by making another new query on the database then recreate a new similar job but I don't want to do this.
I don't know how to fix the command text.
Get Management Studio, and stop trying to do this through Visual Studio.
http://www.microsoft.com/en-us/download/details.aspx?id=43351
Download SQLManagementStudio_x64_ENU.exe (unless you are on x86 for some reason, in which case pick the other one). It's ~1GB, not sure why, sorry about that.
Starting with 2012 SP1, Management Studio Express is fully functional with no license requirement (this is the first time this has happened).
Connect to your server, expand SQL Server Agent > Jobs, and double-click your job. Nice enough UI that will let you edit steps, schedules, etc. without doing any of this hokey generate-an-entire-script-for-a-job stuff...
I'm a systems programmer myself and completely new to DTS or any ETL etc. I have a situation here and would greatly appreciate your support.
we are using MS SQL 2000 till now in project and now efforts are on way to move to 2008.
I can see some .dts files which were created long back with MSSQL2000. VB script(.bas) files were generated from these .dts files and they are run in the VB program, till date.
Problem::
Now as we are moving to MS SQL 2008 ==>
I need to make sure that the VB program(generated from .dts files earlier) are good to be run and MAINTAINED even with MSSQL 2008 R2.
Question:
Firstly can these programs be maintained even in MSSQL2008 (without MSSQL2000) ?
If yes, do i need to any dll's or references etc?
I could see one of my team member migrating the .dts to .dtsx files, is there a way to generate VB files(.bas) from .dtsx ?
ETL solutions in SQL Server 2000 (DTS) and SQL Server 2008 (SSIS) are incompatible, but you could use Execute DTS task from SSIS package. You don't need any dll's for it.
That'll give you some time before you have a chance to rewrite your system. You'll have to do it eventually, because 'Execute DTS' is deprecated component and won't be supported in future versions.
I enjoy using SQL Server Management Studio for change and updating my database. Its easier, faster, and safer than writing changes myself.
I was looking into using some sort of version control for databases and read about using a SQL Server Database Project in Visual Studio 2010.
I scripted out an existing database and imported it into a new SQL Server Database Project. Now, from what I can tell, there is no GUI to edit the database; I can't add columns, change datatypes, or edit existing data without scripting it myself. For instance, in SQL Server Management Studio I can right click on a table-name and select "Design" and then add/edit columns, change datatypes, etc from there.
While Visual Studio's Database Projects has some features SQL Management Studio does not have I don't think I can live without a "table designer".
Is there a table designer built into VS's Database Project I'm just not seeing?
No, there's no table designer.
If you're starting to think about version controlling your database, you ought also to be thinking about writing actual SQL to implement your database objects. That's the route that the Database Projects force you down. If you can't write the SQL for your database changes, how are you going to be able to review and appreciate a diff between how a table was 6 months ago, and how it currently is in your project?
I've been using VS2008 Database Projects for about 10 months now for our version control. Every now and then I do still use the table designer, it is a quick and easy tool. I believe the majority of your question centers around workflow as this is what I found to be the most challenging part about development in a version controlled environment. I would recommend continuing to design your objects in Management Studio or however you're comfortable and then do a create script and import that script into your Database project. There are some quirks when doing this, you'll need to always script the create statement even if you're performing an alter in your environment. As well you'll need to remove any USE statements for your database as the context in which you're importing your scripts will always be in the project you're importing to.
We have found that a successful workflow for us to facilitate code deployment is to have a production branch which is branched to a Main (development branch) and then to test. All new development is done in Main and merged by changeset to each other environment as required.
You can import your scripts from your development environment by right clicking in the solution and clicking import scripts. I recommend that you check all the options to overwrite objects that exist, import extended properties and import permissions.
After changing your DB schema using SSMS's GUI tool, you can use Database project's Schema Compare tool to update your project files (set the source to be your database and target to be your project). This way you can keep using GUI tool to manage the schema and the database project will manage the versioning.
There is no visual table designer in Visual Studio 2010 Database Project. But, concerning version control for databases, there is a workaround - you can use SQL Server Management Studio together with Red Gate's SQL Source Control. It costs some money but definitely is worth it.