Currently We have developed a system for a manual work they have been doing using many excel files.
Is there a best practice for data migration? because I wanted to use backend language like .net to do the validation and insert into tables rather than using SQL to do migration.
Total record in excel is around 12K rows but for many tables so its not needed consider a lot about performance and it is only one time.
I would add a few calculated columns in Excel that would generate SQL Insert / Update scripts. Something like ="INSERT INTO table (column) VALUES ('"&A1&"');"
Then just copy calculated column and run it through SQL client. I used to have a macros to run it directly from Excel through OLEDB that would highlight failed expressions and store SQL Exceptions next to them.
That way the data can be easily tidied, corrected and SQL re-run as needed.
Related
I am looking to see if the capability is there to have a custom SSMS sql query imported in SPSS (Statistical Package for the Social Sciences). I would want to build syntax that generates this query as my new dataset that I can then continue my scripted analysis. I see the basic query capability of one table from a Sql Server but I would like to create a query that joins to many tables. I anticipate the query to be a bit complex with many joins and perhaps data transformations.
Has anybody had experience or a solution to this situation?
I know I could take the query and make a table of it that SPSS can then connect to but my data changes daily and I would need a job in another application to refresh this table before my SPSS syntax would pull it and I would like to eliminate that first step by just having the query that grabs the data at the beginning of my syntax.
Ultimately I am looking to build out my SPSS syntax and schedule it in the Production Facility to run daily.
We need to automate the SQL row count, as of now we are using an Excel formula to fill count(*) query to all tables and copying it and pasting in Oracle SQL Developer and running all at once.
So, what I am looking for: is there any way to get automated by using Python or any programming language which directly connects to a database by asking necessary inputs and get all the process done by itself and get me row count for each table without involving me or any process said above.
After trying a few different packages and methods found online, I am yet to find a solution that works for inserting a dataframe from R into an existing table in SQL Server.
I've had great success doing this with MySQL, but SQL Server seems to be more difficult.
I have managed to write a new table using the DBI package, but I can't find a way to insert into using this method. Looking at the documentation, there doesn't seem to be a way of inserting.
As there are more than 1000 rows of data, using sqlQuery from the RODBC package also seems unfeasable.
Can anybody suggest a working method for inserting large amounts of data from a dataframe into an existing SQL table?
I've had similar needs using R and PostGreSQL using the r-postgres-specific drivers. I imagine similar issues may exist with SQLServer. The best solution I found was to write to a temporary table in the database using either dbWriteTable or one of the underlying functions to write from a stream to load very large tables (for Postgres, postgresqlCopyInDataframe, for example). The latter usually requires more work in terms of defining and aligning SQL data types and R class types to ensure writing, wheres dbWriteTable tends to be a bit easier. Once written to a temporary table, to then issue an SQL statement to insert into your table as you would within the database environment. Below is an example using high-level DBI library database calls:
dbExecute(conn,"start transaction;")
dbExecute(conn,"drop table if exists myTempTable")
dbWriteTable(conn,"myTempTable",df)
dbExecute(conn,"insert into myRealTable(a,b,c) select a,b,c from myTempTable")
dbExecute(conn,"drop table if exists myTempTable")
dbExecute(conn,"commit;")
I have a stored procedure that takes a table name and writes out a series of INSERT statements, one for each row in the table. It's being used to provide sample, "real world" data for our test environment.
It works well but some of these sample rowsets are 10, 20k records. The stored proc writes them out using the PRINT statement and it's hard to copy that many lines and paste them into the management studio to run them. Is there a SQL redirect feature I might be able to use, perhaps to write this output to a table and a way to loop through and run each statement that way? Just a thought.
I'd like to do this all from within the management studio and not have to write a C# program to create a dataset and loop over it, etc. I'm essentially looking for suggestions on a good approach. Thanks very much.
Use EXEC:
PRINT #INSERT_statement
EXEC #INSERT_statement
...to run the query.
But I'd recommend looking at bulk insertion to make the data load faster:
BULK INSERT
Where is your stored procedure getting this data?
You may want to look into importing it as a table and then running your stored procedure against that inserted table. SQL Server Management studio has many options for importing data.
If your stored proc is generating the data - then that's a whole other issue.
Is it possible to search and replace all occurrences of a string in all columns in all tables of a database? I use Microsoft SQL Server.
Not easily, though I can thing of two ways to do it:
Write a series of stored procedures that identify all varchar and text columns of all tables, and generate individual update statements for each column of each table of the form "UPDATE foo SET BAR = REPLACE(BAR,'foobar','quux')". This will probably involve a lot of queries against the system tables, with a lot of experimentation -- Microsoft doesn't go out of its way to document this stuff.
Export the entire database to a single text file, do a search/replace on that, and then re-import the entire database. Given that you're using MS SQL Server, this is actually the easier approach. Microsoft created the Microsoft SQL Server Database Publishing Wizard for other reasons, but it makes a fine tool for exporting all of the tables of a SQL Server database as a text file containing pure SQL DDL and DML. Run the tool to export all of the tables for a database, edit the resulting file as you need, and then feed the file back to sqlcmd to recreate the database.
Given a choice, I'd use the second method, as long as the DPW works with your version of SQL Server. The last time I used the tool, it met my needs (MS SQL Server 2000 / 2005) but it had some quirks when working with database Roles.
In MySQL, you can do it very easily like this:
update [table_name] set [field_name] = replace([field_name],'[string_to_find]','[string_to_replace]');
I have personally tested this successfully on a production server.
Example:
update users set vct_filesneeded = replace(vct_filesneeded,'.avi','.ai');
Ref: http://www.mediacollege.com/computer/database/mysql/find-replace.html
A good starting point for writing such a query is the "Search all columns in all the tables in a database for a specific value" stored procedure. The full code is at the link (not trivial, but copy/paste it and use it, it just works).
From there on it's relatively trivial to amend the code to do a replace of the found values.