I have a folder with multiple .bak files of sql server.
I want to restore them to sql server. How can i do a script so that all backuped files from that folder can be restored at once.
RESTORE DATABASE [db1] FROM DISK = N'C:\folder\db1.bak' WITH FILE = 1, MOVE N'DB_Data' TO N'C:\folder\db2.mdf', MOVE N'DB_log' TO N'C:\folder\db1.LDF', NOUNLOAD, STATS = 10
GO
RESTORE DATABASE [db2] FROM DISK = N'C:\folder\db2.bak' WITH FILE = 1, MOVE N'DB_Data' TO N'C:\folder\db2.mdf', MOVE N'DB_Log' TO N'C:\folder\db2.LDF', NOUNLOAD, STATS = 10
GO
Try something like this.
I guess databases name is not written into .bak file. No idea how to retreive by script. You may use file name or any other temp name for restored databases and after restore rename it under SSMS.
Be aware that .bak files may contain DIFF backups. In case of DIFF backup, it won't work just like that - one by one.
If folder contains FULL BACKUP and TRANSACTION LOG backups it's always the best idea to start restoring database using backup history in SSMS.
Let's talk about restoring multiple databases from multiple BAK files.
I think GAWK may help. It's not the MS way, but may be the fastest way.
Do you know GAWK?
Let you build input file with file names:
dir *.bak > input.txt
Write GAWK program into file program.gawk.txt:
BEGIN {
printf "-- My auto-generated script \n"
}
{
printf "RESTORE DATABASE [substr($1,1,length($1)-4)] FROM DISK = N'$1 \n'"
printf "GO \n\n"
}
END {
printf "-- End of script"
}
Experiment with substr to generate temp name for restored database based on BAK file name. Not sure if substr expression is all right - please check yourself.
$1 will be changed to file name from input.txt
substr($1,1,length($1)-4) is for cutting ".BAK" from filename
Build output script by:
gawk --file=program.gawk.txt input.txt > my_script.sql
..and review it.
Maybe it's possible to do this powershell-way. Don't know.
Download GAWK from: http://gnuwin32.sourceforge.net/packages/gawk.htm
Related
I was having issues with the SQL maintanence plan cleanup task not deleting one particularly large database backup each night, it works for days then fails then starts working again and it always works correctly on the small databases.
I researched this maintanence plan cleanup task issue and tried everything I could think of to get it working, changed extension matching to *, added a \ at the end of folder path, changed age no NONE so all files would be deleted regardless of age, and still sometimes this backup is not getting deleted.
So I implemented this SQL JOB using the script below to see if that would work, but the same issue again, intermittently the large backup file is not deleted, when I run the task manually it seems to always delete it.
My question here is, is there a way to firstly get a list of files that match the delete criteria and write them to a log file before actually attempting to delete the files, that way I could at least see if for some reason the large backup file is not matching the criteria to be deleted in the first place.
Any assistance to otherwise delete the old backup files using T-SQL without using xp_cmdshell and without using a batch or powershell script would be appreciated.
declare #dt datetime
select #dt=dateadd(hh,-22,getdate())
EXECUTE master.dbo.xp_delete_file 0,N'Z:\SQLBackups\',N'BAK',#dt,1
The version of SQL server I'm having the issue with:
Microsoft SQL Server Management Studio 10.50.4042.0
Microsoft Analysis Services Client Tools 10.50.4042.0
Microsoft Data Access Components (MDAC) 6.1.7601.17514
Microsoft MSXML 3.0 6.0
Microsoft Internet Explorer 9.10.9200.17609
Microsoft .NET Framework 2.0.50727.5485
Operating System 6.1.7601
After creating a powershell script to delete the files and finding the error "Another process is using the file" I made this script to check what process that is using the handle64.exe program and found it was the Commvault agent CLBackup.exe that was locking the file.
Backup schedule conflict is the cause of the issue.
$log = ($MyInvocation.MyCommand.Path).TrimEnd("ps1") + "log"
$handlelog = ($MyInvocation.MyCommand.Path).TrimEnd(".ps1") + "-Handle.log"
$1 = gci 'Z:\SQLBackups' | %{gci $_.fullname -Filter '*.BAK' | ? {$_.LastWriteTime -le (get-date).addhours(-22)}}
write-output "$(get-date -format g) Files will be deleted: $1" >> $log
$1 | % {remove-item $_.fullname -force -Confirm:$FALSE}
if (!!($error)) {
write-output "$(get-date -format g) Open Handles on .BAK files:" >> $handlelog
$exec = "$env:SystemRoot\system32\cmd.exe /c" + (Split-Path($MyInvocation.MyCommand.Path)) + "\handle64.exe .BAK -u -nobanner -accepteula"
Invoke-Expression -Command:$exec >> $handlelog
$error >> $log
}
How do I populate the Transaction Processing Performance Council's TPC-DS database for SQL Server? I have downloaded the TPC-DS tool but there are few tutorials about how to use it.
In case you are using windows, you gotta have visual studio 2005 or later. Unzip dsgen in the folder tools there is dsgen2.sln file, open it using visual studio and build the project, will generate tables for you, I've tried that and I loaded tables manually into sql server
I've just succeeded in generating these queries.
There are some tips may not the best but useful.
cp ${...}/query_templates/* ${...}/tools/
add define _END = ""; to each query.tpl
${...}/tools/dsqgen -INPUT templates.lst -OUTPUT_DIR /home/query99/
Let's describe the base steps:
Before go to the next steps double-check that the required TPC-DS Kit has not been already prepared for your DB
Download TPC-DS Tools
Build Tools as described in 'v2.11.0rc2\tools\How_To_Guide-DS-V2.0.0.docx' (I used VS2015)
Create DB
Take the DB schema described in tpcds.sql and tpcds_ri.sql (they located in 'v2.11.0rc2\tools\'-folder), suit it to your DB if required.
Generate data that be stored to database
# Windows
dsdgen.exe /scale 1 /dir .\tmp /suffix _001.dat
# Linux
dsdgen -scale 1 -dir /tmp -suffix _001.dat
Upload data to DB
# example for ClickHouse
database_name=tpcds
ch_password=12345
for file_fullpath in /tmp/tpc-ds/*.dat; do
filename=$(echo ${file_fullpath##*/})
tablename=$(echo ${filename%_*})
echo " - $(date +"%T"): start processing $file_fullpath (table: $tablename)"
query="INSERT INTO $database_name.$tablename FORMAT CSV"
cat $file_fullpath | clickhouse-client --format_csv_delimiter="|" --query="$query" --password $ch_password
done
Generate queries
# Windows
set tmpl_lst_path="..\query_templates\templates.lst"
set tmpl_dir="..\query_templates"
set dialect_path="..\..\clickhouse-dialect"
set result_dir="..\queries"
set tmpl_name="query1.tpl"
dsqgen /input %tmpl_lst_path% /directory %tmpl_dir% /dialect %dialect_path% /output_dir %result_dir% /scale 1 /verbose y /template %tmpl_name%
# Linux
# see for example https://github.com/pingcap/tidb-bench/blob/master/tpcds/genquery.sh
To fix the error 'Substitution .. is used before being initialized' follow this fix.
I'm trying to attach a FileStream enabled database without a log file. My SQL looks something like this:
USE master
CREATE DATABASE MyDB
ON PRIMARY(NAME = N'MyDB', FILENAME = 'C:\myDB.MDF' ),
FILEGROUP myFileGroup CONTAINS FILESTREAM ( NAME = myData, FILENAME = 'C:\myFileGroup')
For Attach
Here is the error I'm receiving:
Msg 5173, Level 16, State 3, Line 2
One or more files do not match the primary file of the database.
If you are attempting to attach a database, retry the operation with the correct files.
If this is an existing database, the file may be corrupted and should be restored from a backup.
Does anyone know if it's possible to attach a FileStream enabled database without the original log file?
Try this blog post:
http://blog.sqlauthority.com/2010/04/26/sql-server-attach-mdf-file-without-ldf-file-in-database/
I would personally go with this one:
CREATE DATABASE TestDb ON
(FILENAME = N'C:\Database\Test\TestDb.mdf')
FOR ATTACH_REBUILD_LOG
GO
And when you have your log rebuilt, you can enable filestream; or try to reattach with the filestream location.
I create database "Test" in folder "d:\test". Database files are "d:\test\Test.mdf" and "d:\Test\Test_log.ldf". I detach database from MS SQL Server 2008 R2, copy all files to new folder ("d:\test_new"), delete log file ("d:\test_new\Test_log.ldf"), and try to attach database again from new location. When I use SQL Server Management Studio, and choose "d:\test_new\Test.mdf" file, it determines that log file is located in "d:\test\Test_log.ldf" (old location). How can I attach this database with rebuilding log in new location? Just imagine, that I cannot copy ldf file again to new location, and that it is still available there, so SQL Server see it anyway. I want to say to SQL Server - "please, forget that log file, and create new log file here". It's be better if you help me with T-SQL script, but if it will be steps in Management studio - I will convert it to script myself.
What I had tried already:
1.
CREATE DATABASE [test]
ON ( FILENAME = N'D:\test_new\test.mdf' )
FOR ATTACH_REBUILD_LOG
attaches log file from old location (FOR ATTACH - the same)
2.
CREATE DATABASE [test]
ON ( FILENAME = N'D:\test_new\test.mdf' )
LOG ON ( FILENAME = N'D:\test_new\test_log.ldf' )
FOR ATTACH_REBUILD_LOG
returns an error: Unable to open the physical file "D:\test_new\test_log.ldf". Operating system error 2: "2(File not found.)".
3.
sp_attach_db and sp_attach_single_file_db
was tried too. And I even had checked their source codes - they just create dynamic SQL and call CREATE DATABASE ... FOR ATTACH statement.
The question is slowly changed to: "Is it possible?"
UPDATE
Well, it looks like it's not possible with current versions of SQL Server. If anybody knows a way to do it - please, I will be very pleased to know it too!
Edit2: To my knowledge, it is not possible for SQL Server to recreate a log file. It can shrink the ldf, but not create it when only the mdf exists.
When you copy your files from d:\test\ to d:\test_new\, do not delete the d:\test_new\Test_log.ldf.
Leave the log file there, because you cannot reattach the new DB without that log file. Afterwards, you can shrink that log to a minimum size.
So, to synthesize:
Copy your files from d:\test\ to d:\test_new\ and leave the log
file there.
Run your create database script that you posted in your question (point 2).
Run the following script to shrink the log to a minimum size
.
USE test
GO
DBCC SHRINKFILE(logicalFileName, 1)
GO
To find out what logicalFileName is, run sp_helpfile, that will give you the logical file name for your log file:
USE test
GO
EXEC sp_helpfile
GO
more info here
Edit:
I think you need first to detach the test database from the old location:
(You might create a script that does it all, from the following commands)
C:\> osql -E
1> sp_detach_db 'test'
2> go
3> quit
C:\>
Then copy the files to the new location.
C:\> copy d:\test\* d:\test_new\*
Next, attach the test DB to the new path location:
C:\> osql -E
1> sp_attach_db #dbname = N'test', #filename1 = N'd:\test_new\Test.mdf', #filename2 = N'd:\test_new\Test_log.ldf'
2> go
3> quit
C:\>
to test if the new database was successfully attached:
C:\> osql -E
1> use test
2> go
3> quit
C:\>
If there are no errors after the go command, then all is ok
Hope this helps
Microsoft article on how to move files
The users must copy BOTH the .mdf and .ldf files. They then have to use the following command (one or the other).
sp_attach_db (deprecated, use the CREATE DATABASE WITH ATTACH in the future)
EXEC sp_attach_db #dbname = 'dbname', #filename1='d:\test_new\test.mdf', #filename2='d:\test_new\test.ldf'
This will result in the databas using the data file (mdf) and transaction log (ldf) from the \test_new directory
CREATE DATABASE FOR ATTACH
CREATE DATABASE dbname ON '(FILENAME=d:\test_new\test.mdf'), (FILENAME='d:\test_new\test.ldf') FOR ATTACH
When using SQL Server Express 2005's User Instance feature with a connection string like this:
<add name="Default" connectionString="Data Source=.\SQLExpress;
AttachDbFilename=C:\My App\Data\MyApp.mdf;
Initial Catalog=MyApp;
User Instance=True;
MultipleActiveResultSets=true;
Trusted_Connection=Yes;" />
We find that we can't copy the database files MyApp.mdf and MyApp_Log.ldf (because they're locked) even after stopping the SqlExpress service, and have to resort to setting the SqlExpress service from automatic to manual startup mode, and then restarting the machine, before we can then copy the files.
It was my understanding that stopping the SqlExpress service should stop all the user instances as well, which should release the locks on those files. But this does not seem to be the case - could anyone shed some light on how to stop a user instance, such that it's database files are no longer locked?
Update
OK, I stopped being lazy and fired up Process Explorer. Lock was held by sqlserver.exe - but there are two instances of sql server:
sqlserver.exe PID: 4680 User Name: DefaultAppPool
sqlserver.exe PID: 4644 User Name: NETWORK SERVICE
The file is open by the sqlserver.exe instance with the PID: 4680
Stopping the "SQL Server (SQLEXPRESS)" service, killed off the process with PID: 4644, but left PID: 4680 alone.
Seeing as the owner of the remaining process was DefaultAppPool, next thing I tried was stopping IIS (this database is being used from an ASP.Net application). Unfortunately this didn't kill the process off either.
Manually killing off the remaining sql server process does remove the open file handle on the database files, allowing them to be copied/moved.
Unfortunately I wish to copy/restore those files in some pre/post install tasks of a WiX installer - as such I was hoping there might be a way to achieve this by stopping a windows service, rather then having to shell out to kill all instances of sqlserver.exe as that poses some problems:
Killing all the sqlserver.exe instances may have undesirable consequencies for users with other Sql Server instances on their machines.
I can't restart those instances easily.
Introduces additional complexities into the installer.
Does anyone have any further thoughts on how to shutdown instances of sql server associated with a specific user instance?
Use "SQL Server Express Utility" (SSEUtil.exe) or the command to detach the database used by SSEUtil.
SQL Server Express Utility,
SSEUtil is a tool that lets you easily interact with SQL Server,
http://www.microsoft.com/downloads/details.aspx?FamilyID=fa87e828-173f-472e-a85c-27ed01cf6b02&DisplayLang=en
Also, the default timeout to stop the service after the last connection is closed is one hour. On your development box, you may want to change this to five minutes (the minimum allowed).
In addition, you may have an open connection through Visual Studio's Server Explorer Data Connections, so be sure to disconnect from any database there.
H:\Tools\SQL Server Express Utility>sseutil -l
1. master
2. tempdb
3. model
4. msdb
5. C:\DEV_\APP\VISUAL STUDIO 2008\PROJECTS\MISSICO.LIBRARY.1\CLIENTS\CORE.DATA.C
LIENT\BIN\DEBUG\CORE.DATA.CLIENT.MDF
H:\Tools\SQL Server Express Utility>sseutil -d C:\DEV*
Failed to detach 'C:\DEV_\APP\VISUAL STUDIO 2008\PROJECTS\MISSICO.LIBRARY.1\CLIE
NTS\CORE.DATA.CLIENT\BIN\DEBUG\CORE.DATA.CLIENT.MDF'
H:\Tools\SQL Server Express Utility>sseutil -l
1. master
2. tempdb
3. model
4. msdb
H:\Tools\SQL Server Express Utility>
Using .NET Refector the following command is used to detach the database.
string.Format("USE master\nIF EXISTS (SELECT * FROM sysdatabases WHERE name = N'{0}')\nBEGIN\n\tALTER DATABASE [{1}] SET OFFLINE WITH ROLLBACK IMMEDIATE\n\tEXEC sp_detach_db [{1}]\nEND", dbName, str);
I have been using the following helper method to detach MDF files attached to SQL Server in unit tests (so that SQ Server releases locks on MDF and LDF files and the unit test can clean up after itself)...
private static void DetachDatabase(DbProviderFactory dbProviderFactory, string connectionString)
{
using (var connection = dbProviderFactory.CreateConnection())
{
if (connection is SqlConnection)
{
SqlConnection.ClearAllPools();
// convert the connection string (to connect to 'master' db), extract original database name
var sb = dbProviderFactory.CreateConnectionStringBuilder();
sb.ConnectionString = connectionString;
sb.Remove("AttachDBFilename");
var databaseName = sb["database"].ToString();
sb["database"] = "master";
connectionString = sb.ToString();
// detach the original database now
connection.ConnectionString = connectionString;
connection.Open();
using (var cmd = connection.CreateCommand())
{
cmd.CommandText = "sp_detach_db";
cmd.CommandType = CommandType.StoredProcedure;
var p = cmd.CreateParameter();
p.ParameterName = "#dbname";
p.DbType = DbType.String;
p.Value = databaseName;
cmd.Parameters.Add(p);
p = cmd.CreateParameter();
p.ParameterName = "#skipchecks";
p.DbType = DbType.String;
p.Value = "true";
cmd.Parameters.Add(p);
p = cmd.CreateParameter();
p.ParameterName = "#keepfulltextindexfile";
p.DbType = DbType.String;
p.Value = "false";
cmd.Parameters.Add(p);
cmd.ExecuteNonQuery();
}
}
}
}
Notes:
SqlConnection.ClearAllPools() was very helpful in eliminating "stealth" connections (when a connection is pooled, it will stay active even though you 'Close()' it; by explicitely clearing pool connections you don't have to worry about setting pooling flag to false in all connection strings).
The "magic ingredient" is call to the system stored procedure sp_detach_db (Transact-SQL).
My connection strings included "AttachDBFilename" but didn't include "User Instance=True", so this solution might not apply to your scenario
I can't comment yet because I don't have high enough rep yet. Can someone move this info to the other answer so we don't have a dupe?
I just used this post to solve my WIX uninstall problem. I used this line from AMissico's answer.
string.Format("USE master\nIF EXISTS (SELECT * FROM sysdatabases WHERE name = N'{0}')\nBEGIN\n\tALTER DATABASE [{1}] SET OFFLINE WITH ROLLBACK IMMEDIATE\n\tEXEC sp_detach_db [{1}]\nEND", dbName, str);
Worked pretty well when using WIX, only I had to add one thing to make it work for me.
I had took out the sp_detach_db and then brought the db back online. If you don't, WIX will leave the mdf files around after the uninstall. Once I brought the db back online WIX would properly delete the mdf files.
Here is my modified line.
string.Format( "USE master\nIF EXISTS (SELECT * FROM sysdatabases WHERE name = N'{0}')\nBEGIN\n\tALTER DATABASE [{0}] SET OFFLINE WITH ROLLBACK IMMEDIATE\n\tALTER DATABASE [{0}] SET ONLINE\nEND", dbName );
This may not be what you are looking for, but the free tool Unlocker has a command line interface that could be run from WIX. (I have used unlocker for a while and have found it stable and very good at what it does best, unlocking files.)
Unlocker can unlock and move/delete most any file.
The downside to this is the apps that need a lock on the file will no longer have it. (But sometimes still work just fine.) Note that this does not kill the process that has the lock. It just removes it's lock. (It may be that restarting the sql services that you are stopping will be enough for it to re-lock and/or work correctly.)
You can get Unlocker from here: http://www.emptyloop.com/unlocker/
To see the command line options run unlocker -H
Here they are for convenience:
Unlocker 1.8.8
Command line usage:
Unlocker.exe Object [Option]
Object:
Complete path including drive to a file or folder
Options:
/H or -H or /? or -?: Display command line usage
/S or -S: Unlock object without showing the GUI
/L or -L: Object is a text file containing the list of files to unlock
/LU or -LU: Similar to /L with a unicode list of files to unlock
/O or -O: Outputs Unlocker-Log.txt log file in Unlocker directory
/D or -D: Delete file
/R Object2 or -R Object2: Rename file, if /L or /LU is set object2 points to a text file containing the new name of files
/M Object2 or -M Object2: Move file, if /L or /LU is set object2 points a text file containing the new location of files
Assuming your goal was to replace C:\My App\Data\MyApp.mdf with a file from your installer, you would want something like unlocker C:\My App\Data\MyApp.mdf -S -D. This would delete the file so you could copy in a new one.