In my current project there is a need to restore a database using wix installer, previously I had managed to work-around this is issue by creating a .sql script of the DB which was 5-10MB.
Now I have a .bak file around 1GB and with .sql is 2.8GB, I tried adding the .sql to the project but it won't even compile.
Any ideas on how I can solve this issue?
The RDBMS is Microsoft SQL Server 2014 Express.
The best approach for this is to use SSDT for the schema and then use bcp to export the data into CSV files that are then installed to a location on disk. Once they are there you can then use the sql:script or sql:string wix elements to call the sql bulk insert command to populate the data:
bulk insert [Database].[dbo].[TableA] from C:\InstallLocation\TableA.Data.asc
You will need to either make sure that the data is imported in the correct order so that you don't get FK errors or temporarily disable the foreign keys in the db:
EXEC sp_MSforeachtable "ALTER TABLE ? NOCHECK CONSTRAINT all"
just don't forget to re-enable!
This is actually a very efficient way of distributing a DB to be installed. For example we have a db that is ~1Gb in size and when using the standard mszip cab compression mode in wix the installer ends up ~60Mb. If the compression is switched to high it ends up ~40Mb (but takes quite a while to build!). Obviously YMMV as it will depend on the data being compressed but whatever it will be significantly smaller than a sql backup or sql scripts.
Related
I need to convert all varchar columns in about 40 tables (filled with the data) to nvarchar columns. It is planned to happen in a dedicated MS SQL server used only for the purpose. The result should be moved to Azure SQL.
Where should be the conversion done: on the old SQL, or after moving it on Azure SQL Server?
According to Remus Rusanu's answer https://stackoverflow.com/a/8157951/1346705, new nvarchar columns are created in the process, and the old varchar columns are dropped. The space can be reclaimed by DBCC CLEANTABLE or using ALTER TABLE ... REBUILD. Are the dropped varchar columns packed into the backup table, or does the backup/restore also remove the dropped columns?
Can the process be somehow automated using a universal SQL script? Or is it necessary to write the script for each individual table?
Context: We are the 3rd party with respect to the enterprise information system. Our product reads from the information system SQL database and presents the data the way that would otherwise be expensive to implement in the IS. The enterprise information system is now migrated to the new version and is to be run on Azure SQL. The database of the IS have been changed heavily, and one of the changes was to abandon the old 8-bit text encoding (varchar) and to use Unicode instead (nvarchar). Our system was used also for collecting data typed manually -- using the same encoding that the old IS used.
Migration is to be done via doing old version of backup (SqlCmd that produces xxx.bak files), restoring on another good old SQL server. Then we run the script that removes all the tables, views, and stored procedures that can be reconstructed from the IS. One of the main reasons is that the SQL code uses features that are not accepted by the new backup tool SqlPackage.exe to produce xxx.bacpac file. Then the bacpac file is restored in Azure SQL.
Where should be the conversion done: on the old SQL, or after moving it on Azure SQL Server?
I would do it on local SQLServer First,Running this on Azure database,might cause you to run into some issues like hitting your DTU limits,disk IO throttling..
Are the dropped varchar columns packed into the backup table, or does the backup/restore also remove the dropped columns?
The space wont be released back to filesystem,also backup doesn't process free spaces,so you will not see much change there.You might want to read more on dbcc cleantable though,before proceeding ..
Can the process be somehow automated using a universal SQL script? Or is it necessary to write the script for each individual table?
It can be automated,may be you can use dynamic sql to see the column type and process further.You will also have to see if any of those columns are part of indexes,if so you have to drop them first
I suggest making the schema changes beforehand on the old instances. Even if you don't bother cleaning up space with DBCC CLEAANTABLE or ALTER...REBUILD, the resultant bacpac size will be the same because, unlike a physical backup/restore, a bacpac file is just a compressed package format of schema and data.
Consider using SQL Server Data Tools (SSDT) to facilitate the schema changes. This will consider all the dependencies (constraints, indexes, etc.) that is a challenge with a "universal" T-SQL solution. SSDT will generally generate a migration script that employs temp tables for such schema changes so the end result won't have wasted space in your old database. However, you will need sufficient unused space in the database to contain the old/new objects side-by-side.
I have developed a SQL Express database. I need to backup all but one table in that database in an automated way. I was thinking i could write a SQL script to do this, trigger it using sqlcmd from a batch file but not sure how to write that SQL script.
I was also thinking, if nothing else possible, i could create a second db that has the tables i want to backup then i write a script that copys data 'into' the second db and then do a auto backup of that entire db. This has the disadvantage of having a procrastinated unpacking of that backup when wanting to use it - its not a small install script.
Is this a possibility, is it the only option or is there tools for SQL Express to do this?
There is no option to exclude just one table while backing up .Few things i could think of
1.Right click database ->Tasks ->generate scripts ->exclude the table you want and choose to save the script and run this every time
2.you could also choose Export option,but since you are using SQL Express,you wont have the option to save this package
Keep the large table in a different database and just backup the original database. You can still use the large table even in a different database, i.e.
SELECT *
FROM MyDb.dbo.SomeTable s
JOIN OtherDb.dbo.LargeTable l
ON (expression);
I'd like to create a database backup using SSMS.
The backup file will be a .bak file, but I would like to ignore 1 column in a certain table, because this column isn't necessary, but it takes up 95% of the backup size.
The column values should all be replaced by 0x00 (column type is varbinary(max), not null).
What's the best way to do this?
FYI: I know how to generate a regular backup using Tasks => Back Up..
There is a long way of doing what you ask. Its basically create a new restored database, remove the non required data and then do a new backup again.
Create a Backup of the production database.
Restore the backup locally on production with a new name
Update the column with 0x00
Shrink the database (Shrink is helpful when doing a restore. This wont reduce the bak file size)
Take the backup of the new database (Also use Backup Compression to reduce the size even more)
Ftp the bak file
If you only needed a few tables, you could have used bcp but that looks out of the picture for your current requirement.
From SQL Server native backups, you can't. You'd have to restore the database to some other location and then migrate usefull data.
You can create a copy of your table without the column and backup using filegroups https://msdn.microsoft.com/en-us/library/ms191539(SQL.90).aspx
I am a C# developer, I am not really good with SQL. I have a simple questions here. I need to move more than 50 millions records from a database to other database. I tried to use the import function in ms SQL, however it got stuck because the log was full (I got an error message The transaction log for database 'mydatabase' is full due to 'LOG_BACKUP'). The database recovery model was set to simple. My friend said that importing millions records using task->import data will cause the log to be massive and told me to use loop instead to transfer the data, does anyone know how and why? thanks in advance
If you are moving the entire database, use backup and restore, it will be the quickest and easiest.
http://technet.microsoft.com/en-us/library/ms187048.aspx
If you are just moving a single table read about and use the BCP command line tools for this many records:
The bcp utility bulk copies data between an instance of Microsoft SQL Server and a data file in a user-specified format. The bcp utility can be used to import large numbers of new rows into SQL Server tables or to export data out of tables into data files. Except when used with the queryout option, the utility requires no knowledge of Transact-SQL. To import data into a table, you must either use a format file created for that table or understand the structure of the table and the types of data that are valid for its columns.
http://technet.microsoft.com/en-us/library/ms162802.aspx
The fastest and probably most reliable way is to bulk copy the data out via SQL Server's bcp.exe utility. If the schema on the destination database is exactly identical to that on the source database, including nullability of columns, export it in "native format":
http://technet.microsoft.com/en-us/library/ms191232.aspx
http://technet.microsoft.com/en-us/library/ms189941.aspx
If the schema differs between source and target, you will encounter...interesting (yes, interesting is a good word for it) problems.
If the schemas differ or you need to perform any transforms on the data, consider using text format. Or another format (BCP lets you create and use a format file to specify the format of the data for export/import).
You might consider exporting data in chunks: if you encounter problems it gives you an easier time of restarting without losing all the work done so far.
You might also consider zipping the exported data files up to minimize time on the wire.
Then FTP the files over to the destination server.
bcp them in. You can use the bcp utility on the destination server for the BULK IMPORT statement in SQL Server to do the work. Makes no real difference.
The nice thing about using BCP to load the data is that the load is what is described as a 'non-logged' transaction, though it's really more like a 'minimally logged' transaction.
If the tables on the destination server have IDENTITY columns, you'll need to use SET IDENTITY statement to disable the identity column on the the table(s) involved for the nonce (don't forget to reenable it). After your data is imported, you'll need to run DBCC CHECKIDENT to get things back in synch.
And depending on what your doing, it can sometimes be helpful to put the database in single-user mode or dbo-only mode for the duration of the surgery: http://msdn.microsoft.com/en-us/library/bb522682.aspx
Another approach I've used to great effect is to use Perl's DBI/DBD modules (which provide access to the bulk copy interface) and write a perl script to suck out the data from the source server, transform it and bulk load it directly into the destination server, without having to save it to disk and move it. Also means you can trap errors and design things for recovery and restart right at the point of failure.
Use BCP to migrate data.
Another approach i have used in the past is to take a backup of the transaction log and shrink the log Prior to the migration. Split the migration script in parts and run the log backup- shrink - migrate iteration a few times.
I have two backup files
1) is named 'backup.sql' with a bunch of SQL defining TABLES
2) is named 'backup' with a bunch of encoded data, which I believe are the ROWS
I need to restore these TABLES + ROWS, but all I am able to figure out is how to restore the tables.
Any tips on dealing with these files? It's the first time I ever deal with SQL Server.
The backup process would not create a file with actual SQL statements, it would create a binary file. So #1 is not a backup file (it's probably a script someone saved to re-create the schema).
I would try to use SQL Server Management Studio to restore the second file and see what happens. I don't think it will allow you to restore an invalid file, but I would take some basic precautions like backing up the system first.
What is the extension for the 'backup' file? Is the filename backup.bak? If you have a backup file created by sql server then it 'should' contain the logic to create both the tables and restore the data, but it could depend on how the backup was created.
---Edit
It is possible for a .SQL file to contain data values as well as the logic to create the tables/columns for a database. I used to run backups of a MySql database in this way a long time ago...it just is not seen very often with SQL server since it has built in backup/restore funcationality.
Seems unlikely they would export all the rows from all tables into CSV file, and given you said it looks encrypted, it's making me think that's your actual backup file.
try this, save a copy of the "backup" file, rename it to backup.bak and run this from SQL Server Management Studio
restore filelistonly from disk='C:\backup.bak'
(assuming your file is saved on the root of the C: drive)
Any results/errors?