Is there a better way/quicker way to export an azure database via SSMS with it being compressed via the Export Data-tier Application? In the Advanced settings, there are only options to select the tables to be exported.
When you take a .bak file, you are able to compress it in Backup Options. Is there an option when you do it via .bacpac?
In the database we are trying to obtain a bacpac for, we have a column of type varbinary(max) in one table and that is where it takes a long time to export/data size. The total size is about 3.5 GB when it is all said and done.
Try using SqlPackage.exe /Action:Export instead; there are more options, for example CompressionOption can be set to Normal, Maximum, Fast, SuperFast, NotCompressed.
The default is Normal, which I believe is what SSMS uses too. Usage and other parameters can be found here: https://learn.microsoft.com/en-us/sql/tools/sqlpackage/sqlpackage-export
An example command to export from a local SQL server with Maximum compression:
sqlpackage.exe /Action:Export /SourceServerName:"." /SourceDatabaseName:"Contoso.Database" /TargetFile:"C:\sqlpackageoutput\output_current_version.bacpac" /p:CompressionOption=Maximum
If you are trying to make the export faster, try SuperFast. If you are aiming for smallest bacpac file then try Maximum.
Related
I am confused with the size of the file I backup with SSMS and Query.
If I create a file from SSMS in its default folder something like "C:\Program Files\Microsoft SQL Server\MSSQL14.NAMEDINSTANCE\MSSQL\Backup" the outfile say Db1.bak is about 198292 KB
Same database if I backup with the query "backup database Db1 to disk='D:\Db1.bak' the file size is just 6256 KB
Sometimes the other database say Db2 gives the same filesize i.e 6256 KB(Both Db1 and Db2 have identical(same) schemas just data in it are different.)
And backup with SSMS gives 33608 KB which seems satisfactory.
I also tried verifying all database in SSMS like this RESTORE VERIFYONLY FROM DISK = 'D:\BACKUP\Db1.bak'
GO and result gives valid in every database check.
I also tried deleting Db1 from SSMS and restoring the less KB file and checked some data of few tables (Not All) and it seems showing all data in tables properly but the filesize dissatisfies me.
Thank You.
I suspect that,like initially mentioned, you have compression on my
default, and using the GUI, with the settings is not making use of
that (and that if you selected to Compress in the GUI, you'd get a
similar size)
If the server option backup compression default is on, even if you don't mention it in your backup command, compression will be applied. So in both cases there would be compressed backup. But it's easy to see, just run this command for both backups:
restore headeronly
from disk = 'here_the_full_path_with_filename';
In the 5th column you'll get the flag if your backup is compressed.
But the cause of this difference is another one, and you'll see it when run restore headeronly: you made multiple backups to the same file.
You used backup command with noinit from SSMS, and the same file name, so now this file contains more than one backup, and restore headeronly will show them all.
In my current project there is a need to restore a database using wix installer, previously I had managed to work-around this is issue by creating a .sql script of the DB which was 5-10MB.
Now I have a .bak file around 1GB and with .sql is 2.8GB, I tried adding the .sql to the project but it won't even compile.
Any ideas on how I can solve this issue?
The RDBMS is Microsoft SQL Server 2014 Express.
The best approach for this is to use SSDT for the schema and then use bcp to export the data into CSV files that are then installed to a location on disk. Once they are there you can then use the sql:script or sql:string wix elements to call the sql bulk insert command to populate the data:
bulk insert [Database].[dbo].[TableA] from C:\InstallLocation\TableA.Data.asc
You will need to either make sure that the data is imported in the correct order so that you don't get FK errors or temporarily disable the foreign keys in the db:
EXEC sp_MSforeachtable "ALTER TABLE ? NOCHECK CONSTRAINT all"
just don't forget to re-enable!
This is actually a very efficient way of distributing a DB to be installed. For example we have a db that is ~1Gb in size and when using the standard mszip cab compression mode in wix the installer ends up ~60Mb. If the compression is switched to high it ends up ~40Mb (but takes quite a while to build!). Obviously YMMV as it will depend on the data being compressed but whatever it will be significantly smaller than a sql backup or sql scripts.
In sql server, From my desktop I connected to the server. And I want to move a data from a database to another. I have used both select into and import wizard. But import wizard seems to be slow. Why?
Is there any methodology changes for transferring data ?
Select into is a SQL query, and it is executed directly.
Import and Export Wizard is a tool which invokes Integration Services (SSIS).
Wizard is slow, but can use various data sources
More about export/import wizard
https://msdn.microsoft.com/en-US/en-en/library/ms141209.aspx
Topic about select into and export/import wizard
https://social.msdn.microsoft.com/forums/sqlserver/en-US/e0524b2a-0ea4-43e7-b74a-e9c7302e34e0/super-slow-performance-while-using-import-export-wizard
I agree with Andrey. The Wizard is super slow. If you perform a Google search on "sql server import and export wizard slow", you will receive nearly 50k hits. You may want to consider a couple of other options.
BCP Utility
Note: I have used this on a number occasions. Very fast processing.
The bcp utility bulk copies data between an instance of Microsoft SQL Server and a data file in a user-specified format. The bcp utility can be used to import large numbers of new rows into SQL Server tables or to export data out of tables into data files. Except when used with the queryout option, the utility requires no knowledge of Transact-SQL. To import data into a table, you must either use a format file created for that table or understand the structure of the table and the types of data that are valid for its columns.
Example:
BULK INSERT TestServer.dbo.EmployeeAddresses
FROM 'D:\Users\Addresses.txt';
GO
OPENROWSET(BULK) Function
The OPENROWSET(BULK) function connects to an OLE DB data source to restore data and it allows access to a remote data by connecting to a remote data source.
Example:
INSERT INTO AllAddress(Address)
SELECT * FROM OPENROWSET(
BULK 'D:\Users\Addresses.txt',
SINGLE_BLOB) AS x;
Reference
https://msdn.microsoft.com/en-us/library/ms175915.aspx
http://solutioncenter.apexsql.com/sql-server-bulk-copy-and-bulk-import-and-export-techniques/
Mysql Store data into many places and it stores data in Small chunk of files for faster retrieve and when we use export wizard what it does is write all metadata and data to our RAM first and depending on our system and increases overhead and same happen in case of importing, and Select into is fast because mysql has to create inbuilt replica of the database that already exist.
in real life, Select into is like photocopy of a page whereas wizard is like re-writing the page manually.
My problem statement is that I have a csv blob and I need to import that blob into a sql table. Is there an utility to do that?
I was thinking of one approach, that first to copy blob to on-premise sql server using AzCopy utility and then import that file in sql table using bcp utility. Is this the right approach? and I am looking for 1-step solution to copy blob to sql table.
Regarding your question about the availability of a utility which will import data from blob storage to a SQL Server, AFAIK there's none. You would need to write one.
Your approach seems OK to me. Though you may want to write a batch file or something like that to automate the whole process. In this batch file, you would first download the file on your computer and the run the BCP utility to import the CSV in SQL Server. Other alternatives to writing batch file are:
Do this thing completely in PowerShell.
Write some C# code which makes use of storage client library to download the blob and once the blob is downloaded, start the BCP process in your code.
To pull a blob file into an Azure SQL Server, you can use this example syntax (this actually works, I use it):
BULK INSERT MyTable
FROM 'container/folder/folder/file'
WITH ( DATA_SOURCE = 'ds_blob',BATCHSIZE=10000,FIRSTROW=2);
MyTable has to have identical columns (or it can be a view against a table that yields identical columns)
In this example, ds_blob is an external data source which needs to be created beforehand (https://learn.microsoft.com/en-us/sql/t-sql/statements/create-external-data-source-transact-sql)
The external data source needs to use a database contained credential, which uses an SAS key which you need to generate beforehand from blob storage https://learn.microsoft.com/en-us/sql/t-sql/statements/create-database-scoped-credential-transact-sql)
The only downside to this mehod is that you have to know the filename beforehand - there's no way to enumerate them from inside SQL Server.
I get around this by running powershell inside Azure Automation that enumerates blobds and writes them into a queue table beforehand
I am a C# developer, I am not really good with SQL. I have a simple questions here. I need to move more than 50 millions records from a database to other database. I tried to use the import function in ms SQL, however it got stuck because the log was full (I got an error message The transaction log for database 'mydatabase' is full due to 'LOG_BACKUP'). The database recovery model was set to simple. My friend said that importing millions records using task->import data will cause the log to be massive and told me to use loop instead to transfer the data, does anyone know how and why? thanks in advance
If you are moving the entire database, use backup and restore, it will be the quickest and easiest.
http://technet.microsoft.com/en-us/library/ms187048.aspx
If you are just moving a single table read about and use the BCP command line tools for this many records:
The bcp utility bulk copies data between an instance of Microsoft SQL Server and a data file in a user-specified format. The bcp utility can be used to import large numbers of new rows into SQL Server tables or to export data out of tables into data files. Except when used with the queryout option, the utility requires no knowledge of Transact-SQL. To import data into a table, you must either use a format file created for that table or understand the structure of the table and the types of data that are valid for its columns.
http://technet.microsoft.com/en-us/library/ms162802.aspx
The fastest and probably most reliable way is to bulk copy the data out via SQL Server's bcp.exe utility. If the schema on the destination database is exactly identical to that on the source database, including nullability of columns, export it in "native format":
http://technet.microsoft.com/en-us/library/ms191232.aspx
http://technet.microsoft.com/en-us/library/ms189941.aspx
If the schema differs between source and target, you will encounter...interesting (yes, interesting is a good word for it) problems.
If the schemas differ or you need to perform any transforms on the data, consider using text format. Or another format (BCP lets you create and use a format file to specify the format of the data for export/import).
You might consider exporting data in chunks: if you encounter problems it gives you an easier time of restarting without losing all the work done so far.
You might also consider zipping the exported data files up to minimize time on the wire.
Then FTP the files over to the destination server.
bcp them in. You can use the bcp utility on the destination server for the BULK IMPORT statement in SQL Server to do the work. Makes no real difference.
The nice thing about using BCP to load the data is that the load is what is described as a 'non-logged' transaction, though it's really more like a 'minimally logged' transaction.
If the tables on the destination server have IDENTITY columns, you'll need to use SET IDENTITY statement to disable the identity column on the the table(s) involved for the nonce (don't forget to reenable it). After your data is imported, you'll need to run DBCC CHECKIDENT to get things back in synch.
And depending on what your doing, it can sometimes be helpful to put the database in single-user mode or dbo-only mode for the duration of the surgery: http://msdn.microsoft.com/en-us/library/bb522682.aspx
Another approach I've used to great effect is to use Perl's DBI/DBD modules (which provide access to the bulk copy interface) and write a perl script to suck out the data from the source server, transform it and bulk load it directly into the destination server, without having to save it to disk and move it. Also means you can trap errors and design things for recovery and restart right at the point of failure.
Use BCP to migrate data.
Another approach i have used in the past is to take a backup of the transaction log and shrink the log Prior to the migration. Split the migration script in parts and run the log backup- shrink - migrate iteration a few times.