Basic info on how to export BLOB as files - sql

I have researched on how to export BLOBs to image. A DB has an IMAGE column storing several thousand images. I thought of exporting the table but I get a BLOB file error in EMS SQL Manager for InterBase and Firebird.
There have been good posts, but I have still not been able to succeed.
SQL scripts to insert File to BLOB field and export BLOB to File
This example has appeared on numerous pages, including Microsoft's site. I am using INTERBASE (Firebird). I have not found anything related to enabling xp_shell for Firebird, or EMS SQL Manager for InterBase and Firebird (which I have also installed). My guess is: its not possible. I also tried Installing SQL Server Express, SQL Server 2008, and SQL Server 2012. I am at a dead end without having even connected to the server. The reason being I have not managed to start the server. Followed the guide at technet.microsoft: How to: Start SQL Server Agent but there are no services on the right pane to me.
PHP file to download entire column (may not post link due to rep limitation).
It has a MySQL connect section that daunts me. There on my computer is the DB as a GDB file, I also have XAMPP. I can figure out a way to use this as a localhost environment. I hope this is making sense.
Last solution is to use bcp, an idea posted on Stack Overflow titled: fastest way to export blobs from table into individual files. I read the documentation, installed it, but cannot connect to server. I use -S PC-PC -U xxx -P xxx (The server must be wrong) But the information I find all uses -T (Windows Authentication)
Summing up. I am using Firebird, as EMS SQL Manager. I try to extract all images from images table into individual files. These tools both have SQL script screens, but it appears to be in conjunction with xp shell. What would you suggest? Am I using the wrong SQL manager to accomplish this?

There are several ways:
Use isql command BLOBDUMP to write a blob to file,
Use a client library (eg Jaybird for Java, Firebird .net provider for C#) to retrieve the data,
With PHP you can use ibase_blob_get in a loop to get bytes from the blob, and write those to a file.
I don't use nor know EMS SQL Manager, so I don't know if (and how) you can export a blob with that.
The example you link to, and almost all tools you mention are for Microsoft SQL Server, not for Firebird; so it is no wonder those don't work.
Example in Java
A basic example to save blobs to disk using Java 8 (might also work on Java 7) would be:
/**
* Example to save images to disk from a Firebird database.
* <p>
* Code assumes a table with the following structure:
* <pre>
* CREATE TABLE imagestorage (
* filename VARCHAR(255),
* filedata BLOB SUB_TYPE BINARY
* );
* </pre>
* </p>
*/
public class StoreImages {
// Replace testdatabase with alias or path of database
private static final String URL = "jdbc:firebirdsql://localhost/testdatabase?charSet=utf-8";
private static final String USER = "sysdba";
private static final String PASSWORD = "masterkey";
private static final String DEFAULT_FOLDER = "D:\\Temp\\target";
private final Path targetFolder;
public StoreImages(String targetFolder) {
this.targetFolder = Paths.get(targetFolder);
}
public static void main(String[] args) throws IOException, SQLException {
final String targetFolder = args.length == 0 ? DEFAULT_FOLDER : args[0];
final StoreImages storeImages = new StoreImages(targetFolder);
storeImages.store();
}
private void store() throws IOException, SQLException {
if (!Files.isDirectory(targetFolder)) {
throw new FileNotFoundException(String.format("The folder %s does not exist", targetFolder));
}
try (
Connection connection = DriverManager.getConnection(URL, USER, PASSWORD);
Statement stmt = connection.createStatement();
ResultSet rs = stmt.executeQuery("SELECT filename, filedata FROM imagestorage")
) {
while (rs.next()) {
final Path targetFile = targetFolder.resolve(rs.getString("FILENAME"));
if (Files.exists(targetFile)) {
System.out.printf("File %s already exists%n", targetFile);
continue;
}
try (InputStream data = rs.getBinaryStream("FILEDATA")) {
Files.copy(data, targetFile);
}
}
}
}
}
Example in C#
Below is an example in C#, it is similar to the code above.
class StoreImages
{
private const string DEFAULT_FOLDER = #"D:\Temp\target";
private const string DATABASE = #"D:\Data\db\fb3\fb3testdatabase.fdb";
private const string USER = "sysdba";
private const string PASSWORD = "masterkey";
private readonly string targetFolder;
private readonly string connectionString;
public StoreImages(string targetFolder)
{
this.targetFolder = targetFolder;
connectionString = new FbConnectionStringBuilder
{
Database = DATABASE,
UserID = USER,
Password = PASSWORD
}.ToString();
}
static void Main(string[] args)
{
string targetFolder = args.Length == 0 ? DEFAULT_FOLDER : args[0];
var storeImages = new StoreImages(targetFolder);
storeImages.store();
}
private void store()
{
if (!Directory.Exists(targetFolder))
{
throw new FileNotFoundException(string.Format("The folder {0} does not exist", targetFolder), targetFolder);
}
using (var connection = new FbConnection(connectionString))
{
connection.Open();
using (var command = new FbCommand("SELECT filename, filedata FROM imagestorage", connection))
using (var reader = command.ExecuteReader())
{
while (reader.Read())
{
string targetFile = Path.Combine(targetFolder, reader["FILENAME"].ToString());
if (File.Exists(targetFile))
{
Console.WriteLine("File {0} already exists", targetFile);
continue;
}
using (var fs = new FileStream(targetFile, FileMode.Create))
{
byte[] filedata = (byte[]) reader["FILEDATA"];
fs.Write(filedata, 0, filedata.Length);
}
}
}
}
}
}

Related

how does one programmatically create a localdb .mdf?

how does one programmatically create a localdb .mdf?
acceptable solutions exclude visual studio, ssms, aspnet_regsql.
a naive stab at a solution might look like this:
static void Main(string[] args)
{
using (var con = new SqlConnection(#"Integrated Security=SSPI;Data Source=(LocalDb)\v11.0;AttachDbFilename=test.mdf"))
{
con.Open();
using (var cmd = new SqlCommand("CREATE DATABASE test", con))
{
cmd.CommandType = CommandType.Text;
cmd.ExecuteNonQuery();
}
}
}
but of course, this fails in SqlConnection.Open with the error
An attempt to attach an auto-named database for file test.mdf failed. A database with the same name exists, or specified file cannot be opened, or it is located on UNC share.
You cannot connect to a database if the specified .mdf doesn't exist.
So... how do you create one?
Had to piece together several answers from Stackoverflow and the great Getting Started with SQL Server 2012 Express LocalDB article from #AaronBertrand
Code assumes Dapper.NET is installed:
PM> Install-Package Dapper
Programmatic creation:
var dbServerName = "SERVER_NAME";
var dbName = "DATABASE_NAME";
var infoResult = SqlLocalDbCommand($"info {dbServerName}");
var needsCreated = infoResult?.Trim().EndsWith($"\"{dbServerName}\" doesn't exist!");
if (needsCreated.GetValueOrDefault(false))
{
var createResult = SqlLocalDbCommand($"create {dbServerName} -s");
var success = createResult?.Trim().EndsWith($"\"{dbServerName}\" started.");
if (false == success)
{
var msg = $"Failed to create database:{Environment.NewLine}{createResult}"
throw new ApplicationException(msg);
}
var master = $#"Server=(localdb)\{dbServerName};Integrated Security=True;"
using (var conn = new SqlConnection(master))
{
var result = conn.Execute($"CREATE DATABASE {dbName}");
}
var #new = $#"Server=(localdb)\{dbServerName};Integrated Security=True;Database={dbName}"
using (var conn = new SqlConnection(#new))
{
//verify i can access my new database
var tables = conn.Query($"SELECT * FROM {dbName}.INFORMATION_SCHEMA.Tables");
}
}
Helper (thanks T30):
/// <summary>
/// Executes a command against SqlLocalDB
/// </summary>
/// <remarks></remarks>
/// <param name="arguments">The arguments to pass to SqlLocalDB.exe</param>
/// <returns></returns>
/// <exception cref="System.ApplicationException">Error returned from process</exception>
private static string SqlLocalDbCommand(string arguments)
{
var process = new Process
{
StartInfo =
{
FileName = "SqlLocalDB",
Arguments = arguments,
UseShellExecute = false,
RedirectStandardOutput = true,
RedirectStandardError = true
}
};
process.Start();
//* Read the output (or the error)
var output = process.StandardOutput.ReadToEnd();
Console.WriteLine(output);
var err = process.StandardError.ReadToEnd();
Console.WriteLine(err);
process.WaitForExit();
if (err.Exists()) throw new ApplicationException(err); //Is LocalDB installed?
return output;
}
Note that with this solution you won't see the mdf files, i'm sure they exist in some user folder but the key take away is that you'll connect by the connection string
(localdb)\SERVER_NAME;Integrated Security=True;Database=DATABASE_NAME
So I take it what you actually want to do is create a database called test in your LocalDB instance, but you don't have an MDF file already created for this database?
If that's the case, the code you have will fail at the connection phase since you've asked it to attach your test.mdf file.
What you would normally do in this situation is make a connection to the master database initially, and then run the create database statement which will create the test database with it's associated MDF file, maybe try changing your connection string so it looks more like this and then running again:
Integrated Security=SSPI;Data Source=(localdb)\V11.0;Initial Catalog=master

Is it possible to get Remote Servers for testing Upload Files functionality?

I have a simle program shown below which is resonsible to upload a file to a Remote Location
public static void main(String[] args) {
String server = "www.myserver.com";
int port = 21;
String user = "user";
String pass = "pass";
FTPClient ftpClient = new FTPClient();
try {
ftpClient.connect(server, port);
ftpClient.login(user, pass);
ftpClient.enterLocalPassiveMode();
ftpClient.setFileType(FTP.BINARY_FILE_TYPE);
File firstLocalFile = new File("D:/Test/Projects.zip");
String firstRemoteFile = "Projects.zip";
InputStream inputStream = new FileInputStream(firstLocalFile);
System.out.println("Start uploading first file");
boolean done = ftpClient.storeFile(firstRemoteFile, inputStream);
inputStream.close();
}
My question is , is it possible to test this program anyway , as i dont have a Remote Server currently .
Means is it possible to get remote server to upload files for temporary purpose ( Sorry but only open source please )
Is anybody aware of such websites ??
Screen shot

TFS 2012 Backup and Restore BuildDefinitions only

I installed a TFS2012 as a test system and doing some tests before we go productive.
This includes to define many BuildDefinitions which was a lot of work.
After the tests are successful, an new server will be installed with TFS2012 on it.
For this new server - which operates then as the productive system - i would like to restore the BuildDefinitions from the test system. But only the BuildDefinitions, not the whole TeamCollections. Because i ran test checkins and i don`t want these on my productive server.
Now, is it possible to backup and restore BuildDefinitions only?
Maybe it is possible directly throught the Sql database?, but i`am a little affraid of references there, pointing on some other tables.
Best Regards, Peter Bucher
Build definitions are not source controlled. The only option is relying on the TFS database backup where can restore or view the tbl_BuildDefinition* tables in the Tfs_DefaultCollection database.
There is a user voice for this feature and also you can use TFS API to do it.
Add a vote on uservoice:
provide a way to version-control build definitions
Using TFS API
How can I copy a TFS 2010 Build Definition?
Finally i decided not to touch the database, because there are references to a lot of other tables.
I used the TFS API v11 (TFS2012) and a bit C# Code, which i fitted to my needs from this base: How can I copy a TFS 2010 Build Definition?
It copies all Build Definitions from one TFS2012 Server to another. For both servers there is the need to specifiy a TeamCollection and a TeamProject.
So, the copy-task has to be done per TeamProject.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Microsoft.TeamFoundation.Build.Client;
using Microsoft.TeamFoundation.Client;
using Microsoft.TeamFoundation.VersionControl.Client;
namespace TFSBuildDefinitionCreator
{
internal class Program
{
private static void Main(string[] args)
{
// Copies build definitions from one server to another.
// Uses the TeamFoundation API V11 (TFS2012).
// Code was used to copy b uild definitions from a test server to a productive.
string sourceServer = "http://testTfs:8080/tfs/MyTeamCollection";
string sourceTeamProject = "MyTeamProject";
string targetServer = "https://productiveTfs:8080/tfs/MyTeamCollection";
string targetTeamProject = "MyTeamProject";
// DropLocation for defininitions: Share on which the build should be dropped.
string defaultDropLocation = "\\\\MyBuildserver\\Builds$";
// Change the DefaultProcessTemplate in the following method below: GetDefaultProcessTemplateByServerPathFromBuildServer.
CopyBuildDefinitions(sourceServer, sourceTeamProject, targetServer, targetTeamProject, defaultDropLocation);
Console.Read();
}
private static IBuildServer GetBuildServerFromServerUrl(string serverUrl)
{
var tfs = TeamFoundationServerFactory.GetServer(serverUrl);
return (IBuildServer)tfs.GetService(typeof(IBuildServer));
}
private static IBuildController GetDefaultBuildControllerFromBuildServer(IBuildServer buildServer)
{
return buildServer.QueryBuildControllers()[0];
}
private static IProcessTemplate GetDefaultProcessTemplateByServerPathFromBuildServer(IBuildServer buildServer, string teamProject)
{
var processTemplates = buildServer.QueryProcessTemplates(teamProject);
var result = processTemplates.First(t => t.ServerPath.Contains("/BuildProcessTemplates/MyDefaultTemplate.xaml"));
return result;
}
private static void CopyBuildDefinitions(string sourceServer, string sourceTeamProject, string targetServer,
string targetTeamProject, string defaultDropLocation)
{
var sourceBuildServer = GetBuildServerFromServerUrl(sourceServer);
var sourceBuildDetails = sourceBuildServer.QueryBuildDefinitions(sourceTeamProject);
foreach (var sourceBuildDetail in sourceBuildDetails)
{
CopyBuildDefinition(sourceBuildDetail, targetServer, targetTeamProject, defaultDropLocation);
}
}
private static void CopyBuildDefinition(IBuildDefinition buildDefinition, string targetServer, string targetTeamProject, string defaultDropLocation)
{
var targetBuildServer = GetBuildServerFromServerUrl(targetServer);
var buildDefinitionClone = targetBuildServer.CreateBuildDefinition(targetTeamProject);
buildDefinitionClone.BuildController = GetDefaultBuildControllerFromBuildServer(targetBuildServer);
buildDefinitionClone.ContinuousIntegrationType = buildDefinition.ContinuousIntegrationType;
buildDefinitionClone.ContinuousIntegrationQuietPeriod = buildDefinition.ContinuousIntegrationQuietPeriod;
// Noch ändern.
//buildDefinitionClone.DefaultDropLocation = buildDefinition.DefaultDropLocation;
buildDefinitionClone.DefaultDropLocation = defaultDropLocation;
buildDefinitionClone.Description = buildDefinition.Description;
buildDefinitionClone.Enabled = buildDefinition.Enabled;
//buildDefinitionClone.Name = String.Format("Copy of {0}", buildDefinition.Name);
buildDefinitionClone.Name = buildDefinition.Name;
//buildDefinitionClone.Process = buildDefinition.Process;
buildDefinitionClone.Process = GetDefaultProcessTemplateByServerPathFromBuildServer(targetBuildServer, targetTeamProject);
buildDefinitionClone.ProcessParameters = buildDefinition.ProcessParameters;
foreach (var schedule in buildDefinition.Schedules)
{
var newSchedule = buildDefinitionClone.AddSchedule();
newSchedule.DaysToBuild = schedule.DaysToBuild;
newSchedule.StartTime = schedule.StartTime;
newSchedule.TimeZone = schedule.TimeZone;
}
foreach (var mapping in buildDefinition.Workspace.Mappings)
{
buildDefinitionClone.Workspace.AddMapping(
mapping.ServerItem, mapping.LocalItem, mapping.MappingType, mapping.Depth);
}
buildDefinitionClone.RetentionPolicyList.Clear();
foreach (var policy in buildDefinition.RetentionPolicyList)
{
buildDefinitionClone.AddRetentionPolicy(
policy.BuildReason, policy.BuildStatus, policy.NumberToKeep, policy.DeleteOptions);
}
buildDefinitionClone.Save();
}
}
}
Hope that helps others.

storing pdf files using filestream on file system

I am first time trying to use filestream to store pdf files on file system using varbinary(MAX) column type of DB.
I have followed following steps.
enabled filestream feature on SQL server 2008 R2.
Create a filegroup for BLOB storage
created table with blob column of type varbinary(max)
Now, I want to use file upload control to select file and when click on upload button it should save the pdf file. Also, how to retrieve the file?
I have tried following code
protected void btnFSupload_Click(object sender, EventArgs e)
{
SqlConnection cn = null;
SqlTransaction tx = null;
SqlCommand cmd = null;
SqlCommand cmd2 = null;
bool bCommit = false;
try
{
// read in the file to be saved as a blob in the database
FileStream input = new FileStream(#"D:\swami.pdf", FileMode.Open, FileAccess.Read);
byte[] buffer = new byte[(int)input.Length];
input.Read(buffer, 0, buffer.Length);
cn = new SqlConnection("server=at-hetal01\\sqlexpress;Initial Catalog=practice;Integrated Security=true;");
cn.Open();
tx = cn.BeginTransaction();
cmd = new SqlCommand("dbo.stp_AddBLOB", cn, tx);
cmd.CommandType = System.Data.CommandType.StoredProcedure;
SqlDataReader r = cmd.ExecuteReader(System.Data.CommandBehavior.SingleRow);
r.Read();
string id = r[0].ToString();
string path = r[1].ToString();
r.Close();
// get the transaction context
cmd2 = new SqlCommand("SELECT GET_FILESTREAM_TRANSACTION_CONTEXT()", cn, tx);
Object obj = cmd2.ExecuteScalar();
byte[] txCtx = (byte[])obj;
// open the filestream to the blob
SafeFileHandle handle = OpenSqlFilestream(path,DESIRED_ACCESS_WRITE,SQL_FILESTREAM_OPEN_NO_FLAGS,txCtx,(UInt32)txCtx.Length,0);
// open a Filestream to write the blob
FileStream output = new FileStream(handle,FileAccess.Write,buffer.Length,false);
output.Write(buffer,0,buffer.Length);
output.Close();
if (handle != null && !handle.IsClosed)
handle.Close();
bCommit = true;
}
catch (Exception ex)
{
Response.Write(ex.Message);
}
finally
{
if (cn != null)
{
switch (bCommit)
{
case true:
tx.Commit();
break;
case false:
tx.Rollback();
break;
}
cn.Close();
}
}
}
Above code shows error as below
The operating system returned the error '0xc000003a({Path Not Found} The path %hs does not exist.)' while attempting 'NtCreateFile' on 'D:\DB\FS\d11132f8-c2a8-452d-ae0c-208164a550d7\beb8e1f1-8116-440b-870b-7cef4281a15d\0000001c-000000e4-010d'. The statement has been terminated.
So, any clue on this?
If you have altered your table using SSMS table designer, the FILESTEAM column attribute will be lost producing the path not found. Make sure the FILESTREAM attribute is set for the file field by running the follwoing statement in your database:
select SERVERPROPERTY('FilestreamShareName') as ShareName,
SERVERPROPERTY('FilestreamConfiguredLevel') as ConfiguredLevel,
SERVERPROPERTY('FilestreamEffectiveLevel') as EffectiveLevel
You'll need to alter the table via a script and NOT SSMS to tie your varchar(max)/filestream field to the FileGroup you should have already created.
When I ran into this issue, I found the answer on StackOverflow, but can't seem to find it again for the reference.
I know this is old, but for future reference:
We checked the SERVERPROPERTY values that #BMP suggested. They were configured correctly, so that didn't help.
However, we went ahead and turned OFF the windows file share part of the file streaming access. Once this was done, the error went away.
In our case it was a web app running on the exact same machine as the sql server which exhibited the problem. I'm not sure if the web app's app pool user didn't have access to the file share created by windows or not.
The details were:
Windows 2003 Server (x86)
IIS 6
SQL Server 2008 R2 Express
UPDATE: Apparently this worked for a few days. It's not working any more.

Import database from File PROGRAMMATICALLY?

Is there a way to programatically import an entire database from a file with SQL?? (either .CSV , .SQL and .DB files are fine)
Thanks!!
EDITED AFTER TO CLARIFY:
I am interested in a solution that is Database independent (has to works with all types of databases (Mysql, SQL Server, PostGres, Oracle...)
MySQL: LOAD DATA INFILE for csv's; for .sql files generated with MySQL, use the shell.
For SQLite: see this SO question.
SQL Server: apparently there's the BULK INSERT command.
You are not going to find a database-independent syntax for an SQL command because there isn't one.
There may be a wrapper library around databases but I'm not aware of it. (Or you could try to use ODBC, but that's connection oriented and wouldn't allow direct access to a file)
Perhaps there is an interactive GUI-related software tool out there to do this.
Note also that loading data directly from a file on a database server into a database almost certainly requires security privileges to do so (otherwise it is a security risk).
Ok so I actually found a solution that is Database INDEPENDENT to import a database from a .Sql file quite easily!! :)
So whichever database you have (Mysql, Sqlite, ...) do the following:
1) export your database into .sql format.
(This .sql file will contain all sql commands such as CREATE TABLE... INSERT INTO table...)
(You may need to remove the lines that start with CREATE TABLE and leave only the lines that start with INSERT...)
2) Then in the language that you are using write some code that read each line of the Sql Lite files and stores it into an Array of String (String[])
3) Then execute each String contained in the array String[] as sql command
I've implemented this in Java:
import java.io.BufferedReader;
import java.io.DataInputStream;
import java.io.FileInputStream;
import java.io.InputStreamReader;
import java.util.LinkedList;
import java.util.List;
import android.content.Context;
import android.database.sqlite.SQLiteDatabase;
public class DatabaseImporter {
private static DatabaseImporter instance;
public static DatabaseImporter getInstance(){
if(DatabaseImporter.instance == null)
instance = new DatabaseImporter();
return DatabaseImporter.instance;
}
private DatabaseImporter (){
}
public void importDatabaseFromFile(Context context, String databaseName , String filePath){
SQLiteDatabase database = //CREATE UR DATABASE WITH THE COMMAND FROM THE DATABASE API YOUR USING
this.executeSqlCommands( database ,
this.readSqlCommandsFromFile(filePath)
);
}
private String[] readSqlCommandsFromFile(String filePath){
String[] sqlCommands = new String[0];
List<String> sqlCommandsList = new LinkedList<String>();
try{
// Open the file that is the first
// command line parameter
FileInputStream fstream = new FileInputStream(filePath);
BufferedReader br = new BufferedReader(new InputStreamReader(fstream));
String strLine;
//Read File Line By Line
while ((strLine = br.readLine()) != null) {
if(!strLine.equals("") && !strLine.equals(" ") && strLine != null)
sqlCommandsList.add(strLine);
}
//Close the input stream
in.close();
}catch (Exception e){//Catch exception if any
System.err.println("Error: " + e.getMessage());
}
sqlCommands = new String[sqlCommandsList.size()];
sqlCommandsList.toArray(sqlCommands);
return sqlCommands;
}
private void executeSqlCommands(SQLiteDatabase database, String[] sqlCommands){
for(int i = 0; i < sqlCommands.length ; i++){
database.execSQL(sqlCommands[i]);
}
}
}
mysql -u -p < dumpfile.sql
importing csv would require a script (using eg. PHP) to put the right fields in the right bit of the query.
If you are using SQL Server check out SSIS