I am not able to get LOAD DATA LOCAL INFILE working as a prepared statement. Error 1064 is thrown, but the syntax displays correctly as a select statement (as text). Local infile is enabled "local_infile=true" in cnf file, and I am able to import files using the same statement shown below replacing variables with actual values. Any help in pointing me in the right direction is gratefully appreciated. Of note, mariadb knowledgebase mentions "LOAD DATA INFILE is unsafe for statement-based replication." Is this the reason the prepared statement and subsequently a stored procedure won't work?
Thanks again.
SET #sql := 'LOAD DATA LOCAL INFILE ? INTO TABLE ?
FIELDS OPTIONALLY ENCLOSED BY \'\"\' TERMINATED BY \'\,\'
LINES TERMINATED BY \'\\n\'
(item, price, pdetail, #store, #purchased)
SET store = ?';
PREPARE stmt_loadmeup FROM #sql;
EXECUTE stmt_loadmeup USING 'full path to file.csv','TEST','TH';
Hi I have a a sql 2000 database with a large number of scanned documents stored as pdfs and word documents stored in an image data type.
I need to export them to files.
I have written code to to do this using xp_cmdshell and bcp. Looking at other questions I have created a fmt file as below:
8.0
1
1 SQLIMAGE 0 0 "" 1 FILEDATA ""
the command is
bcp "select filedata FROM attacheddocuments where pkey = '+ convert (varchar, #imageid) + '" queryout "c:\scans\' + #imagefilename + '" -T -f c:\scans\attached.fmt
however when I run the query it creates all the files but they cannot be opened in either word or acrobat. both report that the file is corrupt.
If instead I run the command
bcp "select filedata FROM attacheddocuments where pkey = '+ convert (varchar, #imageid) + '" queryout "c:\scans\' + #imagefilename + '" -T -N
The pdf files now open ok but the word documents are still corrupt.
Does anyone have any ideas where I am going wrong?
I know this is a really old post, I am having this issue with all files except PDF's.
I have tried without the -N and with. Using a format file and not using a format file. Strange thing is I used to use my script often, had not used it for a while. During that period some SQL updates have came out. Not the scripts only exports PDF documents not being corrupt.
Zip files and just about any file I can run through a repair tool and they are fixed. But that is not a doable option due to volume.
format file
Plus My format file had a 4 instead of 0. That alone made all files but PDF's documents corrupt with corrupt headers.
I have 24 spss files in .sav format in a single folder. All these files have the same structure. I want to run the same syntax on all these files. Is it possible to write a code in spss for this?
You can use the SPSSINC PROCESS FILES user submitted command to do this or write your own macro. So first lets create some very simple fake data to work with.
*FILE HANDLE save /NAME = "Your Handle Here!".
*Creating some fake data.
DATA LIST FREE / X Y.
BEGIN DATA
1 2
3 4
END DATA.
DATASET NAME Test.
SAVE OUTFILE = "save\X1.sav".
SAVE OUTFILE = "save\X2.sav".
SAVE OUTFILE = "save\X3.sav".
EXECUTE.
*Creating a syntax file to call.
DO IF $casenum = 1.
PRINT OUTFILE = "save\TestProcess_SHOWN.sps" /"FREQ X Y.".
END IF.
EXECUTE.
Now we can use the SPSSINC PROCESS FILES command to specify the sav files in the folder and apply the TestProcess_SHOWN.sps syntax to each of those files.
*Now example calling the syntax.
SPSSINC PROCESS FILES INPUTDATA="save\X*.sav"
SYNTAX="save\TestProcess_SHOWN.sps"
OUTPUTDATADIR="save" CONTINUEONERROR=YES
VIEWERFILE= "save\Results.spv" CLOSEDATA=NO
MACRONAME="!JOB"
/MACRODEFS ITEMS.
Another (less advanced) way is to use the command INSERT. To do so, repeatedly GET each sav-file, run the syntax with INSERT, and sav the file. Probably something like this:
get 'file1.sav'.
insert file='syntax.sps'.
save outf='file1_v2.sav'.
dataset close all.
get 'file2.sav'.
insert file='syntax.sps'.
save outf='file2_v2.sav'.
etc etc.
Good luck!
If the Syntax you need to run is completely independent of the files then you can either use: INSERT FILE = 'Syntax.sps' or put the code in a macro e.g.
Define !Syntax ()
* Put Syntax here
!EndDefine.
You can then run either of these 'manually';
get file = 'file1.sav'.
insert file='syntax.sps'.
save outfile ='file1_v2.sav'.
Or
get file = 'file1.sav'.
!Syntax.
save outfile ='file1_v2.sav'.
Or if the files follow a reasonably strict naming structure you can embed either of the above in a simple bit of python;
Begin Program.
imports spss
for i in range(0, 24 + 1):
syntax = "get file = 'file" + str(i) + ".sav.\n"
syntax += "insert file='syntax.sps'.\n"
syntax += "save outfile ='file1_v2.sav'.\n"
print syntax
spss.Submit(syntax)
End Program.
Below is an example of the BCP Statement.
I'm not accustomed to using BCP so your help and candor is greatly appreciated
I am using it with a format file as well.
If I execute from CMD prompt it works fine but from SQL I get the error.
The BCP statement is all on one line and the SQL Server Agent is running as Local System.
The SQL server, and script are on the same system.
I ran exec master..xp_fixeddrives
C,45589
E,423686
I've tried output to C and E with the same result
EXEC xp_cmdshell 'bcp "Select FILENAME, POLICYNUMBER, INSURED_DRAWER_100, POLICY_INFORMATION, DOCUMENTTYPE, DOCUMENTDATE, POLICYYEAR FROM data.dbo.max" queryout "E:\Storage\Export\Data\max.idx" -fmax-c.fmt -SSERVERNAME -T
Here is the format file rmax-c.fmt
10.0
7
1 SQLCHAR 0 255 "$#Y#$" 1 FILENAME
2 SQLCHAR 0 40 "" 2 POLICYNUMBER
3 SQLCHAR 0 40 "" 3 INSURED_DRAWER_100
4 SQLCHAR 0 40 "" 4 POLICY_INFORMATION
5 SQLCHAR 0 40 "" 5 DOCUMENTTYPE
6 SQLCHAR 0 40 "" 6 DOCUMENTDATE
7 SQLCHAR 0 8 "\r\n" 7 POLICYYEAR
Due to formating in this post the last column of the format file is cut off but reads SQL_Latin1_General_CP1_CI_AS for each column other that documentdate.
Does the output path exist? BCP does not create the folder before trying to create the file.
Try this before your BCP call:
EXEC xp_cmdshell 'MKDIR "E:\Storage\Export\Data\"'
First, rule out an xp_cmdshell issue by doing a simple 'dir c:*.*';
Check out my blog on using BCP to export files.
I had problems on my system in which I could not find the path to BCP.EXE.
Either change the PATH variable of hard code it.
Example below works with Adventure Works.
-- BCP - Export query, pipe delimited format, trusted security, character format
DECLARE #bcp_cmd4 VARCHAR(1000);
DECLARE #exe_path4 VARCHAR(200) =
' cd C:\Program Files\Microsoft SQL Server\100\Tools\Binn\ & ';
SET #bcp_cmd4 = #exe_path4 +
' BCP.EXE "SELECT FirstName, LastName FROM AdventureWorks2008R2.Sales.vSalesPerson" queryout ' +
' "C:\TEST\PEOPLE.TXT" -T -c -q -t0x7c -r\n';
PRINT #bcp_cmd4;
EXEC master..xp_cmdshell #bcp_cmd4;
GO
Before changing the path to \110\ for SQL Server 2012 and the name of the database to [AdventureWorks2012], I received the following error.
After making the changes, the code works fine from SSMS. The service is running under NT AUTHORITY\Local Service. The SQL Server Agent is disabled. The output file was created.
Please check, the file might be opened in another application or program.
If it is the case, bcp.exe cannot overwrite the existing file contents.
In my case, I solved The problem in the following way:
my command was :
bcp "select Top 1000 * from abc.dbo.abcd" queryout FileNameWithDirectory -c -t "|" -r "0x0a" -S 192.111.1.111 -U xx -P xxxxx
My FileNameWithDirectory was too long. like "D:\project-abc\R&D\abc-608\FilesNeeded\FilesNeeded\DataFiles\abc.csv".
I change into a simpler directory like : "D:\abc.csv"
Problem solved.
So I guess the problem occurred due to file name exceeding. thus the file was not found.
If it works from the command line but not from the SQL Agent, I think it is an authentication issue.
The SQL Server Agent is running under a account. Make sure that the account has the ability to read the format file and generate the output file.
Also, make sure the account has the ability to execute the xp_cmdshell stored procedure.
Write back with your progress ...
I received this after I shared my output folder, even when there were no files open.
I created a new, unshared folder for output and all was fine.
(might help someone ;-))
In my case this fix was simply running in administrator mode.
This error can be due to insufficient write permissions to the target folder.
This is a common issue, since the user writing the query might have access to a folder, but the SQL Server Agent or logged-in server account which actually invokes bcp.exe may not.
Destination path has to already exist (except for file name).
Remove no_output from your command, if you use one offcourse
SET #sql = 'BCP ....'
EXEC master..xp_cmdshell #sql , no_output
EXEC master..xp_cmdshell #sql
In case anyone else runs into the same problem: I had ...lesPerson" queryout' rather than ...lesPerson" queryout '
If your code is writing the data file, and then reading it with BCP, make sure that you CLOSE THE DATA FILE before trying to read it!
Failure to do so gives: 'Unable to open host data-file'.
Python example:
# Management of temporary bulk insert file.
def openBulkInsertFile(self) :
self.bulkInsertFile = open('c:/tmp/bulkInsertContent.txt', 'w', newline='')
self.csvWriter = csv.writer(self.bulkInsertFile)
def closeBulkInsertFile(self) :
self.bulkInsertFile.close()
When using a Job in SQL the user that uses the SQL express server is the current user logged, you should give write permission to that user in the folder where the Batch writes the output.
This happens usually only with bcp, when using type commands the ownership goes to the computer(Administrator) and the command runs with out problem.
So if you have a long command in your job just look for the bcp parts.
for whatever reason, when I get to the step where I generate a txt file with data from a query using the BCP utility, it hangs on the file creation. then if i try to query the database for those tables, it really won't let me.
does anyone know why this would happen? the query is actually very simple:
SET #cmdQueryout = 'bcp "SELECT X FROM Database.dbo.Details WHERE DetailsId = (SELECT MAX(DetailsId) FROM Database.dbo.Details WHERE CommitDateTime IS NOT NULL AND LEFT(PolicyNumber, 3) != ''NYD'') ORDER BY X, Y, Z" queryout "' + #detailFilePath + '" -c -T'
EXEC master..xp_cmdshell #cmdQueryout
I can see it created the first file but there's no data in it and it stops there.
I can open the file but if I try to delete, it won't let me because BCP is using the file.
The query should not take more than a few seconds to run so why would it stop like this?
EDIT - If I run this by itself in another query window, it works.
But if it's in a SQL job and in a transaction, it does not work.
Found out the issue.
I was using a trusted connection when I needed to specify a username & password.
-Uusername -Ppassword
instead of -T.