Export query result to csv in UTF-8 - sql

I would like to export query result to csv file in utf-8. Now i export to csv in this way:
DECLARE #cmd varchar(1000)
SET #cmd = 'bcp "select * from table" queryout "d:\textfile.csv" -w -T -t; -Slocalhost'
EXEC xp_cmdshell #cmd
How to make file be in UTF-8?

SQL Server does not support code page 65001 (UTF-8 encoding). reference
You need to add -w parameter in bcp utility to specify the encoding is UTF16.

I am sure you have long since solved your problem but as this is still open thought it might help someone else.
I am creating txt files with SQL Server queries and they seem to be "plain ANSI". In the end I used iconv.exe to convert from WINDOWS-1252 to UTF-8
iconv.exe is part of the GnuWin package which can be downloaded from Sourceforge https://sourceforge.net/projects/gnuwin32/

Related

How export CSV with SQLCMD in UTF-16?

I am trying to export the data from different queries by SQLCMD, my files must be exported in UTF-16, currently they do it in UTF8 BOM and I have been able to even in UTF8, but I cannot get it to export in UTF16.
Reviewing the Encoding of the file in Notepad++ this is the format in which it should be exported (UTF16 LE BOM), in the list selection it appears with the name UCS-2 Little Endian
Encoding to export
Encoding in options
SET #cmd= 'sqlcmd -s"-" -f i:1252 -Q "SELECT * FROM Machines" -f o:65001 -o "C:\CSV\report_machines.csv"'
EXEC master..xp_cmdshell #cmd
Check the page (https://www.example-code.com/sql/load_text_file_using_code_page.asp) and tried code 1200, however SQL Server returns the following message:
Sqlcmd: The code page <1200> specified in option -f is invalid or not installed on this system.
I want to avoid using another program, if anyone knows how to solve this problem I would greatly appreciate it.

BCP Imported File in UTF-8 Encoded File

I have generated a CSV file using BULK Copy (BCP) utility in SQL Server 2008. I want a file that will be in UTF-8 Encoding. I am using below code to generate File. Any one can help me to resolve this problem.
DECLARE #Path VARCHAR(100) ,
#sql VARCHAR(8000) ,
#DBName VARCHAR(100) ,
#exePath VARCHAR(100),
SELECT #sql = 'bcp "SELECT * FROM '+#DBName+'..ABC" queryout '+#Path+'\'+'ABC'.csv -c -k -t, -U sa -P abc -T -S ' + ##servername
exec #exePath #sql;
Sql Server 2008 don't support utf-8 but you can use "-w" instead "-c" that will use UTF-16 as mentionned by DiMach, the probleme is that the csv file not will be well formed.
If you trying to export csv file for specific language, for example french (french contains spécial caracters : à é â ç ...) you will need to know that there is an other solution :
First : try to search in the internet the encoding that support all charaters of the target language, for example for french we can found ISO-8859-1
Second : search in the internet a page code related to encoding that you find, for ISO-8859-1 is 1252
Third : execute to following sql request in your sql server to see if your sql server support this encoding
select name, COLLATIONPROPERTY(name, 'CodePage') as Code_Page, description
from sys.fn_HelpCollations() where COLLATIONPROPERTY(name, 'CodePage')=1252 order by Code_Page
fourth : in BCP command use -C 1252 only (delete -c and -w)
I hope that's help
Just replace "-c" parameter with "-w"
SELECT #sql = 'bcp "SELECT * FROM '+#DBName+'..ABC" queryout '+#Path+'\'+'ABC'.csv -w -k -t, -U sa -P abc -T -S ' + ##servername

My bcp hangs after creating an empty file

I am trying to drop a file into a directory on the local machine (same machine running SQL Instance). The content of the table I am trying to drop out is in xml format.
ie. table=xmlOutFiles, fieldName = xmlContent; fieldName contains essentially varchar(max) data that is to become the xml file we need.
When the bcp command is executed it seems to create the file in the #dest location, size = 0 bytes and then the process running from within SMSS just sits there waiting for something!
I cannot do anything with that empty file, like delete it, unless I use task manager to kill the process "bcp.exe".
I have tried multiple combinations of the bcp flags, etc.
Running the bcp command from a system prompt, replacing the "#vars" seems to work but I really need it to be part of my SQL Trigger script and function.
Assistance appreciated!!!!
Select #dest = (Select filename from xmlOutfiles)
Select #cmd = 'bcp "Select xmlContent from ProcureToPay.dbo.XmlOutFiles" queryout '+#dest+' -x -w -T -S' + ##servername
Exec xp_cmdshell #cmd
I have tried executing with -T and -Uusername -Ppassword parameters, etc.
This command works from the DOS prompt:
bcp "Select xmlContent from Procure.To.Pay.dbo.XmlOutFiles" queryout c:\temp\test.xml -x -w -T S<myservernameHere>

Unable to open BCP host data-file

Below is an example of the BCP Statement.
I'm not accustomed to using BCP so your help and candor is greatly appreciated
I am using it with a format file as well.
If I execute from CMD prompt it works fine but from SQL I get the error.
The BCP statement is all on one line and the SQL Server Agent is running as Local System.
The SQL server, and script are on the same system.
I ran exec master..xp_fixeddrives
C,45589
E,423686
I've tried output to C and E with the same result
EXEC xp_cmdshell 'bcp "Select FILENAME, POLICYNUMBER, INSURED_DRAWER_100, POLICY_INFORMATION, DOCUMENTTYPE, DOCUMENTDATE, POLICYYEAR FROM data.dbo.max" queryout "E:\Storage\Export\Data\max.idx" -fmax-c.fmt -SSERVERNAME -T
Here is the format file rmax-c.fmt
10.0
7
1 SQLCHAR 0 255 "$#Y#$" 1 FILENAME
2 SQLCHAR 0 40 "" 2 POLICYNUMBER
3 SQLCHAR 0 40 "" 3 INSURED_DRAWER_100
4 SQLCHAR 0 40 "" 4 POLICY_INFORMATION
5 SQLCHAR 0 40 "" 5 DOCUMENTTYPE
6 SQLCHAR 0 40 "" 6 DOCUMENTDATE
7 SQLCHAR 0 8 "\r\n" 7 POLICYYEAR
Due to formating in this post the last column of the format file is cut off but reads SQL_Latin1_General_CP1_CI_AS for each column other that documentdate.
Does the output path exist? BCP does not create the folder before trying to create the file.
Try this before your BCP call:
EXEC xp_cmdshell 'MKDIR "E:\Storage\Export\Data\"'
First, rule out an xp_cmdshell issue by doing a simple 'dir c:*.*';
Check out my blog on using BCP to export files.
I had problems on my system in which I could not find the path to BCP.EXE.
Either change the PATH variable of hard code it.
Example below works with Adventure Works.
-- BCP - Export query, pipe delimited format, trusted security, character format
DECLARE #bcp_cmd4 VARCHAR(1000);
DECLARE #exe_path4 VARCHAR(200) =
' cd C:\Program Files\Microsoft SQL Server\100\Tools\Binn\ & ';
SET #bcp_cmd4 = #exe_path4 +
' BCP.EXE "SELECT FirstName, LastName FROM AdventureWorks2008R2.Sales.vSalesPerson" queryout ' +
' "C:\TEST\PEOPLE.TXT" -T -c -q -t0x7c -r\n';
PRINT #bcp_cmd4;
EXEC master..xp_cmdshell #bcp_cmd4;
GO
Before changing the path to \110\ for SQL Server 2012 and the name of the database to [AdventureWorks2012], I received the following error.
After making the changes, the code works fine from SSMS. The service is running under NT AUTHORITY\Local Service. The SQL Server Agent is disabled. The output file was created.
Please check, the file might be opened in another application or program.
If it is the case, bcp.exe cannot overwrite the existing file contents.
In my case, I solved The problem in the following way:
my command was :
bcp "select Top 1000 * from abc.dbo.abcd" queryout FileNameWithDirectory -c -t "|" -r "0x0a" -S 192.111.1.111 -U xx -P xxxxx
My FileNameWithDirectory was too long. like "D:\project-abc\R&D\abc-608\FilesNeeded\FilesNeeded\DataFiles\abc.csv".
I change into a simpler directory like : "D:\abc.csv"
Problem solved.
So I guess the problem occurred due to file name exceeding. thus the file was not found.
If it works from the command line but not from the SQL Agent, I think it is an authentication issue.
The SQL Server Agent is running under a account. Make sure that the account has the ability to read the format file and generate the output file.
Also, make sure the account has the ability to execute the xp_cmdshell stored procedure.
Write back with your progress ...
I received this after I shared my output folder, even when there were no files open.
I created a new, unshared folder for output and all was fine.
(might help someone ;-))
In my case this fix was simply running in administrator mode.
This error can be due to insufficient write permissions to the target folder.
This is a common issue, since the user writing the query might have access to a folder, but the SQL Server Agent or logged-in server account which actually invokes bcp.exe may not.
Destination path has to already exist (except for file name).
Remove no_output from your command, if you use one offcourse
SET #sql = 'BCP ....'
EXEC master..xp_cmdshell #sql , no_output
EXEC master..xp_cmdshell #sql
In case anyone else runs into the same problem: I had ...lesPerson" queryout' rather than ...lesPerson" queryout '
If your code is writing the data file, and then reading it with BCP, make sure that you CLOSE THE DATA FILE before trying to read it!
Failure to do so gives: 'Unable to open host data-file'.
Python example:
# Management of temporary bulk insert file.
def openBulkInsertFile(self) :
self.bulkInsertFile = open('c:/tmp/bulkInsertContent.txt', 'w', newline='')
self.csvWriter = csv.writer(self.bulkInsertFile)
def closeBulkInsertFile(self) :
self.bulkInsertFile.close()
When using a Job in SQL the user that uses the SQL express server is the current user logged, you should give write permission to that user in the folder where the Batch writes the output.
This happens usually only with bcp, when using type commands the ownership goes to the computer(Administrator) and the command runs with out problem.
So if you have a long command in your job just look for the bcp parts.

Problem with BCP writing to .txt file from SQL

Im using Sql2008 trying to run this BCP command but it never creates the file.
-- Export query
DECLARE #qry2 VARCHAR(1000)
SET #qry2 = 'SELECT * FROM #SkippedProductsTable'
-- Folder we will be putting the file in
DECLARE #incomingfolder VARCHAR(1000)
SET #incomingfolder = 'c:\Logs'
DECLARE #bcpCommand VARCHAR(2000)
SET #bcpCommand = 'bcp "'+#qry2+'" queryout "'+#incomingfolder+'\SkippedProducts-'+CAST(#StoreMatchCode AS VARCHAR)+'-'+'.txt" -c -T'
PRINT #bcpCommand
EXEC MASTER..xp_cmdshell #bcpCommand, no_output
The created command looks like:
bcp "SELECT * FROM #SkippedProductsTable" queryout "c:\Logs\SkippedProducts-1330-.txt" -c -T
Can anyone suggest what could be going wrong? I've never used BCP before and not really sure where to start looking.
As a start I know that the folder deffinately exists at that location
I think the problem is the SELECT.
You are SELECTing from a table variable that is not declared in the query, so there's nothing for BCP to do.
Table variables only persist for the context they are called in, so even if you have one in a query, and you have dynamic sql or a subproc within that first query, they won't be able to see the table variable.
See this for more info.