in my database, the tables are collation = latin1
in mysqldump command i put --default-character-set=latin1
BUT, in sql ouput file, the accents, like á, appears ├í,
i appears í
I don't have idea and no find solution for this :x
after searching lot, i found the solution in this question
MysqlDump from Powershell and Windows encoding
To resolve, insert
[System.Console]::OutputEncoding = [System.Text.Encoding]::UTF8
in top of your PowerShell command, after, in invoke-expression, insert this -Encoding UTF8
is this the full command
invoke-expression $cmd | Out-File $backuppathandfile -Encoding UTF8
Related
I am loading data into SQL Server using bcp command line utility.
The problem is data is coming is like below. Every field in double quotes and I have to skip last three rows because it has some trailers. How can i solve this?
Thanks in Advance. I appreciate if you could help in this.
bcp database.schema.tablename in Filename.text -T -c -t"|" -r"0x0a" -F 3 -m 2
UHDR 20211110
"DATE","CUSIP","ISIN","SEDOL","TICKER","DESCRIPTION","QUANTITY","RATE","COMMENT","MARKET","FEE"
11/10/2021|""|"CA45826T3010"|"BMVXZT5"|"ITR"|"INTEGRA RESOURCES CORP REGISTERED SHS"|"28712"|"0.0000"|"HTB"|"CA"|"11.5000"
If you want to:
unconditionally remove all " chars.
unconditionally skip the last 3 lines:
(Get-Content -ReadCount 0 Filename.text) -replace '"' |
Select-Object -SkipLast 3 |
Set-Content Filename_CleanedUp.text
Note: -ReadCount 0 is a performance optimization that makes Get-Content read all lines into a single array instead of streaming the lines one by one.
Then pass Filename_CleanedUp.text to your bcp command.
I have 4 files with the same csv header as following
Column1,Column2,Column3,Column4
But I only required data from Column2,Column3,Column4 for import the data into SQL database using BCP . I am using the PowerShell to select the columns that I want and import the required data using BCP but my powershell executed with no error and there are not data updated in my database table. May I know how to set the BCP to import the output from Powershell to database table. Here are my powershell script
$filePath = Get-ChildItem -Path 'D:\test\*' -Include $filename
$desiredColumn = 'Column2','Column3','Column4'
foreach($file in $filePath)
{
write-host $file
$test = import-csv $file | select $desiredColumn
write-host $test
$action = bcp <myDatabaseTableName> in $test -T -c -t";" -r"\n" -F2 -S <MyDatabase>
}
These are the output from the powershell script
D:\test\sample1.csv
#{column2=111;column3=222;column4=333} #{column2=444;column3=555;column4=666}
D:\test\sample2.csv
#{column2=777;column3=888;column4=999} #{column2=aaa;column3=bbb;column4=ccc}
First off, you can't update a table with bcp. It is used to bulk load data. That is, it will either insert new rows or export existing data into a flat file. Changing existing rows, usually called as updating, is out of scope for bcp. If that's what you need, you need to use another a tool. Sqlcmd works fine, and Powershell's got Invoke-Sqlcmd for running arbitary TSQL statements.
Anyway, the BCP utility has notoriously tricky syntax. As far as I know, one cannot bulk load data by passing the data as parameter to bcp, a source file must be used. Thus you need to save the filtered file and pass its name to bcp.
Exporting a filtered CSV is easy enough, just remember to use -NoTypeInformation switch, lest you'll get #TYPE Selected.System.Management.Automation.PSCustomObject as your first row of data. Assuming the bcp arguments are well and good (why -F2 though? And Unix newlines?).
Stripping double quotes requires another an edit to the file. Scrpting Guy has a solution.
foreach($file in $filePath){
write-host $file
$test = import-csv $file | select $desiredColumn
# Overwrite filtereddata.csv, should one exist, with filtered data
$test | export-csv -path .\filtereddata.csv -NoTypeInformation
# Remove doulbe quotes
(gc filtereddata.csv) | % {$_ -replace '"', ''} | out-file filtereddata.csv -Fo -En ascii
$action = bcp <myDatabaseTableName> in filtereddata.csv -T -c -t";" -r"\n" -F2 -S <MyDatabase>
}
Depending on your locale, column separator might be semicolon, colon or something else. Use -Delimiter '<character>' switch to pass whatever you need or change bcp's argument.
Erland's got a helpful page about bulk operations. Also, see Redgate's advice.
Without need to modify the file first, there is an answer here about how bcp can handle quoted data.
BCP in with quoted fields in source file
Essentially, you need to use the -f option and create/use a format file to tell SQL your custom field delimiter (in short, it is no longer a lone comma (,) but it is now (",")... comma with two double quotes. Need to escape the dblquotes and a small trick to handle the first doulbe quote on a line. But it works like a charm.
Also, need the format file to ignore column(s)... just set the destination column number to zero. All with no need to modify the file before load. Good luck!
I'm trying to learn the basics of sql programming, I am working with SQL Server 2014. I have managed to import a file into a table with the command:
BULK INSERT Db.dbo.Co2_table
FROM 'd:\dataset_co2.txt'
with
(
FIRSTROW =2,
ROWTERMINATOR ='\n'
)
GO
I would like to do the dual operation, that is exporting the content of a table to a file. I have tried:
SELECT *
INTO OUTFILE 'C:\datadump\sqldbdump.txt"
FROM dbo.alarms_2_2014
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n';
bcp Db.dbo.Co2_table out "C:\users\ws5.en-cre\desktop\prova.txt" -T –c
sqlcmd -S . -d Db -E -s, -W -Q "SELECT * FROM dbo.Co2_table" > ExcelTest.csv
But none of these seem to work (I get error messages). Any idea?
I suspect you are running those commands from Management Studio. You should use console for this command.This works for me. Also check if you have permissions on that folder.
bcp "select * from Db.dbo.Co2_table" queryout C:\users\ws5.en-cre\desktop\prova.txt -c -T
or
bcp Db.dbo.Co2_table out C:\users\ws5.en-cre\desktop\prova.txt -c -T
Also you have suspicious symbol in c parameter -T –c. It is not a regular dash -.
Thank you for you answers and suggestions, and apologies for my lack of precision and my late reply (in this case I missed the notifications from stackoverflow).
Regarding the question on whether I use mstudio or console, what I do is clicking on “new query” from mstudio, write the code and press execute. So I guess the answer is that I use mstudio.
If I try:
bcp "select * from Db.dbo.Co2_table" queryout
C:\users\ws5.en-cre\desktop\prova.txt -c –T
it says
Msg 102, Level 15, State 1, Line 1 Incorrect syntax near 'queryout'.
I guess in this case one of the problem is that the quotes are missing, but even adding them doesn’t solve the problem.
I am looking for a solution that can be implemented as a script. I am familiar with excel vba macros, I would like to implement something like that.
Thanks,
Alex
I'm trying to use bcp to query out a comma-separated-value file but each time I get an empty file.
Here's my bcp command:
bcp "SELECT * FROM ##OutAK " QUERYOUT D:\Outbound\raw\li14090413.raw -c -T -t -S DB1
I have verified that ##OutAK is NOT empty because select count (*) from ##OutAK is not 0. When open file using HEX editor, I see the following:
0D 0A
I found the problem. It seems BCP is "allergic" with NULL. So, I just put ISNULL() to all the null-able fields and the output file is back to normal now.
I use SQL Server 2005 and I am try to store Cyrillic characters but I can't with SQL code by trying to run this is SQL Server:
INSERT INTO Assembly VALUES('Македонски парлиамент број 1','','');
Or from C# is happening the same problem but inserting/updating the column from SQL Server it work and it is store normally.
The datatype of column is nvarchar.
You have to add N prefix before your string.
When you implicitly declare a string variable it is treated as varchar by default. Adding prefix N denotes that the subsequent string is in Unicode (nvarchar).
INSERT INTO Assembly VALUES(N'Македонски парлиамент број 1','','');
Here is some reading:
http://databases.aspfaq.com/general/why-do-some-sql-strings-have-an-n-prefix.html
https://msdn.microsoft.com/en-IN/library/ms186939.aspx
What is the meaning of the prefix N in T-SQL statements?
I'm not sure if you are doing a static stored procedure or scripting, but maybe the text is not being encoded properly when you save it to disk. I ran into this, and my problem was solved in PowerShell by correcting the encoding of the SQL that I saved to disk for osql processing:
Out-File -FilePath "MyFile.sql" -InputObject $MyRussianSQL -Encoding "Unicode" -Force;
& osql -U myuser -P password -i "MyFile.sql";