I need to create a procedure where I will save some data on my Files Server (as an .csv).
Something like this:
--Note: I need permissions to access "0.0.0.0" address
--Note: I need to create a new file, and not replace an existing one
Select Name, Email From users => save to '\\0.0.0.0\c\userData'
I tried this:
--Note: "1.1.1.1" is the server where my database is, not sure if its the parameter I should use on ServerName
Exec xp_cmdshell
'bcp " Name, Email From users" queryout "\\0.0.0.0\c\userData" -c -t , -S "1.1.1.1" -T'
Got the following errors:
SQLState = S1000, NativeError = 0; Error = [Microsoft][ODBC Driver 13 for SQL Server]Unable to open BCP host data-file
I also saw that you can use something like this:
INSERT INTO OPENROWSET('Microsoft.ACE.OLEDB.12.0','Text;Database=D:\;HDR=YES;FMT=Delimited','SELECT * FROM [FileName.csv]')
SELECT Field1, Field2, Field3 FROM DatabaseName
But it has the downside the file has to exist already.
Q: So how can I achieve this? Export data to a file to a different machine with folder specific permissions/credentials (and create a new file, not overwrite it).
Related
My SQL Query generates a XML output:
select 'TEST.kml' as name,
(select 'TEST' as name, (
select (
select top 10 issue as name,
null as description,
null as 'Point/coordinates',
(
select
null as altitudeMode,
Coordinates as 'coordinates'
for xml path('Polygon'), type)
from Mapping for xml path('Placemark'), type))
for xml path ('Line') , type)
for xml path ('Doc'), root('kml'))
I want to save the output of the query as .XML file on to local drive.Please advise.
Not the most elegant way but it is possible to use bulk copy program and xp_cmdshell to do this. Few things first, xp_cmdshell is blocked by default by SQL Server as part of the security configuration so you will need to enable that first and BCP requires you to have access to the directory that you want to create the file.
To enable xp_cmdshell you'll need run sp_configure and RECONFIGURE, use this:
EXEC sp_configure'xp_cmdshell', 1
RECONFIGURE
GO
EXEC sp_configure 'show advanced options', 1
RECONFIGURE
GO
Then you can run the following:
EXEC xp_cmdshell 'bcp "SELECT * FROM [Database].dbo.[Table] FOR XML AUTO,
ELEMENTS" queryout "C:\test.xml" -c -T'
Just add your query into it and make sure you add [] around your table names.
The Microsoft Documents for xp_cmdshell are here and bcp can be found here
Using bcp is definite choice especially when working with large data sets. Alternatively, you can try using SQL Management Studio - Export Data.
Open the interface - Right Click on database name, then Tasks, then Export Data
The menu is opened. Click Next
Then choose SQL Server Native Client, sql server, database name and authentication method:
Then where to save the data:
Then how we are getting the data (in your case SQL query):
Past the query:
Then we have some settings, click finish.
To save the results of a remote query to a local file, you could use a Powershell script like this example:
$connection = New-Object System.Data.SqlClient.SqlConnection("Data Source=YourServer;Initial Catalog=YourDatabase;Integrated Security=SSPI")
$command = New-Object System.Data.SqlClient.SqlCommand(#("
select 'TEST.kml' as name,
(select 'TEST' as name, (
select (
select top 10 issue as name,
null as description,
null as 'Point/coordinates',
(
select
null as altitudeMode,
Coordinates as 'coordinates'
for xml path('Polygon'), type)
from Mapping for xml path('Placemark'), type))
for xml path ('Line') , type)
for xml path ('Doc'), root('kml');"), $connection);
$connection.Open();
$command.ExecuteScalar() | Out-File -FilePath "C:\KmlFiles\YourFile.kml";
$connection.Close();
The script can be executed from a command prompt by saving the script to a file with a ".ps1" extension and using a command like:
powershell -ExecutionPolicy RemoteSigned -File "C:\PowershellScripts\ExampleExport.ps1"
This command can be scheduled using a Windows Task Scheduler task to automate the export. Alternatively, schedule using a SQL Server agent job with a Powershell or CmdExec step.
I have two scripts. One is named sqlscript.sql and the other is named script.sh I have all of the queries needed written in my sql script. They are just a bunch of update statements. For example:
UPDATE xxDev.SYS_PARAMS SET val = 'serverName' WHERE lower(name) = 'enginebaseurl';
I'm running the .sql script IN the .sh script. When the .sh script runs, I want it to prompt the user for a server name and take that user input and replace it in serverName in the sql statements.
I'm brand new to both bash scripting and this website, so I hope I'm making sense asking this question. I'm using PuTTY if that makes a difference at all.
Suppose you use MySQL, try something like:
# TODO: prompt user for server name and store it into variable serverName
serverName="get from user"
cat <<"EOF" | mysql -u user1 -p passwd -h server1 -P 3306 -D db1
UPDATE xxDev.SYS_PARAMS SET val = '$serverName' WHERE lower(name) = 'enginebaseurl';
EOF
So in this example, you embed the sql script into the .sh so that you don't have to maintain two files.
I would probably use a variable
set #val 'serverName'
UPDATE xxDev.SYS_PARAMS SET val = #val WHERE lower(name) = 'enginebaseurl';
You can split the sqlscript.sql into
set-val.sql
set #val 'serverName'
and the actual update statements. Then you can recreate the set-val.sql from your user input:
echo -n "enter server: "
read server
echo "set #val '$server' > set-val.sql
and then you forward both files to mysql:
cat set-val.sql sqlscript.sql | mysql
You should probably use this only for internal things, it seems a little fragile.
I'm going let you figure out how to pass a shell parameter into your sql command, but here's an incredibly cool way to query the user for the server name. It might even be POSIX compliant.
#!/bin/sh
echo -n "Hit me with that server name: "; read serverName
echo "${serverName}! Outstanding! Pick up \$200 when you pass Go!"
I am trying to drop a file into a directory on the local machine (same machine running SQL Instance). The content of the table I am trying to drop out is in xml format.
ie. table=xmlOutFiles, fieldName = xmlContent; fieldName contains essentially varchar(max) data that is to become the xml file we need.
When the bcp command is executed it seems to create the file in the #dest location, size = 0 bytes and then the process running from within SMSS just sits there waiting for something!
I cannot do anything with that empty file, like delete it, unless I use task manager to kill the process "bcp.exe".
I have tried multiple combinations of the bcp flags, etc.
Running the bcp command from a system prompt, replacing the "#vars" seems to work but I really need it to be part of my SQL Trigger script and function.
Assistance appreciated!!!!
Select #dest = (Select filename from xmlOutfiles)
Select #cmd = 'bcp "Select xmlContent from ProcureToPay.dbo.XmlOutFiles" queryout '+#dest+' -x -w -T -S' + ##servername
Exec xp_cmdshell #cmd
I have tried executing with -T and -Uusername -Ppassword parameters, etc.
This command works from the DOS prompt:
bcp "Select xmlContent from Procure.To.Pay.dbo.XmlOutFiles" queryout c:\temp\test.xml -x -w -T S<myservernameHere>
Below is an example of the BCP Statement.
I'm not accustomed to using BCP so your help and candor is greatly appreciated
I am using it with a format file as well.
If I execute from CMD prompt it works fine but from SQL I get the error.
The BCP statement is all on one line and the SQL Server Agent is running as Local System.
The SQL server, and script are on the same system.
I ran exec master..xp_fixeddrives
C,45589
E,423686
I've tried output to C and E with the same result
EXEC xp_cmdshell 'bcp "Select FILENAME, POLICYNUMBER, INSURED_DRAWER_100, POLICY_INFORMATION, DOCUMENTTYPE, DOCUMENTDATE, POLICYYEAR FROM data.dbo.max" queryout "E:\Storage\Export\Data\max.idx" -fmax-c.fmt -SSERVERNAME -T
Here is the format file rmax-c.fmt
10.0
7
1 SQLCHAR 0 255 "$#Y#$" 1 FILENAME
2 SQLCHAR 0 40 "" 2 POLICYNUMBER
3 SQLCHAR 0 40 "" 3 INSURED_DRAWER_100
4 SQLCHAR 0 40 "" 4 POLICY_INFORMATION
5 SQLCHAR 0 40 "" 5 DOCUMENTTYPE
6 SQLCHAR 0 40 "" 6 DOCUMENTDATE
7 SQLCHAR 0 8 "\r\n" 7 POLICYYEAR
Due to formating in this post the last column of the format file is cut off but reads SQL_Latin1_General_CP1_CI_AS for each column other that documentdate.
Does the output path exist? BCP does not create the folder before trying to create the file.
Try this before your BCP call:
EXEC xp_cmdshell 'MKDIR "E:\Storage\Export\Data\"'
First, rule out an xp_cmdshell issue by doing a simple 'dir c:*.*';
Check out my blog on using BCP to export files.
I had problems on my system in which I could not find the path to BCP.EXE.
Either change the PATH variable of hard code it.
Example below works with Adventure Works.
-- BCP - Export query, pipe delimited format, trusted security, character format
DECLARE #bcp_cmd4 VARCHAR(1000);
DECLARE #exe_path4 VARCHAR(200) =
' cd C:\Program Files\Microsoft SQL Server\100\Tools\Binn\ & ';
SET #bcp_cmd4 = #exe_path4 +
' BCP.EXE "SELECT FirstName, LastName FROM AdventureWorks2008R2.Sales.vSalesPerson" queryout ' +
' "C:\TEST\PEOPLE.TXT" -T -c -q -t0x7c -r\n';
PRINT #bcp_cmd4;
EXEC master..xp_cmdshell #bcp_cmd4;
GO
Before changing the path to \110\ for SQL Server 2012 and the name of the database to [AdventureWorks2012], I received the following error.
After making the changes, the code works fine from SSMS. The service is running under NT AUTHORITY\Local Service. The SQL Server Agent is disabled. The output file was created.
Please check, the file might be opened in another application or program.
If it is the case, bcp.exe cannot overwrite the existing file contents.
In my case, I solved The problem in the following way:
my command was :
bcp "select Top 1000 * from abc.dbo.abcd" queryout FileNameWithDirectory -c -t "|" -r "0x0a" -S 192.111.1.111 -U xx -P xxxxx
My FileNameWithDirectory was too long. like "D:\project-abc\R&D\abc-608\FilesNeeded\FilesNeeded\DataFiles\abc.csv".
I change into a simpler directory like : "D:\abc.csv"
Problem solved.
So I guess the problem occurred due to file name exceeding. thus the file was not found.
If it works from the command line but not from the SQL Agent, I think it is an authentication issue.
The SQL Server Agent is running under a account. Make sure that the account has the ability to read the format file and generate the output file.
Also, make sure the account has the ability to execute the xp_cmdshell stored procedure.
Write back with your progress ...
I received this after I shared my output folder, even when there were no files open.
I created a new, unshared folder for output and all was fine.
(might help someone ;-))
In my case this fix was simply running in administrator mode.
This error can be due to insufficient write permissions to the target folder.
This is a common issue, since the user writing the query might have access to a folder, but the SQL Server Agent or logged-in server account which actually invokes bcp.exe may not.
Destination path has to already exist (except for file name).
Remove no_output from your command, if you use one offcourse
SET #sql = 'BCP ....'
EXEC master..xp_cmdshell #sql , no_output
EXEC master..xp_cmdshell #sql
In case anyone else runs into the same problem: I had ...lesPerson" queryout' rather than ...lesPerson" queryout '
If your code is writing the data file, and then reading it with BCP, make sure that you CLOSE THE DATA FILE before trying to read it!
Failure to do so gives: 'Unable to open host data-file'.
Python example:
# Management of temporary bulk insert file.
def openBulkInsertFile(self) :
self.bulkInsertFile = open('c:/tmp/bulkInsertContent.txt', 'w', newline='')
self.csvWriter = csv.writer(self.bulkInsertFile)
def closeBulkInsertFile(self) :
self.bulkInsertFile.close()
When using a Job in SQL the user that uses the SQL express server is the current user logged, you should give write permission to that user in the folder where the Batch writes the output.
This happens usually only with bcp, when using type commands the ownership goes to the computer(Administrator) and the command runs with out problem.
So if you have a long command in your job just look for the bcp parts.
I am attempting to create a batch file in windows that will take a user's input, and pass that along to a sql file containing the following query, so that I can set a siteid, like in the following sql query:
exec sp_addlinkedserver [sqlserver1]
select * from [sqlserver1].onesource.dbo.admsites where siteid = '123'
I want to then take the results of this query, particularly the admsiteid, and then use the results of the query, and insert that into the originatorid (using another .sql file:
Use Onesource
update OSCsettings set originatorid = 'whatever-the-admsiteid-is'
How would I go about passing along these variables?
sqlcmd with the -v command line
-v var = "value"
You can specify multiple variables in the list.
See:
http://msdn.microsoft.com/en-us/library/ms162773.aspx
and
http://msdn.microsoft.com/en-us/library/ms188714.aspx