I'm currently tasked with reading some data that stored in a flat file into my database and run reports against it. The one problem I'm running into is checking to see if a file actually exists. Is there a simple function to check if the file exists?
Thanks!
Just googling I found this at SQL DBA and this at MS SQL tips.
You are doing ETL in a stored procedure?!! I don't think you should, just because you can.
I recommend you use use SSIS for this. Doing ETL in Stored Proc or TSQL is not a recommended practice, in fact, it is frequently used as an example of what not to do.
I believe you can do something like this:
DECLARE #Path varchar(128) ,
#FileName varchar(128)
SET #Path = 'C:\'
SET #FileName = 'FILE_NAME.EXT'
DECLARE #objFSys int
DECLARE #i int
DECLARE #File varchar(1000)
SET #File = #Path + #FileName
EXEC sp_OACreate 'Scripting.FileSystemObject', #objFSys out
EXEC sp_OAMethod #objFSys, 'FileExists', #i out, #File
IF #i = 1
PRINT 'file exists'
ELSE
PRINT 'file does not exists'
EXEC sp_OADestroy #objFSys
This article goes over this method and a couple others.
Related
Using openrowset, I am loading the XML file to a temp table.
How do I get the file created date?
CREATE TABLE #T
(
IntCol int,
XmlCol xml
);
INSERT INTO #T (XmlCol)
SELECT *
FROM OPENROWSET(BULK 'c:\Test.xml', SINGLE_BLOB) AS x;
SELECT * FROM #t
Not the most concise way to do this, but using Ole Automation in SQL Server is one way to get this information. The following sample uses C:\Temp\testfile.txt as an example. This not really "SQL", don't know if this good enough for you.
DECLARE #hr INT;
DECLARE #dt_created DATETIME;
DECLARE #obj_file INT;
DECLARE #obj_file_system INT;
DECLARE #file_name VARCHAR(100)='C:\Temp\testfile.txt';
-- Create a FileSystemObject. Create this once for all subsequent file manipulation. Don't forget to destroy this object once you're done with file manipulation (cf cleanup)
EXEC #hr = sp_OACreate 'Scripting.FileSystemObject', #obj_file_system OUT;
IF #hr<>0 GOTO __cleanup;
-- Get a handle for the file. Don't forget to release the handle for each file you get a handle for (see cleanup). The return will be different from 0 if the file doesn't exist
EXEC #hr = sp_OAMethod #obj_file_system, 'GetFile', #obj_file out, #file_name;
IF #hr<>0 GOTO __print_created_date;
-- Retrieve the created date.
EXEC sp_OAGetProperty #obj_file, 'DateCreated', #dt_created OUT;
__print_created_date:
SELECT #dt_created AS file_date;
__cleanup:
EXEC sp_OADestroy #obj_file_system;
EXEC sp_OADestroy #obj_file;
Ole Automation needs to be enabled first though (just once):
sp_configure 'show advanced options', 1;
GO
RECONFIGURE;
GO
sp_configure 'Ole Automation Procedures', 1;
GO
RECONFIGURE;
GO
T-SQL is not a language with which you can access file systems. However you can write a stored procedure in C# to accomplish this task. You would read the metadata with the appropriate classes in the .Net Framework. You can write a custom function with CLR integration to get every information you need from your filesystem.
Here is a little working sample to get an file created date with CLR integration in C#:
public class UserDefinedFunctions
{
[SqlFunction]
public static SqlDateTime GetCreatedDate(SqlString filePath)
{
return filePath.IsNull ? SqlDateTime.Null : File.GetCreationTime(filePath.Value);
}
}
Then you've got to deploy the assembly and register it to SQL Server, then create the proper function with Create function commands.
see more in CodeProject Sample
declare #f varchar(1000)='E:\work\share\data.txt';
declare #cmd varchar(1000)='for %a in ('+#f+') do echo %~ta'
IF OBJECT_ID('tempdb..#t1') IS NOT NULL
DROP TABLE #t1
create table #t1(_output varchar(1000))
insert #t1
exec xp_cmdshell #cmd
declare #fileDate varchar(100)
set #fileDate=(select _output from #t1 where _output not like '%echo%')
print #fileDate
and convert #fileDate to dateTime by your settings
This is part of my procedure to return #v_buffer variable.
Problem is that, the size of file i want to upload is greater than 8000 bytes. That's why i need to use varbinary(max) type.
But sp_oamethod read returns me an error.
Is someone know hot to use sp_oamethod to solve my problem?
declare #returnCode int
declare #v_file int
declare #v_buffer varbinary(max)
declare #v_fullpath nvarchar(400) --pdf file phusical location
exec #returncode = sp_oacreate 'adodb.stream', #v_file out
exec #returncode = sp_oamethod #v_file, 'open'
exec #returncode = sp_oasetproperty #v_file, 'type', 1
exec #returncode = sp_oasetproperty #v_file, 'loadfromfile', #v_fullpath
exec #returnCode = sp_oamethod #v_file, 'read', #v_buffer out, -1
if #returncode <> 0
begin
exec sp_oageterrorinfo #v_file
end
exec #returnCode = sp_OAMethod #v_file, 'Close'
exec #returnCode = sp_OADestroy #v_file
Error message returned by sp_oageterrorinfo:
0x8004271A, ODSOLE Extended Procedure, Error in srv_convert.
Please do not use the OLE Automation stored procedures (i.e. sp_OA* ) as they have been deprecated since SQL Server 2005 was released. What you are trying to do is rather simple with SQLCLR (i.e. .NET-based objects that exist within SQL Server). You can create a scalar function to accept a filepath and return its bytes using the File.ReadAllBytes method. Return that byte[] via the SqlBytes type.
You will need to set your Assembly to PERMISSION_SET = EXTERNAL_ACCESS. In order to accomplish that, please do not set the database to TRUSTWORTHY ON as that is an unnecessary security risk. Instead, sign the Assembly (using a password), then create an Asymmetric Key in the master Database from the DLL, then create a Login from that Key, and finally grant the Login the EXTERNAL ACCESS ASSEMBLY permission.
For a lot more information on working with SQLCLR, including many examples, please see the series of articles I am writing on this topic on SQL Server Central: Stairway to SQLCLR (that site does require free registration in order to read their content).
Or, if you don't want to deal with any coding, I created a library of over 270 functions and stored procedures called SQL#. There are several file system related functions, though none of them are available in the Free version. Still, the one that would help here is called: File_GetFileBinary.
If you try to pass a string that is greater than 4000 characters but less than or equal to 8000 characters in length to the sp_OASetProperty or to the sp_OAMethod OLE Automation extended stored procedure, the input string is silently truncated to 4000 characters before it is passed to the object and no error is returned.
If you try to set a property to a string that is greater than 8000 characters through the sp_OASetProperty OLE Automation extended stored procedure, or if you try to pass an input parameter to the sp_OAMethod OLE Automation extended stored procedure that is longer than 8000 characters, you receive the following error message:
hr Source Description
---------- ---------------------------- --------------------------
0x8004271A ODSOLE Extended Procedure Error in srv_convert.
https://support.microsoft.com/en-us/kb/325492
If someone had same problem, i have done it like this:
In my procedure i create temp #IMG table:
create table #IMG (FileID2 nvarchar(50), img image)
If the file size is greater than 8k, i split it to parts in size 8000 and update #IMG:
exec #returnCode = sp_oamethod #v_file, 'read', #v_buffer out, 8000
update #IMG set img = #v_buffer
SELECT #ptrval = TEXTPTR(Img) FROM #IMG WHERE FileID2 = #v_FileID2
--file split section
select #v_Blocks = #v_FileSize/8000+1
if #v_Blocks = 1
begin
WRITETEXT #IMG.Img #ptrval #v_buffer
end
else
begin
WRITETEXT #IMG.Img #ptrval #v_buffer
set #j=#v_blocks-1
while #j>0
begin
exec #returnCode = sp_oamethod #v_file, 'read', #v_buffer out , 8000
set #i=(select DATALENGTH(Img) from #IMG WHERE FileID2= #v_FileID2)
UPDATETEXT #IMG.Img #ptrval #i 0 #v_buffer
set #j=#j-1
end
end
You could use the same workaround as described at 0x8004271A ODSOLE Extended Procedure Error in srv_convert.
In your case, the line
exec #returnCode = sp_oamethod #v_file, 'read', #v_buffer out, -1
should be replaced by
Create table #tmp(dt varbinary(max))
insert into #tmp
exec #hr = sp_oamethod #v_file, 'read', #mode = -1
Select dt from #tmp -- single column/single row.
Drop Table #tmp -- clean up
Caution: this code wasn't tested. However the following code works fine for 32KB xml output:
Create table #tmp(dt xml)
insert into #tmp
exec #hr = sp_OAGetProperty #obj, 'responseXML.XML'
/*
Here is the trick: inserting from the returned result set, i.e.
`insert into <table>(<columns>) select <columns>`
*/
Select dt from #tmp -- single column/single row.
Drop Table #tmp -- clean up
responseXML.XML returns the whole xml document after calling MSXML2.ServerXMLHttp.send. The trick works according to section Result Sets of sp_OAMethod.
When the table variable type is available, the code can be even shorter:
DECLARE #xml(val xml);
insert into #xml
exec #hr = sp_OAGetProperty #obj, 'responseXML.XML'
Select * from #xml; -- just to see the output
I.e. there is no need to clean up.
I received the same issue. Apparently, you cannot use sp_OAMethod with a VARBINARY(MAX) variable.
Try changing: declare #v_buffer varbinary(max)
to declare #v_buffer varbinary(8000).
Good luck!
I am confronted with this devilish error : "Specified cast not valid" ; while calling a stored procedure through Linq.
SET NOCOUNT ON;
declare #packagename VARCHAR(100)
declare #servername VARCHAR(100)
SET #packagename = 'PackageName'
SET #servername = 'servername'
DECLARE #ssisstr varchar(8000)
set #ssisstr = 'C:\"Program Files (x86)"\"Microsoft SQL Server"\100\DTS\Binn\dtexec /sql ' + #packagename + ' /ser ' + #servername + ' ';
----now execute dynamic SQL by using EXEC.
declare #returncode nvarchar(100)
EXEC #returncode = xp_cmdshell #ssisstr;
select #returncode
The return looks like like a table with one column and content:
Microsoft (R) SQL Server Execute Package Utility
Version 10.50.4260.0 for 32-bit
Copyright (C) Microsoft Corporation 2010. All rights reserved.
NULL
...
stored procedure call looks like
using (DataContext context = new DataContext (connection))
{
var returncode = context.StoredProcedure(packagname,servername);
}
This is a bit long for a comment.
Stored procedures return a status value, which is always an integer. So, the statements:
declare #returncode nvarchar(100)
EXEC #returncode = xp_cmdshell #ssisstr;
are highly suspect. This may fix your error:
declare #returncode int;
EXEC #returncode = xp_cmdshell #ssisstr;
However, I'm guessing this will not be doing what you really want. If not, ask another question and describe what you are trying to do.
Integer return type is an acceptable solution. The code was also lacking the following no_output parameter:
DECLARE #returncode int
EXEC #returncode = xp_cmdshell #ssisstr, no_output;
select #returncode
Works, thanks!
if i run below script without char "|" it working but when i am adding char "|" it is not working
how to add char "|" using sql script to text file ?
DECLARE #Text AS VARCHAR(100)
DECLARE #Cmd AS VARCHAR(100)
SET #Text = 'Hello world| '
SET #Cmd ='echo ' + #Text + ' > C:\AppTextFile.txt'
EXECUTE Master.dbo.xp_CmdShell #Cmd
thanks
The pipe character has a special meaning in batch commands, so it must be escaped using the caret character. This should work:
DECLARE #Text AS VARCHAR(100)
DECLARE #Cmd AS VARCHAR(100)
SET #Text = 'Hello world^| '
SET #Cmd ='echo ' + #Text + ' > C:\AppTextFile.txt'
EXECUTE Master.dbo.xp_CmdShell #Cmd
Although this is really not a good way to write data to a text file: usually SQL Server should not have permission to write to the root of the C: drive, and xp_cmdshell is disabled by default. I suggest you look at alternatives like sqlcmd.exe, bcp.exe or a small script in your preferred language (PowerShell, Perl, Python, whatever).
It is generally much easier, safer and more flexible to query data from SQL Server than it is to push it out from the server side. In your specific case, it looks like you want to write out a delimited file, and bcp.exe is intended for that purpose.
Other way to do this
DECLARE #File varchar(300) = 'c:\Temp\out.txt'
DECLARE #Text varchar(8000) = 'Sample text'
DECLARE #OLE INT
DECLARE #FileID INT
EXECUTE sp_OACreate 'Scripting.FileSystemObject', #OLE OUT
EXECUTE sp_OAMethod #OLE, 'OpenTextFile', #FileID OUT, #File, 8, 1
EXECUTE sp_OAMethod #FileID, 'WriteLine', Null, #Text
EXECUTE sp_OADestroy #FileID
EXECUTE sp_OADestroy #OLE
Check for errors using SP result
EXECUTE #result = sp_OAMethod #OLE, 'OpenTextFile', #FileID OUT, #File, 8, 1
if #result<>0 GOTO ON_ERROR
Haven't found the way to read error message though
wrap it with two pairs of single quotes,
DECLARE #Text AS VARCHAR(100)
DECLARE #Cmd AS VARCHAR(100)
SET #Text = '''Hello world| '''
SET #Cmd ='echo ' + #Text + ' > C:\AppTextFile.txt'
EXECUTE Master.dbo.xp_CmdShell #Cmd
Trying to update a table on a linked server (SQL 2000/2005) but my server name will not be known ahead of time. I'm trying this:
DECLARE #Sql NVARCHAR(4000)
DECLARE #ParamDef NVARCHAR(4000)
DECLARE #SERVER_NAME VARCHAR(35)
SET #Sql = 'UPDATE
#server_name_param.dba_sandbox.dbo.SomeTable
SET SomeCol=''data'''
SET #ParamDef = N'#server_name_param VARCHAR(35)'
print #Sql
exec sp_executesql #Sql, #ParamDef, #server_name_param=#SERVER_NAME
Which returns this:
UPDATE
#server_name_param.dba_sandbox.dbo.SomeTable
SET SomeCol='data'
Msg 170, Level 15, State 1, Line 2
Line 2: Incorrect syntax near '.'.
Any ideas? Is there anyway I view the SQL statement that is being executed after the parameters are bound?
You'll have to do this, it can't be parameterised
....
SET #Sql = 'UPDATE ' + #server_name_param + '.dba_sandbox.dbo.SomeTable SET SomeCol=''data'''
....
Edit: There is another way which I used back in my pure DBA days
EXEC sp_setnetname 'AdhocServer', #SERVER_NAME
UPDATE AdhocServer.dba_sandbox.dbo.SomeTable SET SomeCol 'data'
EXEC sp_setnetname 'AdhocServer', 'MeaninglessValue'
sp_setnetname is there from SQL Server 2000 to 2008
Edit2. Permissions:
Try EXECUTE AS LOGIN = 'login_name' , where login_name is a superuser
I've not really used this (I use "AS USER" for testing), so not sure of the finer points...
Edit 3: for concurrency, consider using sp_getapplock and a stored procedure, or some other concurrency control mechanism.
You cannot do this with parameters directly - you would have to use dynamic SQL, or send the server name as a parameter to an SP that does dynamic SQL:
DECLARE #template NVARCHAR(4000)
DECLARE #Sql NVARCHAR(4000)
DECLARE #SERVER_NAME VARCHAR(35)
SET #template = 'UPDATE {#server_name_param}.dba_sandbox.dbo.SomeTable SET SomeCol=''data'''
SET #sql = REPLACE(#template, '{#server_name_param}', #SERVER_NAME)
print #Sql
exec sp_executesql #Sql -- OR EXEC ( #sql )
I like gbn's trick. I didn't know that one and I'm gonna have to research that some more.
Since I didn't know that trick, I've had to use dynamic sql in similar situations in the past (like what Cade posted). When that happens I would normally query an information schema view to make sure the parameter value is a real database object before building the query. That way I'm sure it's not an injection attempt.