import multiple txt files continuously by filename and DateStamp - sql

Question

I would like to set up an automated job to continuously insert multiple txt files into a table.
I have multiple dataloggers that output multiple (every minute) .txt files and are named by their datestamp, i.e. 20130921_1755.txt (YYYYMMDD_HHMM.txt). They also have a field named DateStamp, which contains date values by the second for each record.
I know what I want my query to do....
insert all filenames from directory into table ALLFILENAMES
select maximum date in final table TBLMEASUREMENTS
convert maximum date to filename (2013-09-22 17:53:00 to "20130922_1753.txt")
bulk insert all filenames > max date from table ALLFILENAMES
I have already started the process by using a post found here:
Import Multiple CSV Files to SQL Server from a Folder
I am having trouble trying to sort out how to select which specific files I need imported to the table. Especially since this is an ongoing job, I need to constantly look to the DB to see which files have not been imported, and then import them. Here's my code so far which works to import multiple files.
--some variables
declare #filename varchar(255),
#path varchar(255),
#sql varchar(8000),
#cmd varchar(1000)
--get the list of files to process:
SET #path = 'C:\SQL_txt\1_hr\'
SET #cmd = 'dir ' + #path + '*.txt /b'
INSERT INTO ALLFILENAMES(WHICHFILE)
EXEC Master..xp_cmdShell #cmd
UPDATE ALLFILENAMES SET WHICHPATH = #path where WHICHPATH is null
--cursor loop
declare c1 cursor for SELECT WHICHPATH,WHICHFILE FROM ALLFILENAMES where WHICHFILE like '%.txt%'
open c1
fetch next from c1 into #path,#filename
While ##fetch_status <> -1
begin
--bulk insert won't take a variable name, so make a sql and execute it instead:
set #sql = 'BULK INSERT Temp FROM ''' + #path + #filename + ''' '
+ ' WITH (
FIELDTERMINATOR = ''\t'',
ROWTERMINATOR = ''\n''
) '
print #sql
exec (#sql)
fetch next from c1 into #path,#filename
end
close c1
deallocate c1
I have been playing around with LEFT, LEN and REPLACE to try to convert the max datestamp into a filename, but have had no luck. Any help or suggestions would be useful. Am I going at this wrong? Thanks

Answer 1

I would do this with SSIS/Data Tools.
Import a file from your 'watch' folder, then move the file to a different folder, using:
ForEach Loop Container
Data Flow Task
File System Task
Derived Column (optional but recommended for source tracking)
You can use the mapped filepath variable in a derived column to indicate source file, and unless there's danger of the same files being added to the watch folder multiple times, there's little need to run the 'has this been imported already' check each time.
Many tutorials on SSIS available, here are two:
SSIS - Loop through Flat Files
SSIS - Move and Rename Files