I'm working with SSIS and am needing to read all the Customer Numbers out of the Customer table, and search through a directory to see if a PDF file exists with the Customer Number (e.g. A000134) within the filename. If it does, attach it to an email and send.
Does anyone have any suggestions on how this could be achieved?
How I thought of approaching it was:
1) Loop through the directory and get all the filenames/path and write to a table.
2) Using String Functions, pull the Customer Number out of the filename.
3) Call a Stored Procedure and within the stored proc, loop through all customers who have a file to send (joining on Customer Number between CustomerTable and FilesTable), and use #file_attachments in sp_send_dbmail to reference the file name to send.
Nice problem but i think SSIS is not right toll to solve this.
Still you can achieve using ssis
Load file data in TXT column and use full text search and use
full-text-search query in ssis
Use custom code to look into files using (C# or Vb) for ssis
But My suggestion would be write some stand alone utility to do this job done.
Related
I'm trying to find a specific string on my database. I'm currently using FlameRobin to open the FDB file, but this software doesn't seems to have a properly feature for this task.
I tried the following SQL query but i didn't work:
SELECT
*
FROM
*
WHERE
* LIKE '126278'
After all, what is the best solution to do that? Thanks in advance.
You can't do such thing. But you can convert your FDB file to a text file like CSV so you can search for your string in all the tables/files at the same time.
1. Download a database converter
First step you need a software to convert you databse file. I recommend using Full Convert to do it. Just get the free trial and download it. It is really easy to use and it will export each table in a different CSV file.
2. Find your string in multiple files at the same time
For that task you can use the Find in files feature of Notepad++ to search the string in all CSV files located at the same folder.
3. Open the desired table on FlameRobin
When Notepad++ highlight the string, it shows in what file it is located and the number of the line. Full Convert saves each CSV with the same name as the original table, so you can find it easily whatever database manager software you are using.
Here is Firebird documentation: https://www.firebirdsql.org/file/documentation/reference_manuals/fblangref25-en/html/fblangref25.html
You need to read about
Stored Procedures of "selectable" kind,
execute statement command, including for execute statement variant
system tables, having "relation" in names.
Then in your SP you do enumerate all the tables, then you do enumerate all the columns in those tables, then for every of them you run a usual
select 'tablename', 'columnname', columnname
from tablename
where columnname containing '12345'
over every field of every table.
But practically speaking, it most probably would be better to avoid SQL commands and just to extract ALL the database into a long SQL script and open that script in Notepad (or any other text editor) and there search for the string you need.
apologies if I've phrased this terribly. I only started using SSIS today.
I've written a FOREACH which loops through all the files in a folder, and updates my table f_actuals together with the filename without the extension - this filename is a combination of a PeriodKey and Business Unit. It works well.
However, this is intended to be a daily upload from our system for the entire month for each business unit (so the month-to-date refreshes daily until we close that period), so what I really need is the FOREACH to include something which does the following: -
Checks the filenames due for import in the designated folder against the filenames already in the f_actuals table
Removes all the matches from the f_actuals table
Continues with the FOREACH I've already built
I know this is probably a massively inefficient way to do this (preference would be daily incremental uploads), but the files need to be month-to-date, as our system cannot provide anything else easily.
Hope this makes sense.
Any help greatly appreciated.
You can use an Execute SQL Task within the For Each Loop to do this.
You can either use an SQL statement:
DELETE
FROM f_actuals
WHERE filename = ?
Or perhaps a stored procedure (accepting your filename as a parameter and doing the same thing as the statement above), e.g.:
EXEC DeleteFromActuals ?
For each filename in your loop, you would store this in a variable, and pass the variable as a parameter in the Execute SQL Task (this is what the ? is).
To map the parameter in the Execute SQL Task, go to 'Parameter Mapping', and add a new parameter. Select the variable containing the filename from the dropdown list, choose a data type of VARCHAR, and set the 'Parameter Name' to 0. The 'Direction' should be 'Input', which is the default.
I am new to SQL.
What is the best way to create a TXT file, if a table has records > 0?
The code already exists to remove or add records to this table.
I am looking for ways to create a trigger file (with no content in the file) at a specific network folder.
Preferably, I would want this TXT file to be removed at the end of the day, so the process could repeat itself every morning
On an after delete Trigger do a select count(*) from table or query one of the system catalog views. If its zero, then call a stored proc that poops a file onto your share drive.
To move the file you could create a small package or call a powershell or bcp (after enabling xp_cmdshell though), or you could create a CLR function (after enabling CLR). I guess since the latter two you need to change a server setting, you could just create a package.
Annnd since there is no data you dont actually need to export, you just create a blank file!
I'm trying to use oracle external tables to load flat files into a database but I'm having a bit of an issue with the location clause. The files we receive are appended with several pieces of information including the date so I was hoping to use wildcards in the location clause but it doesn't look like I'm able to.
I think I'm right in assuming I'm unable to use wildcards, does anyone have a suggestion on how I can accomplish this without writing large amounts of code per external table?
Current thoughts:
The only way I can think of doing it at the moment is to have a shell watcher script and parameter table. User can specify: input directory, file mask, external table etc. Then when a file is found in the directory, the shell script generates a list of files found with the file mask. For each file found issue a alter table command to change the location on the given external table to that file and launch the rest of the pl/sql associated with that file. This can be repeated for each file found with the file mask. I guess the benefit to this is I could also add the date to the end of the log and bad files after each run.
I'll post the solution I went with in the end which appears to be the only way.
I have a file watcher than looks for files in a given input dir with a certain file mask. The lookup table also includes the name of the external table. I then simply issue an alter table on the external table with the list of new file names.
For me this wasn't much of an issue as I'm already using shell for most of the file watching and file manipulation. Hopefully this saves someone searching for ages for a solution.
What is the best practice when importing and validating an XML file to a single table (flattened) in SQL Server ?
I've a XML file which contains about 15 complex types which are all related to a single parent element.
The SSIS design could look like this:
But it's getting very complicated with all those (15) joins.
Is it maybe a better idea to just write T-SQL code to :
1) Import the XML into a column which is of the type XML and is linked to a XSD-schema.
2) Use this code:
TRUNCATE TABLE XML_Import
INSERT INTO XML_Import(ImportDateTime, XmlData)
SELECT GETDATE(), XmlData
FROM
(
SELECT *
FROM OPENROWSET (BULK 'c:\XML-Data.xml', SINGLE_BLOB) AS XMLDATA
) AS FileImport (XMLDATA)
delete from dbo.UserFlat
INSERT INTO dbo.UserFlat
SELECT
user.value('(UserIdentifier)', 'varchar(8)') as UserIdentifier,
user.value('(Emailaddress)', 'varchar(70)') as Emailaddress,
businessaddress.value('(Fax)', 'varchar(70)') as Fax,
employment.value('(EmploymentData)', 'varchar(8)') as EmploymentData,
-- More values here ...
FROM
XML_Import CROSS APPLY
XmlData.nodes('//user') AS User(user) CROSS APPLY
user.nodes('BusinessAddress') AS BusinessAddress(businessaddress) CROSS APPLY
user.nodes('Employment') AS Employment(employment)
-- More 'joins' here ...
to fill the 'UserFlat' table ?Some disadvantages are that you have to manually type the SQLcode, but the advantage here is that I have more direct control how the elements are processed and converted. But I don't know if there are any performance differences between processing XML in SSIS and processing the XML with T-SQL XML statements.
Note that some other requirements are:
Error handling : in case of an error, an email must be send to a person.
Able to process multiple input files with a specific file name pattern : XML_{date}_{time}.xml
Move the processed XML files to a different folder.
Please advice.
Based on the requirements that you have mentioned, I would say that you can use best of both the worlds (T-SQL & SSIS).
I feel that T-SQL gives more flexibility in loading the XML data that you have described in the question.
There are lot of different ways you can achieve this. Here is one possible option:
Create a Stored Procedure that would take the path of the XML file as input parameter.
Perform your XML data load operation using the T-SQL way which you feel is easier.
Use SSIS package to perform error handling, file processing, archiving and send email.
Use logging feature available in SSIS. It just requires simple configuration. Here is a samples that show how to configure logging in SSIS How to track status of rows successfully processed or failed in SSIS data flow task?
A sample mock up of your flow would be as shown below in the screenshot. Loop the files using Foreach Loop container. Pass the file path as parameter to Execute SQL Task, which in turn would call the T-SQL that you had mentioned. After processing the file, using the File System Task to move the file to an archive folder.
Sample used in SSIS reading multiple xml files from folder
shows how to loop through files using Foreach loop container. It loops through xml files but uses Data Flow Task because the xml files are in simpler format.
Sample used in How to send the records from a table in an e-mail body using SSIS package? shows how to send e-mail using Send Mail Task.
Sample used in How do I move files to an archive folder after the files have been processed? shows how to move files to an Archive folder.
Sample used in Branching after a file system task in SSIS without failing the package shows how to continue package execution even after a particular task fails. This will help you to proceed with package execution even if Foreach Loop fails so you can send email. Blue arrow in the screenshot indicates on completion of previous task.
Sample used in How do I pick the most recently created folder using Foreach loop container in SSIS package? shows how to perform pattern matching.
Hope that gives you an idea.