Hope you can help me.
I am looking to retrieve multiple file-names into a column from files I wish to upload onto SQL in a custom column on SSIS.
I am aware of using Advanced Editor on the Flat File Source in Component Properties > FileNameColumnName. However, I am not sure how to make sure it picks up all the file names or what to enter in this field?
I can upload the files and all the data it holds but not the filename into a column.
Related
I have a data source which is something like
select
patient_id
from patient_table
the destination is a CSV file.
Now I want to add patient_name to both the source and the destination.
I go to the source and I change the query to
select
patient_id,
patient_name
from patient_table
After I add this when I click on columns the patient_name column is not there.
The same thing happens for my destination. I have a flat file destination with the patient_id column so I add the patient_name column to the actual .csv file and that column is not reflected on the flat file connection manager.
The only way that I've been able to get these new columns to show up is to delete the data flow task, connection managers, sources and destinations and to create everything new from scratch.
Is there any other way to do this?
I just created a simple data flow with an OLE DB source and a flat file destination. After adding a second column to the OLE DB source I double clicked my flat file destination which opened the Flat File Destination Editor. Clicking UPDATE added the 2nd column to the flat file connection.
Are you using the latest tooling available to modify SSIS packages?
I don't have an SSDT installation handy right now, so I'll do the best I can without screenshots (and working from memory).
In the Source object, after you add the column to the text of your query, click on Columns, which you already know. Your new column doesn't show up in the list at the bottom yet, which you also know. Up in the top of that window, there's a grid representation of the result set from the query. Find your new column in that grid and check the box to tell the connector you want that column to enter the data flow.
Now go to the connection manager for the .csv file. Add the new column there.
Once it's in the connection manager, now you should be able to map it in the destination object.
There's a possibility that you'll have to click on the arrow or arrows in your data flow task and map the new column in those, too, but it doesn't always happen like that. I haven't taken the time to figure out why that's necessary sometimes and not others, but you'll know right away because the arrows will have red Xs on them.
And that should get you there.
I have 4 different text files each file with different name and different column in it place in one folder. I want these 4 four files to be inserted or updated into 4 different existing tables. So How to read read these 4 files dynamically and insert them into their respective table dynamically in SSIS.
Well, you need to use Data Flow Task to move data from a Flat File Source to a Table Destination (OLEDB Destination perhaps). Are the columns in your file delimited in any way? For example, with any of these: (;),(|) or something like that? if it is, you can create a FlatFileConnectionManager and set that to split the columns. If not, you might need to use the FixedWidth option to separate your columns. To use the OLEDB Destination, you will need to create a OLEDB connectionManager to point to the table in your database. I could help you more if I had more information about the files you want to read the data from.
EDIT
Well you said at the start you were working with 4 files and 4 tables, so you can create 4 Flat Destination sourcers with 4 OLEDB destinations aswell (1 of each for each flat file). If I understood you correctly, these 4 files can or cannot exist yet. So if you know the names that the files will get, change the Package Property DelayValidation to true, and then create a connection with a sample text file. You do this so the File path gets saved. The tables, in my opinion DO need to exist. Now, when you said:
i want to load all the text files into each different existing table whenever there is files inside the folder.
The only way I know you can do something similar, is to schedule the execution of your package at a certain time with SQL Server Agent Job. Please let me know if this was what you were looking for.
I want to use a format file to help import a comma delimited file using bulk insert. I want to know how you generate format files from a flat file source. The microsoft guidance on this subjects makes it seem as though you can only generate a format file from a SQL table. But I want it to look at text file and tell me what the delimiters are in that file.
Surely this is possible.
Thanks
The format file can, and usually does include more than just delimiters. It also frequently includes column data types, which is why it can only be automatically generated from the Table or view the data is being retrieved from.
If you need to find the delimiters in a flat file, I'm sure there are a number of ways to create a script that could accomplish that, as well as creating a format file.
I am having some difficulties on how to dynamically create a metadata, which need to be extracted from the header line of a flat .csv file in CC.
Usually, I manually define the metadata by select New Metadata --> Extract from flat file in CC. However the metadata of the file may changes with additional columns. Thus, I do not know the metadata of the file and I can not define it in this static approach.
It would be helpful if you could suggest a solution to create metadata dynamically and using this newly created metadata for connecting to other components. Perhaps an example graph file for demonstration would be great.
Thanks,
Andy
I have discovered this kind of solution.
You just have to fill in flat .csv filename into csv readers and writers.
MetaDataMaster.grf - runs the graphs below.
MetaDataCreator.grf - creates metadata according to csv header and
write it into meta_example.fmt file
MetaDataUser.grf - Reads csv according to created meta_example.fmt file - you can add there a reformat and use just some predefined fields.
You can run the 2nd and 3rd graph separately to test it.
I'm trying to read multiple XML files from a folder, to compile all the data they have (all of them have the same XML structure), and than save that data in a CSV file.
I already have a 'read-files' Transformation with the steps: Get File Names and Copy Rows to Result, to get all the XML files. (it's working - I print a file with all the files names)
Then, I enter in a 'for-each-file' Job which has a Transformation with the Get Rows from Result Step, and then another Job to process those files.
I think I'm loosing information from the 'read-files' Transformation to the Transformation in the 'for-each-file' Job which Get all the rows. (I print another file with all the files names, but it is empty)
Can you tell me if I'm thinking in the right way? I have to set some variables, or some option that is disabled? Thanks.
Here is an example of "How to process a Kettle transformation once per filename"
http://www.timbert.net/doku.php?id=techie:kettle:jobs:processtransonceperfile