I am working on a transformation that gets data from a database table. After the Table Input step I would like to rename the primary key field to 'primaryKey'. The only way to get the name of the primary key field is through a parameter ${primaryKeyName}. How can I rename the field saved in the parameter ${primaryKeyName} to primaryKey?
I tried a Select Value step but it would use ${primaryKeyName} literally instead of getting the value of ${primaryKeyName}.
You cannot use variables as field names to rename them unless:
option a) write your own User defined Java Class, fetch the variable name, and rename the field directly in Java;
option b) Use Metadata Injection to manipulate a template transformation: use a select values step in the inner/template transformation and have the outer transformation get the variable's value and use it to inject metadata into the Select values step
option c) it seems obvious, but can't you use an alias on your SQL query?
Related
I've multiple csv files and multiple tables.
The table name is file name and column name is first row of csv file.
Now I want to add default value of empty string to the sink table.
Consider my scenario,
employee:
id int, name varchar, is_active bit NULL
employee.csv:
id|name|is_active
1|raja|
Now I'm trying to copy the csv data to PostgreSQL table its throwing error.
Expected result is default value if its empty value.
You can use NULLIF in PostgreSQL:
NULLIF(argument_1,argument_2);
The NULLIF function returns a null value if argument_1 equals to argument_2, otherwise it returns argument_1.
This way you can replace NULL value with some other value
If your error is related to Type mismatch then consider typecasting the column first
Thanks!
As per the issue, tried to repro the scenario and here is the following outcome which was successfully copied. You have to use
Source Dataset: employee.csv from Azure Blob Storage
Sink Dataset : Here, I have used the sink as Azure SQL DB for some limitations but as you have used PostgreSQL is almost similar.
Copy Activity Settings:
Under the mapping settings there will be type conversion, where you have to import schema else you can dynamically add
Output:
Alternative to use DataFlow - if you have multiple data fields, you need to use the derived column transformation to generate new columns in your data flow or to modify existing fields.
For more details, refer Derived column transformation in mapping data flow.
You can even refer to this Microsoft Q&A post for more insights: Copy Task failure because of conversion failure
Will come directly to the question.
Have 2 parameter like filename and table name. The requirement is to upload the data from the excel sheet to the database table enter in the other parameter. This should be in run time. No hardcoding of field names and that program should be flexible enough to suite any table. Please help.
I can think of two possible approaches:
Dynamic code generation -- write a program which writes a program
Use dynamic type tools
For 1. try googling
For 2. see https://wiki.scn.sap.com/wiki/display/Snippets/Example+-+create+a+dynamic+internal+table - this wiki shows a way (not sure if it is overkill as it creates the type from scratch whereas any table in your SAP system is already a defined type in the Data Dictionary).
You can do easily reference a parameterised table in Open SQL e.g. MODIFY (p_tab) ...
Perhaps you could do a generic SPLIT of a line read in from file by the delimiter into a table of fields - you can then use ASSIGN COMPONENT to match the fields you have read in to the fields in your internal type.
If you are doing this I think a white list of allowed tables would be wise - and auth checks. Otherwise someone could upload SAP standard tables with no authorisation.
I have a series of task that are very similar:
SELECT a,b FROM c
Lookup in another table and change value in column b.
Save new value back to c and if not match, send the result on to an error table.
That part is pretty straight forward and illustrated here:
Source ==> Lookup =match=> SQL Update command
=No match=> SQL Save Error command
(Hope you understand what I mean - but it works!)
I now have to repeat this a number of times, where my source-sql changes. So what I want to do is to insert a Script Component in front of the Source and set my User::Sql variable like:
Variables.Sql = "SELECT d, e FROM f"
All of the above is contained in a Data Flow. When I have created one I can then copy that one and only change the Sql variable in the script and then it should all work.
My problem is: When I insert the Script Command it asks me if it is a Source, Destination or Transscript script. And by only setting the variable it does not produce any rows for output and cannot connect to my Source.
Anyone know how to make that work?
(I have simplified the above. I actually want to update multiple variables and use those in my Source, Lookup and Error update as well - therefore it is not more simple just to change the SQL script in the initial Source! But being able to do the above, I will be able to achieve what I want :-))
You should set your variable containing the SQL query in the control flow, before you execute the dataflow.
Then you need to use that variable as an expression in your Dataflow. You can parametrize the query used in the lookup or any other parameters of your dataflow.
If your dataflows really have always the same structure, you could even generate a list of queries and call your dataflow task in a loop, preventing the duplication of the same tasks.
I have Access as my back end. Student is the database name and its path is c:\Project\student.mdb
And am having few tables in it. When i click a command button in vb.net a new table must be created in the path mentioned. And it should have the fields I declare.
What s the query I need to use ? Is it possible to do like that ?
Yes, it's possible, you can check out the syntax for the create table command.
However, I have to say that creating tables dynamically suggests a bad database design. The database layout should normally remain the same whatever data you put in the database.
If you create tables dynamically, that means that you have data in the table names, but data should go inside the tables, not in table names (or even field names).
Yes it's possible . Just pass the create table command as query and the table name as variable , which you want to add dynamically. for reference of create table command
Create table command
I want to executing a select statement in SSIS, but this select statement takes a parameter from another component, and the column of this select statement must be used as inputs to other components.
For example:
select id from myTable where name = (column from a previous component).
And the "id" content of the above select statement should be a column that future components can use.
If i add an "OLE DB Command" component, it allows me to refer to other components as inputs, but I cannot generate an output from it. It seems OLE DB Command component is only used for update/insert statements?
Any ideas on how to do it?
Actually, this is a case for Lookup. It seems you want to do a lookup by name and return id. Pretty simple. Here's how I created an example of this:
Drag a Data Flow Task onto the design surface. Double-click it to switch to it.
Create a Connection Manager for my database
Drag onto the design surface:
an OLE DB Source
A Lookup Transform
An OLE DB Destination
Connect the Source to the Lookup to the Destination. It's the "Lookup Match Output" we want going to the destination. See figure 1.
Configure the source. My source table just had id and name columns.
Configure the lookup
General tab: Use an OLE DB Connection
Connection tab: specify the same connection, but use the Lookup table. My lookup table was just id and name, but name was made unique, so it makes better sense as a lookup column.
On the columns tab, configure name to map to name, with "id" as an output. Configure the lookup operation to be "add new column", and name that column "lookupId". See figure 2.
Ignore the other two tabs
Configure the output to take all three columns. See figure 3.
That's all. For each row from the source, the name column will be used to match the name column of the lookup table. Each match will contribute its id column as the new lookupId column. All three columns will proceed to the destination.
Figure 1:
Figure 2:
Figure 3: