SSIS filename - file count - sql

I'm currently creating a flat file export for one of our clients, i've managed to get the file in the format they want, i'm trying to get the easiest way of creating a dynamic file name. I've got the date in as a variable and the path ect but they want a count in the file name. For example
File name 1 : TDY_11-02-2013_{1}_T1.txt. The {} being the count. So next weeks file would be TDY_17-02-2013_{2}_T1.txt
I cant see an easy way of doing this!! any idea's??

EDIT:
on my first answer, I thought you meant count of values returned on a query. My bad!
two ways to achieve this, you could loop into the destination folder, select the last file by date, get its value and increase 1, which sound like a lot of trouble. Why not a simple log table on the DB with last execution date and ID and then you compose your file name base on the last row of this table?
where exactly is your problem?
you can make a dynamic file name using expressions:
the count, you can use a "row count" component inside your data flow to assign the result to a variable and use the variable on your expression:

Use Script task and get the number inside the curly braces of the file name and store it in a variable.
Create a variable(FileNo of type int) which stores the number for the file
Pseudo code
string name = string.Empty;
string loction = #"D:\";
/* Get the path from the connection manager like the code below
instead of hard coding like D: above
string flatFileConn =
(string(Dts.Connections["Yourfile"].AcquireConnection(null) as String);
*/
string pattern = string.Empty;
int number = 0;
string pattern = #"{([0-9])}"; // Not sure about the correct regular expression to retrieve the number inside braces
foreach (string s in Directory.GetFiles(loction,"*.txt"))
{
name = Path.GetFileNameWithoutExtension(s);
Match match = Regex.Match(name, pattern );
if (match.Success)
{
dts.Variables["User::FileNo"].Value = int.Parse(match.Value)+1;
}
}
Now once you get the value use it in your file expression in the connection manager
#[User::FilePath] +#[User::FileName]
+"_{"+ (DT_STR,10,1252) #[User::FileNo] + "}T1.txt"

Related

Using a variable content to generate a variable name

I have a script for a game server, and I'm stuck in some shit that look easy to solve
Exists a variable which receive content dynamically based on user action, so we will name this variable: example, and attrib some random value
local example = Potato
Then I have a function which sends a message to a discord webhook
SendWebhookMessage(varNAME, "Message content")
Where varname is the variable containing the link of the webhook.
I want to use the content of variable example to generate the variable name like
webhook_ds_example
So in this case it will be
webhook_ds_potato
Hope you guys could understand and help to solve
local menu = { name = "Baú" }
local cb_take = function(idname)
local citem = chest.items[idname]
local amount = vRP.prompt(source,"Quantidade:","")
amount = parseInt(amount)
if amount >= 0 and amount <= citem.amount then
local new_weight = vRP.getInventoryWeight(user_id)+vRP.getItemWeight(idname)*amount
if new_weight <= vRP.getInventoryMaxWeight(user_id) then
citem.amount = citem.amount - amount
local temp = os.date("%x %X")
vRP.logs("savedata/bau.txt","Bau: "..name.." [ID]: "..user_id.." /"..temp.." [FUNÇÃO]: Retirar / [ITEM]: "..idname.." / [QTD]: "..amount)
local webhook_bau_fac1 = ""
local webhook_bau_fac2 = ""
local webhook_bau_fac3 = ""
local webhook_bau_fac4 = ""
SendWebhookMessage(webhook_bau_..name,"```prolog\n[ID]: "..user_id.." "..identity.name.." "..identity.firstname.." \n[GUARDOU]: "..vRP.format(parseInt(amount)).." "..vRP.itemNameList(itemName).." \n[BAU]: "..chestName.." "..os.date("\n[Data]: %d/%m/%Y [Hora]: %H:%M:%S").." \r```")
You are looking for tables, which let you store lots of different named values in one variable.
local webhook_bau -- make a variable
-- create a table with 4 entries, put it in the variable
webhook_bau = {fac1="", fac2="", fac3="", fac4=""}
-- if you want to start with an empty table, use {} instead
-- change one of them based on the name
webhook_bau[name] = "something"
-- use one of the entries based on the name
SendWebhookMessage(webhook_bau[name], "whatever you want to send")
Table entries "magically" appear when you use them, you don't have to create them first. If you access an entry that doesn't exist, you will read the value nil. You can also delete an entry by putting nil in the entry.
In the example you provided, you have:
SendWebhookMessage(webhook_bau_..name,"```prolog\n[ID]: "..user_id.." "..identity.name.." "..identity.firstname.." \n[GUARDOU]: "..vRP.format(parseInt(amount)).." "..vRP.itemNameList(itemName).." \n[BAU]: "..chestName.." "..os.date("\n[Data]: %d/%m/%Y [Hora]: %H:%M:%S").." \r```")
You are missing quotes around webhook_bau_ which would result in an error trying to concatenate a nil variable.
I'm also not seeing where name is set (only menu.name), so I'm assuming you have that elsewhere in your program, if not, that will also be nil, so just make sure that is set somewhere as well.

How to get the reorder the column with csv input fixed column in pentaho

Scenario:
I have created transformation to load data into table from csv file and I have following columns in csv file:
Customer_Id
Company_Id
Employee_Name
But user may give input file with column ordering (random order) as
Employee_Name
Company_Id
Customer_Id
so, if I try to load file which has random column ordering, will kettle load correct column values as per column names ... ?
Using ETL Metadata Injection you can use a transformation like this, to either normalize the data, or to store it to your database:
Then you just need to send the correct data to that transformation. You can read the header line from the CSV, and use Row Normaliser to convert to the format used by ETL Metadata Injection.
I have included a quick example here: csv_inject on Dropbox, if you make something like this and run it from something that runs it per csv file it should work.
Ooh, thats some nasty javascript!
The way to do this is with metadata injection. Look at the samples, but basically you need a template which reads the file, and writes it back out. you then use another parent transformation to figure out the headings, configure that template and then execute it.
There are samples in the PDI samples folder, and also take a look at the "figuring out file format" example in matt casters blueprints project on github.
You could try something like this as your JavaScript:
//Script here
var seen;
trans_Status = CONTINUE_TRANSFORMATION;
var col_names = ['Customer_Id','Company_Id','Employee_Name'];
var col_pos;
if (!seen) {
// First line
trans_Status = SKIP_TRANSFORMATION;
seen = 1;
col_pos = [-1,-1,-1];
for (var i = 0; i < col_names.length; i++) {
for (var j = 0; j < row.length; j++) {
if (row[j] == col_names[i]) {
col_pos[i] = j;
break;
}
}
if (col_pos[i] === -1) {
writeToLog("e", "Cannot find " + col_names[i]);
trans_Status = ERROR_TRANSFORMATION;
break;
}
}
}
var Customer_Id = row[col_pos[0]];
var Company_Id = row[col_pos[1]];
var Employee_Name = row[col_pos[2]];
Here is the .ktr I tried: csv_reorder.ktr
(edit, here are the test csv files)
1.csv:
Customer_Id,Company_Id,Employee_Name
cust1,comp1,emp1
2.csv:
Employee_Name,Company_Id,Customer_Id
emp2,comp2,cust2
Assuming rejecting the input file is not an option you basically have 4 solutions.
reorder the fields in an external editor (don't use excel if it contains dates)
Use code within your transformation to detect the column headers and reorder the file.
Use metadata injection as proposed by bolav
Create a job. This need to:
a. load the file into a temporary database.
b. use an sql statement to retrieve the fields (use a SELECT with an ORDER By clause)
c. output the file in the correct order

Zoho Creator making a custom function for a report

Trying to wrap my head around zoho creator, its not as simple as they make it out to be for building apps… I have an inventory database, and i have four fields that I call to fill a field called Inventory Number (Inv_Num1) –
First Name (First_Name)
Last Name (Last_Name)
Year (Year)
Number (Number)
I have a Custom Function script that I call through a Custom Action in the form report. What I am trying to do is upload a CSV file with 900 entries. Of course, not all of those have those values (first/last/number) so I need to bulk edit all of them. However when I do the bulk edit, the Inv_Num1 field is not updated with the new values. I use the custom action to populate the Inv_Num1 field with the values of the other 4 fields.
Heres is my script:
void onetime.UpdateInv()
{
for each Inventory_Record in Management
{
FN = Inventory_Record.First_Name.subString(0,1);
LN = Inventory_Record.Last_Name.subString(0,1);
YR = Inventory_Record.Year.subString(2,4);
NO = Inventory_Record.Number;
outputstr = FN + LN + YR + NO;
Inventory_Record.Inv_Num1 = outputstr;
}
}
I get this error back when I try to run this function
Error.
Error in executing UpdateInv workflow.
Error in executing For Each Record task.
Error in executing Set Variable task. Unable to update template variable FN.
Error evaluating STRING expression :
Even though there is a First Name for example, it still thinks there is none. This only happens on the fields I changed with Bulk Edit. If I do each one by hand, then the custom action works—but of course then the Inv_Num1 is already updated through my edit on success functions and makes the whole thing moot.
this may be one year late, you might have found the solution but just to highlight, the error u were facing was just due to the null value in first name.
you just have put a null check on each field and u r good to go.
you can generate the inv_number on the time of bulk uploading also by adding null check in the same code on and placing the code on Add> On Submt.( just the part inside the loop )
the Better option would be using a formula field, you just have to put this formula in that formula field and you'll get your inventory_number autogenerated , you can rename the formula_field to Inv Number or whaterver u want.
Since you are using substring directly in year Field, I am assuming the
year field as string.else you would have to user Year.tostring().substring(2,4) & instead of if(Year=="","",...) you have to put if(Year==null , null,...);
so here's the formula
if(First_Name=="","",First_Name.subString(0,1))+if(Last_Name =="","",Last_Name.subString(0,1)) + if(Year=="","",Year.subString(2,4)+Number
Let me know ur response if u implement this.
Without knowing the datatype its difficult to fix, but making the assumption that your Inventory_Record.number is a numeric data item you are adding a string to a number:
The "+" is used for string Concatenation - Joiner but it also adds two numbers together so think "a" + "b" = "ab" for strings but for numbers 1 + 2 = 3.
All good, but when you do "a" + 2 the system doesn't know whether to add or concatenate so gives an error.

How to use Substring in SSIS

I want to export data from SharePoint list to SQL using SSIS.
In SharePoint list, i have a column as multi select, So i am getting below value in my column
1;#control 1;#3;#control 3
I want to use substring in derived column in such a way that i should get below result
1,3
I want only ID from the given column.
I have tried below code
SUBSTRING(ColumnName,1,FINDSTRING(ColumnName,";#",1) - 1)
But it only gives me answer as
1
Can anyone please help me out.?
Because there is an unknown number of controls selected in your SharePoint Multi-Select, a Derived Column transformation is not going to work for you. You'll have to use a script.
One way to parse your string is with regular expressions. You'll have to add an output to the script transformation and assign your parsed string to that output.
Regex controlExpression = new Regex(#"control ([0-9]+)");
MatchCollection controlMatches = controlExpression.Matches(--YOUR INPUT HERE--);
String output = string.Join(",",
(controlMatches.Cast<Match>().Select(n => n.Groups[1].ToString())).ToArray());

SQL: Use a predefined list in the where clause

Here is an example of what I am trying to do:
def famlist = selection.getUnique('Family_code')
... Where “””...
and testedWaferPass.family_code in $famlist
“””...
famlist is a list of objects
‘selection’ will change every run, so the list is always changing.
I want to return only columns from my SQL search where the row is found in the list that I have created.
I realize it is supposed to look like: in ('foo','bar')
But no matter what I do, my list will not get like that. So I have to turn my list into a string?
('\${famlist.join("', '")}')
Ive tried the above, idk. Wasn’t working for me. Just thought I would throw that in there. Would love some suggestions. Thanks.
I am willing to bet there is a Groovier way to implement this than shown below - but this works. Here's the important part of my sample script. nameList original contains the string names. Need to quote each entry in the list, then string the [ and ] from the toString result. I tried passing as prepared statement but for that you need to dynamically create the string for the ? for each element in the list. This quick-hack doesn't use a prepared statement.
def nameList = ['Reports', 'Customer', 'Associates']
def nameListString = nameList.collect{"'${it}'"}.toString().substring(1)
nameListString = nameListString.substring(0, nameListString.length()-1)
String stmt = "select * from action_group_i18n where name in ( $nameListString)"
db.eachRow( stmt ) { row ->
println "$row.action_group_id, $row.language, $row.name"
}
Hope this helps!