I have a table of 811 records. I want to get five records at a time and assign it to variable. Next time when I run the foreach loop task in SSIS, it will loop another five records and overwrite the variable. I have tried doing with cursor but couldn't find the solution. Any help will be highly appreciated. I have table like this for e.g.
ServerId ServerName
1 Abc11
2 Cde22
3 Fgh33
4 Ijk44
5 Lmn55
6 Opq66
7 Rst77
. .
. .
. .
I want query should take first five names as follows and assign it to variable
ServerId ServerName
1 Abc11
2 Cde22
3 Fgh33
4 Ijk44
5 Lmn55
Then next loop takes another five name and overwrite the variable value and so on till the last record is consumed.
Taking ltn's answer into consideration this is how you can achieve limiting the rows in SSIS.
The Design will look like
Step 1 : Create the variables
Name DataType
Count int
Initial int
Final int
Step 2 : For the 1st Execute SQL Task write the sql to store the count
Select count(*) from YourTable
In the General tab of this task Select the ResultSet as Single Row.
In the ResultSet tab map the result to the variable
ResultName VariableName
0 User::Count
Step 3 : In the For Loop container enter the expression as shown below
Step 4 : Inside the For Loop drag an Execute SQL Task and write the expression
In Parameter Mapping map the initial variable
VariableName Direction DataType ParameterName ParameterSize
User::Initial Input NUMERIC 0 -1
Result Set tab
Result Name Variable Name
0 User::Final
Inside the DFT u can write the sqL to get the particular rows
Click on Parameters and select the variable INITIAL and FINAL
if your data will not be update between paging cycles and the sort order is always the same then you could try an approach similiar to:
CREATE PROCEDURE TEST
(
#StartNumber INT,
#TakeNumber INT
)
AS
SELECT TOP(#TakeNumber)
*
FROM(
SELECT
RowNumber=ROW_NUMBER() OVER(ORDER BY IDField DESC),
NameField
FROM
TableName
)AS X
WHERE RowNumber>=#StartNumber
Related
I have a table of 900 records.
I want to get 10 records at a time and assign it to variable.
Next time when I run the for each loop task in SSIS,
it will loop another 10 records and overwrite the variable.
Any help will be highly appreciated.
I have table like this for e.g
EMPID
0001
00045
00067
00556
00078
00345
00002
00004
00005
00006
00007
00008
this is want I have tried execute sql task to pull 900 records to variable, connect Execute sql task to For each loop, inside for each loop have Data flow task, the source has sql query and destination is table.
select * from Dbo.JPKGD0__STP
where EMPID in ?
but this will pass each empid in 1 loop , so i wanted to pass 10 empids each time.
Please let me know if I need to use different approach/or other tasks to achieve this.
Step (1) - Create variables
You have to create two variables of type int:
#[User::RowCount] >> type int
#[User::Counter] >> type int
#[User::strQuery] >> type string
Assign the following expression to #[User::strQuery]:
"SELECT EMPID
FROM Dbo.JPKGD0__STP
ORDER BY EMPIDASC
OFFSET " + (DT_WSTR,50)#[User::Counter] + " ROWS
FETCH NEXT 10 ROWS ONLY "
Step (2) - Get Row Count
First, add an Execute SQL Task with the following command:
SELECT Count(*) FROM Dbo.JPKGD0__STP;
And store the result in #[User::RowCount] variable (check this link for more information).
Step (3) - For Loop Container
Now, Add a For Loop Container with the following expressions:
InitExpression: #[User::Counter] = 0
EvalExpression: #[User::Counter] < #[User::RowCount]
AssignExpression: #[User::Counter] = #[User::Counter] + 10
Inside the For loop container, add a Data flow task, with an OLE DB Source and a destination. In the OLE DB Source, select the Access Mode as SQL Command from variable and select #[User::strQuery] as a source.
References
Row Offset in SQL Server
SQL Server OFFSET FETCH
SSIS Basics: Using the Execute SQL Task to Generate Result Sets
ORDER BY Clause (Transact-SQL)
In a project we have several SSIS packages (around 200), all the package names are stored in a control table. We need to create a master package which can run all the 200 packages.
Since the max concurrent executable setting was set to 8. So planning to create 8 execute package tasks in a container and was thinking of generating the connection string(Execute package task- File connection String) dynamically using the package names stored in the table.
The control table is in the below format
Id PackageName
---------------
1 Package1
2 Package2
Ideas on how should be implemented helps.
I covered this pattern on https://stackoverflow.com/a/34868545/181965 but you're looking for a package that looks something like this
A sequence container that contains everything that one of those 8 discrete buckets of work would require. In your case, a Variable for
CurrentPackage String
rsObject Object
ContainerId Int32
The containerId will be the values 0 through 7 (since you have 8 buckets of work). As outlined in the other answer, we must scope the variables to the Sequence Container. The default in 2012+ is to create them at the Control Flow level, whereas 2005/2008 would create them at the level of the selected object.
Set up
I created a table and loaded it with 200 rows
CREATE TABLE dbo.so_35415549
(
id int IDENTITY(1,1) NOT NULL
, PackageName sysname
);
INSERT INTO
dbo.so_35415549
(
PackageName
)
SELECT TOP 200
'Package' + CAST(ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) AS varchar(3))
FROM
sys.all_columns AS AC;
Get My Bucket's data
The modulus, modulo, mod whatever you call it operator is our friend here. The mod operator will return the remainder after division. e.g. 10 mod 3 is 1 because 3*3 + 1 = 10
In your case, you'll be modding via 8 so you know the remainder will be bounded between 0 and 7.
SQL Server implements the mod operator as % and you can test the correctness via the following query
SELECT
S.id
, S.PackageName
, S.id % 8 AS ModValue
FROM
dbo.so_35415549 AS S
ORDER BY
1;
Sample output
id PackageName ModValue
1 Package1 1
2 Package2 2
3 Package3 3
4 Package4 4
5 Package5 5
6 Package6 6
7 Package7 7
8 Package8 0
9 Package9 1
10 Package10 2
...
199 Package199 7
200 Package200 0
SQL Get Work List
Using the above query as a template, we will use the following query. Notice the ? in there. That is the placeholder for an Execute SQL Tasks parameterization for an OLE DB Connection Manager.
SELECT
S.PackageName
FROM
dbo.so_35415549 AS S
WHERE
S.id % 8 = ?
ORDER BY
1;
The Parameter we pass in will be #[User::ContainerId]
The Result Set option will be updated from None to Full ResultSet and we push the value into rsObject
FELC Shred Work List
This is a standard shredding of a recordset. We got our variable populated in the previous step so let's enumerate through the results. There will be one column in our result set and you will map that to User::CurrentPackageName
EPT Run Package
This is your Execute Package Task. Use the value of CurrentPackageName and you're set.
I just started out using Informatica and currently I am figuring out how to get this to a target output (flat file to Microsoft SSIS):
ID Letter Parent_ID
---- ------ ---------
1 A NULL
2 B 1
3 C 1
4 D 2
5 E 2
6 F 3
7 G 3
8 H 4
9 I 4
From (assuming that this is a comma-delimited flat file):
c1,c2,c3,c4
A,B,D,H
A,B,D,I
A,B,E
A,C,F
A,C,G
EDIT: Where c1 c2 c3 and c4 being a header.
EDIT: A more descriptive representation of what I want to acheive:
EDIT: Here is what I have so far (Normalizer for achieving the letter column and Sequence Generator for ID)
Thanks in advance.
I'd go with a two-phased approach. Here's the general idea (not a full, step-by-step solution).
Perform pivot to get all values in separate rows (eg. from "A,B,D,H" do a substring and union the data to get four rows)
Perform sort with distinct and insert into target to get IDs assigned. End of mapping one.
In mapping two add a Sequence to add row numbers
Do the pivot again
Use expression variable to refer previous row and previous RowID (How do I get previous row?)
If current RowID doesn't match previous RowID, this is a top node and has no parent.
If previous row exists and the RowID is matching, previous row is a parent. Perform a lookup to get it's ID from DB and use as Parent_ID. Send update to DB.
I am writing some sort of resources management system.
A resource is an instance of a definition. A definition is the metadata, basically it contains the properties.
This is in general my DB:
TypeDefinition
id name
===============
1 CPU
PropertyDefinition
id name typeDefinitionId valueType
================================================
1 frequency 1 int
2 status 1 string
TypeInstance
id name typeDefinitionId
=================================
1 CPU#1 1
2 CPU#2 1
PropertyInstanceValue
id propertyDefinitionId typeInstanceId valueType intValue StringValue FloatValue
========================================================================================
1 1 1 int 10
2 2 1 string Pending
3 1 2 int 20
4 2 2 string Approved
REQUIREMENT:
order all resources according to a specific property value.
For example: order all resources according to their status --> Meaning CPU#2 will appear before CPU#1 because “Approved” is before “Pending”.
If we were to order according to frequency, CPU#1 will appear before CPU#2 because 10 is before 20.
So I need to sort each time according to a different column (intValue / stringValue/ FloatValue / etc), depending on the property's valueType.
Any suggestion?
LIMITATION:
PIVOT is currently the only option we've thought of, but it's not really possible since the DB is huge and I need the query to be as fast as possible.
Thanks a lot in advance,
Michal.
If the problem is that you don't want to dynamically build the query then use this order by structure:
order by case #orderby
when 'status' then status
when 'frequency' then frequency
end
option (recompile)
You will pass the #orderby parameter. The final recompile option is to force the engine to build a new plan according to the passed parameters, that is, assuming you are using a stored procedure.
If you want to order your query results using SQL (as opposed to sorting them in the calling application after results are returned) you would need to generate Dynamic SQL and execute it using sp_executesql.
http://msdn.microsoft.com/en-us/library/ms188001.aspx
If I understand your question correctly, I would approach it as follows:
Create a table of allowed string values, making sure to include a column for specifying sorting precedence (let's call this AllowedValues)
create table [dbo].[AllowedStringValues] ( PropertyDefinitionId int, stringValue varchar(250), sortOrder int)
Create a complex subquery that selects out the proper value based on the row's PropertyDefinition (it looks like it needs to look among 3 columns depending on what datatype it is).
If the value is a string type, inner join the subquery with the AllowedStringValues table
(assuming value was stored into a value column)
inner join AllowedStringValues on ValueType=String AND value = stringValue OR ValueType <> string
Sort by the sort priority in AllowedValues if it is a string, or by the numeric value otherwise.
order by case ValueType
when 'string' then sortOrder
else value
end
I Have an SQL query giving me X results, I want the query output to have a coulmn called
count making the query somthing like this:
count id section
1 15 7
2 3 2
3 54 1
4 7 4
How can I make this happen?
So in your example, "count" is the derived sequence number? I don't see what pattern is used to determine the count must be 1 for id=15 and 2 for id=3.
count id section
1 15 7
2 3 2
3 54 1
4 7 4
If id contained unique values, and you order by id you could have this:
count id section
1 3 2
2 7 4
3 15 7
4 54 1
Looks to me like mikeY's DSum approach could work. Or you could use a different approach to a ranking query as Allen Browne described at this page
Edit: You could use DCount instead of DSum. I don't know how the speed would compare between the two, but DCount avoids creating a field in the table simply to store a 1 for each row.
DCount("*","YourTableName","id<=" & [id]) AS counter
Whether you go with DCount or DSum, the counter values can include duplicates if the id values are not unique. If id is a primary key, no worries.
I frankly don't understand what it is you want, but if all you want is a sequence number displayed on your form, you can use a control bound to the form's CurrentRecord property. A control with the ControlSource =CurrentRecord will have an always-accurate "record number" that is in sequence, and that will update when the form's Recordsource changes (which may or may not be desirable).
You can then use that number to navigate around the form, if you like.
But this may not be anything like what you're looking for -- I simply can't tell from the question you've posted and the "clarifications" in comments.
The only trick I have seen is if you have a sequential id field, you can create a new field in which the value for each record is 1. Then you do a running sum of that field.
Add to your query
DSum("[New field with 1 in it]","[Table Name]","[ID field]<=" & [ID Field])
as counterthing
That should produce a sequential count in Access which is what I think you want.
HTH.
(Stolen from Rob Mills here:
http://www.access-programmers.co.uk/forums/showthread.php?p=160386)
Alright, I guess this comes close enough to constitute an answer: the following link specifies two approaches: http://www.techrepublic.com/blog/microsoft-office/an-access-query-that-returns-every-nth-record/
The first approach assumes that you have an ID value and uses DCount (similar to #mikeY's solution).
The second approach assumes you're OK creating a VBA function that will run once for EACH record in the recordset, and will need to be manually reset (with some VBA) every time you want to run the count - because it uses a "static" value to run its counter.
As long as you have reasonable numbers (hundreds, not thousands) or records, the second approach looks like the easiest/most powerful to me.
This function can be called from each record if available from a module.
Example: incrementingCounterTimeFlaged(10,[anyField]) should provide your query rows an int incrementing from 0.
'provides incrementing int values 0 to n
'resets to 0 some seconds after first call
Function incrementingCounterTimeFlaged(resetAfterSeconds As Integer,anyfield as variant) As Integer
Static resetAt As Date
Static i As Integer
'if reset date < now() set the flag and return 0
If DateDiff("s", resetAt, Now()) > 0 Then
resetAt = DateAdd("s", resetAfterSeconds, Now())
i = 0
incrementingCounterTimeFlaged = i
'if reset date > now increments and returns
Else
i = i + 1
incrementingCounterTimeFlaged = i
End If
End Function
autoincrement in SQL
SELECT (Select COUNT(*) FROM table A where A.id<=b.id),B.id,B.Section FROM table AS B ORDER BY B.ID Asc
You can use ROW_NUMBER() which is in SQL Server 2008
SELECT ROW_NUMBER() OVER (ORDER By ID DESC) RowNum,
ID,
Section
FROM myTable
Then RowNum displays sequence of row numbers.