I'm trying to copy values from one ID to another ID.
The ID and the timestamp are primary keys.
String sqlString = "SELECT * FROM Values WHERE ID = #ID";
var sqlCommand = new SqlCommand(sqlString, sqlConnect);
sqlCommand.Parameters.Add("#ID", SqlDbType.UniqueIdentifier).Value = ID;
using (var reader = sqlCommand.ExecuteReader())
{
while (reader.Read())
{
time = reader["Time"];
dt = DateTime.Parse(time.ToString(), CultureInfo.InvariantCulture);
value = reader["Value"];
insertData(IdDestination, dt, value);
}
}
This function is working, but if I have 10.000 table rows, then it's very slow.
I was thinking about using SqlBulkCopy, but this doesn't work for me because I need to change the Destination ID. Another problem is that sometimes, the values I want to copy already exist, so I need a rollback or commit. Update is also not working because of losing important data.
Does anyone have an idea about what kind of batch processing would work for me?
I'd like to insert 1000 rows or more at the same time without having to call my insert function a thousand times.
Related
I was wondering if there's a way of updating a column inside a dataflow task by running a select on every row?
Here's the situation :
Let's take this as our start position. I collect info from 2 files, then I merge them, and I add a column with the derived column tool. Is there a way of populating this column by performing a select on every row using the values of the the row?
Ex :
SELECT Count(*) AS cnt
FROM TABLE T
WHERE T.COLUMN1 = ROW.COLUMN3
AND T.COLUMN2 = ROW.COLUMN5
I don't know if I'm just not phrasing my need properly but I couldn't get any results
Thank you
You should be able to do this with a Lookup Transformation.
EDIT based on comment:
If you don't want to use a lookup due to the size of the table, you can do exactly what you want with a Script Component. You can create and execute your SQL Command for each row of the dataflow just like you would in any .net application.
I was able to do it with a Script Component
1- I've removed the Derived Column
2- I've created a string variable where I stored the query whith a wildcard string to replace every value that I need to get from the row.
3- I've passed this variable allong with one containing the connecection string info to the Script Component
4- I've added a new column to the Output Columns of the Script Component
5- Added using System.Data.OleDb;
6- Created 2 variables :
string jourFerieQuery;
string dbcsoledbschema;
7- Updated the PostExecute() to put the values of my SSIS variables into the script variables :
public override void PostExecute()
{
base.PostExecute();
jourFerieQuery = Variables.jourFerieQuery;
dbcsoledbschema = Variables.dbcsoledbschema;
}
8- Added a method :
int GetData(string cs, string query)
{
OleDbConnection conn = new OleDbConnection(cs);
conn.Open();
OleDbCommand cmd = new OleDbCommand(query, conn);
DataTable dt = new DataTable();
dt.Load(cmd.ExecuteReader());
conn.Close();
return (int)dt.Rows[0][0];
}
9- Updated the Input0_ProcessInputRow(Input0Buffer Row) :
public override void Input0_ProcessInputRow(Input0Buffer Row)
{
string query = jourFerieQuery.Replace("[1]", Row.CODDEVI).Replace("[2]", Row.DATCRBEZEROCONGE.ToString());
Row.Keep = GetData(dbcsoledbschema, query);
}
My query returns a count that's why the method I've added returns an int
You can do that through a script component (transformation).
Add an output of Ct.
I use System.Data.OleDB as it matches the SSIS package connection string.
I have an application that's built in .NET language.
In this application we mainly read/write to the database (SQL Server 2005).
Sometimes (for a single input) I just use the SQL query, for example:
commandText = "INSERT INTO Test_Table (Number) VALUES ('10')";
command = new System.Data.SqlClient.SqlCommand(commandText, connection);
command.ExecuteNonQuery();
If I want to update a bunch of records in my database, I use the SqlCommandBuilder class, as in this example:
adapter = new System.Data.SqlClient.SqlDataAdapter("SELECT * FROM Test_Table WHERE Number = '9'",connection);
commandbuilder = new System.Data.SqlClient.SqlCommandBuilder(adapter);
dataset = new System.Data.DataSet();
adapter.Fill(dataset,"Example");
dataset.Tables["Example"].Rows[0].["Number"] = 10;
adapter.update(dataset,"Example");
These work great. But now, for some reason I need to insert/update datetimes and use the CONVERT function on it.
The single SQL query works great:
t = System.DateTime.Now;
commandText = "INSERT INTO Test_Table (DateTime) VALUES (datetime, 't.toString()', 103)";
command = new System.Data.SqlClient.SqlCommand(commandText, connection);
command.ExecuteNonQuery();
This works without a problem, however I have no idea how to achieve the same thing using my SqlCommandBuilder scripts. I could change everything to single query, but this would take a week.
I have already tried the following, without success:
t = System.DateTime.Now;
adapter = new System.Data.SqlClient.SqlDataAdapter("SELECT * FROM Test_Table WHERE Number = '9'", connection);
commandbuilder = new System.Data.SqlClient.SqlCommandBuilder(adapter);
dataset = new System.Data.DataSet();
adapter.Fill(dataset, "Example");
dataset.Tables["Example"].Rows[0].["DateTime"] = "CONVERT(datetime,'" + t.toString() + "',103);
adapter.update(dataset, "Example");
This line of code is weird:
dataset.Tables["Example"].Rows[0].["DateTime"] = "CONVERT(datetime,'" + t.toString() + "',103);
Does it compile? Specifically, this:
.Rows[0].["DateTime"]
I think it should be:
.Rows[0]["DateTime"]
But regardles of the syntax...I don't think this is the right way to go. The datatable (in the dataset) expects a datetime object (btw, don't name your attributes by their datatype, it causes confusion) and you are providing it with something that is incompatible. Sytem.DateTime.Now returns a DateTime object, then you are concatenating it with string (again, does this compile?) and I assume you expect it to be injected into the INSERT statement?
Since you said that it would take a week to change everything, I assume that you have a lot of similar code to repair.
I see three possible solutions, all require some work:
Create a database trigger
https://msdn.microsoft.com/en-us/library/ms189799.aspx
Add a default value to the DateTime field in the database and remove the DateTime from the select query.
http://www.w3schools.com/sql/sql_default.asp
https://www.youtube.com/watch?v=INfi7jkdXC8
(start watching at around 2:00)
You can write a function that does the actual text replacing but it can get tricky:
dasdas as
private string ChangeDate(string insertQuery)
{
// find the location of the date column
// replace the actual value with the "CONVERT(datetime,'" + actualValue + "',103)"
// return the new value and store it in the sqlcommandbuilder.insertstatement
}
Admittedly, all three require work and are not really "elegant". I would go for option 2, because it seems less work. But I don't know if this solves your problem..
The code I'm using deletes the last row in the database. From what I have read, once I've used ORDER BY it will set the result back to read only which makes deleterow() or anything else involving updating the database not possible.
Is there a work around so I could make the table ORDER BY Score DESC and still delete the row I need to delete?
String host = "jdbc:derby://localhost:1527/Scores";
String uName = "root";
String uPass= "root";
Connection con = DriverManager.getConnection( host, uName, uPass );
Statement stmt = con.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE,
ResultSet.CONCUR_UPDATABLE);
String SQLd = "select * from ROOT.HISCORES ORDER BY Score DESC";
ResultSet dl = stmt.executeQuery( SQLd );
dl.last();
dl.deleteRow();
}
This does not look like a good practice to me.
Instead of SELECTing data into Java and deleting it from there, just do it at the database level. Do a delete-select and skip most of the Java work.
Something like:
DELETE FROM root.hiscores
WHERE Score = (SELECT MIN(hs.Score) FROM root.hiscores hs)
I want to use this syntax but I cant:
Select (Clng ( ##IDENTITY ) )
Or
Clng ( select ( ##IDENTITY ) )
I want to get last inserted id in current scope and cast it to the Long type..how can I make this in one query?
..
This query worked correctly:
Select ##identity
And give me the last inserted autonumber in current session but I want to cast it to the something else in one query
In the first query , You are converting a value into long while you are retrieving it, which is a valid operation.
When you use second query , when select statement is used, you are getting a record set and not an single value.
You can not apply clng on result set but you can apply on a single value
Updated Code:
Check the sample code below:
string query = "Insert Into Categories (CategoryName) Values (?)";
string query2 = "Select ##Identity";
long ID;
string connect = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=|DataDirectory|Northwind.mdb";
using (OleDbConnection conn = new OleDbConnection(connect))
{
using (OleDbCommand cmd = new OleDbCommand(query, conn))
{
cmd.Parameters.AddWithValue("", Category.Text);
conn.Open();
cmd.ExecuteNonQuery();
cmd.CommandText = query2;
ID = (long)cmd.ExecuteScalar();
}
}
I just gave the C# version of code as i was not much aware of VB or VB.Net . More over , methodology is the same
I need to populate dropdown menues with a table that contains 4 million rows.
How can I do this without having it time out on the select statement?
Is need SQL Injections. ? or anything else ?
Now I tried only get top 100 rows . But my project have lot of users and lot of details in database . So i need to show all values in dropdownlist , my current code is here :
protected void SearchButton_Click(object sender, EventArgs e)
{
var search = YourSeachTextBox.Text.Trim();
if(!String.IsNullOrEmpty(search) && search.Length > 3)
{
using(SqlConnection sqlConnection = new SqlConnection("Your Connection String"))
{
var query = "SELECT TOP 100 * FROM [YourTable] WHERE UserName LIKE #Search";
SqlCommand sqlCommand = new SqlCommand(query,sqlConnection);
sqlCommand.Parameters.AddWithValue("#Search", search + "%");
}
}
}
i assume that you intend to populate drop downs with parts aof 4 million rows?
Then you do have to create indexes on those columns that help to separate them!
If you really intend to populate them with most of the contents at once, things will have to timeout for sure as your clients browsers won't get to handle this!