I have something that looks like this
foreach (var user in NewUsers)
{
var dbUser = new User {FirstName = user.FirstName};
dbUser.Save();
}
That is too many inserts into the database. Can I do something like?
User.BulkInsert(NewUsers);
Thanks.
Check out the Batch Query - this should solve the issue
http://www.subsonicproject.com/docs/BatchQuery
Related
I am currently updating a BackEnd project to .NET Core and having performance issues with my Linq queries.
Main Queries:
var queryTexts = from text in _repositoryContext.Text
where text.KeyName.StartsWith("ApplicationSettings.")
where text.Sprache.Equals("*")
select text;
var queryDescriptions = from text in queryTexts
where text.KeyName.EndsWith("$Descr")
select text;
var queryNames = from text in queryTexts
where !(text.KeyName.EndsWith("$Descr"))
select text;
var queryDefaults = from defaults in _repositoryContext.ApplicationSettingsDefaults
where defaults.Value != "*"
select defaults;
After getting these IQueryables I run a foreach loop in another context to build my DTO model:
foreach (ApplicationSettings appl in _repositoryContext.ApplicationSettings)
{
var applDefaults = queryDefaults.Where(c => c.KeyName.Equals(appl.KeyName)).ToArray();
description = queryDescriptions.Where(d => d.KeyName.Equals("ApplicationSettings." + appl.KeyName + ".$Descr"))
.FirstOrDefault()?
.Text1 ?? "";
var name = queryNames.Where(n => n.KeyName.Equals("ApplicationSettings." + appl.KeyName)).FirstOrDefault()?.Text1 ?? "";
// Do some stuff with data and return DTO Model
}
In my old Project, this part had an execution from about 0,45 sec, by now I have about 5-6 sec..
I thought about using compiled queries but I recognized these don't support returning IEnumerable yet. Also I tried to avoid Contains() method. But it didn't improve performance anyway.
Could you take short look on my queries and maybe refactor or give some hints how to make one of the queries faster?
It is to note that _repositoryContext.Text has compared to other contexts the most entries (about 50 000), because of translations.
queryNames, queryDefaults, and queryDescriptions are all queries not collections. And you are running them in a loop. Try loading them outside of the loop.
eg: load queryNames to a dictionary:
var queryNames = from text in queryTexts
where !(text.KeyName.EndsWith("$Descr"))
select text;
var queryNamesByName = queryName.ToDictionary(n => n.KeyName);
one can write queries like below
var Profile="developer";
var LstUserName = alreadyUsed.Where(x => x.Profile==Profile).ToList();
you can also use "foreach" like below
lstUserNames.ForEach(x=>
{
//do your stuff
});
I'm trying to develop a tiny front-end for a sqlite DB using Razor Pages (NOT MVC!).
Looks like it supports SQL in a different way, not like I used to in the past.
For example, I need to re-assign value to the field Pl_Pos (position of the record in the list). Now I do this like that:
for (int i = PlnPos + 1; i<= PlModelIndexModel.PlnModelTotal; i++)
{
PlnModel PlnModelRedo = await _context.Plane_Models.FirstOrDefaultAsync(m => m.Pl_Model_Pos == i);
PlnModelRedo.Pl_Model_Pos = i - 1;
_context.Attach(PlnModelRedo).State = EntityState.Modified;
await _context.SaveChangesAsync();
}
Is there any other way to do it more efficient and do something like:
Update Pl_Models set Pl._Models.Pl_Pos = Pl_Models set Pl._Models.Pl_Pos - 1 where Pl_Models set Pl._Models.Pl_Pos > PlnPos
I stuck a bit...
Thanks in advance!
I'm not entirely sure what you're asking but it sounds like
you're just after a LINQ statement or foreach loop.
var PlaneModels = _context.Plane_Models.ToList();
foreach(PlaneModel pm in PlaneModels)
{
PlaneModel.Pl_Model_Pos -= 1;
}
_context.Plane_Models.UpdateRange(PlaneModels);
await _context.SaveChangesAsync();
When are you trying to modify the position of the record in the list? When a record is inserted/appended/deleted, or is it used as a display index?
How can I truncate a table with CakePHP 3.x
I get the truncate query by this:
$this->Coupons->schema()->truncateSql($this->Coupons->connection());
but what is the best practice to execute it
This code working well, thanks to #ndm for his comment that helped the answer to be better.
//In Coupons Controller
$this->Coupons->connection()->transactional(function ($conn) {
$sqls = $this->Coupons->schema()->truncateSql($this->Coupons->connection());
foreach ($sqls as $sql) {
$this->Coupons->connection()->execute($sql)->execute();
}
});
In CakePHP4 you can use the following code to truncate a table:
$table = $this->Coupons;
$sqls = $table->getSchema()->truncateSql($table->getConnection());
foreach ($sqls as $sql) {
$table->getConnection()->execute($sql)->execute();
}
I tested following and it's worked:
$connection = $this->Coupons->getConnection();
$connection->query('TRUNCATE coupons');
Reference: https://book.cakephp.org/4/en/orm/database-basics.html#executing-queries
Read more here: https://book.cakephp.org/4/en/orm/database-basics.html#using-transactions
There is a problem. I have old database with some data, and by another side I have new database with new structure.
Now I need best way (ideas) how to copy data from one table to the another. Problem is some tables have max 1000 records some 32 000 some 640 000, and time to copy 5000+ is really long.
Any best practices ? Sample code below ...
public ActionResult ImportTable1()
{
var oldTable1 = context.OLDTABLE.ToList();
foreach (var item in oldTable1)
{
try
{
var cTable = contextNew.NEWTABLE.Where(p => p.fiel1 == item.field1).FirstOrDefault();
if (cTable == null)
{
NEWTABLE nTable = new NEWTABLE
{
field1 = item.field1,
field2 = item.field2
};
contextNew.NEWTABLE.Add(nTable);
}
else
{
cTable.field1 = item.field1
cTable.field2 = item.field2;
contextNew.Entry(cTable).State = EntityState.Modified;
}
IcontextNew.SaveChanges();
}
catch (DbEntityValidationException dbEx)
{
foreach (var validationErrors in dbEx.EntityValidationErrors)
{
foreach (var validationError in validationErrors.ValidationErrors)
{
_progresLog = ("Property: " + validationError.PropertyName + " Error: {1}" + validationError.ErrorMessage);
}
}
}
return PartialView();
}
... so bulk now
public void ExperimentalPartsBulk()
{
string msisDatabase = ConfigurationManager.ConnectionStrings["old"].ToString();
string newDatabase = ConfigurationManager.ConnectionStrings["new"].ToString();
SqlConnection sourceconnection = new SqlConnection(msisDatabase);
SqlConnection sourcedestination = new SqlConnection(newDatabase);
sourceconnection.Open();
SqlCommand cmd = new SqlCommand("Select * from ELEMENTS");
cmd.Connection = sourceconnection;
SqlDataReader reader = cmd.ExecuteReader();
//Connect to Destination DataBase
SqlConnection destinationConnection = new SqlConnection(newDatabase);
destinationConnection.Open();
SqlBulkCopy bulkCopy = new SqlBulkCopy(destinationConnection);
bulkCopy.DestinationTableName = "ELEMENTSNEW";
bulkCopy.ColumnMappings.Clear();
bulkCopy.ColumnMappings.Add("fielString1", "newString1");
bulkCopy.ColumnMappings.Add("fielString2", "newStrin2");
bulkCopy.ColumnMappings.Add("fielFloat1", "newINT1");
bulkCopy.WriteToServer(reader);
reader.Close();
sourceconnection.Close();
sourcedestination.Close();
}
problem now is w differences betwen two tables
fielString1 can be null, newString1 cant be |
fielFloat1 is float now is nullable but newINT1 not
How to import with some conditions or to the different types of field ?
Siwek,
Any loop as shown in the first code sample will failed due to performance issues.... as you pointed!
The right approach here is SQL approach. The idea is to "flush" all data to new DB. Flush mean that ALL records (5,000 or 500,000) are stored to new DB with one action! And avoid any loops during extracting, filtering, editing and saving of data, because 640,000 loops takes long time....
Bulk copy is one possible. Issue with bulk copy is that it's hard for you to filter and edit data in this object.
Use ADO.net DataSet to get data from old DB, filter it, edit it, and save it on memory and flush it to new DB. DataSet take one step per action (extracting, filtering, editing, etc. ! no loops).
Or, try SQL replication. Replication is the SQL mechanism to copy data from DB "A" table "oneTable" to another DB, "B" with a table "AnotherTable" with a different schema and rules. Try it. I can specify more if you think it's a reasonable solution for you. No code needed, it's can be created using wizard on SQL Management studio, and run whenever needed (via SQL Job agent).
You should seriously consider SSIS or bcp. Otherwise you are looking at a scenario whet you're pulling data from the source server all the way down to the client box where the .net code is executing, then pushing all off that data up to the destination server. Think of the bandwidth being consumed. If you can instead do an SSIS export into the destination, at least it would be eliminating an extra layer of concern.
If you absolutely must pull data down to the client, consider writing the data into bcp formatted files, and then bulkcopying them into the destination server.
I'm pretty sure that you'll find that both of these paths are significantly faster than using plain old ADO.NET sort of approaches.
I have a linq object and I want to write the query using linq.
please help me.
INPUT:
var tags = (from row in tempChildData.AsEnumerable()
join tagOrder in tupleInfoDataset.Tables["TagHierarchy"].AsEnumerable() on row.Field<Int64>("TAGID") equals tagOrder.Field<Int64>("TAGID")
join tagName in tupleInfoDataset.Tables["SequenceChoiceList"].AsEnumerable() on tagOrder.Field<Int64>("PARENTTAGID") equals tagName.Field<Int64>("TAGID")
join facet in tupleInfoDataset.Tables["FacetType"].AsEnumerable() on tagName.Field<string>("Tag_Name") equals facet.Field<string>("Facetname")
join tagIdInfo in schDataTogetTagid.AsEnumerable() on row.Field<string>("refTagName").Contains(":") ? row.Field<string>("refTagName").Split(':').Last():row.Field<string>("refTagName") equals tagIdInfo.Field<string>("TAGNAME")
where ( childList.Contains(row.Field<Int64>("TAGID")) && facet.Field<string>("FacetType").ToLower().Equals("ctype"))
select new
{
Tagid = row.Field<Int64>("TAGID"),
TagIdToInsert=tagIdInfo.Field<Int64>("TAGID"),
MaxOccur = row.Field<string>("Maxoccurs"),
MinOccur =Convert.ToInt32(Convert.ToString(row.Field<string>("Minoccur"))),
ParentTagId=tagOrder.Field<Int64>("PARENTTAGID"),
Order=tagOrder.Field<Int64>("TAG_ORDER"),
ParentTagname = tagName.Field<string>("Tag_Name"),
FacetId=facet.Field<Int64>("FacetID")
}).ToList();
var parentTagID = (from tagIdInfo in tupleInfoDataset.Tables["Tuple"].AsEnumerable()
where tagIdInfo.Field<Int64>("TAGID").Equals(key.Key)
select tagIdInfo.Field<Int64>("ConceptID")).ToList();
long parentID =Convert.ToInt64(parentTagID[0]);
Now i want the query out of the above code as:
INSERT INTO TUPLE_MAP (TagId,ParentTagId,ParentTagname,MinOccur,MaxOccur,Order)
VALUES (TagIdToInsert,ParentTagId,ParentTagname,MinOccur,MaxOccur,Order)
Please help me I don't know how to write SQL queries using linq
Maybe something like this:
using(var db=new DataContext("YourConnectionStringHERE"))
{
db.TUPLE_MAP.InsertAllOnSubmit(tags.Select (t =>
new TUPLE_MAP()
{
TagId=t.TagIdToInsert,
ParentTagId=t.ParentTagId,
ParentTagname=t.ParentTagname,
MinOccur=t.MinOccur,
MaxOccur=t.MaxOccur,
Order=t.Order
}));
db.SubmitChanges();
}
Or if you want to use the parentID then something like this:
using(var db=new DataContext("YourConnectionStringHERE"))
{
db.TUPLE_MAP.InsertAllOnSubmit(tags.Select (t =>
new TUPLE_MAP()
{
TagId=t.TagIdToInsert,
ParentTagId=parentID,
ParentTagname=t.ParentTagname,
MinOccur=t.MinOccur,
MaxOccur=t.MaxOccur,
Order=t.Order
}));
db.SubmitChanges();
}
where db is your linq data context
Useful references:
How to: Insert Rows Into the Database (LINQ to SQL)
EDIT
So if you are using the Compact database 3.5 then many something like this:
using (var conn =new SqlCeConnection("Data Source = test.sdf; Password ='pass'"))
{
foreach (var tag in tags)
{
using(var cmd = conn.CreateCommand())
{
cmd.CommandText = #"INSERT INTO TUPLE_MAP (TagId,ParentTagId,ParentTagname,MinOccur,MaxOccur,Order)
VALUES (#TagIdToInsert,#ParentTagId,#ParentTagname,#MinOccur,#MaxOccur,#Order)";
cmd.Parameters.AddWithValue("#TagIdToInsert", tag.TagIdToInsert);
cmd.Parameters.AddWithValue("#ParentTagId", tag.ParentTagId);
cmd.Parameters.AddWithValue("#ParentTagname", tag.ParentTagname);
cmd.Parameters.AddWithValue("#MinOccur", tag.MinOccur);
cmd.Parameters.AddWithValue("#MaxOccur", tag.MaxOccur);
cmd.Parameters.AddWithValue("#Order", tag.Order);
cmd.ExecuteNonQuery();
}
}
}
Useful references:
Why can't I insert a record into my SQL Compact 3.5 database?
SqlCeCommand.Parameters Property
SqlCeCommand Class
SqlParameterCollection.AddWithValue Method
Use linq Pad or sql profiler to see the generated SQL.
You can also use visual studio for that purpose. In the debug mode,hold cursor on the variable "tags", you will be able to see the SQL.
I am assuming you are using Linq to SQL, if you are doing so you would have entity called Tuple_map in you xxxDataContext. Then you would just have to create object of that entity something like this....
using (XXXDataContext context = new XXXDataContext())
{
Tuple_map obj = new Tuple_map();
//Populate obj properties like obj.tabid = from objects you got it from above query
context.Tuple_map.InsertOnSubmit(obj);
context.SubmitChanges();
}