I need Ember Data to update one of my models in bulk. So I set
bulkCommit: true
in the DS.RESTAdapter. But now it even uses bulk commits for updates to a single record!
This is very unexpected behaviour.
So how do I modify Ember Data to only use bulk commits when more than 1 item is committed?
Here is what I've done now:
updateRecords: function(store, type, records) {
var arr;
arr = records.list;
if (arr.length === 1) {
return this.updateRecord(store, type, arr[0]);
} else {
return this._super(store, type, records);
}
}
This checks if records consists of a single item and calls updateRecord then.
createRecords and deleteRecords are changed accordingly.
Related
If I want to make some checks before inserting a row into the database, I know that I can run the following code:
public bool BookSeat(int userId, string seatNumber)
{
If (IsSeatStillAvailable(seatNumber))
{
var ticket = new Ticket(userId, seatNumber);
_dbContext.Tickets(ticket);
_dbContext.SaveChanges();
return true;
}
return false;
}
private bool IsSeatStillAvailable(seatNumber)
{
var seatTaken = _dbcontext.Tickets.Any(w => w.seatNumber == seatNumber);
return !seatTaken;
}
This will do a call to the database to see if the seat is taken and then do a second call to book the seat. BUT in that time it might have already been booked.
I know in simple examples, I can create an index, but my use case is much more complex.
Is there a way that I can generate the SQL statement on the insert so that it can produce an ATOMIC transaction?
To produce something like (excuse the clumsy SQL):
IF (SELECT TOP 1 Count(*) FROM Tickets = 0)
BEGIN
INSERT INTO Tickets (UserId, SeatNumber)
VALUES (#UserId, #SeatNumber);
RETURN true
END;
RETURN false
What you are looking for is concurrency handling and optimistic locking :
https://learn.microsoft.com/en-us/ef/core/saving/concurrency?tabs=data-annotations
I am trying to update hundreds of database records using the TypeORM library. Problem is that sometimes DUPLICATE ERR is returned from SQL when the bulk upload is performed and stops the whole operation. Is possible to set up TypeORM in a way so duplicate entries are ignored and the insert is performed?
The table is using two primary keys:
This is my insert command (TypeORM + Nestjs):
public async saveBulk(historicalPrices: IHistoricalPrice[]) {
if (!historicalPrices.length) {
return;
}
const repoPrices = historicalPrices.map((p) => this.historicalPricesRepository.create(p));
await this.historicalPricesRepository.save(repoPrices, { chunk: 200 });
}
Thanks in advance
You will have to use InsertQueryBuilder to save the entities instead of repository.save method. InsertQueryBuilder will allow you to call an additional method orIgnore() which will add IGNORE literal into your mysql INSERT statement. From mysql official doc:
When INSERT IGNORE is used, the insert operation fails silently for rows containing the unmatched value, but inserts rows that are matched.
One demerit is obviously that you'll have to now chunk the rows on your own. InsertQueryBuilder doesn't provide any options to chunk the entities. Your code should look like this:
for (let i = 0; i < historicalPrices.length; i += 200) {
const chunk = historicalPrices.slice(i, i + 200);
const targetEntity = this.historicalPricesRepository.target;
await this.historicalPricesRepository
.createQueryBuilder()
.insert()
.into(targetEntity)
.values(chunk)
.orIgnore()
.execute();
}
Currently working on an API which given an address returns information about is. Some of the rows in our tables are duplicates, however being as there is over 15 million I cant go and find the duplicates. Instead I have opted to use
var query = `SELECT TOP 1 * from my_TABLE where..conditions`;
This ensures that only one row of the duplicates are returned.
The problem is when this is sent back as a JSON it comes as an array with one object.
In the Server.js file
// create Request object
var request = new sql.Request();
// query to the database
request.query(query, function (err, result) {
if (err) {
console.log("Error while querying database :- " + err);
res.send(err);
}
else {
res.send(result)
}
Returns this:
[{
Address:'our info'
}]
is there a way to have it respond with
{
Address:'our info'
}
Because from DB you've get list of object anyway, even there is only 1 item.
It works as you expected when you try to return json with the first element of your array.
Using Entity Framework, I am updating about 300 rows, and 9 columns about every 30 seconds. Below is how I am currently doing it. My question is, how can I make the code more efficient?
Every once in a while, I feel my database gets hit with the impact and I just want to make it as efficient as possible.
// FOREACH OF MY 300 ROWS
var original = db.MarketDatas.FirstOrDefault(x => x.BBSymbol == targetBBsymbol);
if (original != null)
{
//if (original.BBSymbol.ToUpper() == "NOH7 INDEX")
//{
// var x1 = 1;
//}
original.last_price = marketDataItem.last_price;
original.bid = marketDataItem.bid;
original.ask = marketDataItem.ask;
if (marketDataItem.px_settle_last_dt_rt != null)
{
original.px_settle_last_dt_rt = marketDataItem.px_settle_last_dt_rt;
}
if (marketDataItem.px_settle_actual_rt != 0)
{
original.px_settle_actual_rt = marketDataItem.px_settle_actual_rt;
}
original.chg_on_day = marketDataItem.chg_on_day;
if (marketDataItem.prev_close_value_realtime != 0)
{
original.prev_close_value_realtime = marketDataItem.prev_close_value_realtime;
}
if (marketDataItem.px_settle_last_dt_rt != null)
{
DateTime d2 = (DateTime)marketDataItem.px_settle_last_dt_rt;
if (d1.Day == d2.Day)
{
//market has settled
original.settled = "yes";
}
else
{
//market has NOT settled
original.settled = "no";
}
}
if (marketDataItem.updateTime.Year != 1)
{
original.updateTime = marketDataItem.updateTime;
}
db.SaveChanges();
}
Watching what is being hit in the debugger...
SELECT TOP (1)
[Extent1].[MarketDataID] AS [MarketDataID],
[Extent1].[BBSymbol] AS [BBSymbol],
[Extent1].[Name] AS [Name],
[Extent1].[fut_Val_Pt] AS [fut_Val_Pt],
[Extent1].[crncy] AS [crncy],
[Extent1].[fut_tick_size] AS [fut_tick_size],
[Extent1].[fut_tick_val] AS [fut_tick_val],
[Extent1].[fut_init_spec_ml] AS [fut_init_spec_ml],
[Extent1].[last_price] AS [last_price],
[Extent1].[bid] AS [bid],
[Extent1].[ask] AS [ask],
[Extent1].[px_settle_last_dt_rt] AS [px_settle_last_dt_rt],
[Extent1].[px_settle_actual_rt] AS [px_settle_actual_rt],
[Extent1].[settled] AS [settled],
[Extent1].[chg_on_day] AS [chg_on_day],
[Extent1].[prev_close_value_realtime] AS [prev_close_value_realtime],
[Extent1].[last_tradeable_dt] AS [last_tradeable_dt],
[Extent1].[fut_notice_first] AS [fut_notice_first],
[Extent1].[updateTime] AS [updateTime]
FROM [dbo].[MarketDatas] AS [Extent1]
WHERE ([Extent1].[BBSymbol] = #p__linq__0) OR (([Extent1].[BBSymbol] IS NULL) AND (#p__linq__0 IS NULL))
It seems it updates the same thing multiple times, if I am understanding it correctly.
UPDATE [dbo].[MarketDatas]
SET [last_price] = #0, [chg_on_day] = #1, [updateTime] = #2
WHERE ([MarketDataID] = #3)
UPDATE [dbo].[MarketDatas]
SET [last_price] = #0, [chg_on_day] = #1, [updateTime] = #2
WHERE ([MarketDataID] = #3)
You can reduce this to 2 round trips.
Don't call SaveChanges() in side the loop. Move it outside and call it after you are done processing everything.
Write the select in such a way that it retrieves all the originals in one go and pushes them to a memory collection, then retrieve from that for each item you are updating/inserting.
code
// use this as your source
// to retrieve an item later use TryGetValue
var originals = db.MarketDatas
.Where(x => arrayOftargetBBsymbol.Contains(x.BBSymbol));
.ToDictionary(x => x.BBSymbol, y => y);
// iterate over changes you want to make
foreach(var change in changes){
MarketData original = null;
// is there an existing entity
if(originals.TryGetValue(change.targetBBsymbol, out original)){
// update your original
}
}
// save changes all at once
db.SaveChanges();
You could only execute "db.SaveChanges" after your foreach loop. It think it you would do exactly what your are asking for.
It seems it updates the same thing multiple times, if I am
understanding it correctly.
Entity Framework performs a database round-trip for every entity to update.
Just check the parameter value, they will be different.
how can I make the code more efficient
The major problem is your current solution is not scalable.
It works well when you only have a few entities to update but will become worse and worse are the number of items to update in a batch will increase.
It's often better to make this kind of logic all in the database, but perhaps you cannot do it.
Disclaimer: I'm the owner of the project Entity Framework Extensions
This library can make your code more efficient by allowing you to save multiples entities at once. All bulk operations are supported:
BulkSaveChanges
BulkInsert
BulkUpdate
BulkDelete
BulkMerge
BulkSynchronize
Example:
// Easy to use
context.BulkSaveChanges();
// Easy to customize
context.BulkSaveChanges(bulk => bulk.BatchSize = 100);
// Perform Bulk Operations
context.BulkDelete(customers);
context.BulkInsert(customers);
context.BulkUpdate(customers);
// Customize Primary Key
context.BulkMerge(customers, operation => {
operation.ColumnPrimaryKeyExpression =
customer => customer.Code;
});
I need to save same model($modelcrite) multiple times after changing a id here is the code
protected function saveed($data1,$studentid,$modelcrite,$model)
{
$index = 0;
foreach ($data1 as $key => $value) {
$studentid[$index]=(string)$key;
$modelcrite->setAttribute('st_id',$key);
if ($modelcrite->validate()){
$modelcrite->setScenario('insert');
$modelcrite->save();
}
else {
$this->Delete($model->ass_id);
return FALSE;
}
$index=$index+1;
}
return TRUE;
}
but the problem is when second time save the $modelcrite value it is updating the database.I need to save it as new one
please can any one tell how can i do that.thank you.
I believe that you confusion comes from the fact that an ActiveRecord model does not represent a table, but instead represents a row in the table. So changing the terminology a bit, when you pass in the new model(row) $modelcrite ActiveRecord recognizes that this is a new model(row), and the save performs an insert. At this point now Active record realizes that the model(row) exists in the table, and any additional saves against that model(row) will generate an update.
With that background in mind what you need to do is in your foreach loop create a new model for the insert, and set the attributes of the new model to $modelcrite, then set the student_id, then validate and save.
Something like this:
foreach ($data1 as $key => $value) {
$newModel = new CriteModel();
$newModel->attributes = $modelcrite->attributes;
$studentid[$index]=(string)$key;
$newModel->setAttribute('st_id',$key);
if ($newModel->validate()){
$newModel->save();