I want to sync data from remote api! something like 1M record! but the whole process talks about 5Mins.
as a user experience, that's very bad thing to do! I want the whole process takes less than 1S!
I mainly use .net core web api 6.0 with SQLite, EF Core!
I search a lot and I used BulkInsert! and BlukSaveChangesAsync and same it talks a long time!
Same it's very bad user experience. I tried the following commented solutions and same problem! I want to make it very fast! as the user! does not feel that there is sync in background or thow.
Note: also I stopped all indexes while inserting the data, to make the process faster! and same problem.
Note: My app is Monolithic.
I know I can use something like Azure function but that would be considered as over engineering.
I want the simpliest way to solve this! I searched a lot in YouTube, GitHub and Stack overflow and I found nothing that would help me as I wish.
Note: I'm writing the data in two tables!
first table: contains only 5 rows.
second table: contains 3 rows.
`
public async Task<IEnumerable<DatumEntity>> SyncCities()
{
var httpClient = _httpClientFactory.CreateClient("Cities");
var httpResponseMessage = await httpClient.GetAsync(
"API_KEY_WITH_SOME_CREDS");
if (httpResponseMessage.IsSuccessStatusCode)
{
using var contentStream =
await httpResponseMessage.Content.ReadAsStreamAsync();
var result = await JsonSerializer.DeserializeAsync<Result>(contentStream);
var datums = result!.Data;
if (datums.Any())
{
//First solution
//_context.Datums.AddRange(datums);
//await _context.SaveChangesAsync();
//second solution
//await _context.BulkInsertAsync(datums);
//await _context.BulkSaveChangesAsync();
//Thread solution
//ThreadPool.QueueUserWorkItem(async delegate
//{
// _context.Datums.AddRange(datums);
// await _context.BulkSaveChangesAsync();
//});
}
return datums;
}
return Enumerable.Empty<DatumEntity>();
}
Tried: I tried bulkInsert! tried ThreadPool!stopped all indexes! I tried a lot of things. and nothing helped me as I tought!
I want the whole process takes less than 1S as the user does not move away from the application! because the bad user experience.
This ThreadPool solved the issue for me:
if (datums.Any())
{
ThreadPool.QueueUserWorkItem(async _ =>
{
using (var scope = _serviceScopeFactory.CreateScope())
{
var context = scope.ServiceProvider
.GetRequiredService<CitiesDbContext>();
context.Datums.AddRange(datums);
await context.SaveChangesAsync();
};
});
}
Related
What's the best way to handle transaction errors in Asp.net core and entity framework?
At this time I have come with something like:
using (var transaction = _dbContext.Database.BeginTransaction())
{
try
{
await _dbContext.MyTable1.AddAsync(table1Entity);
await _dbContext.SaveChangesAsync();
Entity2 e2 = new Entity2();
e2.Table1Id = table1Entity.Id;
await _dbContext.SaveChangesAsync();
await transaction.CommitAsync();
return new CreatedAtRouteResult(...);
}
catch (Exception ex)
{
System.Console.Write(ex);
await transaction.RollbackAsync();
var context = HttpContext.Features.Get<IExceptionHandlerFeature>();
return Problem(
detail: context.Error.StackTrace,
title: context.Error.Message);
}
}
But don't really know if this is a good practice. How would you do this?
There is nothing wrong with the way you are handling the transactions, but there are some improvements you can make here:
Remove the data access code from your controller and move it into a separate class.
Do not return the technical details of the error, but a user friendly message.
AddAsync only exists for special use cases, all other cases should use the non-async method Add.
From the EF docs:
"This method is async only to allow special value generators, such as
the one used by
'Microsoft.EntityFrameworkCore.Metadata.SqlServerValueGenerationStrategy.SequenceHiLo',
to access the database asynchronously. For all other cases the non
async method should be used."
I think your code was good, except you don't need to call:
await transaction.RollbackAsync();
in catch block. Failed transaction will auto-rollback when disposed.
Link: https://learn.microsoft.com/en-us/ef/core/saving/transactions
I want to ask about increase performance when i do multiple future http request in single page. In case , i want to build a dashboard page. In dashboard, i've 4 endpoints url that return different result in every endpoint and should be shown in dashboard page.
here example code when load data
var client = new http.Client();
Future main() async {
var newProducts = await client.get("${endpoint}/product?type=newly&limit=5");
ProductListResponse newProductResponse = ProductListResponse.fromJson(json.decode(newProducts.body));
var bestSeller = await client.get("${endpoint}/product?type=best-seller&limit=5");
ProductListResponse bestSellerResponse = ProductListResponse.fromJson(json.decode(bestSeller.body));
var outOfStock = await client.get("${endpoint}/product?type=out-of-stock&limit=5");
ProductListResponse outOfStockResponse = ProductListResponse.fromJson(json.decode(outOfStock.body));
var lastRequest = await client.get("${endpoint}/product-request?type=newly&limit=5");
ProductRequestListResponse productRequestResponse = ProductRequestListResponse.fromJson(json.decode(lastRequest.body));
}
every endpoint when i hit manually using postman it takes 200ms for return the result. But when i implement in flutter app, it took almost 2s.
can i improve performance when getting data?
The reason why your code run so slow is that you are making those HTTP requests one by one. Each await will take quite some time.
You can either not use await and implement the logic using callbacks (.then) or you can combine the Futures into one using Future.wait and use await for that combined Future.
Your code will look something like this:
var responses = await Future.wait([
client.get("${endpoint}/product?type=newly&limit=5"),
client.get("${endpoint}/product?type=best-seller&limit=5"),
client.get("${endpoint}/product?type=out-of-stock&limit=5"),
client.get("${endpoint}/product-request?type=newly&limit=5")
]);
I have ASP.NET Core web application setup with SignalR scaled-out with Redis.
Using the built-in groups works fine:
Clients.Group("Group_Name");
and survives multiple load-balancers. I'm assuming that SignalR persists those groups in Redis automatically so all servers know what groups we have and who are subscribed to them.
However, in my situation, I can't just rely on Groups (or Users), as there is no way to map the connectionId (Say when overloading OnDisconnectedAsync and only the connection id is known) back to its group, and you always need the Group_Name to identify the group. I need that to identify which part of the group is online, so when OnDisconnectedAsync is called, I know which group this guy belongs to, and on which side of the conversation he is.
I've done some research, and they all suggested (including Microsoft Docs) to use something like:
static readonly ConcurrentDictionary<string, ConversationInformation> connectionMaps;
in the hub itself.
Now, this is a great solution (and thread-safe), except that it exists only on one of the load-balancer server's memory, and the other servers have a different instance of this dictionary.
The question is, do I have to persist connectionMaps manually? Using Redis for example?
Something like:
public class ChatHub : Hub
{
static readonly ConcurrentDictionary<string, ConversationInformation> connectionMaps;
ChatHub(IDistributedCache distributedCache)
{
connectionMaps = distributedCache.Get("ConnectionMaps");
/// I think connectionMaps should not be static any more.
}
}
and if yes, is it thread-safe? if no, can you suggest a better solution that works with Load-Balancing?
Have been battling with the same issue on this end. What I've come up with is to persist the collections within the redis cache while utilising a StackExchange.Redis.IDatabaseAsync alongside locks to handle concurrency.
This unfortunately makes the entire process sync but couldn't quite figure a way around this.
Here's the core of what I'm doing, this attains a lock and return back a deserialised collection from the cache
private async Task<ConcurrentDictionary<int, HubMedia>> GetMediaAttributes(bool requireLock)
{
if(requireLock)
{
var retryTime = 0;
try
{
while (!await _redisDatabase.LockTakeAsync(_mediaAttributesLock, _lockValue, _defaultLockDuration))
{
//wait till we can get a lock on the data, 100ms by default
await Task.Delay(100);
retryTime += 10;
if (retryTime > _defaultLockDuration.TotalMilliseconds)
{
_logger.LogError("Failed to get Media Attributes");
return null;
}
}
}
catch(TaskCanceledException e)
{
_logger.LogError("Failed to take lock within the default 5 second wait time " + e);
return null;
}
}
var mediaAttributes = await _redisDatabase.StringGetAsync(MEDIA_ATTRIBUTES_LIST);
if (!mediaAttributes.HasValue)
{
return new ConcurrentDictionary<int, HubMedia>();
}
return JsonConvert.DeserializeObject<ConcurrentDictionary<int, HubMedia>>(mediaAttributes);
}
Updating the collection like so after I've done manipulating it
private async Task<bool> UpdateCollection(string redisCollectionKey, object collection, string lockKey)
{
var success = false;
try
{
success = await _redisDatabase.StringSetAsync(redisCollectionKey, JsonConvert.SerializeObject(collection, new JsonSerializerSettings
{
ReferenceLoopHandling = ReferenceLoopHandling.Ignore
}));
}
finally
{
await _redisDatabase.LockReleaseAsync(lockKey, _lockValue);
}
return success;
}
and when I'm done I just ensure the lock is released for other instances to grab and use
private async Task ReleaseLock(string lockKey)
{
await _redisDatabase.LockReleaseAsync(lockKey, _lockValue);
}
Would be happy to hear if you find a better way of doing this. Struggled to find any documentation on scale out with data retention and sharing.
I'm using autofac and the interfaces are correctly resolved but this code fails with "No connection could be made because the target machine actively refused it 127.0.0.1:8081"
using (var store = GetService<IDocumentStore>())
{
using (var session = store.OpenSession())
{
session.Store(new Entry { Author = "bob", Comment = "My son says this", EntryId = Guid.NewGuid(), EntryTime = DateTime.Now, Quote = "I hate you dad." });
session.SaveChanges();
}
}
Here is the registration
builder.Register<IDocumentStore>(c =>
{
var store = new DocumentStore { Url = "http://localhost:8081" };
store.Initialize();
return store;
}).SingleInstance();
When I navigate to http://localhost:8081 I do get the silverlight management UI. Although I'm running a Windows VM and vmware and Silverlight5 don't play together. That's another issue entirely. Anyways does anyone see what I'm doing wrong here or what I should be doing differently? Thanks for any code, tips, or tricks.
On a side note, can I enter some dummy records from a command line interface? Any docs or examples of how I can do that?
Thanks All.
Just curious, are you switching RavenDB to listen on 8081? The default is 8080. If you're getting the management studio to come up, I suspect you are.
I'm not too familiar with autofac but, it looks like you're wrapping your singleton DocumentStore in a using statement.
Try:
using (var session = GetService<IDocumentStore>().OpenSession())
{
}
As far as dummy records go, the management studio will ask you if you want to generate some dummy data if your DB is empty. If you can't get silverlight to work in the VM, I'm not sure if there's another automated way to do it.
Perhaps using smuggler:
http://ravendb.net/docs/server/administration/export-import
But you'd have to find something to import.
We seem to have verified that RavenDB is getting stale results even when we use various flavors of "WaitForNonStaleResults". Following is the fully-functional sample code (written as a standalone test so that you can copy/paste it and run it as is).
public class Cart
{
public virtual string Email { get; set; }
}
[Test]
public void StandaloneTestForPostingOnStackOverflow()
{
var testDocument = new Cart { Email = "test#abc.com" };
var documentStore = new EmbeddableDocumentStore { RunInMemory = true };
documentStore.Initialize();
using (var session = documentStore.OpenSession())
{
using (var transaction = new TransactionScope())
{
session.Store(testDocument);
session.SaveChanges();
transaction.Complete();
}
using (var transaction = new TransactionScope())
{
var documentToDelete = session
.Query<Cart>()
.Customize(x => x.WaitForNonStaleResultsAsOfLastWrite())
.First(c => c.Email == testDocument.Email);
session.Delete(documentToDelete);
session.SaveChanges();
transaction.Complete();
}
RavenQueryStatistics statistics;
var actualCount = session
.Query<Cart>()
.Statistics(out statistics)
.Customize(x => x.WaitForNonStaleResultsAsOfLastWrite())
.Count(c => c.Email == testDocument.Email);
Assert.IsFalse(statistics.IsStale);
Assert.AreEqual(0, actualCount);
}
}
We have tried every flavor of WaitForNonStaleResults and there is no change. Waiting for non-stale results seems to work fine for the update, but not for the delete.
Update
Some things which I have tried:
Using separate sessions for each action. Outcome: no difference. Same successes and fails.
Putting Thread.Current.Sleep(500) before the final query. Outcome: success. If I sleep the thread for a half second, the count comes back zero like it should.
Re: my comment above on stale results, AllowNonAuthoritiveInformation wasn't working. Needing to put WaitForNonStaleResults in each query, which is the usual "answer" to this issue, feels like a massive "code smell" (as much as I normally hate the term, it seems completely appropriate here).
The only real solution I've found so far is:
var store = new DocumentStore(); // do whatever
store.DatabaseCommands.DisableAllCaching();
Performance suffers accordingly, but I think slower performance is far less of a sin than unreliable if not outright inaccurate results.
This is an old question, but I recently ran across this problem as well. I was able to work around it by changing the convention on the DocumentStore used by the session to make it wait for non stale as of last write:
session.DocumentStore.DefaultQueryingConsistency = ConsistencyOptions.AlwaysWaitForNonStaleResultsAsOfLastWrite;
This made it so that I didn't have to customize every query run after. That said, I believe this only works for queries. It definitely doesn't work on patches as I have found out through testing.
I would also be careful about this and only use it around the code that's needed as it can cause performance issues. You can set the store back to its default with the following:
session.DocumentStore.DefaultQueryingConsistency = ConsistencyOptions.None;
The problem isn't related to deletes, it is related to using TransactionScope. The problem here is that DTC transaction complete in an asynchronous manner.
To fix this issue, what you need to do is call:
session.Advanced.AllowNonAuthoritiveInformation = false;
Which will force RavenDB to wait for the transaction to complete.