I have a very simple Cosmos DB query that I am making from an asp.net core 3 Razor Pages application. The same query I make in Data Explorer in Azure will return results in 0.02ms. When I run it through the application, setting up stopwatches to see the duration of the calls, it can be anywhere from 400ms to 2000ms.
QueryDefinition queryDefinition = new QueryDefinition("SELECT * FROM Cache where Cache.JoinCode = #jc").WithParameter("#jc", JoinCode);
var query = _container.GetItemQueryIterator<HostCache>(queryDefinition);
List<HostCache> results = new List<HostCache>();
while (query.HasMoreResults)
{
var response = await query.ReadNextAsync();
results.AddRange(response.ToList());
}
return results.FirstOrDefault();
The long running request is the await query.ReadNextAsync();. Is there anything I can do to speed that up? Maybe I'm doing it wrong?
First, I would highly recommend that you (or anyone using Cosmos DB .Net SDK) to watch this video on Cosmos DB Youtube Channel: https://www.youtube.com/watch?v=McZIQhZpvew. This provides really useful information about the best practices to follow when working with this SDK.
This video will explain why the first request takes so much time and how you can speed that up.
To summarize for the purpose of this answer, creating an instance of Cosmos Client (with "Direct" connection mode) does not do much. When you make the 1st request with that client, the initialization happens and at that time SDK makes a few network requests to get necessary information about establishing "Direct" (TCP) connection. That's why it takes a great deal of time with the 1st request. After the 1st request, the information is cached by the SDK so subsequent requests take much less time than the 1st one.
To do the initialization while creating Cosmos client, you would need to use CreateAndInitializeAsync method of the CosmosClient. Here's an example of the same from the documentation page:
using Microsoft.Azure.Cosmos;
List<(string, string)> containersToInitialize = new List<(string, string)>
{ ("DatabaseName1", "ContainerName1"), ("DatabaseName2", "ContainerName2") };
CosmosClient cosmosClient = await CosmosClient.CreateAndInitializeAsync("connection-string-from-portal",
containersToInitialize)
Related
I've hooked up Miniprofiler to my local ASP.NET Core project and it works as expected. Now I need it to work in a hosted environment where there are multiple instances of the same website and there are no sticky sessions. It is my understanding that this should be supported if you just set the storage option when configuring the profiler. However, setting the storage does not seem to do anything. I initialize the storage like this:
var redisConnection = "...";
MiniProfiler.DefaultOptions.Storage = new RedisStorage(redisConnection);
app.UseMiniProfiler();
After doing this, I expected that I could open a profiled page and a result would be added to my redis cache. I would then also expect that a new instance of my website would list the original profiling result. However, nothing is written to the cache when generating new profile results.
To test the connection, I tried manually saving a profiler instance (storage.Save()) and it gets saved to the storage. But again, the saved result is not loaded when showing profiler results (and regardless, none of the examples I've seen requires you to do this). I have a feeling that I've missed some point about how the storage is supposed to work.
It turns out that my assumption that MiniProfiler.DefaultOptions.Storage would be used was wrong. After changing my setup code to the following, it works.
// Startup.cs ConfigureServices
var redisConnection = "...";
services.AddMiniProfiler(o =>
{
o.RouteBasePath = "/profiler";
o.Storage = new RedisStorage(redisConnection); // This is new
});
// Startup.cs Configure
app.UseMiniProfiler();
I have a Web Api 2 service that will be deployed across 4 production servers. When a request doesn't pass validation a custom response object is generated and returned to the client.
A rudimentary example
if (!ModelState.IsValid)
{
var responseObject = responseGenerator.GetResponseForInvalidModelState(ModelState);
return Ok(responseObject);
}
Currently the responseGenerator is aware of what environment it is in and generates the response accordingly. For example, in development it'll return a lot detail but in production it'll only return a simple failure status.
How can I implement a "switch" that turns details on without requiring a round trip to the database each time?
Due to the nature of our environment using a config file isn't realistic. I've considered using a flag in the database and then caching it at the application layer but environmental constraints make refreshing the cache on all 4 servers very painful.
I ended up going with the parameter suggestion and then implementing a token system on the back end. If a Debug token is present in the request the service validates it against the database. If it's a valid and active token it returns the additional detail.
This allows us to control things from our end while keeping things simple for the vendors and only adds that extra round trip to the database during debugging.
C#:
Stopwatch stopwatch2 = Stopwatch.StartNew();
TableDataInsertAllResponse response = await BigQueryService.Tabledata.InsertAll(request, _account.ProjectId, table.DataSetId, tableId).ExecuteAsync(ct);
stopwatch2.Stop();
JAVA:
long start = System.currentTimeMillis();
TableDataInsertAllResponse response = mBigquery.tabledata().insertAll(mAccount.getProjectId(), tid.getDatasetId(),tid.getDefaultTableId(), request).execute();
logger.fatal(String.format("%s~BigQuery.InsertAll#Back~%s~%s~%s~%s",FormatHelper.getCurrentDateTime(), rowList.size(), tid,System.currentTimeMillis()-start, payLoad));
As we compared these two against a same dataset and same machine, C# is usually 3-5 times slower than Java call (>500ms vs ~100ms). we traced as deep as we could into the BQ API source code, and found they are basically with the same process of data: serialized data to json and then do gzip and then send out through http calls. it just does not make sense that C# http calls could be that slow. We think we need technical support from BigQuery side on our specific cases to see what happens after BQ receives our request. If any very specific payload differences lead to the performance difference. What level of support package do we need to get?
Edit
Some add-on: Right now, our c# solution is all async (using async and await in c#). I found a post BigQuery streaming 'insertAll' performance with PHP for PHP. I'm having the the same problem with this post. our application cannot wait for more than 100ms for a request. We can forget about comparing it to Java, but we just need the c# calls can run more quicker. I'm wondering if any improvement on InsertAll() or on our side can be done for c#.
I want to use breezejs api for storing data in local storage (indexdb or websql) and also want to sync local data with sql server.
But I am failed to achieve this and also not able to find sample app of this type of application using breezejs, knockout and mvc api.
My requirement is:
1) If internet is available, the data will come from sql server by using mvc web api.
2) If internet is shutdown, the application will retrieve data from cached local storage (indexdb or websql).
3) As soon as internet is on, the local data will sync to sql server.
Please let me know Can I achieve this requirement by using breezejs api or not?
If yes, please provide me some and links and sample.
If no, what other can we use for achieving this type of requirement?
Thanks.
Please help me to meet this requirement.
You can do this, but I would suggest simply using localstorage. Basically, every time you read from the server or save to the server, you export the entities and save that to local storage. THen, when you need to read in the data, if the server is unreachable, you read the data from localstorage and use importentities to get it into the manager and then query locally.
function getData() {
var query = breeze.EntityQuery
.from("{YourAPI}");
manager.executeQuery.then(saveLocallyAndReturnPromise)
.fail(tryLocalRestoreAndReturnPromise)
// If query was successful remotely, then save the data in case connection
// is lost
function saveLocallyAndReturnPromise(data) {
// Should add error handling here. This code
// assumes tis local processing will be successful.
var cacheData = manager.exportEntities()
window.localStorage.setItem('savedCache',cacheData);
// return queried data as a promise so that this detour is
// transparent to viewmodel
return Q(data);
}
function tryLocalRestoreAndReturnPromise(error) {
// Assume any error just means the server is inaccessible.
// Simplified for example, but more robust error handling is
// warranted
var cacheData = window.localStorage.getItem('savedCache');
// NOTE: should handle empty saved cache here by throwing error;
manager.importEntities(cacheData); // restore saved cache
var query = query.using(breeze.FetchStrategy.FromLocalCache);
return manager.executeQuery(query); // this is a promise
}
}
This is a code skeleton for simplicity. You should check catch and handle errors, add an isConnected function to determine connectivity, etc.
If you are doing editing locally, there are a few more hoops to jump through. Every time you make a change to the cache, you will need to export either the whole cache or the changes (probably depending on the size of the cache). When there is a connection, you will need to test for local changes first and, if found, save them to the server before requerying the server. In addition, any schema changes made while offline complicate matters tremendously, so be aware of that.
Hope this helps. A robust implementation is a bit more complex, but this should give you a starting point.
if(isset($_GET['actionid']) && isset($_GET['profileid']))
{
$actionid = $_GET['actionid'];
$profileid = $_GET['profileid'];
$res = $database->news_poll($profileid,$actionid);
$k = 0;
while(!$NROW =$res->fetch_array())
{
usleep('50000000');
$res = $database->news_poll($profileid,$actionid);
}
$action = actiontype_encode($NROW,'0',$json,$encode,$database);
$data['action'] = $action;
echo json_encode($data);
}
this is my script for polling the server for new data.
but the working browser stops working only for my site. I guess the problem is that when a particular browser subscribes for the new data the connection is kept open so no further request can be made by the browser to same server. please explain if any problem.
If there is any way at all you can, I recommend setting yourself up with NodeJS and SocketIO for long polling. Your web server needs to keep a request open for every connected user, and that is more than Apache/PHP can handle for very long.
If that's not possible I recommend short polling, doing a normal ajax request every 3 seconds. That's not perfect but manageble.
I answered a similar question recently with more details.
Regardless of language I strongly advise against writing your own long polling server, unless you want that to be your project for a couple of years. I have been in a project that used a home grown long polling server written C and then re-written in Java, and it was not pretty.
I figured out the problem is that Apache serves multiple requests from a single client one at a time. So when a request is made to the long polling script at backend for new data that request hangs other requests from the same browser to the same server.
To overcome this drawback one needs to use node.js or tornado.