I am trying to display document revisions in a paginated list. This will only work when I know the total results count so I can calculate the page count.
For common document queries this is possible with session.Query(...).Statistics(out var stats).
My current solution is the following:
// get paged revisions
List<T> items = await session.Advanced.Revisions.GetForAsync<T>(id, (pageNumber - 1) * pageSize, pageSize);
double totalResults = items.Count;
// get total results in case items count equals page size
if (pageSize <= items.Count || pageNumber > 1)
{
GetRevisionsCommand command = new GetRevisionsCommand(id, 0, 1, true);
session.Advanced.RequestExecutor.Execute(command, session.Advanced.Context);
command.Result.Results.Parent.TryGet("TotalResults", out totalResults);
}
Problem with this approach is that I need an additional request just to get the TotalResults property which indeed has already been returned by the first request. I just don't know how to access it.
You are right the TotalResults is returned from server but not parsed on the client side.
I opened an issue to fix that: https://issues.hibernatingrhinos.com/issue/RavenDB-15552
You can also get the total revisions count for document by using the /databases/{dbName}/revisions/count?id={docId} endpoint, client code for example:
var docId = "company/1";
var dbName = store.Database;
var json = await store.GetRequestExecutor().HttpClient
.GetAsync(store.Urls.First() + $"/databases/{dbName}/revisions/count?id={docId}").Result.Content.ReadAsStringAsync();
using var ctx = JsonOperationContext.ShortTermSingleUse();
using var obj = ctx.ReadForMemory(json, "revision-count");
obj.TryGet("RevisionsCount", out long revisionsCount);
Another way could be getting all the revisions:
var revisions = await session.Advanced.Revisions.GetForAsync<Company>(docId, 0, int.MaxValue);
Then using the revisions.Count as the total count.
Related
I am using the following script to add rows of files from a student loop in the Google spreadsheet if credits are less than x. The script was working good but as the data in the spreadsheet is being added daily, now the script is throwing "Exceeded maximum execution time" error (we have more than 2000 files). As I am new to scripting I don't know how to optimize the code.
Could someone help me to optimize the code or any solution so that the execution time take less than 5 min. Every time you compare to an email, it has to be compared to many emails. Please Help!
function updated() {
//Final file data (Combined)
var filecombined = SpreadsheetApp.openById("XXXXXXXXXX");
var sheet2 = filecombined.getSheets();
//Folder with all the files
var parentFolder = DriveApp.getFolderById("YYYYYYYYYYYY");
var files = parentFolder.getFiles();
//Current Date
var fecha = new Date();
//Path for each file in the folder
while (files.hasNext()) {
var idarchivo = files.next().getId();
var sps = SpreadsheetApp.openById(idarchivo);
var sheet = sps.getSheetByName('STUDENT PROFILE');
var data = sheet.getDataRange().getValues();
var credits = data[5][1];
//Flat; bandera:1 (new row), bandera:2 (update row)
var bandera = 1;
//Take data from final file (Combined)
var data2 = sheet2[0].getDataRange().getValues();
//If credits are less than X: write
if (credits < 120) {
var email = data[2][1];
var lastrow = filecombined.getLastRow();
var u = 0;
//comparison loop by email, if found it, update and exit the loop
while (u < lastrow) {
u = u + 1;
if (email == data2[u - 1][1]) {
sheet2[0].getRange(u, 3).setValue(credits);
sheet2[0].getRange(u, 4).setValue(fecha);
u = lastrow;
bandera = 2;
}
}
//if that email does not exist, write a new row
if (bandera == 1) {
var nombre = data[0][1];
sheet2[0].getRange(lastrow + 1, 1).setValue(nombre);
sheet2[0].getRange(lastrow + 1, 2).setValue(email);
sheet2[0].getRange(lastrow + 1, 3).setValue(credits);
sheet2[0].getRange(lastrow + 1, 4).setValue(fecha);
}
}
}
SpreadsheetApp.flush();
}
The questioner's code is taking taking more than 4-6 minutes to run and is getting an error Exceeded maximum execution time.
The following answer is based solely on the code provided by the questioner. We don't have any information about the 'filecombined' spreadsheet, its size and triggers. We are also in the dark about the various student spreadsheets, their size, etc, except that we know that there are 2,000 of these files. We don't know how often this routine is run, nor how many students have credits <120.
getvalues and setvalues statements are very costly; typically 0.2 seconds each. The questioners code includes a variety of such statements - some are unavoidable but others are not.
In looking at optimising this code, I made two major changes.
1 - I moved line 27 var data2 = sheet2[0].getDataRange().getValues();
This line need only be executed once and I relocated it at the top of the code just after the various "filecombined" commands. As it stood, this line was being executed once for every student spreadsheet; this along may have contributed to several minutes of execution time.
2) I converted certain setvalue commands to an array, and then updated the "filecombined" spreadsheet from the array once only, at the end of the processing. Depending on the number of students with low credits and who are not already on the "filecombined" sheet, this may represent a substantial saving.
The code affected was lines 47 to 50.
line47: sheet2[0].getRange(lastrow+1, 1).setValue(nombre);
line48: sheet2[0].getRange(lastrow+1, 2).setValue(email);
line49: sheet2[0].getRange(lastrow+1, 3).setValue(credits);
line50: sheet2[0].getRange(lastrow+1, 4).setValue(fecha);
There are setvalue commands also executed at lines 38 and 39 (if the student is already on the "filecombined" spreadsheet), but I chose to leave these as-is. As noted above, we don't know how many such students there might be, and the cost of these setvalue commands may be minor or not. Until this is clear, and in the light of other time savings, I chose to leave them as-is.
function updated() {
//Final file data (Combined)
var filecombined = SpreadsheetApp.openById("XXXXXXXXXX");
var sheet2 = filecombined.getSheets();
//Take data from final file (Combined)
var data2 = sheet2[0].getDataRange().getValues();
// create some arrays
var Newdataarray = [];
var Masterarray = [];
//Folder with all the files
var parentFolder = DriveApp.getFolderById("YYYYYYYYYYYY");
var files = parentFolder.getFiles();
//Current Date
var fecha = new Date();
//Path for each file in the folder
while (files.hasNext()) {
var idarchivo = files.next().getId();
var sps = SpreadsheetApp.openById(idarchivo);
var sheet = sps.getSheetByName('STUDENT PROFILE');
var data = sheet.getDataRange().getValues();
var credits = data[5][1];
//Flat; bandera:1 (new row), bandera:2 (update row)
var bandera = 1;
//If credits are less than X: write
if (credits < 120){
var email = data[2][1];
var lastrow = filecombined.getLastRow();
var u = 0;
//comparison loop by email, if found it, update and exit the loop
while (u < lastrow) {
u = u + 1;
if (email == data2[u-1][1]){
sheet2[0].getRange(u, 3).setValue(credits);
sheet2[0].getRange(u, 4).setValue(fecha);
u = lastrow;
bandera = 2;
}
}
//if that email does not exist, write a new row
if(bandera == 1){
var nombre = data[0][1];
Newdataarray = [];
Newdataarray.push(nombre);
Newdataarray.push(email);
Newdataarray.push(credits);
Newdataarray.push(fecha);
Masterarray.push(Newdataarray);
}
}
}
// update the target sheet with the contents of the array
// these are all adding new rows
lastrow = filecombined.getLastRow();
sheet2[0].getRange(lastrow+1, 1, Masterarray.length, 4);
sheet2[0].setValues(Masterarray);
SpreadsheetApp.flush();
}
As I mentioned in my comment, the biggest issue you have is that you repeatedly search an array for a value, when you could use a much faster lookup function.
// Create an object that maps an email address to the (last) array
// index of that email in the `data2` array.
const knownEmails = data2.reduce(function (acc, row, index) {
var email = row[1]; // email is the 2nd element of the inner array (Column B on a spreadsheet)
acc[email] = index;
return acc;
}, {});
Then you can determine if an email existed in data2 by trying to obtain the value for it:
// Get this email's index in `data2`:
var index = knownEmails[email];
if (index === undefined) {
// This is a new email we didn't know about before
...
} else {
// This is an email we knew about already.
var u = ++index; // Convert the array index into a worksheet row (assumes `data2` is from a range that started at Row 1)
...
}
To understand how we are constructing knownEmails from data2, you may find the documentation on Array#reduce helpful.
I would like to develop an API using HAPI JS and MYSQL using limit and offset parameters
Could anyone please suggest me resources on how to develop this.
Thanks.
Example: GET /schools?offset=10&limit=20
Paging depends on very basic principles.
First, you need to know in a particular query, which you'll show in the page you have, how many rows do you have?
For example, you listing blog post on your page, get total row
SELECT count(*) FROM blog_post_table; // cosnt numRows = Query....,
Let's say it's 168;
Now, you need to decide how many rows will be shown in a page. On your example, there is a query parameter "limit=20" which answers our question.
const perPage = request.query.limit || 20;
Based on the limit parameter in the query string or at least 20 items per page, validation is your homework, Joi.
Then, we should calculate how many pages will be there based on our perPage information.
const totalPages = Match.ceil(numRows / perPage);
It results in 9, we used Math.ceil, you get it I assume, you can't divide 168 with 20 because there are 8 remaining items. The result w 8.4, means we need one more page to show remaining 8 items.
So far so good, now we have to calculate our offset, where we start from reading data. In the first page, of course, it starts from 0, but what about second or fourth or ninth?
Now we need a current page parameter, and we should carry this information in query strings (I discarded your offset=10 definition on purpose), let's say there is parameter in query string named page
const currentPage = request.query.page || 1; // easy, yet harmful, it needs validation
Now we know our current page, now we can calculate our limit based on those figures.
const offset = perPage * (currentPage - 1);
To realize, put numbers on the places
const offset = 20 * (1-1); // page 1 offset = 0;
const offset = 20 * (2-1); // page 2 offset = 20;
.
.
.
const offset = 20 * (9-1); // page 9 offset = 160;
Now we can make our main query based on our calculations.
const sql = `SELECT * FROM blog_post_table LIMIT ${perPage} OFFSET ${offset}`;
Fetch rows and send to your page. Some additional information will be useful if you want to show a proper paging interface.
Let's say your response data is like this;
const response = {
'results': ['array of elements from db'],
'currentPage': currentPage, // learn object literals, it unnecessary,
'nextPage': currentPage + 1, // don't be that naive, validate this,
'lastPage': totalPages, // it is obvious
'previousPage': currentPage - 1, // what if we're in the very first page, then we can go to 0, validate!
}
BONUS:
Little code that generates previous and next pages, not a perfect solution just made it up quickly.
const getOtherPages = (totalPages, currentPage) => {
previousPage = null
nextPage = null
if currentPage > totalPages or currentPage < 1:
return previousPage, nextPage
if currentPage > 1:
previousPage = currentPage - 1
if (currentPage + 1) < totalPages:
nextPage = currentPage + 1
return [previousPage, nextPage]
}
// Usage
const [previousePage, nextPage] = getOtherPages(totalPages, currentPage);
So I have a standard service reference proxy calss for MS CRM 2013 (i.e. right-click add reference etc...) I then found the limitation that CRM data calls limit to 50 results and I wanted to get the full list of results. I found two methods, one looks more correct, but doesn't seem to work. I was wondering why it didn't and/or if there was something I'm doing incorrectly.
Basic setup and process
crmService = new CrmServiceReference.MyContext(new Uri(crmWebServicesUrl));
crmService.Credentials = System.Net.CredentialCache.DefaultCredentials;
var accountAnnotations = crmService.AccountSet.Where(a => a.AccountNumber = accountNumber).Select(a => a.Account_Annotation).FirstOrDefault();
Using Continuation (something I want to work, but looks like it doesn't)
while (accountAnnotations.Continuation != null)
{
accountAnnotations.Load(crmService.Execute<Annotation>(accountAnnotations.Continuation.NextLinkUri));
}
using that method .Continuation is always null and accountAnnotations.Count is always 50 (but there are more than 50 records)
After struggling with .Continutation for a while I've come up with the following alternative method (but it seems "not good")
var accountAnnotationData = accountAnnotations.ToList();
var accountAnnotationFinal = accountAnnotations.ToList();
var index = 1;
while (accountAnnotationData.Count == 50)
{
accountAnnotationData = (from a in crmService.AnnotationSet
where a.ObjectId.Id == accountAnnotationData.First().ObjectId.Id
select a).Skip(50 * index).ToList();
accountAnnotationFinal = accountAnnotationFinal.Union(accountAnnotationData).ToList();
index++;
}
So the second method seems to work, but for any number of reasons it doesn't seem like the best. Is there a reason .Continuation is always null? Is there some setup step I'm missing or some nice way to do this?
The way to get the records from CRM is to use paging here is an example with a query expression but you can also use fetchXML if you want
// Query using the paging cookie.
// Define the paging attributes.
// The number of records per page to retrieve.
int fetchCount = 3;
// Initialize the page number.
int pageNumber = 1;
// Initialize the number of records.
int recordCount = 0;
// Define the condition expression for retrieving records.
ConditionExpression pagecondition = new ConditionExpression();
pagecondition.AttributeName = "address1_stateorprovince";
pagecondition.Operator = ConditionOperator.Equal;
pagecondition.Values.Add("WA");
// Define the order expression to retrieve the records.
OrderExpression order = new OrderExpression();
order.AttributeName = "name";
order.OrderType = OrderType.Ascending;
// Create the query expression and add condition.
QueryExpression pagequery = new QueryExpression();
pagequery.EntityName = "account";
pagequery.Criteria.AddCondition(pagecondition);
pagequery.Orders.Add(order);
pagequery.ColumnSet.AddColumns("name", "address1_stateorprovince", "emailaddress1", "accountid");
// Assign the pageinfo properties to the query expression.
pagequery.PageInfo = new PagingInfo();
pagequery.PageInfo.Count = fetchCount;
pagequery.PageInfo.PageNumber = pageNumber;
// The current paging cookie. When retrieving the first page,
// pagingCookie should be null.
pagequery.PageInfo.PagingCookie = null;
Console.WriteLine("#\tAccount Name\t\t\tEmail Address");while (true)
{
// Retrieve the page.
EntityCollection results = _serviceProxy.RetrieveMultiple(pagequery);
if (results.Entities != null)
{
// Retrieve all records from the result set.
foreach (Account acct in results.Entities)
{
Console.WriteLine("{0}.\t{1}\t\t{2}",
++recordCount,
acct.EMailAddress1,
acct.Name);
}
}
// Check for more records, if it returns true.
if (results.MoreRecords)
{
// Increment the page number to retrieve the next page.
pagequery.PageInfo.PageNumber++;
// Set the paging cookie to the paging cookie returned from current results.
pagequery.PageInfo.PagingCookie = results.PagingCookie;
}
else
{
// If no more records are in the result nodes, exit the loop.
break;
}
}
I have the following method to return search results based on a supplied query
private List<Item> GetResults(QueryBase qBase)
{
using (IndexSearchContext sc = SearchManager.GetIndex("story").CreateSearchContext())
{
var hits = sc.Search(qBase, int.MaxValue);
var h1 = hits.FetchResults(0, 25);
var h2 = h1.Select(r => r.GetObject<Item>());
var h3 = h2.Where(item => item != null);
return h3.ToList();
}
}
The index being searched indexes web and master content. If I pass in a query that I know matches a single published item and break at the line beginning var h2 = then I see the variable hits has 2 items. This I expect because actually the items are both the same item, one from web and one from master.
However, the variable h1 only has a single result. The result from web has been omitted.
This is the case whether I'm debugging in the context of web or master. Can anyone explain?
When fetching the items using FetchResults method, Sitecore groups the items from lucene by the id of the item. First of the items becomes a SearchResult in the resulting SearchResultCollection object, and other items become Subresults for this results.
For example if you have a home item with id {110D559F-DEA5-42EA-9C1C-8A5DF7E70EF9} with one published version and 4 versions in different languages for the home item, what you'll get from lucene is a single result and 4 subresults for this result:
using (IndexSearchContext sc = SearchManager.GetIndex("story").CreateSearchContext())
{
var hits = sc.Search(qBase, int.MaxValue);
var h1 = hits.FetchResults(0, 25);
foreach (SearchResult result in h1)
{
var url = result.Url;
foreach (SearchResult subresult in result.Subresults)
{
var subUrl = subresult.Url; // other versions of this item
}
}
}
The urls for results and subresults in my case would be:
sitecore://web/{110D559F-DEA5-42EA-9C1C-8A5DF7E70EF9}?lang=en&ver=1
sitecore://master/{110D559F-DEA5-42EA-9C1C-8A5DF7E70EF9}?lang=en&ver=1 (subresult)
sitecore://master/{110D559F-DEA5-42EA-9C1C-8A5DF7E70EF9}?lang=ja-JP&ver=1 (subresult)
sitecore://master/{110D559F-DEA5-42EA-9C1C-8A5DF7E70EF9}?lang=de-DE&ver=1 (subresult)
sitecore://master/{110D559F-DEA5-42EA-9C1C-8A5DF7E70EF9}?lang=da&ver=1 (subresult)
so for retrieving the all the items with their versions you can use this code:
private List<Item> GetResults(QueryBase qBase)
{
using (IndexSearchContext sc = SearchManager.GetIndex("story").CreateSearchContext())
{
var hits = sc.Search(qBase, int.MaxValue);
var h1 = hits.FetchResults(0, 25);
var h2 = h1.Select(r => r.GetObject<Item>()).ToList();
// add other versions of item to the resulting list
foreach (IEnumerable<SearchResult> subresults in h1.Select(sr => sr.Subresults))
{
h2.AddRange(subresults.Select(r => r.GetObject<Item>()));
}
var h3 = h2.Where(item => item != null);
return h3.ToList();
}
}
You can not assume with item will be returned as the first one from the lucene and which items will be returned as subresults. If you want to get any specific item you need to pass version number, language and database to the query.
I have a situation where my user is presented with a grid, and it, by default, will just get the first 15 results. However they may type in a name and search for an item across all pages.
Alone, either of these works fine, but I am trying to figure out how to make them work as a single query. This is basically what it looks like...
// find a filter if the user is searching
var filters = request.Filters.ToFilters();
// determine the name to search by
var name = filters.Count > 0 ? filters[0] : null;
// we need to be able to catch some query statistics to make sure that the
// grid view is complete and accurate
RavenQueryStatistics statistics;
// try to query the items listing as quickly as we can, getting only the
// page we want out of it
var items = RavenSession
.Query<Models.Items.Item>()
.Statistics(out statistics) // output our query statistics
.Search(n => n.Name, name)
.Skip((request.Page - 1) * request.PageSize)
.Take(request.PageSize)
.ToArray();
// we need to store the total results so that we can keep the grid up to date
var totalResults = statistics.TotalResults;
return Json(new { data = items, total = totalResults }, JsonRequestBehavior.AllowGet);
The problem is that if no name is given, it does not return anything; Which is not the desired result. (Searching by 'null' doesn't work, obviously.)
Do something like this:
var q= RavenSession
.Query<Models.Items.Item>()
.Statistics(out statistics); // output our query statistics
if(string.IsNullOrEmpty(name) == false)
q = q.Search(n => n.Name, name);
var items = q.Skip((request.Page - 1) * request.PageSize)
.Take(request.PageSize)
.ToArray();