Repast: query an agent set and count the number of agents in while loop - repast-simphony

I want to achieve a logic like this:
while (count loading_docks with [status == "free"] > 0 and trucks with [status == "free" and type == "20'" and capacity < 1000] > 0) {
match a truck satisfying above 3 condidtions to a free dock for unloading cargo;
}
as can be seen, the query needs to be repetively called and updated in the while loop, and the second query is composed of 3 conditions (Which is not easy with AndQuery() method).
This is very easy to implement in Netlogo. What is the suitable and shorter way to achieve in repast?
UPDATE - the initial attempt
public void match_dock() {
for (Truck t: this.getTruck_queue()) {
if (this.Count_freeDock() > 0) {
Query<Object> fit_dock = new AndQuery(
new PropertyEquals(context, "status", 1),
new PropertyGreaterThanEquals(context, "max_veh", t.getTruck_type()));
double min = 10000;
Dock match = null;
for(Object o: fit_dock.query()) {
if (((Dock)o).getMax_veh() < min) {
match = (Dock)o;
}
}
match.setStatus(2);
match.getServe_list().add(t.getReq_id());
t.setServe_dock(match.getId());
// if (t.getServe_dock() != -1) {
// this.getTruck_queue().remove(t);
// }
}
}
}
public int Count_freeDock() {
List<Dock> free_list = new ArrayList<Dock>();
Query<Object> free_dock = new PropertyEquals<Object>(context, "status", 1);
for (Object o : free_dock.query()) {
if (o instanceof Dock) {
free_list.add((Dock)o);
}
}
return free_list.size();
}
There are three issues to fix:
1) The query of a particular agent set has to consider three conditions; AndQuery only composes two conditions. is there a Query method which allows more than two conditions to be considered at the same time?
current problem:
Query<Object> pre_fit = new AndQuery(
new PropertyEquals(context, "status", 1),
new PropertyGreaterThanEquals(context, "max_veh", t.getTruck_type()));
Query<Object> fit_dock = new AndQuery(pre_fit, new PropertyEquals(context, "ops_type", 3));
The initial composition of two conditions works fine and queries fast. However, when I add the third condition "ops_type", the query speed becomes hugely slow. What's the reason behind? Or is this a correct way to compose three conditions?
2) Is there simpler way to query the size (count) of a particular agent set, other than writing a custom count function (as shown in example)?
3) what is the shortest way to add(or copy) the queried agent set into a list for related list operations?
update entire code block:
public void match_dock() {
Iterator<Truck> truck_list = this.getTruck_queue().iterator();
while(truck_list.hasNext() && this.Count_freeDock() > 0) {
Truck t = truck_list.next();
// Query<Object> pre_fit = new AndQuery(
// new PropertyEquals(context, "status", 1),
// new PropertyGreaterThanEquals(context, "max_veh", t.getTruck_type()));
// Query<Object> ops_fit = new OrQuery<>(
// new PropertyEquals(context, "ops_type", 3),
// new PropertyEquals(context, "ops_type", this.getOps_type(t.getOps_type())));
// Query<Object> fit_dock = new AndQuery(pre_fit, new PropertyEquals(context, "ops_type", 3));
// Query<Object> fit_dock = new AndQuery(pre_fit, ops_fit);
Query<Object> pre_fit = new AndQuery(
new PropertyEquals(context, "status", 1),
new PropertyGreaterThanEquals(context, "max_veh", t.getTruck_type()));
Query<Object> q = new PropertyEquals(context, "ops_type", 3);
double min = 10000;
Dock match = null;
for (Object o : q.query(pre_fit.query())) {
// for(Object o: fit_dock.query()) {
if (((Dock)o).getMax_veh() < min) {
match = (Dock)o;
}
}
try {
match.setStatus(2);
match.getServe_list().add(t.getReq_id());
t.setServe_dock(match.getId());
if (t.getServe_dock() != -1) {
System.out.println("truck id " + t.getReq_id() + "serve dock: " + t.getServe_dock());
t.setIndock_tm(this.getTick());
truck_list.remove();
}
}
catch (Exception e){
// System.out.println("No fit dock found");
}
}
}
public int Count_freeDock() {
List<Dock> free_list = new ArrayList<Dock>();
Query<Object> free_dock = new PropertyEquals<Object>(context, "status", 1);
for (Object o : free_dock.query()) {
if (o instanceof Dock) {
free_list.add((Dock)o);
}
}
// System.out.println("free trucks: " + free_list.size());
return free_list.size();
}
UPDATE on 5/5
I have moved the query outside the while loop for better detection.
I found the slow speed could be largely due to the use of "PropertyGreaterThanEquals". regardless whether the queried field is int or double.
when you query using "PropertyGreaterThanEquals", the query runs very slow regardless wether the queried field is int or double. However, it returns correct result.
when you query using "PropertyEquals", the query runs in less than one second regardless wether the queried field is int or double. however, it returns result which is not correct since it needs to consider ">=".
public void match_dock() {
System.out.println("current tick is: " + this.getTick());
Iterator<Truck> truck_list = this.getTruck_queue().iterator();
Query<Object> pre_fit = new AndQuery(
new PropertyEquals(context, "status", 1),
new PropertyGreaterThanEquals(context, "max_veh", 30));
//new PropertyEquals(context, "max_veh", 30));
Query<Object> q = new PropertyEquals(context, "hv_spd", 240);
for (Object o : q.query(pre_fit.query())) {
if (o instanceof Dock) {
System.out.println("this object is: " + ((Dock)o).getId());
}
}
}

For 1, you could try chaining the queries like so:
Query<Object> pre_fit = new AndQuery(
new PropertyEquals(context, "status", 1),
new PropertyGreaterThanEquals(context, "max_veh", t.getTruck_type()));
Query<Object> q = new PropertyEquals(context, "ops_type", 3);
for (Object o : q.query(pre_fit.query())) { ...
I think this can be faster than the embedding the AndQuery, but I'm not entirely sure.
For 2, I think some of the Iterables produced by a Query are in fact Java Sets. You could try to cast to one of those and then call size(). If its not a set then you do in fact have to iterate as the query filter conditions are actually applied as part of the iteration.
For 3, I think there are some Java methods for this. new ArrayList(Iterable), and some methods in Collections.

Related

Google BigQuery returns only partial table data with C# application using .net Client Library

I am trying to execute the query (Basic select statement with 10 fields). My table contains more than 500k rows. C# application returns the response with only 4260 rows. However Web UI returns all the records.
Why my code returns only partial data, What is the best way to select all the records and load into C# Data Table? If there is any code snippet it would be more helpful to me.
using Google.Apis.Auth.OAuth2;
using System.IO;
using System.Threading;
using Google.Apis.Bigquery.v2;
using Google.Apis.Bigquery.v2.Data;
using System.Data;
using Google.Apis.Services;
using System;
using System.Security.Cryptography.X509Certificates;
namespace GoogleBigQuery
{
public class Class1
{
private static void Main()
{
try
{
Console.WriteLine("Start Time: {0}", DateTime.Now.ToString());
String serviceAccountEmail = "SERVICE ACCOUNT EMAIL";
var certificate = new X509Certificate2(#"KeyFile.p12", "notasecret", X509KeyStorageFlags.Exportable);
ServiceAccountCredential credential = new ServiceAccountCredential(
new ServiceAccountCredential.Initializer(serviceAccountEmail)
{
Scopes = new[] { BigqueryService.Scope.Bigquery, BigqueryService.Scope.BigqueryInsertdata, BigqueryService.Scope.CloudPlatform, BigqueryService.Scope.DevstorageFullControl }
}.FromCertificate(certificate));
BigqueryService Service = new BigqueryService(new BaseClientService.Initializer()
{
HttpClientInitializer = credential,
ApplicationName = "PROJECT NAME"
});
string query = "SELECT * FROM [publicdata:samples.shakespeare]";
JobsResource j = Service.Jobs;
QueryRequest qr = new QueryRequest();
string ProjectID = "PROJECT ID";
qr.Query = query;
qr.MaxResults = Int32.MaxValue;
qr.TimeoutMs = Int32.MaxValue;
DataTable DT = new DataTable();
int i = 0;
QueryResponse response = j.Query(qr, ProjectID).Execute();
string pageToken = null;
if (response.JobComplete == true)
{
if (response != null)
{
int colCount = response.Schema.Fields.Count;
if (DT == null)
DT = new DataTable();
if (DT.Columns.Count == 0)
{
foreach (var Column in response.Schema.Fields)
{
DT.Columns.Add(Column.Name);
}
}
pageToken = response.PageToken;
if (response.Rows != null)
{
foreach (TableRow row in response.Rows)
{
DataRow dr = DT.NewRow();
for (i = 0; i < colCount; i++)
{
dr[i] = row.F[i].V;
}
DT.Rows.Add(dr);
}
}
Console.WriteLine("No of Records are Readed: {0} # {1}", DT.Rows.Count.ToString(), DateTime.Now.ToString());
while (true)
{
int StartIndexForQuery = DT.Rows.Count;
Google.Apis.Bigquery.v2.JobsResource.GetQueryResultsRequest SubQR = Service.Jobs.GetQueryResults(response.JobReference.ProjectId, response.JobReference.JobId);
SubQR.StartIndex = (ulong)StartIndexForQuery;
//SubQR.MaxResults = Int32.MaxValue;
GetQueryResultsResponse QueryResultResponse = SubQR.Execute();
if (QueryResultResponse != null)
{
if (QueryResultResponse.Rows != null)
{
foreach (TableRow row in QueryResultResponse.Rows)
{
DataRow dr = DT.NewRow();
for (i = 0; i < colCount; i++)
{
dr[i] = row.F[i].V;
}
DT.Rows.Add(dr);
}
}
Console.WriteLine("No of Records are Readed: {0} # {1}", DT.Rows.Count.ToString(), DateTime.Now.ToString());
if (null == QueryResultResponse.PageToken)
{
break;
}
}
else
{
break;
}
}
}
else
{
Console.WriteLine("Response is null");
}
}
int TotalCount = 0;
if (DT != null && DT.Rows.Count > 0)
{
TotalCount = DT.Rows.Count;
}
else
{
TotalCount = 0;
}
Console.WriteLine("End Time: {0}", DateTime.Now.ToString());
Console.WriteLine("No. of records readed from google bigquery service: " + TotalCount.ToString());
}
catch (Exception e)
{
Console.WriteLine("Error Occurred: " + e.Message);
}
Console.ReadLine();
}
}
}
In this Sample Query get the results from public data set, In table contains 164656 rows but response returns 85000 rows only for the first time, then query again to get the second set of results. (But not known this is the only solution to get all the results).
In this sample contains only 4 fields, even-though it does not return all rows, in my case table contains more than 15 fields, I get response of ~4000 rows out of ~10k rows, I need to query again and again to get the remaining results for selecting 1000 rows takes time up to 2 minutes in my methodology so I am expecting best way to select all the records within single response.
Answer from User #:Pentium10
There is no way to run a query and select a large response in a single shot. You can either paginate the results, or if you can create a job to export to files, then use the files generated in your app. Exporting is free.
Step to run a large query and export results to files stored on GCS:
1) Set allowLargeResults to true in your job configuration. You must also specify a destination table with the allowLargeResults flag.
Example:
"configuration":
{
"query":
{
"allowLargeResults": true,
"query": "select uid from [project:dataset.table]"
"destinationTable": [project:dataset.table]
}
}
2) Now your data is in a destination table you set. You need to create a new job, and set the export property to be able to export the table to file(s). Exporting is free, but you need to have Google Cloud Storage activated to put the resulting files there.
3) In the end you download your large files from GCS.
It my turn to design the solution for better results.
Hoping this might help someone. One could retrieve next set of paginated result using PageToken. Here is the sample code for how to use PageToken. Although, I liked the idea of exporting for free. Here, I write rows to flat file but you could add them to your DataTable. Obviously, it is a bad idea to keep large DataTable in memory though.
public void ExecuteSQL(BigqueryService bqservice, String ProjectID)
{
string sSql = "SELECT r.Dealname, r.poolnumber, r.loanid FROM [MBS_Dataset.tblRemitData] R left join each [MBS_Dataset.tblOrigData] o on R.Dealname = o.Dealname and R.Poolnumber = o.Poolnumber and R.LoanID = o.LoanID Order by o.Dealname, o.poolnumber, o.loanid limit 100000";
QueryRequest _r = new QueryRequest();
_r.Query = sSql;
QueryResponse _qr = bqservice.Jobs.Query(_r, ProjectID).Execute();
string pageToken = null;
if (_qr.JobComplete != true)
{
//job not finished yet! expecting more data
while (true)
{
var resultReq = bqservice.Jobs.GetQueryResults(_qr.JobReference.ProjectId, _qr.JobReference.JobId);
resultReq.PageToken = pageToken;
var result = resultReq.Execute();
if (result.JobComplete == true)
{
WriteRows(result.Rows, result.Schema.Fields);
pageToken = result.PageToken;
if (pageToken == null)
break;
}
}
}
else
{
List<string> _fieldNames = _qr.Schema.Fields.ToList().Select(x => x.Name).ToList();
WriteRows(_qr.Rows, _qr.Schema.Fields);
}
}
The Web UI automatically flattens the data. This means that you see multiple rows for each nested field.
When you run the same query via the API, it won't be flattened, and you get fewer rows, as the nested fields are returned as objects. You should check if this is the case at you.
The other is that indeed you need to paginate through the results. Paging through list results has this explained.
If you want to do only one job, than you should write your query ouput to a table, than export the table as JSON, and download the export from GCS.

Mvel dynamic expression

Do you know if it's possible to dynamically evaluate an expression with Mvel. For example :
VariableResolverFactory functionFactory = new MapVariableResolverFactory();
MVEL.eval("def SUM(op1,op2,op3) { result=0B; if(op1) result+=op2; else result+=op3; } return result; ",functionFactory);
ParserContext ctx = new ParserContext()
Serializable s = MVEL.compileExpression("SUM(op1,op2,op3)", ctx);
contextMapFct.put("op1", "5 > 3"); // just as an example if it's useless
contextMapFct.put("op2", new BigDecimal(10));
contextMapFct.put("op3", new BigDecimal(30));
Object obj= MVEL.executeExpression(s, contextMapFct, this.functionFactory);
Few changes done
1.) Brace added at the last, was closing before return result;.
2.) int result=0, declaration added.
3.) if(op1 == 'true'), it is not coming as boolean, but as String
VariableResolverFactory functionFactory = new MapVariableResolverFactory();
MVEL.eval(
"def SUM(op1,op2,op3) { int result=0; if(op1 == 'true') result+=op2; else result+=op3; return result; }",
functionFactory);
ParserContext ctx = new ParserContext();
Serializable s = MVEL.compileExpression("SUM(op1,op2,op3)", ctx);
Map contextMapFct = new HashMap();
contextMapFct.put("op1", "5 > 3"); // just as an example if it's useless
contextMapFct.put("op2", new BigDecimal(10));
contextMapFct.put("op3", new BigDecimal(30));
Object obj = MVEL.executeExpression(s, contextMapFct, functionFactory);
System.out.println(obj);
output
30

Why does this controller double the inserts when I try to archive the results of the Bing Search API?

I'm trying to archive my search results for a term by
Using the Bing API in an async controller
Inserting them into database using Entity Framework
using the Bing API and insert them into a database using entity framework. For whatever reason it is returning 50 results, but then it enters 100 results into the database.
My Controller Code:
public class DHWebServicesController : AsyncController
{
//
// GET: /WebService/
private DHContext context = new DHContext();
[HttpPost]
public void RunReportSetAsync(int id)
{
int iTotalCount = 1;
AsyncManager.OutstandingOperations.Increment(iTotalCount);
if (!context.DHSearchResults.Any(xx => xx.CityMarketComboRunID == id))
{
string strBingSearchUri = #ConfigurationManager.AppSettings["BingSearchURI"];
string strBingAccountKey = #ConfigurationManager.AppSettings["BingAccountKey"];
string strBingUserAccountKey = #ConfigurationManager.AppSettings["BingUserAccountKey"];
CityMarketComboRun cityMarketComboRun = context.CityMarketComboRuns.Include(xx => xx.CityMarketCombo).Include(xx => xx.CityMarketCombo.City).First(xx => xx.CityMarketComboRunID == id);
var bingContainer = new Bing.BingSearchContainer(new Uri(strBingSearchUri));
bingContainer.Credentials = new NetworkCredential(strBingUserAccountKey, strBingAccountKey);
// now we can build the query
Keyword keyword = context.Keywords.First();
var bingWebQuery = bingContainer.Web(keyword.Name, "en-US", "Moderate", cityMarketComboRun.CityMarketCombo.City.Latitude, cityMarketComboRun.CityMarketCombo.City.Longitude, null, null, null);
var bingWebResults = bingWebQuery.Execute();
context.Configuration.AutoDetectChangesEnabled = false;
int i = 1;
DHSearchResult dhSearchResult = new DHSearchResult();
List<DHSearchResult> lst = new List<DHSearchResult>();
var webResults = bingWebResults.ToList();
foreach (var result in webResults)
{
dhSearchResult = new DHSearchResult();
dhSearchResult.BingID = result.ID;
dhSearchResult.CityMarketComboRunID = id;
dhSearchResult.Description = result.Description;
dhSearchResult.DisplayUrl = result.DisplayUrl;
dhSearchResult.KeywordID = keyword.KeywordID;
dhSearchResult.Created = DateTime.Now;
dhSearchResult.Modified = DateTime.Now;
dhSearchResult.Title = result.Title;
dhSearchResult.Url = result.Url;
dhSearchResult.Ordinal = i;
lst.Add(dhSearchResult);
i++;
}
foreach (DHSearchResult c in lst)
{
context.DHSearchResults.Add(c);
context.SaveChanges();
}
AsyncManager.Parameters["message"] = "The total number of results was "+lst.Count+". And there are " + context.DHSearchResults.Count().ToString();
}
else
{
AsyncManager.Parameters["message"] = "You have already run this report";
}
AsyncManager.OutstandingOperations.Decrement(iTotalCount);
}
public string RunReportSetCompleted(string message)
{
string str = message;
return str;
}
}
Here is how I am calling it from my asp.net mvc 4 page.
#Ajax.ActionLink("Run Report", "GatherKeywordsFromBing", "DHWebServices",
new { id=item.CityMarketComboRunID},
new AjaxOptions { OnSuccess = "ShowNotifier();", UpdateTargetId = "TopNotifierMessage", HttpMethod = "POST", InsertionMode = InsertionMode.Replace, LoadingElementId = strCityMarketComboProgressID, LoadingElementDuration = 1000 },
new { #class = "ViewLink" })
<span class="ProgressIndicator" id="#strCityMarketComboProgressID"><img src="#Url.Content("~/Content/img/SmallBall.gif")" alt="loading" /></span>
For whatever reason all of
Try saving only once:
foreach (DHSearchResult c in lst)
{
context.DHSearchResults.Add(c);
}
context.SaveChanges();
Also there's nothing asynchronous in your code, so there's no point of using asynchronous controller. Not only that it won't improve anything but it might make things worse.

RavendDB faceted search results formatting

Currently I'm playing with faceted search after reading RavenDB doc about it.
The result returned is OK, but there's a small problem. Since the result comes as
IDictionary<string, IEnumerable<FacetValue>>
it's necessary to iterate over it and do some fancy string manipulation to format the result and show it in a PartialView. More specifically this facet:
new Facet
{
Name = "Value_Range",
Mode = FacetMode.Ranges,
Ranges =
{
"[NULL TO Dx500.0]",
"[Dx500.0 TO Dx1000.0]",
"[Dx1000.0 TO Dx2500.0]",
"[Dx2500.0 TO Dx5000.0]",
"[Dx5000.0 TO NULL]",
}
}
View code:
#fv.Range
This is this "beautiful" string that gets output on the view: [Dx400.0 TO Dx600.0]
RavenDB uses the Dx prefix above to do a number to string conversion.
Controller code where the facet result is passed to a specific ViewModel:
var facetResults = DocumentSession.Query<Realty>("RealtyFacets")
//.Where(x => x.Value >= 100 && x.Value <= 1000)
.ToFacets("facets/RealtyFacets").ToArray();
var model = new RealtyFacetsViewModel();
model.Cities = facetResults[0];
model.Purposes = facetResults[1];
model.Types = facetResults[2];
model.Values = facetResults[3];
return PartialView("RealtyFacets", model);
Is there any other/better way of getting results from a faceted search so that no string manipulation must be done to format the returned data?
After Ayende's suggestion, I did this in my controller:
foreach (var val in facetResults[3].Value)
{
switch(val.Range)
{
case "[Dx0.0 TO Dx200.0]":
val.Range = string.Format("{0:C2} {1} {2:C2}",
0, #Localization.to, 200);
break;
case "[Dx200.0 TO Dx400.0]":
val.Range = string.Format("{0:C2} {1} {2:C2}",
200, #Localization.to, 400);
break;
case "[Dx400.0 TO Dx600.0]":
val.Range = string.Format("{0:C2} {1} {2:C2}",
400, #Localization.to, 600);
break;
case "[Dx600.0 TO Dx800.0]":
val.Range = string.Format("{0:C2} {1} {2:C2}",
600, #Localization.to, 800);
break;
case "[Dx800.0 TO Dx1000000.0]":
val.Range = string.Format("{0:C2} {1} {2:C2}",
800, #Localization.to, 1000000);
break;
}
}
model.Values = facetResults[3];
As per #MattWarren suggestion, I ended up using:
foreach (var val in facetResults[3].Value)
{
// Original string format: [Dx5000.0 TO Dx10000.0]
var limits = val.Range.Split(new string[] { "TO", "[", "]", " " },
StringSplitOptions.RemoveEmptyEntries);
// Leveraging RavenDB NumberUtil class...
val.Range = string.Format("{0:C0} {1} {2:C0}",
Raven.Abstractions.Indexing.NumberUtil.StringToNumber(limits.ElementAt(0)),
#Localization.to,
Raven.Abstractions.Indexing.NumberUtil.StringToNumber(limits.ElementAt(1)));
}
Leniel,
In your code, create a dictionary that would map between the facet value and the display string.
RavenDB currently have no way to influence the facet value.

How can I do this all in one query? Nhibernate

I have a list of Ids and I want to get all the rows back in one query. As a list of objects(So a List of Products or whatever).
I tried
public List<TableA> MyMethod(List<string> keys)
{
var query = "SELECT * FROM TableA WHERE Keys IN (:keys)";
var a = session.CreateQuery(query).SetParameter("keys", keys).List();
return a; // a is a IList but not of TableA. So what do I do now?
}
but I can't figure out how to return it as a list of objects. Is this the right way?
List<TableA> result = session.CreateQuery(query)
.SetParameterList("keys", keys)
.List<TableA>();
Howeever there could be a limitation in this query if number of ":keys" exceed more than 1000 (incase of oracle not sure with other dbs) so i would recommend to use ICriteria instead of CreateQuery- native sqls.
Do something like this,
[TestFixture]
public class ThousandIdsNHibernateQuery
{
[Test]
public void TestThousandIdsNHibernateQuery()
{
//Keys contains 1000 ids not included here.
var keys = new List<decimal>();
using (ISession session = new Session())
{
var tableCirt = session.CreateCriteria(typeof(TableA));
if (keys.Count > 1000)
{
var listsList = new List<List<decimal>>();
//Get first 1000.
var first1000List = keys.GetRange(0, 1000);
//Split next keys into 1000 chuncks.
for (int i = 1000; i < keys.Count; i++)
{
if ((i + 1)%1000 == 0)
{
var newList = new List<decimal>();
newList.AddRange(keys.GetRange(i - 999, 1000));
listsList.Add(newList);
}
}
ICriterion firstExp = Expression.In("Key", first1000List);
ICriterion postExp = null;
foreach (var list in listsList)
{
postExp = Expression.In("Key", list);
tableCirt.Add(Expression.Or(firstExp, postExp));
firstExp = postExp;
}
tableCirt.Add(postExp);
}
else
{
tableCirt.Add(Expression.In("key", keys));
}
var results = tableCirt.List<TableA>();
}
}
}