Java/Mongo Query with geoLocation and within specified timeframe - mongodb-query

I am having some issues retrieving the correct date within a particular radius from MongoDB. I have a json example shown at the bottom. My goal is to search for all items close to a defined geolocation as well as filtering on a starttime. This also includes creating the correct index. There is currently not a lot guidance out and about. Thanks in advance for any help.
In summary, what I'm trying to do is:
retrieve all items from MongoDB that are within 3 miles of a provided geolocation
filter results for items with a starttime after x but before y.
Create an index that works for my query.
Regards,
Lance
If I just wanted to query on dates I could do the following:
DateTime maxdateTime = new DateTime(); //Joda Time
//Adjust my datetime, simply add 2 hours to the time
DateTime mindateTime = new DateTime(); Joda Time
//Adjust my datetime, subtract 1 day
Date maxDate = new Date(maxdateTime.getMillis());
Date minDate = new Date(mindateTime.getMillis());
DBCollection coll = db.getCollection("BikeRides");
coll.ensureIndex( { startTime:1, } )
BasicDBObject query = new BasicDBObject();
query.put("startTime", BasicDBObjectBuilder.start("$gte", minDate).add("$lte", maxDate).get());
DBCursor cursor = coll.find(query);
try {
while(cursor.hasNext()) {
System.out.println(cursor.next());
}
} finally {
cursor.close();
}
If I just wanted to query on GeoLocation I could do the following:
//A GeoLoc is passed into this method
int distance = 3;
int earthRadius = 3963.192;
DBCollection coll = db.getCollection("BikeRides");
coll.ensureIndex( { Location.GeoLoc : "2d" } )
db.runCommand( { geoNear: "BikeRides",
nearSphere: [ "Location.GeoLoc.latitude": geoLoc.longitude, "Location.GeoLoc.longitude": geoLoc.latitude ]
maxDistance: distance / earthRadius
} )
I attempted this with Jongo as well but without any success:
DateTime nowDateTime = new DateTime(DateTimeZone.UTC); // Joda time
DateTime maxStartTime = nowDateTime.plusMinutes(TIME_IN_MINUTES);
DateTime minStartTime = nowDateTime.minusDays(1); //This will cut out most old bike rides
Long now = nowDateTime.toInstant().getMillis();
Long max = maxStartTime.toInstant().getMillis();
Long min = minStartTime.toInstant().getMillis();
//Get the objects using Jongo
MongoCollection bikeRidesCollection = MongoDatabase.Get_DB_Collection(MONGO_COLLECTIONS.BIKERIDES, "Location.GeoLoc");
//Currently not placing a limit on this result. If we find this is an issue we can add later.
bikeRidesCollection.ensureIndex("{Location.GeoLoc: '2d', RideStartTime: 1}");
Iterable<BikeRide> all = bikeRidesCollection
.find("{Location.GeoLoc: {$near: [#, #], $maxDistance: #}, RideStartTime: {$lte: #, $gte: #}}", //, $maxDistance: #
location.getGeoLoc().longitude,
location.getGeoLoc().latitude,
RADIUS_IN_MILES/EarthRadiusInMILES,
max ,
min )
.as(BikeRide.class);
List<BikeRide> closeBikeRides = Lists.newArrayList(all);
My sample Json:
{
"BikeRides": [
{
"StartTime": "2013-03-08T00:01:00.000 UTC",
"bikeRideName": "Strawberry Ride",
"id": "513afc2d0364b81b8abfa86e",
"location": {
"city": "Portland",
"geoLoc": {
"longitude": "-122.71446990966797",
"latitude": "45.49216842651367"
},
"state": "OR",
"streetAddress": "4214 SE 36th"
},
"startTime": "2013-03-07T16:01:00-08:00"
}
]
}

Solved. To get Jongo to work I simply needed to use the correct maxDistance. Instead of EarthRadiusInMILES, I should have defined this; Double ONE_DEGREE_IN_MILES = 69.11;
Note: 1° of latitude = about 69.11 miles

Related

Filter datetime value within a JSON collection of a collection inside CosmoDB using SQL

Using Microsoft CosmoDBs SQL like syntax. I have a collection of entries that follow a schema like this (simplified for this post)
{"id":"123456",
"activities": {
"activityA": {
"loginType": "siteA",
"lastLogin": "2018-02-06T19:42:22.205Z"
},
"activityB": {
"loginType": "siteB",
"lastLogin": "2018-03-07T11:39:50.346Z"
},
"activityC": {
"loginType": "siteC",
"lastLogin": "2018-04-08T15:21:15.312Z"
}
}
}
Without knowing the exact index into the activities entry activities list/sub collection, how can I query to get back all items in the Cosmo db collection that have a "lastLogin" matching a date range?
If I only wanted to search on the first item in the activities list, I could do something like this using index 0.
SELECT * FROM c where (c.activities[0].lastLogin > '2018-01-01T00:00:00') and (c.activities[0].lastLogin <= '2019-02-15T00:00:00')
But I want to search all entries in the list. Would be nice if there was something like this:
SELECT * FROM c where (c.activities[?].lastLogin > '2018-01-01T00:00:00') and (c.activities[?].lastLogin <= '2019-02-15T00:00:00')
But that doesn't exist.
The answer is that you can not iterate over a non list collection. Had the collection item been structured like this
{"id":"123456",
"activities": [
{ "label": "activityA",
"loginType": "siteA",
"lastLogin": "2018-02-06T19:42:22.205Z"
},
{
"label": "activtyB",
"loginType": "siteB",
"lastLogin": "2018-03-07T11:39:50.346Z"
},
etc...
It would be easy to crease a UDF to iterate over with something like this
UDF: filterActivityList
function(activityList, targetDateTimeStart, targetDateTimeEnd) {
var s, _i, _len;
for (_i = 0, _len = activityList.length; _i < _len; _i++) {
s = activityList[_i];
if ((s.lastLogin >= targetDateTimeStart) && (s.lastLogin < targetDateTimeEnd))
{
return true;
}
}
return false;
}
Then to query:
select * from c WHERE udf.filterActivityList(c.activities, '2018-01-01T00:00:00', '2018-02-01T00:00:00');
If I were to leave the structure as a JSON hierarchy instead of converting it to a JSON list then I would have to write another udf to accept the top level node of the hierarchy as an input parameter and have it convert the notes under it to a list, then apply the udf.filterActivityList UDF to the result. From my experience this approach is resource intensive and takes a very long time for Cosmo to process.

React BigCalendar date conversion

I am using react-big-calendar. The events data has date start/end in epoch format. It doesn't render correctly. How can set the accessor properties to work with this JSON format?
actionItems =
[
{
"id": 3312,
"name": "Event Name",
"startDate": 1518415200000,
"endDate": 1519797600000,
"duration": "4 weeks",
},
]
my current calendar component declaration
<BigCalendar
events={actionItems}
views={allViews}
showMultiDayTimes
defaultDate={new Date()}
/>
You can use map function to get events in proper format
const mapToRBCFormat = e => Object.assign({}, e, {
start: new Date(e.startDate),
end: new Date(e.endDate))
})
<BigCalendar
events={actionItems.map(mapToRBCFormat)}
views={allViews}
showMultiDayTimes
defaultDate={new Date()}
/>
Epoch time is just a number and react big calendar accepts javascript data object. So you need to convert it into javascript date object using following.You can map function and use the object to render it and you need to multiply with 1000 to get time format. for more visit https://www.epochconverter.com/programming/#javascript
// availableSlots is your object.
var freeSlots = availableSlots.map(obj => {
var slotObj = {};
delete obj.duration;
slotObj['start'] = new Date(obj.start * 1000);
slotObj['end'] = new Date(obj.end * 1000);
slotObj['title'] = "Book"; // extra field
return slotObj;
});
Hope this works :)

Bluemix SQLDB Query - Can't figure out JSON Parameter Markings

In my Nodered Bluemix application, I'm trying to make a SqlDB query, but I can't find sufficient documentation or examples on how to use the parameter markings in the query. Are there any examples and further insight into what I am doing wrong? Here is the flow I am having trouble with:
[
{
"id":"7924a83a.03355",
"type":"websocket-listener",
"path":"/ws/dbdata",
"wholemsg":"false"
},
{
"id":"b84efad2.9a2a58",
"type":"function",
"name":"Parse JSON",
"func":"msg.payload = JSON.parse(msg.payload);\nvar begin = msg.payload[0].split(\" \");\nbegin[1] = begin[1]+\":00\";\nvar date1 = begin[0].split(\"-\");\nvar processStart = date1[2]+\"-\"+date1[0]+\"-\"+date1[1]+\" \"+begin[1];\n\nvar end = msg.payload[0].split(\" \");\nend[1] = end[1]+\":00\";\nvar date2 = end[0].split(\"-\");\nvar processEnd = date2[2]+\"-\"+date2[0]+\"-\"+date2[1]+\" \"+end[1];\n\nmsg.payload[0] = processStart;\nmsg.payload[1] = processEnd;\nreturn msg;",
"outputs":1,"noerr":0,"x":381.79998779296875,"y":164.8000030517578,"z":"3f9da5d2.b3f0aa",
"wires":[["4f92b16a.cf981"]]
},
{
"id":"3e20f8a4.06451",
"type":"websocket in",
"name":"dbInput",
"server":"7924a83a.03355",
"client":"",
"x":159.8000030517578,"y":164.8000030517578,"z":"3f9da5d2.b3f0aa",
"wires":[["b84efad2.9a2a58"]]
},
{
"id":"68a4a35.5983f5c",
"type":"debug",
"name":"",
"active":true,"console":"false",
"complete":"true",
"x":970.7999877929688,"y":162.8000030517578,"z":"3f9da5d2.b3f0aa",
"wires":[]
},
{
"id":"5a0aed1c.34279c",
"type":"sqldb in",
"service":"LabSensors-sqldb",
"query":"",
"params":"{msg.begin},{msg.end}",
"name":"db Request",
"x":787.7999877929688,"y":163.8000030517578,"z":"3f9da5d2.b3f0aa",
"wires":[["68a4a35.5983f5c"]]
},
{
"id":"e08c4a85.e95e68",
"type":"debug",
"name":"",
"active":true,"console":"false",
"complete":"true",
"x":791.7999877929688,"y":233.8000030517578,"z":"3f9da5d2.b3f0aa",
"wires":[]
},
{
"id":"4f92b16a.cf981",
"type":"function",
"name":"Construct Query",
"func":"msg.begin = msg.payload[0];\nmsg.end = msg.payload[1];\nmsg.payload = \"SELECT * FROM IOT WHERE TIME >= '?' AND TIME < '?'\";\nreturn msg;",
"outputs":1,"noerr":0,"x":583.7999877929688,"y":163.8000030517578,"z":"3f9da5d2.b3f0aa",
"wires":[["5a0aed1c.34279c",
"e08c4a85.e95e68"]]
}
]
In the node-red documentation for the SQLDB query node it says:
"Parameter Markers is a comma delimited set of json paths. These will replace any question marks that you place in your query, in the order that they appear."
Have you tried removing the curly braces, i.e. to set the "params" field in the node to just "msg.begin,msg.end"?
You just need remove single quotes this is a correct sentence:
msg.payload = "SELECT * FROM IOT WHERE TIME >= ? AND TIME < ?";

Counting cascaded distincts in RavenDB

I'm facing an index problem for which I can't see a solution yet.
I have the following document structure per board:
{
"Name": "Test Board",
...
"Settings": {
"Admins": [ "USER1", "USER2" ],
"Members": [ "USER3", "USER4", "USER5" ]
...
},
...
"CreatedBy": "USER1",
"CreatedOn": "2014-09-26T18:14:20.0858945"
...
}
Now I'd like to be able to retrieve the count of all users which are somewhere registered in a board. Of course this should not only count the number of user occurences but rather count the number of distinct users. One user can be member of multiple boards.
This operation should perform as fast as possible since it is displayed in a global statistics dashboard visible on each page. Therefor I chose to try it with an index instead of retrieving all boards and their users and do the work on client side.
Trying to achieve this by using a Map/Reduce index:
Map = boards => from board in boards
select new
{
Aggregation = "ALL",
Users = new object[]
{
board.CreatedBy,
board.Settings.Admins,
board.Settings.Members
},
NumberOfUsers = 1
};
Reduce = results => from res in results
group res by new
{
res.Aggregation
}
into g
select new
{
g.Key.Aggregation,
Users = g.Select(x => x.Users),
NumberOfUsers = g.Sum(x => x.Users.Length)
};
Obviously this results in a wrong count. I don't have any experience with Reduce yet so I appreciate any tip! The solution will be probably pretty easy...
What would be the best way to globally distinct CreatedBy, Admins and Members of all documents and return the count?
Use an index like this:
from board in docs.Boards
select new
{
Users = board.Settings.Admins.Count + board.Settings.Members.Count + 1 /* created by */
}
from r in results
group r by "all" into g
select new
{
Users = g.Sum(x=>x.Users)
}
The best I could come up so far is:
Map = boards => from board in boards
select new
{
Users = new object[]
{
board.CreatedBy,
board.Settings.Admins,
board.Settings.Members
}
};
Reduce = results => from r in results
group r by "all" into g
select new
{
Users = g.SelectMany(x => x.Users)
};
And then query for the distinct user count:
var allUsersQuery = _documentSession.Query<AllUsersIndex.Result, AllUsersIndex>();
return allUsersQuery.Any() ? allUsersQuery.First().Users.Distinct().Count() : 0;
At least the query only returns a list of all usernames on all boards instead of bigger object trees. But the uniqueness still has to be done client-side.
If there is any better way please let me know. It would be beautiful to have only one integer returned from the server...
Then use this:
from board in docs.Boards
from user in board.Settings.Admins.Concat(board.Settings.Members).Concat(new[]{board.CreatedBy})
select new
{
User = user,
Count = 1
}
from r in results
group r by r.User into g
select new
{
User = g.Key,
Count = g.Sum(x=>x.Count)
}
I'm not really happy about the fanout, but this will give you all the discint users and the number of times they appear.
If you want just the number of distinct users, just get the total results from the index.

facets with ravendb

i am trying to work with the facet ability in ravendb but getting strange results.
i have a documents like :
{
"SearchableModel": "42LC2RR ",
"ModelName": "42LC2RR",
"ModelID": 490578,
"Name": "LG 42 Television 42LC2RR",
"Desctription": "fffff",
"Image": "1/4/9/8/18278941c",
"MinPrice": 9400.0,
"MaxPrice": 9400.0,
"StoreAmounts": 1,
"AuctionAmounts": 0,
"Popolarity": 3,
"ViewScore": 0.0,
"ReviewAmount": 2,
"ReviewScore": 45,
"Sog": "E-TV",
"SogID": 1,
"IsModel": true,
"Manufacrurer": "LG",
"ParamsList": [
"1994267",
"46570",
"4134",
"4132",
"4118",
"46566",
"4110",
"180676",
"239517",
"750771",
"2658507",
"2658498",
"46627",
"4136",
"169941",
"169846",
"145620",
"169940",
"141416",
"3190767",
"3190768",
"144720",
"2300706",
"4093",
"4009",
"1418470",
"179766",
"190025",
"170557",
"170189",
"43768",
"4138",
"67976",
"239516",
"3190771",
"141195"
],
}
where the ParamList each represents a property of the product and in our application we have in cache what each param represents.
when searching for a specific product i would like to count all the returning attributes to be able to add the amount of each item after the search.
After searching lg in televisions category i want to get :
Param:4134 witch is a representative of LCD and the amount :65.
but unfortunately i am getting strange results. only some params are counted and some not.
on some searchers where i am getting results back i dont get any amounts back.
i am using the latest stable version of RavenDB.
index :
from doc in docs
from param in doc.ParamsList
select new {Name=doc.Name,Description=doc.Description,SearchNotVisible = doc.SearchNotVisible,SogID=doc.SogID,Param =param}
facet :
DocumentStore documentStore = new DocumentStore { ConnectionStringName = "Server" };
documentStore.Initialize();
using (IDocumentSession session = documentStore.OpenSession())
{
List<Facet> _facets = new List<Facet>
{
new Facet {Name = "Param"}
};
session.Store(new FacetSetup { Id = "facets/Params", Facets = _facets });
session.SaveChanges();
}
usage example :
IDictionary<string, IEnumerable<FacetValue>> facets = session.Advanced.DatabaseCommands.GetFacets("FullIndexParams", new IndexQuery { Query = "Name:lg" }, "facets/Params");
i tried many variations without success.
does anyone have ideas what am i doing wrong ?
Thanks
Use this index, it should resolve your problem:
from doc in docs
select new {Name=doc.Name,Description=doc.Description,SearchNotVisible = doc.SearchNotVisible,SogID=doc.SogID,Param = doc.ParamsList}
What analyzer you set for "Name" field. I see you search by Name "lg". By default, Ravendb use KeywordAnalyzer, means you must search by exact name. You should set another analyzer for Name or Description field (StandardAnalyzer for example).