just started using Keen.IO…very nice product. I have a question please on data modeling:
I am tracking mobile app registration events that have the following attributes:
Device Id,
Platform,
Manufacturer,
Marketing Version,
Language and
Latitude and Longitude
The problem is that the mobile app sends this registration every time the device changes location. In a way, it is to maintain the device latest location and keep track of historic changes. I created a new collection for devices and it seems to be OK as I am able to select_unique the keen.location.coordinates to draw the devices on a map.
The question though is how would I know the latest location update? So now I have several events in the collection for the same device (uniquely identified by a device id) but no way of finding the latest!
Thank you for any pointer.
Regards,
Khaled
It is technically possible to use existing Keen query types to retrieve the latest position of a device as you require, the problem is doing so is inefficient and doesn't scale well (mainly because Keen is designed to store/query event data, not entity data). You'd have to plug the Device Id you retrieve from the select_unique into individual queries for each device to find the latest location. Doing so would bump you up against concurrency and/or rate limiting fairly quickly.
That being said, that leaves you with a few options:
Use Keen's S3 integration. This feature writes all of your raw events to S3 for consumption however you see fit. You could use this data to create a pipeline to a separate entity database which would allow you to retrieve the device's latest location. Note: there is an extra charge for this feature.
Post device/location data to a separate entity database at the same time you're sending the event to Keen, and use that database as the source of the device's latest location.
Either of these options will work, it just depends on how you'd like to implement/manage data collection in your application.
I think you can achieve this by combining a select_unique query with a group_by clause and looking at the last item in the result.
Though, as terrhorn noted – it's not a scalable solution. I wouldn't use it for anything other than dashboarding.
Here's a quick example:
var query = new Keen.Query("select_unique", {
eventCollection: "devices",
targetProperty: "location",
group_by: "device_id"
});
The result will look something like this:
{
"result": [
{
"device_id": "4252f729-7bdc-a487-be15-984999a96683",
"result": [
"location_1",
"location_2"
]
}
]
}
NOTE: This is theoretical – I'm not sure if Keen sorts the result of the group_by by the order in which the events came in or not – but it's a good assumption to test. I don't see why they wouldn't.
Related
This is more of a general programming question.
I'm trying to create an app, think of it as a Yelp clone. I have most of it working but I'm missing one important feature. The data of the places around me. For now I'm only focused on food, so I'd like it if I search something like "Pizza", it'd show me all the pizza joints near me.
I was originally planning to use Google Places API. However if you havent heard, they're changing their pricing and lowering the free tier and upping the cost by a huge margin.
There's also the problem of saving the data. One workaround I saw a user suggest was to just keep using Google's API, but every time you make the query, store the data in your own DB as well (I only need address and name and latitude and longitude) so eventually, you'd have what you need in a sense. However I also want to have something like a simple rating system for each place like Yelp, but Google (and all other places like MapBox, Here Maps, etc) states something along the lines of "info from their API should not be stored or cached for more than 24hrs" but it's very broad and not specific.
So what I was planning to do was, call the Google API, grab the 3 info I need (Address, Name, Lat/Lng), add more fields to store the rating, likes, whatever else the user will add. Then store it in my database, but that doesn't seem like a solution now.
So does anyone have any ideas or advice? Or know of a service where I can get the details of all the food places? And if possible, can anyone confirm that storing the Name, Address, Lat&Lng is a violation of their policy since in my eyes, it's public data, but something like the rating that Google provides, or the pictures that Google provides, now that's Google property.
For obtaining places you can use OpenStreetMap, e.g. using Overpass API. Since larger traffic can be expected you should run your own database(s) instead of using the public APIs.
However OSM doesn't contain ratings. So you have to combine this data with some other publicly available rating system.
I've trying to setup ThingsBoard for a few days and up to now it's going well. I'm stuck on how to push attributes (sometimes in an aggregate mode - avg, sum, etc) and telemetry data from a Device to its related asset.
Any simple way to do it? I didn't find any API for such.
Thanks
The following is the API for pushing telemetry data for asset, but you also need to be atleast a customer user to use the api i.e need to set authorization token in header as x-authoization: Bearer {JWTtoken}
API: /api/plugins/telemetry/ASSET/{assetId}/timeseries/SERVER_SCOPE/
method : POST
request(example):
{
"ts": 1563456910227,
"values": {
"longitude": "78.381984",
"latitude": "17.448645",
"fuel": "30",
"speed": "42",
"vehicleType": "bus",
"status": "On route"
}
}
Thingsboard supports MQTT, CoAP and HTTP. See https://thingsboard.io/docs/user-guide/telemetry/#device-telemetry-upload-api for details.
I had a similar problem and was able to fix it by:
(1) in GUI adding a relation from the DEVICE to an associated ENTITY (in my case it was CUSTOMER entity Type)
N.B. here is a related question with a possible solution for creating the relation in code.
then
(2) updating the root rule chain in thingsboard (also via the GUI). Step by step instructions and a picture of my working rule chain canvas can be found in here.
After taking these steps I was not only able to see latest device telemetry show up on latest telemetry for the related customer in the GUI, but also was able to get back the device id in api response to my request for latest telemetry from all devices belonging to a customer.
In rule chain you can do it easily,
first filter the message to make sure you are selecting the right one, then use " Enrichment node" called (duplicate to related) then use save telemetry like attached.
enter image description here
Push Data from Device to Asset - Basic Example
You can leverage Rule Engine to do that in real time.
First of all you have to establish a relation between a device and an asset which can be done via the Web UI or REST APIs. Below a screenshot for asset Building A that contains the device Thermostat A:
To push Thermostat A data to the related asset Building A, use a Rule Chain where key steps are changing the message originator from device to related asset, and then saving timeseries (or attributes) for the asset. You can decide whether to perform a double saving (first on device and then on asset, as I've done in the rule chain below) or to store data only at asset level. I consider good practice to store raw data as soon as they're uploaded, and to do additional persistence after further computations if needed.
Push data from Device to Asset - Advanced Example
An advanced version of the above example is described here and it shows the calculation of delta temperature between outdoor and indoor thermostats (the devices) installed into a warehouse (the asset).
Professional Edition Rule Nodes
There are some rule nodes suitable for your purposes, such as Duplicate To Related or Aggregate Stream, but they are available only on the Thingsboard PE edition.
Concise:
Is there a way (with YouTrack-API or somehow) to change issue ID. Say from s-5 to s-30?
Full story:
We created YouTrack project to track scientific equipment, which we produce. One device - one issue. With custom fields, we created, it is VERY convenient way to handle "device stories".
The only problem we have, is that devices have their own id numbers. If we could change issue ID so, that it corresponds to device IDs, it would be very helpful. Especially helpful, when we reference a device from other issues and other projects, and to have links with real device ids, etc.
Currently it's not possible. There's quite an old feature request on this: https://youtrack.jetbrains.com/issue/JT-11067.
I want to query for user stories and the Initiative that each story falls under. My fetch property looks like:
...
model: 'UserStory',
fetch: ['Name', 'PortfolioItem', 'Parent'],
...
This fetches the PortfolioItem/Feature object, PortfolioItem/FeatureGroup object but not the PortfolioItem/Initiative. The FeatureGroup object does not show a 'Parent' property.
In short, how can I fetch the parent's parent without querying separately for Initiatives and comparing the '_ref' or something like that?
For server performance reasons the hierarchical relationships will only be populated for one level with each request. You'll have to make subsequent requests to build the remaining tiers.
If you hit the new Lookback API (unreleased when Kyle first answered, now in open preview), the snapshots returned include a field _ItemHierarchy which will include the ObjectID of all ancestors including all the way up through Portfolio Items.
You can find information on the LBAPI here. There is support for querying it in the App SDK 2.0's SnapshotStore. Note that SDK 2.0p6 (releasing soon) has some improvements.
Old, old question. But I recently did the same thing for the entire tree. I didn't use the lookback api. I created a model and stored the data and went through the parentage of each portfolio item.
Perhaps not the most efficient way, but it works.
How unique is MPMediaItemPropertyPersistentID? Will it even work when synching the list of IDs to another device connected to the same iTunes Account?
I want to implement a iCloud synchronized playlist solution which stores the IDs as a list and I need to know if this will be possible.
As per the documentation (emphasis mine)...
The value of the MPMediaItemPropertyPersistentID identifier persists
across application launches and across syncs that do not change the
sync status of the media item. The value is not guaranteed to
persist across a sync/unsync/sync cycle.
As such, given that it won't even persist on this basis I'd be surprised if it persisted across devices in a sufficiently robust manner, if at all.
In my understanding this is not possible. I think this ID is persistent only for each device. It is no unique identifier for a specific song in the iTunes Store. It is only a ID for your own synced songs.
As you read the documentation, you'll see how fragile this ID could be.
"The value is not guaranteed to persist across a sync/unsync/sync cycle."
So if you sync your song database with iTunes and maybe delete a song from your iOS device and the sync again and put it back to your device, you may not get the same ID again for this song. And for sure not across other devices.
So I think, what you're trying to do will not work, until you get a worldwide identical Identifier for each song of the iTunes catalogue or your own iTunes catalogue on the Mac (where the Mac has to handle the IDs).
The other answers are a little big vague, so here is an answer from my own experiences and tests:
1) You can't use MPMediaItemPropertyPersistentID to get an ID that is equal between devices.
2) The MPMediaItemPropertyPersistentID will change when the device is synced with another iTunes library or all music is removed from the device and then synced again.
The ID get's created and stored by iTunes when the song is synced on the device. If it's unsynced, the ID get's deleted.
In case somebody else lands here, like me, using a Google search:
I confirmed what middaparka said above after an iOS upgrade of my device, when my music app tried to use persistentIDs from before the upgrade. The IDs had changed, and I ended up (unwittingly) listening to many songs from my library that I don't normally listen to...
So I took middaparka's advice and constructed a persistentKey by exclusive-oring the hashes from title, artistName, albumTitle and duration. Building the persistentKey during Core Database initialization will be save time later, by avoiding multiple string comparisons when fetching items in "normal operating code."
The persistentKey strategy worked properly for songs. However, when I made a hash for albums from title, artist and releaseYear, I ended up with one collision.
I had two self-titled albums by different artists released in 1976. When the hashes for the album title and the artist were exclusive-ored, they cancelled each other out. I ended up using the hash for the duration instead of the artist, and that worked.
I may end up refining the algorithm for generating the persistentKeys later...