I'm attempting to query an IoT-hub for devices where reported and desired properties differ. The purpose of this is to be able to notify users when devices fail to update for a given period of time. Running this query
SELECT * FROM c WHERE properties.desired != properties.reported
generates alot of false positives since both desired and reported contains a $metadata property with timestamps that always differ.
So to be clear, I want to list all devices where any of the "real" values (not metadata) differ from desired to reported.
The simply workaround to avoid the '$' properties for comparing between the desired and reported properties is to create a separate complex object within the desired and reported properties. This complex object will represent a state between the real and shadow device.
Example:
"Config": {
"abc": 123,
"status": "inprocess",
"battery": {
"level": 90
}
}
In this case, the query string for query all devices where their Config is different from desired to reported properties looks the following:
SELECT deviceId FROM devices WHERE is_defined(properties.desired.Config) and is_defined(properties.reported.Config) and properties.desired.Config != properties.reported.Config
UPDATE:
Another option (workaround) is using an Azure IoT Hub eventing for changes in the device twin. These notification changes can be routed to the custom endpoint e.g. Event Hub and consumed by EventHubTrigger function. The routing query:
is_object($body.properties.reported) OR is_object($body.properties.desired)
The function can easy obtained a device twin and comparing its properties such as desired and reported after their cleanup metadata parts.
The result of the comparing properties can be stored in the device twin tags, e.g.:
"tags": {
"Watchdog": {
"timestamp": "2019-08-12T14:24:36.1805155Z",
"status": "inprocess"
}
}
Note, that the tags property is not visible by device.
Once we have a watchdog status in the device tags, we can query devices for its status, e.g.:
"query": "SELECT deviceId FROM devices WHERE is_defined(devices.tags.Watchdog) and devices.tags.Watchdog.status='inprocess' "
The following code snippet shows an example of the function:
using Microsoft.Azure.Devices;
using Microsoft.Azure.EventHubs;
using Microsoft.Azure.WebJobs;
using Microsoft.Extensions.Logging;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using System;
using System.Text;
using System.Threading.Tasks;
namespace FunctionApp14
{
public static class Function2
{
static RegistryManager registryManager = RegistryManager.CreateFromConnectionString(Environment.GetEnvironmentVariable("AzureIoTHubShariedAccessPolicy"));
[FunctionName("Function2")]
public static async Task Run([EventHubTrigger("%myTwinChanges%", Connection = "myTwinChangesEventHubConnectionString", ConsumerGroup = "local")]EventData message, ILogger log)
{
var msg = Encoding.UTF8.GetString(message.Body.Array);
log.LogInformation($"C# IoT Hub trigger function processed a message: {msg}");
if (message.SystemProperties["iothub-message-source"]?.ToString() == "twinChangeEvents")
{
var twinChnages = JsonConvert.DeserializeAnonymousType(msg, new { properties = new JObject() });
if (twinChnages?.properties != null)
{
// deviceId
var connectionDeviceId = message.SystemProperties["iothub-connection-device-id"].ToString();
// device twin
var twin = await registryManager.GetTwinAsync(connectionDeviceId);
// cleanup and compare the twin properties
twin.Properties.Desired.ClearMetadata();
twin.Properties.Reported.ClearMetadata();
var desired = JObject.Parse(twin.Properties.Desired.ToJson());
var reported = JObject.Parse(twin.Properties.Reported.ToJson());
var status = JToken.DeepEquals(desired, reported) ? "ok" : "inprocess";
log.LogWarning($"desired-reported status = {status}");
// put the state on the tags
var twinPatch = JsonConvert.SerializeObject(new { tags = new { Watchdog = new { timestamp = DateTime.UtcNow, status = status } } });
await registryManager.UpdateTwinAsync(connectionDeviceId, twinPatch, twin.ETag);
}
}
await Task.CompletedTask;
}
}
}
You might need to do this in code. I had a piece of code that I used for something similar, I put it in a GitHub repo for this question.
The code is pretty naive, it compares the string values of the properties (excluding the metadata). It should be pretty easy to change it to check the property keys/values if needed.
I am using Google diff-match-patch JAVA plugin to create patch between two JSON strings and storing the patch to database.
diff_match_patch dmp = new diff_match_patch();
LinkedList<Patch> diffs = dmp.patch_make(latestString, originalString);
String patch = dmp.patch_toText(diffs); // Store patch to DB
Now is there any way to use this patch to re-create the originalString by passing the latestString?
I google about this and found this very old comment # Google diff-match-patch Wiki saying,
Unpatching can be done by just looping through the diff, swapping
DIFF_INSERT with DIFF_DELETE, then applying the patch.
But i did not find any useful code that demonstrates this. How could i achieve this with my existing code ? Any pointers or code reference would be appreciated.
Edit:
The problem i am facing is, in the front-end i am showing a revisions module that shows all the transactions of a particular fragment (take for example an employee details), like which user has updated what details etc. Now i am recreating the fragment JSON by reverse applying each patch to get the current transaction data and show it as a table (using http://marianoguerra.github.io/json.human.js/). But some JSON data are not valid JSON and I am getting JSON.parse error.
I was looking to do something similar (in C#) and what is working for me with a relatively simple object is the patch_apply method. This use case seems somewhat missing from the documentation, so I'm answering here. Code is C# but the API is cross language:
static void Main(string[] args)
{
var dmp = new diff_match_patch();
string v1 = "My Json Object;
string v2 = "My Mutated Json Object"
var v2ToV1Patch = dmp.patch_make(v2, v1);
var v2ToV1PatchText = dmp.patch_toText(v2ToV1Patch); // Persist text to db
string v3 = "Latest version of JSON object;
var v3ToV2Patch = dmp.patch_make(v3, v2);
var v3ToV2PatchTxt = dmp.patch_toText(v3ToV2Patch); // Persist text to db
// Time to re-hydrate the objects
var altV3ToV2Patch = dmp.patch_fromText(v3ToV2PatchTxt);
var altV2 = dmp.patch_apply(altV3ToV2Patch, v3)[0].ToString(); // .get(0) in Java I think
var altV2ToV1Patch = dmp.patch_fromText(v2ToV1PatchText);
var altV1 = dmp.patch_apply(altV2ToV1Patch, altV2)[0].ToString();
}
I am attempting to retrofit this as an audit log, where previously the entire JSON object was saved. As the audited objects have become more complex the storage requirements have increased dramatically. I haven't yet applied this to the complex large objects, but it is possible to check if the patch was successful by checking the second object in the array returned by the patch_apply method. This is an array of boolean values, all of which should be true if the patch worked correctly. You could write some code to check this, which would help check if the object can be successfully re-hydrated from the JSON rather than just getting a parsing error. My prototype C# method looks like this:
private static bool ValidatePatch(object[] patchResult, out string patchedString)
{
patchedString = patchResult[0] as string;
var successArray = patchResult[1] as bool[];
foreach (var b in successArray)
{
if (!b)
return false;
}
return true;
}
In JavaScript my Worklight Client can pass arbitrary objects to an adapter:
var payload = { able: 3488, baker: "wibble"};
var invocationData = {
adapter : 'xxx',
procedure : 'xxx',
parameters : [payload],
compressResponse : false
};
var options = {
onSuccess: onCallSuccess,
onFailure: onCallFailure
};
WL.Client.invokeProcedure(invocationData, options);
and the adapter can access the object
function xxx(p1) {
return {'answer': p1.able};
}
In the native APIs it seems that we are limited to primitive types:
public void setParameters(java.lang.Object[] parameters)
This method
sets the request parameters. The order of the object in the array will
be the order sending them to the adapter
Parameters: Object -
parameters An array of objects of primitive types ( String, Integer,
Float, Boolean, Double). The order of the objects in the array is the
order in which they are sent to the adapter.
Hence if my adapters are to be used by both JavaScript and Native clients they will need to accept any complex objects as JSON Strings. Unless there is an alternative I'm missing?
I don't see a better alternative than simply stringifying the object as you have suggested. Are you using objects other than JSON in your native side? If so what is the structure of the object?
I'm very new to breeze/knockout, but I'm 99% of the way to doing what I need to do. I'm using the Hot Towel template, and I'm successfully retrieving a list of items via breeze. The entity (ITBAL) is a database first Entity Framework entity. When I look at the JSON coming back in Fiddler, I see the correct data. The problem is that all of the properties of data.results are dependentobservables, and not the raw values themselves.
We have a custom grid control that is trying to display the data.results array. Because it is not expecting observables, it is simply displaying "function dependentobservable" instead of the value.
I tried to unwrap the object, but keep getting the circular reference error. I don't know why that is, since ITBAL isn't associated with anything.
The data as Fiddler reports it:
[{"$id":"1","$type":"WebUIHtml5HotTowel.Models.ITBAL, WebUIHtml5HotTowel","IBITNO":"A100 ","IBWHID":"1 ","IBITCL":"50","IBITSC":"3 ","IBSUSP":" ","IBVNNO":"100 ","IBPRLC":" ","IBSCLC":" ","IBCCCD":"P","IBPICD":" ","IBSAFL":"Y","IBSTCS":399.99000,"IBUSCS":0.00000,"IBAVCS":414.95214,"IBLCST":7.00000,"IBLCCC":20.0,"IBLCDT":110923.0,"IBLSCC":20.0,"IBLSDT":130111.0,"IBLXCC":19.0,"IBLXDT":990102.0,"IBMXO1":2100.000,"IBMXO2":0.000,"IBMXO3":0.000,"IBMNO1":5.000,"IBMNO2":0.000,"IBMNO3":0.000,"IBFOQ1":0.000,"IBFOQ2":0.000,"IBFOQ3":0.000,"IBOHQ1":327.000,"IBOHQ2":0.000,"IBOHQ3":0.000,"IBAQT1":1576.000,"IBAQT2":0.000,"IBAQT3":0.000,"IBBOQ1":50.000,"IBBOQ2":0.000,"IBBOQ3":0.000,"IBPOQ1":448.000,"IBPOQ2":0.000,"IBPOQ3":0.000,"IBIQT1":1446.000,"IBIQT2":0.000,"IBIQT3":0.000,"IBRMD1":10.000,"IBRMD2":0.000,"IBRMD3":0.000,"IBRYD1":10.000,"IBRYD2":0.000,"IBRYD3":0.000,"IBISM1":0.000,"IBISM2":0.000,"IBISM3":0.000,"IBISY1":0.000,"IBISY2":0.000,"IBISY3":0.000,"IBAMD1":0.000,"IBAMD2":0.000,"IBAMD3":0.000,"IBAYD1":0.000,"IBAYD2":0.000,"IBAYD3":0.000,"IBMMD1":0.000,"IBMMD2":0.000,"IBMMD3":0.000,"IBMYD1":0.000,"IBMYD2":0.000,"IBMYD3":0.000,"IBSMD1":1.0,"IBSMD2":0.0,"IBSMD3":0.0,"IBSYD1":1.0,"IBSYD2":0.0,"IBSYD3":0.0,"IBBLME":335.000,"IBBLYO":2680.000,"IBBLLY":1441.000,"IBNMTY":8.0,"IBNMLY":11.0,"IBQSMD":21.000,"IBQSYD":21.000,"IBQSLY":20.000,"IBISMD":16318.19,"IBISYD":16318.19,"IBISLY":45714.87,"IBCSMD":373.46,"IBCSYD":373.46,"IBCSLY":67.00,"IBDQMD":0.000,"IBDQYD":0.000,"IBDQLY":0.000,"IBDSMD":0.00,"IBDSYD":0.00,"IBDSLY":0.00,"IBDCMD":0.00,"IBDCYD":0.00,"IBDCLY":0.00,"IBNOMD":18.0,"IBNOYD":18.0,"IBNOLY":18.0,"IBPKMD":15.0,"IBPKYD":15.0,"IBPKLY":14.0,"IBINUS":" ","IBIAID":0.0,"IBSAID":0.0,"IBCQT1":1527.000,"IBCQT2":0.000,"IBCQT3":0.000,"IBFCST":"Y","IBDRSH":" ","IBWMIU":"JP","IBFL15":" ","IBUS20":" ","IBLPR1":0.00000,"IBLPR2":0.00000,"IBLPR3":0.00000,"IBLPR4":0.00000,"IBLPR5":0.00000,"IBLPCD":" ","IBABCC":"B","IBPRCL":0.0,"IBQBCL":" ","IBACDC":"Y","IBTDCD":" ","IBDOUM":" ","IBTP01":0.0,"IBTP02":0.0,"IBTP03":0.0,"IBTP04":0.0,"IBLMCC":20.0,"IBLMDT":130513.0,"IBTMPH":"Y","IBCOMC":" ","IBCOMF":0.00000,"IBITCT":" ","IBEOQT":0.000,"IBITCM":0.0,"IBBRVW":" ","IBPTID":" ","IBQTLT":0.0000,"IBCTY1":"AUS","IBCTY2":"AUS","IBTXCD":"1","IBREVS":"Y","IBITXC":" ","IBMNOQ":0.000,"IBSTUS":0.000,"IBUS30":" ","IBPSLN":" ","IBPLIN":"N","IBUPDP":"Y","IBDFII":"2011-08-11T00:00:00.000","IBLHRK":"A","IBPLNC":" "}]
My Controller:
[BreezeController]
public class ItemInquiryController : ApiController
{
readonly EFContextProvider<AplusEntities> _contextProvider = new EFContextProvider<AplusEntities>();
[System.Web.Http.HttpGet]
public string Metadata()
{
return _contextProvider.Metadata();
}
[HttpGet]
public IQueryable<ITBAL> ItemBalances(string itemNumber, string warehouse)
{
return _contextProvider.Context.ITBALs.Where(i => i.IBITNO == itemNumber && i.IBWHID == warehouse)
.OrderBy(i => i.IBWHID)
.ThenBy(i => i.IBITNO);
}
}
The relevant portion from the viewmodel:
var manager = new breeze.EntityManager("api/ItemInquiry");
var store = manager.metadataStore;
var itbalInitializer = function (itbal) {
itbal.CompositeKey = ko.computed(function () {
return itbal.IBITNO() + itbal.IBWHID();
});
};
store.registerEntityTypeCtor("ITBAL", null, itbalInitializer);
var index = "0" + (args.pageNum * args.pageSize);
var query = new breeze.EntityQuery("ItemBalances")
.withParameters({ itemNumber: "A100", warehouse: "1" })
.take(args.pageSize);
if (index > 0) {
query = query.skip(index);
}
manager.executeQuery(query).then(function (data) {
vm.itbals.removeAll();
var itbals = data.results;//[0].Data;
itbals.forEach(function (itbal) {
vm.itbals.push(itbal);
});
vm.totalRecords(1);
itemBalancesGrid.mergeData(vm.itbals(), args.pageNum, parseInt(vm.totalRecords()));
}).fail(function (e) {
logger.log(e, null, loggerSource, true, 'error');
});
I figure I must be missing something fairly simple, but it is escaping me.
UPDATE: I removed the BreezeController attribute from the ApiController, and it works correctly.
Jon, removing the [Breeze] attribute effectively disables breeze for your application so I don't think that is the long term answer to your problem.
If you don't actually want entities for this scenario - you just want data - than a Breeze projection that mentions just the data to display in the grid would seem to be the best choice. Projections return raw data that are not wrapped in KO observables and are not held in the Breeze EntityManager cache.
If you want the data as cached entities and you also want to display them in a grid that doesn't like KO observable properties ... read on.
You can unwrap a KO'd object with ko.toJS. However, the grid is likely to complain about circular references (or throw an "out of memory" exception as some grids do) ... even if the entity has no circular navigation paths. The difficulty stems from the fact that every Breeze entity has an inherent circularity by way of its entityAspect property:
something.entityAspect.entity //'entity' points back to 'something'
Because you are using Knockout for your model library and because you say ITBAL has no navigation properties ("is not related to anything"), I think the following will work for you:
manager.executeQuery(query).then(success) ...
function success(data) {
var unwrapped = ko.toJS(data.results).map(
function(entity) {
delete entity.entityAspect;
return entity;
});
vm.itbals(unwrapped);
vm.totalRecords(1); // huh? What is that "parseInt ..." stuff?
itemBalancesGrid.mergeData(vm.itbals(), args.pageNum, parseInt(vm.totalRecords()));
})
ko.toJS is a Knockout function that recursively unwraps an object or collection of objects, returning copies of values. Then we iterate over the copied object graphs, deleting their entityAspect properties. The array of results is stuffed into the vm.itbals observable and handed along.
You can imagine how to generalize this to remove anything that is giving you trouble.
Extra
What the heck is vm.totalRecords? I sense that this is supposed to be the total number of matching records before paging. You can get that from Breeze by adding .inlineCount() to the breeze query definition. You get the value after the query returns from the data.inlineCount property.
Do you really need vm.itbals()? If all you do here is pass values to the grid, why not do that and cut out the middle man?
The following success callback combines these thoughts
function success(data) {
var unwrapped = ko.toJS(data.results).map(
function(entity) {
delete entity.entityAspect;
return entity;
});
itemBalancesGrid.mergeData(unwrapped, args.pageNum, data.inlineCount);
})
I apologize if this is a stupid/newb question, but when a Rally query result is returned with a _ref (using the Javascript SDK 1.32), is there a way to directly get the object associated with the _ref?
I see that I can use getRefFromTypeAndObjectId to get the type and the object ID, and then query on that type and object ID to get the object, however I wondered if there was something like getObjectFromRef or some other such way to more directly get back the object associated with the reference.
Excellent question. The getRallyObject method on RallyDataSource should do what you need.
var ref = '/defect/12345.js';
rallyDataSource.getRallyObject(ref, function(result) {
//got it
var name = result.Name;
}, function(response) {
//oh noes... errors
var errors = response.Errors;
});
In SDK 2.0 you use the load method of a data model to read a specific object. Check out this example: http://developer.help.rallydev.com/apps/2.0p5/doc/#!/guide/appsdk_20_data_models