I have a URL that produces JSON,
{
"status": "success",
"totalRecords": 55,
"records": [
{
"timestamp": 1393418044341,
"load": 40,
"deviceId": 285
},
{
"timestamp": 1393418104337,
"load": 42,
"deviceId": 285
},
{
"timestamp": 1393418164328,
"load": 24.5,
"deviceId": 285
},
{
"timestamp": 1393418224322,
"load": 42.5,
"deviceId": 285
},
It goes on and on, producing data every 30 seconds or so.
I have used Pentaho data-integration to parse and extract each of the data and put them into individual groups - timestamp, load and deviceId.
When I saved this it produced a .ktr file.
From this i have used the report-designer to upload the .ktr file and make charts with the data and then I uploaded the charts to the BI Server.
BUT
Can I just take the data, feed it into the BI Server and produce charts, bypassing the report-designer?
Yes you can do this - and using report designer would definately be the wrong way.
However you've inadvertantly made the right choice in building the first bit in PDI! Thats a good move.
Next step is to install CTools, add your .ktr to a CDA datasource (within CDE) and then using CDE define your charts and finally a refresh interval on the dashboard.
There's lots of good CTools tutorials around if you havent used it yet - it is also easily installed from the marketplace, or via ctools-installer.sh
Related
I am new to Steam analytics and I need help here to achieve a specific task.
I have telemetry data coming from iot hub in this format.
Basically i will be getting machines telemetry data and the stage of the operations on that machine streamed to iot hub.
The stages will be indicated with tag ex:"stageid":"stage1". I need to calculate the time taken for each stage using stream analytics based on timestamp and stage tag.
packet Ex:
[{
"Payload": {
"devid": "01",
"locid": "loc01",
"machid": "mac01",
"stageid": "stage1",
"timestamp": "2020-01-24T09:22:00.3270000Z"
},
"Payload": {
"devid": "02",
"locid": "loc01",
"machid": "mac01",
"stageid": "stage1",
"timestamp": "2020-01-24T09:22:00.3270000Z"
}
}]
[{
"Payload": {
"devid": "01",
"locid": "loc01",
"machid": "mac01",
"stageid": "stage2",
"timestamp": "2020-01-24T09:26:00.3270000Z"
},
"Payload": {
"devid": "02",
"locid": "loc01",
"machid": "mac01",
"stageid": "stage2",
"timestamp": "2020-01-26T09:24:00.3270000Z"
}
}]
pls help me can we achieve this with query and what could be the query or what is the other best approach?
Thanks,
Per my knowledge,your needs can't be implemented by ASA built-in features. ASA is a real-time collect data and analytics service.In other words,data need to be processed in the real-time.Current data can't wait for next dataset to do some calculate or merge things. Even if you could use windows function and group by,i believe the frequency of messages pushed by the device is also variable.
As a workaround,my idea is using iot-hub azure function trigger.Inside trigger,you could use code to parse message and save key columns(stageid,timestamp,devid) into some storage,maybe azure table storage. Before every insert,just grab latest row of current device to calculate the time taken with current message so that you could produce that time to store in other residence.In the end, update the latest row for every device.
in all Amadeus Self-Service APIs only some Airport codes work. For example "FRA" works, but "TXL" doesn't. Is this because the API is in Beta and I only use the Sandbox version?
Example:
https://test.api.amadeus.com/v1/shopping/flight-destinations?origin=FRA&oneWay=false&nonStop=false
WORKS
https://test.api.amadeus.com/v1/shopping/flight-destinations?origin=TXL&oneWay=false&nonStop=false
{
"errors": [
{
"status": 500,
"code": 141,
"title": "SYSTEM ERROR HAS OCCURRED",
"detail": "DATA DOMAIN NOT FOUND FOR REQUEST"
}
]
}
The APIs available in the test environment have a limit set of data (cache or fake data).
In the test environment, this API doesn't have data for TXL as origin, for Germany you have FRA and MUC.
So far our data set covers more the US and some big cities in the world. We will publish soon the list of available data on our portal.
You can find the list of available data in the test environment on our GitHub page.
I'm using wit ai for a bot and I think it's amazing. However, I must provide the customer with screens in my web app to train and manage the app. And here I found a big problem (or maybe I'm just lost). The documentation of the REST API is not enough to design a client that acts like the wit console (not even close). it's like a tutorial of what endpoints you can hit and an overview of the parameters, but no clean explanation of the structure of the response.
For example, there is no endpoint to get the insights edge. Also and most importantly, no clear documentation about the response structure when hitting the message endpoints (i.e. the structure the returned entities: are they prebuilt or not, and if they are, is the value a string or an object or array, and what the object might contain [e.g. datetime]). Also the problem of the deprecated guide and the new guide (the new guide should be done and complete by now). I'm building parts of the code based on my testing. Sometimes when I test something new (like adding a range in the datetime entity instead of just a value), I get an error when I try to set the values to the user since I haven't parsed the response right, and the new info I get makes me modify the DB structure at my end sometimes.
So, the bottom line, is there a complete reference that I can implement a complete client in my web app (my web app is in Java by the way and I couldn't find a client library that handles the latest version of the API)? Again, the tool is AWESOME but the documentation is not enough, or maybe I'm missing something.
The document is not enough of course but I think its pretty straightforward. And from what I read there is response structure under "Return the meaning of a sentence".
It's response in JSON format. So you need to decode the response first.
Example Request:
$ curl -XGET 'https://api.wit.ai/message?v=20170307&q=how%20many%20people%20between%20Tuesday%20and%20Friday' \
-H 'Authorization: Bearer $TOKEN'
Example Response:
{
"msg_id": "387b8515-0c1d-42a9-aa80-e68b66b66c27",
"_text": "how many people between Tuesday and Friday",
"entities": {
"metric": [ {
"metadata": "{'code': 324}",
"value": "metric_visitor",
"confidence": 0.9231
} ],
"datetime": [ {
"value": {
"from": "2014-07-01T00:00:00.000-07:00",
"to": "2014-07-02T00:00:00.000-07:00"
},
"confidence": 1
}, {
"value": {
"from": "2014-07-04T00:00:00.000-07:00",
"to": "2014-07-05T00:00:00.000-07:00"
},
"confidence": 1
} ]
}
}
You can read more about response structure under Return the meaning of a sentence
I do an up scale with a code like this on an Azure SQL Database:
ALTER DATABASE [MYDATABASE] Modify (SERVICE_OBJECTIVE = 'S1');
How is it possible to know from a c# code when Azure has completed the job and the table is already available?
Checking for SERVICE_OBJECTIVE value is not enough, the process still continue further.
Instead of performing this task in T-SQL I would perform the task from C# using an API call over to the REST API, you can find all of the details on MSDN.
Specifically, you should look at the Get Create or Update Database Status API method which allows you to call the following URL:
GET https://management.azure.com/subscriptions/{subscription-id}/resourceGroups/{resource-group-name}/providers/Microsoft.Sql/servers/{server-name}/databases/{database-name}}/operationResults/{operation-id}?api-version={api-version}
The JSON body allows you to pass the following parameters:
{
"id": "{uri-of-database}",
"name": "{database-name}",
"type": "{database-type}",
"location": "{server-location}",
"tags": {
"{tag-key}": "{tag-value}",
...
},
"properties": {
"databaseId": "{database-id}",
"edition": "{database-edition}",
"status": "{database-status}",
"serviceLevelObjective": "{performance-level}",
"collation": "{collation-name}",
"maxSizeBytes": {max-database-size},
"creationDate": "{date-create}",
"currentServiceLevelObjectiveId":"{current-service-id}",
"requestedServiceObjectiveId":"{requested-service-id}",
"defaultSecondaryLocation": "{secondary-server-location}"
}
}
In the properties section, the serviceLevelObjective property is the one you can use to resize the database. To finish off you can then perform a GET on the Get Database API method where you can compare both the currentServiceLevelObjectiveId and requestedServiceObjectiveId properties to ensure your command has been successful.
Note: Don't forget to pass all of the common parameters required to make API calls in Azure.
I updated our BigQuery client to new client Google API and suddenly I started seeing this error when uploading via JSON:
"errors": [
{
"reason": "invalid",
"location": "Offset:0 / Line:1 / Column:159 / Field:q1",
"message": "Could not convert value to string"
},
Job reference:
"jobReference": {
"projectId": "dot-metrics",
"jobId": "job_8e0511a40c1845cca5717daf78b605f7"
},
This worked before we updated our client, afterwards it just stopped working so it must be some change inside BigQuery. Any help is appreciated!
This looks like a regression in a recent release that broke importing null values in json. A fix should be forthcoming.
Note if you drop the null fields (i.e. instead of "field: null" you just don't include "field" at all) it should continue to work.