Explanation of retry logic in Azure Durable Functions - error-handling

I'm new to Azure Durable Functions and am trying to understand the retry logic and error handling.
I have a very simple orchestration function that executes 100 action functions in a fan-in fan-out pattern. My expectation is that when an action function breaks for whatever reason it is retried based on the retry options. In my case I'm expecting that I get a count of 100 in the final orchestration line (Write-Information $results.count), but for every action function that errors out due to my random error throwing, it seems no retry is done and the results is always less than 100.
Why is this happening, also why am I seeing the orchestrator output $results.count multiple times?
Orchestration
param($Context)
$range = 1..100
$retryOptions = New-DurableRetryOptions -FirstRetryInterval (New-TimeSpan -Seconds 1) -MaxNumberOfAttempts 10
$tasks =
foreach ($num in $range) {
try{
Invoke-DurableActivity -FunctionName 'RandomWaitActivity' -Input $num -NoWait -RetryOptions $retryOptions
}
catch{
Write-Output $_.Exception.Message
}
}
$results = Wait-ActivityFunction -Task $tasks
Write-Information $results.count
Action
param($number)
Write-Output "Received $number"
$random = Get-Random -Minimum 1 -Maximum 20
if($random -eq 13){
Throw "13 is a unlucky number"
}
Start-Sleep -Milliseconds $random
return $number

Well, If you want to add retry in your, you can configure retry policy in your function.json.
In this retry policy you can set maxRetryCount which represent the number of times a function will retry before stopping.
function.json :-
{
"bindings": [
{
"authLevel": "anonymous",
"type": "httpTrigger",
"direction": "in",
"name": "Request",
"methods": [
"get",
"post"
]
},
{
"type": "http",
"direction": "out",
"name": "Response"
}
],
"retry": {
"strategy": "fixedDelay",
"maxRetryCount": 1,
"delayInterval": "00:00:10"
}
}
Here Since I have set the retry count to 1 the function will try to execute twice.
Refer this MsDOC on retry policy.

Related

Get-AzPolicyExemption and -Match

I am trying to loop around all of our subscriptions and get Policy Exemptions, but only get the ones that we have created. The loop appears fine, but the Match element appears to bring back some Exemptions that don't meet the -Match criteria.
$allSubscriptions = Get-AzSubscription
$baseFolder = "C:\source\PowerShell Exemptions Dump\"
# loop subscriptions
foreach($sub in $allSubscriptions){
$subName = $sub.Name
# Get Exemptions at Sub level
Set-AzContext -Subscription $subName
# Write to File
$exemptionsIn = Get-AzPolicyExemption|ConvertTo-Json
$fileName = $baseFolder + $subName + ".json"
$exemptionsOut = ''
foreach($ex in $exemptionsIn|ConvertFrom-Json){
if($ex.Properties.PolicyAssignmentId -Match "abc") {
$exemptionsOut += $ex|ConvertTo-Json
}
}
if ($exemptionsOut -ne '') {
$exemptionsOut | Out-File -filepath $fileName
$exemptionsOut = ''
}
}
It does work to a certain extent i.e. if a Subscription has a 0% match in everything it brings back, then it doesn't create a file. but it appears if it finds one match, then it saves Exemptions to the file that don't match.
Here is some example Json that was saved to one of the files:
[
{
"Properties": {
"PolicyAssignmentId": "/providers/Microsoft.Management/managementGroups/abc-mg/providers/Microsoft.Authorization/policyAssignments/abc-mg",
"PolicyDefinitionReferenceIds": "",
"ExemptionCategory": "Waiver",
"DisplayName": "abc - abc-mg Policy Assignment",
"Description": "AIB Testing",
"ExpiresOn": "\/Date(1662134400000)\/",
"Metadata": ""
},
"SystemData": null,
"Name": "456",
"ResourceId": "/subscriptions/123/providers/Microsoft.Authorization/policyExemptions/789",
"ResourceName": "456",
"ResourceGroupName": null,
"ResourceType": "Microsoft.Authorization/policyExemptions",
"SubscriptionId": "123"
},
{
"Properties": {
"PolicyAssignmentId": "/providers/Microsoft.Management/managementGroups/root-mg/providers/Microsoft.Authorization/policyAssignments/111",
"PolicyDefinitionReferenceIds": "installEndpointProtection",
"ExemptionCategory": "Waiver",
"DisplayName": "root-mg - Azure Security Benchmark",
"Description": "currently use sophos and not defender",
"ExpiresOn": null,
"Metadata": ""
},
"SystemData": null,
"Name": "345",
"ResourceId": "/providers/Microsoft.Management/managementGroups/root-mg/providers/Microsoft.Authorization/policyExemptions/345",
"ResourceName": "345",
"ResourceGroupName": null,
"ResourceType": "Microsoft.Authorization/policyExemptions",
"SubscriptionId": null
}
]
Finally, I don't appear to get all Exemptions back in this loop i.e. some are set at Resource Group or Resource Level. Do I need to drill further beyond Set-AzContext?
After reproducing the same code from my end, I could able to see the expected results. However, make sure you are checking in the right file and the location to which you are sending your data to.
Finally, I don't appear to get all Exemptions back in this loop i.e. some are set at Resource Group or Resource Level.
This might be due to the scope that you are looking into. After setting the scope to the required level I could able to get the expected results. Below is the code that worked for me.
$Resource = Get-AzResource -ResourceGroupName <YOUR_RESOURCEGROUP_NAME>
for($I=0;$I -lt $Resource.ResourceId.Count;$I++)
{
$a=Get-AzPolicyExemption -Scope $Resource.ResourceId[$I]
for($J=0;$J -lt $a.Count;$J++)
{
If($a.ResourceId[$J] -Match $Resource.ResourceId[$I])
{
$exemptionsIn = Get-AzPolicyExemption -Scope $Resource.ResourceId[$I] | ConvertTo-Json
$fileName = "sample2" + ".json"
$exemptionsOut = ''
foreach($ex in $exemptionsIn|ConvertFrom-Json){
if($ex.Properties.PolicyAssignmentId -Match "Swetha*") {
$exemptionsOut += $ex|ConvertTo-Json
}
}
if ($exemptionsOut -ne '') {
$exemptionsOut | Out-File -filepath $fileName
$exemptionsOut = ''
}
}
}
}
I have few policy exemptions in my subscription but above script gave me the results at Resource level which -Match with Swetha.
RESULTS:

Karate API : * def from response is throwing syntax error

I am setting up an E2E test and chaining my request/responses. I am defining variables from each response and using them in the next call.
Its working up to a point, and then a problem surfaces when defining off the 2nd response.
If I def operationId, operationSubject, or operationStatus (e.g response.operationId), it works.
If I store anything from the results (e.g response.results.0.personId) it throws this error
Expected ; but found .0
response.results.0.personId
My response:
{
"operationId": "922459ecxxxxx",
"operationSubject": "BATCH_ENROLLMENT",
"operationStatus": "PROCESSED",
"results": {
"0": {
"personId": "367a73b5xxxx",
"status": "PRE_AUTH",
"email": "mquinter+TEST.69387488#email.com",
"loanNumber": null
},
"1": {
"personId": "56f060fd-e34xxxxxx",
"status": "PRE_AUTH",
"email": "mquintxxxx#email.com",
"loanNumber": null
}
}
}
That's not how to access data in JSON. See this similar question: https://stackoverflow.com/a/71847841/143475
Maybe you meant to do this:
* def foo = response.results[0].personId
https://stackoverflow.com/users/143475/peter-thomas
I see the issue - It wasn't finding the response because I wasn't giving it enough time before the next call.
I put a sleep in there and its working as expected.
Thanks

Handling multiple rows returned by IMPORTJSON script on GoogleSheets

I am trying to populate a google sheet using an API. But the API has more than one row to be returned for a single query. Following is the JSON returned by API.
# https://api.dictionaryapi.dev/api/v2/entries/en/ABANDON
[
{
"word": "abandon",
"phonetics": [
{
"text": "/əˈbændən/",
"audio": "https://lex-audio.useremarkable.com/mp3/abandon_us_1.mp3"
}
],
"meanings": [
{
"partOfSpeech": "transitive verb",
"definitions": [
{
"definition": "Cease to support or look after (someone); desert.",
"example": "her natural mother had abandoned her at an early age",
"synonyms": [
"desert",
"leave",
"leave high and dry",
"turn one's back on",
"cast aside",
"break with",
"break up with"
]
},
{
"definition": "Give up completely (a course of action, a practice, or a way of thinking)",
"example": "he had clearly abandoned all pretense of trying to succeed",
"synonyms": [
"renounce",
"relinquish",
"dispense with",
"forswear",
"disclaim",
"disown",
"disavow",
"discard",
"wash one's hands of"
]
},
{
"definition": "Allow oneself to indulge in (a desire or impulse)",
"example": "they abandoned themselves to despair",
"synonyms": [
"indulge in",
"give way to",
"give oneself up to",
"yield to",
"lose oneself in",
"lose oneself to"
]
}
]
},
{
"partOfSpeech": "noun",
"definitions": [
{
"definition": "Complete lack of inhibition or restraint.",
"example": "she sings and sways with total abandon",
"synonyms": [
"uninhibitedness",
"recklessness",
"lack of restraint",
"lack of inhibition",
"unruliness",
"wildness",
"impulsiveness",
"impetuosity",
"immoderation",
"wantonness"
]
}
]
}
]
}
]
By using the following calls via IMPORTJSON,
=ImportJSON(CONCATENATE("https://api.dictionaryapi.dev/api/v2/entries/en/"&$A2), "/phonetics/text", "noHeaders")
=ImportJSON(CONCATENATE("https://api.dictionaryapi.dev/api/v2/entries/en/"&$A2), "/meanings/partOfSpeech", "noHeaders")
=ImportJSON(CONCATENATE("https://api.dictionaryapi.dev/api/v2/entries/en/"&$A2), "/meanings/definitions/definition", "noHeaders")
=ImportJSON(CONCATENATE("https://api.dictionaryapi.dev/api/v2/entries/en/"&$A2), "/meanings/definitions/synonyms", "noHeaders")
=ImportJSON(CONCATENATE("https://api.dictionaryapi.dev/api/v2/entries/en/"&$A2), "/meanings/definitions/example", "noHeaders")
I am able to get the following in GoogleSheets,
Whereas, the actual output according to JSON should be,
As you can see a complete row is being overwritten. How can this be fixed?
EDIT
Following is the link to sheet for viewing only.
I believe your goal as follows.
You want to achieve the bottom image in your question on Google Spreadsheet.
Unfortunately, I couldn't find the method for directly retrieving the bottom image using ImportJson. So in this answer, I would like to propose a sample script for retrieving the values you expect using Google Apps Script. I thought that creating a sample script for directly achieving your goal might be simpler rather than modifying ImportJson.
Sample script:
function SAMPLE(url) {
var res = UrlFetchApp.fetch(url, {muteHttpExceptions: true});
if (res.getResponseCode() != 200) return res.getContentText();
var obj = JSON.parse(res.getContentText());
var values = obj[0].meanings.reduce((ar, {partOfSpeech, definitions}, i) => {
definitions.forEach(({definition, example, synonyms}, j) => {
var v = [definition, Array.isArray(synonyms) ? synonyms.join(",") : synonyms, example];
var phonetics = obj[0].phonetics[i];
ar.push(j == 0 ? [(phonetics ? phonetics.text : ""), partOfSpeech, ...v] : ["", "", ...v]);
});
return ar;
}, []);
return values;
}
When you use this script, please put =SAMPLE(CONCATENATE("https://api.dictionaryapi.dev/api/v2/entries/en/"&$A2)) to a cell as the custom formula.
Result:
When above script is used, the following
Note:
In this sample script, when the structure of the JSON object is changed, it might not be able to be used. So please be careful this.
References:
Class UrlFetchApp
Custom Functions in Google Sheets

TableData.insertAll with templateSuffix - frequent 503 errors

We are using TableData.insertAll with a templateSuffix and are experiencing frequent 503 errors with our usage pattern.
We set the templateSuffix based on two pieces of information - the name of the event being inserted and the data of the event being inserted. E.g. 'NewPlayer20160712'. The table ID is set to 'events'.
In most cases this works as expected, but relatively often it will fail and return an error. Approximately 1 in every 200 inserts will fail, which seems way too often for expected behaviour.
The core of our event ingestion service looks like this:
//Handle all rows in rowsBySuffix
async.mapLimit(Object.keys(rowsBySuffix), 5, function(suffix) {
//Construct request for suffix
var request = {
projectId: "tactile-analytics",
datasetId: "discoducksdev",
tableId: "events",
resource: {
"kind": "bigquery#tableDataInsertAllRequest",
"skipInvalidRows": true,
"ignoreUnknownValues": true,
"templateSuffix": suffix, // E.g. NewPlayer20160712
"rows": rowsBySuffix[suffix]
},
auth: jwt // valid google.auth.JWT instance
};
//Insert all rows into BigQuery
var cb = arguments[arguments.length-1];
bigquery.tabledata.insertAll(request, function(err, result) {
if(err) {
console.log("Error insertAll. err=" + JSON.stringify(err) + ", request.resource=" + JSON.stringify(request.resource));
}
cb(err, result);
});
}, arguments[arguments.length-1]);
A typical error would look like this:
{
   "code": 503,
   "errors": [
      {
         "domain": "global",
         "reason": "backendError",
         "message": "Error encountered during execution. Retrying may solve the problem."
      }
   ]
}
The resource part for the insertAll that fails looks like this:
{
   "kind": "bigquery#tableDataInsertAllRequest",
   "skipInvalidRows": true,
   "ignoreUnknownValues": true,
   "templateSuffix": "GameStarted20160618",
   "rows": [
      {
         "insertId": "1f4786eaccd1c16d7ce865fea4c7af89",
         "json": {
            "eventName": "gameStarted",
            "eventSchemaHash": "unique-schema-hash-value",
            "eventTimestamp": 1466264556,
            "userId": "f769dc78-3210-4fd5-a2b0-ca4c48447578",
            "sessionId": "821f8f40-ed08-49ff-b6ac-9a1b8194286b",
            "platform": "WEBPLAYER",
            "versionName": "1.0.0",
            "versionCode": 12345,
            "ts_param1": "2016-06-04 00:00",
            "ts_param2": "2014-01-01 00:00",
            "i_param0": 598,
            "i_param1": 491,
            "i_param2": 206,
            "i_param3": 412,
            "i_param4": 590,
            "i_param5": 842,
            "f_param0": 5945.442,
            "f_param1": 1623.4111,
            "f_param2": 147.04747,
            "f_param3": 6448.521,
            "b_param0": true,
            "b_param1": false,
            "b_param2": true,
            "b_param3": true,
            "s_param0": "Im guesior ti asorne usse siorst apedir eamighte rel kin.",
            "s_param1": "Whe autiorne awayst pon, lecurt mun.",
            "eventHash": "1f4786eaccd1c16d7ce865fea4c7af89",
            "collectTimestamp": "1468346812",
            "eventDate": "2016-06-18"
         }
      }
   ]
}
We have noticed that, if we avoid including the name of the event in the suffix (e.g. the NewPlayer part) and instead just have the date as the suffix, then we never experience these errors.
Is there any way that this can be made to work reliably?
Backend errors happen, we usually see 5 from 10000 requests. We simply retry, and we have more constant rate, and we can provide a reconstructable use case we put a ticket on the Bigquery issue tracker. This way if there is something wrong with our project it can be investigated.
https://code.google.com/p/google-bigquery/

Subcribe a channel / real-time notification

Question about real-time notification..
Post: https://xxxiot.cumulocity.com/cep/realtime
Body:
[
{
"channel": "/meta/handshake",
"version": "1.0",
"mininumVersion": "1.0beta",
"supportedConnectionTypes": ["long-polling","callback-polling"],
"advice":{"timeout":120000,"interval":30000}
}
]
My Response:
[
{
"minimumVersion": "1.0",
"supportedConnectionTypes": [
"smartrest-long-polling",
"long-polling"
],
"successful": true,
"channel": "/meta/handshake",
"ext": {
"ack": true
},
"clientId": "5o0ghvle7yy4ix41on423v6k3j87",
"version": "1.0"
}
]
After received the clientId.. I have run the following command:
Post: https://xxxiot.cumulocity.com/cep/realtime
Body:
[
{
"channel": "/meta/subscribe",
"clientId": "5o0ghvle7yy4ix41on423v6k3j87",
"subscription": "/alarms/overHeatAlarms"
}
]
Response:
[
{
"error": "403:denied_by_security_policy:create_denied",
"subscription": "/alarms/overHeatAlarms",
"successful": false,
"channel": "/meta/subscribe"
}
]
Where is the problem? I'm trying to subcribing to "overheatAlarms"!
It may be that it does not exist? Can I read the existing information?
Thanks,
Alim
Yes, your suspicion is correct. There are basically two options for you:
Subscribe to all alarms or alarms from a particular device: Use "/cep/realtime" and channel "/alarms/* resp. channel "/alarms/[device ID]".
Create a processing rule that filters out overheat alarms and subscribe to that rule: Use "/cep/notifications" and channel "/[module name]/[statement name]".
The module name is what you enter as name when you click "New module". The statement name is what you add to the statement, e.g.
#Name('overHeatAlarms')
select * from AlarmsCreated where [your condition for overheat alarms]
(If you don't put a name there, they will be name statement_1, statement_2, ....)
To get notifications from Java, have a look at an example of getting notifications for changes in devices. In the subscribe() method, you pass "*" or the device ID. To get the notification, pass an implementation of SubscriptionListener, in particular the onNotification method. You can modify "channelPrefix" to "/alarms/" or "/measurements/" to get other notifications.
Thanks, André.
I've tested following Code Snippet.. it works, but it is not the best solution :-)
MeasurementApi measurementApi = getMeasurementApi();
MeasurementFilter measurementFilter = new MeasurementFilter();
while (true) {
Calendar cal = Calendar.getInstance();
Date toDate = cal.getTime();
cal.add(Calendar.SECOND, -25);
Date fromDate = cal.getTime();
measurementFilter.byDate(fromDate, toDate);
measurementFilter.byFragmentType(TemperatureMeasurement.class);
measurementFilter.bySource(new GId(DEVICE_SIMULATOR));
MeasurementCollection mc = measurementApi
.getMeasurementsByFilter(measurementFilter);
MeasurementCollectionRepresentation measurements = mc.get();
for (; measurements != null; measurements = mc
.getNextPage(measurements)) {
for (MeasurementRepresentation measurement : measurements
.getMeasurements()) {
TemperatureMeasurement temperatureSensor = measurement
.get(TemperatureMeasurement.class);
System.out.println(measurement.getSource().getId() + " "
+ measurement.getTime()+ " " + temperatureSensor.getTemperature() );
}
}
}