I am trying to loop around all of our subscriptions and get Policy Exemptions, but only get the ones that we have created. The loop appears fine, but the Match element appears to bring back some Exemptions that don't meet the -Match criteria.
$allSubscriptions = Get-AzSubscription
$baseFolder = "C:\source\PowerShell Exemptions Dump\"
# loop subscriptions
foreach($sub in $allSubscriptions){
$subName = $sub.Name
# Get Exemptions at Sub level
Set-AzContext -Subscription $subName
# Write to File
$exemptionsIn = Get-AzPolicyExemption|ConvertTo-Json
$fileName = $baseFolder + $subName + ".json"
$exemptionsOut = ''
foreach($ex in $exemptionsIn|ConvertFrom-Json){
if($ex.Properties.PolicyAssignmentId -Match "abc") {
$exemptionsOut += $ex|ConvertTo-Json
}
}
if ($exemptionsOut -ne '') {
$exemptionsOut | Out-File -filepath $fileName
$exemptionsOut = ''
}
}
It does work to a certain extent i.e. if a Subscription has a 0% match in everything it brings back, then it doesn't create a file. but it appears if it finds one match, then it saves Exemptions to the file that don't match.
Here is some example Json that was saved to one of the files:
[
{
"Properties": {
"PolicyAssignmentId": "/providers/Microsoft.Management/managementGroups/abc-mg/providers/Microsoft.Authorization/policyAssignments/abc-mg",
"PolicyDefinitionReferenceIds": "",
"ExemptionCategory": "Waiver",
"DisplayName": "abc - abc-mg Policy Assignment",
"Description": "AIB Testing",
"ExpiresOn": "\/Date(1662134400000)\/",
"Metadata": ""
},
"SystemData": null,
"Name": "456",
"ResourceId": "/subscriptions/123/providers/Microsoft.Authorization/policyExemptions/789",
"ResourceName": "456",
"ResourceGroupName": null,
"ResourceType": "Microsoft.Authorization/policyExemptions",
"SubscriptionId": "123"
},
{
"Properties": {
"PolicyAssignmentId": "/providers/Microsoft.Management/managementGroups/root-mg/providers/Microsoft.Authorization/policyAssignments/111",
"PolicyDefinitionReferenceIds": "installEndpointProtection",
"ExemptionCategory": "Waiver",
"DisplayName": "root-mg - Azure Security Benchmark",
"Description": "currently use sophos and not defender",
"ExpiresOn": null,
"Metadata": ""
},
"SystemData": null,
"Name": "345",
"ResourceId": "/providers/Microsoft.Management/managementGroups/root-mg/providers/Microsoft.Authorization/policyExemptions/345",
"ResourceName": "345",
"ResourceGroupName": null,
"ResourceType": "Microsoft.Authorization/policyExemptions",
"SubscriptionId": null
}
]
Finally, I don't appear to get all Exemptions back in this loop i.e. some are set at Resource Group or Resource Level. Do I need to drill further beyond Set-AzContext?
After reproducing the same code from my end, I could able to see the expected results. However, make sure you are checking in the right file and the location to which you are sending your data to.
Finally, I don't appear to get all Exemptions back in this loop i.e. some are set at Resource Group or Resource Level.
This might be due to the scope that you are looking into. After setting the scope to the required level I could able to get the expected results. Below is the code that worked for me.
$Resource = Get-AzResource -ResourceGroupName <YOUR_RESOURCEGROUP_NAME>
for($I=0;$I -lt $Resource.ResourceId.Count;$I++)
{
$a=Get-AzPolicyExemption -Scope $Resource.ResourceId[$I]
for($J=0;$J -lt $a.Count;$J++)
{
If($a.ResourceId[$J] -Match $Resource.ResourceId[$I])
{
$exemptionsIn = Get-AzPolicyExemption -Scope $Resource.ResourceId[$I] | ConvertTo-Json
$fileName = "sample2" + ".json"
$exemptionsOut = ''
foreach($ex in $exemptionsIn|ConvertFrom-Json){
if($ex.Properties.PolicyAssignmentId -Match "Swetha*") {
$exemptionsOut += $ex|ConvertTo-Json
}
}
if ($exemptionsOut -ne '') {
$exemptionsOut | Out-File -filepath $fileName
$exemptionsOut = ''
}
}
}
}
I have few policy exemptions in my subscription but above script gave me the results at Resource level which -Match with Swetha.
RESULTS:
Related
I'm new to Azure Durable Functions and am trying to understand the retry logic and error handling.
I have a very simple orchestration function that executes 100 action functions in a fan-in fan-out pattern. My expectation is that when an action function breaks for whatever reason it is retried based on the retry options. In my case I'm expecting that I get a count of 100 in the final orchestration line (Write-Information $results.count), but for every action function that errors out due to my random error throwing, it seems no retry is done and the results is always less than 100.
Why is this happening, also why am I seeing the orchestrator output $results.count multiple times?
Orchestration
param($Context)
$range = 1..100
$retryOptions = New-DurableRetryOptions -FirstRetryInterval (New-TimeSpan -Seconds 1) -MaxNumberOfAttempts 10
$tasks =
foreach ($num in $range) {
try{
Invoke-DurableActivity -FunctionName 'RandomWaitActivity' -Input $num -NoWait -RetryOptions $retryOptions
}
catch{
Write-Output $_.Exception.Message
}
}
$results = Wait-ActivityFunction -Task $tasks
Write-Information $results.count
Action
param($number)
Write-Output "Received $number"
$random = Get-Random -Minimum 1 -Maximum 20
if($random -eq 13){
Throw "13 is a unlucky number"
}
Start-Sleep -Milliseconds $random
return $number
Well, If you want to add retry in your, you can configure retry policy in your function.json.
In this retry policy you can set maxRetryCount which represent the number of times a function will retry before stopping.
function.json :-
{
"bindings": [
{
"authLevel": "anonymous",
"type": "httpTrigger",
"direction": "in",
"name": "Request",
"methods": [
"get",
"post"
]
},
{
"type": "http",
"direction": "out",
"name": "Response"
}
],
"retry": {
"strategy": "fixedDelay",
"maxRetryCount": 1,
"delayInterval": "00:00:10"
}
}
Here Since I have set the retry count to 1 the function will try to execute twice.
Refer this MsDOC on retry policy.
Question about real-time notification..
Post: https://xxxiot.cumulocity.com/cep/realtime
Body:
[
{
"channel": "/meta/handshake",
"version": "1.0",
"mininumVersion": "1.0beta",
"supportedConnectionTypes": ["long-polling","callback-polling"],
"advice":{"timeout":120000,"interval":30000}
}
]
My Response:
[
{
"minimumVersion": "1.0",
"supportedConnectionTypes": [
"smartrest-long-polling",
"long-polling"
],
"successful": true,
"channel": "/meta/handshake",
"ext": {
"ack": true
},
"clientId": "5o0ghvle7yy4ix41on423v6k3j87",
"version": "1.0"
}
]
After received the clientId.. I have run the following command:
Post: https://xxxiot.cumulocity.com/cep/realtime
Body:
[
{
"channel": "/meta/subscribe",
"clientId": "5o0ghvle7yy4ix41on423v6k3j87",
"subscription": "/alarms/overHeatAlarms"
}
]
Response:
[
{
"error": "403:denied_by_security_policy:create_denied",
"subscription": "/alarms/overHeatAlarms",
"successful": false,
"channel": "/meta/subscribe"
}
]
Where is the problem? I'm trying to subcribing to "overheatAlarms"!
It may be that it does not exist? Can I read the existing information?
Thanks,
Alim
Yes, your suspicion is correct. There are basically two options for you:
Subscribe to all alarms or alarms from a particular device: Use "/cep/realtime" and channel "/alarms/* resp. channel "/alarms/[device ID]".
Create a processing rule that filters out overheat alarms and subscribe to that rule: Use "/cep/notifications" and channel "/[module name]/[statement name]".
The module name is what you enter as name when you click "New module". The statement name is what you add to the statement, e.g.
#Name('overHeatAlarms')
select * from AlarmsCreated where [your condition for overheat alarms]
(If you don't put a name there, they will be name statement_1, statement_2, ....)
To get notifications from Java, have a look at an example of getting notifications for changes in devices. In the subscribe() method, you pass "*" or the device ID. To get the notification, pass an implementation of SubscriptionListener, in particular the onNotification method. You can modify "channelPrefix" to "/alarms/" or "/measurements/" to get other notifications.
Thanks, André.
I've tested following Code Snippet.. it works, but it is not the best solution :-)
MeasurementApi measurementApi = getMeasurementApi();
MeasurementFilter measurementFilter = new MeasurementFilter();
while (true) {
Calendar cal = Calendar.getInstance();
Date toDate = cal.getTime();
cal.add(Calendar.SECOND, -25);
Date fromDate = cal.getTime();
measurementFilter.byDate(fromDate, toDate);
measurementFilter.byFragmentType(TemperatureMeasurement.class);
measurementFilter.bySource(new GId(DEVICE_SIMULATOR));
MeasurementCollection mc = measurementApi
.getMeasurementsByFilter(measurementFilter);
MeasurementCollectionRepresentation measurements = mc.get();
for (; measurements != null; measurements = mc
.getNextPage(measurements)) {
for (MeasurementRepresentation measurement : measurements
.getMeasurements()) {
TemperatureMeasurement temperatureSensor = measurement
.get(TemperatureMeasurement.class);
System.out.println(measurement.getSource().getId() + " "
+ measurement.getTime()+ " " + temperatureSensor.getTemperature() );
}
}
}
I need to interact with an API that is expecting an array of objects, among other parameters. Example:
{
"fields": {
"somefield": "somevalue",
"someobject": {
"name": "foobar"
},
"versions": [
{
"name": "1.0"
}
]
}
}
With the help of this answer, I've tried two different ways of handling this. I've combined them into a single code example:
$versionName = New-Object -TypeName PSObject
$versionName | Add-Member -Name "name" -MemberType NoteProperty -Value "1.0"
$versionName2 = #{}
$versionName2.name = "1.0"
$postIssueBody = #{}
$postIssueBody.fields = #{}
$postIssueBody.fields.somefield = "somevalue"
$postIssueBody.fields.someobject = #{}
$postIssueBody.fields.someobject.name = "foobar"
$postIssueBody.fields.version = #($versionName)
$postIssueBody.fields.version2 = #()
$postIssueBody.fields.version2 += [pscustomobject]$versionName2
$postIssueRequestJson = $postIssueBody | ConvertTo-Json
$postIssueRequestJson
This results in the following output:
{
"fields": {
"somefield": "somevalue",
"someobject": {
"name": "foobar"
},
"version": [
"#{name=1.0}"
],
"version2": [
"#{name=1.0}"
]
}
}
As you can see, that's not going to fly as valid JSON. What is the best way to handle this assignment so that the version names are properly formed after going through ConvertTo-Json?
The ConvertTo-Json function has a switch called Depth. It informs the Convert function how deep it should go when converting data to the JSON format. By default, it is set at 2. Since the data that isn't being converted properly sits at a depth of 3, we simply need to set the depth to that, like so:
$postIssueRequestJson = $postIssueBody | ConvertTo-Json -Depth 3
And now we have well-formed JSON.
{
"fields": {
"somefields": "somevalue",
"someobject": {
"name": "foobar"
},
"versions": [
{
"name": "1.0"
}
]
}
}
Ok, I think I understand. So you need a string that starts with "versions": and is followed by an array of objects, yes? So, let's start with an empty array.
$Array = #()
Then we can create objects, and add those to the array:
$Array += [PSCustomObject]#{"Name1"="1.0.0"}
$Array += [PSCustomObject]#{"Name2"="3.10.0"}
Now we have a PowerShell array with PSCustomObjects in it. We can pipe that to ConvertTo-JSON and it will output:
[
{
"Name1": "1.0.0"
},
{
"Name2": "3.10.0"
}
]
Which is the array of objects you wanted. If you want an object to have that as it's value you could simply create another object to do that with:
$Versions = [PSCustomObject]#{'versions'=$Array}
Then you can convert that to JSON if you want and get:
{
"versions": [
{
"Name1": "1.0.0"
},
{
"Name2": "3.10.0"
}
]
}
That's what you were looking for, right? Or if you really want it on one line:
PS C:\> ($Versions|convertto-json).split() -join ""
{"versions":[{"Name1":"1.0.0"},{"Name2":"3.10.0"}]}
To be formatted exactly as your first example we would have to get rid of the { } surrounding that result I suppose, you can do that with Trim() as such:
PS C:\> ($Versions|convertto-json).trim("{}").split() -join ""
"versions":[{"Name1":"1.0.0"},{"Name2":"3.10.0"}]
Edit: Ok, so you just need to add objects as property values of other objects as needed, much like I did for setting the array as a value in the object in my example.
I think the easiest way to understand what needs to be done is to take your example (minus the last comma, since that makes it throw errors), and pipe it into ConvertFrom-JSON and assign it a variable. Then you can see how that is formed in Powersehll. Once I do that (I named my variable $JSON), I can see that $JSON has 1 NoteProperty of 'fields'. That NoteProperty has 3 NoteProperties of 'somefield', 'someobject', and 'versions'. When I do a $JSON.fields|Get-Member I find out more about those.
somefield is just a String. That will be easy enough to deal with.
someobject is a PSCustomObject, basically a HashTable where name=foobar.
versions just shows that it's a System.Object, so I'll do $JSON.fields.versions.GetType() and it shows that the basetpe is System.Array. After looking at versions it looks like an array with 1 object in it, and that object has one noteproperty that is a string (like the first object we had).
So, there's two ways to go about doing this. You can either try and create your objects and array in-line, or you can make them ahead of time, starting at the deepest nested layer, and work your way up. I'll be showing you the later.
$name = [PSCustomObject]#{'name'='1.0'}
$versions=#($name)
$Someobject = [PSCustomObject]#{'name'='foobar'}
$Fields = [PSCustomObject]#{
'somefields'='somevalue'
'someobject'=$someobject
'versions'=$versions}
$NewJSON = [PSCustomObject]#{'Fields'=$fields}
$NewJSON | ConvertTo-Json
It seems that if I am running a word or phrase through an ngram filter, the original word does not get indexed. Instead, I only get chunks of the word up to my max_gram value. I would expect the original word to get indexed as well. I'm using Elasticsearch 0.20.5. If I set up an index using a filter with ngrams like so:
CURL -XPUT 'http://localhost:9200/test/' -d '{
"settings": {
"analysis": {
"filter": {
"my_ngram": {
"max_gram": 10,
"min_gram": 1,
"type": "nGram"
},
"my_stemmer": {
"type": "stemmer",
"name": "english"
}
},
"analyzer": {
"default_index": {
"filter": [
"standard",
"lowercase",
"asciifolding",
"my_ngram",
"my_stemmer"
],
"type": "custom",
"tokenizer": "standard"
},
"default_search": {
"filter": [
"standard",
"lowercase"
],
"type": "custom",
"tokenizer": "standard"
}
}
}
}
}'
Then I put a long word into a document:
CURL -XPUT 'http://localhost:9200/test/item/1' -d '{
"foo" : "REALLY_REALLY_LONG_WORD"
}'
And I query for that long word:
CURL -XGET 'http://localhost:9200/test/item/_search' -d '{
"query":
{
"match" : {
"foo" : "REALLY_REALLY_LONG_WORD"
}
}
}'
I get 0 results. I do get a result if I query for a 10 character chunk of that word. When I run this:
curl -XGET 'localhost:9200/test/_analyze?text=REALLY_REALLY_LONG_WORD
I get tons of grams back, but not the original word. Am I missing a configuration to make this work the way I want?
If you would like to keep the complete word of phrase, use a multi-field mapping for the value where you keep one "not analyzed" or with keyword-tokenizer instead.
Also, when searching a field with nGram-tokenized values, you should probably also use the nGram-tokenizer for the search, then the n-character limit will also apply for the search-phrase, and you will get the expected results.
I am trying to load a CSV file to BigQuery using a python script modelled on the python sample code here: https://developers.google.com/bigquery/docs/developers_guide
But I'm running into the following error when I try to load a table with the REST API:
{'status': '200', 'content-length': '1492', 'expires': 'Fri, 01 Jan 1990 00:00:00 GMT', 'server': 'HTTP Upload Server Built on Jun 14 2012 02:12:09 (1339665129)', 'etag': '"tcivyOj9QvKAbuEJ5MEMf9we85w/-mxYhUDjvvydxcebR8fXI6l_5RQ"', 'pragma': 'no-cache', 'cache-control': 'no-cache, no-store, must-revalidate', 'date': 'Fri, 06 Jul 2012 22:30:55 GMT', 'content-type': 'application/json'}
{
"kind": "bigquery#job",
"etag": "\"tcivyOj9QvKAbuEJ5MEMf9we85w/-mxYhUDjvvydxcebR8fXI6l_5RQ\"",
"id": "firespotter.com:firespotter:job_d6b99265278b4c0da9c3033acf39d6b2",
"selfLink": "https://www.googleapis.com/bigquery/v2/projects/firespotter.com:firespotter/jobs/job_d6b99265278b4c0da9c3033acf39d6b2",
"jobReference": {
"projectId": "firespotter.com:firespotter",
"jobId": "job_d6b99265278b4c0da9c3033acf39d6b2"
},
"configuration": {
"load": {
"schema": {
"fields": [
{
"name": "date",
"type": "STRING"
},
{
"name": "time",
"type": "STRING"
},
{
"name": "call_uuid",
"type": "STRING"
},
{
"name": "log_level",
"type": "STRING"
},
{
"name": "file_line",
"type": "STRING"
},
{
"name": "message",
"type": "STRING"
}
]
},
"destinationTable": {
"projectId": "385479794093",
"datasetId": "telephony_logs",
"tableId": "table_name"
},
"createDisposition": "CREATE_IF_NEEDED",
"writeDisposition": "WRITE_TRUNCATE",
"encoding": "UTF-8"
}
},
"status": {
"state": "DONE",
"errorResult": {
"reason": "notFound",
"message": "Not Found: Dataset 385479794093:telephony_logs"
},
"errors": [
{
"reason": "notFound",
"message": "Not Found: Dataset 385479794093:telephony_logs"
}
]
}
}
The projectId listed in the error "385479794093" is not the projectId that I pass in, it's the "project number". The projectId should be "firespotter.com:firespotter":
{
"kind": "bigquery#datasetList",
"etag": "\"tcivyOj9QvKAbuEJ5MEMf9we85w/ZMa8z6LKMgWZIqLWh3ti2SsSs4g\"",
"datasets": [
{
"kind": "bigquery#dataset",
"id": "firespotter.com:firespotter:telephony_logs",
"datasetReference": {
"datasetId": "telephony_logs",
"projectId": "firespotter.com:firespotter"
}
}
]
}
Why does the REST API insist on supplying its own incorrect projectId, when I pass the correct value in three different places? Is there another place where I need to pass in or set the Project ID?
For reference, here is the relevant code snippet:
PROJECT = 'firespotter.com:firespotter'
DATASET = 'telephony_logs'
FLOW = OAuth2WebServerFlow(
client_id='385479794093.apps.googleusercontent.com',
client_secret='<a_secret_here>',
scope='https://www.googleapis.com/auth/bigquery',
user_agent='firespotter-upload-script/1.0')
def loadTable(http, projectId, datasetId, tableId, file_path, replace=False):
url = "https://www.googleapis.com/upload/bigquery/v2/projects/" + projectId + "/jobs"
# Create the body of the request, separated by a boundary of xxx
mime_data = ('--xxx\n' +
'Content-Type: application/json; charset=UTF-8\n' + '\n' +
'{\n' +
' "projectId": "' + projectId + '",\n' +
' "configuration": {\n' +
' "load": {\n' +
' "schema": {\n' +
' "fields": [\n' +
' {"name":"date", "type":"STRING"},\n' +
' {"name":"time", "type":"STRING"},\n' +
' {"name":"call_uuid", "type":"STRING"},\n' +
' {"name":"log_level", "type":"STRING"},\n' +
' {"name":"file_line", "type":"STRING"},\n' +
' {"name":"message", "type":"STRING"}\n' +
' ]\n' +
' },\n' +
' "destinationTable": {\n' +
' "projectId": "' + projectId + '",\n' +
' "datasetId": "' + datasetId + '",\n' +
' "tableId": "' + tableId + '"\n' +
' },\n' +
' "createDisposition": "CREATE_IF_NEEDED",\n' +
' "writeDisposition": "' + ('WRITE_TRUNCATE' if replace else 'WRITE_APPEND') + '",\n' +
' "encoding": "UTF-8"\n' +
' }\n' +
' }\n' +
'}\n' +
'--xxx\n' +
'Content-Type: application/octet-stream\n' +
'\n')
# Append data from the specified file to the request body
f = open(file_path, 'r')
header_line = f.readline() # skip header line
mime_data += f.read()
# Signify the end of the body
mime_data += ('--xxx--\n')
headers = {'Content-Type': 'multipart/related; boundary=xxx'}
resp, content = http.request(url, method="POST", body=mime_data, headers=headers)
print str(resp) + "\n"
print content
# --- Main ----------------------------------------------
def main(argv):
csv_path = args[0]
# If the credentials don't exist or are invalid, run the native client
# auth flow. The Storage object will ensure that if successful the good
# credentials will get written back to a file.
storage = Storage('bigquery2_credentials.dat') # Choose a file name to store the credentials.
credentials = storage.get()
if credentials is None or credentials.invalid:
credentials = run(FLOW, storage)
# Create an httplib2.Http object to handle our HTTP requests and authorize it
# with our good credentials.
http = httplib2.Http()
http = credentials.authorize(http)
loadTable(http, PROJECT, DATASET, 'table_name', csv_path, replace=True)
if __name__ == '__main__':
main(sys.argv)
Did you recently set the project id to firespotter.com:firespotter? If the dataset was created before the project was named, there will be a mismatch between the old project id and the new. There is an automated system that updates project ids, but it is possible that it hasn't run yet or is having a problem (I'm on vacation right now, so can't check). Hopefully, if you retry again some time soon it will just work. If not, let us know.
There are a few questions here:
Why did my load job fail? Just to check, was that the entire request you sent? If so, it looks like there's no data to be loaded, i.e. sourceUris is empty. If so, that's the problem, and we're apparently returning the world's worst error message.
Why the numeric project ID? BigQuery uses the project name and the associated numeric ID interchangeably, so all you're seeing is that we tend to convert project names to IDs on the way in. Just to confirm, if you visit the Google APIs Console and look up your project, do you see that same numeric ID in the url?
Why does the project ID get specified in multiple places? First, it seems that you specified the project ID as a top-level attribute in the job; that shouldn't be necessary. (I suspect that it simply overrides whatever project ID you specify in the job reference itself.) That leaves two locations -- once as part of the job reference and the other as part of the table reference. These actually signify two different things -- the one in the job specifies what project you're inserting a job into, i.e. who's paying for the job, and the one in the table specifies what project the resulting table lives in, i.e. who owns the resulting data. In general, these will be the same, but the API allows them to be distinct. (This could be useful if, for example, you built a service that needed to insert data into tables ultimately owned by customers.)