Invoking Rest API using Powershell - CosmosDb - api

I was trying to deploy cosmos database using Cosmos DB REST Api. I'm using a function to build the authorisation header and I got the script from https://gallery.technet.microsoft.com/scriptcenter/How-to-query-Azure-Cosmos-0a9aa517 link. It works perfectly fine for GET & POST however when I tried to execute a PUT command I'm always getting below error.
Invoke-RestMethod : The remote server returned an error: (401)
Unauthorized.
Im trying to update the offer for Cosmos collection but it always ends with the error and I couldn't understand whats the reason. I also checked my headers and authorisation with Microsoft documentation and looks fine to me. Refer https://learn.microsoft.com/en-us/rest/api/documentdb/replace-an-offer for Uri and headers required. My request and response are below
Request
PUT https: //mycosmosdb.documents.azure.com:443/offers/mycollection HTTP/1.1
authorization: type % 3dmaster % 26ver % 3d1.0 % 26sig % 3dIgWkszNS % 2b94fUEyrG8frByB2PWSc1ZEszc06GUeuW7s % 3d
x - ms - version: 2017 - 02 - 22
x - ms - date: Wed, 02 Aug 2017 08: 40: 37 GMT
User - Agent: Mozilla / 5.0(Windows NT; Windows NT 10.0; en - US)WindowsPowerShell / 5.1.15063.483
Content - Type: application / json
Host: mycosmosdb.documents.azure.com
Content - Length: 269
{
"offerVersion": "V2",
"offerType": "Invalid",
"content": {
"offerThroughput": 500,
"offerIsRUPerMinuteThroughputEnabled": false
},
"resource": "dbs/xterf==/colls/STuexopre=/",
"offerResourceId": "STuexopre=",
"id": "xiZw",
"_rid": "xiZw"
}
Response
HTTP / 1.1 401 Unauthorized
Transfer - Encoding: chunked
Content - Type: application / json
Content - Location: https: //mycosmosdb.documents.azure.com/offers/variantstockquantity
Server: Microsoft - HTTPAPI / 2.0
x - ms - activity - id: 6f7be3c8 - cfa2 - 4d5e - ad69 - fb14ef218980
Strict - Transport - Security: max - age = 31536000
x - ms - gatewayversion: version = 1.14.57.1
Date: Wed, 02 Aug 2017 08: 40: 35 GMT
163{
"code": "Unauthorized",
"message": "The input authorization token can't serve the request. Please check that the expected payload is built as per the protocol, and check the key being used. Server used the following payload to sign: 'put\noffers\mycollection\nwed, 02 aug 2017 08:40:37 gmt\n\n'\r\nActivityId: 6f7be3c8-cfa2-4d5e-ad69-fb14ef218980"
}
0
My Powershell Code
Function Generate-MasterKeyAuthorizationSignature
{
[CmdletBinding()]
Param
(
[Parameter(Mandatory=$true)][String]$verb,
[Parameter(Mandatory=$true)][String]$resourceLink,
[Parameter(Mandatory=$true)][String]$resourceType,
[Parameter(Mandatory=$true)][String]$dateTime,
[Parameter(Mandatory=$true)][String]$key,
[Parameter(Mandatory=$true)][String]$keyType,
[Parameter(Mandatory=$true)][String]$tokenVersion
)
$hmacSha256 = New-Object System.Security.Cryptography.HMACSHA256
$hmacSha256.Key = [System.Convert]::FromBase64String($key)
If ($resourceLink -eq $resourceType) {
$resourceLink = ""
}
$payload = "$($verb.ToLowerInvariant())`n$($resourceType.ToLowerInvariant())`n$resourceLink`n$($dateTime.ToLowerInvariant())`n`n"
$hashPayload = $hmacSha256.ComputeHash([System.Text.Encoding]::UTF8.GetBytes($payload))
$signature = [System.Convert]::ToBase64String($hashPayload);
[System.Web.HttpUtility]::UrlEncode("type=$keyType&ver=$tokenVersion&sig=$signature")
}
Function Modify-Offer
{
[CmdletBinding()]
Param
(
[Parameter(Mandatory=$true)][String]$DocumentDBApi,
[Parameter(Mandatory=$true)][String]$EndPoint,
[Parameter(Mandatory=$true)][String]$MasterKey,
[Parameter(Mandatory=$true)][String]$CollectionName
)
$Verb = "PUT"
$ResourceType = "offers";
$ResourceLink = "offers"
$body = '{
"offerVersion": "V2",
"offerType": "Invalid",
"content": {
"offerThroughput": 500,
"offerIsRUPerMinuteThroughputEnabled": false
},
"resource": "dbs/xterf==/colls/STuexopre=/",
"offerResourceId": "STuexopre=",
"id": "xiZw",
"_rid": "xiZw"
}'
$dateTime = [DateTime]::UtcNow.ToString("r")
$authHeader = Generate-MasterKeyAuthorizationSignature -verb $Verb -resourceLink $ResourceLink -resourceType $ResourceType -key $MasterKey -keyType "master" -tokenVersion "1.0" -dateTime $dateTime
$header = #{authorization=$authHeader;"x-ms-version"=$DocumentDBApi;"x-ms-date"=$dateTime}
$contentType= "application/json"
$queryUri = "$EndPoint$ResourceLink/$CollectionName"
$result = Invoke-RestMethod -Method $Verb -ContentType $contentType -Uri $queryUri -Headers $header -Body $body
$result | ConvertTo-Json -Depth 10
}
Modify-Offer -EndPoint $CosmosDBEndPoint -MasterKey $MasterKey -DocumentDBApi $DocumentDBApiVersion -CollectionName $ColName
Can someone throw me some help as why my PUT requests are failed with authorisation error, what I'm missing and how can I correct it.

Response message clearly states used payload for verification. Tracing '$payLoad' in Generate-MasterKeyAuthorizationSignature will quickly revel the issue.
You need to address at-least below two issues for this to work
RepalceOffer documentation states RID of the offer, instead you are
passing the collection name.
ResourceLin hardcoded: $ResourceLink
= "offers" in Modify-Offer where as it needs to point to the RID of the resource.
Here is slightly modified code which should do job
Function Generate-MasterKeyAuthorizationSignature
{
[CmdletBinding()]
Param
(
[Parameter(Mandatory=$true)][String]$verb,
[Parameter(Mandatory=$true)][String]$resourceLink,
[Parameter(Mandatory=$true)][String]$resourceType,
[Parameter(Mandatory=$true)][String]$dateTime,
[Parameter(Mandatory=$true)][String]$key,
[Parameter(Mandatory=$true)][String]$keyType,
[Parameter(Mandatory=$true)][String]$tokenVersion
)
$hmacSha256 = New-Object System.Security.Cryptography.HMACSHA256
$hmacSha256.Key = [System.Convert]::FromBase64String($key)
If ($resourceLink -eq $resourceType) {
$resourceLink = ""
}
$payLoad = "$($verb.ToLowerInvariant())`n$($resourceType.ToLowerInvariant())`n$resourceLink`n$($dateTime.ToLowerInvariant())`n`n"
$hashPayLoad = $hmacSha256.ComputeHash([System.Text.Encoding]::UTF8.GetBytes($payLoad))
$signature = [System.Convert]::ToBase64String($hashPayLoad);
Write-Host $payLoad
[System.Web.HttpUtility]::UrlEncode("type=$keyType&ver=$tokenVersion&sig=$signature")
}
Function Modify-Offer
{
[CmdletBinding()]
Param
(
[Parameter(Mandatory=$true)][String]$DocumentDBApi,
[Parameter(Mandatory=$true)][String]$EndPoint,
[Parameter(Mandatory=$true)][String]$MasterKey,
[Parameter(Mandatory=$true)][String]$OfferRID
)
$Verb = "PUT"
$ResourceType = "offers";
$body = '{
"offerVersion": "V2",
"offerType": "Invalid",
"content": {
"offerThroughput": 600,
"offerIsRUPerMinuteThroughputEnabled": false
},
"resource": "dbs/xterf==/colls/STuexopre=/",
"offerResourceId": "STuexopre=",
"id": "xiZw",
"_rid": "xiZw"
}'
$dateTime = [DateTime]::UtcNow.ToString("r")
$authHeader = Generate-MasterKeyAuthorizationSignature -verb $Verb -resourceLink $OfferRID -resourceType $ResourceType -key $MasterKey -keyType "master" -tokenVersion "1.0" -dateTime $dateTime
$header = #{authorization=$authHeader;"x-ms-version"=$DocumentDBApi;"x-ms-date"=$dateTime}
$contentType= "application/json"
$queryUri = "$EndPoint$ResourceType/$OfferRID"
$result = Invoke-RestMethod -Method $Verb -ContentType $contentType -Uri $queryUri -Headers $header -Body $body
$result | ConvertTo-Json -Depth 10
}
Modify-Offer -EndPoint $CosmosDBEndPoint -MasterKey $MasterKey -DocumentDBApi $DocumentDBApiVersion -OfferRID $ColName
Other alternative recommended approach if possible is to consume client SDK in Powershell. Here is a sample code which updates first offer of the account.
Add-Type -Path "...\Microsoft.Azure.Documents.Client.dll"
$client=New-Object Microsoft.Azure.Documents.Client.DocumentClient($CosmosDBEndPoint, $MasterKey)
$offersEnum=$client.ReadOffersFeedAsync().Result.GetEnumerator();
if ($offersEnum.MoveNext())
{
$targetOffer=$offersEnum.Current
$offerUpdated=New-Object Microsoft.Azure.Documents.OfferV2($targetOffer, 600, $FALSE)
$client.ReplaceOfferAsync($offerUpdated).Result
}

Related

Databricks: cannot get personal access token. Error 400 Invalid resource ID

$ADB_WORKSPACE_ID = "5555555555555"
$ADB_WORKSPACE_URL = "adb-5555555555555.5.azuredatabricks.net"
$adbGlobalToken = (az account get-access-token --resource 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d | ConvertFrom-Json).accessToken
$azureApiToken= (az account get-access-token --resource "https://management.core.windows.net/" | ConvertFrom-Json).accessToken
$headers = #{
"Authorization" = "Bearer $adbGlobalToken";
"X-Databricks-Azure-SP-Management-Token" = $azureApiToken;
"X-Databricks-Azure-Workspace-Resource-Id" = $ADB_WORKSPACE_ID;
}
$body = #{
"comment" = "This is an example token";
"lifetime_seconds" = 300;
}
$uri = "https://${ADB_WORKSPACE_URL}/api/2.0/token/create"
write-host $uri
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
$json = Invoke-RestMethod -Method Post -Uri $uri -Headers $headers -ContentType "application/json" -Body $body
write-host $json
Above code gives me the following error:
Invoke-RestMethod :
Error 400 Invalid resource ID.
HTTP ERROR 400
Problem accessing /api/2.0/token/create. Reason:
Invalid resource ID.
(FictiveDatabricks URL & ID )
Like the error stated, I provided the wrong resource id.
The correct way to get it is the following:
$ADB_WORKSPACE_ID = "/subscriptions/$SubscriptionId/resourceGroups/$ResourceGroupName/providers/Microsoft.Databricks/workspaces/$WorkspaceName"
ResourceId is not the same as the WorkSpaceId.
ResourceId can also be found in the Databricks resource json in the "id" field.

Getting Invalid JWT when creating project

When I get the token from the itwin-developer-console, the below PowerShell works. However, when I generate my own token, which appears to be valid per all the parsing I do of it, I get 401 Invalid JWT. I can use the token I generate to query and manipulate work area connections with no problems as long as I provide the product-settings-service scope. I've limited the scopes to just be projects:read projects:modify like the token I get from the console, but no joy. I notice that my token does not populate entitlements. Could that be it?
$url = "https://api.bentley.com/projects/"
$authVal = "Bearer $($oidcClientToken.access_token)"
$today = Get-Date -Format "yyyy-MM-dd"
$projectName = $today + " " + (Get-RandomString -Length 12 -Characters "ABCDEFGHIJKLMNOPQRSTUVWXYZ")
$bodyCreate = ConvertTo-Json -Depth 4 #{
project = #{
displayName = $projectName
projectNumber = $projectName
industry = "Oil and Gas"
projectType = "Offshore Structures"
billingCountry = "US"
status = "active"
allowExternalTeamMembers = $true
}
}
$resp5 = Invoke-RestMethod -ContentType "Application/Json" -Method Post -Uri $url -Body $bodyCreate -Headers #{'Authorization' = $authVal; 'Content-Type' = 'application/json'; 'Accept' = 'application/vnd.bentley.itwin-platform.v1+json' }
You need a different token for calling iTwin Platform APIs, please read the Authorization documentation on this.
Specifically in this case, the token needs to be issued by https://ims.bentley.com instead of https://imsoidc.bentley.com, which is currently used by itwin.js applications. The same client will work.

ARM Template deploymentScript PowerShell try catch ignored?

I am using an ARM Template of type "Microsoft.Resources/deploymentScripts". That uses a PowerShell script that should add an identity to a role in Azure AD. When I run that a second time the post requests, the last Invoke-RestMethod, to add the member fails. That intentional and okay because the member is already there. I wrapped that in a try catch and look for the specific error. This works fine locally but not when deployed on Azure. It still stops at the line of the invoke, 41, and does not seem to respect the try catch around that. What am I doing wrong?
param([string] $spObjectId, $roleName, $tenantId, $clientId, $clientSecret)
Write-Host $ErrorActionPreference
$ErrorActionPreference = "Continue"
$Body = #{
'tenant' = $tenantId
'client_id' = $clientId
'scope' = 'https://graph.microsoft.com/.default'
'client_secret' = $clientSecret
'grant_type' = 'client_credentials'
}
$url = "https://login.microsoftonline.com/$tenantId/oauth2/v2.0/token"
Write-Host $url
$Params = #{
'Uri' = $url
'Method' = 'Post'
'Body' = $Body
'ContentType' = 'application/x-www-form-urlencoded'
}
$AuthResponse = Invoke-RestMethod #Params
$Headers = #{
'Authorization' = "Bearer $($AuthResponse.access_token)"
}
$roles = Invoke-RestMethod -Uri 'https://graph.microsoft.com/v1.0/directoryRoles' -Headers $Headers
$role = $roles.value | Where-Object { $_.displayName -eq "$roleName" }
$roleId = $role.id
$Body = #{
"#odata.id"= "https://graph.microsoft.com/v1.0/directoryObjects/$spObjectId"
}
$json = $Body | ConvertTo-Json
$url = "https://graph.microsoft.com/v1.0/directoryRoles/$roleId/members/`$ref"
try {
Invoke-RestMethod -Method 'Post' -Uri $url -Body $json -Headers $Headers -ContentType 'application/json'
}
catch {
$errordetails = $_.ErrorDetails.Message | ConvertFrom-Json
if($errordetails.error.message -ne "One or more added object references already exist for the following modified properties: 'members'.")
{
throw $_.Exception
}
}
Write-Host "Done"
The error in the Azure Portal
The provided script failed with the following error:
Microsoft.PowerShell.Commands.HttpResponseException: Response status
code does not indicate success: 400 (Bad Request). at
System.Management.Automation.MshCommandRuntime.ThrowTerminatingError(ErrorRecord
errorRecord) at ,
/mnt/azscripts/azscriptinput/AddMemberToRole.ps1: line 41 at
, : line 1 at ,
/mnt/azscripts/azscriptinput/DeploymentScript.ps1: line 192. Please
refer to https://aka.ms/DeploymentScriptsTroubleshoot for more
deployment script information. (Code: DeploymentScriptError)
Ideally, Invoke-RestMethod's error is captured in the try catch block. Your code looks correct and similar to what has to be done:
try{ $restp=Invoke-RestMethod (...) } catch {$err=$_.Exception}
$err | Get-Member -MemberType Property
TypeName: System.Net.WebException
Name MemberType Definition
---- ---------- ----------
Message Property string Message {get;}
Response Property System.Net.WebResponse Response {get;}
Status Property System.Net.WebExceptionStatus Status {get;}
I suspect two reasons for this behavior:
The ARM template is having issues such that try catch block is neglected. Make sure the syntax is correct and expected (ARM works in local).
An old deployed script is messing with this script. Try changing the resource name and redeploy.

EDIT the Azure databrics cluster's SPARK configuration using PowerShell and REST API

I am trying to EDIT the Azure databrics cluster's SPARK configuration using PowerShell and REST API. However I am getting an error which I am unable to understand/fix. I have provided the 'required' fields as parameters, however, the error states that I haven't passed them
CODE:
$DBAPIRootUrl = "dec" # example: https://uksouth.azuredatabricks.net
$DBAPIKey = "abc" # Example dapi601e67891a9d1f7886e40916479aaa
[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]::Tls12
$ClustersAPIListUrl = $DBAPIRootUrl.Trim('/') + "/api/2.0/clusters/list"
$ClustersAPIEditUrl = $DBAPIRootUrl.Trim('/') + "/api/2.0/clusters/edit"
$headers = #{
Authorization = "Bearer $DBAPIKey"
"Content-Type" = "application/json"
}
$response = Invoke-WebRequest -Uri $ClustersAPIListUrl -Method GET -Headers $headers #-Body $parameters
$json_response = ($response.Content | ConvertFrom-Json)
$jsonDoc = [pscustomobject]#{
cluster_id = $json_response.clusters.cluster_id
spark_version = $json_response.clusters.spark_version
node_type_id = $json_response.clusters.node_type_id
spark_conf = "
javax.jdo.option.ConnectionPassword
datanucleus.fixedDatastore false
javax.jdo.option.ConnectionURL jdbc:sqlserver://metadatasrvr.database.windows.net:1433;database=emptydb
datanucleus.schema.autoCreateAll true
spark.hadoop.hive.metastore.schema.verification false
datanucleus.autoCreateSchema true
spark.sql.hive.metastore.jars maven
javax.jdo.option.ConnectionDriverName com.microsoft.sqlserver.jdbc.SQLServerDriver
spark.sql.hive.metastore.version 1.2.0
javax.jdo.option.ConnectionUserName"
}
$jsonDoc | ConvertTo-Json
#$parameters | ConvertTo-Json
$response = Invoke-WebRequest -Uri $ClustersAPIEditUrl -Method POST -Headers $headers -Body $jsonDoc
ERROR:
Invoke-WebRequest : {"error_code":"INVALID_PARAMETER_VALUE","message":"Missing required fields: cluster_id, size"}
At line:21 char:13
+ $response = Invoke-WebRequest -Uri $ClustersAPIEditUrl -Method POST - ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
The error message clearly explains ""error_code":"INVALID_PARAMETER_VALUE","message":"Missing required fields: cluster_id, size"}".
Note: While editing Databricks cluster, make sure to pass the "cluster_id" and "node_type_id" as a mandatory expecting fields.
To Edit the configuration of a cluster to match the provided attributes and size.
An example request:
{
"cluster_id": "1202-211320-brick1",
"num_workers": 10,
"spark_version": "5.3.x-scala2.11",
"node_type_id": "Standard_D3_v2"
}
Reference: Databricks - REST API EDIT clusters
Hope this helps.

Fortify API Start Scan with Default - How to send package

I am trying to use the API from https://api.emea.fortify.com/swagger/ui/index#/
called Start Scan with Default.
I cannot find any documentation to suggest how to set the post up.
This is what I have so far, but I get an error and of course I am not sending the files to scan either, so I know it is not right.
I have tried a Get request, which works so I know it is authenticated etc.
I just need to know are the parameters correctly formatted and how do I upload the actual files to scan.
POST /api/v3/releases/43579/static-scans/start-scan-with-defaults?releaseId=43579& fragNo=22& offset=22& isRemediationScan=false& notes=hello HTTP/1.1
Host: api.emea.fortify.com
Content-Type: application/json
Authorization: Bearer [TOKEN HERE]
User-Agent: PostmanRuntime/7.13.0
Accept: */*
Cache-Control: no-cache
Postman-Token: 57e40c1d-c99c-40a4-a79b-06ef9a678a07,8ef4ad1e-327f-4eee-b6bb-bddb21b18d50
Host: api.emea.fortify.com
accept-encoding: gzip, deflate
content-length:
Connection: keep-alive
cache-control: no-cache
Response:
{
"errors": [
{
"errorCode": null,
"message": "Unexpected error processing request"
}
]
}
UPDATE
I have found this repo on Git written in Java, which I have tried to recreate in PowerShell with no success.
https://github.com/fod-dev/fod-uploader-java
My PowerShell:
[System.Net.WebRequest]::DefaultWebProxy = [System.Net.WebRequest]::GetSystemWebProxy()
[System.Net.WebRequest]::DefaultWebProxy.Credentials = [System.Net.CredentialCache]::DefaultNetworkCredentials
$zipDetails = Get-Content C:\Users\patemanc\Desktop\types.zip -Encoding Byte
Write-Host $zipDetails.Length
$releaseId = "43576"
$url = "https://api.emea.fortify.com/api/v3/releases/$releaseId/static-scans/start-scan-with-defaults?"
$url += "releaseId=$releaseId"
$url += "&fragNo=-1"
$url += "&offset=0"
$url += "&isRemediationScan=false"
$url += "&notes=PowrShell Test"
$long_lived_access_token = "ENTER TOKEN HERE"
$headers = #{Authorization = "bearer:$long_lived_access_token"}
$response = Invoke-WebRequest -ContentType "application/octet-stream" -Uri $url -Method POST -Body $zipDetails -Headers $headers -UseBasicParsing
Write-Host "Here is the end"
Write-Host $response
Error Response:
79212
Invoke-WebRequest : The underlying connection was closed: An unexpected error occurred on a send.
At line:22 char:13
+ $response = Invoke-WebRequest -ContentType "application/json" -Uri $ ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
Why postman? If you use some plugin to run it, from Jenkins for example, it works fine. I don't know how the plugins call it from the API.