I have a bcp export
EXEC xp_cmdshell
'bcp "SELECT * FROM ##tmptable" queryout "c:\phonegapdownload\phonegap.js" -c -t, -T'
my problem is that it adds line number to the export file
1,{ "id": "500020", "type": "0" },
2,{ "id": "500025", "type": "0"},
3,{ "id": "500043", "type": "0"},
can anyone tell me how to stop it doing this
thanks
My hunch is that it's returning the row id rather than the row number. Perhaps change to just get the row you need and see if that fixes it.
EXEC xp_cmdshell
'bcp "SELECT [yourJsonRowName] FROM ##tmptable" queryout "c:\phonegapdownload\phonegap.js" -c -t, -T'
Related
I am trying to loop around all of our subscriptions and get Policy Exemptions, but only get the ones that we have created. The loop appears fine, but the Match element appears to bring back some Exemptions that don't meet the -Match criteria.
$allSubscriptions = Get-AzSubscription
$baseFolder = "C:\source\PowerShell Exemptions Dump\"
# loop subscriptions
foreach($sub in $allSubscriptions){
$subName = $sub.Name
# Get Exemptions at Sub level
Set-AzContext -Subscription $subName
# Write to File
$exemptionsIn = Get-AzPolicyExemption|ConvertTo-Json
$fileName = $baseFolder + $subName + ".json"
$exemptionsOut = ''
foreach($ex in $exemptionsIn|ConvertFrom-Json){
if($ex.Properties.PolicyAssignmentId -Match "abc") {
$exemptionsOut += $ex|ConvertTo-Json
}
}
if ($exemptionsOut -ne '') {
$exemptionsOut | Out-File -filepath $fileName
$exemptionsOut = ''
}
}
It does work to a certain extent i.e. if a Subscription has a 0% match in everything it brings back, then it doesn't create a file. but it appears if it finds one match, then it saves Exemptions to the file that don't match.
Here is some example Json that was saved to one of the files:
[
{
"Properties": {
"PolicyAssignmentId": "/providers/Microsoft.Management/managementGroups/abc-mg/providers/Microsoft.Authorization/policyAssignments/abc-mg",
"PolicyDefinitionReferenceIds": "",
"ExemptionCategory": "Waiver",
"DisplayName": "abc - abc-mg Policy Assignment",
"Description": "AIB Testing",
"ExpiresOn": "\/Date(1662134400000)\/",
"Metadata": ""
},
"SystemData": null,
"Name": "456",
"ResourceId": "/subscriptions/123/providers/Microsoft.Authorization/policyExemptions/789",
"ResourceName": "456",
"ResourceGroupName": null,
"ResourceType": "Microsoft.Authorization/policyExemptions",
"SubscriptionId": "123"
},
{
"Properties": {
"PolicyAssignmentId": "/providers/Microsoft.Management/managementGroups/root-mg/providers/Microsoft.Authorization/policyAssignments/111",
"PolicyDefinitionReferenceIds": "installEndpointProtection",
"ExemptionCategory": "Waiver",
"DisplayName": "root-mg - Azure Security Benchmark",
"Description": "currently use sophos and not defender",
"ExpiresOn": null,
"Metadata": ""
},
"SystemData": null,
"Name": "345",
"ResourceId": "/providers/Microsoft.Management/managementGroups/root-mg/providers/Microsoft.Authorization/policyExemptions/345",
"ResourceName": "345",
"ResourceGroupName": null,
"ResourceType": "Microsoft.Authorization/policyExemptions",
"SubscriptionId": null
}
]
Finally, I don't appear to get all Exemptions back in this loop i.e. some are set at Resource Group or Resource Level. Do I need to drill further beyond Set-AzContext?
After reproducing the same code from my end, I could able to see the expected results. However, make sure you are checking in the right file and the location to which you are sending your data to.
Finally, I don't appear to get all Exemptions back in this loop i.e. some are set at Resource Group or Resource Level.
This might be due to the scope that you are looking into. After setting the scope to the required level I could able to get the expected results. Below is the code that worked for me.
$Resource = Get-AzResource -ResourceGroupName <YOUR_RESOURCEGROUP_NAME>
for($I=0;$I -lt $Resource.ResourceId.Count;$I++)
{
$a=Get-AzPolicyExemption -Scope $Resource.ResourceId[$I]
for($J=0;$J -lt $a.Count;$J++)
{
If($a.ResourceId[$J] -Match $Resource.ResourceId[$I])
{
$exemptionsIn = Get-AzPolicyExemption -Scope $Resource.ResourceId[$I] | ConvertTo-Json
$fileName = "sample2" + ".json"
$exemptionsOut = ''
foreach($ex in $exemptionsIn|ConvertFrom-Json){
if($ex.Properties.PolicyAssignmentId -Match "Swetha*") {
$exemptionsOut += $ex|ConvertTo-Json
}
}
if ($exemptionsOut -ne '') {
$exemptionsOut | Out-File -filepath $fileName
$exemptionsOut = ''
}
}
}
}
I have few policy exemptions in my subscription but above script gave me the results at Resource level which -Match with Swetha.
RESULTS:
I cannot find any doc for Fish Shell regarding using Command Substitution more than once.
I'm trying to assign the state, city from the JSON result set (jq parser) piped from a curl API query of LocationIQ. 2 Command Substitution 1:(curl) and 2:(jq). I don't need the location variable assignment if I can get the address variable assignment
Purpose of Function:
#Take 2 arguments (Latitude, Longitude) and return 2 variables $State, $City
The JSON:
{
"address": {
"city": "Aurora",
"country": "United States of America",
"country_code": "us",
"county": "Kane County",
"postcode": "60504",
"road": "Ridge Road",
"state": "Illinois"
},
"boundingbox": [
"41.729347",
"41.730247",
"-88.264466",
"-88.261979"
],
"display_name": "Ridge Road, Aurora, Kane County, Illinois, 60504, USA",
"importance": 0.2,
"lat": "41.729476",
"licence": "https://locationiq.com/attribution",
"lon": "-88.263423",
"place_id": "333878957973"
}
My Function:
function getLocation
set key 'hidden'
set exifLat $argv[1]
set exifLon $argv[2]
set location (curl -s "https://us1.locationiq.com/v1/reverse.phpkey=$key&lat=$exifLat&lon=$exifLon&format=json" | set address (jq --raw-output '.address.state,.address.city') )
echo "Location: $location
echo "state: $address[1]"
echo "city: $address[2]"
end
Error: fish Command substitution not allowed
Works fine using only the curl Command substitution ->removing the: set address & parens for jq.
set location (curl -s "https://us1.locationiq.com/v1/reverse.phpkey=$key&lat=$exifLat&lon=$exifLon&format=json" | jq --raw-output '.address.state,.address.city')
I'm still pretty novice - maybe there is a better way to achieve my desired result: Assign the JSON State to a variable and City to a variable?
I originally tried (slicing the location[17] - City, location[19] - State) and getting inconsistent results as the fields seem to be dynamic and affecting how many results which affects the ordering.
Any help appreciated!
I find the nested set confusing. Did you intend to do use $location to hold the downloaded JSON data, and $address to hold the results of jq? If yes, split them out into separate statements
set url "https://us1.locationiq.com/v1/reverse.phpkey=$key&lat=$exifLat&lon=$exifLon&format=json"
set location (curl -s $url)
set address (echo $location | jq --raw-output '.address.state,.address.city')
It is possible to store SQL query in JSON array of objects?? Because when i have something like this:
[{
"id": "1",
"query": "SELECT ID FROM table"
},
{
"id": "2",
"query": "SELECT ID FROM table"
},
{
"id": "3",
"query": "SELECT USER FROM table"
}
]
JSON file in VSCode is ok no error it is getting nasty when i want to store complex queries with joins etc.
for example this query even if i format it correctly it will generate error in JSON file about formatting
(just example i not it is not valid)
SELECT user, id, , count(price) as numrev
FROM price
where id = 1 and user = 0
group by user, id, price
that it can't be stored in string
It is bit easy to do, but requires on extra step.
Simply convert/encode you raw SQL queries in base64 text.
Decode the text before you execute the queries in you code.
If the JSON file is created automatically by a program/code
All most all programming languages proved base64 encode / decode functions as part of the core if not download compatible package / library to achieve this automation
var queries = [{
"id": "1",
"query": "U0VMRUNUIElEIEZST00gdGFibGU="
},
{
"id": "2",
"query": "U0VMRUNUIElEIEZST00gdGFibGU="
},
{
"id": "3",
"query": "U0VMRUNUIFVTRVIgRlJPTSB0YWJsZQ=="
},
{
"id": "4",
"query": "U0VMRUNUIHVzZXIsIGlkLCAsIGNvdW50KHByaWNlKSBhcyBudW1yZXYKICBGUk9NIHByaWNlCiAgd2hlcmUgaWQgPSAxIGFuZCB1c2VyID0gMCAKICBncm91cCBieSB1c2VyLCBpZCwgcHJpY2U="
}
];
for (i = 0; i < queries.length; i++) {
console.log("id = " + queries[i].id + ", query = " + atob(queries[i].query));
}
when you parse you JSON array make sure to decode before you execute the SQL queries.
let me know this one helped you.. ☺☻☺
FYI , refer http://www.utilities-online.info/base64/
enter image description here
Having found that for one specific sheets document I was trying to reference as an external table, the heading row was being included in the data when executing queries*. I decided to drop the table and recreate it using a definitions file which would definitely expose the options from the docs. It didn't seem to work as no schema is created, despite being defined in the file.
I've recreated the issue with a simple sheet with 3 columns and a frozen header row and the following test.def file:
{
"autodetect": false,
"schema": {
"fields": [
{"name": "c1", "type": "STRING", "mode": "nullable"},
{"name": "c2", "type": "STRING", "mode": "nullable"},
{"name": "c3", "type": "STRING", "mode": "nullable"},
]
},
"sourceFormat": "GOOGLE_SHEETS",
"sourceUris": [
"https://docs.google.com/spreadsheets/..."
],
"googleSheetsOptions": {
"skipLeadingRows": 1
}
}
and then I try to create the file using:
bq mk myproject:mydataset.mytable < test.def
the table is created but no schema is present - what am I doing wrong?
this issue remains but I cannot identify why 95% of the time the table is created OK and the first/header row correctly excluded from the data returned by a query but in one specific case, created the same way as all others, the header row is returned in the data ...
Odd :(
M
OK so the correct syntax is:
bq mk --external_table_definition=myfile.def project:dataset.table
This also allows you to tell google to skip leading rows on the sheet (as of tiem of writing not possible from BQ UI)
M
I have an array:
[
{
"AssetId": 14462955,
"Name": "Cultural Item"
},
{
"AssetId": 114385498,
"Name": "Redspybot"
},
{
"AssetId": 29715011,
"Name": "American Cowboy"
},
{
"AssetId": 98253651,
"Name": "Mahem"
}
]
I would like to loop through each object in this array, and pick out the value of each key called AssetId and output it.
How would I do this using jq for the command line?
The command-line tool jq writes to STDOUT and/or STDERR. If you want to write the .AssetId information to STDOUT, then one possibility would be as follows:
jq -r ".[] | .AssetId" input.json
Output:
14462955
114385498
29715011
98253651
A more robust incantation would be: .[] | .AssetId? but your choice will depend on what you want if there is no key named "AssetId".
You can also do it via this command.
jq ".[].AssetId" input.json
if array like be that which is in my case
{
"resultCode":0,
"resultMsg":"SUCCESS",
"uniqueRefNo":"111222333",
"list":[
{
"cardType":"CREDIT CARD",
"isBusinessCard":"N",
"memberName":"Bank A",
"memberNo":10,
"prefixNo":404591
},
{
"cardType":"DEBIT CARD",
"isBusinessCard":"N",
"memberName":"Bank A",
"memberNo":10,
"prefixNo":407814
},
{
"cardType":"CREDIT CARD",
"isBusinessCard":"N",
"memberName":"Bank A",
"memberNo":10,
"prefixNo":413226
}
]
}
you can get the prefixNo with below jq command.
jq ".list[].prefixNo" input.json
For more specific case on array iterating on jq you can check this blogpost
you have a couple of choices to do the loop itself. you can apply peak's awesome answer and wrap a shell loop around it. replace echo with the script you want to run.
via xargs
$ jq -r ".[] | .AssetId" input.json | xargs -n1 echo # this would print
14462955
114385498
29715011
98253651
via raw loop
$ for i in $(jq -r ".[] | .AssetId" input.json)
do
echo $i
done
14462955
114385498
29715011
98253651
An alternative using map:
jq "map ( .AssetId ) | .[]"
For your case jq -r '.[].AssetId' should work
You can also use online JQ Parser : https://jqplay.org/
If you want to loop through the each value then can use below :
for i in $(echo $api_response | jq -r ".[].AssetId")
do
echo echo $i
done