I'm using a script that uses curl to obtain specific array values from a configuration. Tom Fenech helped with the column formatting. I've run into another issue where the column formatting is skewed by the tr and cut commands. The command output of one array set contains square brackets surrounding the object's key. I'm using tr or cut to remove the surrounding brackets, and they do the job, however the column format is skewed. Here's my code:
# get the device groups
d_groups=`curl -H "X-Person-Token: $auth_token" -H "X-Person-Email: $auth_email" -k "$api_host/api/v1/device_groups"`
grp_uuid=`echo $d_groups | jq '.[] | .uuid'`
grp_name=`echo $d_groups | jq '.[] | .name'`
grp_desc=`echo $d_groups | jq '.[] | .description'`
grp_dev=`echo $d_groups | jq '.[] | .devices'`
echo "========== DEVICES =========="
paste <(printf 'DEVICE_GROUP_NAME\n%s\n' "$grp_name") <(printf 'DESCRIPTION\n%s\n' "$grp_desc") <(printf 'DEVICE_GROUP_UUID\n%s\n' "$grp_uuid") <(printf 'DEVICES\n%s\n' "$grp_dev" | cut -c2-39) | column -t
echo ""
exit 0
I've also tried using 'tr -d "[]"' with the same results as using cut. Here are the results of the above script:
========== DEVICES ==========
DEVICE_GROUP_NAME DESCRIPTION DEVICE_GROUP_UUID DEVICES
"Auto_API_GP1" "Auto_API_GP1" "03be550b-3744-484e-88c5-d25e1ed865c0"
"Auto_API_GP2" "Auto_API_GP2" "3a8e2ee4-3a59-4fba-aaf0-0d527d20fe13" "1a7a2092-29bd-4178-88e9-1373ee5886de"
"Auto_API_GP3" "Auto_API_GP3" "ba1ead89-34f5-4084-a9a2-8a681e83164d"
"81a3969c-1cc3-4f13-8602-afcb981d5295"
"8bbfd1a7-e048-4148-abeb-354763b6aef7"
What I'm expecting for results:
========== DEVICES ==========
DEVICE_GROUP_NAME DESCRIPTION DEVICE_GROUP_UUID DEVICES
"Auto_API_GP1" "Auto_API_GP1" "03be550b-3744-484e-88c5-d25e1ed865c0" "1a7a2092-29bd-4178-88e9-1373ee5886de"
"Auto_API_GP2" "Auto_API_GP2" "3a8e2ee4-3a59-4fba-aaf0-0d527d20fe13" "81a3969c-1cc3-4f13-8602-afcb981d5295"
"Auto_API_GP3" "Auto_API_GP3" "ba1ead89-34f5-4084-a9a2-8a681e83164d" "8bbfd1a7-e048-4148-abeb-354763b6aef7"
I'm an absolute beginner at this, any insight is very much appreciated. Thanks!
So I stopped staring at the script long enough to get lunch and coffee and figured that the code I posted above was removing the brackets on the first output but was either entering a new line or replacing the bracket with a new line. So I simply replaced
tr -d '[]'
with
tr -s '[]' '\n'
and it now works as expected.
I do humbly except any corrections or a more efficient way. :)
Related
I am loading data into SQL Server using bcp command line utility.
The problem is data is coming is like below. Every field in double quotes and I have to skip last three rows because it has some trailers. How can i solve this?
Thanks in Advance. I appreciate if you could help in this.
bcp database.schema.tablename in Filename.text -T -c -t"|" -r"0x0a" -F 3 -m 2
UHDR 20211110
"DATE","CUSIP","ISIN","SEDOL","TICKER","DESCRIPTION","QUANTITY","RATE","COMMENT","MARKET","FEE"
11/10/2021|""|"CA45826T3010"|"BMVXZT5"|"ITR"|"INTEGRA RESOURCES CORP REGISTERED SHS"|"28712"|"0.0000"|"HTB"|"CA"|"11.5000"
If you want to:
unconditionally remove all " chars.
unconditionally skip the last 3 lines:
(Get-Content -ReadCount 0 Filename.text) -replace '"' |
Select-Object -SkipLast 3 |
Set-Content Filename_CleanedUp.text
Note: -ReadCount 0 is a performance optimization that makes Get-Content read all lines into a single array instead of streaming the lines one by one.
Then pass Filename_CleanedUp.text to your bcp command.
If I run this very straight query on my json data from an aws command I get a correct result as to how many aws server instances I have in an account:
aws ec2 describe-instances | jq -r '.Reservations[].Instances[].InstanceId'
Produces a list of 47 instance IDs which corresponds to the number of server instances I have in the account. For example:
i-01adbf1408ef1a333
i-0f92d078ce975c138
i-0e4e117c44b17b417
and on up to 47 instances
This next query still produces the correct number of results:
aws ec2 describe-instances | jq -r '.Reservations[].Instances[] | [( .InstanceId ) ]'
However, if I add a query to include the name tags of the servers I get dramatically less number of server instances reported:
aws ec2 describe-instances | jq -r '.Reservations[].Instances[] | [( (.Tags[]|select(.Key=="Name")|.Value), .InstanceId ) ]'
This is the output of that command:
"i-08d3c05eed1316c9d"
"USAMZLAB10003","i-79eebb29"
"EOMLABAMZ1306","i-dbc98af4"
"USAMZLAB10002","i-d1dc1d83"
"i-0366c9bf18d27eb96"
"i-04d061334bc2f2d6b"
"USAMZLAB10007","i-f7a680a7"
"i-090e84eff4fece2b3"
"EOMLABAMZ1303","i-7cc98a53"
"EOMLABCSE713","i-08233926"
"i-0705eb3039cd56e04"
jq: error (at <stdin>:5013): Cannot iterate over null (null)
For some reason that query reports that there are only 11 aws server instances (when there should be 47). It does report that there are servers with and without name tags. But it's not reporting the correct number of servers.
It also produces the jq error "Cannot iterate over null".
I have put the original JSON into this paste:
Original JSON
How can I make the error more verbose so I can find out what's going on?
And why does adding the name tag to the query dramatically reduce the number of results?
In your json, not all instances have a set of Tags thus the error. You would have to handle it or substitute an empty array in its place with (.Tags // []). But overall, I would write it like this:
.Reservations[].Instances[] | [ (.Tags // [] | from_entries.Name), .InstanceId ]
How can I make the error more verbose so I can find out what's going on?
You could use debug.
why does adding the name tag to the query dramatically reduce the number of results?
Because your jq program is at variance with your expectations; specifically, you have overlooked what happens when .Tags evaluates to null. To understand the mismatch, consider:
$ jq -n '{} | .Tags[]|select(.Key=="Name")|.Value'
Another issue is the handling of empty arrays. You might like to handle the case of empty arrays along the lines suggested by the following:
$ jq -n '{Tags: []} | (.Tags[] | select(.Key=="Name")|.Value) // null'
$ null
One solution
If you want null to appear whenever there isn't a tag:
.Reservations[].Instances[]
| [ ((.Tags // [])[] | select(.Key=="Name") | .Value) // null,
.InstanceId ]
Given your input, the first two lines of the output would be:
[null,"i-08d3c05eed1316c9d"]
["USAMZLAB10003","i-79eebb29"]
Variant using try
.Reservations[].Instances[]
| [ try (.Tags[] | select(.Key=="Name")|.Value) // null,
.InstanceId ]
Is there a way to get full dependency map of all contracts from the Pact Broker (preferably in json format)?
There is an API call used in the graph: https://<broker-url>/groups/<service>.csv to get data to draw the graph, but that is not great for parsing and requires a call to find all services and then a call for each service to get the dependencies.
It would be nice to have one call with a full dependency map in json format.
Yes! There is a HAL browser built in to broker, which enables you to follow the graph programmatically.
For example, you could run a query like this and filter with jq on the subset of properties you need, and re-order the output:
curl -v -u 'dXfltyFMgNOFZAxr8io9wJ37iUpY42M:O5AIZWxelWbLvqMd8PkAVycBJh2Psyg1' https://test.pact.dius.com.au/pacts/latest | jq '.pacts[]._embedded | select(.consumer.name | contains("AWSSummiteer")) | .consumer.name + "->" + .provider.name'
Which produces something like:
"AWSSummiteerSentimentSNSProvider->AWSSummiteerTwitterSNSProvider"
"AWSSummiteerTwitterSNSConsumer->AWSSummiteerTwitterSNSProvider"
"AWSSummiteerTwitterSNSProvider->Twitter"
"AWSSummiteerWeb->AWSSummiteerIoT"
"AWSSummiteerWeb->AWSSummiteerIoTPresignedUrl"
"AWSSummiteerWeb->AWSSummiteerSentimentSNSProvider"
"AWSSummiteerWeb->AWSSummiteerTwitterSNSConsumer"
"AWSSummiteerWeb->AWSSummiteerWeb"
which you could pipe into graphviz to create pretty charts but of course you could translate this into any format you like.
Here is the full graphviz visualisation:
echo "digraph { ranksep=3; ratio=auto; overlap=false; node [ shape = plaintext, fontname = "Helvetica" ];" > latest.dot ; curl -v -u 'dXfltyFMgNOFZAxr8io9wJ37iUpY42M:O5AIZWxelWbLvqMd8PkAVycBJh2Psyg1' https://test.pact.dius.com.au/pacts/latest | jq '.pacts[]._embedded | select(.consumer.name | contains("AWSSummiteer")) | .consumer.name + "->" + .provider.name' | tr -d '"' | sed 's/-/_/g' | sed 's/_>/->/g' >> latest.dot; echo "}" >> latest.dot
dot latest.dot -otest.png -Tpng
which creates this pretty picture:
Since that is the standard way to present output in the Perl 6 documentation, I'm using the whateverable bots to evaluate expressions via the #perl6 IRC channel or the #whateverable channel. Produced output is something like this:
10:28:19 jmerelo | p6: say 333444777 ~~ /(3+)/ │
10:28:19 evalable6 | jmerelo, rakudo-moar 5ce24929f: OUTPUT: «｢333｣ 0 => ｢333｣»
(in the WeeChat console program). From that output, I cut and paste to the document, erasing the parts I'm not interested in.
I was wondering if there was some easy way to parse and save that output directly, either server-based (some Whateverable bots save to gists, for instance), or client-based via scriptint the irssi or weechat platform.
I think the most convenient solution in this case would be to bypass irc bots and define a bash function. Something like this:
d6() { echo -n '# OUTPUT: «'; perl6 -e "$1" | sed -z 's/\n//g'; echo '»'; }
Then you can use it like this:
d6 'say 42'
Which will produce this output:
# OUTPUT: «42»
Of course, you'd need a different solution for other operating systems.
As a bonus, you can also put it into the clipboard automatically:
d6 'say 42' | tee >(xclip -selection clipboard)
I wrote a command line
sc query PlugPlay | FIND "SERVICE_NAME" | FIND "STATE"
to list only the service name and its status but it's not giving any output.
Please correct me how to list the service name and its STATE (running or stopped) only.
You can do this with Windows' built-in findstr command. If you give it multiple words to find, separated by spaces, it will print lines that match any word (i.e. findstr "a b" is equivalent to grep -E 'a|b').
sc query plugplay | findstr "SERVICE_NAME STATE"
Running two pipes like that is not an "or" operation, it is an "and" operation. It will only output lines that include both SERVICE_NAME and STATE (which will be none, so no output is correct). If you run just the first find it gives
C:\>sc query PlugPlay | FIND "SERVICE_NAME"
SERVICE_NAME: PlugPlay
C:\>
and thus the STATE information is already removed.
The windows find command is too simple and limited to do what you want, but it can be achieved using the unix grep command. From cygwin for instance:
$ sc query PlugPlay | grep -E 'SERVICE_NAME|STATE'
SERVICE_NAME: PlugPlay
STATE : 4 RUNNING
$