Is there a way to get full dependency map of all contracts from the Pact Broker (preferably in json format)?
There is an API call used in the graph: https://<broker-url>/groups/<service>.csv to get data to draw the graph, but that is not great for parsing and requires a call to find all services and then a call for each service to get the dependencies.
It would be nice to have one call with a full dependency map in json format.
Yes! There is a HAL browser built in to broker, which enables you to follow the graph programmatically.
For example, you could run a query like this and filter with jq on the subset of properties you need, and re-order the output:
curl -v -u 'dXfltyFMgNOFZAxr8io9wJ37iUpY42M:O5AIZWxelWbLvqMd8PkAVycBJh2Psyg1' https://test.pact.dius.com.au/pacts/latest | jq '.pacts[]._embedded | select(.consumer.name | contains("AWSSummiteer")) | .consumer.name + "->" + .provider.name'
Which produces something like:
"AWSSummiteerSentimentSNSProvider->AWSSummiteerTwitterSNSProvider"
"AWSSummiteerTwitterSNSConsumer->AWSSummiteerTwitterSNSProvider"
"AWSSummiteerTwitterSNSProvider->Twitter"
"AWSSummiteerWeb->AWSSummiteerIoT"
"AWSSummiteerWeb->AWSSummiteerIoTPresignedUrl"
"AWSSummiteerWeb->AWSSummiteerSentimentSNSProvider"
"AWSSummiteerWeb->AWSSummiteerTwitterSNSConsumer"
"AWSSummiteerWeb->AWSSummiteerWeb"
which you could pipe into graphviz to create pretty charts but of course you could translate this into any format you like.
Here is the full graphviz visualisation:
echo "digraph { ranksep=3; ratio=auto; overlap=false; node [ shape = plaintext, fontname = "Helvetica" ];" > latest.dot ; curl -v -u 'dXfltyFMgNOFZAxr8io9wJ37iUpY42M:O5AIZWxelWbLvqMd8PkAVycBJh2Psyg1' https://test.pact.dius.com.au/pacts/latest | jq '.pacts[]._embedded | select(.consumer.name | contains("AWSSummiteer")) | .consumer.name + "->" + .provider.name' | tr -d '"' | sed 's/-/_/g' | sed 's/_>/->/g' >> latest.dot; echo "}" >> latest.dot
dot latest.dot -otest.png -Tpng
which creates this pretty picture:
Related
I try to use geomesa with redis. I thought that redis enables statistics on geomesa by default.
my redis geomesa db:
./geomesa-redis describe-schema -u localhost:6379 -c geomesa -f SignalBuilder
INFO Describing attributes of feature 'SignalBuilder'
geo | Point (Spatio-temporally indexed) (Spatially indexed)
time | Date (Spatio-temporally indexed) (Attribute indexed)
cam | String (Attribute indexed) (Attribute indexed)
imei | String
dir | Double
alt | Double
vlc | Double
sl | Integer
ds | Integer
dir_y | Double
poi_azimuth_x | Double
poi_azimuth_y | Double
User data:
geomesa.attr.splits | 0
geomesa.feature.expiry | time(30 days)
geomesa.id.splits | 0
geomesa.index.dtg | time
geomesa.indices | z3:7:3:geo:time,z2:5:3:geo,attr:8:3:time,attr:8:3:cam,attr:8:3:cam:time
geomesa.stats.enable | true
geomesa.table.partition | time
geomesa.z.splits | 0
geomesa.z3.interval | week
from doc: https://www.geomesa.org/documentation/stable/user/datastores/query_planning.html#stats-collected
Stat generation can be enabled or disabled through the simple feature type user data
using the key geomesa.stats.enable
Cached statistics, and thus cost-based query planning, are currently
only implemented for the Accumulo and Redis data stores.
*Total count, *Min/max (bounds) for the default geometry, *default date
and any indexed attributes, *Histograms for the default geometry,
default date and any indexed attributes, *Frequencies for any indexed
attributes...
Why the return time is increased when increased amount of data?
./geomesa-redis export -u localhost:6379 -c geomesa -f SignalBuilder -q "cam like '%' and bbox(geo,38,56,39,57)" --hints STATS_STRING='Enumeration(cam)'
INFO Running export - please wait...
id,stats:String,*geom:Geometry
stat,"{""5798a065-d51e-47a1-b04b-ab48df9f1324"":203215}",POINT (0 0)
INFO Feature export complete to standard out for 1 features in 2056ms
next request
/geomesa-redis export -u localhost:6379 -c geomesa -f SignalBuilder -q "cam like '%' and bbox(geo,38,56,39,57)" --hints STATS_STRING='Enumeration(cam)'
INFO Running export - please wait...
id,stats:String,*geom:Geometry
stat,"{""5798a065-d51e-47a1-b04b-ab48df9f1324"":595984}",POINT (0 0)
INFO Feature export complete to standard out for 1 features in 3418ms
How to understand that statistics are collected and saved, and used when returning hints stats, like STATS_STRING='MinMax(time)' or STATS_STRING='Enumeration(cam)'?
And how to use sampling with geotools?
I try next
geomesa-cassandra export -P 10.200.217.24:9042 -u cassandra -p cassandra \
-k geomesa -c gsm_events -f SignalBuilder \
-q "cam like '%' and time DURING 2021-12-27T16:50:38.004Z/2022-01-26T16:50:38.004Z" \
--hints SAMPLE_BY='cam';SAMPLING=0.000564
but it does not work.
Thank you for any answer.
When you run an export with a query hint for stats, GeoMesa will always run a query. If you want to use the cached statistics, use the stats-* commands instead. In code, you'd use the stats method which all GeoMesa data stores implement.
I'm doing this:
apply: init
#terraform apply -auto-approve
BUCKET=$(shell terraform output -json | jq '.S3_Bucket.value')
DYNAMODB=$(shell terraform output -json | jq '.dynamo_db_lock.value')
#echo $${BUCKET}
make shows both variables being set (I was using := but that doesn't work for me, since i need them set when i execute apply) but it's still echoing blank:
...
Outputs:
S3_Bucket = "bucket1"
dynamo_db_lock = "dyna1"
BUCKET="bucket"
DYNAMODB="dyna1"
<-- echo should print it out here
I want to use that variable for a sed afterwards...
Thanks all!
Each makefile recipe line is run in its own shell. Thus, shell variables set in one recipe line will not effect other recipe lines. You can get around this by catenating all the lines as so:
apply: init
#terraform apply -auto-approve
#BUCKET=$(shell terraform output -json | jq '.S3_Bucket.value'); \
DYNAMODB=$(shell terraform output -json | jq '.dynamo_db_lock.value'); \
echo $${BUCKET}
If I run this very straight query on my json data from an aws command I get a correct result as to how many aws server instances I have in an account:
aws ec2 describe-instances | jq -r '.Reservations[].Instances[].InstanceId'
Produces a list of 47 instance IDs which corresponds to the number of server instances I have in the account. For example:
i-01adbf1408ef1a333
i-0f92d078ce975c138
i-0e4e117c44b17b417
and on up to 47 instances
This next query still produces the correct number of results:
aws ec2 describe-instances | jq -r '.Reservations[].Instances[] | [( .InstanceId ) ]'
However, if I add a query to include the name tags of the servers I get dramatically less number of server instances reported:
aws ec2 describe-instances | jq -r '.Reservations[].Instances[] | [( (.Tags[]|select(.Key=="Name")|.Value), .InstanceId ) ]'
This is the output of that command:
"i-08d3c05eed1316c9d"
"USAMZLAB10003","i-79eebb29"
"EOMLABAMZ1306","i-dbc98af4"
"USAMZLAB10002","i-d1dc1d83"
"i-0366c9bf18d27eb96"
"i-04d061334bc2f2d6b"
"USAMZLAB10007","i-f7a680a7"
"i-090e84eff4fece2b3"
"EOMLABAMZ1303","i-7cc98a53"
"EOMLABCSE713","i-08233926"
"i-0705eb3039cd56e04"
jq: error (at <stdin>:5013): Cannot iterate over null (null)
For some reason that query reports that there are only 11 aws server instances (when there should be 47). It does report that there are servers with and without name tags. But it's not reporting the correct number of servers.
It also produces the jq error "Cannot iterate over null".
I have put the original JSON into this paste:
Original JSON
How can I make the error more verbose so I can find out what's going on?
And why does adding the name tag to the query dramatically reduce the number of results?
In your json, not all instances have a set of Tags thus the error. You would have to handle it or substitute an empty array in its place with (.Tags // []). But overall, I would write it like this:
.Reservations[].Instances[] | [ (.Tags // [] | from_entries.Name), .InstanceId ]
How can I make the error more verbose so I can find out what's going on?
You could use debug.
why does adding the name tag to the query dramatically reduce the number of results?
Because your jq program is at variance with your expectations; specifically, you have overlooked what happens when .Tags evaluates to null. To understand the mismatch, consider:
$ jq -n '{} | .Tags[]|select(.Key=="Name")|.Value'
Another issue is the handling of empty arrays. You might like to handle the case of empty arrays along the lines suggested by the following:
$ jq -n '{Tags: []} | (.Tags[] | select(.Key=="Name")|.Value) // null'
$ null
One solution
If you want null to appear whenever there isn't a tag:
.Reservations[].Instances[]
| [ ((.Tags // [])[] | select(.Key=="Name") | .Value) // null,
.InstanceId ]
Given your input, the first two lines of the output would be:
[null,"i-08d3c05eed1316c9d"]
["USAMZLAB10003","i-79eebb29"]
Variant using try
.Reservations[].Instances[]
| [ try (.Tags[] | select(.Key=="Name")|.Value) // null,
.InstanceId ]
Since that is the standard way to present output in the Perl 6 documentation, I'm using the whateverable bots to evaluate expressions via the #perl6 IRC channel or the #whateverable channel. Produced output is something like this:
10:28:19 jmerelo | p6: say 333444777 ~~ /(3+)/ │
10:28:19 evalable6 | jmerelo, rakudo-moar 5ce24929f: OUTPUT: «｢333｣ 0 => ｢333｣»
(in the WeeChat console program). From that output, I cut and paste to the document, erasing the parts I'm not interested in.
I was wondering if there was some easy way to parse and save that output directly, either server-based (some Whateverable bots save to gists, for instance), or client-based via scriptint the irssi or weechat platform.
I think the most convenient solution in this case would be to bypass irc bots and define a bash function. Something like this:
d6() { echo -n '# OUTPUT: «'; perl6 -e "$1" | sed -z 's/\n//g'; echo '»'; }
Then you can use it like this:
d6 'say 42'
Which will produce this output:
# OUTPUT: «42»
Of course, you'd need a different solution for other operating systems.
As a bonus, you can also put it into the clipboard automatically:
d6 'say 42' | tee >(xclip -selection clipboard)
I'm using a script that uses curl to obtain specific array values from a configuration. Tom Fenech helped with the column formatting. I've run into another issue where the column formatting is skewed by the tr and cut commands. The command output of one array set contains square brackets surrounding the object's key. I'm using tr or cut to remove the surrounding brackets, and they do the job, however the column format is skewed. Here's my code:
# get the device groups
d_groups=`curl -H "X-Person-Token: $auth_token" -H "X-Person-Email: $auth_email" -k "$api_host/api/v1/device_groups"`
grp_uuid=`echo $d_groups | jq '.[] | .uuid'`
grp_name=`echo $d_groups | jq '.[] | .name'`
grp_desc=`echo $d_groups | jq '.[] | .description'`
grp_dev=`echo $d_groups | jq '.[] | .devices'`
echo "========== DEVICES =========="
paste <(printf 'DEVICE_GROUP_NAME\n%s\n' "$grp_name") <(printf 'DESCRIPTION\n%s\n' "$grp_desc") <(printf 'DEVICE_GROUP_UUID\n%s\n' "$grp_uuid") <(printf 'DEVICES\n%s\n' "$grp_dev" | cut -c2-39) | column -t
echo ""
exit 0
I've also tried using 'tr -d "[]"' with the same results as using cut. Here are the results of the above script:
========== DEVICES ==========
DEVICE_GROUP_NAME DESCRIPTION DEVICE_GROUP_UUID DEVICES
"Auto_API_GP1" "Auto_API_GP1" "03be550b-3744-484e-88c5-d25e1ed865c0"
"Auto_API_GP2" "Auto_API_GP2" "3a8e2ee4-3a59-4fba-aaf0-0d527d20fe13" "1a7a2092-29bd-4178-88e9-1373ee5886de"
"Auto_API_GP3" "Auto_API_GP3" "ba1ead89-34f5-4084-a9a2-8a681e83164d"
"81a3969c-1cc3-4f13-8602-afcb981d5295"
"8bbfd1a7-e048-4148-abeb-354763b6aef7"
What I'm expecting for results:
========== DEVICES ==========
DEVICE_GROUP_NAME DESCRIPTION DEVICE_GROUP_UUID DEVICES
"Auto_API_GP1" "Auto_API_GP1" "03be550b-3744-484e-88c5-d25e1ed865c0" "1a7a2092-29bd-4178-88e9-1373ee5886de"
"Auto_API_GP2" "Auto_API_GP2" "3a8e2ee4-3a59-4fba-aaf0-0d527d20fe13" "81a3969c-1cc3-4f13-8602-afcb981d5295"
"Auto_API_GP3" "Auto_API_GP3" "ba1ead89-34f5-4084-a9a2-8a681e83164d" "8bbfd1a7-e048-4148-abeb-354763b6aef7"
I'm an absolute beginner at this, any insight is very much appreciated. Thanks!
So I stopped staring at the script long enough to get lunch and coffee and figured that the code I posted above was removing the brackets on the first output but was either entering a new line or replacing the bracket with a new line. So I simply replaced
tr -d '[]'
with
tr -s '[]' '\n'
and it now works as expected.
I do humbly except any corrections or a more efficient way. :)