In TensorFlow, I found that I can do the following,
from tensorflow.core import framework
from google.protobuf import json_format
graph_def = framework.graph_pb2.GraphDef()
node_def = framework.node_def_pb2.NodeDef()
graph_def.node.extend([node_def])
print json_format.MessageToJson(graph_def)
Which prints
{
"node": [
{}
]
}
Now, my node is not actually set to an Operation. I can't figure out how to make node_def be an operation. I can construct OpDefs via,
from tensorflow.python.ops import gen_array_ops
const_op_def = gen_array_ops._InitOpDefLibrary()._ops['Const'].op_def
Which is of class <class 'tensorflow.core.framework.op_def_pb2.OpDef'>
I would like to make my NodeDef register as this OpDef.
EDIT:
>>> print json_format.MessageToJson(gen_array_ops._InitOpDefLibrary()._ops['Const'].op_def)
{
"outputArg": [
{
"typeAttr": "dtype",
"name": "output"
}
],
"name": "Const",
"attr": [
{
"type": "tensor",
"name": "value"
},
{
"type": "type",
"name": "dtype"
}
]
}
I believe you are looking for a way to set message fields inside of the GraphDef and NodeDef protos. How to modify ProtoBuf objects in Python is detailed here
Related
Hi, I am working on an api with geotiff and geojson files. Do you have a way to use geotiff.py to convert an image file (.tif) to geojson without using GDAL?
Thanks in advance for your help.
I tried to create this api (unfinished) but I don't have much knowledge of non-GDAL uses. use python
#app.route("imgeo/upload-img.html")
def upload_file():
if request.method == 'POST':
geo_tiff = GeoTiff(tiff_archivo, as_crs=none, banda=1)
zarr_array = geo_tiff.read()
def server_info():
ubication = [
{
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"geometry": {
"type": "Polygon",
"coordinates": [-74.006393, 40.714172] # for example
},
"properties": {
"name": "your ubication",
"description": "your ubication"
}
}
]
}
]
return jsonify(ubication)
if __name__ == '__main__':
# Start the application
app.run(debug=True)
I have been working on my own validator for JSON schema and FINALLY have most of how unevaluatedProperties are supposed to work,... I think. That's one tricky piece there! However I really just want to confirm one thing. Given the following schema and JSON, what is the expected outcome... I have tried it with a https://www.jsonschemavalidator.net and gotten an answer, but I was hoping I could get a more definitive answer.
The focus is the faz property is in fact being evaluated, but the command to disallow unevaluatedProperties comes from a deeply nested schema.
Thoguhts?
Here is the schema...
{
"type": "object",
"properties": {
"foo": {
"type": "object",
"properties": {
"bar": {
"type": "string"
}
},
"unevaluatedProperties": false
}
},
"anyOf": [
{
"properties": {
"foo": {
"properties": {
"faz": {
"type": "string"
}
}
}
}
}
]
}
Here is the JSON...
{
"foo": {
"bar": "test",
"faz": "test"
}
}
That schema will successfully evaluate against the provided data. The unevaluatedProperties keyword will be aware of properties evaluated in subschemas of adjacent keywords, and is evaluated after all other applicator keywords, so it will see the annotation produced from within the anyOf subschema, also.
Evaluating this keyword is easy if you follow the specification literally -- it uses annotations to decide what to do. You just need to make sure that all keywords either produce annotations correctly or propagate annotations correctly that were produced by other keywords, and then all the information is available to generate the correct result.
The result produced by my implementation is:
{
"annotations" : [
{
"annotation" : [
"faz"
],
"instanceLocation" : "/foo",
"keywordLocation" : "/anyOf/0/properties/foo/properties"
},
{
"annotation" : [
"foo"
],
"instanceLocation" : "",
"keywordLocation" : "/anyOf/0/properties"
},
{
"annotation" : [
"bar"
],
"instanceLocation" : "/foo",
"keywordLocation" : "/properties/foo/properties"
},
{
"annotation" : [],
"instanceLocation" : "/foo",
"keywordLocation" : "/properties/foo/unevaluatedProperties"
},
{
"annotation" : [
"foo"
],
"instanceLocation" : "",
"keywordLocation" : "/properties"
}
],
"valid" : true
}
This is not an answer but a follow up example which I feel is in the same vein. I feel this guides us to the answer.
Here we have a single object being validated. But the unevaluated command resides in two different schemas each a part of a different "adjacent keyword subschemas"(from the core spec http://json-schema.org/draft/2020-12/json-schema-core.html#rfc.section.11)
How should this be resolved. If all annotations must be evaluated then in what order do I evaluate? The oneOf first or the anyOf? According the spec an unevaluated command(properties or items) generate annotation results which means that that result would affect any other unevaluated command.
http://json-schema.org/draft/2020-12/json-schema-core.html#unevaluatedProperties
"The annotation result of this keyword is the set of instance property names validated by this keyword's subschema."
This is as far as I am understanding the spec.
According to the two validators I am using this fails.
Schema
{
"$schema": "https://json-schema.org/draft/2019-09/schema",
"type": "object",
"properties": {
"foo": {
"type": "string"
}
},
"oneOf": [
{
"properties": {
"faz": {
"type": "string"
}
},
"unevaluatedProperties": true
}
],
"anyOf": [
{
"properties": {
"bar": {
"type": "string"
}
},
"unevaluatedProperties": false
}
]
}
Data
{
"bar": "test",
"faz": "test",
}
We are running batch spark jobs using AWS EMR clusters. Those jobs run periodically and we would like to orchestrate those via AWS Step Functions.
As of November 2019 Step Functions has support for EMR natively. When adding a Step to the cluster we can use the following config:
"Some Step": {
"Type": "Task",
"Resource": "arn:aws:states:::elasticmapreduce:addStep.sync",
"Parameters": {
"ClusterId.$": "$.cluster.ClusterId",
"Step": {
"Name": "FirstStep",
"ActionOnFailure": "CONTINUE",
"HadoopJarStep": {
"Jar": "command-runner.jar",
"Args": [
"spark-submit",
"--class",
"com.some.package.Class",
"JarUri",
"--startDate",
"$.time",
"--daysToLookBack",
"$.daysToLookBack"
]
}
}
},
"Retry" : [
{
"ErrorEquals": [ "States.ALL" ],
"IntervalSeconds": 1,
"MaxAttempts": 1,
"BackoffRate": 2.0
}
],
"ResultPath": "$.firstStep",
"End": true
}
Within the Args List of the HadoopJarStep we would like to set arguments dynamically. e.g. if the input of the state machine execution is:
{
"time": "2020-01-08",
"daysToLookBack": 2
}
The strings in the config starting with "$." should be replaced accordingly when executing the State Machine, and the step on the EMR cluster should run command-runner.jar spark-submit --class com.some.package.Class JarUri --startDate 2020-01-08 --daysToLookBack 2. But instead it runs command-runner.jar spark-submit --class com.some.package.Class JarUri --startDate $.time --daysToLookBack $.daysToLookBack.
Does anyone know if there is a way to do this?
Parameters allow you to define key-value pairs, so as the value for the "Args" key is an array, you won't be able to dynamically reference a specific element in the array, you would need to reference the whole array instead. For example "Args.$": "$.Input.ArgsArray".
So for your use-case the best way to achieve this would be to add a pre-processing state, before calling this state. In the pre-processing state you can either call a Lambda function and format your input/output through code or for something as simple as adding a dynamic value to an array you can use a Pass State to reformat the data and then inside your task State Parameters you can use JSONPath to get the array which you defined in in the pre-processor. Here's an example:
{
"Comment": "A Hello World example of the Amazon States Language using Pass states",
"StartAt": "HardCodedInputs",
"States": {
"HardCodedInputs": {
"Type": "Pass",
"Parameters": {
"cluster": {
"ClusterId": "ValueForClusterIdVariable"
},
"time": "ValueForTimeVariable",
"daysToLookBack": "ValueFordaysToLookBackVariable"
},
"Next": "Pre-Process"
},
"Pre-Process": {
"Type": "Pass",
"Parameters": {
"FormattedInputsForEmr": {
"ClusterId.$": "$.cluster.ClusterId",
"Args": [
{
"Arg1": "spark-submit"
},
{
"Arg2": "--class"
},
{
"Arg3": "com.some.package.Class"
},
{
"Arg4": "JarUri"
},
{
"Arg5": "--startDate"
},
{
"Arg6.$": "$.time"
},
{
"Arg7": "--daysToLookBack"
},
{
"Arg8.$": "$.daysToLookBack"
}
]
}
},
"Next": "Some Step"
},
"Some Step": {
"Type": "Pass",
"Parameters": {
"ClusterId.$": "$.FormattedInputsForEmr.ClusterId",
"Step": {
"Name": "FirstStep",
"ActionOnFailure": "CONTINUE",
"HadoopJarStep": {
"Jar": "command-runner.jar",
"Args.$": "$.FormattedInputsForEmr.Args[*][*]"
}
}
},
"End": true
}
}
}
You can use the States.Array() intrinsic function. Your Parameters becomes:
"Parameters": {
"ClusterId.$": "$.cluster.ClusterId",
"Step": {
"Name": "FirstStep",
"ActionOnFailure": "CONTINUE",
"HadoopJarStep": {
"Jar": "command-runner.jar",
"Args.$": "States.Array('spark-submit', '--class', 'com.some.package.Class', 'JarUri', '--startDate', $.time, '--daysToLookBack', '$.daysToLookBack')"
}
}
}
Intrinsic functions are documented here but I don't think it explains the usage very well. The code snippets provided in the Step Functions console are more useful.
Note that you can also do string formatting on the args using States.Format(). For example, you could construct a path using an input variable as the final path segment:
"Args.$": "States.Array('mycommand', '--path', States.Format('my/base/path/{}', $.someInputVariable))"
How can I reference a schema (B in this case) from itself no matter where it is nested?
I would like to be able to validate against schema A and against schema B. E.g.
from jsonschema import validate
validate(a, A)
validate(b, B)
Why? For unit tests and for speed improvements (if one of the B's fails I don't need to validate A).
Schema A references schema B and B references itself (see examples below).
I have tried the following:
a) Using definitions in A works only if validating against A. Validating B fails with jsonschema.exceptions.RefResolutionError: Unresolvable JSON pointer: 'definitions/b'. Which makes sense because the definition only exists on A.
from jsonschema import validate
B = {
"anyOf": [
{
"type": "object"
},
{
"type": "array",
"items": {"$ref": "#/definitions/b"}
}
]
}
A = {
"definitions": {
"b": B
},
"type": "array",
"items": {"$ref": "#/definitions/b"}
}
validate([], A)
validate([], B) # RefResolutionError
b) Using "$id": "http://example.org/B" in B and reference itself with: {"$ref": "http://example.org/B"} makes an actual (unnecessary http request) to a URL that does not exist. There is no need to make that request as the information is present in the document. Attempts using a non-url such as just #B fails with jsonschema.exceptions.RefResolutionError: unknown url type.
from jsonschema import validate
B = {
"$id": "http://example.org/schema/B",
"anyOf": [
{
"type": "object"
},
{
"type": "array",
"items": {"$ref": "http://example.org/schema/B"}
}
]
}
A = {
"type": "array",
"items": {"$ref": "http://example.org/schema/B"}
}
validate([{}], A) # RefResolutionError: HTTP Error 404: Not Found
validate([{}], B) # RefResolutionError: HTTP Error 404: Not Found
I had a problem kinda like this. I found upgrading my jsonschema module from somewhere in version 3 to jsonschema >= 4.3.2 fixed it immediately so may simply be a bug.
pip install --upgrade jsonschema
In a markdown cell in an ipython3 notebook (4.0.0) I include an svg that is located together with the notebook file:
<img src="NewTux.svg"/>
In the normal notebook view it is displayed as expected.
However, when I try to export to pdf the image does not show up.
What puzzles me is that a matplotlib plot (with %config InlineBackend.figure_format = 'svg') perfectly shows both on screen AND in the exported PDF.
How can I get a PDF including also the svgs which are not plotted but just included as a figure in markdown?
(A workaround is to print to pdf in browser, but then I miss LaTeX-formatting and formula and color in the syntax-highlighting of the code sections).
Minimum working example for the ipython file is:
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"%matplotlib inline\n",
"%config InlineBackend.figure_format = 'svg'\n",
"import numpy as np\n",
"import matplotlib.pyplot as pp\n",
"\n",
"x = np.arange(0,10,0.05)\n",
"y = np.sin(x)\n",
"\n",
"pp.plot(x,y)\n",
"pp.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<img src=\"NewTux.svg\">"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.4.3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
Where I downloaded: NewTux.svg from the Wikimedia commons.