I have an application that requests JSON objects from various other applications via their REST APIs. The response from any application comes in the following format:
{
data : {
key1: { val: value, defBy: "ontology class"}
key2: ...,
}
}
The following code depicts an object from App1:
{
data : {
key1: { val: "98404506-385576361", defBy: "abc:SHA-224"}
}
}
The following code depicts an object from App2:
{
data : {
key2: { val: "495967838-485694812", defBy: "xyz:SHA3-224"}
}
}
Here, DefBy refers to the algorithm used to encrypt the string in val. When my application receives such objects, it parses the JSON and converts each kv in the object into RDF such that:
// For objects from App1:
key1 rdf:type osba:key
key1 osba:generatedBy abc:SHA-224
...
// For objects from App2
key2 rdf:type osba:key
key2 osba:generatedBy xyz:SHA3-224
I need to query the generated RDF data in a way that I can specify if osba:generatedBy of any key belongs to the SHA family, then return the subject as a valid query-result, such that: where {?k osba:generatedBy ???}
Please note the following points:
I also receive objects with other encryption algorithms such as MD5, etc.
I don't know in advance what encryption algorithm will be used by a new application joining the network nor what NS it uses. For example, in the above objects, one uses abc:, and the other uses xyz:.
I can't use SPARQL filtering because the value could be SecureHashAlgorithm instead of SHA
My problem is that I can't define an upper (referenced) ontology in advance and map the value stored in defBy: of the incoming objects, because I don't know in advance what ontology is used nor what encryption algorithm the value represents.
I read about Automatic Ontology Integration, Alignment, Mapping, etc,. but I can't find the rationale of this concept to my problem.
Any solutions?
3) I can't use SPARQL filtering because the value could be SecureHashAlgorithm instead of SHA
SPARQL filtering supports matching against regular expressions as defined by xpath. Thus, something along the line of
SELECT ?key
WHERE { ?key osba:generatedBy ?generator
FILTER regex(?generator, "^s(ecure)?h(ash)?a(lgorithm)?.*", "i") }
(note: untested) should do the job. To build a good regex I can recommend http://regexr.com/
In case it's necessary: You can convert an IRI to a string (for matching) with the str() function.
Related
I'd like to create a JSON strategy using rust proptest library. However, I do not want to create an arbitrary JSON. I'd like to create it according to a schema (more specifically, OpenAPI schema). This means that keys of the JSON are known and I do not want to create them using any strategy, but I'd like to create the values using the strategy (pretty-much recursively).
I already implemented the strategy for primitive types, but I do not how to create a JSON object strategy.
I would like the strategy to have the type BoxedStratedy<serde_json::Value> or be able to map the strategy to this type because the JSON objects can contain other objects, and thus I need to be able to compose the strategies.
I found a HashMapStrategy strategy, however, it can be only created by a hash_map function that takes two strategies - one for generating keys and one for values. I thought that I could use Just strategy for the keys, but it did not lead anywhere. Maybe prop_filter_map could be used.
Here is the code. There are tests too. One is passing because it tests only primitive type and the other is failing since I did not find a way to implement generate_json_object function.
I tried this but the types do not match. Instead of a strategy of map from string to JSON value, it is a strategy of a map from string to BoxedStrategy.
fn generate_json_object(object: &ObjectType) -> BoxedStrategy<serde_json::Value> {
let mut json_object = serde_json::Map::with_capacity(object.properties.len());
for (name, schema) in &object.properties {
let schema_kind = &schema.to_item_ref().schema_kind;
json_object.insert(name.clone(), schema_kind_to_json(schema_kind));
}
Just(serde_json::Value::Object(json_object)).boxed()
}
One can create a vector of strategies, which implements a Strategy trait and can be boxed. So to create a serde_json::Value::Object, we create a vector of tuples. The first element will be a Just of key and the second element will be a boxed strategy of value. The boxed strategy of value can be created by schema_kind_to_json function. After we have a vector of tuples which implement a Strategy, we can use .prop_map to transform it to a serde_json::Value::Object.
fn generate_json_object(object: &ObjectType) -> BoxedStrategy<serde_json::Value> {
let mut vec = Vec::with_capacity(object.properties.len());
for (name, schema) in &object.properties {
let schema_kind = &schema.to_item_ref().schema_kind;
vec.push((Just(name.clone()), schema_kind_to_json(schema_kind)));
}
vec.prop_map(|vec| serde_json::Value::Object(serde_json::Map::from_iter(vec)))
.boxed()
}
We already have flatbuffer library embedded in our software code for simple schemas with JSON output data generation.
More update: We are generating the header files using flatc compiler against the schema and integrate these files inside of our code along with FB library for further serialization/deserialization.
Now we also need to have the following schema tree to be supported.
namespace SampleNS;
/// user defined key value pairs to add custom metadata
/// key namespacing is the responsibility of the user
table KeyValue {
key:string (key, required);
value:string (required);
}
enum SchemaVersion:byte {
V1,
V2
}
table Sometable {
value1:ubyte;
value2:ushort (key);
}
table ComponentData {
inputs: [Sometable];
outputs: [Sometable];
}
table Node {
name:string (key);
/// IO definition
data:ComponentData;
/// nested child
child:[Components];
}
table Components {
type:ubyte;
index:ubyte;
nodes:[Node];
}
table GroupMasterData {
schemaversion:SchemaVersion = sampleNS::SchemaVersion::V1;
metainfo:[KeyValue];
/// List of expected components in the system
components:[Components];
}
root_type GroupMasterData;
As from above, table Components is nested recursively. The intention is components may have childs that have the same fields.
I have few queries:
Flatc didnt gave me any error during schema compilation for such
recursive nested tables. But is this supported during the field
access for such tables?
I tried to generate a sample json data file based on above data but I
could not see the field for schemaversion. I learned FB doesn't
serialize the default values. so, I removed the default value that I
assigned in the schema. But, it still doesnt write into the json data
file. On this I also learned we can forcefully write into the file
using force_defaults option. I don't know where is this is to be
put: in the attribute or elsewhere?
Can I create a struct of enum field?
Is their any API to set Flatbuffer options that we otherwise pass to the compiler arguments? or if not, may be I think we have to tinker with the FB library code. Please suggest.
Method 1:
In our serialization method, we do this:
flatbuffers::Parser* parser = new flatbuffers::Parser();
parser->opts.output_default_scalars_in_json = true;
Is this the right method or should I use any other API?
Yes, trees (and even DAG) structures are fully supported. The type definition is recursive, but the data will eventually have leaf nodes with an empty vector of children, presumably.
The integer value for V1 is 0, and that is also the default value for all fields with no explicit default assigned. Use --defaults-json to see this field when converting. Note that explicit versions in a schema is an anti-pattern, since schemas are naturally evolvable without breaking backwards compatibility.
You can put enum fields in structs, yes. Is that what you mean?
I'm new to ReasonML, but I read through most of the official documents. I could go through the casual trial and errors for this, but since I need to write codes in ReasonML right now, I'd like to know the best practices of iterating keys and values of reason record types.
I fully agree with #Shawn that you should use a more appropriate data structure. A list of tuples, for example, is a nice and easy way to pass in a user-defined set of homogeneous key/value pairs:
fooOnThis([
("test1", ["a", "b", "c"]),
("test2", ["c"]),
])
If you need heterogeneous data I would suggest using a variant to specify the data type:
type data =
| String(string)
| KvPairs(list((string, data)));
fooOnThis([
("test1", [String("a"), String("b"), String("c")]),
("test2", [String("c"), KvPairs([("innerTest", "d")])]),
])
Alternatively you can use objects instead of records, which seems like what you actually want.
For the record, a record requires a pre-defined record type:
type record = {
foo: int,
bar: string,
};
and this is how you construct them:
let value = {
foo: 42,
bar: "baz",
};
Objects on the other hand are structurally typed, meaning they don't require a pre-defined type, and you construct them slightly differently:
let value
: {. "foo": int, "bar": string }
= {"foo": 42, "bar": "baz"};
Notice that the keys are strings.
With objects you can use Js.Obj.keys to get the keys:
let keys = Js.Obj.keys(value); // returns [|"foo", "bar"|]
The problem now is getting the values. There is no Js.Obj API for getting the values or entries because it would either be unsound or very impractical. To demonstrate that, let's try making it ourselves.
We can easily write our own binding to Object.entries:
[#bs.val] external entries: Js.t({..}) => array((string, _)) = "Object.entries";
entries here is a function that takes any object and returns an array of tuples with string keys and values of a type that will be inferred based on how we use them. This is neither safe, because we don't know what the actual value types are, or particularly practical as it will be homogeneously typed. For example:
let fields = entries({"foo": 42, "bar": "baz"});
// This will infer the value's type as an `int`
switch (fields) {
| [|("foo", value), _|] => value + 2
| _ => 0
};
// This will infer the value's type as an `string`, and yield a type error
// because `fields` can't be typed to hold both `int`s and `string`s
switch (fields) {
| [|("foo", value), _|] => value ++ "2"
| _ => ""
};
You can use either of these switch expressions (with unexpected results and possible crashes at runtime), but not both together as there is no unboxed string | int type to be inferred in Reason.
To get around this we can make the value an abstract type and use Js.Types.classify to safely get the actual underlying data type, akin to using typeof in JavaScript:
type value;
[#bs.val] external entries: Js.t({..}) => array((string, value)) = "Object.entries";
let fields = entries({"foo": 42, "bar": "baz"});
switch (fields) {
| [|("foo", value), _|] =>
switch (Js.Types.classify(value)) {
| JSString(str) => str
| JSNumber(number) => Js.Float.toString(number)
| _ => "unknown"
}
| _ => "unknown"
};
This is completely safe but, as you can see, not very practical.
Finally, we can actually modify this slightly to use it safely with records as well, by relying on the fact that records are represented internally as JavaScript objects. All we need to do is not restrict entries to objects:
[#bs.val] external entries: 'a => array((string, value)) = "Object.entries";
let fields = keys({foo: 42, bar: 24}); // returns [|("foo", 42), ("bar", 24)|]
This is still safe because all values are objects in JavaScript and we don't make any assumptions about the type of the values. If we try to use this with a primitive type we'll just get an empty array, and if we try to use it with an array we'll get the indexes as keys.
But because records need to be pre-defined this isn't going to be very useful. So all this said, I still suggest going with the list of tuples.
Note: This uses ReasonML syntax since that's what you asked for, but refers to the ReScript documentation, which uses the slightly different ReScript syntax, since the BuckleScript documentation has been taken down (Yeah it's a mess right now, I know. Hopefully it'll improve eventually.)
Maybe I am not understanding the question or the use case. But as far as I know there is no way to iterate over key/value pairs of a record. You may want to use a different data model:
hash table https://caml.inria.fr/pub/docs/manual-ocaml/libref/Hashtbl.html
Js.Dict (if you're working in bucklescript/ReScript) https://rescript-lang.org/docs/manual/latest/api/js/dict
a list of tuples
With a record all keys and value types are known so you can just write code to handle each one, no iteration needed.
I have appsettings.json with code:
"Serilog": {
"WriteTo": [
{
"Name": "RollingFile",
"Args": {
"pathFormat": "/home/www-data/aissubject/storage/logs/log-{Date}.txt"
}
}
]
}
How can I read value of "pathFormat" key?
What you're referring to is a JSON array. How you access that varies depending on what you're doing, but I'm assuming that since you're asking this, you're trying to get it directly out of IConfiguration, rather than using the options pattern (as you likely should be).
IConfiguration is basically a dictionary. In order to create the keys of that dictionary from something like JSON, the JSON is "flattened" using certain conventions. Each level will be separated by a colon. Arrays will be flattened by adding a colon-delimited component containing the index. In other words, to get at pathFormat in this particular example, you'd need:
Configuration["Serilog:WriteTo:0:Args:pathFormat"]
Where the 0 portion denotes that you're getting the first item in the array. Again, it's much better and more appropriate to use the options pattern to map the configuration values onto an actual object, which would let you actually access this as an array rather than a magic string like this.
I'm building a generic API with content and a schema that can be user-defined. I want to add filtering logic to API responses, so that users can query for specific objects they've stored in the API. For example, if a user is storing event objects, they could do things like filter on:
Array contains: Whether properties.categories contains Engineering
Greater than: Whether properties.created_at is older than 2016-10-02
Not equal: Whether properties.address.city is not Washington
Equal: Whether properties.name is Meetup
etc.
I'm trying to design filtering into the query string of API responses, and coming up with a few options, but I'm not sure which syntax for it is best...
1. Operator as Nested Key
/events?properties.name=Harry&properties.address.city.neq=Washington
This example is uses just a nested object to specific the operators (like neq as shown). This is nice in that it is very simple, and easy to read.
But in cases where the properties of an event can be defined by the user, it runs into an issue where there is a potential clash between a property named address.city.neq using a normal equal operator, and a property named address.city using a not equal operator.
Example: Stripe's API
2. Operator as Key Suffix
/events?properties.name=Harry&properties.address.city+neq=Washington
This example is similar to the first one, except it uses a + delimiter (which is equivalent to a space) for operations, instead of . so that there is no confusion, since keys in my domain can't contain spaces.
One downside is that it is slightly harder to read, although that's arguable since it might be construed as more clear. Another might be that it is slightly harder to parse, but not that much.
3. Operator as Value Prefix
/events?properties.name=Harry&properties.address.city=neq:Washington
This example is very similar to the previous one, except that it moves the operator syntax into the value of the parameter instead of the key. This has the benefit of eliminating a bit of the complexity in parsing the query string.
But this comes at the cost of no longer being able to differentiate between an equal operator checking for the literal string neq:Washington and a not equal operator checking for the string Washington.
Example: Sparkpay's API
4. Custom Filter Parameter
/events?filter=properties.name==Harry;properties.address.city!=Washington
This example uses a single top-level query paramter, filter, to namespace all of the filtering logic under. This is nice in that you never have to worry about the top-level namespace colliding. (Although in my case, everything custom is nested under properties. so this isn't an issue in the first place.)
But this comes at a cost of having a harder query string to type out when you want to do basic equality filtering, which will probably result in having to check the documentation most of the time. And relying on symbols for the operators might lead to confusion for non-obvious operations like "near" or "within" or "contains".
Example: Google Analytics's API
5. Custom Verbose Filter Parameter
/events?filter=properties.name eq Harry; properties.address.city neq Washington
This example uses a similar top-level filter parameter as the previous one, but it spells out the operators with word instead of defining them with symbols, and has spaces between them. This might be slightly more readable.
But this comes at a cost of having a longer URL, and a lot of spaces that will need to be encoded?
Example: OData's API
6. Object Filter Parameter
/events?filter[1][key]=properties.name&filter[1][eq]=Harry&filter[2][key]=properties.address.city&filter[2][neq]=Washington
This example also uses a top-level filter parameter, but instead of creating a completely custom syntax for it that mimics programming, it instead builds up an object definition of filters using a more standard query string syntax. This has the benefit of bring slightly more "standard".
But it comes at the cost of being very verbose to type and hard to parse.
Example Magento's API
Given all of those examples, or a different approach, which syntax is best? Ideally it would be easy to construct the query parameter, so that playing around in the URL bar is doable, but also not pose problems for future interoperability.
I'm leaning towards #2 since it seems like it is legible, but also doesn't have some of the downsides of other schemes.
I might not answer the "which one is best" question, but I can at least give you some insights and other examples to consider.
First, you are talking about "generic API with content and a schema that can be user-defined".
That sound a lot like solr / elasticsearch which are both hi level wrappers over Apache Lucene which basically indexes and aggregates documents.
Those two took totally different approaches to their rest API, I happened to work with both of them.
Elasticsearch :
They made entire JSON based Query DSL, which currently looks like this :
GET /_search
{
"query": {
"bool": {
"must": [
{ "match": { "title": "Search" }},
{ "match": { "content": "Elasticsearch" }}
],
"filter": [
{ "term": { "status": "published" }},
{ "range": { "publish_date": { "gte": "2015-01-01" }}}
]
}
}
}
Taken from their current doc. I was surprised that you can actually put data in GET...
It actually looks better now, in earlier versions it was much more hierarchical.
From my personal experience, this DSL was powerful, but rather hard to learn and use fluently (especially older versions). And to actually get some result you need more than just play with URL. Starting with the fact that many clients don't even support data in GET request.
SOLR :
They put everything into query params, which basically looks like this (taken from the doc) :
q=*:*&fq={!cache=false cost=5}inStock:true&fq={!frange l=1 u=4 cache=false cost=50}sqrt(popularity)
Working with that was more straightforward. But that's just my personal taste.
Now about my experiences. We were implementing another layer above those two and we took approach number #4. Actually, I think #4 and #5 should be supported at the same time. Why? Because whatever you pick people will be complaining, and since you will be having your own "micro-DSL" anyway, you might as well support few more aliases for your keywords.
Why not #2? Having single filter param and query inside gives you total control over DSL. Half a year after we made our resource, we got "simple" feature request - logical OR and parenthesis (). Query parameters are basically a list of AND operations and logical OR like city=London OR age>25 don't really fit there. On the other hand parenthesis introduced nesting into DSL structure, which would also be a problem in flat query string structure.
Well, those were the problems we stumbled upon, your case might be different. But it is still worth to consider, what future expectations from this API will be.
Matomo Analytics has an other approach to deal with segment filter and its syntaxe seems to be more readable and intuitive, e.g:
developer.matomo.org/api-reference/reporting-api-segmentation
Operator
Behavior
Example
==
Equals
&segment=countryCode==IN Return results where the country is India
!=
Not equals
&segment=actions!=1 Return results where the number of actions (page views, downloads, etc.) is not 1
<=
Less than or equal to
&segment=actions<=4 Return results where the number of actions (page views, downloads, etc.) is 4 or less
<
Less than
&segment=visitServerHour<12 Return results where the Server time (hour) is before midday.
=#
Contains
&segment=referrerName=#piwik Return results where the Referer name (website domain or search engine name) contains the word "piwik".
!#
Does not contain
&segment=referrerKeyword!#yourBrand Return results where the keyword used to access the website does not contain word "yourBrand".
=^
Starts with
&segment=referrerKeyword=^yourBrand Return results where the keyword used to access the website starts with "yourBrand" (requires at least Matomo 2.15.1).
=$
Ends with
&segment=referrerKeyword=$yourBrand Return results where the keyword used to access the website ends with "yourBrand" (requires at least Matomo 2.15.1).
and you can have a close look at how they parse the segment filter here: https://github.com/matomo-org/matomo/blob/4.x-dev/core/Segment/SegmentExpression.php
#4
I like how Google Analytics filter API looks like, easy to use and easy to understand from a client's point of view.
They use a URL encoded form, for example:
Equals: %3D%3D filters=ga:timeOnPage%3D%3D10
Not equals: !%3D filters=ga:timeOnPage!%3D10
Although you need to check documentation but it still has its own advantages. IF you think that the users can get accustomed to this then go for it.
#2
Using operators as key suffixes also seems like a good idea (according to your requirements).
However I would recommend to encode the + sign so that it isn't parsed as a space. Also it might be slightly harder to parse as mentioned but I think you can write a custom parser for this one. I stumbled across this gist by jlong some time back. Perhaps you'll find it useful to write your parser.
You could also try Spring Expression Language (SpEL)
All you need to do is to stick to the said format in the document, the SpEL engine would take care of parsing the query and executing it on a given object. Similar to your requirement of filtering a list of objects, you could write the query as:
properties.address.city == 'Washington' and properties.name == 'Harry'
It supports all kind of relational and logical operators that you would need. The rest api could just take this query as the filter string and pass it to SpEL engine to run on an object.
Benefits: it's readable, easy to write, and execution is well taken care of.
So, the URL would look like:
/events?filter="properties.address.city == 'Washington' and properties.name == 'Harry'"
Sample code using org.springframework:spring-core:4.3.4.RELEASE :
The main function of interest:
/**
* Filter the list of objects based on the given query
*
* #param query
* #param objects
* #return
*/
private static <T> List<T> filter(String query, List<T> objects) {
ExpressionParser parser = new SpelExpressionParser();
Expression exp = parser.parseExpression(query);
return objects.stream().filter(obj -> {
return exp.getValue(obj, Boolean.class);
}).collect(Collectors.toList());
}
Complete example with helper classes and other non-interesting code:
import java.util.Arrays;
import java.util.List;
import java.util.stream.Collectors;
import org.springframework.expression.Expression;
import org.springframework.expression.ExpressionParser;
import org.springframework.expression.spel.standard.SpelExpressionParser;
public class SpELTest {
public static void main(String[] args) {
String query = "address.city == 'Washington' and name == 'Harry'";
Event event1 = new Event(new Address("Washington"), "Harry");
Event event2 = new Event(new Address("XYZ"), "Harry");
List<Event> events = Arrays.asList(event1, event2);
List<Event> filteredEvents = filter(query, events);
System.out.println(filteredEvents.size()); // 1
}
/**
* Filter the list of objects based on the query
*
* #param query
* #param objects
* #return
*/
private static <T> List<T> filter(String query, List<T> objects) {
ExpressionParser parser = new SpelExpressionParser();
Expression exp = parser.parseExpression(query);
return objects.stream().filter(obj -> {
return exp.getValue(obj, Boolean.class);
}).collect(Collectors.toList());
}
public static class Event {
private Address address;
private String name;
public Event(Address address, String name) {
this.address = address;
this.name = name;
}
public Address getAddress() {
return address;
}
public void setAddress(Address address) {
this.address = address;
}
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
}
public static class Address {
private String city;
public Address(String city) {
this.city = city;
}
public String getCity() {
return city;
}
public void setCity(String city) {
this.city = city;
}
}
}
I decided to compare the approaches #1/#2 (1) and #3 (2) and concluded that (1) is preferred (at least, for Java server side).
Assume, some parameter a must be equal 10 or 20. Our URL query in this case must look like ?a.eq=10&a.eq=20 for (1) and ?a=eq:10&a=eq:20 for (2). In Java HttpServletRequest#getParameterMap() will return the next values: { a.eq: [10, 20] } for (1) and { a: [eq:10, eq:20] } for (2). Later we must convert returned maps, for example, to SQL where clause. And we should get: where a = 10 or a = 20 for both (1) and (2). Briefly, it looks something like that:
1) ?a=eq:10&a=eq:20 -> { a: [eq:10, eq:20] } -> where a = 10 or a = 20
2) ?a.eq=10&a.eq=20 -> { a.eq: [10, 20] } -> where a = 10 or a = 20
So, we got the next rule: when we pass through URL query two parameters with the same name we must use OR operand in SQL.
But let's assume another case. The parameter a must be greater than 10 and less than
20. Applying the rule above we will have the next conversion:
1) ?a.gt=10&a.ls=20 -> { a.gt: 10, a.lt: 20 } -> where a > 10 and a < 20
2) ?a=gt:10&a=ls:20 -> { a: [gt.10, lt.20] } -> where a > 10 or(?!) a < 20
As you can see, in (1) we have two parameters with different names: a.gt and a.ls. This means our SQL query will have AND operand. But for (2) we still have the same names and it must be converted to the SQL with OR operand!
This means that for (2) instead of using #getParameterMap() we must directly parse the URL query and analyze repeated parameter names.
I know this is old school, but how about a sort of operator overloading?
It would make the query parsing a lot harder (and not standard CGI), but would resemble the contents of an SQL WHERE clause.
/events?properties.name=Harry&properties.address.city+neq=Washington
would become
/events?properties.name=='Harry'&&properties.address.city!='Washington'||properties.name=='Jack'&&properties.address.city!=('Paris','New Orleans')
paranthesis would start a list. Keeping strings in quotes would simplify parsing.
So the above query would be for events for Harry's not in Washington or for Jacks not in Paris or in New Orleans.
It would be a ton of work to implement... and the database optimization to run those queries would be a nightmare, but if you're looking for a simple and powerful query language, just imitate SQL :)
-k