How to design generic filtering operators in the query string of an API? - sql

Question

I'm building a generic API with content and a schema that can be user-defined. I want to add filtering logic to API responses, so that users can query for specific objects they've stored in the API. For example, if a user is storing event objects, they could do things like filter on:
Array contains: Whether properties.categories contains Engineering
Greater than: Whether properties.created_at is older than 2016-10-02
Not equal: Whether properties.address.city is not Washington
Equal: Whether properties.name is Meetup
etc.
I'm trying to design filtering into the query string of API responses, and coming up with a few options, but I'm not sure which syntax for it is best...
1. Operator as Nested Key
/events?properties.name=Harry&properties.address.city.neq=Washington
This example is uses just a nested object to specific the operators (like neq as shown). This is nice in that it is very simple, and easy to read.
But in cases where the properties of an event can be defined by the user, it runs into an issue where there is a potential clash between a property named address.city.neq using a normal equal operator, and a property named address.city using a not equal operator.
Example: Stripe's API
2. Operator as Key Suffix
/events?properties.name=Harry&properties.address.city+neq=Washington
This example is similar to the first one, except it uses a + delimiter (which is equivalent to a space) for operations, instead of . so that there is no confusion, since keys in my domain can't contain spaces.
One downside is that it is slightly harder to read, although that's arguable since it might be construed as more clear. Another might be that it is slightly harder to parse, but not that much.
3. Operator as Value Prefix
/events?properties.name=Harry&properties.address.city=neq:Washington
This example is very similar to the previous one, except that it moves the operator syntax into the value of the parameter instead of the key. This has the benefit of eliminating a bit of the complexity in parsing the query string.
But this comes at the cost of no longer being able to differentiate between an equal operator checking for the literal string neq:Washington and a not equal operator checking for the string Washington.
Example: Sparkpay's API
4. Custom Filter Parameter
/events?filter=properties.name==Harry;properties.address.city!=Washington
This example uses a single top-level query paramter, filter, to namespace all of the filtering logic under. This is nice in that you never have to worry about the top-level namespace colliding. (Although in my case, everything custom is nested under properties. so this isn't an issue in the first place.)
But this comes at a cost of having a harder query string to type out when you want to do basic equality filtering, which will probably result in having to check the documentation most of the time. And relying on symbols for the operators might lead to confusion for non-obvious operations like "near" or "within" or "contains".
Example: Google Analytics's API
5. Custom Verbose Filter Parameter
/events?filter=properties.name eq Harry; properties.address.city neq Washington
This example uses a similar top-level filter parameter as the previous one, but it spells out the operators with word instead of defining them with symbols, and has spaces between them. This might be slightly more readable.
But this comes at a cost of having a longer URL, and a lot of spaces that will need to be encoded?
Example: OData's API
6. Object Filter Parameter
/events?filter[1][key]=properties.name&filter[1][eq]=Harry&filter[2][key]=properties.address.city&filter[2][neq]=Washington
This example also uses a top-level filter parameter, but instead of creating a completely custom syntax for it that mimics programming, it instead builds up an object definition of filters using a more standard query string syntax. This has the benefit of bring slightly more "standard".
But it comes at the cost of being very verbose to type and hard to parse.
Example Magento's API
Given all of those examples, or a different approach, which syntax is best? Ideally it would be easy to construct the query parameter, so that playing around in the URL bar is doable, but also not pose problems for future interoperability.
I'm leaning towards #2 since it seems like it is legible, but also doesn't have some of the downsides of other schemes.

Answer 1

I might not answer the "which one is best" question, but I can at least give you some insights and other examples to consider.
First, you are talking about "generic API with content and a schema that can be user-defined".
That sound a lot like solr / elasticsearch which are both hi level wrappers over Apache Lucene which basically indexes and aggregates documents.
Those two took totally different approaches to their rest API, I happened to work with both of them.
Elasticsearch :
They made entire JSON based Query DSL, which currently looks like this :
GET /_search
{
"query": {
"bool": {
"must": [
{ "match": { "title": "Search" }},
{ "match": { "content": "Elasticsearch" }}
],
"filter": [
{ "term": { "status": "published" }},
{ "range": { "publish_date": { "gte": "2015-01-01" }}}
]
}
}
}
Taken from their current doc. I was surprised that you can actually put data in GET...
It actually looks better now, in earlier versions it was much more hierarchical.
From my personal experience, this DSL was powerful, but rather hard to learn and use fluently (especially older versions). And to actually get some result you need more than just play with URL. Starting with the fact that many clients don't even support data in GET request.
SOLR :
They put everything into query params, which basically looks like this (taken from the doc) :
q=*:*&fq={!cache=false cost=5}inStock:true&fq={!frange l=1 u=4 cache=false cost=50}sqrt(popularity)
Working with that was more straightforward. But that's just my personal taste.
Now about my experiences. We were implementing another layer above those two and we took approach number #4. Actually, I think #4 and #5 should be supported at the same time. Why? Because whatever you pick people will be complaining, and since you will be having your own "micro-DSL" anyway, you might as well support few more aliases for your keywords.
Why not #2? Having single filter param and query inside gives you total control over DSL. Half a year after we made our resource, we got "simple" feature request - logical OR and parenthesis (). Query parameters are basically a list of AND operations and logical OR like city=London OR age>25 don't really fit there. On the other hand parenthesis introduced nesting into DSL structure, which would also be a problem in flat query string structure.
Well, those were the problems we stumbled upon, your case might be different. But it is still worth to consider, what future expectations from this API will be.

Answer 2

Matomo Analytics has an other approach to deal with segment filter and its syntaxe seems to be more readable and intuitive, e.g:
developer.matomo.org/api-reference/reporting-api-segmentation
Operator
Behavior
Example
==
Equals
&segment=countryCode==IN Return results where the country is India
!=
Not equals
&segment=actions!=1 Return results where the number of actions (page views, downloads, etc.) is not 1
<=
Less than or equal to
&segment=actions<=4 Return results where the number of actions (page views, downloads, etc.) is 4 or less
<
Less than
&segment=visitServerHour<12 Return results where the Server time (hour) is before midday.
=#
Contains
&segment=referrerName=#piwik Return results where the Referer name (website domain or search engine name) contains the word "piwik".
!#
Does not contain
&segment=referrerKeyword!#yourBrand Return results where the keyword used to access the website does not contain word "yourBrand".
=^
Starts with
&segment=referrerKeyword=^yourBrand Return results where the keyword used to access the website starts with "yourBrand" (requires at least Matomo 2.15.1).
=$
Ends with
&segment=referrerKeyword=$yourBrand Return results where the keyword used to access the website ends with "yourBrand" (requires at least Matomo 2.15.1).
and you can have a close look at how they parse the segment filter here: https://github.com/matomo-org/matomo/blob/4.x-dev/core/Segment/SegmentExpression.php

Answer 3

#4
I like how Google Analytics filter API looks like, easy to use and easy to understand from a client's point of view.
They use a URL encoded form, for example:
Equals: %3D%3D filters=ga:timeOnPage%3D%3D10
Not equals: !%3D filters=ga:timeOnPage!%3D10
Although you need to check documentation but it still has its own advantages. IF you think that the users can get accustomed to this then go for it.
#2
Using operators as key suffixes also seems like a good idea (according to your requirements).
However I would recommend to encode the + sign so that it isn't parsed as a space. Also it might be slightly harder to parse as mentioned but I think you can write a custom parser for this one. I stumbled across this gist by jlong some time back. Perhaps you'll find it useful to write your parser.

Answer 4

You could also try Spring Expression Language (SpEL)
All you need to do is to stick to the said format in the document, the SpEL engine would take care of parsing the query and executing it on a given object. Similar to your requirement of filtering a list of objects, you could write the query as:
properties.address.city == 'Washington' and properties.name == 'Harry'
It supports all kind of relational and logical operators that you would need. The rest api could just take this query as the filter string and pass it to SpEL engine to run on an object.
Benefits: it's readable, easy to write, and execution is well taken care of.
So, the URL would look like:
/events?filter="properties.address.city == 'Washington' and properties.name == 'Harry'"
Sample code using org.springframework:spring-core:4.3.4.RELEASE :
The main function of interest:
/**
* Filter the list of objects based on the given query
*
* #param query
* #param objects
* #return
*/
private static <T> List<T> filter(String query, List<T> objects) {
ExpressionParser parser = new SpelExpressionParser();
Expression exp = parser.parseExpression(query);
return objects.stream().filter(obj -> {
return exp.getValue(obj, Boolean.class);
}).collect(Collectors.toList());
}
Complete example with helper classes and other non-interesting code:
import java.util.Arrays;
import java.util.List;
import java.util.stream.Collectors;
import org.springframework.expression.Expression;
import org.springframework.expression.ExpressionParser;
import org.springframework.expression.spel.standard.SpelExpressionParser;
public class SpELTest {
public static void main(String[] args) {
String query = "address.city == 'Washington' and name == 'Harry'";
Event event1 = new Event(new Address("Washington"), "Harry");
Event event2 = new Event(new Address("XYZ"), "Harry");
List<Event> events = Arrays.asList(event1, event2);
List<Event> filteredEvents = filter(query, events);
System.out.println(filteredEvents.size()); // 1
}
/**
* Filter the list of objects based on the query
*
* #param query
* #param objects
* #return
*/
private static <T> List<T> filter(String query, List<T> objects) {
ExpressionParser parser = new SpelExpressionParser();
Expression exp = parser.parseExpression(query);
return objects.stream().filter(obj -> {
return exp.getValue(obj, Boolean.class);
}).collect(Collectors.toList());
}
public static class Event {
private Address address;
private String name;
public Event(Address address, String name) {
this.address = address;
this.name = name;
}
public Address getAddress() {
return address;
}
public void setAddress(Address address) {
this.address = address;
}
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
}
public static class Address {
private String city;
public Address(String city) {
this.city = city;
}
public String getCity() {
return city;
}
public void setCity(String city) {
this.city = city;
}
}
}

Answer 5

I decided to compare the approaches #1/#2 (1) and #3 (2) and concluded that (1) is preferred (at least, for Java server side).
Assume, some parameter a must be equal 10 or 20. Our URL query in this case must look like ?a.eq=10&a.eq=20 for (1) and ?a=eq:10&a=eq:20 for (2). In Java HttpServletRequest#getParameterMap() will return the next values: { a.eq: [10, 20] } for (1) and { a: [eq:10, eq:20] } for (2). Later we must convert returned maps, for example, to SQL where clause. And we should get: where a = 10 or a = 20 for both (1) and (2). Briefly, it looks something like that:
1) ?a=eq:10&a=eq:20 -> { a: [eq:10, eq:20] } -> where a = 10 or a = 20
2) ?a.eq=10&a.eq=20 -> { a.eq: [10, 20] } -> where a = 10 or a = 20
So, we got the next rule: when we pass through URL query two parameters with the same name we must use OR operand in SQL.
But let's assume another case. The parameter a must be greater than 10 and less than
20. Applying the rule above we will have the next conversion:
1) ?a.gt=10&a.ls=20 -> { a.gt: 10, a.lt: 20 } -> where a > 10 and a < 20
2) ?a=gt:10&a=ls:20 -> { a: [gt.10, lt.20] } -> where a > 10 or(?!) a < 20
As you can see, in (1) we have two parameters with different names: a.gt and a.ls. This means our SQL query will have AND operand. But for (2) we still have the same names and it must be converted to the SQL with OR operand!
This means that for (2) instead of using #getParameterMap() we must directly parse the URL query and analyze repeated parameter names.

Answer 6

I know this is old school, but how about a sort of operator overloading?
It would make the query parsing a lot harder (and not standard CGI), but would resemble the contents of an SQL WHERE clause.
/events?properties.name=Harry&properties.address.city+neq=Washington
would become
/events?properties.name=='Harry'&&properties.address.city!='Washington'||properties.name=='Jack'&&properties.address.city!=('Paris','New Orleans')
paranthesis would start a list. Keeping strings in quotes would simplify parsing.
So the above query would be for events for Harry's not in Washington or for Jacks not in Paris or in New Orleans.
It would be a ton of work to implement... and the database optimization to run those queries would be a nightmare, but if you're looking for a simple and powerful query language, just imitate SQL :)
-k