I am invoking an API operation to fetch a list of objects and then, for each object in that list, I am invoking another API operation to fetch additional details of that object and adding those details into the object. The goal is to return a list of objects with all the properties (or details) that I need. In the example below, the /allObjects call will get a list of all objects with 2 properties - id and k1. But my application needs 3 properties, id, k1 and k4. So I invoke another API method to get detailed information for each object, which includes the k4 property. I copy that detailed property into the original result-set's object and return the "enriched" result-set. But the latency cost of this can be high. What is the best way of achieving this without slowing down the application too much? In the real world, I am dealing with 500-2000 objects.
GET /allObjects yields JSON results =>
{
"data" : [
{
"id" : "123",
"k1" : "v1"
}, {
"id" : "456",
"k1" : "v1"
}
]
}
for (obj in results.data) {
GET /object/{obj.id} yields JSON result =>
{
"data" : {
"id" : "123",
"k1" : "v1",
"k2" : "v2",
"k3" : "v3",
"k4" : "v4"
}
}
// Add k4 property to original result-set object, and assign
// it the value of the k4 property in the individual result object.
obj.k4 = result.data.k4;
}
return results.data;
Your requirement is such that, you have no other option but to go for a mash-up(unless you can convince the API developers to combine the two API).
You could, however, opt to make a mashup service with a low latency cost to stand in between your application to abstract out the mashup logic, conversely, you could opt to use a language that is custom made for this kind of work to program your application. Both of these options can be accommodated with Ballerina, I've written a post here showing how easy it is to do that using it.
Related
For my project I need a new property for json schemas.
I named it "isPropertyOf" and basically what I want is that if I have this:
fruits = {
"banana": "yellow fruit",
"apple": "round fruit"
}
schema = {
"type": "object",
"properties": {
"fav_fruit": {
"type": "string",
"isPropertyOf": fruits
}
}
}
Then schema would validate only objects like {"fav_fruit":"banana"} or {"fav_fruit":"apple"}, but not {"fav_fruit":"salami"}
(I know that for this very example using an enum would make more sense, but assume "fruits" is also used for other stuff and I would rather avoid redundancy)
I read docs about this and figured I need to use jsonschema.validators.extend. I tried something like that:
def is_property_of_callable(validator_instance, property_value, instance, schema):
keys = [k for k in property_value]
return instance in keys
mapping = {"isPropertyOf": is_property_of_callable}
my_validator = jsonschema.validators.extend(jsonschema.Draft7Validator, mapping)
instance = {"fav_fruit": "tobacco"}
my_validator(schema).is_valid(instance)
I was ready to see something go wrong, but apparently the validator wasn't seeing any issue. I tried with an obviously wrong instance that even the standard validator of jsonschema wouldn't accept
my_validator(schema).is_valid({"fav_fruit": 0})
But apparently it looked ok to it.
I thought maybe the way I added this property was so broken that it was making the validator accept anything, so I tried a minimal case of validator extension:
minimal_validator = jsonschema.validators.extend(jsonschema.Draft7Validator, {})
minimal_validator(schema).is_valid(instance)
And this one is also happy with 0 being my favourite fruit.
What am I doing wrong here? How can I make that work?
I am using spring data mongodb sdk to query mongo db.
The document in mongoDb looks like this:
{
"data": {
"suggestions":[
{
"key": "take",
"value": 1
},
{
"key": "donttake",
"value": 0
}
]
}
}
In my api request I have a structure similar to "suggestions" element above.
I want to create a query criteria where "is" clause should be the value of "suggestions" element in the api request.
I tried the following code using spring data mongo db:
JsonParser jsonParser = new JsonParser();
ObjectMapper objMapper = new ObjectMapper();
String jsonArrayString = objMapper.writeValueAsString(apirequest.getSuggestions());
JsonArray arrayFromString = jsonParser.parse(jsonArrayString).getAsJsonArray();
criteria = Criteria.where("data.suggestions").is(arrayFromString);
The problem with this code is that when I debug and see the query that gets created using criteria above, I goes in as $java: [{"key": "take", "value": 1}]
Therefore, it can't match it with the mongo document and doesn't fetch me any result.
Is there another way to query and array of documents in mongodb from spring data mongo ?
I followed a completely different approach by reading some information on querying arrays in mongodb available at
https://docs.mongodb.com/manual/tutorial/query-array-of-documents/
I used elemMatch to solve this problem as follows:
Let's say my API request gets mapped to and object suggestions and keyVal is an object that stores the keyVal pair.
for (KeyVal keyVal: suggestions()){
Criteria c =
Criteria.where("key").is(keyVal.key()).and("value").is(keyVal.value());
criteria = Criteria.where("data.suggestions").elemMatch(c);
}
Then criteria can be used in a mongo Query
Also, keep in mind that elemMatch doesn't care about the ordering of elements inside a document in an array. So that way, elemMatch solves the purpose well.
How can i get my json from web api to format only value or null for Option types and Discriminated Unions preferably using Newtonsoft.
I am currently using Newtonsoft and only have to add this to web api for it to work:
config.Formatters.JsonFormatter.SerializerSettings <- new JsonSerializerSettings()
When i consume the data on my side, i can easily convert it back to an F# item using: JsonConvert.DeserializeObject<'a>(json)
The api will be consumed by NON .NET clients as well so i would like a more standard formatted json result.
I would like to to fix my issue, w/o having to add code or decorators to all of my records/DU in order for it to work. I have lots of records with lots of properties, some are Option.
ex (this is how DU is serializing):
// When value
"animal": {
"case": "Dog"
}
// When no value
"animal": null
This is what I need:
// When value
"animal": "Dog"
// When no value
"animal": null
This is how an Option type is serializing:
"DocumentInfo": {
"case": "Some",
"fields": [
{
"docId": "77fb9dd0-bfbe-42e0-9d29-d5b1f5f0a9f7",
"docType": "Monkey Business",
"docName": "mb.doc",
"docContent": "why cant it just give me the values?"
}
]
}
This is what I need:
"DocumentInfo": {
"docId": "77fb9dd0-bfbe-42e0-9d29-d5b1f5f0a9f7",
"docType": "Monkey Business",
"docName": "mb.doc",
"docContent": "why cant it just give me the values?"
}
Thank you :-)
You could try using Chiron. I haven't used it myself so I can't give you an extensive example, but https://neoeinstein.github.io/blog/2015/12-13-chiron-json-ducks-monads/index.html has some bits of sample code. (And see https://neoeinstein.github.io/blog/2016/04-02-chiron-computation-expressions/index.html as well for some nicer syntax). Basically, Chiron knows how to serialize and deserialize the basic F# types (strings, numbers, options, etc.) already, and you can teach it to serialize any other type by providing that type with two static methods, ToJson and FromJson:
static member ToJson (x:DocumentInfo) = json {
do! Json.write "docId" x.docId
do! Json.write "docType" x.docType
do! Json.write "docName" x.docName
do! Json.write "docContent" x.docContent
}
static member FromJson (_:DocumentInfo) = json {
let! i = Json.read "docId"
let! t = Json.read "docType"
let! n = Json.read "docName"
let! c = Json.read "docContent"
return { docId = i; docType = t; docName = n; docContent = c }
}
By providing those two static methods on your DocumentInfo type, Chiron will automatically know how to serialize a DocumentInfo option. At least, that's my understanding -- but the Chiron documentation is sadly lacking (by which I mean literally lacking: it hasn't been written yet), so I haven't really used it myself. So this may or may not be the answer you need, but hopefully it'll be of some help to you even if you don't end up using it.
I have found the solution that allows me to use Newtonsoft (JSON.NET), apply custom converters for my types where needed and not require any changes to my DU's or Records.
The short answer is, create a custom converter for Json.Net and use the Read/Write Json overrides:
type CustomDuConverter() =
inherit JsonConverter() (...)
Unfortunately the ones I have found online that were already created doesn't work as is for my needs listed above, but will with slight modification. A great example is to look at: https://gist.github.com/isaacabraham/ba679f285bfd15d2f53e
To apply your custom serializer in Web Api for every call, use:
config.Formatters.JsonFormatter.SerializerSettings.Converters.Add(new CustomDuConverter())
To deserialize use (example that will deserialize to DU):
JsonConvert.DeserializeObject<Animal>("Dog", customConverter)
ex:
type Animal = Dog | Cat
json:
"animal": "Dog"
This will allow you to create a clean Api for consumers and allow you to consume 3rd party Json data into your types that use Option, etc.
Using Spring Data REST with JPA in version 2.0.2.RELEASE.
How can I disable Hypertext Application Language (HAL) in the JSON ? http://stateless.co/hal_specification.html
I have tried many things already, but to no avail. For example, I have set Accept and Content-type headers to "application/json" instead of "application/hal+json" but I still receive the JSON content with hyper links.
For example, I'd like to get something like:
{
"name" : "Foo",
"street" : "street Bar",
"streetNumber" : 2,
"streetLetter" : "b",
"postCode" : "D-1253",
"town" : "Munchen",
"country" : "Germany",
"phone" : "+34 4410122000",
"vat" : "000000001",
"employees" : 225,
"sector" : {
"description" : "Marketing",
"average profit": 545656665,
"average employees": 75,
"average profit per employee": 4556
}
}
Instead of:
{
"name" : "Foo",
"street" : "street Bar",
"streetNumber" : 2,
"streetLetter" : "b",
"postCode" : "D-1253",
"town" : "Munchen",
"country" : "Germany",
"phone" : "+34 4410122000",
"vat" : "000000001",
"employees" : 225,
"_links" : {
"self" : {
"href" : "http://localhost:8080/app/companies/1"
},
"sector" : {
"href" : "http://localhost:8080/app/companies/1/sector"
}
}
}
Thanks for your help.
(Hyper)media types
The default settings for Spring Data REST use HAL as the default hypermedia representation format, so the server will return the following for the given Accept headers:
No header -> application/hal+json -> HAL
application/hal+json -> application/hal+json -> HAL
application/json -> application/json -> HAL (this is what the default configures)
application/x-spring-data-verbose+json -> application/x-spring-data-verbose+json -> a Spring Data specific format (using links for the links container and content as wrapper for the collection items.
If you configure RepositoryRestConfiguration.setDefaultMediaType(…) to a non-HAL format, the server will return the Spring Data specific JSON format unless you explicitly ask for application/hal+json. Admittedly the configuration option is probably a bit misleading, so I filed DATAREST-294 to improve this. The issue was resolved in 2.1 RC1 (Dijkstra) 2014.
Note that we effectively need a hypermedia format in place to be able to express relations between managed resources and enable discoverability of the server. So there's no way you'll be able to get rid of it completely. This is mostly due to the fact that you could easily crash the server if you expose entities that have bidirectional relationships or make up an enormous object graph.
Inlining related entities
If you never want to have sectors linked to and always inline them, one option is to simply exclude the SectorRepository from being exported as a REST resource in the first place. You can achieve this by annotating the repository interface with #RepositoryRestResource(exported = false).
To get a representation returned as you posted in your lower example have a look at the projections feature introduced in Spring Data REST 2.1 M1. It basically allow you to craft optional views on a resource that can differ from the default one via a simple interface.
You'd basically define an interface:
#Projection(name = "foo", types = YourDomainClass.class)
interface Inlined {
// list all other properties
Sector getSector();
}
If you either put this interface into a (sub)package of your domain class or manually register it via RepositoryRestConfiguration.projectionConfiguration() the resources exposing YourDomainClass will accept a request parameter projection so that passing in foo in this example would render the inlined representation as you want it.
This commit has more info on the feature in general, this commit has an example projection defined.
So you want 2 things:
1) get rid of _links field
2) include the related sector field
Possible solution (works for me :D)
1) get rid of _links
For this create the class below:
[... package declaration, imports ...]
public class MyRepositoryRestMvcConfiguration extends RepositoryRestMvcConfiguration {
public MyRepositoryRestMvcConfiguration(ApplicationContext context, ObjectFactory<ConversionService> conversionService) {
super(context, conversionService);
}
#Bean
protected LinkCollector linkCollector() {
return new LinkCollector(persistentEntities(), selfLinkProvider(), associationLinks()) {
public Links getLinksFor(Object object, List<Link> existingLinks) {
return new Links();
}
};
}
}
and use it e.g.:
[... package declaration, imports ...]
#SpringBootApplication
#Import({MyRepositoryRestMvcConfiguration.class})
public class MyApplication {
public static void main(String[] args) {
SpringApplication.run(MyApplication.class, args);
}
}
I'm pretty sure (99%, but not tested) that you won't need this class for removing the _links for the related entity/entities included the way next point (2) is showing.
2) include the related sector field
For this you could use Excerpts (especially made for this scenario). Because the Spring example is so eloquent and it's silly to just copy it here I'll just point it: https://docs.spring.io/spring-data/rest/docs/3.1.x/reference/html/#projections-excerpts.excerpting-commonly-accessed-data.
But just for the record and your convenience I'll paste the main parts of the spring example:
#Projection(name = "inlineAddress", types = { Person.class })
interface InlineAddress {
String getFirstName();
String getLastName();
Address getAddress();
}
see at Projection javadoc that types means The type the projection type is bound to.
The excerpt could be used this way:
#RepositoryRestResource(excerptProjection = InlineAddress.class)
interface PersonRepository extends CrudRepository<Person, Long> {}
in order to get this (when also using MyRepositoryRestMvcConfiguration):
{
"firstName" : "Frodo",
"lastName" : "Baggins",
"address" : {
"street": "Bag End",
"state": "The Shire",
"country": "Middle Earth"
}
}
For you the sector is the equivalent of address.
Final notes
When returning arrays the _links field won't be removed (it's too intrusive to do it); in the end you'll have something like this:
{
"_embedded" : {
"persons" : [ {person1}, {person2}, ..., {personN} ]
},
"_links" : {
e.g. first, next, last, self, profile
},
"page" : {
"size" : 1,
"totalElements" : 10,
"totalPages" : 10,
"number" : 0
}
}
As you can see even if we'd have _links removed that still won't be enough; one would probably also want _embedded replaced by persons which would lead to less maintainable code (too much spring intrusive overrides). But if one really wants these too he should start checking RepositoryRestMvcConfiguration and RepositoryEntityController.getCollectionResource.
Spring is evolving so I feel the need to point that this works with at least:
spring-data-rest-webmvc 3.1.3.RELEASE
or, if you prefeer spring boot version:
spring-boot-starter-parent 2.1.1.RELEASE
If you want to delete _links do the following (worked for me):
Go to your pom.xml and delete the following dependency:
spring-boot-starter-data-rest
Make an "Update project" to update the pom.xml changes.
Now it will be use your own controller for the api rest, deleting _self, _links..., that alike this:
[... package declaration, imports ...]
#RestController
#RequestMapping("/series")
public class SerieController {
#Autowired
private SerieRepositorio serieRepositorio;
public SerieController(SerieRepositorio serieRepositorio) {
this.serieRepositorio = serieRepositorio;
}
#GetMapping
public Iterable<Serie> getAllSeries() {
return serieRepositorio.findAll();
}
}
I am trying to save (RavenDB build 960) the names and values of form data items passed into a Nancy Module via its built in Request.Form.
If I save a straightforward instance of a dynamic object (with test properties and values) then everything works and both the property names and values are saved. However, if I use Nancy's Request.Form then only the dynamic property names are saved.
I understand that I will have to deal with further issues to do with restoring the correct types when retrieving the dynamic data (RavenJObjects etc) but for now, I want to solve the problem of saving the dynamic names / values in the first place.
Here is the entire test request and code:
Fiddler Request (PUT)
Nancy Module
Put["/report/{name}/add"] = parameters =>
{
reportService.AddTestDynamic(Db, parameters.name, Request.Form);
return HttpStatusCode.Created;
};
Service
public void AddTestDynamic(IDocumentSession db, string name, dynamic data)
{
var testDynamic = new TestDynamic
{
Name = name,
Data = data
};
db.Store(testDynamic);
db.SaveChanges();
}
TestDynamic Class
public class TestDynamic
{
public string Name;
public dynamic Data;
}
Dynamic contents of Request.Form at runtime
Resulting RavenDB Document
{
"Name": "test",
"Data": [
"username",
"age"
]
}
Note: The type of the Request.Form is Nancy.DynamicDictionary. I think this may be the problem since it inherits from IEnumerable<string> and not the expected IEnumerable<string, object>. I think that RavenDB is enumerating the DynamicDictionary and only getting back the dynamic member-names rather than the member name / value pairs.
Can anybody tell me how or whether I can treat the Request.Form as a dynamic object with respect to saving it to RavenDB? If possible I want to avoid any hand-crafted enumeration of DynamicDictionary to build a dynamic instance so that RavenDB can serialise correctly.
Thank You
Edit 1 #Ayende
The DynamicDictionary appears to implement the GetDynamicMemberNames() method:
Taking a look at the code on GitHub reveals the following implementation:
public override IEnumerable<string> GetDynamicMemberNames()
{
return dictionary.Keys;
}
Is this what you would expect to see here?
Edit 2 #TheCodeJunkie
Thanks for the code update. To test this I have:
Created a local clone of the NancyFx/Nancy master branch from
GitHub
Added the Nancy.csproj to my solution and referenced the project
Run the same test as above
RavenDB Document from new DynamicDictionary
{
"Name": "test",
"Data": {
"$type": "Nancy.DynamicDictionary, Nancy",
"username": {},
"age": {}
}
}
You can see that the resulting document is an improvement. The DynamicDictionary type information is now being correctly picked up by RavenDB and whilst the dynamic property-names are correctly serialized, unfortunately the dynamic property-values are not.
The image below shows the new look DynamicDictionary in action. It all looks fine to me, the new Dictionary interface is clearly visible. The only thing I noticed was that the dynamic 'Results view' (as opposed to the 'Dynamic view') in the debugger, shows just the property-names and not their values. The 'Dynamic view' shows both as before (see image above).
Contents of DynamicDictionary at run time
biofractal,
The problem is the DynamicDictionary, in JSON, types can be either objects or lists ,they can't be both.
And for dynamic object serialization, we rely on the implementation of GetDynamicMemberNames() to get the properties, and I assume that is isn't there.