Why does this AWS API Gateway Mapping Template not output the correct variable? - velocity

I am using AWS API Gateway. I have a response from my backend and I am trying to map the response into a different output using a Mapping Template.
I am trying to return each of the hits.hits._source.display elements. But the mapping template returns the entire object with the literal "_source.display" string added to the end.
The original unmodified server response is like -
{
"took":7,
"timed_out":false,
"_shards":{
"total":1,
"successful":1,
"skipped":0,
"failed":0
},
"hits":{
"total":{
"value":10000,
"relation":"gte"
},
"max_score":1.0,
"hits":[
{
"_index":"address",
"_type":"_doc",
"_id":"GAVIC411064535",
"_score":1.0,
"_source":{
"id":"GAVIC411064535",
"display":"1/28 George Street, Casterton VIC 3311",
"location":{
"lat":-37.59205672,
"lon":141.38825665
},
"component":{
"buildingName":"",
"number":"1/28",
"street":"George Street",
"locality":"Casterton",
"state":"VIC",
"postcode":"3311"
}
}
},
{
"_index":"address",
"_type":"_doc",
"_id":"GAVIC411066597",
"_score":1.0,
"_source":{
"id":"GAVIC411066597",
"display":"52 St Georges Road, Corio VIC 3214",
"location":{
"lat":-37.59205672,
"lon":141.38825665
},
"component":{
"buildingName":"",
"number":"52",
"street":"St Georges Road",
"locality":"Corio ",
"state":"VIC",
"postcode":"3214"
}
}
},
My Mapping Template so far looks like this -
#set($inputRoot = $input.path('$'))
{
#foreach($elem in $inputRoot.hits.hits)
{
"address": $elem._source.display,
}#if($foreach.hasNext),#end
#end
}
The resulting output looks like this. You can see that its returing the entire $elem rather than returning the properties I've asked for. That is ._source.display
It has also literally printed ._source.display to the end.
{
"address": {_index=address, _type=_doc, _id=GAVIC411064535, _score=1.0, _source={id=GAVIC411064535, display=1/28 George Street, Casterton VIC 3311, location={lat=-37.59205672, lon=141.38825665}, component={buildingName=, number=1/28, street=George Street, locality=Casterton, state=VIC, postcode=3311}}}._source.display,
},
{
"address": {_index=address, _type=_doc, _id=GAVIC411066597, _score=1.0, _source={id=GAVIC411066597, display=52 St Georges Road, Corio VIC 3214, location={lat=-37.59205672, lon=141.38825665}, component={buildingName=, number=52, street=Georges Road, locality=Corio , state=VIC, postcode=3214}}}._source.display,
},
I have truncated the server response and mapping template output for brevity.
The desired mapping template output is -
{
"address" : "28 George Street, Casterton VIC 3311",
"address" : "52 St Georges Road, Corio VIC 3214"
}

This is a bug in version 1.7 which has been fixed in 2.x versions (in July 2016... AWS really ought to upgrade its libraries sometimes).
You can work around this bug like this:
#foreach($elem in $inputRoot.hits.hits)
{
"address": "$elem.get('_source').display"
}#if($foreach.hasNext),#end
#end

Related

How to get selected json objects from a nested json in postgres sql

I have the data stored in a table named 'userfields' the below format in one of the columns named 'params' in Postgres SQL. I want to select two json objects and the output should be in the same format
{
"default": {
"model": "core.location",
"pk": 119
},
"field1": {
"name": "Vista Community Clinic- The Gary Center, S. Harbour Blvd",
"full_address": "201 S. Harbor Boulevard, \nLa Habra, CA 90631"
}
"fields2": {
"name": "xyz- The Gary Center, S. Harbour Blvd",
"full_address": "abc, \nLa Habra, CA 90631"
}
}
I have tried and achieved for one json object which will return for example default
The query I have used is
select json_extract_path(params::json,'default') as selectedparams from userfields.
I am not able to select two json objects like default and feild1
The output I have got is
{
"model": "core.location",
"pk": 119
}
The output I want is I want to get default and field1 as the nested JSON by using PostgreSQL13 query like below
Desired Result:
{
"default": {
"model": "core.location",
"pk": 119
},
"field1": {
"name": "Vista Community Clinic- The Gary Center, S. Harbour Blvd",
"full_address": "201 S. Harbor Boulevard, \nLa Habra, CA 90631"
}
}

How to get new Search Engine results in the past 24h using a SERP API?

Assume I am in possession of a SERP API, which given a keyword, returns me the Google results of that keyword in JSON format (for example: https://serpapi.com/):
{
"organic_results": [
{
"position": 1,
"title": "Coffee - Wikipedia",
"link": "https://en.wikipedia.org/wiki/Coffee",
"displayed_link": "https://en.wikipedia.org › wiki › Coffee",
"snippet": "Coffee is a brewed drink prepared from roasted coffee beans, the seeds of berries from certain Coffea species. From the coffee fruit, the seeds are ...",
"sitelinks":{/*snip*/}
,
"rich_snippet":
{
"bottom":
{
"extensions":
[
"Region of origin: Horn of Africa and ‎South Ara...‎",
"Color: Black, dark brown, light brown, beige",
"Introduced: 15th century"
]
,
"detected_extensions":
{
"introduced_th_century": 15
}
}
}
,
"about_this_result":
{
"source":
{
"description": "Wikipedia is a free content, multilingual online encyclopedia written and maintained by a community of volunteers through a model of open collaboration, using a wiki-based editing system. Individual contributors, also called editors, are known as Wikipedians.",
"source_info_link": "https://en.wikipedia.org/wiki/Wikipedia",
"security": "secure",
"icon": "https://serpapi.com/searches/6165916694c6c7025deef5ab/images/ed8bda76b255c4dc4634911fb134de53068293b1c92f91967eef45285098b61516f2cf8b6f353fb18774013a1039b1fb.png"
}
,
"keywords":
[
"coffee"
]
,
"languages":
[
"English"
]
,
"regions":
[
"the United States"
]
}
,
"cached_page_link": "https://webcache.googleusercontent.com/search?q=cache:U6oJMnF-eeUJ:https://en.wikipedia.org/wiki/Coffee+&cd=4&hl=en&ct=clnk&gl=us",
"related_pages_link": "https://www.google.com/search?q=related:https://en.wikipedia.org/wiki/Coffee+Coffee"
},
/* Results 2,3,4... */
]}
What is a good way to get new results from the past 24h? I added the &tbs=qdr:d query parameter, which only shows the results from the past day. That's a good first step.
The 2nd step is to filter out only relevant results. When there are no relevant results, Google shows this message box:
What is their algorithm to show this box?
Idea 1: "grep -i {exact_keywords}"
For example, if I search a keyword like "Alexander Pope", the 24h Google query might return results about the pope, written by a guy called Alexander. That's not super relevant. My naive idea is to grep (case insensitive) the exact keyword "Alexander Pope".
But that might leave out some good results.
Any other ideas?

JToken "becomes" null during iteration

I am developing a sort of gateway API for an Ionic mobile-app, which has to talk to some online API that doesn't accept front-end calls.
One call returns a JSON object, with a "data"-array of JObjects. A response of the server looks like this (below JSON just shows one result, to give an idea of what the server's response looks like):
{
"currentPage": 1,
"numberOfPages": 1,
"totalResults": 19,
"data": [
{
"id": "BznahA",
"name": "Anheuser-Busch InBev",
"nameShortDisplay": "Anheuser-Busch InBev",
"description": "Anheuser-Busch operates 12 breweries in the United States, 14 in China and one in the United Kingdom. Anheuser-Busch's operations and resources are focused on adding to life's enjoyment not only through the responsible consumption of beer by adults, but through theme park entertainment and packaging. In the United States, the company holds a 48.5 percent share of U.S. beer sales. Worldwide, Anheuser-Busch's beer sales volume was 128.4 million barrels in 2007. The St. Louis-based company's subsidiaries include one of the largest U.S. manufacturers of aluminum beverage containers and one of the world's largest recyclers of aluminum beverage cans. Anheuser-Busch also has interests in malt production, rice milling, real estate development, turf farming, metalized and paper label printing, bottle production and transportation services.",
"website": "http://www.anheuser-busch.com/",
"established": "1852",
"isOrganic": "N",
"images": {
"icon": "https://brewerydb-images.s3.amazonaws.com/brewery/BznahA/upload_0FkKKl-icon.png",
"medium": "https://brewerydb-images.s3.amazonaws.com/brewery/BznahA/upload_0FkKKl-medium.png",
"large": "https://brewerydb-images.s3.amazonaws.com/brewery/BznahA/upload_0FkKKl-large.png",
"squareMedium": "https://brewerydb-images.s3.amazonaws.com/brewery/BznahA/upload_0FkKKl-squareMedium.png",
"squareLarge": "https://brewerydb-images.s3.amazonaws.com/brewery/BznahA/upload_0FkKKl-squareLarge.png"
},
"status": "verified",
"statusDisplay": "Verified",
"createDate": "2012-01-03 02:41:44",
"updateDate": "2018-11-02 02:15:01",
"isMassOwned": "Y",
"isInBusiness": "Y",
"isVerified": "N",
"locations": [
{
"id": "ZApGug",
"name": "Main Brewery",
"streetAddress": "One Busch Place",
"locality": "Saint Louis",
"region": "Missouri",
"postalCode": "63118",
"phone": "(314) 577-2626",
"website": "http://www.anheuser-busch.com/",
"latitude": 38.5985037,
"longitude": -90.2093428,
"isPrimary": "Y",
"inPlanning": "N",
"isClosed": "N",
"openToPublic": "Y",
"locationType": "macro",
"locationTypeDisplay": "Macro Brewery",
"countryIsoCode": "US",
"yearOpened": "1852",
"status": "verified",
"statusDisplay": "Verified",
"createDate": "2012-01-03 02:41:44",
"updateDate": "2018-11-02 02:14:56"
Now, my gateway API processes this into self-made objects and returns it to the mobile app. The methods are defined as such:
var resp = await _clientFactory.CreateClient().GetAsync(
apiCall.Uri
);
var token = JToken.Parse(await resp.Content.ReadAsStringAsync());
var data = token["data"]; // gets data array from parsed JToken
// iterates through data, makes models and returns
return data.Select(ParseBrewery).ToList();
This next method is where the NullPointer error occurs. This method takes the JTokens from the data in the method above. However, the breweryToken parameter throws a NullPointer exception.
// takes JToken and assigns element to returned model
private static Brewery ParseBrewery(JToken breweryToken) => new Brewery()
{
Id = breweryToken["id"].ToObject<string>(),
Name = breweryToken["name"].ToObject<string>(),
NameShortDisplay = breweryToken["nameShortDisplay"]?.ToObject<string>(),
CreateDate = breweryToken["createDate"].ToObject<DateTimeOffset>(),
UpdateDate = breweryToken["updateDate"]?.ToObject<DateTimeOffset>(),
Status = breweryToken["status"].ToObject<string>(),
InBusiness = breweryToken["inBusiness"]?.ToObject<string>() != "N",
IsVerified = breweryToken["isVerified"].ToObject<string>() != "N",
Locations = JToken.Parse(breweryToken["locations"].ToString())?.Select(ParseLocation).ToList()
};
The ParseLocation-method handles the location object (a subobject in each object in the data JToken).
// takes JToken (parsed in ParseBrewery method) and assigns values to returned model
private static Location ParseLocation(JToken locationToken) => new Location()
{
Id = locationToken["id"].ToObject<string>(),
Name = locationToken["name"].ToObject<string>(),
StreetAddress = locationToken["streetAddress"].ToObject<string>(),
City = locationToken["locality"].ToObject<string>(),
Region = locationToken["region"].ToObject<string>(),
PostalCode = locationToken["postalCode"].ToObject<string>(),
Website = locationToken["website"].ToObject<string>(),
Latitude = locationToken["latitude"].ToObject<double>(),
Longitude = locationToken["longitude"].ToObject<double>(),
CreateDate = locationToken["createDate"].ToObject<DateTimeOffset>(),
Status = locationToken["status"].ToObject<string>(),
Country = ParseCountry(locationToken["country"])
};
This method again handles a subobject of the location object.
// takes JToken from ParseLocation method and returns filled model
private static Country ParseCountry(JToken countryToken) => new Country()
{
IsoCode = countryToken["isoCode"].ToObject<string>(),
Name = countryToken["name"].ToObject<string>(),
IsoThree = countryToken["isoThree"]?.ToObject<string>(),
NumberCode = countryToken["numberCode"]?.ToObject<int>(),
CreateDate = countryToken["createDate"].ToObject<DateTimeOffset>()
};
The problem I am running into is, as I mentioned at the corresponding method, somehow the JToken from data seems to become null at some point or another. Debugging through the code shows there is definitely data being returned, but after two or three iterations, it just jumps to the end of the call and throws a NullPointer exception.
I have tried running the variable data through JToken.Parse separately, but I keep running into the same issue. Parsing each separate item in the ParseBrewery method call also returns an error.
I am at a point where I do not know what else I could try to get this to work. I have simply no idea why this could be happening, but I am quite sure it must be a tiny thing as another part of the API which works exactly the same way does return data succesfully.
Hopefully I have provided you with enough info and example code to help understand my problem. If not or if you have any questions, of course feel free to ask and I will answer to the best of my ability!
Thankyou in advance.

Handling null value inside Map

I'm learning to use BLOC pattern for consume API from newsapi.org. I have result JSON and Model like this from https://newsapi.org/v2/everythingq=flutter&apiKey=7f8eef4261bc4c29b26c4d5e93e8b5d6:
Json
{
"status": "ok",
"totalResults": 533,
"articles": [
{
"source": {
"id": "cnn",
"name": "CNN"
},
"author": "Sandee LaMotte, CNN",
"title": "Burnout linked to potentially deadly irregular heartbeat, study says",
"description": "If you're feeling bone-deep mental and physical exhaustion, or what is otherwise known as burnout, new research suggests you could be at a higher risk for a potentially fatal heart flutter.",
"url": "https://www.cnn.com/2020/01/13/health/burnout-linked-to-atrial-fibrillation-wellness/index.html",
"urlToImage": "https://cdn.cnn.com/cnnnext/dam/assets/200113120102-02-burnout-stock-super-tease.jpg",
"publishedAt": "2020-01-14T00:15:07Z",
"content": "(CNN)If you're feeling bone-deep mental and physical exhaustion, or what is otherwise known as burnout, new research suggests you could be at a higher risk for a potentially fatal heart flutter.\r\nAtrial fibrillation, also called AFib or AF, is the most common… [+4971 chars]"
},
{
"source": {
"id": "the-verge",
"name": "The Verge"
},
"author": "Ashley Carman",
"title": "The sex toy banned from CES last year is unlike anything we’ve ever seen",
"description": "Lora DiCarlo’s Osé personal massager made it to CES this year, after being banned and having its award revoked in 2019. The device is certainly innovative, using suction and movement to stimulate the G-spot and clitoris instead of vibration.",
"url": "https://www.theverge.com/2020/1/6/21051259/lora-dicarlo-sex-tech-toy-massager-awards-ban-ose-ces-2020",
"urlToImage": "https://cdn.vox-cdn.com/thumbor/G5zPZiQZgwqS9FqLqzUE3eyyLyA=/0x146:2040x1214/fit-in/1200x630/cdn.vox-cdn.com/uploads/chorus_asset/file/19576504/loradicarlo1.jpg",
"publishedAt": "2020-01-06T14:25:23Z",
"content": "Its certainly innovative\r\nAshley Carman / The Verge\r\nLora DiCarlo likely didnt intend to change CES forever, but the sexual wellness company and its personal massager accomplished just that last year. After having an award revoked and being banned from the sh… [+3125 chars]"
},
{
"source": {
"id": null,
"name": "Codemagic.io"
},
"author": null,
"title": "Flutter vs. Other Mobile Development Frameworks: A UI and Performance Experiment",
"description": "In this experiment we created the same app five times, each time with a different framework. The frameworks I used were native Android, native iOS, Flutter, Xamarin Forms and React Native",
"url": "https://blog.codemagic.io/flutter-vs-ios-android-reactnative-xamarin/",
"urlToImage": "https://blog.codemagic.io/uploads/Codemagic-io_Blog_Flutter-Versus-Other-Mobile-Development-Frameworks_1.png",
"publishedAt": "2020-01-03T21:26:48Z",
"content": "Written by Bram De Coninck\r\nIve already read a lot of articles about how Flutter compares to other mobile development frameworks, but most of them were written from a theoretical point of view. Its already been well established that Flutter is a solid choice … [+24309 chars]"
},
{
"source": {
"id": null,
"name": "Insider.com"
},
"author": "dschild#businessinsider.com (Darcy Schild), Darcy Schild",
"title": "Kylie Jenner shared a sneak peek of her new makeup collection inspired by her daughter, Stormi",
"description": "Kylie Jenner/YouTube; Kylie Jenner/Instagram Kylie Jenner, the makeup mogul behind Kylie Cosmetics, is launching a collection of new eyeshadows, Lip Kits, lip glosses, and a blush inspired by her 1-year-old daughter, Stormi. Jenner gave her followers a sneak …",
"url": "https://www.insider.com/kylie-jenner-cosmetics-stormi-collection-when-to-buy-2020-1",
"urlToImage": "https://i.insider.com/5e24bb4b3ac0c9452948fd10?width=1200&format=jpeg",
"publishedAt": "2020-01-19T20:24:58Z",
"content": "Kylie Jenner is launching a makeup collection inspired by her 1-year-old daughter, Stormi Webster.\r\nThe Stormi Collection by Kylie Cosmetics will be available for purchase starting February 1, Jenner said in an Instagram Story post on Saturday.\r\nIn November 2… [+2710 chars]"
},
]
}
NewsModel
class NewsModel {
final Map<String, dynamic> source;
final String author;
final String title;
final String description;
final String url;
final String urlToImage;
final String publishedAt;
final String content;
NewsModel({
this.source,
this.author,
this.title,
this.description,
this.url,
this.urlToImage,
this.publishedAt,
this.content,
});
NewsModel.fromJson(Map<String, dynamic> parsedJson)
: source = parsedJson['source']== null
? "Unknown Source"
: parsedJson['source'],
author = parsedJson['author'] == null
? "Unknown Author"
: parsedJson['author'],
title =
parsedJson['title'] == null ? "Unknown Title" : parsedJson['title'],
description = parsedJson['description'] == null
? "Unknown Description"
: parsedJson['description'],
url = parsedJson['url'] == null ? "Unknown Url" : parsedJson['url'],
urlToImage = parsedJson['urlToImage'] == null
? "https://homepages.cae.wisc.edu/~ece533/images/peppers.png"
: parsedJson['urlToImage'],
publishedAt = parsedJson['publishedAt'] == null
? "Unknown PublishedAt"
: parsedJson['publishedAt'],
content = parsedJson['content'] == null
? "Unknown Content"
: parsedJson['content'];
Map<String, dynamic> toMapForDb() {
return <String, dynamic>{
"source": source,
"author": author,
"title": title,
"description": description,
"url": url,
"urlToImage": urlToImage,
"publishedAt": publishedAt,
"content": content,
};
}
}
final newsModel = NewsModel();
source in this json is MAP,But one of its value is null (id). How to handling null value in my model ?
I'm trying handling this null value like this
source = parsedJson['source']['id'] == null? "Unknown Source" : parsedJson['source'],
But i get the error message Error From Bloc =type 'String' is not a subtype of type Map<String, dynamic>
My temporary solution to handling null value return Text(news.source["id"] ?? "null"); . But i want handling inside my model.
If you need BLOC source code
BLOC
import 'package:network/models/news_model.dart';
import 'package:repositories/repositories/news_repositories.dart';
import 'package:rxdart/subjects.dart';
class NewsBloc {
final _newsRepositories = NewsRepositories();
final _newsFetch = PublishSubject<List<NewsModel>>();
Stream<List<NewsModel>> get allEverythingNews => _newsFetch.stream;
Future<void> getEverythingNews() async {
try {
List<NewsModel> news = await _newsRepositories.getNewsEverything();
final newsAddSink = _newsFetch.sink.add(news);
return newsAddSink;
} catch (e) {
print("Error From Bloc =" + e.toString());
_newsFetch.sink.addError(e);
}
}
void dispose() {
_newsFetch.close();
}
}
final newsBloc = NewsBloc();
As can be seen from json you are getting object as a source and in your model you create a map to store source but when source is null then you are trying to assign String value to it. That’s why you are getting error.
Try To assign map like:
source = parsedJson['source']== null
? {"id": 0, "name": "Unknown Source"}
: parsedJson['source'],
By this error:
Error From Bloc =type 'String' is not a subtype of type Map
It is seems that the value provided in the NewsModel.fromJson() is a string and not a Map.
You need to do this:
NewsModel.fromJson(json.decode(STRING_FROM_API_RESPONSE))
Note that json is from import 'dart:convert';

Recessively query Redis based on data type

I am representing JSON objects in Redis by flattening the JSON path keys like so:
JSON object
{
"users": [
{
"name": "John Smith",
"address": {
"street": "12 Rainbow Avenue",
"country": "Fuzzyland"
}
}, {
"name": "Billy Bob",
"address": {
"street": "8 Rocky Lane",
"country": "Fraggleville"
}
}
]
}
Redis entries
users // SET [0, 1]
users:0 // SET [name, address]
users:0:name // STRING "John Smith"
users:0:address // SET [street, country]
users:0:address:street // STRING "12 Rainbow Avenue"
users:0:address:country // STRING "Fuzzyland"
users:1 // SET [name, address]
users:1:name // STRING "Billy Bob"
users:1:address // SET [street, country]
users:1:address:street // STRING "8 Rocky Lane"
users:1:address:country // STRING "Fraggleville"
As you can see, for properties that are arrays or object literals I am using a Redis SET to store the available keys.
It is easy enough for me to convert the JSON object to this structure using Node. However I am not experienced enough with Redis to write recursive queries that stitch together a complex result.
For example, say I want to get all the keys and values for users:1. How do I write a query that looks at the value of users:1 sees that it is a SET so uses the values from that as another set of query keys?