The app I'm working on responds to most requests with JSON objects or collections thereof. We're using Jbuilder to construct those responses. The amount of data rendered is fairly large (several thousand objects in various nested structures - once formatted and fully expanded, there are as many as 10,000 lines of JSON for a typical response). This rendering is taking a significant amount of time - about 1/3 of the total request time, according to NewRelic.
I'm looking for some kind of guide, set of tips, or other resource that will help me make sure I'm getting the best possible performance out of JBuilder. I'm also curious if there are performance comparisons available for Jbuilder vs. RABL or other similar tools.
Edit: I've found a GitHub Issue that complains about Jbuilder performance, but the only actual suggestion anyone's made is 'don't use Jbuilder'. Well, actually, they used slightly stronger language, but there's still no word on why Jbuilder is so slow, what, if anything, can be done to get around it, or how other tools for the same task compare.
jbuilder builds up a big hash containing your data and then uses ActiveSupport::JSON to turn it into json. There are faster json emitters as the following micro benchmark shows (make sure you have the multijson and yajl-ruby gems installed)
require 'benchmark'
require 'active_support'
require 'multi_json'
sample = {menu: {
header: "SVG Viewer",
items: [
{id: "Open"},
{id: "OpenNew", label: "Open New"},
nil,
{id: "ZoomIn", label: "Zoom In"},
{id: "ZoomOut", label: "Zoom Out"},
{id: "OriginalView", label: "Original View"},
nil,
{id: "Quality"},
{id: "Pause"},
{id: "Mute"},
nil,
{id: "Find", label: "Find..."},
{id: "FindAgain", label: "Find Again"},
{id: "Copy"},
{id: "CopyAgain", label: "Copy Again"},
{id: "CopySVG", label: "Copy SVG"},
{id: "ViewSVG", label: "View SVG"},
{id: "ViewSource", label: "View Source"},
{id: "SaveAs", label: "Save As"},
nil,
{id: "Help"},
{id: "About", label: "About Adobe CVG Viewer..."}
]
}}
MultiJson.engine = :yajl
Benchmark.bmbm(5) do |x|
x.report 'activesupport' do
1000.times {ActiveSupport::JSON.encode(sample)}
end
x.report 'yajl' do
1000.times {MultiJson.encode(sample)}
end
end
On my machine this produces
user system total real
activesupport 1.050000 0.010000 1.060000 ( 1.068426)
yajl 0.020000 0.000000 0.020000 ( 0.021169)
ie to encode the sample object 1000 times active support took a hair over 1 second, MultiJson (using the yajl engine) took 21ms.
JBuilder is hardcoded to use ActiveSupport::JSON, but MultiJSON (a gem that lets you switch between json libraries) is a trivial drop in and is already a dependency of ActiveSupport - see my fork of jbuilder. I've opened a pull request, but until then you could try using this fork (or create your own - it's a one line change)
Consider switching to Rabl and adding some caching. Given you have thousands of objects in nested structures, some nodes of your resulting JSON can be rendered as partials and cached - the performance gain can be huge.
Apart from this Rabl performance is slightly better than performance of JBuilder, but I find Rabl syntax sometimes confusing and I'd switch to JBuilder once it has fragment caching implemented.
As stated before JBuilder builds a hash, then serializes that hash to JSON.
The same with caching, there is the main hash and the cached hash get merged into the main hash which still needs to be converted to JSON.
My solution was TurboStreamer. TurboStreamer outputs directly to an IO/Stream/String therefore skipping the serialization step that JBuilder (and at first glance this still applies to Rabl, and to_json depending on usage).
For us this has significantly reduced render time & GC times (due to building the hash in jbuilder) and allows us to start streaming JSON out to the client as we get our results. The downside is TurboStreamer is a little more verbose and explicit.
Performance Test A (no caching involved):
source
results
Performance Test B (mostly all caching):
source
results
Related
Backstory:
I'm building an e-commerce web app (online store)
Now I got to the point of choosing a database system and an appropriate design.
I got stuck with developing a design for product attributes
I've been considering of choosing NoSQL (MongoDB) or SQL database systems
I need you advice and help
The problem:
When you choose a product type (e.g. table) it should show you the corresponding filters for such a type (e.g. height, material etc.). When you choose another type, say "car", it provides you with the car specific filter attributes (e.g. fuel, engine volume)
For example, here on one popular online store if you choose a data storage type you get a filter fo this type attributes, such as hard drive size or connection type
Question
What approach is the best for such a problem? I described some below, but maybe you have your own thoughts in regard to it
MongoDB
Possible solution:
You can implement such product attrs structure pretty easy.
You can create one collection with a field attrs for each product and put there whatever you want, like they suggest here (field "details"):
https://docs.mongodb.com/ecosystem/use-cases/product-catalog/#non-relational-data-model
The structure will be
Problem:
With such a solution you don't have product types at all so you can't filter the products out by their types. Each product contains it's own arbitrary structure in attrs field and don't follow any pattern
Ir maybe I can somehow go with this approach?
SQL
There are solutions like single table where all the products store in one table and you end up with as many fields as an attribute number of all the products taken together.
Or for every product type you create a new table
But I won't consider these ones. One is very bulky and another one isn't much flexible and requires a dynamic scheme design
Possible solution
There is one pretty flexible solution called EAV https://en.wikipedia.org/wiki/Entity%E2%80%93attribute%E2%80%93value_model
Our schema would be:
EAV
Such a design may be done on MongoDB system, but I'm not sure it's been made for such a normalised structure
Problem
The schema is going to get really huge and really hard to query and grasp
If you choose SQL database, take a look PostgreSQL which supports JSON features. Not necessarily you need to follow Database normalization.
If you choose MongoDB, you need to store attrs array with generic {key:"field", value:"value"} pairs.
{id:1, attrs:[{key: "prime", value: true}, {key:"height", value:2}, {key:"material", value:"wood"},{key:"color", "value":"brown"}]}
{id:2, attrs:[{key: "prime", value: true}, {key:"fuel", value:"gas"}, {key:"volume", "value":3}]}
{id:3, attrs:[{key: "prime", value: true}, {key:"fuel", value:"diesel"}, {key:"volume", "value":1.5}]}
Then you define Multi-key index like this:
db.collection.createIndex({"attrs.key":1, "attrs.value":1})
If you want apply step-by-step filters, use MongoDB aggregation with $elemMatch operator
☑ Prime
☑ Fuel
☐ Other
...
☑ Volume 3
☐ Volume 1.5
Query's representation
db.collection.aggregate([
{
$match: {
$and: [
{
attrs: {
$elemMatch: {
key: "prime",
value: true
}
}
},
{
attrs: {
$elemMatch: {
key: "fuel"
}
}
},
{
attrs: {
$elemMatch: {
key: "volume",
"value": 3
}
}
}
]
}
}
])
MongoPlayground
I'm building a Zapier app for a platform that have dynamic fields. I have an API that returns the list of fields for one of my resource (for example) :
[
{ name: "First Name", key: "first_name", type: "String" },
{ name: "Civility", key: "civility", type: "Multiple" }
]
I build my action's inputFields based on this API :
create: {
[...],
operation: {
inputFields: [
fetchFields()
],
[...]
},
}
The API returns type that are list of values (i.e : Civility), but to get these values I have to make another API call.
For now, what I have done is in my fetchFields function, each time I encounter a type: "Multiple", I do another API call to get the possible values and set it as choices in my input field. However this is expensive and the page on Zapier takes too much time to display the fields.
I tried to use the z.dehydrate feature provided by Zapier but it doesn't work for input choices.
I can't use a dynamic dropdown here as I can't pass the key of the field possible value I'm looking for. For example, to get back the possible values for Civility, I'll need to pass the civility key to my API.
What are the options in this case?
David here, from the Zapier Platform team.
Thanks for writing in! I think what you're doing is possible, but I'm also not 100% that I understand what you're asking.
You can have multiple API calls in the function (which it sounds like you are). In the end, the function should return an array of Field objects (as descried here).
The key thing you might not be aware of is that subsequent steps have access to a partially-filled bundle.inputData, so you can have a first function that gets field options and allows a user to select something, then a second function that runs and pulls in fields based on that choice.
Otherwise, I think a function that does 2 api calls (one to fetch the field types and one to turn them into Zapier field objects) is the best bet.
If this didn't answer your question, feel free to email partners#zapier.com or join the slack org (linked at the bottom of the readme) and we'll try to solve it there.
Here is my situation, context, and dilemma.
Situation
I'm fairly new to Laravel and still learning the ropes. I recently installed TNTSearch and Laravel Scout and was able to create a model index using the below config. I created the index using the console command php artisan tntsearch:import "App\Models\Product" and can fuzzy search successfully with App\Models\Product::search($keyword)->get().
config/scout.php
'tntsearch' => [
'storage' => storage_path() . '/index',
'fuzziness' => 'auto',
'fuzzy' => [
'prefix_length' => 2,
'max_expansions' => 50,
'distance' => 4,
],
'asYouType' => true
],
Context
I have an SQL database table with over 30k+ product records segmented per province (Canadian project), and instead of searching the whole index and later filter by market, I’d like to create one index per market and launch a search for a given market. I believe it will speed up the search and avoid returning results which will later be discarded! So basically having one product index file per province (i.e. products_on.index, products_qc.index, ...)
Dilemma/Issue
I am unable to find how to create such an index, have it update automatically and also how to use it. I scoured the Internet for tutorial/guidance and could only find scarce information I can hardly put together. I’d appreciate if someone could point me in the right direction or guide me on how to implement such a thing.
No answer is wrong, and any bits and pieces of information can help me greatly to “get up to speed.”
EDIT (July 30th, 2018):
I still haven't found the answer to my request but the more I search, the more I'm concluding search indexes are "tied" to a model, and it is not possible to have more than one index per model. So I would have to create one model extension per market from the original Listings model (Listings_QC, Listings_ON, ...). Then create an index per markets and search from those (Listings_QC::search(...)).
I'm not keen to create models based on data! Is this a good approach/practice?
RESOLVED !
My inexperience with Laravel search index in general lead me in the wrong direction!
I finally found a document explaining how to use searchBoolean() to search using "and". Modified my config as below to add the searchBoolean:
'tntsearch' => [
'storage' => storage_path() . '/index',
'fuzziness' => 'auto',
'fuzzy' => [
'prefix_length' => 2,
'max_expansions' => 50,
'distance' => 4,
],
'asYouType' => true,
'searchBoolean' => true
],
Then specify the market using the model's method toSearchableArray(), and add the market to any requested seach keyword.
For example, listing search with 'Alsace' for a QC market, I launch the search as
Listings::search('Alsace QC')->get().
Voilà! May help others hitting the same "wall"!
I am having trouble passing params to ajax get request.
Let's suppose i have to pass params {category: 'cat', type: 'type', searchKey: 'key' } to the url /search and I have the code below:
action$.ofType('QUERY')
.debounceTime(500)
.switchMap(action =>
ajax.get('/search', {//pass some parameters},)
.map(result => ({
type: 'FETCH_SUCCESS',
payload: result.response,
})),
As I am new to RxJs, Please suggest me the right way of doing this.
While it is technically permissible to provide a request body (and corresponding Content-Type header like application/x-www-form-urlencoded) for GET requests, nearly all servers assume GET do not contain one. Instead, POST (creation of a document) or PUT (updating a document) is used when a body is neccesary.
However, if what you're asking for is simply regular old query params, that's pretty normal but there is no built-in support in RxJS for converting an Object to a query string--mostly because there is no official spec on how complex objects/arrays should be serialized so every server has notable differences once you do more than simple key -> value.
Instead, you just include them in the URL. I realize you said "without building url" but the lack of a spec means RxJS will likely never add support because it's highly opinionated. You can just manually generate the string yourself or use a third-party utility that has a convention you like.
ajax.get(`/search?query=${query}&something=${something`)
// or using something like https://www.npmjs.com/package/query-string
ajax.get(`/search?${queryString.stringify(params)}`)
If you're interested in further understanding the opinionated nature of query params, consider how you might serialize { foo: [{ bar: 1 }, { bar: 2 }] }. Some might say it should be ?foo[0][bar]=1&foo[1][bar]=2 but I have also seen ?foo[][bar]=1&foo[][bar]=2, ?foo[bar]=1&foo[bar]=2, and other variants. Thing get even more hairy when dealing with duplicates. ?foo=true&foo=false should foo be true or false? (it's an opinion hehe)
I am a complete beginner in PostgreSQL. And I was really amazed by the hstore datatype provided by Postgres. Well, I am using the Rails 3 framework and developing a simple app that uses PostgreSQL. I want to store an array of hashes in a field.
For Eg.:
authors: [
{
name: "abc",
email: "abc#example.com"
},
{
name: "xyz",
email: "xyz#example.com"
}
]
Is this possible in PostgreSQL using Rails 3? If so, can somebody give insights on how?
Thanks
It's certainly possible to create an array-of-hstore column in Rails 4 with e.g. a column spec like this in the table creation:
t.hstore :properties, :array => true
However, there's an encoding bug in Rails 4.0 that unfortunately renders them unusable; essentially you can read from them and they present correctly as arrays of hashes, but not write.
I've opened an issue (with fix patch) at https://github.com/rails/rails/issues/11135 which hopefully will be incorporated soon.
I'm not sure if they'll allow you to have an hstore array, but there are a few active record extensions that add hstore and array types. e.g.:
https://github.com/funny-falcon/activerecord-postgresql-arrays
https://github.com/engageis/activerecord-postgres-hstore
https://github.com/tlconnor/activerecord-postgres-array
Don't miss the Rails 4-related improvements, too:
http://blog.remarkablelabs.com/2012/12/a-love-affair-with-postgresql-rails-4-countdown-to-2013