How do you do something like gun.get({startkey, endkey}) ?
Previously: https://github.com/amark/gun/issues/479
#qwe123wsx #sebastianmacias apologies for the delay! Originally posted at: https://github.com/amark/gun/issues/479
The wire spec has a protocol for this but it isn't implemented yet. It looks something like this:
gun.on('out', {get: {'#': {'>': 'a', '<': 'b'}}});
However this doesn't work yet. I would recommend instead:
(1) Pagination behavior is very different from one app to another and will be hard for us to create a "one-size-fits-all" solution, so it would be highly helpful if you could implement your own* pagination and make it available as a user-module, then we can learn from your experience (what worked, what didn't) and make the best solution part of core.
(2) Your app will probably work fine without pagination in the meanwhile, while it can be built (it is targeted for after 1.0), and then as your app becomes more popular, it should be fairly easy to add in without much refactor, once you need it and it is available.
... * How to build your own?
Lots of good articles on this, best one I've seen yet is from Neo4j on how to do it in a graph database (which applies to gun as well) https://graphaware.com/neo4j/2014/08/20/graphaware-neo4j-timetree.html .
Another rough idea is you model your data based on pagination or time. So rather than having ALL tweets go into user's tweet table, instead, the user's tweet table is a table of DAYS (or weeks), and then you put the tweet inside the week table. Now when you load the data, you can scan/skip based off of week very easily while it being super bandwidth efficient.
Rough PSEUDO code:
function onTweetSend(tweet){
gun.get('user').get('alice').get('tweets').get(Date.uniqueYear() + Date.uniqueWeek()).set(tweet)
}
function paginateUserTweet(howMany, cb){
var range = convertToArrayOfUniqueWeekNamesFromToday(howMany);
var all = [];
range.forEach(function(week){
gun.get('user').get('alice').get('tweets').get(week).load(function(tweets){
all.push(tweets);
if(all.length < range.length){ return }
all = flattenArray(all);
cb(all);
});
});
}
Now we can use https://gun.eco/docs/RAD#lex
gun.get(...).get({'.': {'>': startkey, '<': endkey}, '%': 50000}).map().once(...)
Related
What I had till today:
I have get_jwt.feature and I call it as a part of karate-config.js. Since I have used one account test#test.com I needed only one jwt and I can reuse it across scenarios. callSingle worked as a charm in this case.
Today:
Suddenly I have need for jwt's from two accounts which I dont want to generate for each scenario, callSingle falls short of this task as it does exactly what its supposed to be doing. Now I have hacky idea, I can simply make two files, get_jwt.feature and get_jwt_user2.feature, and single call them each.
So my question: Is there a better way of doing this?
You can use "2 levels" of calls. So point the callSingle() to a JS function that calls get_jwt.feature 2 times, maybe with different arguments and then return a JSON. Pseudo-code below. First is get_jwts.js:
function fn(users) {
var jwt1 = karate.call('get_jwt.feature', users.user1);
var jwt2 = karate.call('get_jwt.feature', users.user2);
return { jwt1: jwt1, jwt2: jwt2 };
};
Then in karate-config.js
config.jwts = karate.callSingle('classpath:get_jwts.js', users);
And now you should be able to do this:
* print jwts.jwt1
* print jwts.jwt2
You can also do a feature-->feature call chain. Do let me know if this works !
EDIT: see Babu's answer in the comments, looks like you can pass an array to callSingle() ! so that may be quite convenient :)
So I'm working on a rails app for a building that keeps track of water usage/collection and electricity use/solar generation, etc. These are stored as measurement rows, attached to sensors, which are attached to programs (location in the building, essentially) and subtypes (attached to types - water, electricity).
I'm doing some graphing with chartkick, and the database calls related to this are way too slow. They'll be much faster on the production servers, but there will also be far more data.
Here's the helper method that has the chart generation and database call in it:
def stackedSubtypeChart(grouping)
rsubs = #resource.subtypes
.order(:usage?) #add usage types after gen types
.map{|stype| [
stype.name,stype.measurements #this takes too long!
.where("date >= ?", params[:start]) #(4 calls!!)
.where("date <= ?", params[:stop])
.group_by_period(grouping, :date).maximum(:amount)]}
rsubs = rsubs.map {|stype|
{name: stype[0],
data: stype[1]}}
ret = column_chart rsubs,
stacked: true,
library: { :series => {0 => { type: "line"}}}
end
#resource is defined in the controller as:
#resource = Type.includes(:subtypes => :sensors).find_by_resource('electricity')
I've commented the line that's responsible for there being multiple calls, which is definitely part of the problem. This takes two seconds to load on my (admittedly very very old) computer with a month of data.
I could really use help with both changing the map so that this is one call instead of however-many-subtypes calls, and with reducing what I'm pulling in so each call isn't taking half a second. I don't have a ton of experience optimizing this sort of thing and I'm not really sure how to start doing more than I have here already.
Might be helpful to look into ActiveRecord Explain to dig into the SQL. There's a good screencast that explains (pun totally intended) pretty well.
After a lot of bashing my head against a wall, I stumbled across this, which is a much faster single query that grabs all the data + data connections I need. It's a little hard to format but it works.
rsubs = Measurement
.where("measurements.date >= ? AND measurements.date <= ?",
offset(params[:start], -1, grouping),
offset(params[:stop], 1, grouping))
.joins(sensor: {subtype: :type})
.where("types.resource = ?", #rname)
.order('subtypes."usage?"')
.group_by_period(grouping, :date).group("subtypes.id, subtypes.name").maximum(:amount)
I make a query via createBuilder() and when executing it (getQuery()->execute()->toArray())
I got 10946 elements. I want to paginate it, so I pass it to:
$paginator = new \Phalcon\Paginator\Adapter\QueryBuilder(array(
"builder" => $builder,
"limit" => $limit,
"page" => $current_page
));
$limit is 25 and $current_page is 1, but when doing:
$paginator->getPaginate();
$page->total_items;
returns 1.
Is that a bug or am I missing something?
UPD: it seems like when counting items it uses created sql with limit. There is no difference what limit is, limit divided by items per page always equals 1. I might be mistaken.
UPD2: Colleague helped me to figure this out, the bug was in the query phalcon produces: count() of the group by counts grouped elements. So a workaround looks like:
$dataCount = $builder->getQuery()->execute()->count();
$page->next = $page->current + 1;
$page->before = $page->current - 1 > 0 ? $page->current - 1 : 1;
$page->total_items = $dataCount;
$page->total_pages = ceil($dataCount / 100);
$page->last = $page->total_pages;
I know this isn't much of an answer but this is most likely to be a bug. Great guys at Phalcon took on a massive job that is too big to do it properly in their little free time and things like PHQL, Volt and other big but non-core components do not receive as much attention as we'd like. Also given that most time in the past 6 months was spent on v2 there are nearly 500 bugs about stuff like that and it's counting. I came across considerable issues in ORM, Volt, Validation and Session, which in the end made me stick to other not as cool but more proven solutions. When v2 comes out I'm sure all attention will on the bug list and testing, until then we are mostly on our own. Given that it's all C right now, only a few enthusiast get involved, with v2 this will also change.
If this is the only problem you are hitting, the best approach is to update your query to get the information you need yourself without getPaginate().
I have this use case where I get the symbolized deep associations from a certain model, and I have to perform certain queries that involve using outer joins. How can one do it WITHOUT resorting to write the full SQL by hand?
Answers I don't want:
- using includes (doesn't solve deep associations very well ( .includes(:cars => [:windows, :engine => [:ignition]..... works unexpectedly ) and I don't want its side-effects)
- writing the SQL myself (sorry, it's 2013, cross-db support, etc etc..., and the objects I fetch are read_only, more side-effects)
I'd like to have an Arel solution. I know that using the arel_table's from the models I can construct SQL expressions, there's also a DSL for the joins, but somehow i cannot use it in the joins method from the model:
car = Car.arel_table
engine = Engine.arel_table
eng_exp = car.join(engine).on(car[:engine_id].eq(engine[:id]))
eng_exp.to_sql #=> GOOD! very nice!
Car.joins(eng_exp) #=> Breaks!!
Why this doesn't work is beyond me. I don't know exactly what is missing. But it's the closest thing to a solution I have now. If somebody could help me completing my example or provide me with a nice work-around or tell me when will Rails include such an obviously necessary feature will have my everlasting gratitude.
This is an old question, but for the benefit of anyone finding it through search engines:
If you want something you can pass into .joins, you can either use .create_join and .create_on:
join_on = car.create_on(car[:engine_id].eq(engine[:id]))
eng_join = car.create_join(engine, join_on, Arel::Nodes::OuterJoin)
Car.joins(eng_join)
OR
use the .join_sources from your constructed join object:
eng_exp = car.join(engine, Arel::Nodes::OuterJoin).on(car[:engine_id].eq(engine[:id]))
Car.joins(eng_exp.join_sources)
I found a blog post that purports to address this problem: http://blog.donwilson.net/2011/11/constructing-a-less-than-simple-query-with-rails-and-arel/
Based on this (and my own testing), the following should work for your situation:
car = Car.arel_table
engine = Engine.arel_table
sql = car.project(car[Arel.star])
.join(engine, Arel::Nodes::OuterJoin).on(car[:engine_id].eq(engine[:id]))
Car.find_by_sql(sql)
If you don't mind adding a dependency and skipping AREL altogether, you could use Ernie Miller's excellent Squeel gem. It would be something like
Car.joins{engine.outer}.where(...)
This would require that the Car model be associated with Engine like so:
belongs_to :engine
I asked on SO a few days ago what was the simplest quickest way to build a wrapper around a recently completed database. I took the advice and used sqlmetal to build linq classes around my database design.
Now I am having two problems. One, I don't know LINQ. And, two, I have been shocked to realize how hard it is to learn. I have a book on LINQ (Linq In Action by Manning) and it has helped some but at the end of the day it is going to take me a couple of weeks to get traction and I need to make some progress on my project today.
So, I am looking for some help getting started.
Click HERE To see my simple database schema.
Click HERE to see the vb class that was generated for the schema.
My needs are simple. I have a console app. The main table is the SupplyModel table. Most of the other tables are child tables of the SupplyModel table.
I want to iterate through each of Supply Model records. I want to grab the data for a supply model and then DoStuff with the data. And I also need to iterate through the child records, for each supply model, for example the NumberedInventories and DoStuff with that as well.
I need help doing this in VB rather than C# if possible. I am not looking for the whole solution...if you can supply a couple of code-snippets to get me on my way that would be great.
Thanks for your help.
EDIT
For the record I have already written the following code...
Dim _dataContext As DataContext = New DataContext(ConnectionStrings("SupplyModelDB").ConnectionString)
Dim SMs As Table(Of Data.SupplyModels) = _dataContext.GetTable(Of Data.SupplyModels)()
Dim query = From sm In SMs Where sm.SupplyModelID = 1 Select sm
This code is working...I have a query object and I can use ObjectDumper to enumerate and dump the data...but I still can't figure it out...because ObjectDumper uses reflection and other language constructs I don't get. It DOES enumerate both the parent and child data just like I want (when level=2).
PLEASE HELP...I'M stuck. Help!
Seth
in C# it would be:
var result = from s in _dataContent.SupplyModels where s.SupplyModelID==1 select s;
foreach(SupplyModel item in result)
{
// do stuff
foreach(SupplyModelChild child in item.SupplyModelChilds)
{
//do more stuff on the child
}
}
and a VB.NET version (from the Telerik code converter)
Dim result As var = From s In _dataContent.SupplyModels _
Where s.SupplyModelID = 1 _
Select s
For Each item As SupplyModel In result
' do stuff
'do more stuff on the child
For Each child As SupplyModelChild In item.SupplyModelChilds
Next
Next