SQLAlchemy/SQLite - How to iterate a large dataset? - sql

Consider the following query
return (self.session
.query(ItemInfo.field1, ItemInfo.field2)
.group_by(ItemInfo.field1, ItemInfo.field2)
.yield_per(yield_per)
)
This worked pretty slow so I decided to get rid off the group_by phase, staying with the following simple query:
return (self.session
.query(ItemInfo.field1, ItemInfo.field2)
.yield_per(yield_per)
)
The problem is that I do get a batch of items at once and then I need to some time for the next one.
Why?

Related

PowerApps: UpdateIf only updates first 100 records (SQL-Datasource)

i am trying to update a SQL-table with a button containing the function UpdateIf in PowerApps.
The button contains the following code:
UpdateIf(
'[dbo].[INVINFO]',
IVI_ID_NR = Dropdown_Inventur.Selected.Result && IVI_MANDANT = Dropdown_Mandant.Selected.Result,
{IVI_ERFSET_MENGE: IVI_LAGER_IST - IVI_VORTRAG_MENGE}
)
My Problem is that it only updates the first 100 rows.
Based on the Dropdown-selection, 500-3000 rows should be updated.
I can't find anything about this 100-rows limitation in the Microsoft-documentation, so my question is: Am i doing anything wrong? Is there an alternative to UpdateIf? (i tried ForAll & Patch, but it was very slow, so i dumped it)
Thanks :)
I found a workaround for this. I just made a stored procedure in the sql server that does this calculation. It works much faster and updates all rows, not just 100.

How do I write a query using the VersionOne API to return all the hours (actuals) recorded under an Epic?

I need to write a query using the VerisonOne API to return all the time (effort) recorded against tasks under a specific Epic. My goal is to have the query be a one line statement I can enter into the address bar of my browser.
I've tried the following using the rest-1.v1 query:
http://<>/VersionOne/rest-1.v1/Data/Epic?sel=Epic.ID.Number,SubsAndDown:PrimaryWorkitem[AssetState=%27Closed%27].Actuals.Value.#Sum&where=Epic.ID.Number=%27E-06593%27
http://<>/VersionOne/rest-1.v1/Data/Story?sel=Story.ID.Number,Story.Name,SuperAndUp.Number,SuperAndUp.Actuals.#Sum&where=Story.SuperAndUp.ID.Number=%27E-06593%27
Below is the output from the first query above. (similar results from the second query)
Assets total="1" pageSize="2147483647" pageStart="0"
Asset href="/VersionOne/rest-1.v1/Data/Epic/1481442" id="Epic:1481442"
Attribute name="SubsAndDown:PrimaryWorkitem[AssetState='Closed'].Actuals.Value.#Sum"/
/Asset
/Assets
Actual results were no hours returned. I expected to have ~4,320 hours returned (the total under the Epic E-06593) after the ...#Sum"/
On your first query
http://<>/VersionOne/rest-1.v1/Data/Epic?sel=Epic.ID.Number,SubsAndDown:PrimaryWorkitem[AssetState=%27Closed%27].Actuals.Value.#Sum&where=Epic.ID.Number=%27E-06593%27,
If you change to AssetState!=Closed then you will get results. Beware there could be another AssetState that might mess with your total hours.
You might want filter down to AssetState= "64" or "Active".
See here for https://community.versionone.com/VersionOne_Connect/Developer_Library/Getting_Started/Platform_Concepts/Asset_State

vuefire dynamic query issues

I have an odd issue....
When using dynamic queries with VueFire, the query appears to run and returns data in to an array called customSiteSearch. I can console out the array and see the data there as expected.
The bit that I am struggling with is, when I evaluate the length of the customSiteSearch array after the query has run, it always returns 0 the first time. When I run the function again for a second time it appears to work. What could be the cause of this?
my code:
customSiteExists(customSite){
console.log('starting customSiteExists')
this.$bindAsArray('customSiteSearch', db.ref('slinksites/').orderByChild('customsite').equalTo(customSite) )
console.log('search result', this.customSiteSearch)
console.log(this.customSiteSearch.length)
},

Rails - get distinct events, sorted by the start date of associated event instances

I've spent several hours going through StackOverflow and playing around with this query, but still can't get it to work! Hopefully an expert here on SO can make the pain go away...
I have two models, Event and EventInstance. An Event has_many EventInstances.
What I want to do is easily get a list of Events (not EventInstances), where:
Events are distinct and not repeated
Events are sorted by the start_date of the nearest EventInstance
Event instances have the attribute :active => true
Only event instances that have a start date in the future are returned
I currently have the query
Event.joins(:event_instances).select('distinct events.*').where('event_instances.start_date >= ?', Time.now).where('event_instances.active = true')
This returns a list of events, but not sorted by date. Excellent - so I am almost there!
If I change the query to add this on the end:
.order('event_instances.start_date')
I get the error:
PG::InvalidColumnReference: ERROR: for SELECT DISTINCT, ORDER BY expressions must appear in select list
So I moved it to the select statement:
select('distinct event_instances.start_date, events.*')
Now I get
PG::UndefinedFunction: ERROR: function count(date, events) does not exist
I've tried moving methods around, using includes, everything but I still can't get it to work. Any help would be really appreciated! Thank you.
try changing
.order('event_instances.start_date')
to
.order(:event_instances.start_date)
or if you need descending order add the .reverse_order method to the end of the query
This is the exact query which worked for my models Post and PostComments on both MySQL and PostgreSQL:
Post.joins(:post_comments).select('distinct post_comments.body, post_comments.created_at').order('post_comments.created_at desc')
So for you, it's equivalent should work too. If it still doesn't then please update your post with the fields of your model.

phalcon querybuilder total_items always returns 1

I make a query via createBuilder() and when executing it (getQuery()->execute()->toArray())
I got 10946 elements. I want to paginate it, so I pass it to:
$paginator = new \Phalcon\Paginator\Adapter\QueryBuilder(array(
"builder" => $builder,
"limit" => $limit,
"page" => $current_page
));
$limit is 25 and $current_page is 1, but when doing:
$paginator->getPaginate();
$page->total_items;
returns 1.
Is that a bug or am I missing something?
UPD: it seems like when counting items it uses created sql with limit. There is no difference what limit is, limit divided by items per page always equals 1. I might be mistaken.
UPD2: Colleague helped me to figure this out, the bug was in the query phalcon produces: count() of the group by counts grouped elements. So a workaround looks like:
$dataCount = $builder->getQuery()->execute()->count();
$page->next = $page->current + 1;
$page->before = $page->current - 1 > 0 ? $page->current - 1 : 1;
$page->total_items = $dataCount;
$page->total_pages = ceil($dataCount / 100);
$page->last = $page->total_pages;
I know this isn't much of an answer but this is most likely to be a bug. Great guys at Phalcon took on a massive job that is too big to do it properly in their little free time and things like PHQL, Volt and other big but non-core components do not receive as much attention as we'd like. Also given that most time in the past 6 months was spent on v2 there are nearly 500 bugs about stuff like that and it's counting. I came across considerable issues in ORM, Volt, Validation and Session, which in the end made me stick to other not as cool but more proven solutions. When v2 comes out I'm sure all attention will on the bug list and testing, until then we are mostly on our own. Given that it's all C right now, only a few enthusiast get involved, with v2 this will also change.
If this is the only problem you are hitting, the best approach is to update your query to get the information you need yourself without getPaginate().