I am utilizing the Rally 2.0p4 API and attempting to aggregate the data to get a list of iterations with sums of the story points per iteration. The only way I have found at present to do this is just query the HierarchicalRequirement model and loop all the data and populate it to an array. This seems less then ideal, is there not a way to just get back totals from the call from the server.
If you are wanting this data summarized by Iteration and/or Release, check out the:
IterationCumulativeFlowData
ReleaseCumulativeFlowData
Objects in the Webservices API documentation:
https://rally1.rallydev.com/slm/doc/webservice/
These objects will provide a daily summary of:
CardCount (# Stories/Defects)
TaskEstimateTotal
CardEstimateTotal
CardToDoTotal
By State, within each Iteration or Release as specified by OID.
In case anyone else comes looking, this is one way that api call can appear:
https://rally1.rallydev.com/slm/webservice/1.30/iterationcumulativeflowdata.js?query=(%20IterationObjectID%20=%20%2211203475854%22%20)&fetch=CardCount,CardToDoTotal,CardEstimateTotal,IterationObjectID
That call syntax is delicate (requires a space before the closing paren on 'query', for example).
Related
I have created a simple pipeline that operates as such:
Generates an access token via an Azure Function. No problem.
Uses a Lookup activity to create a table to iterate through the rows (4 columns by 0.5M rows). No problem.
For Each activity (sequential off, batch-size = 10):
(within For Each): Set some variables for checking important values.
(within For Each): Pass values through web activity to return a json.
(within For Each): Copy Data activity mapping parts of the json to the sink-dataset (postgres).
Problem: The pipeline slows to a crawl after approximately 1000 entries/inserts.
I was looking at this documentation regarding the limits of ADF.
ForEach items: 100,000
ForEach parallelism: 20
I would expect that this falls within in those limits unless I'm misunderstanding it.
I also cloned the pipeline and tried it by offsetting the query in one, and it tops out at 2018 entries.
Anyone with more experience be able to give me some idea of what is going on here?
As a suggestion, whenever I have to fiddle with variables inside a foreach, I made a new pipeline for the foreach process, and call it from within the foreach. That way I make sure that the variables get their own context for each iteration of the foreach.
Have you already checked that the bottleneck is not at the source or sink? If the database or web service is under some stress, then going sequential may help if your scenario allows that.
Hope this helped!
I have been running into a consistent problem using the LBAPI which I feel is probably a common use case given its purpose. I am generating a chart which uses LBAPI snapshots of a group of Portfolio Items to calculate the chart series. I know the minimum and maximum snapshot dates, and need to query once a day in between these two dates. There are two main ways I have found to accomplish this, both of which are not ideal:
Use the _ValidFrom and _ValidTo filter properties to limit the results to snapshots within the selected timeframe. This is bad because it will also load snapshots which I don't particularly care about. For instance if a PI is revised several times throughout the day, I'm really only concerned with the last valid snapshot of that day. Because some of the PIs I'm looking for have been revised several thousand times, this method requires pulling mostly data I'm not interested in, which results in unnecessarily long load times.
Use the __At filter property and send a separate request for each query date. This method is not ideal because some charts would require several hundred requests, with many requests returning redundant results. For example if a PI wasn't modified for several days, each request within that time frame would return a separate instance of the same snapshot.
My workaround for this was to simulate the effect of __At, but with several filters per request. To do this, I added this filter to my request:
Rally.data.lookback.QueryFilter.or(_.map(queryDates, function(queryDate) {
return Rally.data.lookback.QueryFilter.and([{
property : '_ValidFrom',
operator : '<=',
value : queryDate
},{
property : '_ValidTo',
operator : '>=',
value : queryDate
}]);
}))
But of course, a new problem arises... Adding this filter results in much too large of a request to be sent via the LBAPI, unless querying for less than ~20 dates. Is there a way I can send larger filters to the LBAPI? Or will I need to break theis up into several requests, which only makes this solution slightly better than the second of the latter.
Any help would be much appreciated. Thanks!
Conner, my recommendation is to download all of the snapshots even the ones you don't want and marshal them on the client side. There is functionality in the Lumenize library that's bundled with the App SDK that makes this relatively easy and the TimeSeriesCalculator will also accomplish this for you with even more features like aggregating the data into series.
I understand how to get cumulative flow data on releases with the ReleaseCumulativeFlowData object - however this requires a ReleaseObjectID. I am looking for a way to get the same data for all the items that are not scheduled in a release, and it does not appear that I can query for where the ReleaseObjectID is null.
Is there any way using CumulativeFlow data to get the number of story points for unscheduled stories on a given day- or is my best bet to either parse the revision history logs using the 1.x API, or use the Lookback API?
Basically, what I am trying to get to is to be able to represent how the total scope of a project has changed over time, including items that are scheduled, as well as items that are estimated in the backlog but are not yet scheduled. - As far as I can tell, there is not an out-of-the-box way to get this information (without revision logs or diving into learning the Lookback API right now), but I am crossing my fingers that I am wrong.
I recommend learning the Lookback API, as this is exactly the sort of question it was designed to answer.
You can find the docs here: https://rally1.rallydev.com/analytics/doc/
For example, if you say:
find:{
_ProjectHierarchy:279050021,
_TypeHierarchy: "HierarchicalRequirement",
Children: null,
ScheduleState:{$lt:"In-Progress"},
__At:"current"
},
fields:["ObjectID", "Name", "PlanEstimate"]
You're looking for snapshots for items under the project with OID 279050021, that are Stories (HierarchicalRequirements), with no children (so are leaf stories), that are in a schedule state earlier than "In-Progress" and we should look for snapshots that are valid today ("current"), but you could have put any ISO 8601 date in there as a string. The fields parameter then specifies which fields of the snapshots to return. While you're learning what's there, it can be helpful to use fields=true and use this Chrome plugin for pretty-printing the JSON response: https://chrome.google.com/webstore/detail/jsonview-and-jsonlint-for/mfjgkleajnieiaonjglfmanlmibchpam However, you should specify the exact list of fields you want when going to production, since fields=true is limited to 200 results.
As a full URL this looks like:
https://rally1.rallydev.com/analytics/v2.0/service/rally/workspace/41529001/artifact/snapshot/query.js?find={_ProjectHierarchy:279050021, _TypeHierarchy: "HierarchicalRequirement", Children: null, ScheduleState:{$lt:"In-Progress"}, __At:"current"}&fields=["ObjectID", "Name", "PlanEstimate"]
But make sure to swap in your own workspace OID (for 41529001) and project OID (for 279050021) or the above URL won't work for you.
I'm optimizing the memory load (~2GB, offline accounting and analysis routine) of this line:
l2 = Photograph.objects.filter(**(movie.get_selectors())).values()
Is there a way to convince django to skip certain columns when fetching values()?
Specifically, the routine obtains all rows of the table matching certain criteria (db is optimized and performs it very quickly), but it is a bit too much for python to handle - there is a long string referenced in each row, storing the urls for thumbnails.
I only really need three fields from each row, but, if all the fields are included, it suddenly consumes about 5kB/row which sadly pushes the RAM to the limit.
The values(*fields) function allows you to specify which fields you want.
Check out the QuerySet method, only. When you declare that you only want certain fields to be loaded immediately, the QuerySet manager will not pull in the other fields in your object, till you try to access them.
If you have to deal with ForeignKeys, that must also be pre-fetched, then also check out select_related
The two links above to the Django documentation have good examples, that should clarify their use.
Take a look at Django Debug Toolbar it comes with a debugsqlshell management command that allows you to see the SQL queries being generated, along with the time taken, as you play around with your models on a django/python shell.
I am making an application that represents a cell phone bill using Core Data. I have three entities: Bill, Line, and Calls. Bills can have many lines, and lines can have many calls. All of this is set up with relationships. Right now, I have a table view that displays all of the bills. When you double click on a bill, a sheet comes down with a popup box that lists all of the lines on the bill. Below the popup box is a box that has many labels that display various information about that line. Below that information I want to list the top 5 numbers called by that line in that month. Lines has a to-many relationship with Calls, which has two fields, number and minutes. I have all of the calls for the selected line loaded into an NSArrayController with a sort descriptor that properly arranges the values. How do I populate 5 labels with the top 5 values of this array controller?
EDIT: The array of calls is already unique, when I gather the data, I combine all the individual calls into total minutes per number for each month. I just need to sort and display the first 5 records of this combined array.
I may be wrong (and really hope I am), but it looks like you'll need to use brute force on this one. There are no set / array operators that can help, nor does NSPredicate appear to help.
I think this is actually a bit tricky and it looks like you'll have to do some coding. The Core Data Programming Guide says:
If you execute a fetch directly, you
should typically not add
Objective-C-based predicates or sort
descriptors to the fetch request.
Instead you should apply these to the
results of the fetch. If you use an
array controller, you may need to
subclass NSArrayController so you can
have it not pass the sort descriptors
to the persistent store and instead do
the sorting after your data has been
fetched.
I think this applies to your case because it's important to consider whether sorting or filtering takes place first in a fetch request (when the fetch requests predicate and sort descriptors are set). This is because you'll be tempted to use the #distinctUnionOfObjects set/array operator. If the list is collapsed to uniques before sorting, it won't help. If it's applied after sorting, you can just set the fetch request's limit to 5 and there're your results.
Given the documentation, I don't know that this is how it will work. Also, in this case, it might be easier to avoid NSArrayController for this particular problem and just use NSTableViewDataSource protocol, but that's beyond the scope of this Q&A.
So, here's one way to do it:
Create a predicate to filter for the
selected bill's line items.*
Create a sort descriptor to sort the
line items by their telephone number
(which are hopefully in a
standardized format internally, else
trouble awaits) via #"call.number" in your case.
Create a fetch request for the line
item entity, with the predicate and
sort descriptors then execute it**.
With those sorted results, it would be nice if you could collapse and "unique" them easily, and again, you'll be tempted to use #distinctUnionOfObjects. Unfortunately, set/array operators won't be any help here (you can't use them directly on NSArray/NSMutableArray or NSSet/NSMutableSet instances). Brute force it is, then.
I'd create a topFive array and loop through the results, adding the number to topFive if it's not there already, until topFive has 5 items or until I'm out of results.
Displaying it in your UI (using Bindings or not) is, as I said, beyond the scope of this Q&A, so I'll leave it there. I'd LOVE to hear if there's a better way to do this - it's definitely one of those "looks like it should be easy but it's not" kind of things. :-)
*You could also use #unionOfObjects in your key path during the fetch itself to get the numbers of the calls of the line items of the selected bill, which would probably be more efficient a fetch, but I'm getting tired of typing, and you get the idea. ;-)
**In practice I'd probably limit the fetch request to something reasonable - some bills (especially for businesses and teenagers) can be quite large.