How do i add a calculated field (a field which does calculation of two or more data fields) in dimple.js?
eg. I have two fields
1. "Sales Value"
2. "Sales Volume"
Now i have to calculate a field ASP = Sales Value /Sales Volume.
I'm afraid dimple doesn't have a built in way to handle this. I assume dimple is aggregating the data for you - hence the difficulty. But here you have no option but to pre-aggregate to the level of a data point and add the calculated field yourself. For example if your data had Brand, SKU and Channel but your chart was at the Brand, Channel level you would need to pre-process the data like this:
// var chartData is going to be the aggregated level you use for your chart.
// var added is a dictionary of aggregated data returning a row index
// for each Brand/Channel combination.
var chartData = [],
added = {};
// Aggregate to the Brand/Channel level
data.forEach(function (d) {
var key = d["Brand"] + "|" + d["Channel"],
i = added[key];
// Check the output index
if (i !== undefined) {
// Brand/Channel have been added already so add the measures
chartData[i]["Sales Value"] += parseFloat(d["Sales Value"]);
chartData[i]["Sales Volume"] += parseFloat(d["Sales Volume"]);
} else {
// Get the index for the row we are about to add
added[key] = chartData.length;
// Insert a new output row for the Brand/Channel
chartData.push({
"Brand": d["Brand"],
"Channel": d["Channel"],
"Sales Value": parseFloat(d["Sales Value"]) || 0,
"Sales Volume": parseFloat(d["Sales Volume"]) || 0
});
}
});
// Calculate ASP
chartData.forEach(function (d) {
d["ASP"] = d["Sales Value"] / d["Sales Volume"];
});
// Draw the chart using chartData instead of data
...
Related
I am fairly new to SSIS and need a little help getting started. I have several reports that come out of our mainframe. The reports are not in a columnar format. The date record is at the top then there might be some initial data then there might be a little more. So I need to read in each line look to see what the text reads and figure out if I need the data or move to the next row.
This is a VERY rough example of what the report I want to import into a SQL table.
DATE: 01/08/2020 FACILITY NAME PAGE1
REVENUE USAGE FOR ACCOUNTING PERIOD 02
----TOTAL---- ----TOTAL---- ----OTHER---- ----INSURANCE---- ----INSURANCE2----
SERVICE CODE - 123456789 DESCRIPTION: WIDGETS
CURR 2,077
IP 0.0000 3 2,345 0.00
143
OP 0.0000 2 1,231 0.00
YTD 5
IP 0.0000
76
OP 0.0000
etc . . . .. .
SERVICE CODE
After the SERVICE CODE the data will start to repeat like it is above. This is the basic idea of a report.
I want to get the Date then the Service Code, Description, Current IP Volume, Current IP Dollar, Current OP Volume, Current OP Dollar, YTD IP Volume, YTD IP Dollar, YTD OP Volume, YTD OP Dollar . . then repeat.
Just to clarify, I am not asking anyone to do this for me. I want to learn how to do this. I have looked on how to do this but every example I have looked at talks about doing this with a CSV, tab, or Excel file. i do not have that type of file so I was asking what I need to look at. I currently use Monarch to format the file, but again I want to learn more about SSIS and this is a perfect way to learn. Asking the vendor to redo the report is not an option plus I want to learn how to do this. Thank you I just wanted to get that out there.
Any help would be greatly appreciated.
Rodger
As stated in comments, you could do this using a script task. The basics steps are:
Define a DataTable to store your data.
Use a StreamReader to read your report.
Process this using a combination of conditionals, String Methods, and parsing to extract the relevant fields from the relevant line:
Write the DataTable to the database using SqlBulkCopy
The following would go inside your Main method in your script task:
//Define a table to store your data
var table = new DataTable
{
Columns =
{
{ "ServiceCode", typeof(string) },
{ "Description", typeof(string) },
{ "CurrentIPVolume", typeof(int) },
{ "CurrentIPDollar,", typeof(decimal) },
{ "CurrentOPVolume", typeof(int) },
{ "CurrentOPDollar", typeof(decimal) },
{ "YTDIPVolume", typeof(int) },
{ "YTDIPDollar,", typeof(decimal) },
{ "YTDOPVolume", typeof(int) },
{ "YTDOPDollar", typeof(decimal) }
}
};
var filePath = #"Your File Path";
using (var reader = new StreamReader(filePath))
{
string line = null;
DataRow row = null;
// As YTD and Curr are identical, we will need a flag later to mark our position within the record
bool ytdFlag= false;
//Loop through every line in the file
while ((line = reader.ReadLine()) != null)
{
//if the line is blank, move on to the next
if (string.IsNullOrWhiteSpace(line)
continue;
// If the line starts with service code, then it marks the start of a new record
if (line.StartsWith("SERVICE CODE"))
{
//If the current value for row is not null then this is
//not the first record, so we need to add the previous
//record to the tale before continuing
if (row != null)
{
table.Rows.Add(row);
ytdFlag= false; // New record, reset YTD flag
}
row = table.NewRow();
//Split the line now based on known values:
var tokens = line.Split(new string[] { "SERVICE CODE - ", "DESCRIPTION: "}, StringSplitOptions.None);
row[0] = tokens[0];
row[1] = tokens[1];
}
if (line.StartsWith("CURR"))
{
//Process the row --> "CURR 2,077"
//Not sure what 2,077 is, but this will parse it
int i = 0;
if (int.TryParse(line.Substring(4).Trim().Replace(",", ""), out i))
{
//Do something with your int
Console.WriteLine(i);
}
}
if (line.StartsWith(" IP"))
{
//Start at after IP then split the line into the 4 numbers
var tokens = line.Substring(3).Split(new [] { " "}, StringSplitOptions.RemoveEmptyEntries);
//If we have gone past the CURR record, then at to YTD Columns
if (ytdFlag)
{
row[6] = int.Parse(tokens[1]);
row[7] = decimal.Parse(tokens[1]);
}
//Otherwise we are still in the CURR section:
else
{
row[2] = int.Parse(tokens[1]);
row[3] = decimal.Parse(tokens[1]);
}
}
if (line.StartsWith(" OP"))
{
//Start at after OP then split the line into the 4 numbers
var tokens = line.Substring(3).Split(new [] { " "}, StringSplitOptions.RemoveEmptyEntries);
//If we have gone past the CURR record, then at to YTD Columns
if (ytdFlag)
{
row[8] = int.Parse(tokens[1]);
row[9] = decimal.Parse(tokens[1]);
}
//Otherwise we are still in the CURR section:
else
{
row[4] = int.Parse(tokens[1]);
row[5] = decimal.Parse(tokens[1]);
}
//After we have processed an OP record, we must set the YTD Flag to true.
//Doesn't matter if it is the YTD OP record, since the flag will be reset
//By the next line that starts with SERVICE CODE anyway
ytdFlag= true;
}
}
}
//Now that we have processed the file, we can write the data to a database
using (var sqlBulkCopy = new SqlBulkCopy("Your Connection String"))
{
sqlBulkCopy.DestinationTableName = "dbo.YourTable";
//If necessary add column mappings, but if your DataTable matches your database table
//then this is not required
sqlBulkCopy.WriteToServer(table);
}
This is a very quick example, far from the finished article, and I have done little or no testing, but it should give you the gist of how it could be done, and get you started on one possible solution.
It can definitely be cleaned up and refactored, but I have tried to make it as clear as possible what is going on, rather than trying to write the most efficient code ever. It should also (hopefully) demonstrate what a monumental pain this is to do, and very minor report changes things like an extra space be "OP" will break the whole thing.
So again, I would re-iterate, if you can get the data in a standard flat file format, with one line per record, you should. I do however appreciate that sometimes these things are out of your control, and I have had to write incredibly ugly import routines like this in the past, so I feel your pain if you can't get the data in a consumable format.
I am trying to add columnSummary to my table using Handsontable. But it seems that the function does not fire. The stretchH value gets set and is set properly. But it does not react to the columnSummary option:
this.$refs.hot.hotInstance.updateSettings({stretchH: 'all',columnSummary: [
{
destinationRow: 0,
destinationColumn: 2,
reversedRowCoords: true,
type: 'custom',
customFunction: function(endpoint) {
console.log("TEST");
}
}]
}, false);
I have also tried with type:'sum' without any luck.
Thanks for all help and guidance!
columnSummary cannot be changed with updateSettings: GH #3597
You can set columnSummary settings at the initialization of Handsontable.
One workaround would be to somehow manage your own column summary, since Handsontable one could give you some headeache. So you may try to add one additional row to put your arithmetic in, but it is messy (it needs fixed rows number and does not work with filtering and sorting operations. Still, it could work well under some circumstances.
In my humble opinion though, a summary column has to be fully functionnal. We then need to set our summary row out of the table data. What comes to mind is to take the above mentioned additional row and take it away from the table data "area" but it would force us to make that out of the table row always looks like it still was in the table.
So I thought that instead of having a new line we could just have to add our column summary within column header:
Here is a working JSFiddle example.
Once the Handsontable table is rendered, we need to iterate through the columns and set our column summary right in the table cell HTML content:
for(var i=0;i<tableConfig.columns.length;i++) {
var columnHeader = document.querySelectorAll('.ht_clone_top th')[i];
if(columnHeader) { // Just to be sure column header exists
var summaryColumnHeader = document.createElement('div');
summaryColumnHeader.className = 'custom-column-summary';
columnHeader.appendChild( summaryColumnHeader );
}
}
Now that our placeholders are set, we have to update them with some arithmetic results:
var printedData = hotInstance.getData();
for(var i=0;i<tableConfig.columns.length;i++) {
var summaryColumnHeader = document.querySelectorAll('.ht_clone_top th')[i].querySelector('.custom-column-summary'); // Get back our column summary for each column
if(summaryColumnHeader) {
var res = 0;
printedData.forEach(function(row) { res += row[i] }); // Count all data that are stored under that column
summaryColumnHeader.innerText = '= '+ res;
}
}
This piece of code function may be called anytime it should be:
var hotInstance = new Handsontable(/* ... */);
setMySummaryHeaderCalc(); // When Handsontable table is printed
Handsontable.hooks.add('afterFilter', function(conditionsStack) { // When Handsontable table is filtered
setMySummaryHeaderCalc();
}, hotInstance);
Feel free to comment, I could improve my answer.
I am creating a dashboard in DC.js. One of the visualizations is a survival curve showing the percentage of survival on the y-axis and the time in weeks on the x-axis
Each record in the dataset contains a deathAfter column called recidiefNa. This shows the number of weeks after death occurred, and shows -99 for survival.
See sketches for example dataset and desired chart form:
I created this code to create the dimensions and groups and draw the desired chart.
var recDim = cf1.dimension(dc.pluck('recidiefNa'));//sets dimension
var recGroup = recDim.group().reduceCount();
var resDim = cf1.dimension(dc.pluck('residuNa'));
var resGroup = resDim.group().reduceCount();
var scChart = dc.compositeChart("#scStepChart");
scChart
.width(600)
.height(400)
.x(d3.scale.linear().domain([0,52]))
.y(d3.scale.linear().domain([0,100]))
.clipPadding(10)
.brushOn(false)
.xAxisLabel("tijd in weken")
.yAxisLabel("percentage vrij van residu/recidief")
.compose([
dc.lineChart(scChart)
.dimension(recDim)
.group(recGroup)
.interpolate("step-after")
.renderDataPoints(true)
.renderTitle(true)
.keyAccessor(function(d){return d.key;})
.valueAccessor(function(d){return (d.value/cf1.groupAll().reduceCount().value()*100);}),
dc.lineChart(scChart)
.dimension(resDim)
.group(resGroup)
.interpolate("step-after")
.renderDataPoints(true)
.colors(['orange'])
.renderTitle(true)
.keyAccessor(function(d){return d.key;})
.valueAccessor(function(d){return (d.value/cf1.groupAll().reduceCount().value()*100 );})
])
.xAxis().ticks(4);
scChart.render();
This gives the following result:
As you can see my first problem is that I need the line to extend until the y-axis showing x=0weeks and y=100% as the first datapoint.
So that's question number one: is there a way to get that line to look more like my sketch(starting on the y-axis at 100%?
My second and bigger problem is that it is showing the inverse of the percentage I need (eg. 38 instead of 62). This is because of the way the data is structured (which is somehting i rather not change)
First I tried changing the valueaccessor to 100-*calculated number. Which is obviously the normal way to solve this issue. However my result was this:
As you can see now the survival curve is a positive incline which is never possible in a survival curve. This is my second question. Any ideas how to fix this?
Ah, it wasn't clear from the particular example that each data point should be based on the last, but your comment makes that clear. It sounds like what you are looking for is a kind of cumulative sum - in your case, a cumulative subtraction.
There is an entry in the FAQ for this.
Adapting that code to your use case:
function accumulate_subtract_from_100_group(source_group) {
return {
all:function () {
var cumulate = 100;
return source_group.all().map(function(d) {
cumulate -= d.value;
return {key:d.key, value:cumulate};
});
}
};
}
Use it like this:
var decayRecGroup = accumulate_subtract_from_100_group(recGroup)
// ...
dc.lineChart(scChart)
// ...
.group(decayRecGroup)
and similarly for the resGroup
While we're at it, we can concatenate the data to the initial point, to answer your first question:
function accumulate_subtract_from_100_and_prepend_start_point_group(source_group) {
return {
all:function () {
var cumulate = 100;
return [{key: 0, value: cumulate}]
.concat(source_group.all().map(function(d) {
cumulate -= d.value;
return {key:d.key, value:cumulate};
}));
}
};
}
(ridiculous function name for exposition only!)
EDIT: here is #Erik's final adapted answer with the percentage conversion built in, and a couple of performance improvements:
function fakeGrouper(source_group) {
var groupAll = cf1.groupAll().reduceCount();
return {
all:function () {
var cumulate = 100;
var total = groupAll.value();
return [{key: 0, value: cumulate}]
.concat(source_group.all().map(function(d) {
if(d.key > 0) {
cumulate -= (d.value/total*100).toFixed(0);
}
return {key:d.key, value:cumulate};
}));
}
};
}
So I want to allow the user to conditionally turn columns on/off in a Cardboard app I built. I have two problems.
I tried using the 'columns' attribute in the config but I can't seem to find a default value for it that would allow ALL columns to display(All check boxes checked) based on the attribute, ie. the default behavior if I don't include 'columns' in the config object at all (tried null, [] but that displays NO columns).
So that gets to my second problem, if there is no default value is there a simple way to only change that value in the config object or do I have to encapsulate the entire variable in 'if-else' statements?
Finally if I have to manually build the string I need to parse the values of an existing custom attribute (a drop list) we have on the portfolio object. I can't seem to get the rally.forEach loop syntax right. Does someone have a simple example?
Thanks
Dax - Autodesk
I found a example in the online SDK from Rally that I could modify to answer the second part (This assumes a custom attribute on Portfolio item called "ADSK Kanban State" and will output values to console) :
var showAttributeValues = function(results) {
for (var property in results) {
for (var i=0 ; i < results[property].length ; i++) {
console.log("Attribute Value : " + results[property][i]);
}
}
};
var queryConfig = [];
queryConfig[0] = {
type: 'Portfolio Item',
key : 'eKanbanState',
attribute: 'ADSK Kanban State'
};
rallyDataSource.findAll(queryConfig, showAttributeValues);
rally.forEach loops over each key in the first argument and will execute the function passed as the second argument each time.
It will work with either objects or arrays.
For an array:
var array = [1];
rally.forEach(array, function(value, i) {
//value = 1
//i = 0
});
For an object:
var obj = {
foo: 'bar'
};
rally.forEach(obj, function(value, key) {
//value = 'bar'
//key = 'foo'
});
I think that the code to dynamically build a config using the "results" collection created by your query above and passed to your sample showAttributeValues callback, is going to look a lot like the example of dynamically building a set of Table columns as shown in:
Rally App SDK: Is there a way to have variable columns for table?
I'm envisioning something like the following:
// Dynamically build column config array for cardboard config
var columnsArray = new Array();
for (var property in results) {
for (var i=0 ; i < results[property].length ; i++) {
columnsArray.push("'" + results[property][i] + "'");
}
}
var cardboardConfig = {
{
attribute: 'eKanbanState',
columns: columnsArray,
// .. rest of config here
}
// .. (re)-construct cardboard...
Sounds like you're building a neat board. You'll have to provide the board with the list of columns to show each time (destroying the old board and creating a new one).
Example config:
{
attribute: 'ScheduleState'
columns: [
'In-Progress',
'Completed'
]
}
I'm creating a Dojo line chart from a dojo.data.ItemFileReadStore using a dojox.charting.DataSeries. I'm using the third parameter (value) of the constructor of DataSeries to specify a method which will generate the points on the chart. e.g.
function formatLineGraphItem(store,item)
{
var o = {
x: graphIndex++,
y: store.getValue(item, "fileSize"),
};
return o;
}
The graphIndex is an integer which is incremented for every fileSize value. This gives me a line chart with the fileSize shown against a numeric count. This works fine.
What I'd like is to be able to specify the x axis label to use instead of the value of graphIndex i.e. the under lying data will still be 1,2,3,4 but the label will show text (in this case the time at which the file size was captured).
I can do this by passing in an array of labels into the x asis when I call chart.addAxis() but this requires me to know the the values before I iterate through the data. e.g.
var dataSeriesConfig = {query: {id: "*"}};
var xAxisLabels = [{text:"2011-11-20",value:1},{text:"2011-11-21",value:2},{text:"2011-11-22",value:3}];
var chart1 = new dojox.charting.Chart("chart1");
chart1.addPlot("default", {type: "Lines", tension: "4"});
chart1.addAxis("x", {labels: xAxisLabels});
chart1.addAxis("y", {vertical: true});
chart1.addSeries("Values", new dojox.charting.DataSeries(dataStore, dataSeriesConfig, formatLineGraphItem));
chart1.render();
The xAxisLabels array can be created by preparsing the dataSeries but it's not a very nice work around.
Does anyone have any ideas how the formatLineGraphItem method could be extended to provide the x axis labels. Or does anyone have any documentation on what values the object o can contain?
Thanks in advance!
This will take a unix timestamp, multiply the value by 1000 (so that it has microseconds for JavaScript, and then pass the value to dojo date to format it).
You shouldn't have any problems editing this to the format you need.
You provided examples that your dates are like "1", "2", "3", which is clearly wrong. Those aren't dates.. so this is the best you can do unless you edit your question.
chart1.addAxis("x",{
labelFunc: function(n){
if(isNaN(dojo.number.parse(n)) || dojo.number.parse(n) % 1 != 0){
return " ";
}
else {
// I am assuming that your timestamp needs to be multiplied by 1000.
var date = new Date(dojo.number.parse(n) * 1000);
return dojo.date.locale.format(date, {
selector: "date",
datePattern: "dd MMMM",
locale: "en"
});
}
},
maxLabelSize: 100
}