I'm using angularjs-nvd3-directives to create stacked area chart.
Now my problem is I'm polling hundreds of data from server and I need to display on the the N latest data. How will I do that?
Here's the HTML file
<div ng-controller="GraphController as viewAll">
<nvd3-stacked-area-chart data="viewAll.data"
id="graph" showXAxis="true" showYAxis="true"
showLegend="true" interactive="true"
tooltips="true" forcex="[xFunction()]">
<svg></svg> </nvd3-stacked-area-chart> </div>
Try to define a second array that holds only the last N values and pass that as the 'data' parameter instead of the full data array.
$scope.viewAll.visData = $scope.viewAll.data.slice(Math.max($scope.viewAll.data.length - N, 1))
In your html:
<nvd3-stacked-area-chart
data="viewAll.visData"
...
>
Related
I have a column named Con_TYPE in which there are multiple types of connections such as fiberoptic, satellite, 3g etc.
And I want to sort them only into 2 rows:
fiberoptic
5
others
115
Can anybody help me?
Thanks in advance
You can use Calculated dimension or Mapping load
Lets imagine that the data, in its raw form, looks like this:
dimension: Con_TYPE
measure: Sum(value)
Calculated dimension
You can add expressions inside the dimension. If we have a simple if statement as an expression then the result is:
dimension: =if(Con_TYPE = 'fiberoptic', Con_TYPE, 'other')
measure: Sum(Value)
Mapping load
Mapping load is a script function so we'll have to change the script a bit:
// Define the mapping. In our case we want to map only one value:
// fiberoptic -> fiberoptic
// we just want "fiberoptic" to be shown the same "fiberoptic"
TypeMapping:
Mapping
Load * inline [
Old, New
fiberoptic, fiberoptic
];
RawData:
Load
Con_TYPE,
value,
// --> this is where the mapping is applied
// using the TypeMapping, defined above we are mapping the values
// in Con_TYPE field. The third parameter specifies what value
// should be given if the field value is not found in the
// mapping table. In our case we'll chose "Other"
ApplyMap('TypeMapping', Con_TYPE, 'Other') as Con_TYPE_Mapped
;
Load * inline [
Con_TYPE , value
fiberoptic, 10
satellite , 1
3g , 7
];
// No need to drop "TypeMapping" table since its defined with the
// "Mapping" prefix and Qlik will auto drop it at the end of the script
And we can use the new field Con_TYPE_Mapped in the ui. And the result is:
dimension: Con_TYPE_Mapped
measure: Sum(Value)
Pros/Cons
calculated dimension
+ easy to use
+ only UI change
- leads to performance issues on mid/large datasets
- have to be defined (copy/paste) per table/chart. Which might lead to complications if have to be changed across the whole app (it have to be changed in each object where defined)
mapping load
+ no performance issues (just another field)
+ the mapping table can be defined inline or loaded from an external source (excel, csv, db etc)
+ the new field can be used across the whole app and changing the values in the script will not require table/chart change
- requires reload if the mapping is changed
P.S. In both cases selecting Other in the tables will correctly filter the values and will show data only for 3g and satellite
I have been trying to display a chart in splunk. I uploaded my json data through Splunk HTTP Forwarder and running the query:
After I uploaded the json data, I have got fields such as
"message":{"acplbuild":"ACPL 1.20.1","coresyncbuild":"4.3.10.25","testregion":"EU_Stage","client":"EU_Mac","date":"2019-08-27","iteration":"20","localCreateTime":"6.672","createSyncTime":"135.768","createSearchTime":"0.679","filetype":"CPSD","filesize":"690_MB","filename":"690MB_NissPoetry.cpsd","operation":"upload","upload_DcxTime":"133.196","upload_manifest_time":"133.141","upload_journal_time":"1.753","upload_coresync_time":"135.225","upload_total_time":142.44},"severity":"info"}
I am trying to run the following query
index="coresync-ue1" host="acpsync_allacpl_7" message.testregion=EU_STAGE message.client=EU_Mac message.operation="upload" |eval roundVal = round(message.upload_total_time, 2) | chart median(roundVal) by message.acplbuild
I am getting no values. It should display rounded off median values as a chart. Can someone point me if I am doing anything wrong here.
I used the same data as specified by you and I faced an issue while rounding off the upload_total_time value. So, I first converted it to number, and then the Splunk search query worked.
Input Data Set
{"message":{"acplbuild":"ACPL 1.20.1","coresyncbuild":"4.3.10.25","testregion":"EU_Stage","client":"EU_Mac","date":"2019-08-27","iteration":"20","localCreateTime":"6.672","createSyncTime":"135.768","createSearchTime":"0.679","filetype":"CPSD","filesize":"690_MB","filename":"690MB_NissPoetry.cpsd","operation":"upload","upload_DcxTime":"133.196","upload_manifest_time":"133.141","upload_journal_time":"1.753","upload_coresync_time":"135.225","upload_total_time":142.44},"severity":"info"}
{ "message":{"acplbuild":"ACPL 1.20.2","coresyncbuild":"4.3.10.25","testregion":"EU_Stage","client":"EU_Mac","date":"2019-08-27","iteration":"20","localCreateTime":"6.672","createSyncTime":"135.768","createSearchTime":"0.679","filetype":"CPSD","filesize":"690_MB","filename":"690MB_NissPoetry.cpsd","operation":"upload","upload_DcxTime":"133.196","upload_manifest_time":"133.141","upload_journal_time":"1.753","upload_coresync_time":"135.225","upload_total_time":152.44123},"severity":"info"}
{ "message":{"acplbuild":"ACPL 1.20.3","coresyncbuild":"4.3.10.25","testregion":"EU_Stage","client":"EU_Mac","date":"2019-08-27","iteration":"20","localCreateTime":"6.672","createSyncTime":"135.768","createSearchTime":"0.679","filetype":"CPSD","filesize":"690_MB","filename":"690MB_NissPoetry.cpsd","operation":"upload","upload_DcxTime":"133.196","upload_manifest_time":"133.141","upload_journal_time":"1.753","upload_coresync_time":"135.225","upload_total_time":160.456},"severity":"info"}
Splunk Search Query
source="sample.json" index="splunk_answers" sourcetype="_json"
| convert num(message.upload_total_time) as total_upld_time
| eval roundVal = round(total_upld_time,2)
| chart median(roundVal) by message.acplbuild
Statistics View
Visualization View
I'm trying to split a large csv data using a condition. To automate this process, I'm pulling a list of unique conditions from a column in the data set and wanting to use this list within a loop to specify condition and also rename the export file.
I've converted the array of values into a list and have tried fitting my function into a loop, however, I believe syntax is the main error.
# df1718 is my df
# znlist is my list of values (e.g. 0 1 2 3 4)
# serial is specified at the top e.g. '4'
for x in znlist:
dftemps = df1718[(df1718.varname == 'RoomTemperature') & (df1718.zone == x)]
dftemps.to_csv('E:\\path\\test%d_zone(x).csv', serial)
So in theory, I would like each iteration to export the data relevant to the next zone in the list and the export file to be named test33_zone0.csv (for example). Thanks for any help!
EDIT:
The error I am getting is: "delimiter" must be string, not int
So if the error is in saving the file try this
dftemps.to_csv('E:\\path\\test{}_zone{}.csv'.format(str(serial),str(x)))
This page link is for a Socrata API Docs page for a public NYC dataset:
On the top right of the page, there is a colophon/cartouche that lists some information about the dataset:
"About This Dataset"
Its last line enables a copy of the code needed to "Embed Theses Docs".
I have tried it. It works but the entire page is embedded.
I would like to embed this colophon every time I access the dataset so that I show this information in my report.
I know where this information is located in the html tree:
<body class="dev foundry 200" ...>
...
<div class="container-fluid content">
...
<div id="foundry-docs">
...
<div class="pull-right sidebar metadata">
<div class="panel panel-info about">
...
< div class="panel-body">
<ul>
<li>...</li> ==> items (9) needed
So, I tried to scrape that information using beautifulsoup:
from bs4 import BeautifulSoup
import requests
data_api_page = 'https://dev.socrata.com/foundry/data.cityofnewyork.us/qiz3-axqb'
page = requests.get( data_api_page )
print(page.status_code)
print(page.headers['content-type'])
soup = BeautifulSoup(page.text, 'html.parser')
all_divs = soup.find(name='div', attrs={'class':'panel panel-info about'})
for tag in all_divs.children:
print(tag)
Nothing is returned (even with find_all): what am I doing wrong?
Thanks for your help!
PS: The other reason, besides annotating a report with this info, is that I want to retrieve the dataset row count before accessing the dataset in order to bypass the 1000-records limit of the Socrata API (v2.1 has the same limit as the prior version) and retrieve the entire dataset.
A couple of things might be useful and doesn't include scraping. There is a metadata API end-point where you can retrieve a lot of the descriptions of the data. Here is the metadata endpoint for that NYC data: http://data.cityofnewyork.us/api/views/metadata/v1/qiz3-axqb.
Unfortunately, the metadata API does not include row count. For that, might be simpler to assemble a SoQL query that simply returns the count, such as: https://data.cityofnewyork.us/resource/qiz3-axqb.json?$select=count(date)
Yes Tom: the metadata API has a ton of info, except the row count...
This is what I am doing to retrieve the entire dataset via the authenticated client:
# First call to obtain the dataset size: count of rows:
LIM = client.get(MVD_collisions, select='COUNT(*) as tot')
LIM = int(LIM[0]['tot'])
# Retrieval call:
results = client.get(MVD_collisions, limit=LIM)
One day, I'll figure out how to scrape the row count from the summary data at the bottom of DATAPAGE = 'https://data.cityofnewyork.us/Public-Safety/NYPD-Motor-Vehicle-Collisions/h9gi-nx95'...
I have this better solution, which does not involve sending a SQL count over all rows:
def get_rows_from_metadata(metadata):
rows = -1 #something went wrong if output
for c in metadata['columns']:
if c['name'] == 'UNIQUE KEY':
try:
rows = int(c['cachedContents']['not_null'])
except:
rows = 0
break
return rows
dataset_rows = get_rows_from_metadata(metadata)
I receive data in the form
id1|attribute1a,attribute1b|attribute2a|attribute3a,attribute3b,attribute3c....
id2||attribute2b,attribute2c|..
I'm trying to merge it all into a form where I just have a bag of tuples of an id field followed by a tuple containing a list of all my other fields merged together.
(id1,(attribute1a,attribute1b,attribute2a,attribute3a,attribute3b,attribute3c...))
(id2,(attribute2b,attribute2c...))
Currently I fetch it like
my_data = load '$input' USING PigStorage(|) as
(id:chararray, attribute1:chararray, attribute2:chararray)...
then I've tried all combinations of FLATTEN, TOKENIZE, GENERATE, TOTUPLE, BagConcat, etc. to massage it into the form I want, but I'm new to pig and just can't figure it out. Can anyone help? Any open source UDF libraries are fair game.
Load each line as an entire string, and then use the features of the built-in STRPLIT UDF to achieve the desired result. This relies on there being no tabs in your list of attributes, and assumes that | and , are not to be treated any differently in separating out the different attributes. Also, I modified your input a little bit to show more edge cases.
input.txt:
id1|attribute1a,attribute1b|attribute2a|,|attribute3a,attribute3b,attribute3c
id2||attribute2b,attribute2c,|attribute4a|,attribute5a
test.pig:
my_data = LOAD '$input' AS (str:chararray);
split1 = FOREACH my_data GENERATE FLATTEN(STRSPLIT(str, '\\|', 2)) AS (id:chararray, attr:chararray);
split2 = FOREACH split1 GENERATE id, STRSPLIT(attr, '[,|]') AS attributes;
DUMP split2;
Output of pig -x local -p input=input.txt test.pig:
(id1,(attribute1a,attribute1b,attribute2a,,,attribute3a,attribute3b,attribute3c))
(id2,(,attribute2b,attribute2c,,attribute4a,,attribute5a))