I have 2 Tensors of same length, data and groupIds. I want to split data into several groups by the corresponding values in groupId. For example,
const data = tf.tensor([1,2,3,4,5]);
const groupIds = tf.tensor([0,1,1,0,0]);
// expected result: [tf.tensor([1,4,5]), tf.tensor([2,3])]
In Tensorflow there is tf.dynamic_partition which does exactly that. Tensorflow.js doesn't seem to have a similar method. I also looked into mask or filtering as work-arounds, but they don't exist either. Does anyone have an idea how to implement this?
To partition your tensor, you can first iterate over your ids tensor to get the number of subtensor to create and the index of the elements it should contain. This information can be stored in an object where the key is the number of the partition in the ids array and the value is an array of indexes.
const data = tf.tensor([6,2,8,4,5]);
const ids = tf.tensor([0,1,1,0,2]);
const data2 = tf.tensor([[6,2],[8,4], [5, 4], [6, 5]]);
const ids2 = tf.tensor([0,1,1,0]);
const filterT = (t, p) => {
t.print()
p.print()
const l = p.unstack().reduce((a, b, i) => {
const v = b.dataSync()[0]
if (Object.keys(a).includes(v.toString())) {
a[v].push(i)
} else {
a[v] = [i]
}
return a
}, {})
const r = Object.keys(l).map(k => t.gather(tf.tensor1d(l[k], 'int32')))
r.forEach(e => e.print())
}
filterT(data, ids)
filterT(data2, ids2)
<html>
<head>
<!-- Load TensorFlow.js -->
<script src="https://cdnjs.cloudflare.com/ajax/libs/tensorflow/0.12.4/tf.js"> </script>
</head>
<body>
</body>
</html>
Related
I struggle accessing the row-elements of a Frame.
One idea I have is to filter the dataframe down to a row, convert it to a vec or something similar and access the elements this way ?!
In Panadas I used to just use ".at / .loc / .iloc / etc."; with Polars in Rust I have no clue.
Any suggestions on what the proper way to do this is ?
Thanks to #isaactfa ... he got me onto the right track. I ended up getting the row not with "get_row" but rather with "get" ... this is probably due to my little RUST understanding (my 2nd week).
Here is a working code sample:
use polars::export::arrow::temporal_conversions::date32_to_date;
use polars::prelude::*;
fn main() -> Result<()> {
let days = df!(
"date_string" => &["1900-01-01", "1900-01-02", "1900-01-03", "1900-01-04", "1900-01-05",
"1900-01-06", "1900-01-07", "1900-01-09", "1900-01-10"])?;
let options = StrpTimeOptions {
date_dtype: DataType::Date, // the result column-datatype
fmt: Some("%Y-%m-%d".into()), // the source format of the date-string
strict: false,
exact: true,
};
// convert date_string into dtype(date) and put into new column "date_type"
// we convert the days DataFrame to a LazyFrame ...
// because in my real-world example I am getting a LazyFrame
let mut new_days_lf = days.lazy().with_column(
col("date_string")
.alias("date_type")
.str()
.strptime(options),
);
// Getting the weekday as a number:
// This is what I wanted to do ... but I get a string result .. need u32
// let o = GetOutput::from_type(DataType::Date);
// new_days_lf = new_days_lf.with_column(
// col("date_type")
// .alias("weekday_number")
// .map(|x| Ok(x.strftime("%w").unwrap()), o.clone()),
// );
// This is the convoluted workaround for getting the weekday as a number
let o = GetOutput::from_type(DataType::Date);
new_days_lf = new_days_lf.with_column(col("date_type").alias("weekday_number").map(
|x| {
Ok(x.date()
.unwrap()
.clone()
.into_iter()
.map(|opt_name: Option<i32>| {
opt_name.map(|datum: i32| {
// println!("{:?}", datum);
date32_to_date(datum)
.format("%w")
.to_string()
.parse::<u32>()
.unwrap()
})
})
.collect::<UInt32Chunked>()
.into_series())
},
o,
));
new_days_lf = new_days_lf.with_column(
col("weekday_number")
.shift_and_fill(-1, 9999)
.alias("next_weekday_number"),
);
// now we convert the LazyFrame into a normal DataFrame for further processing:
let mut new_days_df = new_days_lf.collect()?;
// convert the column to a series
// to get a column by name we need to collect the LazyFrame into a normal DataFrame
let col1 = new_days_df.column("weekday_number")?;
// convert the column to a series
let col2 = new_days_df.column("next_weekday_number")?;
// now I can use series-arithmetics
let diff = col2 - col1;
// create a bool column based on "element == 2"
// add bool column to DataFrame
new_days_df.replace_or_add("weekday diff eq(2)", diff.equal(2)?.into_series());
// could not figure out how to filter the eager frame ...
let result = new_days_df
.lazy()
.filter(col("weekday diff eq(2)").eq(true))
.collect()
.unwrap();
// could not figure out how to access ROW elements
// thus I used "get" instead af of "get_row"
// getting the date where diff is == 2 (true)
let filtered_row = result.get(0).unwrap();
// within the filtered_row get element with an index
let date = filtered_row.get(0).unwrap();
println!("\n{:?}", date);
Ok(())
}
What I need is to sort data I get from an API into different arrays but add '0' value where there is no value at 1 type but is at the other type/s. Now is this possible with array.filter since its faster then a bunch of for and if loops ?
So let's say I get following data from SQL to the API:
Day Type Amount
-----------------------
12.1.2022 1 11
12.1.2022 2 4
13.1.2022 1 5
14.1.2022 2 9
16.1.2022 2 30
If I run this code :
this.data = result.Data;
let date = [];
const data = { 'dataType1': [], 'dataType2': [], 'dataType3': [], 'dataType4': [] }
/*only writing example for 2 types since for 4 it would be too long but i desire
answer that works for any amount of types or for 4 types */
this.data.forEach(x => {
var lastAddress = date[date.length - 1]
if (x.type == 1) {dataType1.push(x.Amount) }
if (x.type == 2) {dataType2.push(x.Amount) }}
lastAddress != x.Day ? date.push(x.Day) : '';
The array I get for type1 is [11,5]
and for type2 I get [4,9,30].
And for dates i get all the unique dates.
But the data I would like is: [11,5,0,0] and [4,0,9,30]
The size of array also has to match the size of Day array at the end.
which would be unique dates.. in this case:
[12.1.2022, 13.1.2022, 14.1.2022, 16.1.2022]
I have already tried to solve this with some for, if and while loops but it gets way too messy, so I'm looking for an alternative.
Also i have 4 types but for reference i only wrote sample for 2.
you can
first get the uniq values
loop over the data to create an array of object with
{date:string,values:number[]}
//create a function:
matrix(data:any[])
{
const uniqTypes=data.reduce((a,b)=>a.indexOf(b.type)>=0?a:
[...a,b.type],[])
const result=[]
this.data.forEach((x:any)=>{
let index=result.findIndex(r=>r.date==x.date)
if (index<0)
{
result.push({date:x.date,values:[...uniqTypes].fill(0)})
index=result.length-1;
}
result[index].values[uniqTypes.indexOf(x.type)]=x.amount
})
return result
}
//and use like
result=this.matrix(this.data);
NOTE: You can create the uniqType outside the function as variable and pass as argument to the function
stackblitz
const type1 = [];
const type2 = [];
data.forEach(item=>{
if(item.type==1){
type1.push(item.amount)
type2.push(0)
}else{
type1.push(0)
type2.push(item.amount)
}
})
console.log(type1)
console.log(type2)
It works fine with a pair of array. I don't need to use Ramda in this example.
const addChild1 = (x , y) => `${x}.addChild(${y}); `
const result1 = addChild(["containerA"], ["a1","a2", "a3"])
console.log(result1) //containerA.addChild(a1,a2,a3)
I couldn't make it work with 2d matrix of strings. I used R.map, but I get 'undefined' in the second argument.
const addChild2 = R.map ((x , y) => `${x}.addChild(${y}); `)
const result2 = addChild2(["containerA", "containerB", "containerC"], [["a1","a2", "a3"], ["b1","b2", "b3"], ["c1","c2", "c3"]])
console.log(result2) //["containerA.addChild(undefined); ","containerB.addChild(undefined); ","containerC.addChild(undefined); "]
How can I avoid the 'undefined'? Desirable output is the below :
["containerA.addChild("a1","a2", "a3"); ","containerB.addChild("b1","b2", "b3"); ","containerC.addChild("c1","c2", "c3");"]
map takes string as second parameters, so in this case, only ["containerA", "containerB", "containerC"] got in the loop
You should use zipWith in this case instead of map
const addChild2 = R.zipWith((x, y) => `${x}.addChild(${y}); `)
const result2 = addChild2(
["containerA", "containerB", "containerC"],
[
["a1", "a2", "a3"],
["b1", "b2", "b3"],
["c1", "c2", "c3"],
]
)
console.log(result2)
<script src="//cdnjs.cloudflare.com/ajax/libs/ramda/0.25.0/ramda.min.js"></script>
I am trying to build a Google Apps Script web app that will pull data from a Google sheet and display it in rows and columns in an HTML page in the browser.
By following the samples etc I wrote this code that WORKS!
function doGet(){
return HtmlService
.createTemplateFromFile('Report3')
.evaluate();
}
function getData(){
var spreadsheetId = '1Z6G2PTJviFKbXg9lWfesFUrvc3NSIAC7jGvhKiDGdcY';
var rangeName = 'Payments!A:D';
var values = Sheets
.Spreadsheets
.Values
.get(spreadsheetId,rangeName)
.values;
return values;
}
the data lying in columns A,B,C,D is getting pulled and being displayed correctly through the following HTML template
<? var data = getData(); ?>
<table>
<? for (var i = 0; i < data.length; i++) { ?>
<tr>
<? for (var j = 0; j < data[i].length; j++) { ?>
<td><?= data[i][j] ?></td>
<? } ?>
</tr>
<? } ?>
</table>
Instead of getting all the rows and all the columns from A,B,C,D I would like to run an SQL Query to retrieve some of the columns with a WHERE clause like SQL. I understand that the =QUERY() function that works in the spreadsheet does not work inside the GAS. So my next attempt was to retrieve SOME of the rows by using a getBatch method .. and this is where I get ERRORs
in this case, i want to exclude column C and get only A,B and D,E
the code that throws an error is as follows :
function getData2(){
var spreadsheetId = '1Z6G2PTJviFKbXg9lWfesFUrvc3NSIAC7jGvhKiDGdcY';
/* var rangeName1 = 'Payments!D'; */
/* var rangeName2 = 'Payments!A'; */
var values = Sheets
.Spreadsheets
.Values
.batchGet(spreadsheetId,{ranges: ['Payments!D:E', 'Payments!A:B']})
.values;
return values;
}
In the corresponding HTML template, all that changes is getData is replaced with getData2
<? var data = getData2(); ?>
with this code, I get the following error :
TypeError: Cannot read property "length" from undefined. (line 6, file
"Code", project "Report003")
Now I have two questions :
What is wrong in my code and how can i fix it?
Is it possible to use SQLite to simplify the process of extracting the desired rows and columns
I have seen this question but I am not able to understand the answer adequately
I finally understood what this solution was and modified it as given below. Now we can use any SQL that is supported by the QUERY() function.
function mostSQL(){
var spreadsheetId = '1Z6G2PTJviFKbXg9lWfesFUrvc3NSIAC7jGvhKiDGdcY';
var targetRange = 'Payments!A:G';
var SQL = 'select A, G where G >= 700 and G <= 800'
var Query = '=QUERY('+targetRange+',\"'+SQL+'\")'
var currentDoc = SpreadsheetApp.openById(spreadsheetId)
var tempSheet = currentDoc.insertSheet();
var pushQuery = tempSheet.getRange(1, 1).setFormula(Query);
var pullResult = tempSheet.getDataRange().getValues();
currentDoc.deleteSheet(tempSheet);
return pullResult;
}
You can use Google Visualization API Query Language to perform data manipulations with the query to the data source. The syntax of the query language is similar to SQL
code.gs
function doGet() {
// SpreadsheetApp.openById("SSID"); // To define the oAUTH Scope - https://www.googleapis.com/auth/spreadsheets
var output = HtmlService.createTemplateFromFile('index');
output.token = ScriptApp.getOAuthToken();
return output
.evaluate()
.setTitle('SQL Query');
}
index.html
<!DOCTYPE html>
<html>
<head>
<base target="_top">
</head>
<body>
<div id="dataTable"><h4>Loading...</h4></div>
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.0/jquery.min.js"></script>
<script src="https://www.gstatic.com/charts/loader.js"></script>
<script>
google.load('visualization', '1.0', {packages: ['corechart','table']});
google.setOnLoadCallback(loadEditor);
function loadEditor() {
var queryString = encodeURIComponent("SELECT A,B,D,E where A!= 'JACK'");
var SSID = "ADD YOUR SPREADSHEET"
var SHEET_NAME = "SHEET NAME"
var query = new google.visualization.Query('https://spreadsheets.google.com/tq?key='+SSID+'&sheet='+SHEET_NAME+'&oauth_token=<?=ScriptApp.getOAuthToken()?>&headers=1&tq=' + queryString);
query.send(handleSampleDataQueryResponse);
}
function handleSampleDataQueryResponse(response) {
console.log(response)
var data = response.getDataTable();
console.log(data);
var chartTbl = new google.visualization.Table(document.getElementById('dataTable'));
chartTbl.draw(data);
}
</script>
</body>
</html>
I love lodash and I use it in many projects.
I always have this problem and I can't find a solution for it.
I would like to do something with lodash, save a temporary state, and continue.
For example lets assume I have moviesList which is a list of movies with their id and profit, and I want to
index movies by their id and keep the result in moviesById
filter profitable movies and keep the result in overMillionProfit
And I want to do so without breaking the chain.
Something like (wishful thinking):
var moviesList = [{id : 1}, {id:2}, ...];
var moviesById = {};
var overMillionProfit = [];
_(moviesList)
.keyBy('id')
.do((unwrappedValue)=> moviesById = unwrappedValue)
.values()
.filter( (m) => m.profit > 1000000)
.do((unwrappedValue) => overMillionProfit = unwrappedValue)
Currently, I find it necessary to break the chain like so:
var moviesList = [{id : 1}, {id:2}, ...];
var moviesById = _.keyBy(moviesList,'id');
var overMillionProfit = _.filter(moviesList, ...);
Perhaps for this scenario it is better, but I want to be able to do it like the above in some cases.
Is there a way to do this?
You can use _.tap() for that:
var moviesList = [{id : 1}, {id:2}, {id:3}];
var moviesById;
var overMillionProfit;
var result = _(moviesList)
.keyBy('id')
.tap((unwrappedValue)=> moviesById = unwrappedValue)
.values()
.filter( (m) => m.id > 1)
.tap((unwrappedValue) => overMillionProfit = unwrappedValue)
.value();
console.log('moviesById\n', moviesById);
console.log('overMillionProfit\n', overMillionProfit);
console.log('result\n', result);
<script src="https://cdnjs.cloudflare.com/ajax/libs/lodash.js/4.17.4/lodash.min.js"></script>