I've been building some simple linear regression models in Tensorflow.js with various types of data sets. However, I would now like to see what the relationship is between dates and price in my dataset.
In previous models, I normalise the price or other features so that the tensor is expressed as a vector between 0 and 1.
How would one do this dates, where the first date would have to be 0 and the last date in the range 1? Furthermore I would need to denormalise the Tensor afterwards.
I can convert the date into a unix timestamp using a library like date.fns... but I wondered if there might be a cleaner way to do this.
My normalise and denormalise functions :
function normalise (tensor) {
const min = tensor.min();
const max = tensor.max();
const normalisedTensor = tensor.sub(min).div(max.sub(min))
return {
tensor : normalisedTensor,
min,
max
}
}
function denormalise(tensor, min, max) {
const denormalisedTensor = tensor.mul(max.sub(min)).add(min);
return denormalisedTensor
}
The basic principle is that you need to convert the date formats (like 2021-03-21) to numbers, then you can use the normalise and denormalise functions that you already have. In your solution you have done that, however, you can write it in a simpler way. You can use date-fns, but the standard JavaScript Date object is also sufficient so I will use that.
Assuming your input is an array of date strings like 2021-05-01, then you can do this:
const dateStrings = ['2021-05-01', '2021-05-07', '2021-05-31'];
const timestamps = dateStrings.map(dateString => new Date(dateString).valueOf());
const unnormalisedTensor = tf.tensor1d(timestamps);
Then you can use the functions you have for normalisation (and denormalisation).
You can convert back from timestamps to date strings like this:
const timestamps = await unnormalisedTensor.array();
const dateStrings = timestamps.map(timestamp => new Date(timestamp).toISOString().substring(0,10));
I achieved this by using date.fns to convert the dates into timestamps and then normalising with the following function
const getTimestamps = (data) => {
data.forEach(record => {
let newDate = record.time
newDate = new Date(newDate);
const timestamp = getUnixTime(newDate);
console.log(newDate, timestamp)
record.timestamp = timestamp
}
Related
I am trying to run a query at the repository level of my nestjs application.
The date format of the column (runDateTime) in my DB is isoString. I want to find all the records that have runDateTime of today's date.
I have a function called Parsebackenddate that essentially converts iso string to YYYY-MM-DD. how do i use this function with "schedule.runDateTime" in order to compare both dates in the format YYYY-MM-DD?
Or is there an alternative?
if (getToday) {
const todayDate = dayjs().format('YYYY-MM-DD');
query.where('schedule.runDateTime = :todayDate', {
todayDate,
});
``
Much appreciated.
first, you have to get a date from the database and hold it in a class variable then compare.
something like that may this could help.
var _orderBean = new date_beans();
_orderBean.dueDate = _response[a].dueDate.toISOString().slice(0,10); //YYYY-MM-DD
then compare
var d = new Date(); //todaydate
if(d.toISOString().slice(0,10)>_orderBean.dueDate)
// TodayDate > DueDate
I have a date field that I want to ensure is in a valid format and if so is the user over 18. The format is YYYY-MM-DD.
Here is one of my validators - the one that is failing:
body('birthday', 'Date format should be: YYYY-MM-DD')
.isRFC3339()
.custom(date => {
const over18 = moment().diff(date, 'years') >= 18;
if(!over18) {
return Promise.reject('You must be 18 or over!');
}
}),
Currently what happens is if the date is not a RFC3339 date the validation chain continues. This is problematic because moment produces an error if I pass an ill formatted date.
How do I break the chain after the call to .isRFC3339() so that if the date is invalid the custom validator will not run? I couldn't find anything in the docs
You can use momentjs strict mode together with String + Format parsing using moment.ISO_8601 (or moment.HTML5_FMT.DATE) special formats.
Your code could be like the following:
body('birthday', 'Date format should be: YYYY-MM-DD')
// .isRFC3339() // no more needed
.custom(date => {
const mDate = moment(date, moment.ISO_8601, true);
const over18 = moment().diff(mDate, 'years') >= 18;
if(!mDate.isValid()) {
return Promise.reject('Date is not YYYY-MM-DD');
if(!over18) {
return Promise.reject('You must be 18 or over!');
}
}),
I'm using the following to get the end of day for a date coming from a date picker:
var date = DateTime.fromISO('2018-05-05').endOf('day');
What I want to end up with is
"2018-05-05T23:59:59+02:00"
however, I cannot get rid of the milliseconds:
console.log(date.toISO({suppressMilliseconds: true}));
// => outputs "2018-05-05T23:59:59.999+02:00"
Is there a more elegant way to do this besides simply setting the millisecond to 0:
date.c.millisecond = 0;
console.log(date.toISO({suppressMilliseconds: true}));
// => outputs "2018-05-05T23:59:59+02:00"
Right, suppressMilliseconds only applies if they're 0. (See here).
But there's a much easier way to round down the second:
DateTime.fromISO('2018-05-05')
.endOf('day')
.startOf('second')
.toISO({ suppressMilliseconds: true })
You should never mutate the Luxon object like in your workaround.
const { DateTime } = require("luxon");
let time = DateTime.now().set({milliseconds: 0});
time = time.toISO({suppressMilliseconds: true});
console.log(time);
It should help and solve the problem.
The way I did :
DateTime.now().toISO().replace(/\.\d{0,3}/, "");
Using RegExp to remove the "." and millis
I was having
2021-02-22T18:03:29.519Z
With the replace RegExp
2021-02-22T18:05:44Z
I am creating a dashboard in DC.js. One of the visualizations is a survival curve showing the percentage of survival on the y-axis and the time in weeks on the x-axis
Each record in the dataset contains a deathAfter column called recidiefNa. This shows the number of weeks after death occurred, and shows -99 for survival.
See sketches for example dataset and desired chart form:
I created this code to create the dimensions and groups and draw the desired chart.
var recDim = cf1.dimension(dc.pluck('recidiefNa'));//sets dimension
var recGroup = recDim.group().reduceCount();
var resDim = cf1.dimension(dc.pluck('residuNa'));
var resGroup = resDim.group().reduceCount();
var scChart = dc.compositeChart("#scStepChart");
scChart
.width(600)
.height(400)
.x(d3.scale.linear().domain([0,52]))
.y(d3.scale.linear().domain([0,100]))
.clipPadding(10)
.brushOn(false)
.xAxisLabel("tijd in weken")
.yAxisLabel("percentage vrij van residu/recidief")
.compose([
dc.lineChart(scChart)
.dimension(recDim)
.group(recGroup)
.interpolate("step-after")
.renderDataPoints(true)
.renderTitle(true)
.keyAccessor(function(d){return d.key;})
.valueAccessor(function(d){return (d.value/cf1.groupAll().reduceCount().value()*100);}),
dc.lineChart(scChart)
.dimension(resDim)
.group(resGroup)
.interpolate("step-after")
.renderDataPoints(true)
.colors(['orange'])
.renderTitle(true)
.keyAccessor(function(d){return d.key;})
.valueAccessor(function(d){return (d.value/cf1.groupAll().reduceCount().value()*100 );})
])
.xAxis().ticks(4);
scChart.render();
This gives the following result:
As you can see my first problem is that I need the line to extend until the y-axis showing x=0weeks and y=100% as the first datapoint.
So that's question number one: is there a way to get that line to look more like my sketch(starting on the y-axis at 100%?
My second and bigger problem is that it is showing the inverse of the percentage I need (eg. 38 instead of 62). This is because of the way the data is structured (which is somehting i rather not change)
First I tried changing the valueaccessor to 100-*calculated number. Which is obviously the normal way to solve this issue. However my result was this:
As you can see now the survival curve is a positive incline which is never possible in a survival curve. This is my second question. Any ideas how to fix this?
Ah, it wasn't clear from the particular example that each data point should be based on the last, but your comment makes that clear. It sounds like what you are looking for is a kind of cumulative sum - in your case, a cumulative subtraction.
There is an entry in the FAQ for this.
Adapting that code to your use case:
function accumulate_subtract_from_100_group(source_group) {
return {
all:function () {
var cumulate = 100;
return source_group.all().map(function(d) {
cumulate -= d.value;
return {key:d.key, value:cumulate};
});
}
};
}
Use it like this:
var decayRecGroup = accumulate_subtract_from_100_group(recGroup)
// ...
dc.lineChart(scChart)
// ...
.group(decayRecGroup)
and similarly for the resGroup
While we're at it, we can concatenate the data to the initial point, to answer your first question:
function accumulate_subtract_from_100_and_prepend_start_point_group(source_group) {
return {
all:function () {
var cumulate = 100;
return [{key: 0, value: cumulate}]
.concat(source_group.all().map(function(d) {
cumulate -= d.value;
return {key:d.key, value:cumulate};
}));
}
};
}
(ridiculous function name for exposition only!)
EDIT: here is #Erik's final adapted answer with the percentage conversion built in, and a couple of performance improvements:
function fakeGrouper(source_group) {
var groupAll = cf1.groupAll().reduceCount();
return {
all:function () {
var cumulate = 100;
var total = groupAll.value();
return [{key: 0, value: cumulate}]
.concat(source_group.all().map(function(d) {
if(d.key > 0) {
cumulate -= (d.value/total*100).toFixed(0);
}
return {key:d.key, value:cumulate};
}));
}
};
}
I'm using DataTables 1.10.5. When I'm trying to sort on dates using the recommended moment.js (as per http://datatables.net/blog/2014-12-18), thinks work fine:
http://jsfiddle.net/9gohzd9t/1/
However, when I add a link (a href) to that date, it sorts on the link instead of the date:
http://jsfiddle.net/dnsL2oc4/1/
Any idea on how to properly fix this without too much hacking around?
The problem lies in the unshift method of datetime-moment.js. Moment tries to convert 12-01-2001 to a valid date in the given "DD-MM-YYYY"-Format, which it can't obviously. So you have to strip the html away from the date, probably with a function like this:
function strip(html) {
var tmp = document.createElement("DIV");
tmp.innerHTML = html;
return tmp.textContent || tmp.innerText || "";
}
And then strip the string in the unshift method (Replace datetime-moment.js with the code below):
$.fn.dataTable.moment = function (format, locale) {
var types = $.fn.dataTable.ext.type;
// Add type detection
types.detect.unshift(function (d) {
return moment(strip(d), format, locale, true).isValid() ?
'moment-' + format :
null;
});
// Add sorting method - use an integer for the sorting
types.order['moment-' + format + '-pre'] = function (d) {
return moment(strip(d), format, locale, true).unix();
};
};