Reference: https://github.com/zhengchuanpan/GMAN/blob/master/PeMS/utils.py
I am trying to do a time series forecasting model with time features encoded like the way they did in the file above. (Lines 70-76)
If time features are encoded this way, doesn't that introduce unnecessary ordinality for the model?
For example, if Sunday is encoded as 1 and a Thursday is encoded as 5 - will that make Thursday more important than Sunday while these features are nominal and not ordinal.
Is this understanding correct? If yes, could you help to understand why that decision was taken during model design? And, how to avoid this ordinality?
I have tried using One Hot Encoding (day of the week - 7, minute of the hour - 12 (5 min interval), hour of the day - 24) resulting in 43 features which is pushing the dataset with time series sequences to be 75 Gb and my computer is not liking that. Can someone point me to the right understanding/design here?
I am preparing time series data to build an RNN model (LSTM). The data is collected from sensors installed in a mechanical plant. Consider I have data for input and output temperature of a compressor along with the time stamps.
Like this there is data for around 20 parameters recorded along with their time stamps. Problem is there is a difference in the time stamps at which data is collected.
So how do I ideally match the time stamps to create a single dataframe with all the parameters and a single time stamp?
Since an RNN doesn't know anything about time deltas but only about time steps, you will need to quantify / interpolate your data.
Find the smallest time delta Δt in all of your series
Resample all of your 20 series to Δt/2* or smaller (Nyquist-Theorem)
* Actually you'd need to do a Fourier transform and then use twice the cutoff frequency as sampling rate. Δt/2 might IMHO be a good approximation.
So I have been trying to wrap my head around mpeg-ts timing, and the PCR (program clock reference). I understand that this is used for video/audio synchronisation at the decoder.
My basic understanding so far is that everything is driven by a 27 Mhz clock (oscillator). This clock loops at a rate of 27 Mhz, counting from 0 - 299 and keeps repeating itself. Each time this "rollover" from 299 back to 0 occurs, then a 33-bit PCR counter is incremented by 1. In effect, the 33-bit PCR counter is therefore itself running at a rate of 90 kHz. So another way of saying this is that the 27 Mhz clock is divided by 300, giving us a 90 kHz clock.
This 90 kHz clock is then used for the 33-bit PCR counter.
I understand that historically 90 kHz was chosen because mpeg-1 used a 90kHz timebase. [see source here]
Anyway... I have read that the PCR 33-bit count values range from 0x000000000 all the way through to 0x1FFFFFFFF. And according to this, it shows what these values mean in terms of time as we humans understand it (Hours, Mins, Secs, etc):
00:00:00.000 (0x000000000)
to
26:30:43.717 (0x1FFFFFFFF)
So ultimately, my question is relating to how do these hex codes get translated into those time stamps. What would the equations be if someone gave me a hex code, and now I need to reproduce the time stamp?
I would appreciate any help :)
==========
I am closer to an answer myself. Looking at the range from 0x000000000 to 0x1FFFFFFFF, this is basically 0 to 8589934591 in decimal. Since the PCR clock is 90Khz, to get the number of seconds it takes to go from 0 to 8589934591 we can do 8589934591/90000 which gives us 95443.71768 seconds.
Unless you are creating a strict bitrate encoder for broadcast over satellite or terrestrial radio, the PCR doesn't matter that much.
Scenario:
You are broadcasting to a wireless receiver with no return channel, The receiver has a clock running at what it thinks is 90000 ticks per second. Your encoder is also running at 90000 tickets per second. How can you be sure the receiver and the broadcaster have the EXACT same definition of a second? Maybe one side is running a little fast or slow. To keep the clocks in sync, the encoder sends the current time occasionally, This value is the PCR. For example, if you are broadcasting at 15,040,000 bits per second, the receiver receives a 188 byte packet every 0.0000125 seconds. Every now and then (100 ms) the encoder will insert its current time. The receiver can compare this time to its internal clock and determine if is running faster or slower than the broadcast encoder. To keep the strict 235,000 packets per second (15,040,000/(188*8) = 235,000) the encoder will insert null packets. On the internet, the null packets take bandwidth, and have no value, so they are eliminated. Hence the PCR has almost no value anymore because its time is no longer relative the the reception rate.
To answer your question. Set the 27hz value to zero, use a recent DTS minus a small static amount (like 100ms), for the 90khz value.
I want to make the route timing (arrival) updated with the current traffic in the city.So I fetched Multiple Time Matrix from Google API.Now while routing the Arrival time should be calculated from the Matrix fetched for that time Interval.
For Example : I have fetched 3 Time Matrix. 10:00 {N * N} 12:00 {N * N} and 14:00{N * N}
If previousCustomer's arrival time is 11:00 then for next customer the time should be fetched from Matrix 12:00{}.
So How could I do this using Optaplanner ?
Let me know if anyone one need more explanation. Thanks
Using that matrix from OptaPlanner isn't hard. Just refactor RoadLocation.getDistanceTo(RoadLocation) to something like RoadLocation.getDistanceTo(RoadLocation, LocalDateTime startingTime) and adjust your score rules accordingly.
The big problem is memory while scaling out. If you have 10k locations, then only 1 time interval already costs almost 2GB RAM memory...
How can you keep track of time in a simple embedded system, given that you need a fixed-point representation of the time in seconds, and that your time between ticks is not precisely expressable in that fixed-point format? How do you avoid cumulative errors in those circumstances.
This question is a reaction to this article on slashdot.
0.1 seconds cannot be neatly expressed as a binary fixed-point number, just as 1/3 cannot be neatly expressed as a decimal fixed-point number. Any binary fixed-point representation has a small error. For example, if there are 8 binary bits after the point (ie using an integer value scaled by 256), 0.1 times 256 is 25.6, which will be rounded to either 25 or 26, resulting in an error in the order of -2.3% or +1.6% respectively. Adding more binary bits after the point reduces the scale of this error, but cannot eliminate it.
With repeated addition, the error gradually accumulates.
How can this be avoided?
One approach is not to try to compute the time by repeated addition of this 0.1 seconds constant, but to keep a simple integer clock-tick count. This tick count can be converted to a fixed-point time in seconds as needed, usually using a multiplication followed by a division. Given sufficient bits in the intermediate representations, this approach allows for any rational scaling, and doesn't accumulate errors.
For example, if the current tick count is 1024, we can get the current time (in fixed point with 8 bits after the point) by multiplying that by 256, then dividing by 10 - or equivalently, by multiplying by 128 then dividing by 5. Either way, there is an error (the remainder in the division), but the error is bounded since the remainder is always less than 5. There is no cumulative error.
Another approach might be useful in contexts where integer multiplication and division is considered too costly (which should be getting pretty rare these days). It borrows an idea from Bresenhams line drawing algorithm. You keep the current time in fixed point (rather than a tick count), but you also keep an error term. When the error term grows too large, you apply a correction to the time value, thus preventing the error from accumulating.
In the 8-bits-after-the-point example, the representation of 0.1 seconds is 25 (256/10) with an error term (remainder) of 6. At each step, we add 6 to our error accumulator. Based on this so far, the first two steps are...
Clock Seconds Error
----- ------- -----
25 0.0977 6
50 0.1953 12
At the second step, the error value has overflowed - exceeded 10. Therefore, we increment the clock and subtract 10 from the error. This happens every time the error value reaches 10 or higher.
Therefore, the actual sequence is...
Clock Seconds Error Overflowed?
----- ------- ----- -----------
25 0.0977 6
51 0.1992 2 Yes
76 0.2969 8
102 0.3984 4 Yes
There is almost always an error (the clock is precisely correct only when the error value is zero), but the error is bounded by a small constant. There is no cumulative error in the clock value.
A hardware-only solution is to arrange for the hardware clock ticks to run very slightly fast - precisely fast enough to compensate for cumulative losses caused by the rounding-down of the repeatedly added tick-duration value. That is, adjust the hardware clock tick speed so that the fixed-point tick-duration value is precisely correct.
This only works if there is only one fixed-point format used for the clock.
Why not have 0.1 sec counter and every ten times increment your seconds counter, and wrap the 0.1 counter back to 0?
In this particular instance, I would have simply kept the time count in tenths of a seconds (or milliseconds, or whatever time scale is appropriate for the application). I do this all the time in small systems or control systems.
So a time value of 100 hours would be stored as 3_600_000 ticks - zero error (other than error that might be introduced by hardware).
The problems that are introduced by this simple technique are:
you need to account for the larger numbers. For example, you may have to use a 64-bit counter rather than a 32-bit counter
all your calculations need to be aware of the units used - this is the area that is most likely going to cause problems. I try to help with this problem by using time counters with a uniform unit. For example, this particular counter needs only 10 ticks per second, but another counter might need millisecond precision. In that case, I'd consider making both counters millisecond precision so they use the same units even though one doesn't really need that precision.
I've also had to play some other tricks this with timers that aren't 'regular'. For example, I worked on a device that required a data acquisition to occur 300 times a second. The hardware timer fired once a millisecond. There's no way to scale the millisecond timer to get exactly 1/300th of a second units. So We had to have logic that would perform the data acquisition on every 3, 3, and 4 ticks to keep the acquisition from drifting.
If you need to deal with hardware time error, then you need more than one time source and use them together to keep the overall time in sync. Depending on your needs this can be simple or pretty complex.
Something I've seen implemented in the past: the increment value can't be expressed precisely in the fixed-point format, but it can be expressed as a fraction. (This is similar to the "keep track of an error value" solution.)
Actually in this case the problem was slightly different, but conceptually similar—the problem wasn't a fixed-point representation as such, but deriving a timer from a clock source that wasn't a perfect multiple. We had a hardware clock that ticks at 32,768 Hz (common for a watch crystal based low-power timer). We wanted a millisecond timer from it.
The millisecond timer should increment every 32.768 hardware ticks. The first approximation is to increment every 33 hardware ticks, for a nominal 0.7% error. But, noting that 0.768 is 768/1000, or 96/125, you can do this:
Keep a variable for "fractional" value. Start it on 0.
wait for the hardware timer to count 32.
While true:
increment the millisecond timer.
Add 96 to the "fractional" value.
If the "fractional" value is >= 125, subtract 125 from it and wait for the hardware timer to count 33.
Otherwise (the "fractional" value is < 125), wait for the hardware timer to count 32.
There will be some short term "jitter" on the millisecond counter (32 vs 33 hardware ticks) but the long-term average will be 32.768 hardware ticks.