Input mix for Keras: Timeseries and Features - input

Question

I have several time series Features (ECG, HRV and breathing) and separate features made from those time series (e.g. SDNN, RMSSD,...).
I follow Francois Chollet with the naming. For a 3D timeseries input tensor they use [samples,timestep,features]
The time series have 15000 values(samples) per timestep [[15000x1],[15000x1],...] while the separate features have 1 value (sample) per timestep.Those extra features with length [1] are different for each timestep. [[0.3],[0.35],[0.34],...].
ECG, HRV, F1, F2, ...
-------------------------------------------------------------
Sequence 1 |
Step 1 | [[15000x1],[1000x1],[1x1],[1x1],...]
Step 2 | [[15000x1],[1000x1],[1x1],[1x1],...]
Step 3 | [[15000x1],[1000x1],[1x1],[1x1],...]
Sequence 2 |
Step 1 | [[15000x1],[1000x1],[1x1],[1x1],...]
Step 2 | [[15000x1],[1000x1],[1x1],[1x1],...]
Step 3 | [[15000x1],[1000x1],[1x1],[1x1],...]
How would you best approach to learn on all those inputs with Keras?
Just zero pad the separate features from 1 to 15000 and add them to the time series
Pad the separate features with themselves (using the one value repeatedly)
As the data is normalized between 0 and 1, using a value way outside of that range of the extra features to pad. E.g. 1000
Have only the time series as 3D tensor input and the separate features as an additional input (as extra layers) and merge them into the learner (multi-input)
Extra question. How does zero padding influence the learner due to the "false" additional information? Especially for the 1 to 15000 part for the separate feature from above. Another example: the HRV and breathing signals are shorter than the ECG due to different sample frequency. Here I would use rather an interpolation instead of zero padding. Would you agree, or does zero padding not influence the learner?
Thanks

Answer 1

Assumption 1
Due to the ambiguity, I'm assuming this (please comment if not and I'll change it)
I'm calling ECG, HRV, etc. features that vary per step
Your feature with the highest frequency has 15000 steps, while the other features have less steps
You have a separate feature that is not sequential and has no steps. (I'll call it the separate feature in this answer)
Extra question:
Yes! Interpolate the less frequent features and make an input tensor like:
(numberOfSequences_maybePatient, 15000 steps, features_ECG_HRV_etc)
You need to keep a correlation between the features on when they happen, and this is made by synchronizing the steps.
Will zero padding influence the results?
Yes it will, unless you use "masking" (a masking layer). But this will only make sense for handling samples (different sequence or patients) with different lengths, not features with different length/sample rate.
Example, the following case would work well with zero padding and masking:
sequence 1: length 100 (all features included, ECG, HRV, etc.)
sequence 2: length 200 (all features included, ECG, HRV, etc.)
How to deal with the separate feature?
There are a number of possible ways. One of the most simple ones, and probably very effective, is to make it a constant sequence with all the 15000 steps. This approach does not require thinking about how the feature relates to the rest of the data, and leaves the task to the model
Suppose the separate feature value is 2 for the first sequence and 4 for the second sequence, make then this data array:
ECG, HRV, separate
--------------------------------------------------------
| [
sequence 1: | [
step 1 | [ecg1, hrv1, 2],
step 2 | [ecg2, hrv2, 2],
step 3 | [ecg3, hrv3, 2]
| ]
|
sequence 2: | [
step 1 | [ecg4, hrv4, 4],
step 2 | [ecg5, hrv5, 4],
step 3 | [ecg6, hrv6, 4]
| ]
| ]
You can also input is as an additional input in the model:
regularSequences = Input((15000,features))
separateFeature = Input((1,)) #assuming 1 value per sequence
And then you decide if you want to sum it somewhere, multiply it somewhere, etc. This approach might be more effective than the other if you have an idea of what this feature means and how it relates to the rest of the data to select the best operations and where.
Assumption 2
Taking this description from your updated answer:
ECG, HRV, F1, F2, ...
-------------------------------------------------------------
Sequence 1 |
Step 1 | [[15000x1],[1000x1],[1x1],[1x1],...]
Step 2 | [[15000x1],[1000x1],[1x1],[1x1],...]
Step 3 | [[15000x1],[1000x1],[1x1],[1x1],...]
Sequence 2 |
Step 1 | [[15000x1],[1000x1],[1x1],[1x1],...]
Step 2 | [[15000x1],[1000x1],[1x1],[1x1],...]
Step 3 | [[15000x1],[1000x1],[1x1],[1x1],...]
Then:
You have 15000 features for a single time step in the ECG. (Are you sure this is not a sequence of 15000 steps?)
You have 1000 features for a single time step in HRV. (Are you sure this is not a sequence of 1000 steps?)
You have several other individual features per time step.
Well, organizing this data is quite easy (but mind the questions I asked above), just pack all features together in each time step:
The shape of your input data will be: (sequences, steps, 16002)
ECG, HRV, F1, F2, ...
-------------------------------------------------------------
[
Sequence 1 | [
Step 1 | [ecg1,ecg2,...,ecg15000,hrv1,hrv2,...hrv1000,F1,F2,...]
Step 2 | [ecg1,ecg2,...,ecg15000,hrv1,hrv2,...hrv1000,F1,F2,...]
Step 3 | [ecg1,ecg2,...,ecg15000,hrv1,hrv2,...hrv1000,F1,F2,...]
]
Sequence 2 | [
Step 1 | [ecg1,ecg2,...,ecg15000,hrv1,hrv2,...hrv1000,F1,F2,...]
Step 2 | [ecg1,ecg2,...,ecg15000,hrv1,hrv2,...hrv1000,F1,F2,...]
Step 3 | [ecg1,ecg2,...,ecg15000,hrv1,hrv2,...hrv1000,F1,F2,...]
]