NumPy Tensordot axes=2 - numpy

I know there are many questions about tensordot, and I've skimmed some of the 15 page mini-book answers that people I'm sure spent hours making, but I haven't found an explanation of what axes=2 does.
This made me think that np.tensordot(b,c,axes=2) == np.sum(b * c), but as an array:
b = np.array([[1,10],[100,1000]])
c = np.array([[2,3],[5,7]])
np.tensordot(b,c,axes=2)
Out: array(7532)
But then this failed:
a = np.arange(30).reshape((2,3,5))
np.tensordot(a,a,axes=2)
If anyone can provide a short, concise explanation of np.tensordot(x,y,axes=2), and only axes=2, then I would gladly accept it.

In [70]: a = np.arange(24).reshape(2,3,4)
In [71]: np.tensordot(a,a,axes=2)
Traceback (most recent call last):
File "<ipython-input-71-dbe04e46db70>", line 1, in <module>
np.tensordot(a,a,axes=2)
File "<__array_function__ internals>", line 5, in tensordot
File "/usr/local/lib/python3.8/dist-packages/numpy/core/numeric.py", line 1116, in tensordot
raise ValueError("shape-mismatch for sum")
ValueError: shape-mismatch for sum
In my previous post I deduced that axis=2 translates to axes=([-2,-1],[0,1])
How does numpy.tensordot function works step-by-step?
In [72]: np.tensordot(a,a,axes=([-2,-1],[0,1]))
Traceback (most recent call last):
File "<ipython-input-72-efdbfe6ff0d3>", line 1, in <module>
np.tensordot(a,a,axes=([-2,-1],[0,1]))
File "<__array_function__ internals>", line 5, in tensordot
File "/usr/local/lib/python3.8/dist-packages/numpy/core/numeric.py", line 1116, in tensordot
raise ValueError("shape-mismatch for sum")
ValueError: shape-mismatch for sum
So that's trying to do a double axis reduction on the last 2 dimensions of the first a, and the first 2 dimensions of the second a. With this a that's a dimensions mismatch. Evidently this axes was intended for 2d arrays, without much thought given to 3d ones. It is not a 3 axis reduction.
These single digit axes values are something that some developer thought would be convenient, but that does not mean they were rigorously thought out or tested.
The tuple axes gives you more control:
In [74]: np.tensordot(a,a,axes=[(0,1,2),(0,1,2)])
Out[74]: array(4324)
In [75]: np.tensordot(a,a,axes=[(0,1),(0,1)])
Out[75]:
array([[ 880, 940, 1000, 1060],
[ 940, 1006, 1072, 1138],
[1000, 1072, 1144, 1216],
[1060, 1138, 1216, 1294]])

Related

running temporal fusion transformer default dataset shape error

I ran default code of Temporal fusion transformer in google colab which downloaded at github.
After clone, when I ran the step 2, there's no way to test training.
python3 -m script_train_fixed_params volatility outputs yes
The problem is shape error in the below.
Computing best validation loss
Computing test loss
/usr/local/lib/python3.7/dist-packages/keras/engine/training_v1.py:2079: UserWarning: `Model.state_updates` will be removed in a future version. This property should not be used in TensorFlow 2.0, as `updates` are applied automatically.
updates=self.state_updates,
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/content/drive/MyDrive/tft_tf2/script_train_fixed_params.py", line 239, in <module>
use_testing_mode=True) # Change to false to use original default params
File "/content/drive/MyDrive/tft_tf2/script_train_fixed_params.py", line 156, in main
targets = data_formatter.format_predictions(output_map["targets"])
File "/content/drive/MyDrive/tft_tf2/data_formatters/volatility.py", line 183, in format_predictions
output[col] = self._target_scaler.inverse_transform(predictions[col])
File "/usr/local/lib/python3.7/dist-packages/sklearn/preprocessing/_data.py", line 1022, in inverse_transform
force_all_finite="allow-nan",
File "/usr/local/lib/python3.7/dist-packages/sklearn/utils/validation.py", line 773, in check_array
"if it contains a single sample.".format(array)
ValueError: Expected 2D array, got 1D array instead:
array=[-1.43120418 1.58885804 0.28558148 ... -1.50945972 -0.16713021
-0.57365613].
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.
I've tried to modify code which is predict dataframe shpae of 'data_formatters/volatility.py", line 183, in format_predictions' because I guessed that's where the problem arises.), but I can't handle that.
You have to change line
183 in volatitlity.py
output[col] = self._target_scaler.inverse_transform(predictions[col].values.reshape(-1, 1))
and line 216 in electricity.py
sliced_copy[col] = target_scaler.inverse_transform(sliced_copy[col].values.reshape(-1, 1))
Afterwards the example electricity works fine. And I guess this should be the same with volatility.

TypeError: 1st argument must be a real sequence 2 signal.spectrogram

I'm trying to take a signal from an electrical reading and decompose it into its spectrogram, but I keep getting a weird error. Here is the code:
f, t, Sxx = signal.spectrogram(i_data.values, 130)
plt.pcolormesh(t, f, Sxx)
plt.ylabel('Frequency [Hz]')
plt.xlabel('Time [sec]')
plt.show()
And here is the error:
convert_to_spectrogram(i_data.iloc[1000,:10020].dropna().values)
Traceback (most recent call last):
File "<ipython-input-140-e5951b2d2d97>", line 1, in <module>
convert_to_spectrogram(i_data.iloc[1000,:10020].dropna().values)
File "<ipython-input-137-5d63a96c8889>", line 2, in convert_to_spectrogram
f, t, Sxx = signal.spectrogram(wf, 130)
File "//anaconda3/lib/python3.7/site-packages/scipy/signal/spectral.py", line 750, in spectrogram
mode='psd')
File "//anaconda3/lib/python3.7/site-packages/scipy/signal/spectral.py", line 1836, in _spectral_helper
result = _fft_helper(x, win, detrend_func, nperseg, noverlap, nfft, sides)
File "//anaconda3/lib/python3.7/site-packages/scipy/signal/spectral.py", line 1921, in _fft_helper
result = func(result, n=nfft)
File "//anaconda3/lib/python3.7/site-packages/mkl_fft/_numpy_fft.py", line 335, in rfft
output = mkl_fft.rfft_numpy(x, n=n, axis=axis)
File "mkl_fft/_pydfti.pyx", line 609, in mkl_fft._pydfti.rfft_numpy
File "mkl_fft/_pydfti.pyx", line 502, in mkl_fft._pydfti._rc_fft1d_impl
TypeError: 1st argument must be a real sequence 2
My reading has a full cycle of 130 observations and its stored as individual values of a pandas df. The wave I am using in particular can be found here. Anyone have any ideas what this error means?
(Small disclaimer, I do not know much about signal processing, so please forgive me if this is a naive question)
Python 3.6.9, scipy 1.3.3
Downloading your file and reading it with pandas.read_csv, I could generate the following spectrogram.
import matplotlib.pyplot as plt
import pandas as pd
from scipy.signal import spectrogram
i_data = pd.read_csv('wave.csv')
f, t, Sxx = spectrogram(i_data.values[:, 1], 130)
plt.pcolormesh(t, f, Sxx)
plt.ylabel('Frequency [Hz]')
plt.xlabel('Time [sec]')
plt.show()

python-xarray: rolling mean example

I have a file which is monthly data for one year (12 points). The data starts in December and ends in November. I'm hoping to create a 3-month running mean file which would be DJF, JFM, ..., SON (10 points)
I noticed there is a DataArray.rolling function which returns a rolling window option and I think would be useful for this. However, I haven't found any examples using the rolling function. I admit i'm not familiar with bottleneck, pandas.rolling_mean or the more recent pandas.rolling so my entry level is fairly low.
Here's some code to test:
import numpy as np
import pandas as pd
import xarray as xr
lat = np.linspace(-90, 90, num=181); lon = np.linspace(0, 359, num=360)
# Define monthly average time as day in middle of month
time = pd.date_range('15/12/1999', periods=12, freq=pd.DateOffset(months=1))
# Create data as 0:11 at each grid point
a = np.linspace(0,11,num=12)
# expand to 2D
a2d = np.repeat(tmp[:, np.newaxis], len(lat), axis=1)
# expand to 3D
a3d = np.repeat(a2d[:, :, np.newaxis], len(lon), axis=2)
# I'm sure there was a cleaner way to do that...
da = xr.DataArray(a3d, coords=[time, lat, lon], dims=['time','lat','lon'])
# Having a stab at the 3-month rolling mean
da.rolling(dim='time',window=3).mean()
# Error output:
Traceback (most recent call last):
File "<ipython-input-132-9d64cc09c263>", line 1, in <module>
da.rolling(dim='time',window=3).mean()
File "/Users/Ray/anaconda/lib/python3.6/site-packages/xarray/core/common.py", line 478, in rolling
center=center, **windows)
File "/Users/Ray/anaconda/lib/python3.6/site-packages/xarray/core/rolling.py", line 126, in __init__
center=center, **windows)
File "/Users/Ray/anaconda/lib/python3.6/site-packages/xarray/core/rolling.py", line 62, in __init__
raise ValueError('exactly one dim/window should be provided')
ValueError: exactly one dim/window should be provided
You are very close. The rolling method takes a key/value pair that maps as dim/window_size. This should work for you.
da.rolling(time=3).mean()

Plotting Single Quiver Arrow with basemap

New to Python, I know this is an easy question with a one line answer but I can't figure it out.
I'm trying to plot a single quiver arrow on top of a basemap with a single longitude and latitude coordinate point but I'm getting errors when trying to plot the quiver.
Here's what I have so far:
from mpl_toolkits.basemap import Basemap
import matplotlib.pyplot as plt
lat,lon=28.31393296,-96.63100599
u,v=6.16,-3.02
m=Basemap(projection='mill',llcrnrlon=-96.9,llcrnrlat=28.245,
urcrnrlon=-96.587,urcrnrlat=28.485,resolution='i')
m.drawcoastlines()
m.fillcontinents(color='coral',lake_color='aqua')
m.drawmapboundary(fill_color='aqua')
m.quiver(lon,lat,u,v,latlon=True)
plt.show()
And I'm getting the error:
Traceback (most recent call last):
File "<ipython-input-46-c277f7618784>", line 1, in <module>
runfile('C:/animationtest.py', wdir='C:')
File "C:\Anaconda\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 685, in runfile
execfile(filename, namespace)
File "C:\Anaconda\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 71, in execfile
exec(compile(scripttext, filename, 'exec'), glob, loc)
File "C:/animationtest.py", line 41, in <module>
m.quiver(lon,lat,ve,vn,latlon=True)
File "C:\Anaconda\lib\site-packages\mpl_toolkits\basemap\__init__.py", line 556, in with_transform
x1, u = self.shiftdata(x, u)
File "C:\Anaconda\lib\site-packages\mpl_toolkits\basemap\__init__.py", line 4713, in shiftdata
raise ValueError('1-d or 2-d longitudes required')
ValueError: 1-d or 2-d longitudes required
Can someone please explain why this doesn't work?
EDIT: Figured out the solution, had to convert lat lon to map projection with the line
x,y=m(lon,lat)
And then plot the quiver with
m.quiver(x,y,u,v)
Does exactly what I needed to do

scipy.test() fails with many assertion errors

New to Scipy.
I've installed scipy and numpy, but ween I run the scipy.test() I get many test fails.
I tried to find some common reasons and I have found several:
>>> import scipy
>>> scipy.test()
Running unit tests for scipy
NumPy version 1.6.2
NumPy is installed in /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy
SciPy version 0.10.1
SciPy is installed in /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy
Python version 2.7.3 (default, Apr 19 2012, 00:55:09) [GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)]
nose version 1.1.2
...
AssertionError:
Arrays are not almost equal to 7 decimals
...
AssertionError:
Not equal to tolerance rtol=1e-10, atol=1e-305
...
AssertionError:
Arrays are not almost equal to 4 decimals
...
AssertionError:
Max |adiff|: 92783.6
Max |rdiff|: 0.353942
Sounds to me like MPI problem or something, but I would't really know.
Any help would be greatly appreciated.
Best, Jonathan.
-------- EDIT ---------
Tried it with scipy.test(verbose=10).
Most tests seems to work fine.
The fails starts with an ERROR. this is what I get:
ERROR: Failure: DeprecationWarning (
The scipy.maxentropy module is deprecated in scipy 0.10, and scheduled to be
removed in 0.11.
If you are using some of the functionality in this module and are of the
opinion that it should be kept or moved somewhere - or you are even interested
to maintain/improve this whole module - please ask on the scipy-dev mailing
list.
The logsumexp function has already been moved to scipy.misc.)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/loader.py", line 390, in loadTestsFromName
addr.filename, addr.module)
File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/importer.py", line 39, in importFromPath
return self.importFromDir(dir_path, fqname)
File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/importer.py", line 86, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/maxentropy/__init__.py", line 19, in <module>
DeprecationWarning)
DeprecationWarning:
The scipy.maxentropy module is deprecated in scipy 0.10, and scheduled to be
removed in 0.11.
If you are using some of the functionality in this module and are of the
opinion that it should be kept or moved somewhere - or you are even interested
to maintain/improve this whole module - please ask on the scipy-dev mailing
list.
The logsumexp function has already been moved to scipy.misc.
======================================================================
FAIL: test_asum (test_blas.TestFBLAS1Simple)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/lib/blas/tests/test_blas.py", line 58, in test_asum
assert_almost_equal(f([3,-4,5]),12)
File "/Library/Python/2.7/site-packages/numpy/testing/utils.py", line 467, in assert_almost_equal
raise AssertionError(msg)
AssertionError:
Arrays are not almost equal to 7 decimals
ACTUAL: 0.0
DESIRED: 12
======================================================================
FAIL: test_dot (test_blas.TestFBLAS1Simple)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/lib/blas/tests/test_blas.py", line 67, in test_dot
assert_almost_equal(f([3,-4,5],[2,5,1]),-9)
File "/Library/Python/2.7/site-packages/numpy/testing/utils.py", line 467, in assert_almost_equal
raise AssertionError(msg)
AssertionError:
Arrays are not almost equal to 7 decimals
ACTUAL: 0.0
DESIRED: -9
======================================================================
FAIL: test_nrm2 (test_blas.TestFBLAS1Simple)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/lib/blas/tests/test_blas.py", line 78, in test_nrm2
assert_almost_equal(f([3,-4,5]),math.sqrt(50))
File "/Library/Python/2.7/site-packages/numpy/testing/utils.py", line 467, in assert_almost_equal
raise AssertionError(msg)
AssertionError:
Arrays are not almost equal to 7 decimals
ACTUAL: 0.0
DESIRED: 7.0710678118654755
======================================================================
FAIL: test_basic.TestNorm.test_overflow
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/linalg/tests/test_basic.py", line 581, in test_overflow
assert_almost_equal(norm(a), a)
File "/Library/Python/2.7/site-packages/numpy/testing/utils.py", line 451, in assert_almost_equal
return assert_array_almost_equal(actual, desired, decimal, err_msg)
File "/Library/Python/2.7/site-packages/numpy/testing/utils.py", line 846, in assert_array_almost_equal
header=('Arrays are not almost equal to %d decimals' % decimal))
File "/Library/Python/2.7/site-packages/numpy/testing/utils.py", line 677, in assert_array_compare
raise AssertionError(msg)
AssertionError:
Arrays are not almost equal to 7 decimals
(mismatch 100.0%)
x: array(-0.0)
y: array([ 1.00000002e+20], dtype=float32)
======================================================================
FAIL: test_basic.TestNorm.test_stable
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/linalg/tests/test_basic.py", line 586, in test_stable
assert_almost_equal(norm(a) - 1e4, 0.5)
File "/Library/Python/2.7/site-packages/numpy/testing/utils.py", line 467, in assert_almost_equal
raise AssertionError(msg)
AssertionError:
Arrays are not almost equal to 7 decimals
ACTUAL: -10000.0
DESIRED: 0.5
======================================================================
FAIL: test_basic.TestNorm.test_types
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/linalg/tests/test_basic.py", line 568, in test_types
assert_allclose(norm(x), np.sqrt(14), rtol=tol)
File "/Library/Python/2.7/site-packages/numpy/testing/utils.py", line 1213, in assert_allclose
verbose=verbose, header=header)
File "/Library/Python/2.7/site-packages/numpy/testing/utils.py", line 677, in assert_array_compare
raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=2.38419e-06, atol=0
(mismatch 100.0%)
x: array(1.0842021724855044e-19)
y: array(3.7416573867739413)
======================================================================
FAIL: test_asum (test_blas.TestFBLAS1Simple)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/linalg/tests/test_blas.py", line 99, in test_asum
assert_almost_equal(f([3,-4,5]),12)
File "/Library/Python/2.7/site-packages/numpy/testing/utils.py", line 467, in assert_almost_equal
raise AssertionError(msg)
AssertionError:
Arrays are not almost equal to 7 decimals
ACTUAL: 0.0
DESIRED: 12
======================================================================
FAIL: test_dot (test_blas.TestFBLAS1Simple)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/linalg/tests/test_blas.py", line 109, in test_dot
assert_almost_equal(f([3,-4,5],[2,5,1]),-9)
File "/Library/Python/2.7/site-packages/numpy/testing/utils.py", line 467, in assert_almost_equal
raise AssertionError(msg)
AssertionError:
Arrays are not almost equal to 7 decimals
ACTUAL: 0.0
DESIRED: -9
======================================================================
FAIL: test_nrm2 (test_blas.TestFBLAS1Simple)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/linalg/tests/test_blas.py", line 127, in test_nrm2
assert_almost_equal(f([3,-4,5]),math.sqrt(50))
File "/Library/Python/2.7/site-packages/numpy/testing/utils.py", line 467, in assert_almost_equal
raise AssertionError(msg)
AssertionError:
Arrays are not almost equal to 7 decimals
ACTUAL: 0.0
DESIRED: 7.0710678118654755
----------------------------------------------------------------------
Ran 5101 tests in 56.140s
FAILED (KNOWNFAIL=12, SKIP=42, errors=1, failures=9)
<nose.result.TextTestResult run=5101 errors=1 failures=9>
I am really sorry to say I have no idea what the errors mean.
The Scipy failures seem typical of what comes out from using a wrong (= ABI incompatibility with Veclib) Fortran compiler on OSX. The Numpy failures are pretty likely unrelated to these --- difficult to say without seeing them.
The mess with binary interface incompatibilities and Fortran compilers on OSX is unfortunate and applies to all Fortran code, but not very easily fixable by Scipy developers --- except that if an incorrect Fortran compiler is detected, the build should be made to fail with an informative error message...
You may find these useful: http://projects.scipy.org/scipy/ticket/1476 http://projects.scipy.org/scipy/ticket/1496
Can you run scipy.test(verbose=10), please? This will print more output and help in finding ot what's wrong.
Looks like you are running into exactly the situation described in this mailing list.
In the discussion it is mentioned that this might relate to a bug in SciPy. There is a bug report describing your situation. You may want to follow this report.