numpy memmap runtime error.... 64bits system with 2Gigas limit? - numpy

Question

I'm trying to create a large file with numpy memmap
big_file = np.memmap(fnamemm, dtype=np.float32, mode='w+', shape=(np.prod(dims[1:]), len_im), order='F')
The system is a Windows 10-64bits operating in a 64bits python
In [2]: sys.maxsize
Out[2]: 9223372036854775807
With enough virtual memory (maximum of 120000Megas)
However every time I try to create a file which resulting size should exceed 2Gigas I get a runtime error
In [29]: big_file = np.memmap(fnamemm, dtype=np.int16, mode='w+', shape=(np.prod(dims[1:]), len_im), order=order)
C:\Users\nuria\AppData\Local\Continuum\anaconda3\envs\caiman\lib\site-packages\numpy\core\memmap.py:247: RuntimeWarning: overflow encountered in long_scalars
bytes = long(offset + size*_dbytes)
---------------------------------------------------------------------------
OSError Traceback (most recent call last)
<ipython-input-29-66578da2d3f6> in <module>()
----> 1 big_file = np.memmap(fnamemm, dtype=np.int16, mode='w+', shape=(np.prod(dims[1:]), len_im), order=order)
~\AppData\Local\Continuum\anaconda3\envs\caiman\lib\site-packages\numpy\core\memmap.py in __new__(subtype, filename, dtype, mode, offset, shape, order)
248
249 if mode == 'w+' or (mode == 'r+' and flen < bytes):
--> 250 fid.seek(bytes - 1, 0)
251 fid.write(b'\0')
252 fid.flush()
OSError: [Errno 22] Invalid argument
This error does not happen when the files sizes are under 2Gigas...
I have replicated the same problem with another windows 7 also 64bits
Have I forgotten something? Why is memmap acting as I have a 32bits system?
EDIT: The error is not exactly a runtime error. the variable "bytes" gets an runtime warning when trying to get the length of the file, resulting I guess in a bad argument that raises the Errno 22

Answer 1

I had a similar error and it turned out that it was because one of the shape=(A,B) argument was with int32 instead of int64. Try the following:
len_im64 = np.array(len_im,dtype='int64')
big_file = np.memmap(fnamemm, dtype=np.float32, mode='w+', shape=(np.prod(dims[1:]).astype('int64'), len_im), order='F')
It fixed it for me.

Answer 2

Even though the system is 64 bit, problem may be because the application is built with 32 bit target. Check your shell execution mode (32 bit or 64 bit).
For such applications you have make them large address aware. Then the 32 applications can get access to 4GB memory on 64 bit machines.
How to do that? Here is someone's article.
https://github.com/pyinstaller/pyinstaller/issues/1288
Note: If your application is already built with 64 bit target.. ignore this and put in comment, Will delete this answer.