Memory Error: Numpy.genfromtxt()

I have a 50,000x5,000 matrix(float) file. when use x = np.genfromtxt(readFrom, dtype=float) to load the file into memory, I am getting the following error message: File 'C:\Pytho

Solution 1:

Depending on your OS and Python version, it's quite likely that you'll never be able to allocate a 1GB array (mgilson's answer is spot on here). The problem is not that you're running out of memory, but that you're running out of contiguous memory. If you're on a 32-bit machine (especially running Windows), it will not help to add more memory. Moving to a 64-bit architecture would probably help.

Using smaller data types can certainly help; depending on the operations you use, a 16-bit float or even an 8-bit int might be sufficient.

If none of this works, then you're forced to admit that the data just doesn't fit in memory. You'll have to process it piecewise (in this case, storing the data as an HDF5 array might be very useful).

Solution 2:

You're actually using 8 byte floats since python's float corresponds to C's double (at least on most systems):

a=np.arange(10,dtype=float)
print(a.dtype)  #np.float64

You should specify your data type as np.float32. Depending on your OS, and whether it is 32bit or 64bit, (and whether you're using 32bit python vs. 64bit python), the address space available for numpy to use could be smaller than your 4Gb which could be an issue here as well.