I'm running the following basic code:
dfMain.to_csv('./January_filtered_International_WE.csv')
which used to run normally until yesterday. This morning I upgraded to pandas 0.25.0 while running code and now I cannot write my 500k rows dataframe to a csv. I can mention that I left Jupyter Notebook running in order to do some processing, so this morning when I opened it I had the dataFrame already, processed.
Versions (using Windows 10)
Jupyter notebook : 5.7.8
Python : 3.6.7
Pandas : 0.25.0
I would like to save my DataFrame in a fast and efficient manner as I will load it several times in the future. I do not want to close the notebook as this will delete the dataFrame.
I tried:
downgrading to Pandas 0.24.2 (previous version used) but still getting the __init__() got an unexpected keyword argument 'tupleize_cols'
use pd.to_pickle but got a memoryError
use pd.to_hdf but got a memoryError
using msgbox instead but apparently it does not support DataFrames (got an error)
upgrade Jupyter notebook, but got the following error:
ERROR: ipython 5.8.0 has requirement prompt-toolkit<2.0.0,>=1.0.4, but
you'll have prompt-toolkit 2.0.9 which is incompatible
so naturally I did pip install prompt-toolkit 1.0.16 but then got this message:
ERROR: jupyter-console 6.0.0 has requirement prompt-toolkit<2.1.0,>=2.0.0, but you'll have prompt-toolkit 1.0.16 which is incompatible.
As an alternative I went into PyCharm and took a random DataFrame.to_csv and it worked. This makes me think the issue is with Jupyter Notebook.
Any help on how to save the DataFrame (~12 GB) is appreciated!
Re-installing Jupyter did the trick in my case
I kept getting the same error, but updating Jupyter fixed it
Related
I want to run modules of Geopandas in Spyder. Apparently Geopandas is compatible with Sypder 4.2.5, (not with any higher version) and I could run code with this combination. However, in one of my code I had to use "input" command and the problem starts there. Sypder 4.2.5 crashes if I try to run input command. From the internet, I came to know that there was a bug in spyder and it was fixed in Spyder 5.3. Now I have no idea how to fix this problem. If I upgrade Spyder, Geopandas will not work. If I don't upgrade spyder, 'input' will not work.
I was trying to run something like the following code
def Coditions_R3():
print("This is R3")
def Coditions_R4():
print("This is R4")
System = input('Please Enter drone system: \n' )
print(System)
if (System == 'R3'):
Coditions_R3()
elif (System == 'R4'):
Coditions_R4()
Can anyone help? is there any way around to run geopandas with higher Spyder versions? or use something else in place of input?
I cannot execute df.info(). it results in an error:
TypeError: Cannot interpret '<attribute 'dtype' of 'numpy.generic' objects>' as a data type.
I've read some solutions regarding the versions of the libraries, but I'm not sure what should I do.
I've checked the versions in Jupyter:
Numpy version: 1.21.5
pandas version: 1.0.1
Thank you for any help!
I'm trying to reload another model to another jupyter notebook using this code:
import tensorflow as tf
reloaded = tf.saved_model.load('m_translator')
result = reloaded.tf_translate(input_text)
and I recently got this error:
KeyError Traceback (most recent call last)
File ~\anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py:4177, in Graph._get_op_def(self, type)
4176 try:
-> 4177 return self._op_def_cache[type]
4178 except KeyError:
KeyError: 'NormalizeUTF8'
FileNotFoundError: Op type not registered 'NormalizeUTF8' in binary running on LAPTOP-D3PPA576. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) `tf.contrib.resampler` should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.
You may be trying to load on a different device from the computational device. Consider setting the `experimental_io_device` option in `tf.saved_model.LoadOptions` to the io_device such as '/job:localhost'.
I had this issue before. To solve it, you need to install tensorflow_text. You should try to :
>>> tf.__version__
2.8.2
>>>!pip install tensorflow-text==2.8.2
In addition to installing tensorflow_text library, what helped me with a similar problem was importing the library at the top of the notebook:
import tensorflow_text
I'm trying to convert a Pandas dataframe to a Pyspark dataframe, and getting the following pyarrow-related error:
import pandas as pd
import numpy as np
data = np.random.rand(1000000, 10)
pdf = pd.DataFrame(data, columns=list("abcdefghij"))
df = spark.createDataFrame(pdf)
/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/session.py:714: UserWarning: createDataFrame attempted Arrow optimization because 'spark.sql.execution.arrow.enabled' is set to true; however, failed by the reason below:
'JavaPackage' object is not callable
Attempting non-optimization as 'spark.sql.execution.arrow.fallback.enabled' is set to true.
I've tried different versions of pyarrow (0.10.0, 0.14.1, 0.15.1 and more) but with the same result. How can I debug this?
I had the same issue, changed the cluster setting to emr-5.30.1 and arrow version to 0.14.1 and it resolved the issue
Can you try upgrading your pyspark to >= 3.0.0? I had the above error with all versions of arrow, but bumping to the newer pyspark fixed it for me.
There is a version conflict with older versions of Spark (ex: 2.4.x) and newer versions of arrow.
I've studied Julia for several months and I am not an expert in programming. Recently, I re-installed Windows on my laptop and re-installed Julia, Anaconda(for python 3) and Atom, so far.
In Julia, I installed PyPlot.jl by Pkg.add("PyPlot") and want to test plotting with PyPlot. So, I put using PyPlot in Julia and it give an error during initialization of module PyPlot. I've got the following error message:
ERROR conda.core.link:_execute_actions(330): An error occurred while installing package 'defaults::qt-5.6.2-vc9_3'.
UnicodeDecodeError('utf8', ' 1\xb0\xb3 \xc6\xc4\xc0\xcf\xc0\xcc \xba\xb9\xbb\xe7\xb5\xc7\xbe\xfa\xbd\xc0\xb4\xcf\xb4\xd9.\r\n', 9, 10, 'invalid start byte')
Attempting to roll back.
UnicodeDecodeError('utf8', ' 1\xb0\xb3 \xc6\xc4\xc0\xcf\xc0\xcc \xba\xb9\xbb\xe7\xb5\xc7\xbe\xfa\xbd\xc0\xb4\xcf\xb4\xd9.\r\n', 9, 10, 'invalid start byte')
Please let me know how I can solve the problem.
I solved the problem myself by using python in Anaconda.
ENV["PYTHON"]="C:/ProgramData/Anaconda3/python"
In the double quotation marks, put the location of the python which is installed by Anaconda.
After that, apply
Pkg.build("PyCall")
Pkg.build("PyPlot")
I now can use PyPlot using PyPlot as before. It seems there was some problem to use python included in Julia.