Write CSV file in append mode on Azure Databricks - pandas

Question

Wanted to write csv file in append mode on Azure Databricks. The below code is working fine on my local machine.(Jupyter notebook)
df = pd.read_csv("/dbfs/mnt/dev/tmp/ml_p/csv_append.csv")
df+6
Ans
[1]: https://i.stack.imgur.com/sXsgH.png
when I opened the same csv file and wanted to save the file after performing the operation.
I got , OSError: [Errno 95] Operation not supported
with open('/dbfs/mnt/dev/tmp/ml_p/csv_append.csv', 'a') as f:
(df + 6).to_csv(f, header=False)
Is there is another alternative to write the CSV file in append mode? or Can I achieve the same using pyspark.

Answer 1

There are some limitations on what operations could be done with files on DBFS (especially via /dbfs mount point), and you hit this limit. The workaround would be to copy file from DBFS to local file system, modify it the same as you do it, and then upload back. Copying of the file could be done with dbutils.fs commands, like:
dbutils.fs.cp("dbfs:/mnt/dev/tmp/ml_p/csv_append.csv", "file:/tmp/csv_append.csv")
df = pd.read_csv("/tmp/csv_append.csv")
df+6
with open('/tmp/csv_append.csv', 'a') as f:
(df + 6).to_csv(f, header=False)
dbutils.fs.mv("file:/tmp/csv_append.csv","dbfs:/mnt/dev/tmp/ml_p/csv_append.csv")