cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

OSError: Invalid argument when attempting to save a pandas dataframe to csv

kll
New Contributor III

I am attempting to save a pandas DataFrame to as csv to a directory I created in Databricks workspace or in the `cwd`.

import pandas as pd
 
import os
 
df.to_csv("data.csv", index=False)
 
df.to_csv(str(os.getcwd()) + "/data.csv", index=False)
 
 
 
 

OSError: [Errno 22] Invalid argument
 
---------------------------------------------------------------------------
 
OSError                                   Traceback (most recent call last)
 
File :2
 
      1 # Save CSV to Databricks filesystem
 
----> 2 df.to_csv("pmf_remax.csv", index=False)

I am also saving a file using `_file` ` method and while it doesn't throw an error, i don't see the file in the `cwd`.

***.to_file(str(os.getcwd()) + '/us_df.geojson', driver='GeoJSON')

when i save a file, where does it save it? I tried to run the command `find . -name example.txt` from the root directory in the web terminal and didn't find it.

1 ACCEPTED SOLUTION

Accepted Solutions

Ajay-Pandey
Esteemed Contributor III

Hi @Keval Shah​ ,

You can save your dataframe to csv in dbfs storage.

Please refer below code that might help you-

df = pd.read_csv(StringIO(data), sep=',')
#print(df)
df.to_csv('/dbfs/FileStore/ajay/file1.txt')

View solution in original post

1 REPLY 1

Ajay-Pandey
Esteemed Contributor III

Hi @Keval Shah​ ,

You can save your dataframe to csv in dbfs storage.

Please refer below code that might help you-

df = pd.read_csv(StringIO(data), sep=',')
#print(df)
df.to_csv('/dbfs/FileStore/ajay/file1.txt')

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!