โ07-15-2022 02:57 PM
I am a newbie to data bricks and trying to write results into the excel/ CSV file using the below command but getting
DataFrame' object has no attribute 'to_csv'
errors while executing.
I am using a notebook to execute my SQL queries and now want to store results in the CSV or excel file
%python
df =spark.sql ("""select * from customer""")
and now I want to store the query results in the excel/csv file.I have tried the below code but it's not working
df.coalesce(1).write.option("header","true").option("sep",",").mode("overwrite").csv("file:///C:/New folder/mycsv.csv")
AND
df.write.option("header", "true").csv("file:///C:/New folder/mycsv.csv")
โ07-15-2022 08:02 PM
Are you using Databricks on AWS, Azure or GCP?
If AWS, then you need to create an S3 bucket and mount that S3 bucket on dbfs location.
I'm assuming that customer table exists in your databricks account.
Next, run this code and it will write your df to S3 location.
Lastly, download the csv file from your S3 location to local.
df = spark.sql("select * from customers")
df.write.option("header",true).csv(<dbfs_path>)
More about dbfs: here
โ07-16-2022 11:07 AM
No, I am not using AWS. I am using Azure and want to download SQL results to the local drive. your above post not helpful
โ07-17-2022 04:51 PM
If your query results are upto 1000 records, then you can download from the cell.. for more than that you'll have to write it to your cloud storage and then download from there.. Else you can run query in batches of 1000 to download it..
โ07-17-2022 08:28 AM
Use โdownload query resultsโ in your DBSQL or notebook cell to download to local disk.
โ09-05-2022 01:23 AM
Hi there @Dipak Bachhavโ
Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.
We'd love to hear from you.
Thanks!
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group