topic Is there a function similar to display that downloads a dataframe? in Data Engineering

Is there a function similar to display that downloads a dataframe?

jonathan-dufaul — Tue, 24 Jan 2023 15:28:41 GMT

I find myself constantly having to do display(df), and then "recompute with <5g records and download). I was just hoping I could skip the middleman and download from get go.

ideally it'd be a function like download(df,num_rows="max") where num_rows is either a number or the maximum (<5gb)

Re: Is there a function similar to display that downloads a dataframe?

Hubert-Dudek — Tue, 24 Jan 2023 18:26:35 GMT

Question where do you want to download it to? If to cloud location, use regular DataFrameWriter. You can install, for example, Azure Storage Explorer on your computer. Some cloud storage you can even mount in your system as a folder or network share.

Re: Is there a function similar to display that downloads a dataframe?

jonathan-dufaul — Tue, 24 Jan 2023 19:15:06 GMT

oh no the download button on the dataframe itself to the local computer. I know how to write to every single location I need, but for quick data inspection > 1k rows it's a few clicks and a wasted query.