topic Re: DownLoad CSV files from Delta Lake in Data Engineering

DownLoad CSV files from Delta Lake

Madalian — Tue, 30 Apr 2024 09:00:46 GMT

We have around 1800 tables in Parq format (Delta Lake). These 1800 tables are very big, we have all these 1800 tables are converted into tables. But we have a requirement that, we need to download in CSV. (from PowerBI / any other reporting tool). Customer want to download full set of tables. I tried following ways.

1. Created paramterized Notebook to filter table and download from result set.

This method is not so liking by users.

Thanks

Re: DownLoad CSV files from Delta Lake

jose_gonzalez — Tue, 30 Apr 2024 23:51:33 GMT

@Madalian did you have time to check Kaniz's response? does this help or you still need help? please let us know

Re: DownLoad CSV files from Delta Lake

Madalian — Wed, 01 May 2024 03:07:34 GMT

Dear Kaniz,

Thank you. one doubt 1) converting tables data into CSV and saving again one more of storage layer.

IS there any way on fly we can convert these tables into CSV's. and export into PowerBI? and again i see in powerBI has limitations around > 5 gb downloads. we have some tables 10 GB +.

Thanks