topic Re: Upload CSV files on Databricks by code (note UI) in Data Engineering

Upload CSV files on Databricks by code (note UI)

Rodrigo_Brandet — Fri, 10 Sep 2021 19:43:01 GMT

Hello everyone.

I have a process on databricks when I need to upload a CSV file everyday manually.

I would like to know if there is a way to import this data (as panda in python, for example) with no necessary to upload this file everyday manually utilizing UI.

Thanks a lot.

Re: Upload CSV files on Databricks by code (note UI)

Ryan_Chynoweth — Fri, 10 Sep 2021 21:39:11 GMT

You can use our DBFS Put REST API endpoint. Note that this uploads the data but you would still probably need a Databricks job to load the data into a table.

You could also use the REST APIs or Python client for the cloud storage you are using as well.

Re: Upload CSV files on Databricks by code (note UI)

Alexis — Sat, 11 Sep 2021 08:50:35 GMT

Hi Rodrigo

if you're using Azure databricks you can also try Autoloader this will upload you data into a spark dataframe or directly into a panda's dataframe. Be aware that you'll need setup your workspace with the necessary permissions.

Here a link with more info Ingest CSV data with Auto Loader - Azure Databricks - Workspace | Microsoft Docs

Happy learning!

Re: Upload CSV files on Databricks by code (note UI)

-werners- — Mon, 13 Sep 2021 13:57:03 GMT

Autoloader is indeed a valid option,

or use of some kind of ETL tool which fetches the file and put it somewhere on your cloud provider, like Azure Data Factory or AWS Glue etc.