- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-13-2025 10:30 AM - edited 07-13-2025 10:45 AM
Hi @giuseppe_esq to answer your question about:
is there any way to create an automated process, that will allow me to import all of the CSV files (e.g. Customer, Employees, Products, Orders etc), stored locally, into newly created Databricks tables please? I think there's a couple of routes we could look at for this. First of all, if you're on a windows machine, you could use something like "Task Scheduler" and create a python script (or script of your choice) to move/create the files you want into something like ADLS2 (or leverage one of the inbuilt Fivetran connectors?) and then let Databricks scan the storage. The key here, I guess, is using the task scheduler to automate this for you (to move it from the local machine).
You could also look at https://docs.databricks.com/aws/en/dev-tools/sdk-python. This looks really cool! I haven't used it yet but this seems like we could do something here. Check this post out: https://community.databricks.com/t5/data-engineering/is-it-possible-to-load-data-only-using-databric...
This seems like a promising route to explore 🙏. Hopefully, those routes may be of use!
Would love to see what the other community members recommend 👌
All the best,
BS