Load data from Aurora to Databricks directly

[creditorwatch]

Hi,

Does anyone know how to link Aurora to Databricks directly and load data into Databricks automatically on a schedule without any third-party tools in the middle?

[Kaniz]

Hi @creditorwatch, To ingest data into Databricks directly from Amazon Aurora and automate the process on a schedule, you have a few options. 

Let’s explore them:

Auto Loader (Recommended):

• Auto Loader is a powerful feature in Databricks that efficiently processes new data files as they arrive in cloud storage without additional setup.
• It provides a Structured Streaming source called cloudFiles. Given an input directory path on the cloud file storage, it automatically processes new files as they arrive.
• You can use Auto Loader for incremental data ingestion from cloud object storage (such as Amazon S3).
• To set up Auto Loader, follow the tutorial: Run your first ETL workload on Databricks.
• You can even schedule Auto Loader to run as a batch job using Trigger.AvailableNow in Databricks Job....

Delta Live Tables and Auto Loader:

• Combine Delta Live Tables with Auto Loader to simplify deployment of scalable, incremental ingestion infrastructure.
• Delta Live Tables emphasizes production-ready infrastructure deployment.
• If you have data arriving once every few hours, you can leverage Auto Loader in a scheduled job usin...2.

COPY INTO (SQL):

• COPY INTO allows SQL users to idempotently and incrementally ingest data from cloud object storage into Delta tables.
• Use COPY INTO when you’re dealing with thousands of files.
• If you expect millions of files over time, Auto Loader is a better choice3.

Remember that these options allow you to ingest data directly into Databricks without relying on third-party tools. Choose the one that best fits your use case and requirements! 🚀