topic Re: Automate data bricks batch which copy data from file/database and put it in some other file/data in Data Engineering

Automate data bricks batch which copy data from file/database and put it in some other file/database

Manoranjan — Wed, 02 Jul 2025 14:14:21 GMT

Hi,

I have created a data bricks batch which copy data from a file/database table and put it in some other file/database table.

I want to automate the functional test cases of this data brick batch.

Can someone suggest, is there any tool available for this kind of job or any approach that can help me?

nayan_wylde — Wed, 02 Jul 2025 17:39:25 GMT

Create a workflow for your batch load notebook and then schedule it.

Manoranjan — Thu, 03 Jul 2025 04:18:18 GMT

I want to keep my automation framework outside the data bricks environment.

is there any way where I can update the record to parquet files/Unity catalog/Delta tables from my framework.

Also start the data bricks batch from outside framework as well.

Please suggest