Automate data bricks batch which copy data from file/database and put it in some other file/database

[Manoranjan]

Hi,

I have created a data bricks batch which copy data from a file/database table and put it in some other file/database table.

I want to automate the functional test cases of this data brick batch.

Can someone suggest, is there any tool available for this kind of job or any approach that can help me?

[nayan_wylde]

Create a workflow for your batch load notebook and then schedule it.

https://docs.databricks.com/gcp/en/jobs/jobs-quickstart

[Manoranjan]

I want to keep my automation framework outside the data bricks environment.

is there any way where I can update the record to parquet files/Unity catalog/Delta tables from my framework.

Also start the data bricks batch from outside framework as well.

Please suggest