cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Can i run data engineering/data transformation jobs using databricks SQL endpoints or are databricks clusters (interactive or jobs) the recommended way to do this?

aladda
Honored Contributor II
Honored Contributor II
1 REPLY 1

Ryan_Chynoweth
Honored Contributor III

Generally, interactive clusters and jobs are better suited for data engineering and transformations as they support more than just SQL. However, if you are using pure SQL, then endpoints can be used for data transformations. All of the Spark SQL functionality is available in DSQL so you can create tables, insert/update data, and run multiple SQL statements within a single DSQL query.

Note that DSQL queries are mostly purposed for visualizations and data acquisition.