cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Can i run data engineering/data transformation jobs using databricks SQL endpoints or are databricks clusters (interactive or jobs) the recommended way to do this?

Anand_Ladda
Honored Contributor II
1 REPLY 1

Ryan_Chynoweth
Honored Contributor III

Generally, interactive clusters and jobs are better suited for data engineering and transformations as they support more than just SQL. However, if you are using pure SQL, then endpoints can be used for data transformations. All of the Spark SQL functionality is available in DSQL so you can create tables, insert/update data, and run multiple SQL statements within a single DSQL query.

Note that DSQL queries are mostly purposed for visualizations and data acquisition.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.