cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Can i run data engineering/data transformation jobs using databricks SQL endpoints or are databricks clusters (interactive or jobs) the recommended way to do this?

aladda
Databricks Employee
Databricks Employee
1 REPLY 1

Ryan_Chynoweth
Esteemed Contributor

Generally, interactive clusters and jobs are better suited for data engineering and transformations as they support more than just SQL. However, if you are using pure SQL, then endpoints can be used for data transformations. All of the Spark SQL functionality is available in DSQL so you can create tables, insert/update data, and run multiple SQL statements within a single DSQL query.

Note that DSQL queries are mostly purposed for visualizations and data acquisition.

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local communityโ€”sign up today to get started!

Sign Up Now