cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
brickster_2018
by • Databricks Employee
  • 4620 Views
  • 1 replies
  • 0 kudos

Resolved! How to track the history of schema changes for a Delta table

I have a Delta table that had schema changes in multiple commits. I wanted to track all these schema changes that happened on the Delta table. The "DESCRIBE HISTORY" is not useful as it logs the schema change made by ALTER TABLE operations.

Data Engineering
Reply
  • 4620 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
brickster_2018
Databricks Employee
  • 0 kudos

When a write operation is performed with columns added. we are not explicitly showing that in DESCRIBE HISTORY output. Only an entry is made for write. and in the operation Parameters, it's not showing anything about schema evolution. whereas if we d...

  • 0 kudos
aladda
by • Databricks Employee
  • 1062 Views
  • 1 replies
  • 0 kudos

Can i run data engineering/data transformation jobs using databricks SQL endpoints or are databricks clusters (interactive or jobs) the recommended way to do this?

Data Engineering
Reply
  • 1062 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Ryan_Chynoweth
Esteemed Contributor
  • 0 kudos

Generally, interactive clusters and jobs are better suited for data engineering and transformations as they support more than just SQL. However, if you are using pure SQL, then endpoints can be used for data transformations. All of the Spark SQL fun...

  • 0 kudos
MoJaMa
by • Databricks Employee
  • 1044 Views
  • 1 replies
  • 1 kudos

How can I understand Photon pricing for Data Engineering and estimate costs for a workload?

Data Engineering
Reply
  • 1044 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
MoJaMa
Databricks Employee
  • 1 kudos

Use the calculator herehttps://databricks.com/product/aws-pricing/instance-typesOpen two windows side by side, pick Photon and Non-Photon instances of the same type and compare.

  • 1 kudos
Labels