cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Mikki007
by New Contributor II
  • 7309 Views
  • 3 replies
  • 0 kudos

How to extract the start and end time of the command line cell of the notebook using REST API in Azure Databricks?

HiI have a notebook with many command line cells in it.I want to extract the execution time of each cell using Databricks REST API? How can I do that?Please note - I managed to get the Start & End time of the Job using REST API (/2.1/jobs/runs/get) f...

  • 7309 Views
  • 3 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Milind Keer​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers y...

  • 0 kudos
2 More Replies
Hubert-Dudek
by Esteemed Contributor III
  • 1797 Views
  • 1 replies
  • 29 kudos

The New DBR LTS runtime is here. Databricks Runtime 11.3 is set to be LTS!   

The New DBR LTS runtime is here. Databricks Runtime 11.3 is set to be LTS! 

11.3 LTS
  • 1797 Views
  • 1 replies
  • 29 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 29 kudos

Great post, thank you for sharing this @Hubert Dudek​ 

  • 29 kudos
HQJaTu
by New Contributor III
  • 7387 Views
  • 10 replies
  • 1 kudos

Resolved! Azure Databricks container runtime broken in 9.1 LTS, how to fix?

For stability, I've stuck with LTS. Last Friday my containers stopped working with error message:Py4JException: An exception was raised by the Python Proxy. Return Message: Traceback (most recent call last): File "/databricks/spark/python/lib/py4j-...

  • 7387 Views
  • 10 replies
  • 1 kudos
Latest Reply
HQJaTu
New Contributor III
  • 1 kudos

This is getting worse. Now JDBC write to SQL is failing for same reason. I haven't yet found a solution for this.Am I not supposed to use containers? Python?This is not cool.

  • 1 kudos
9 More Replies
SepidehEb
by Databricks Employee
  • 4223 Views
  • 6 replies
  • 7 kudos

Resolved! How to get a minor DBR image?

In short, we aim to add a step to a CI job that would run tests in a container, which supposedly should mimic DBR of our clusters – currently we use 7.3 . We consider using one of databricksruntime images (possibly a standard:7.x for now, https://hub...

  • 4223 Views
  • 6 replies
  • 7 kudos
Latest Reply
Atanu
Databricks Employee
  • 7 kudos

Hi @Sepideh Ebrahimi​ , since cluster is Databricks proprietary, you ca not run it locally. as @Werner Stinckens​  said, you can build your own image but that has to be run in cluster. but there is databricks connect (https://docs.databricks.com/dev-...

  • 7 kudos
5 More Replies
Anonymous
by Not applicable
  • 1488 Views
  • 2 replies
  • 0 kudos

Changing default Delta behavior in DBR 8.x for writes

Is there anyway to add a Spark Config that reverts the default behavior when doing tables writes from Delta to Parquet in DBR 8.0+? I know you can simply specify .format("parquet") but that could involve a decent amount of code change for some client...

  • 1488 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Thanks @Ryan Chynoweth​ !

  • 0 kudos
1 More Replies
Labels