- 7993 Views
- 3 replies
- 0 kudos
Azure Databricks Monitoring & Alerting (Data Observability) Tools / Frameworks for Enterprise
I am trying to evaluate options for Monitoring and Alerting tools like New Relic, Datadog, Grafana with Databricks on Azure . No one supports when reached out to them. I would like to hear from the databricks team on the recommended tool / framework ...
- 7993 Views
- 3 replies
- 0 kudos
- 0 kudos
I'd recommend this new tool we've been trying out. It's really helpful for monitoring and provides good insights on how Azure Databricks clusters, pools & jobs are doing – like if they're healthy or having issues. It brings everything together, makin...
- 0 kudos
- 13179 Views
- 4 replies
- 1 kudos
Dynamic Number of Tasks in Databricks Workflow
Do Databricks workflows support creating a workflow with a dynamic number of tasks?For example, let's say we have a DAG like this:T1 -> T2(1) -> T2(2) -> ..... -> T3 T2(n-1) -> T2(n)...
- 13179 Views
- 4 replies
- 1 kudos
- 1 kudos
@FlexException Databricks API supports job creation and execution Task Parameters and Values in Databricks Workflows | by Ryan Chynoweth | MediumOne possibility is after running earlier job, process the output to create a dynamic number of tasks in s...
- 1 kudos
- 3882 Views
- 1 replies
- 0 kudos
How to access Spark UI metrics in an automated way (API)
I am doing some automated testing; and would like ultimately to access per job/stage/task metrics as shown in the UI (e.g. spark UI -> sql dataframe) -> plan visualization in an automated way (API is ideal; but some ad-hoc metrics pipelines from loca...
- 3882 Views
- 1 replies
- 0 kudos
- 0 kudos
Thanks for the response. This enables the event logs. But the event logs seem to be empty. Would you know where I can get the spark metrics as seen from the spark ui.
- 0 kudos
- 7618 Views
- 7 replies
- 0 kudos
Trouble on Accessing Azure Storage from Databricks (Python)
I used the same accessing method shown in https://community.databricks.com/t5/data-engineering/to-read-data-from-azure-storage/td-p/32230 but kept get the error below.org.apache.spark.SparkSecurityException: [INSUFFICIENT_PERMISSIONS] Insufficient pr...
- 7618 Views
- 7 replies
- 0 kudos
- 0 kudos
Hi,you can find storage account firewall information by accessing resource in azure portal Please mind that if you are using Unity Catalog you should NOT mount Storage Account, you should rather use abstraction of Storage Creadentials and External Lo...
- 0 kudos
- 5325 Views
- 6 replies
- 0 kudos
can't execute the code
When I was executing the code, I was getting this error:"Notebook detached×Exception when creating execution context: java.net.SocketTimeoutException: Connect Timeout"Can someone help me?
- 5325 Views
- 6 replies
- 0 kudos
- 1973 Views
- 0 replies
- 0 kudos
raise Py4JJavaError while changing data type of a column
HiI'm using Azure databricks 10.4 LTS (includes Apache Spark 3.2.1, Scala 2.12). I'm trying to convert 2 columns from string data type to timestamp data type . My date columns are in below format2/18/2021 7:20:12 PMSo I wrote following commandfrom py...
- 1973 Views
- 0 replies
- 0 kudos
- 5304 Views
- 3 replies
- 0 kudos
Query example for databricks Query History API
Hi I am trying to get query history data from my SQL warehouse. Following previous examples is not working. databricks_workspace_url = "xxx"token = "xxx"start_time = 1707091200end_time = 1707174000api_endpoint = f"{databricks_workspace_url}/api/2.0/s...
- 5304 Views
- 3 replies
- 0 kudos
- 0 kudos
@Cheryl - you can use query_start_time=2023-01-01T00:00:00Z as a parameter to filter for the time frame. available filter criteria are given below - https://docs.databricks.com/api/workspace/queryhistory/list#filter_by-query_start_time_range
- 0 kudos
- 2067 Views
- 0 replies
- 0 kudos
CLI: Cannot Configure Additional Profiles in web terminal
Hello,I'm having a hard adding secrets since the change from the legacy CLI. Executing "databricks configure" as referenced in the personal access token section of this document does nothing. I have verified that the CLI is installed as v0.216.0.
- 2067 Views
- 0 replies
- 0 kudos
- 3417 Views
- 1 replies
- 0 kudos
Traversing to previous rows and getting the data based on condition
Sample Input data setClusterIdEventEventTime1212-18-r9u1kzn1RUNNING2024-02-02T11:38:30.168+00:001212-18-r9u1kzn1TERMINATING2024-02-02T13:43:33.933+00:001212-18-r9u1kzn1STARTING2024-02-02T15:50:05.174+00:001212-18-r9u1kzn1RUNNING2024-02-02T15:54:21.51...
- 3417 Views
- 1 replies
- 0 kudos
- 4542 Views
- 1 replies
- 2 kudos
6.4 Extended Support (includes Apache Spark 2.4.5, Scala 2.11 Connect Timeout
"Notebook detached Exception when creating execution context: java.net.SocketTimeout Exception: Connect Timeout" when trying to connect my cluster to a notebook. Then "Error trying to handle that request We failed to handle that request, please try a...
- 4542 Views
- 1 replies
- 2 kudos
- 2 kudos
Hello @Retired_mod I am facing same issue I tried changing DBR but it is still giving me error and the cluster is not startingRegardsMS
- 2 kudos
- 29503 Views
- 7 replies
- 3 kudos
Trying to use pdf2image on databricks
Trying to use pdf2image on databricks, but its failing with "PDFInfoNotInstalledError: Unable to get page count. Is poppler installed and in PATH?"I've installed pdf2image & poppler-utils by running the following in a cell:%pip install pdf2image%pip ...
- 29503 Views
- 7 replies
- 3 kudos
- 3 kudos
Seems like this thread has died, but for posterity, databricks provides the following code for installing poppler on a cluster. The code is sourced from the dbdemos accelerators, specifically the "LLM Chatbot With Retrieval Augmented Generation (RAG)...
- 3 kudos
- 12877 Views
- 8 replies
- 0 kudos
failed to initialise azure-event-hub with azure AAD(service principal)
We have been trying to authenticate azure-event-hub with azure AD(service principal) instead of shared access key(connection string) and read events from azure-event-hub and it is failing to initialise azure-event-hubs. And throwing no such method ex...
- 12877 Views
- 8 replies
- 0 kudos
- 0 kudos
@swathi-dataops I have added ServicePrincipalCredentialsAuth and ServicePrincipalAuthBase as a normal classes instead of creating a separate jar for these 2 classes and packaged them as a part of my project jar.And used the below code for configuring...
- 0 kudos
- 5698 Views
- 4 replies
- 0 kudos
MY exam got suspended
Hello Team,I encountered Pathetic experience while attempting my 1st Data Bricks certification. I was continuously in front of the camera and an alert appeared and then my exam resumed.Later a support person asked me to show the full room and I showe...
- 5698 Views
- 4 replies
- 0 kudos
- 0 kudos
Thanks @vinay076 You will receive notice of the reschedule via an email from Webassessor.
- 0 kudos
- 1450 Views
- 0 replies
- 0 kudos
Is Unity Catalog enabled in AWS Government
Hey Databricks afficiandos, does anyone know if AWS Government cloud has Unity Catalog enabled?
- 1450 Views
- 0 replies
- 0 kudos
- 2574 Views
- 0 replies
- 0 kudos
DLT table from a text source
I am trying to create a delta live table by reading a text source. I get an error message that states that both source and target should be in delta format. Am I missing something?
- 2574 Views
- 0 replies
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
