cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

pSdatabricks
by New Contributor II
  • 1805 Views
  • 3 replies
  • 0 kudos

Azure Databricks Monitoring & Alerting (Data Observability) Tools / Frameworks for Enterprise

I am trying to evaluate options for Monitoring and Alerting tools like New Relic, Datadog, Grafana with Databricks on Azure . No one supports when reached out to them. I would like to hear from the databricks team on the recommended tool / framework ...

  • 1805 Views
  • 3 replies
  • 0 kudos
Latest Reply
Sruthivika
Visitor
  • 0 kudos

I'd recommend this new tool we've been trying out. It's really helpful for monitoring and provides good insights on how Azure Databricks clusters, pools & jobs are doing – like if they're healthy or having issues. It brings everything together, makin...

  • 0 kudos
2 More Replies
Meshynix
by New Contributor II
  • 300 Views
  • 4 replies
  • 0 kudos

Not able to create external table in a schema under a Catalog.

Problem StatementCluster 1 (Shared Cluster) is not able to read the file location at "dbfs:/mnt/landingzone/landingzonecontainer/Inbound/" and hence we are not able to create an external table in a schema inside Enterprise Catalog.Cluster 2 (No Isola...

  • 300 Views
  • 4 replies
  • 0 kudos
Latest Reply
YuliyanBogdanov
New Contributor
  • 0 kudos

Hi @Meshynix,Can you provide the code snippet you execute to create your tables, this would give us a better insight for both use cases. Also can you provide the error that is being returned in the first use case. This would help a lot.

  • 0 kudos
3 More Replies
FlexException
by New Contributor II
  • 1090 Views
  • 5 replies
  • 0 kudos

Dynamic Number of Tasks in Databricks Workflow

Do Databricks workflows support creating a workflow with a dynamic number of tasks?For example, let's say we have a DAG like this:T1 ->    T2(1) ->             T2(2) ->              .....                 -> T3             T2(n-1) ->             T2(n)...

  • 1090 Views
  • 5 replies
  • 0 kudos
Latest Reply
tanyeesern
Visitor
  • 0 kudos

@FlexException Databricks API supports job creation and execution Task Parameters and Values in Databricks Workflows | by Ryan Chynoweth | MediumOne possibility is after running earlier job, process the output to create a dynamic number of tasks in s...

  • 0 kudos
4 More Replies
superspan
by Visitor
  • 73 Views
  • 2 replies
  • 0 kudos

How to access Spark UI metrics in an automated way (API)

I am doing some automated testing; and would like ultimately to access per job/stage/task metrics as shown in the UI (e.g. spark UI -> sql dataframe) -> plan visualization in an automated way (API is ideal; but some ad-hoc metrics pipelines from loca...

  • 73 Views
  • 2 replies
  • 0 kudos
Latest Reply
superspan
Visitor
  • 0 kudos

Thanks for the response. This enables the event logs. But the event logs seem to be empty. Would you know where I can get the spark metrics as seen from the spark ui.

  • 0 kudos
1 More Replies
hossein_kolahdo
by New Contributor II
  • 154 Views
  • 2 replies
  • 0 kudos

Accessing data from a legacy hive metastore workspace on a new Unity Catalog workspace

Hello,For the purposes of testing I'm interested in creating a new workspace with Unity Catalog enabled, and from there I'd like to access (external - S3) tables on an existing legacy hive metastore workspace (not UC enabled). The goal is for both wo...

Data Engineering
hivemetastore
unitycatalog
Workspaces
  • 154 Views
  • 2 replies
  • 0 kudos
Latest Reply
hossein_kolahdo
New Contributor II
  • 0 kudos

@Kaniz From looking at the documentation none address my particular use case which I illustrated (2 workspaces on one account, 1 with UC and the other not). Was there a particular part on any of the docs you're suggesting can help here?

  • 0 kudos
1 More Replies
Geoff123
by New Contributor II
  • 198 Views
  • 8 replies
  • 0 kudos

Trouble on Accessing Azure Storage from Databricks (Python)

I used the same accessing method shown in https://community.databricks.com/t5/data-engineering/to-read-data-from-azure-storage/td-p/32230 but kept get the error below.org.apache.spark.SparkSecurityException: [INSUFFICIENT_PERMISSIONS] Insufficient pr...

  • 198 Views
  • 8 replies
  • 0 kudos
Latest Reply
Wojciech_BUK
Contributor III
  • 0 kudos

Hi,you can find storage account firewall information by accessing resource in azure portal Please mind that if you are using Unity Catalog you should NOT mount Storage Account, you should rather use abstraction of Storage Creadentials and External Lo...

  • 0 kudos
7 More Replies
Kibour
by New Contributor II
  • 482 Views
  • 1 replies
  • 0 kudos

Import from repo

Hi all,I am trying the new "git folder" feature, with a repo that works fine from the "Repos". In the new folder location, my imports from my own repo don't work anymore. Anyone faced something similar?Thanks in advance for sharing your experience

  • 482 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @Kibour,  Ensure that the Databricks Repos feature is enabled. Sometimes, this issue arises when Repos are not properly activated.Additionally, verify that Repos are allowed for DBR 8.4+ versions.Confirm that your folder structure adheres to the c...

  • 0 kudos
Marinagomes
by Visitor
  • 138 Views
  • 1 replies
  • 0 kudos

raise Py4JJavaError while changing data type of a column

HiI'm using Azure databricks 10.4 LTS (includes Apache Spark 3.2.1, Scala 2.12). I'm trying to convert 2 columns from string data type to timestamp data type . My date columns are in below format2/18/2021 7:20:12 PMSo I wrote following commandfrom py...

  • 138 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @Marinagomes,  Try Using try_to_timestamp: Instead of to_timestamp, consider using try_to_timestamp. It returns null for malformed expressions, which can help identify problematic rows.

  • 0 kudos
databrick53
by New Contributor
  • 387 Views
  • 6 replies
  • 0 kudos

can't execute the code

When I was executing the code, I was getting this error:"Notebook detached×Exception when creating execution context: java.net.SocketTimeoutException: Connect Timeout"Can someone help me?

  • 387 Views
  • 6 replies
  • 0 kudos
Latest Reply
toolhater
New Contributor
  • 0 kudos

as of last night 3/27 it looks like it was working again

  • 0 kudos
5 More Replies
Cheryl
by New Contributor
  • 130 Views
  • 3 replies
  • 0 kudos

Query example for databricks Query History API

Hi I am trying to get query history data from my SQL warehouse. Following previous examples is not working. databricks_workspace_url = "xxx"token = "xxx"start_time = 1707091200end_time = 1707174000api_endpoint = f"{databricks_workspace_url}/api/2.0/s...

  • 130 Views
  • 3 replies
  • 0 kudos
Latest Reply
shan_chandra
Honored Contributor III
  • 0 kudos

@Cheryl - you can use query_start_time=2023-01-01T00:00:00Z  as a parameter to filter for the time frame. available filter criteria are given below - https://docs.databricks.com/api/workspace/queryhistory/list#filter_by-query_start_time_range    

  • 0 kudos
2 More Replies
AxelBrsn
by New Contributor II
  • 109 Views
  • 2 replies
  • 0 kudos

Use DLT from another pipeline

Hello, I have a question.Context :I have a Unity Catalog organized with three schemas (bronze, silver and gold). Logically, I would like to create tables in each schemas.I tried to organize my pipelines on the layers, which mean that I would like to ...

  • 109 Views
  • 2 replies
  • 0 kudos
Latest Reply
YuliyanBogdanov
New Contributor
  • 0 kudos

Hey @AxelBrsn,Unfortunately this is a limitation with DLT as far as my experience goes. You should organize the pipelines in a way that they encompass the full Bronze/Silver/Gold flow, since you don't have control over the schema if you want to make ...

  • 0 kudos
1 More Replies
HelloDatabricks
by New Contributor
  • 344 Views
  • 5 replies
  • 8 kudos

Connect Timeout - Error when trying to run a cell

Hello everybody.Whenever I am trying to run a simple cell I receive the following error message now:Notebook detached. Exception when creating expectation context: java.net.SocketTimeoutException: Connect Timeout.After that error message the cluster ...

  • 344 Views
  • 5 replies
  • 8 kudos
Latest Reply
MarijaS
New Contributor
  • 8 kudos

today is ok

  • 8 kudos
4 More Replies
satishnavik
by New Contributor II
  • 593 Views
  • 3 replies
  • 0 kudos

How to connect Databricks Database with Springboot application using JPA

facing issue with integrating our Spring boot JPA supported application with Databricks.Below are the steps and setting we did for the integration.When we are starting the spring boot application we are getting a warning as :HikariPool-1 - Driver doe...

  • 593 Views
  • 3 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @satishnavik, It seems you’re encountering issues while integrating your Spring Boot JPA application with Databricks. Let’s address the warnings and exceptions you’re facing. Warning: Driver Does Not Support Network Timeout for Connections The...

  • 0 kudos
2 More Replies