Hello,For the purposes of testing I'm interested in creating a new workspace with Unity Catalog enabled, and from there I'd like to access (external - S3) tables on an existing legacy hive metastore workspace (not UC enabled). The goal is for both wo...
@Kaniz From looking at the documentation none address my particular use case which I illustrated (2 workspaces on one account, 1 with UC and the other not). Was there a particular part on any of the docs you're suggesting can help here?
I used the same accessing method shown in https://community.databricks.com/t5/data-engineering/to-read-data-from-azure-storage/td-p/32230 but kept get the error below.org.apache.spark.SparkSecurityException: [INSUFFICIENT_PERMISSIONS] Insufficient pr...
Hi,you can find storage account firewall information by accessing resource in azure portal Please mind that if you are using Unity Catalog you should NOT mount Storage Account, you should rather use abstraction of Storage Creadentials and External Lo...
Hi all,I am trying the new "git folder" feature, with a repo that works fine from the "Repos". In the new folder location, my imports from my own repo don't work anymore. Anyone faced something similar?Thanks in advance for sharing your experience
Hi @Kibour,
Ensure that the Databricks Repos feature is enabled. Sometimes, this issue arises when Repos are not properly activated.Additionally, verify that Repos are allowed for DBR 8.4+ versions.Confirm that your folder structure adheres to the c...
HiI'm using Azure databricks 10.4 LTS (includes Apache Spark 3.2.1, Scala 2.12). I'm trying to convert 2 columns from string data type to timestamp data type . My date columns are in below format2/18/2021 7:20:12 PMSo I wrote following commandfrom py...
Hi @Marinagomes,
Try Using try_to_timestamp: Instead of to_timestamp, consider using try_to_timestamp. It returns null for malformed expressions, which can help identify problematic rows.
I am doing some automated testing; and would like ultimately to access per job/stage/task metrics as shown in the UI (e.g. spark UI -> sql dataframe) -> plan visualization in an automated way (API is ideal; but some ad-hoc metrics pipelines from loca...
Hi @superspan,
Event logs provide detailed information about Spark application execution, including task-level data.
Unfortunately, directly configuring spark.eventLog.dir in a Databricks notebook won’t work. Instead, follow these steps:
Step 1: ...
When I was executing the code, I was getting this error:"Notebook detached×Exception when creating execution context: java.net.SocketTimeoutException: Connect Timeout"Can someone help me?
Problem StatementCluster 1 (Shared Cluster) is not able to read the file location at "dbfs:/mnt/landingzone/landingzonecontainer/Inbound/" and hence we are not able to create an external table in a schema inside Enterprise Catalog.Cluster 2 (No Isola...
Hi @Meshynix,Can you provide the code snippet you execute to create your tables, this would give us a better insight for both use cases. Also can you provide the error that is being returned in the first use case. This would help a lot.
Hi I am trying to get query history data from my SQL warehouse. Following previous examples is not working. databricks_workspace_url = "xxx"token = "xxx"start_time = 1707091200end_time = 1707174000api_endpoint = f"{databricks_workspace_url}/api/2.0/s...
@Cheryl - you can use query_start_time=2023-01-01T00:00:00Z as a parameter to filter for the time frame. available filter criteria are given below - https://docs.databricks.com/api/workspace/queryhistory/list#filter_by-query_start_time_range
Hello, I have a question.Context :I have a Unity Catalog organized with three schemas (bronze, silver and gold). Logically, I would like to create tables in each schemas.I tried to organize my pipelines on the layers, which mean that I would like to ...
Hey @AxelBrsn,Unfortunately this is a limitation with DLT as far as my experience goes. You should organize the pipelines in a way that they encompass the full Bronze/Silver/Gold flow, since you don't have control over the schema if you want to make ...
Hello everybody.Whenever I am trying to run a simple cell I receive the following error message now:Notebook detached. Exception when creating expectation context: java.net.SocketTimeoutException: Connect Timeout.After that error message the cluster ...
Hello,I'm having a hard adding secrets since the change from the legacy CLI. Executing "databricks configure" as referenced in the personal access token section of this document does nothing. I have verified that the CLI is installed as v0.216.0.
facing issue with integrating our Spring boot JPA supported application with Databricks.Below are the steps and setting we did for the integration.When we are starting the spring boot application we are getting a warning as :HikariPool-1 - Driver doe...
Hi @satishnavik, It seems you’re encountering issues while integrating your Spring Boot JPA application with Databricks.
Let’s address the warnings and exceptions you’re facing.
Warning: Driver Does Not Support Network Timeout for Connections
The...
Sample Input data setClusterIdEventEventTime1212-18-r9u1kzn1RUNNING2024-02-02T11:38:30.168+00:001212-18-r9u1kzn1TERMINATING2024-02-02T13:43:33.933+00:001212-18-r9u1kzn1STARTING2024-02-02T15:50:05.174+00:001212-18-r9u1kzn1RUNNING2024-02-02T15:54:21.51...
Hi @RajNath , Handling event times and aggregations in large datasets can be challenging, but Structured Streaming in Databricks provides powerful tools to address this.
Let’s break down your requirements and explore how you can achieve them:
Ru...
I am new to databricks delta sharing. In case of delta sharing, i don't see any cluster running. Tried looking for documentation but only hint i got is, it usage delta sharing server but what is the cost of it and how to configure and optimize for la...
Hi @RajNath, Let’s dive into the world of Delta Sharing and explore how it works, its cost implications, and optimization strategies.
What is Delta Sharing?
Delta Sharing is a secure data-sharing platform developed by Databricks. It allows you to ...
"Notebook detached Exception when creating execution context: java.net.SocketTimeout Exception: Connect Timeout" when trying to connect my cluster to a notebook. Then "Error trying to handle that request We failed to handle that request, please try a...