- 1798 Views
- 8 replies
- 1 kudos
Resolved! JDCB Error trying a get schemas call.
Hi Community,I have a free demo version and can create a jdbc connection and get metadata (schema, table, and columns structure info). Everything works as described in the docs, but when working with someone who has a paid version of databricks the s...
- 1798 Views
- 8 replies
- 1 kudos
- 1 kudos
@JeffSeaman , please let us know if any of my suggestions help get you on the right track. If they do, kindly mark the post as "Accepted Solution" so others can benefit as well. Cheers, Louis.
- 1 kudos
- 2037 Views
- 7 replies
- 14 kudos
Resolved! How to get pipeline update duration programmatically
I'm looking to track how much time is being spent running updates for my DLT pipelines.When querying the list pipeline updates REST API endpoint I can see start and end times being returned, however, these fields are not listed in the documentation. ...
- 2037 Views
- 7 replies
- 14 kudos
- 14 kudos
Originally went with the approach of exporting to and reading from the event log table, which has been helpful for getting other metrics as well.Also found today that there is a new system table is in public preview which exposes the durations I was ...
- 14 kudos
- 13204 Views
- 5 replies
- 0 kudos
How to Determine the Cost for Each Query Run Against SQL Warehouse Serverless?
Hello Everyone.First of all, I would like to thank you to databricks to enable system tables for customers. It does help a lot. I am working on cost optimization topic. Particularly sql warehouse serverless. I am not sure all of you have tried system...
- 13204 Views
- 5 replies
- 0 kudos
- 1514 Views
- 4 replies
- 2 kudos
Databricks Pipeline SDK misisng fields
Looking at the Databricks Java SDK for Pipeline events, I see that the Rest API returns a details field that has the same information as event log details. But this is not surfaced in SDK, should be a small change to add it. Is that something which c...
- 1514 Views
- 4 replies
- 2 kudos
- 2 kudos
The start and end time fields in the Pipeline Updates API are currently present in the Databricks REST API but are not yet supported (i.e., not included or mapped) in the Databricks Java SDK as of September 2025.This means:You can see these fields (s...
- 2 kudos
- 2260 Views
- 1 replies
- 2 kudos
Resolved! DLT or DataBricks for CDC and NRT
We are currently delivering a large-scale healthcare data migration project involving:One-time historical migration of approx. 80 TB of data, already completed and loaded into Delta Lake.CDC merge logic is already developed and validated using Apache...
- 2260 Views
- 1 replies
- 2 kudos
- 2 kudos
For cost-sensitive, large-scale healthcare data streaming scenarios, using Delta Live Tables (DLT) for both CDC and streaming (Option C) is generally the most scalable, manageable, and cost-optimized approach. DLT offers native support for structured...
- 2 kudos
- 3005 Views
- 1 replies
- 1 kudos
Resolved! Embedding Dashboards on Databricks Apps
Hi Team,I recently tried the Hello World template and embedded the <iframe> from the dashboard that I created. It works properly fine before I added some code to have a Login Form (I used Dash Plotly on creating the Login Form) before the dashboard a...
- 3005 Views
- 1 replies
- 1 kudos
- 1 kudos
in databricks, I recently tried the Hello World template and embedded the <iframe> from the dashboard that I created. It works properly fine before I added some code to have a Login Form (I used Dash Plotly on creating the Login Form) before the...
- 1 kudos
- 2266 Views
- 1 replies
- 1 kudos
Resolved! Assistance Required with Auto Liquid Clustering Implementation Challenges
Hi Databricks Team,We are currently implementing Auto Liquid Clustering (ALC) on our Delta tables as part of our data optimization efforts. During this process, we have encountered several challenges and would appreciate your guidance on best practic...
- 2266 Views
- 1 replies
- 1 kudos
- 1 kudos
To implement Auto Liquid Clustering (ALC) on Delta tables in Databricks, especially when transitioning from external partitioned tables to unpartitioned managed tables, a careful and ordered process is crucial to avoid data duplication and ensure con...
- 1 kudos
- 3121 Views
- 1 replies
- 1 kudos
Resolved! Understanding High I/O Wait Despite High CPU Utilization in system.compute Metrics
Hi everyone,I'm working on building a hardware metrics dashboard using the system.compute schema in Databricks, specifically leveraging the cluster, node_type, and node_timeline tables.While analyzing the data, I came across something that seems cont...
- 3121 Views
- 1 replies
- 1 kudos
- 1 kudos
Your observation highlights a subtlety in interpreting CPU metrics, especially in distributed environments like Databricks, where cluster and node-level behaviors can diverge from typical single-server intuition. Direct Answer No, seeing both high cp...
- 1 kudos
- 2028 Views
- 1 replies
- 0 kudos
Resolved! LakeFlow Connect->GA4 - creation of Liquid Clustered stream table
Hello While creating new Data Ingestion from GA4, can we set-up Liquid Clustering (either Manual or Automatical) on destination table which will contain fetched data from GA4?
- 2028 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes, in Databricks, it is possible to set up Liquid Clustering—both manual and automatic—on destination tables that store data ingested from Google Analytics 4 (GA4). This feature significantly improves table management and query performance compared...
- 0 kudos
- 630 Views
- 1 replies
- 0 kudos
Dabs Databricks asset bundles
Hi!I am relatively new to Dabs, but getting on quite well.I have managed to deploy both a job that uses a notebook defined in the bundle itself and a job that points to a notebook living in an azure devops git repo. While these are two viable solutio...
- 630 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @IONA ,you need to add a step into your CD pipeline to copy the notebook:- checkout: self- script: |cp path/to/notebook_in_repo/notebook.py .bundle/notebook.pydisplayName: 'Copy notebook into bundle'- script: |databricks bundle deploydisplayName: ...
- 0 kudos
- 4643 Views
- 2 replies
- 0 kudos
How to deal with deleted files in source directory in DLT?
We have a DLT pipeline that uses the autoloader to detect files added to a source storage bucket. It reads these updated files and adds new records to a bronze streaming table. However we would also like to automatically delete records from the bronz...
- 4643 Views
- 2 replies
- 0 kudos
- 1614 Views
- 2 replies
- 4 kudos
Resolved! How to invoke Databricks AI Assistant from a notebook cell?
Hello Community,I am exploring the Databricks AI Assistant and wondering if there is a way to invoke or interact with it directly from a notebook cell instead of using the workspace sidebar UI.Is there any built-in command (like %assistant) to open o...
- 1614 Views
- 2 replies
- 4 kudos
- 4 kudos
@shashankB There are no command like %assistant exists today to interact with Databricks Assistant. As @szymon_dybczak mentioned in the reply the exiting modes that you can interact with Assistant today.Also there is no published Assistant‑specific ...
- 4 kudos
- 3082 Views
- 1 replies
- 1 kudos
Resolved! How to Efficiently Sync MLflow Traces and Asynchronous User Feedback with a Delta Table
I’m building a custom UI table (using Next.js and FastAPI) to display MLflow trace data from a Retrieval-Augmented Generation (RAG) application running on Databricks Managed MLflow 3.0. The table needs to show answer generation speed (from CHAT_MODEL...
- 3082 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello! Here are the answers to your questions: - Yes! See databricks managed mlflow tracing - enable production monitor or endpoint config to collect traces in a delta table - We have example code for implementing async feedback collection - Definit...
- 1 kudos
- 7032 Views
- 3 replies
- 2 kudos
SQL While do loops
Hello. Could you please suggest a workaround for a while do loop in Databricks SQL?WHILE LSTART>0 DO SET LSTRING=CONCAT(LSTRING, VSTRING2)Thank you in advance
- 7032 Views
- 3 replies
- 2 kudos
- 2 kudos
@elgeo Here are two alternatives.1. Use a recursive CTEWITH RECURSIVE loop_cte (lstart, lstring) AS ( SELECT 5 AS lstart, '' AS lstring UNION ALL SELECT lstart - 1, CONCAT(lstring, 'VSTRING2') FROM loop_cte WHERE lstart > 1 ) SELECT * FROM ...
- 2 kudos
- 802 Views
- 1 replies
- 1 kudos
SQL Stored Procedure in Databricks
Hello, Is there a sql server equivalent stored procedure in Databricks which supports while loop along with delay as below. or are there any other alternative to achieve the same.while (select count(*) from schema.mart_daily with (nolock)) = 0 begin ...
- 802 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @ashishasr ,Yes, support for stored procedures is in public preview:CREATE PROCEDURE | Databricks on AWSIn the definition of stored procedure you can use compound statement ( SQL compound statement (BEGIN ... END) with the definition of the SQL Pr...
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 554 | |
| 349 | |
| 287 |
