- 2956 Views
- 4 replies
- 2 kudos
Resolved! Using built-in display method modules
The builtin `display` function is very helpful. but we're moving code from notebooks into python modules.Here, it seems that `display` is defined differently which results in poor visualization.Example:```df = spark.createDataFrame([{'x': 1}])display...
- 2956 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @holunder42, The behavior you are seeing is expected. The display() function is not a standard Python built-in. It is injected into the notebook's global namespace by the Databricks runtime when a notebook cell executes. When you move code into an...
- 2 kudos
- 2045 Views
- 7 replies
- 1 kudos
Resolved! Genie generates MEASURE expression with "filter" clause
Genie generated a query against a metric view that introduces a "filter" clause as a second parameter to MEASURE:SELECT `countryName`, MEASURE(`deviceCount`, `isActive` = true) AS `online`, MEASURE(`deviceCount`, `isActive` = false) AS `offline...
- 2045 Views
- 7 replies
- 1 kudos
- 1 kudos
Hi @Malthe, The behavior you are seeing is indeed the LLM generating invalid SQL syntax. The MEASURE() function takes exactly one argument, which is a reference to a measure column defined in a metric view. There is no second "filter" parameter, and ...
- 1 kudos
- 1576 Views
- 2 replies
- 0 kudos
- 1576 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @peterlewis, Databricks notebooks support LaTeX mathematical notation inside %md (markdown) cells. The rendering engine uses MathJax, so standard LaTeX math syntax works. Here is a rundown of how to use it and some common patterns. INLINE MATH Wra...
- 0 kudos
- 872 Views
- 2 replies
- 0 kudos
How can i get enhanced autoscalar for job clusters
Is the enhanced autoscalar https://docs.databricks.com/aws/en/ldp/auto-scaling#what-is-enhanced-autoscaling available for job clusters as well?I do not see the option to set the autoscaling mode for job clusters
- 872 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @aranjan99, There are two distinct autoscaling features in Databricks, and it helps to clarify the difference since the naming can be confusing. OPTIMIZED AUTOSCALING (for job and all-purpose clusters) If your workspace is on the Premium plan (or ...
- 0 kudos
- 2254 Views
- 4 replies
- 0 kudos
Resolved! LIKE Operator Fails with COLLATE UNICODE_CI Columns
Hi All,We are experiencing problem while we are querying the table which has collate unicode_ci attribute enabled, Kindly find the below findings and let me know if any alternative approach to fix this issue?Problem StatementThe LIKE operator in a q...
- 2254 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi, This is a known limitation in Databricks SQL. The LIKE operator only supports a subset of collations, and UNICODE_CI is NOT one of them. WHICH COLLATIONS DOES LIKE SUPPORT? Per the official LIKE documentation, the supported collations are limited...
- 0 kudos
- 1468 Views
- 2 replies
- 2 kudos
Resolved! Databricks UI and Backend Deployment Issue
Hi everyone,I’m deploying a frontend (Streamlit) and backend (FastAPI) as two separate Databricks Apps within the same workspace, both with user authentication enabled.The frontend makes a server-side HTTP request to the backend app URL when a user s...
- 1468 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @MrChromatic, The behavior you are seeing is expected. When your Streamlit frontend app makes an HTTP request to your FastAPI backend app URL, that request goes through the Databricks authentication proxy just like any browser request would. Since...
- 2 kudos
- 2701 Views
- 8 replies
- 1 kudos
Resolved! Issue on Service Credential creation for Azure Databricks access connector
Question: Why I'm getting the following error and how can we fix it?In step 6 of Create service credentials - Azure Databricks | Microsoft Learn when I enter the resource id of my Azure Databricks access connector, I get the following error:/subscrip...
- 2701 Views
- 8 replies
- 1 kudos
- 1 kudos
Hi @Saf4Databricks, The error message is the key clue here. When you enter the Azure Access Connector resource ID and get back: "is not a valid IAM role ARN. Valid ARNs normally look like arn:aws:iam::<account>:role/<iam-role-name>" This tells you th...
- 1 kudos
- 1324 Views
- 4 replies
- 1 kudos
Are streaming tables suitable for Gold layer Star schema?
Based on docs, we can't use identity columns and ANALYZE TABLE on streaming tables. So, should we avoid using streaming tables for Gold layer Star schema?https://docs.databricks.com/aws/en/ldp/developer/ldp-sql-ref-create-streaming-table#limitations
- 1324 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @yit337, You are on the right track noticing those limitations. The short answer is: for a Gold layer star schema, materialized views are generally the better fit, though streaming tables are not completely ruled out depending on the specific tabl...
- 1 kudos
- 596 Views
- 1 replies
- 0 kudos
Streaming Table data leakage to historical permanent table
Data Leakage in Historical Table from Streaming TableEnvironmentPlatform: Azure Databricks + Azure Event HubsStreaming Framework: Spark Structured StreamingStorage: Delta LakePipeline Event Hubs → stream_messages (live 24hr rolling window) → message...
- 596 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Akash_Varuna, The count discrepancies you are seeing between stream_messages and messages are almost certainly caused by the 24-hour rolling window on your stream_messages table expiring data while the load_messages job is paused during your main...
- 0 kudos
- 4470 Views
- 2 replies
- 1 kudos
Resolved! How to stop task retry
I would like to stop automatic retries, but the max retries configuration does not seem to work.Could you please tell me how to disable retries? I would also like to understand why the task retries automatically.I did not set any scheduler. I created...
- 4470 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @senkii, There are two separate retry mechanisms in Databricks that can cause tasks to run again, and distinguishing between them is important for your situation. 1. TASK-LEVEL RETRIES (Workflows setting) This is the "Retries" setting you configur...
- 1 kudos
- 1550 Views
- 2 replies
- 0 kudos
Resolved! Zerobus Kafka-compatible API
Hi Team,I went through a recording where it was mentioned that a Kafka‑compatible API is planned for a Beta release in Q1. Do we have any rough timeline on when this feature might be available?We already have Kafka producer topics, and we would like ...
- 1550 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @developer3535, I see @stbjelcevic already confirmed the Q1 2026 timeline for the Kafka-compatible API Beta. I wanted to add some context on what you can do in the meantime and where to look for updates. CURRENT ZEROBUS INGEST INTERFACES While wai...
- 0 kudos
- 3271 Views
- 2 replies
- 2 kudos
Architecture Advice: DLT Strategy for Daily Snapshots to SCD2 with "Grace Period" Deletes
Hi everyone,I’m looking for architectural advice on building a Silver layer in DLT. I am dealing with inventory data and need to handle historical tracking, "sold" logic based on disappearance, and storage cost optimization.Here's how the situation l...
- 3271 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @samuelperezh, Building on @aleksandra_ch's reply, I wanted to add some additional detail around each of your three questions, especially around the grace period implementation and the backfill strategy. 1. GRACE PERIOD PATTERN As aleksandra_ch no...
- 2 kudos
- 1975 Views
- 2 replies
- 1 kudos
how does Job cluster auto scaling work
Can you share the metrics databricks uses during job cluster auto scaling?Is Databricks looking at queued tasks, slot utilization etc or just looking at CPU utilizations?The autoscaling docuemnt https://docs.databricks.com/aws/en/compute/configure?u...
- 1975 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @aranjan99, The autoscaling behavior on job clusters depends on your workspace pricing tier. Here is a breakdown of the metrics and mechanics involved. WHAT METRICS DRIVE SCALING DECISIONS Job cluster autoscaling uses Spark scheduler signals, not ...
- 1 kudos
- 1542 Views
- 2 replies
- 1 kudos
Resolved! Identity column has null values
I want to update a dimension table in the gold model from a silver table by using create_auto_cdc_from_snapshot_flow and SCD2. In the target table, I have defined an IDENTITY column, which should be populated automatically.The dlt flow runs successf...
- 1542 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @yit337, The reason your identity column values are NULL is that the target table created by create_auto_cdc_from_snapshot_flow is a streaming table, and streaming tables do not support identity columns. This is a documented limitation: https://do...
- 1 kudos
- 3557 Views
- 4 replies
- 1 kudos
Resolved! How to use multi-threading and batch inserts for large UPSERT to PostgreSQL from Databricks?
Hi everyone,We have a Databricks (Unity Catalog) pipeline where we process large datasets in Spark and need to load incremental data into a PostgreSQL target table.Our scenario is:Initial full load (~300 million rows) to PostgreSQL using bulk COPY is...
- 3557 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @Saikumar_Manne, With 190M+ daily rows going into PostgreSQL via INSERT ON CONFLICT DO UPDATE, there are several levers to pull. Here is a breakdown of the approaches and tuning options. APPROACH 1: STAGING TABLE + MERGE (RECOMMENDED FOR THIS VOLU...
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
