- 1344 Views
- 2 replies
- 0 kudos
Resolved! How to auto‑terminate DLT-managed clusters after pipeline execution?
We have Data bricks Jobs that run a combination of DLT pipelines and notebook tasks.For the notebook tasks, we use a job cluster, and we are able to auto‑terminate it after execution using auto termination: 10However, the DLT-managed clusters behave ...
- 1344 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @anusha98, The behavior you are seeing, where the cluster stays running for a long time after pipeline completion, is almost certainly because your pipeline is running in Development mode. In Development mode, the default cluster shutdown delay is...
- 0 kudos
- 650 Views
- 2 replies
- 1 kudos
Manager hierarchy
for employee manager hierarchy some employees are not having correct manager mapping, what can be the reason?
- 650 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @spjti, Incorrect manager mappings in an employee hierarchy typically come from a few common root causes. Since the question is fairly open-ended, here is a checklist of the most likely issues and how to diagnose them in Databricks. COMMON REASONS...
- 1 kudos
- 1407 Views
- 6 replies
- 0 kudos
Same Delta Table, Different Behavior: Dev vs Prod Workspace in Databricks
I recently ran into an interesting Databricks behavior while implementing a row-count comparison using Delta Time Travel (VERSION AS OF).Platform: AzureScenario:Same Unity CatalogSame fully qualified tableSame table ID, location, and Delta formatYet ...
- 1407 Views
- 6 replies
- 0 kudos
- 0 kudos
Hi @Smriti2, The error "TIME TRAVEL is not allowed. Operation not supported on Streaming Tables" when the table is not intentionally a streaming table typically points to the table having pipeline-related metadata attached to it in the prod workspace...
- 0 kudos
- 1743 Views
- 4 replies
- 0 kudos
Databricks Dataframe: Merge exception for string column in delta table with change detectio enabled.
Hi,I am trying to append change data records into delta table using dataframe. However, I am encountering an issue while appending new records to existing data in delta table. The exception raised isCannot merge columns 'ColumnX' into 'ColumnX'Both n...
- 1743 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @crami, The "Cannot merge columns 'ColumnX' into 'ColumnX'" error you are seeing is caused by passing "delta.enableChangeDataFeed" as a write option on the DataFrame writer. Change Data Feed (CDF) is a table-level property, not a per-write option....
- 0 kudos
- 1189 Views
- 1 replies
- 0 kudos
Inconsistent UNRESOLVED_COLUMN._metadata error on Serverless compute during MERGE operations
Hi.I've been facing this problem for the last couple of days.We're experiencing intermittent failures with the error [UNRESOLVED_COLUMN.WITHOUT_SUGGESTION] A column, variable, or function parameter with name '_metadata' cannot be resolved. SQLSTATE: ...
- 1189 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @ruicarvalho_de, The behavior you are seeing is related to how the _metadata virtual column interacts with Spark Connect (which Serverless compute uses under the hood) and how deferred analysis changes when that hidden column is resolved. Here is ...
- 0 kudos
- 2348 Views
- 5 replies
- 0 kudos
databricks list job runs api limit doesnt work
noticed that the limit parameter on the list job runs api: https://docs.databricks.com/api/workspace/jobs/listrunsdoesnt work. no matter what limit i set, i always get all the runs which can be too many if i want all runs.If i set offset as > 1, then...
- 2348 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi @aranjan99, To add some detail to the earlier reply: the confusion here comes from the fact that the REST API and the Databricks CLI handle the "limit" parameter differently. REST API BEHAVIOR In the REST API (POST or GET to /api/2.1/jobs/runs/lis...
- 0 kudos
- 813 Views
- 3 replies
- 2 kudos
Access Lab
Hi Team,How do I access lab? Do I need to take subscription?
- 813 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @ShyamalaT, As @Advika and @szymon_dybczak mentioned, the hands-on labs that accompany Databricks Academy courses do require either an Instructor-Led Training (ILT) enrollment or a Databricks Academy Labs subscription. Here is a quick summary of y...
- 2 kudos
- 1192 Views
- 4 replies
- 0 kudos
Row count is higher than expected after breaking cases into 15-minute segments
I’m building a fact table in Databricks where the grain is one row per case per 15-minute segment. For example, if a case lasts 90 minutes, I expect about 6 rows.I generate the 15-minute segments and MERGE into a Delta table (incremental load). Howev...
- 1192 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @CuriousEngineer, This is a common scenario, and there are good solutions. Nice job including the code and the minimal reproducible example -- that makes it much easier to help. There are two places in your query that are very likely introducing d...
- 0 kudos
- 5418 Views
- 8 replies
- 2 kudos
Resolved! Continuous Job - How to set max_retries
Hello Community,I have a couple of continuous workflows (jobs) running in production, and they’ve been working well so far. However, we’re seeing some transient failures that are causing the entire job to restart — which I’d prefer to avoid.While we ...
- 5418 Views
- 8 replies
- 2 kudos
- 2 kudos
Hi @Kirankumarbs, Thanks for posting this. I can see from the thread that the existing replies confirmed the limitation but did not offer much in the way of alternatives. Let me provide a fuller picture. THE SHORT ANSWER You cannot set a custom max_r...
- 2 kudos
- 2282 Views
- 3 replies
- 4 kudos
Resolved! Where is Lakeflow Designer in the databricks UI
Hello everyone,Yesterday I saw a demo of Lakeflow Designer, and it looked very interesting. I tried to find this option in the UI, but I was not able to locate it.Could anyone please guide me on where I can access or try out this feature?I would real...
- 2282 Views
- 3 replies
- 4 kudos
- 4 kudos
If you are talking about the No-code version that is described here, then that is still in Private Preview as far as I am aware. Talk to your Databricks Account if you want to opt into the preview. If you do get access to it, you should be able to fi...
- 4 kudos
- 2394 Views
- 2 replies
- 1 kudos
Resolved! Share dashboard with external customers
I’m new to Databricks and still exploring the platform. One of our requirements is to share a Databricks dashboard with multiple external customers (for example, Customer 1 and Customer 2).The key requirement is data isolation:When a user from Custom...
- 2394 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @RJ1 - Welcome to Databricks Platform! Great question. This is a very common requirement. At a high level, you’ll want to separate the concerns: How users authenticate and are identified (who is the user / which customer are they from?)How data i...
- 1 kudos
- 1344 Views
- 3 replies
- 2 kudos
Resolved! Data governance solution
I am here looking for Data governance solution for organization. I also searched this on many different website and found many solutions. We are bit confused to which one to choose. One of my friend suggest me this platform, and i am hoping i will ge...
- 1344 Views
- 3 replies
- 2 kudos
- 2 kudos
You can use Unity Catalog for Databricks Lakehouse. You can use Collibra/Open Metadata along with Unity Catalog for complete governance
- 2 kudos
- 1034 Views
- 2 replies
- 4 kudos
Resolved! Looking for resources to learn Databricks
Hi Community Members,I'm working as a Power BI Developer and interested to upskill into Databricks platform as a Data Analyst and Data Engineering.Request to share the resources(documentation/video tutorials) in a sequential order.Thank You!Best Rega...
- 1034 Views
- 2 replies
- 4 kudos
- 4 kudos
For Bite-size overviews check the demo center - https://www.databricks.com/resources/demos/library This youtube channel is great for more detailed oriented discussion around specific features .https://www.youtube.com/@nextgenlakehouseFor more structu...
- 4 kudos
- 10782 Views
- 6 replies
- 2 kudos
Resolved! Understanding Autoscaling in Databricks: Under What Conditions Does Spark Add a New Worker Node?
I’m currently working with Databricks autoscaling configurations and trying to better understand how Spark decides when to spin up additional worker nodes. My cluster has a minimum of one worker and can scale up to five. I know that tasks are assigne...
- 10782 Views
- 6 replies
- 2 kudos
- 2 kudos
Is the above information true for job clusters as well? Looks like the enhanced auto scalar is only available for pipelines
- 2 kudos
- 594 Views
- 2 replies
- 3 kudos
Resolved! Am i publishing article in a correct way or not?
Hello Community,I’d like to check with the contributors whether the article I recently published follows the correct approach. Did I choose the right options and the appropriate place to publish it in the Databricks Community?https://community.databr...
- 594 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Kirankumarbs ,Yes, you did everything in correct manner. You put your article in correct place which is "Community Articles".Anyway, thanks for sharing with us
- 3 kudos
