- 1118 Views
- 6 replies
- 0 kudos
Same Delta Table, Different Behavior: Dev vs Prod Workspace in Databricks
I recently ran into an interesting Databricks behavior while implementing a row-count comparison using Delta Time Travel (VERSION AS OF).Platform: AzureScenario:Same Unity CatalogSame fully qualified tableSame table ID, location, and Delta formatYet ...
- 1118 Views
- 6 replies
- 0 kudos
- 0 kudos
Hi @Smriti2, The error "TIME TRAVEL is not allowed. Operation not supported on Streaming Tables" when the table is not intentionally a streaming table typically points to the table having pipeline-related metadata attached to it in the prod workspace...
- 0 kudos
- 1334 Views
- 4 replies
- 0 kudos
Databricks Dataframe: Merge exception for string column in delta table with change detectio enabled.
Hi,I am trying to append change data records into delta table using dataframe. However, I am encountering an issue while appending new records to existing data in delta table. The exception raised isCannot merge columns 'ColumnX' into 'ColumnX'Both n...
- 1334 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @crami, The "Cannot merge columns 'ColumnX' into 'ColumnX'" error you are seeing is caused by passing "delta.enableChangeDataFeed" as a write option on the DataFrame writer. Change Data Feed (CDF) is a table-level property, not a per-write option....
- 0 kudos
- 808 Views
- 1 replies
- 0 kudos
Inconsistent UNRESOLVED_COLUMN._metadata error on Serverless compute during MERGE operations
Hi.I've been facing this problem for the last couple of days.We're experiencing intermittent failures with the error [UNRESOLVED_COLUMN.WITHOUT_SUGGESTION] A column, variable, or function parameter with name '_metadata' cannot be resolved. SQLSTATE: ...
- 808 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @ruicarvalho_de, The behavior you are seeing is related to how the _metadata virtual column interacts with Spark Connect (which Serverless compute uses under the hood) and how deferred analysis changes when that hidden column is resolved. Here is ...
- 0 kudos
- 1856 Views
- 5 replies
- 0 kudos
databricks list job runs api limit doesnt work
noticed that the limit parameter on the list job runs api: https://docs.databricks.com/api/workspace/jobs/listrunsdoesnt work. no matter what limit i set, i always get all the runs which can be too many if i want all runs.If i set offset as > 1, then...
- 1856 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi @aranjan99, To add some detail to the earlier reply: the confusion here comes from the fact that the REST API and the Databricks CLI handle the "limit" parameter differently. REST API BEHAVIOR In the REST API (POST or GET to /api/2.1/jobs/runs/lis...
- 0 kudos
- 544 Views
- 3 replies
- 2 kudos
Access Lab
Hi Team,How do I access lab? Do I need to take subscription?
- 544 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @ShyamalaT, As @Advika and @szymon_dybczak mentioned, the hands-on labs that accompany Databricks Academy courses do require either an Instructor-Led Training (ILT) enrollment or a Databricks Academy Labs subscription. Here is a quick summary of y...
- 2 kudos
- 955 Views
- 4 replies
- 0 kudos
Row count is higher than expected after breaking cases into 15-minute segments
I’m building a fact table in Databricks where the grain is one row per case per 15-minute segment. For example, if a case lasts 90 minutes, I expect about 6 rows.I generate the 15-minute segments and MERGE into a Delta table (incremental load). Howev...
- 955 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @CuriousEngineer, This is a common scenario, and there are good solutions. Nice job including the code and the minimal reproducible example -- that makes it much easier to help. There are two places in your query that are very likely introducing d...
- 0 kudos
- 4061 Views
- 8 replies
- 2 kudos
Resolved! Continuous Job - How to set max_retries
Hello Community,I have a couple of continuous workflows (jobs) running in production, and they’ve been working well so far. However, we’re seeing some transient failures that are causing the entire job to restart — which I’d prefer to avoid.While we ...
- 4061 Views
- 8 replies
- 2 kudos
- 2 kudos
Hi @Kirankumarbs, Thanks for posting this. I can see from the thread that the existing replies confirmed the limitation but did not offer much in the way of alternatives. Let me provide a fuller picture. THE SHORT ANSWER You cannot set a custom max_r...
- 2 kudos
- 1758 Views
- 3 replies
- 4 kudos
Resolved! Where is Lakeflow Designer in the databricks UI
Hello everyone,Yesterday I saw a demo of Lakeflow Designer, and it looked very interesting. I tried to find this option in the UI, but I was not able to locate it.Could anyone please guide me on where I can access or try out this feature?I would real...
- 1758 Views
- 3 replies
- 4 kudos
- 4 kudos
If you are talking about the No-code version that is described here, then that is still in Private Preview as far as I am aware. Talk to your Databricks Account if you want to opt into the preview. If you do get access to it, you should be able to fi...
- 4 kudos
- 1731 Views
- 2 replies
- 1 kudos
Resolved! Share dashboard with external customers
I’m new to Databricks and still exploring the platform. One of our requirements is to share a Databricks dashboard with multiple external customers (for example, Customer 1 and Customer 2).The key requirement is data isolation:When a user from Custom...
- 1731 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @RJ1 - Welcome to Databricks Platform! Great question. This is a very common requirement. At a high level, you’ll want to separate the concerns: How users authenticate and are identified (who is the user / which customer are they from?)How data i...
- 1 kudos
- 933 Views
- 3 replies
- 2 kudos
Resolved! Data governance solution
I am here looking for Data governance solution for organization. I also searched this on many different website and found many solutions. We are bit confused to which one to choose. One of my friend suggest me this platform, and i am hoping i will ge...
- 933 Views
- 3 replies
- 2 kudos
- 2 kudos
You can use Unity Catalog for Databricks Lakehouse. You can use Collibra/Open Metadata along with Unity Catalog for complete governance
- 2 kudos
- 755 Views
- 2 replies
- 2 kudos
Resolved! Looking for resources to learn Databricks
Hi Community Members,I'm working as a Power BI Developer and interested to upskill into Databricks platform as a Data Analyst and Data Engineering.Request to share the resources(documentation/video tutorials) in a sequential order.Thank You!Best Rega...
- 755 Views
- 2 replies
- 2 kudos
- 2 kudos
For Bite-size overviews check the demo center - https://www.databricks.com/resources/demos/library This youtube channel is great for more detailed oriented discussion around specific features .https://www.youtube.com/@nextgenlakehouseFor more structu...
- 2 kudos
- 10003 Views
- 6 replies
- 2 kudos
Resolved! Understanding Autoscaling in Databricks: Under What Conditions Does Spark Add a New Worker Node?
I’m currently working with Databricks autoscaling configurations and trying to better understand how Spark decides when to spin up additional worker nodes. My cluster has a minimum of one worker and can scale up to five. I know that tasks are assigne...
- 10003 Views
- 6 replies
- 2 kudos
- 2 kudos
Is the above information true for job clusters as well? Looks like the enhanced auto scalar is only available for pipelines
- 2 kudos
- 467 Views
- 2 replies
- 3 kudos
Resolved! Am i publishing article in a correct way or not?
Hello Community,I’d like to check with the contributors whether the article I recently published follows the correct approach. Did I choose the right options and the appropriate place to publish it in the Databricks Community?https://community.databr...
- 467 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Kirankumarbs ,Yes, you did everything in correct manner. You put your article in correct place which is "Community Articles".Anyway, thanks for sharing with us
- 3 kudos
- 318 Views
- 0 replies
- 1 kudos
Better Diff for Jupyter Notebooks in Bitbucket
Comparing versions of Jupyter Notebooks (new preferred format on Databricks) in Bitbucket is much more difficult than the previous format. TPlease use the link below vote on adding better Jupyter Notebooks comparison to Bitbucket.Enable rich renderin...
- 318 Views
- 0 replies
- 1 kudos
- 5702 Views
- 6 replies
- 1 kudos
Cannot import editable installed module in notebook
Hi,I have the following directory structure:- mypkg/ - setup.py - mypkg/ - __init__.py - module.py - scripts/ - main # notebook From the `main` notebok I have a cell that runs:%pip install -e /path/to/mypkgThis command appears to succ...
- 5702 Views
- 6 replies
- 1 kudos
- 1 kudos
Sorry to triple post but I have another update: it seems to work for standalone clusters, but it refuses to build the wheel (I get a write permission error) on the job clusters.
- 1 kudos
