- 618 Views
- 2 replies
- 0 kudos
DLT Pipeline and Pivot tables
TLDR:Can DLT determine a dynamic schema - one which is generated from the results of a pivot?IssueI know you cant use spark `.pivot` in DLT pipeline and that if you wish to pivot data you need to do that outside of the DLT decorated functions. I have...
- 618 Views
- 2 replies
- 0 kudos
- 0 kudos
Thank you for the reply - I have tried this (it was suggested in earlier solutions); but that may well be a side effect of the above function.query = f""" SELECT pivot_key, {select_clause} FROM data...
- 0 kudos
- 499 Views
- 3 replies
- 1 kudos
Cost attribution based on table history statistics
Hello all,I have a job that processes 50 tables - 25 belong to finance, 20 belong to master data, 5 belong to supply chain data domains.Now, imagine the job ran for 14 hours and did cost me 1000 euros on a day. If I like to attribute the per day cost...
- 499 Views
- 3 replies
- 1 kudos
- 1 kudos
Root Cause / Why executionTimeMs isn’t idealexecutionTimeMs includes everything the job did:Waiting for resourcesShuffle, GC, or network latencyContention with other concurrent jobsUsing this to allocate costs can misattribute costs, especially if so...
- 1 kudos
- 2639 Views
- 15 replies
- 17 kudos
Resolved! Ingesting 100 TB raw CSV data into the Bronze layer in Parquet + Snappy
Problem i am trying to solve:Bronze is the landing zone for immutable, raw data.At this stage, i am trying to sse a columnar format (Parquet or ORC) → good compression, efficient scans. and then apply lightweight compression (e.g., Snappy) → balances...
- 2639 Views
- 15 replies
- 17 kudos
- 17 kudos
@szymon_dybczak @BS_THE_ANALYST @Coffee77 @TheOC the use case summary is as eblow The use case: A telecom operator wants to minimize unnecessary truck rolls (sending technicians to customer sites), which cost $100–$200 per visit.Data sources feeding...
- 17 kudos
- 1604 Views
- 10 replies
- 4 kudos
Maven libraries in VNet injected, UC enabled workspace on Standard Access Mode Cluster
Hi!As the title suggests, I want to install Maven libaries on my cluster with access mode 'Standard'. Our workspace is VNet injected and has Unity Catalog enabled.The coordinates have been allowlisted by the account team according to these instructio...
- 1604 Views
- 10 replies
- 4 kudos
- 4 kudos
We have resolved the Metastore issue, which also seemed to have resolved the JAR issue. I don't have a clue why this resolves it. The network people might have used service tags which also opened the workspace to the odbc connections?
- 4 kudos
- 1406 Views
- 4 replies
- 1 kudos
Resolved! read json files on unity catalog
Hello Guys, I have some issue when i load several json files which have a same schema on databricks. when i do2025_07_17_19_55_00_2025_07_31_21_55_00_17Q51D_alice_out.json 516.13 KB2025_07_17_19_55_00_2025_07_31_21_55_00_17Q51D_bob_out.json 516.13 K...
- 1406 Views
- 4 replies
- 1 kudos
- 1 kudos
Hello @szymon_dybczak , Its Ok i have check the history of the table. I'm so confuse about the command display() output and the really output write operationThanx
- 1 kudos
- 735 Views
- 3 replies
- 2 kudos
Read polars from recently created csv file
More and more python packages transition to use polars instead of e.g. pandas. There is a problem with this in databricks when trying to read a csv file with it using pl.read_csv("filename.csv") when the file has been created in the same notebook cel...
- 735 Views
- 3 replies
- 2 kudos
- 2 kudos
Hello @victorNilsson Thank you for letting me know how to replicate the issue, I was able to get the same error this time. I've given the problem another go and think I have been able to fix it by specifying the output path as "/tmp/test.csv". By wri...
- 2 kudos
- 1792 Views
- 13 replies
- 6 kudos
Resolved! Job File Event Trigger not firing for SftpCommit and SftpCreate
Hi there, We are using Azure Storage Account and their SFTP feature. We have 3rd parties we work with that submit reports to us via SFTP into Azure Blob Storage. We have setup a File Trigger for that external location. Everything works fine if you up...
- 1792 Views
- 13 replies
- 6 kudos
- 6 kudos
Hi Dimitry, You need to go to the external_location. Then turn off file events for that external_location.Then you still select File Trigger. It will then evaluate the external_location. It will give you a message that you can only track up to 10k Fi...
- 6 kudos
- 1052 Views
- 9 replies
- 2 kudos
Databricks Lakebridge: Azure SQL DB to Databricks (Error while import)
Hi community experts,I am getting error "cannot import name 'recon' from 'databricks.labs.lakebridge.reconcile.execute'" importing modules as shown in attached screenshot. I am follwing steps as mentioned in your partner training module "Lakebridge f...
- 1052 Views
- 9 replies
- 2 kudos
- 2 kudos
Hi @shubham007 ,They made refactoring to that module in last month so that's why it stopped working. Probably Lakebridge for SQL Source System Migration module was recorded before that change.And why they made change? It is explained here:Split recon...
- 2 kudos
- 1326 Views
- 11 replies
- 3 kudos
Resolved! Unreliable file events on Azure Storage (SFTP) for job trigger
Hi allI got a job trigger by a file event on the external location.The location and jobs triggers are working fine when uploading file via Azure Portal.I need SFTP trigger, so I went into the event grid, found subscription for the storage account on ...
- 1326 Views
- 11 replies
- 3 kudos
- 3 kudos
UpdateAppears that even uploading via UI does not trigger it any more. It did trigger weeks ago.I have just uploaded a file in UI and saw this message in the storage queue:{"topic":"/subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.Storage/st...
- 3 kudos
- 555 Views
- 1 replies
- 0 kudos
Databricks Lakebridge: Azure SQL DB to Databricks (Error in Data and Schema Validation)
Hi community experts,I am getting error while Data and Schema Validation with the Reconciler. As attached here screenshots. Please help resolve this issue.Output:
- 555 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @shubham007 ,As stated in another thread. I think this error could be related to misconfiguration on your side. Lakebridge is trying to find following table in your SQL Server instance -> None.SalesLT.customerBut look at which database reconciliat...
- 0 kudos
- 1739 Views
- 1 replies
- 2 kudos
Databrick Genie
I have curated silver and gold tables in Advana that feed downstream applications. Other organizations also create tables for their own use. Can Databricks Genie query across tables from different pipelines within the same organization and across mul...
- 1739 Views
- 1 replies
- 2 kudos
- 2 kudos
Can you explain the landscape a bit more? The term "pipelines" means something specific in Databricks. You mention "across multiple organizations." What does that mean? Are you guys using Unity Catalog, are all the tables/data in Unity? Please elab...
- 2 kudos
- 611 Views
- 4 replies
- 4 kudos
Resolved! Does the free version of Databricks not support external storage data sources?
1、Can the data I use with the free version of Databricks on Azure only be stored on Azure, AWS, and Google Cloud Storage?2、Assuming the network is connected, can the paid version be used to access other publicly stored data (i.e., custom storage spac...
- 611 Views
- 4 replies
- 4 kudos
- 4 kudos
Not sure if this is a cheeky way to get around bringing files in: https://community.databricks.com/t5/data-engineering/connect-to-azure-data-lake-storage-using-databricks-free-edition/m-p/127900#M48116 but I answered a similar thing on a different po...
- 4 kudos
- 548 Views
- 1 replies
- 1 kudos
Identify updated rows during incremental refresh in DLT Materialized Views
Hello, every time that I run a delta live table materialized view in serverless , I get a log of "COMPLETE RECOMPUTE". I realised I was using current_timestamp as a column in my MV to identify rows which got updated in the last refresh. But that make...
- 548 Views
- 1 replies
- 1 kudos
- 1 kudos
@ManoramTaparia The issue is that current_timestamp() makes your MV non-deterministic, forcing complete recomputes. Here's how to fix it:Solution: Use the Source Table's Change TrackingOption 1: Leverage Source Table's Timestamp Column@Dlt.table(name...
- 1 kudos
- 765 Views
- 5 replies
- 2 kudos
- 765 Views
- 5 replies
- 2 kudos
- 2 kudos
Hello @yinan Good day!!Databricks, being a cloud-based platform, does not have direct built-in support for reading data from a truly air-gapped (completely offline, no network connectivity) Cloudera Distribution for Hadoop (CDH) environment. In such...
- 2 kudos
- 344 Views
- 2 replies
- 0 kudos
Using Databricks to transform cloudera lakehouse on-prem without bringing the data to cloud
I am looking for a solution to connect databricks to cloudera lakehouse hosted on-prem and transform the data using databricks without bringing the data to databricks delta tables or cloud storage. once the transformation is done the data need to be ...
- 344 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello, What is your data volume? You can connect using jdbc/odbc but this process will be slower if the data volume is too high.Another way of connecting is if your cloudera storage is in HDFS then you can also connect through HDFS API as well.
- 0 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 493 | |
| 349 | |
| 287 |
