- 6261 Views
- 5 replies
- 2 kudos
apply_changes_from_snapshot with expectations
Hi,Question: Are expectations supposed to function in conjunction with create_streaming_table() and apply_changes_from_snapshot?Our team is investigating Delta Live Tables and we have a working prototype using Autoloader to ingest some files from a m...
- 6261 Views
- 5 replies
- 2 kudos
- 2 kudos
Also facing the same issue - did you find a solution?Thinking I will have to apply expectations at the next stage of the pipeline until this is worked out
- 2 kudos
- 1731 Views
- 1 replies
- 0 kudos
Performance Issue with OPTIMIZE Command for Historical Data Migration Using Liquid Clustering
Hello Databricks Community,I’m experiencing performance issues with the OPTIMIZE command when migrating historical data into a table with liquid clustering. Specifically, I am processing one year’s worth of data at a time. For example:The OPTIMIZE co...
- 1731 Views
- 1 replies
- 0 kudos
- 0 kudos
Did you got any solution? If Yes please post it.
- 0 kudos
- 1409 Views
- 2 replies
- 0 kudos
how can I verify that the result of a dlt will have enough rows before updating the table?
I have a dlt/lakeflow pipeline that creates a table, and I need to make sure that it will only update the resulting materialized view if it will have more than one million records.I've found this, but it seems to only work if I have already updated t...
- 1409 Views
- 2 replies
- 0 kudos
- 0 kudos
Thank you for the quick reply.Is there a common/recommended/possible way to work around this limitation? I don't mind not using the expectation api if it doesn't support logic that's based on aggregations.
- 0 kudos
- 6672 Views
- 9 replies
- 5 kudos
Resolved! Databricks Autoloader BadRecords path Issue
I have one file that has 100 rows and in which two rows are bad data and the remaining 98 rows is good data, but when I use the bad records' path, it completely moves the file to the bad records' path, which has good data as well, and it should move ...
- 6672 Views
- 9 replies
- 5 kudos
- 5 kudos
Why Entire Files Go to badRecordsPath When you enable badRecordsPath in Autoloader or in Spark’s file readers (with formats like CSV/JSON), here’s what happens: Spark expects each data file to be internally well-formed with respect to the declared s...
- 5 kudos
- 8087 Views
- 8 replies
- 4 kudos
Resolved! Schema evolution for JSON files with AutoLoader
I am using Auto Loader to ingest JSON files into a managed table. Auto Loader saves only the first-level fields as new columns, while nested structs are stored as values within those columns.My goal is to support schema evolution when loading new fi...
- 8087 Views
- 8 replies
- 4 kudos
- 4 kudos
@yit awesome. Glad that you got this solved. I look forward to the next problem .All the best,BS
- 4 kudos
- 3456 Views
- 5 replies
- 0 kudos
How to replace ${param} by :param
Hello,We previously used ${param} in our SQL queries:SELECT * FROM json.`${source_path}/file.json`However, this syntax is now deprecated. The recommended approach is to use :param instead.But when I attempt to replace ${param} with :param, I encounte...
- 3456 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi @ZD Please try this syntax in Your notebook for SQL:%sqldeclare _my_path = 'some_path';select _my_path;
- 0 kudos
- 1898 Views
- 2 replies
- 1 kudos
Resolved! Job cluster has no permission to create folder in Unity Catalog Volume
Hello everybody,I want to run a job that collects some csv files from a SFTP server and saves them on my Unity Catalog Volume. While my personal cluster defined like the following has access to create folders on the volume my job cluster doesn't.Defi...
- 1898 Views
- 2 replies
- 1 kudos
- 1 kudos
Thank you, that helped although I had to use "SINGLE_USER" instead of "DATA_SECURITY_MODE_DEDICATED". According to the docs (https://docs.databricks.com/api/workspace/clusters/create) "SINGLE_USER" is an alias for "DATA_SECURITY_MODE_DEDICATED".
- 1 kudos
- 3046 Views
- 4 replies
- 2 kudos
Resolved! Not able to read data from volume and data is in JSON format
Not able to read data from volume and data is in JSON format data = spark.read.json("/Volumes/mydatabricksaviral/datatesting/datavolume/mytest.json") display(data) ############################################# Py4JJavaError: An error occurred while...
- 3046 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @Aviral-Bhardwaj ,please double check:- if volume path is correct- if you have READ VOLUME permission on this volume- if your cluster has access to unity catalog- if json file exists
- 2 kudos
- 1374 Views
- 1 replies
- 0 kudos
Resolved! Low Level Design for Moving Data from Databricks A to Databricks B
Hello Techie,May someone please help me with Low level design point what all we should considered while moving data from One Delta lake instance to another delta lake.For example :-Service principle creation.IP Whitelisting.Any gitlab / devops relate...
- 1374 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Pratikmsbsvm Here's a brief low-level design checklist for Delta Lake to Delta Lake data migration:1. Security & Authentication- Create service principals for both environments- Set up Azure Key Vault for credential management- Configure IP white...
- 0 kudos
- 1812 Views
- 1 replies
- 0 kudos
finishes execution in 6 seconds but the notebook still shows "waiting"
Issue:Although my SQL query completes execution in approximately 2-3 seconds, the notebook interface continues to show "waiting" for an extended period before displaying results. Only way to see the results of my cell execution is by refreshing the w...
- 1812 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @root92 This is a known Databricks interface issue, not related to query performance or account type.Most Likely Causes:WebSocket connection timeout between browser and DatabricksBrowser memory issues with long-running notebook sessionsNetwork pro...
- 0 kudos
- 2180 Views
- 5 replies
- 0 kudos
in-home built predictive optimization
Hello allHas anyone attempted to look at the internals of predictive optimization and built an in-home solution mimicking its functionality? I understood that there are no plans from Databricks to roll-out this feature for external tables, and hence,...
- 2180 Views
- 5 replies
- 0 kudos
- 0 kudos
@LinlinH thanks for the details. Can you please share any Github link where the community work is put so I can verify if any code can be re-used...
- 0 kudos
- 4905 Views
- 2 replies
- 0 kudos
Collecting Job Usage Metrics Without Unity Catalog
hi,I would like to request assistance on how to collect usage metrics and job execution data for my Databricks environment. We are currently not using Unity Catalog, but I would still like to monitor and analyze usageCould you please provide guidance...
- 4905 Views
- 2 replies
- 0 kudos
- 4677 Views
- 2 replies
- 0 kudos
Append-only table from non-streaming source in Delta Live Tables
I have a DLT pipeline, where all tables are non-streaming (materialized views), except for the last one, which needs to be append-only, and is therefore defined as a streaming table.The pipeline runs successfully on the first run. However on the seco...
- 4677 Views
- 2 replies
- 0 kudos
- 0 kudos
@Oliver_Angelil - I was wondering if you found a solution? I have a similar use-case. I want to create a archive table using DLT from non-streaming (MV). I would prefer a DLT solution. I was able to get it to work using traditional merge approach (no...
- 0 kudos
- 3260 Views
- 3 replies
- 1 kudos
Resolved! Deploying Data Assets Bundle with VSCode Add-in
Deploying a bundle containing a Pipeline that references a DLT Notebook.In the YAML I am passing the relative path to the Notebook from the repository root (where the YAML lives).Deploying says 'success' but when validating, the Pipeline cannot find ...
- 3260 Views
- 3 replies
- 1 kudos
- 1 kudos
Hello @simonB2025 May you share a snippet of your folder/project structure and where the notebook resides, so we can suggest toyou the exact solution. Thank you!Best, Ilir
- 1 kudos
- 3787 Views
- 1 replies
- 1 kudos
Resolved! connecting to unity catalog with power BI
i created a group of powerBI users, i grant consumer access permission to this groupe on unity catalog i started a shared compute cluster (Standard_D4ds_v5). when trying to connect from power BI, i will connect to tables only if i grant access to wor...
- 3787 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Miloud_G Instead of granting full workspace access, you can grant "Databricks SQL access" entitlement, which provides limited workspace access specifically for BI tools.Go to Workspace Settings → Identity and Access → Groups/Users/SPs Manage → Se...
- 1 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
