- 53 Views
- 2 replies
- 1 kudos
Lakeflow SDP (DLT) produce external tables, or only UC-managed
As I understand it, streaming tables and materialized views produced by Lakeflow Spark Declarative Pipelines (DLT) are always Unity Catalog managed tables , there's no LOCATION/path option on create_streaming_table or apply_changes.Is that correct? A...
- 53 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @nidhin, What you’re saying is basically correct for a Unity Catalog-enabled Lakeflow Spark Declarative Pipelines setup. In that model, pipelines publish streaming tables and materialized views into the target catalog and schema, the data is store...
- 1 kudos
- 58 Views
- 0 replies
- 0 kudos
STTM as a Metadata Contract in Databricks
One pattern I keep seeing in data engineering projects:STTM is treated as documentation.But in reality, STTM can become much more than that.A well-structured Source-to-Target Mapping can act as a metadata contract between business, engineering, QA, a...
- 58 Views
- 0 replies
- 0 kudos
- 68 Views
- 0 replies
- 0 kudos
Adhoc Table Refresh in Lakeflow Spark Declarative Pipelines (SDP)
Hi,It is currently not possible to specify a list of tables to refresh and their refresh policies (full/normal) in a Lakeflow Job.It can be done via the REST API, but it's messy.For example, if you need some tables or views refreshed more regularly, ...
- 68 Views
- 0 replies
- 0 kudos
- 210 Views
- 2 replies
- 0 kudos
Resolved! From STTM to Databricks Pipelines: Can Metadata Become the Source Code of Data Engineering?
I’ve been exploring a metadata-driven approach to data engineering through a project called Data Engineering Copilot.The idea is to treat Source-to-Target Mapping (STTM) documents as structured metadata rather than static documentation.Instead of man...
- 210 Views
- 2 replies
- 0 kudos
- 0 kudos
This is a good discussion topic, but from my experience right now it is both meta data driven and most traditional excel based STMs.A few observations:How most teams manage STTM todayLevel 1 (Most Common)STTM in Excel, Word, or Confluence.Engineers m...
- 0 kudos
- 103 Views
- 1 replies
- 1 kudos
SQL Warehouse stuck on "Cluster Start-up Delayed
Hi everyone,I'm running into an issue with my Starter Warehouse on Databricks and would appreciate any help or pointers.Problem: My SQL Warehouse has been stuck in a Starting state with the following warning:Cluster Start-up Delayed. Please wait whil...
- 103 Views
- 1 replies
- 1 kudos
- 1 kudos
This typically points to delayed compute provisioning behind the SQL Warehouse, often due to temporary capacity/resource availability or a transient startup issue.A few things I would try:1. Stop and restart the SQL WarehouseIf it has been stuck for ...
- 1 kudos
- 101 Views
- 1 replies
- 0 kudos
Managing IPYNB cell timestamps in source control
We're in the process of converting over our Databricks notebooks from .py file to .ipynb. We have disabled storing notebook output in source control at the workspace level.However, what we're discovering is that every cell in our notebooks has 3 time...
- 101 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @emorgoch, Thanks for raising this. This appears to be a regression rather than expected behaviour. Internally, the issue has been identified around .ipynb handling in Git folders, and the intended fix is to stop serialising these execution timest...
- 0 kudos
- 132 Views
- 1 replies
- 0 kudos
Resolved! Table history time travel
I have noticed what seems to be unexpected behavior with the history of Unity Catalog managed tables and would like to understand whether this is expected.As a test, I created a table with two versions:Version 0Version 1 (created approximately 200 ho...
- 132 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @MVMZ, What you’re seeing is expected for Unity Catalog managed tables. The key detail is that for Unity Catalog managed tables, Databricks blocks time travel queries when the requested version is older than delta.deletedFileRetentionDuration, whi...
- 0 kudos
- 124 Views
- 2 replies
- 0 kudos
Issue with MongoDB Void Null Type in Databricks
Facing an issue with the MongoDB Void/Null type in Databricks, which requires explicit casting or conversion to an array or struct of strings. Looking for guidance on how to handle this data type when reading from MongoDB and writing to a Delta table...
- 124 Views
- 2 replies
- 0 kudos
- 0 kudos
You can handle VOID columns directly if you are on Databricks Runtime 18.2 or later for batch writes.More details hereYou can explicitly cast or replace VOID/NULL columns with appropriate types when reading from MongoDBdf = df.withColumn("files_col",...
- 0 kudos
- 435 Views
- 2 replies
- 1 kudos
Resolved! PII tags in Spark Declarative Pipelines
I need to add PII tags at both the table and column levels for a streaming table created using Spark Declarative Pipelines.I tried applying Unity Catalog tags with the following code inside the SDP Python pipeline:spark.sql(f"""ALTER TABLE {table_nam...
- 435 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @amirabedhiafi : Is it not possible to pass the StructField to the schema and then pass it to thedlt.createStreamingTable (name, schema)I tried passing the description of the columns to it and that works. I am wondering , why tags do not work
- 1 kudos
- 18462 Views
- 9 replies
- 5 kudos
Resolved! Apply unitycatalog tags programmatically
Hello,I'm interested in the "Tags" feature of columns/schemas/tables of the UnityCatalog (described here: https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/tags)I've been able to play with them by hand and would now lik...
- 18462 Views
- 9 replies
- 5 kudos
- 5 kudos
Hi, Just re-opening this thread here.Is there a way to programatically add tags to our streaming tables with SDP ?dlt.createStreamingTable(table, schema) and the schema is then A Struct field with metadata , comments and Tags?has anyone tried that?
- 5 kudos
- 168 Views
- 1 replies
- 1 kudos
Resolved! DatabricksConnect from Python/AKS environment calling Databricks Cluster: Spark Query Call Hangs
I have Python 3.12 Pod in AKS using DatabricksConnect 18.1.1 connecting to Databricks cluster 18.1.All works great and normally I see no issues running series of Spark queries But once a while, even without any load on dedicated cluster we have, quer...
- 168 Views
- 1 replies
- 1 kudos
- 1 kudos
The execution and result streaming generally happens over the gRPC route. You can force the gRPC route to send periodic frames to keep the connection look active in the AKS network infrastructure side.You can add the following variables into the AKS ...
- 1 kudos
- 212 Views
- 2 replies
- 2 kudos
Resolved! Databricks Connect - Will I ever have to Stop clean up Spark session when creating new per request
I have API that triggers Spark calculations - with API hosted by Python 3.12 pod in AKS and connects to Databricks cluster using Databricks 18.1.1.Initially I was using getOrCreate call on my API requests and all works.But problem is - as Spark sessi...
- 212 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi there, I Short answer You should call spark.stop() when you're done with each session. What you're doing now (not calling it) works, but it's not ideal — you're relying on the server-side idle timeout to clean up after you, and in the meantime e...
- 2 kudos
- 190 Views
- 2 replies
- 0 kudos
Can multiple questions be added to the same sql query in genie space
Hi, Can we add multiple sample questions to one SQL query in the sql queries instructions so Genie learns to handle similar variations?
- 190 Views
- 2 replies
- 0 kudos
- 0 kudos
Thanks for the response Ashwin, will add more sql queries with different phrasing and would test it
- 0 kudos
- 695 Views
- 0 replies
- 1 kudos
Databricks Omnigent Points to the Next Layer of AI Engineering
A lot of AI announcements these days start to sound similar after a while. A new model is better. A new agent is faster. A new framework can do more. And most of the time, the conversation stays focused on the tool itself.That is why Omnigent caught ...
- 695 Views
- 0 replies
- 1 kudos
- 583 Views
- 4 replies
- 5 kudos
Resolved! json file existing in volume but not showing in UI
I have some json files existing in a specific volume when I try to search for them they don't appear but when I query the the volume using python I am able to get them and read their content.Any help ?
- 583 Views
- 4 replies
- 5 kudos
- 5 kudos
Hi,The global workspace search won't return results for files stored in Unity Catalog Volumes. Its indexing is focused on workspace assets and catalog-managed objects, rather than the underlying files within a Volume.To locate files in a Volume, navi...
- 5 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 575 | |
| 349 | |
| 287 |
