Hi everyone,I am looking to create a notebook that, when executed by a user, performs the following actions:Retrieves all Databricks jobs created by the current userChecks whether a specific role already has permissions on those jobsAutomatically add...
Hello Community,We have a case in our project that we would like to solve in an elegant and scalable manner. As always, I would really appreciate your suggestions and experience.In short:We have a multi-step job consisting of 4 stages. In one of the ...
I don't think there is anything native for this in Databricks. The closest match would have been system tables (system.lakeflow.job_run_timeline / job_task_run_timeline) but I don't think it will have the necessary grain for what your pattern. There'...
Has anyone else seen full refresh snapshots trigger outside of their configured refresh window in Lakeflow Connect?Here's our situation:- We have a full refresh window configured to restrict snapshot operations to off-hours- On at least one occasion,...
Hello @Sumit_7 ,I have tested a few scenarios and logged a ticket with Databricks and discovered the following:Common MisconceptionThe start_window setting does not define a bounded time window during which full refreshesare contained. It is simply a...
Has anyone else run into a situation where a breaking schema change on a SQL Server source table leaves their Lakeflow Connect pipeline in a state it can't recover from — even after destroying and recreating the pipeline?Here's what happened to us:- ...
Hey all,Following up.I was able to recover. The one step I was missing is resetting CDC in the source side. After that, I was able to destroy and recreate the bundle and successfully refresh all tables. Thanks!
Hi everyone,I am facing a data consistency issue in my Databricks incremental pipeline where records are being skipped because of a time gap between when a record is processed and when the physical file is finalized in Azure Blob Storage (ABFS).Our A...
You can handle it as belowFix the Bronze Write - The 20+ minutes commit gap suggests metadata contention or "Small File Issues" in the bronze delta tables. You can optimize tables manually or enable Optimized Write and Auto Optimize if feasible. This...
Dear Experts,I have a requirement to implement PostgreSQL CDC using Databricks Lakeflow Connect. While setting up the tables, I am unable to see the list of available tables, even though the connection settings appear to be correct.Could you please s...
Hi @harisrinivasay, @szymon_dybczak is correct. You must enter the database name. Lakeflow Connect can only connect to and query that database, and list the schemas and tables if you provide the correct name. If the name is incorrect or if you don’t ...
Hi everyone,I’m working on a use case where I need to retain 30 days of historical data in a Delta table and use it to build trend reports.I’m looking for the best approach to reliably maintain this historical data while also making it suitable for r...
Hey @Raj_DB , The TLDR is time travel is great for short-term ops and debugging, but brittle as your primary reporting history, and its cost profile is harder to control and reason about than a purpose-built history table. Docs 1,2 explicitly say De...
We are currently performing an assessment for a client’s Redshift to Databricks migration, and we would like to better understand the enhanced capabilities of Lakebridge for this use case.We would appreciate clarification on the following points:Scop...
There is a nice course on Partner Academy as well . It uses SQL Server as a target system for migration but you can follow the same steps for Redshift as well . https://partner-academy.databricks.com/learn/courses/4326/lakebridge-for-sql-source-syste...
The code : import dltfrom pyspark.sql.functions import col@dlt.table( name="silver_customers", comment="Cleaned customers data from bronze")@dlt.expect("valid_email", "email IS NOT NULL")@dlt.expect("valid_customer_id", "customer_id IS NOT NULL...
@helius_205 I doubt, do check the execution mode ~ should be triggered. Also it's a normal read instead of readStream. Read Docs for better understanding.
What I tried:Updating the description via UI (AI Suggested Description / manual editI’m running into an issue while trying to update the description for the schema.Context:Type: SCHEMA_FOREIGN_SQLSERVERError message:Failed to save description. Please...
Hi, Yes 100%, if you use Lakeflow connect, it will ingest the data and they will become managed tables. Which will support the descriptions and comments. You should also get some query improvement as you're actually moving the data rather than queryi...
Scenario: Delta Sharing with Materialized ViewProvider Side Setup :->A Delta Share was created.->A materialized view was added to the share.->Recipients Created-> 1). Open Delta Sharing recipient Accessed using Python (import delta_sharing)->2)...
Hi @ittzzmalind, This is expected behaviour and is mainly due to how Delta Sharing handles materialized views for open (non-Databricks) recipients versus Databricks-to-Databricks recipients. For Databricks-to-Databricks recipients, the shared materia...
I need to load many tables into Bronze layer connecting to sql server DB. How can i pass the tables names dynamically in DLT. Means one code pass many tables and load into bronze layer
Hi Ashwin Thanks for the quick response. Yes I want to pass all the tables through config parameter/param file and load that into bronze layerI will try this approach. Thanks
Hi everyone,I’m trying to write Delta tables to my S3-compatible (non-AWS) endpoint, and it was writing perfectly fine last week with the same setup. Now, without any changes on my end, it’s failing and giving me anUnknownException: (com.amazonaws.se...
Hi @demo-user , Can you share more information about your setup: Cluster type and DBR versionS3-compatible storage implementation (MinIO / something else?) AFAIK this is not supposed to work as Delta client in DBR relies on AWS STS to perform S3 comm...
Hi AllI have a view creation script in DataBricks which uses a lateral view to access columns in a structure held within an array field. It is working fine but I have noted that the LATERAL VIEW is now depracated and that I should be using a TABLE RE...
You can useCREATE OR REPLACE VIEW newview AS SELECT t1.field1, item.field2, item.field3 FROM table1 AS t1 INNER JOIN table2 AS t2 ON t1.id = t2.id , LATERAL EXPLODE(t1.structure) AS structureitem(item)
Hi Team,We’ve applied column-level tags to a table in Azure Databricks and attached the table in our Genie workspace. However, when querying via Genie, the column tag information is not being returned correctly (missing/incomplete results), despite t...
Hi @sreya_sahithi,This is an important distinction about how Genie works: Genie queries the actual data rows in the tables attached to its space — it does not natively query Unity Catalog metadata such as column-level tags. Column tags live in INFORM...
