- 619 Views
- 4 replies
- 1 kudos
Resolved! Unable to View Tables While Setting Up PostgreSQL CDC via Lakeflow Connect
Dear Experts,I have a requirement to implement PostgreSQL CDC using Databricks Lakeflow Connect. While setting up the tables, I am unable to see the list of available tables, even though the connection settings appear to be correct.Could you please s...
- 619 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @harisrinivasay, @szymon_dybczak is correct. You must enter the database name. Lakeflow Connect can only connect to and query that database, and list the schemas and tables if you provide the correct name. If the name is incorrect or if you don’t ...
- 1 kudos
- 265 Views
- 0 replies
- 0 kudos
Stop Refreshing. Start Querying.
How Databricks Metric Views Are Replacing Power BI Import Models — and What Your Team Needs to Do About It.IntroductionPower BI Import models work — until scheduled refreshes, size limits, and governance sprawl become too big to ignore. Databricks Un...
- 265 Views
- 0 replies
- 0 kudos
- 601 Views
- 4 replies
- 0 kudos
Resolved! Sharepoint Connector Site Limitation
Hey All!We are trying out the Beta connector for SharePoint and found that the connector will not work at the root-level site. Is there a reason for this limitation. It is unfortunately a hard blocker for us to use the native connector. MUST_START...
- 601 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi Scott, Just asking our product team the quesiton. By the root level site do you mean content that is stored on the root level site? Or do you mean everything across your root tennant. ie you want to ingest all files across your tennant in a single...
- 0 kudos
- 1137 Views
- 2 replies
- 2 kudos
Resolved! How Deep clone works
Hi,For DR purposes, we have setup Deep clone using delta share. Each time the deep clone job runs, it executes the query create or replace table {schema}.{table} deep clone {delta_share}.{schema}.{table} The 1st time the job ran, it took few hours to...
- 1137 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @DineshOjha, Deep clone is incremental, not a full re-copy every time, even when you use CREATE OR REPLACE TABLE … DEEP CLONE … against a Delta Sharing table. On the first DEEP CLONE, Databricks must read the entire source table (via Delta Sharing...
- 2 kudos
- 523 Views
- 2 replies
- 1 kudos
Resolved! Environment-Specific Schemas in SQL Files
Hi Databricks Community,We are looking for the recommended best practice for managing environment-specific configurations (catalogs and schemas) within our SQL files when using Databricks Asset Bundles.Our Setup:We use Databricks Asset Bundles to pac...
- 523 Views
- 2 replies
- 1 kudos
- 1 kudos
Thank you LR, we tried this approach and it worked but for this we would need to define the variable at multiple places, soch as the databricks.yml, the jobs file and the main code.So we decided to put all these variables in a single config file whic...
- 1 kudos
- 269 Views
- 1 replies
- 0 kudos
DLT ERROR CODE: INGESTION_GATEWAY_CONNECTION_ERROR.SOURCE_CONNECTION_FAILURE
we are using lakeflow to sync data from ms sql server to databricks, then we randomly started getting this connection error in our lakeflow pipeline. not sure how to resolve this{"id": "68dcb1f0-39af-11f1-aa8c-00163e2cf055","sequence": {"data_plane_i...
- 269 Views
- 1 replies
- 0 kudos
- 0 kudos
The error indicates connection pool timeout. The Lake flow Ingestion Gateway cannot acquire a connection from the source SQL Server within the 10 second default window.You can check on the belowSQL Server Connection Exhaustion - Lake flow maintains p...
- 0 kudos
- 451 Views
- 2 replies
- 1 kudos
Resolved! Auto CDC fLow without CDF?
Auto CDC flow works with source table CDF enabled, but fails when CDF is disabled.The source table is updated via INSERT OVERWRITE.IS CDF mandatory?
- 451 Views
- 2 replies
- 1 kudos
- 1 kudos
Yes, @kevinzhang29 . For Auto CDC with a Delta source table, a change data feed (CDF) (i.e., a CDC feed) is required. AUTO CDC is explicitly designed to read from a CDC/change feed source such as Delta CDF, not from plain snapshots. When you don’t ha...
- 1 kudos
- 2069 Views
- 5 replies
- 1 kudos
Liquid clustering on partitioned delta table where new data is added by replacing the partition
Hi There,Currently we have the bronze, silver & gold delta tables partition by a string and yearmonth column, we maintain the entire historic data of 5 years and replace the partitions (latest 2 months) to add the new data.We use overwrite mode to re...
- 2069 Views
- 5 replies
- 1 kudos
- 1 kudos
Hi @SusmithaBadam, we are in Private Preview for a new feature that allows you to convert a table from Partitioning to Liquid Clustering. Here is the User Guide. Please reach out to your account team to get enrolled!
- 1 kudos
- 1402 Views
- 2 replies
- 3 kudos
Liquid Clustering with Partitioning
Hi Team,Can we use Partitioning and Liquid Clustering in Conjunction? Essentially, partitioning the table first on a specific field and then apply liquid clustering (on other fields)?Alternatively, can we define the order priority of the cluster key ...
- 1402 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Devsh_on_point, we are in Private Preview for a feature that helps you convert a Partitioned table to Liquid Clustering. Here is the User Guide. Reach out to your account team to get enrolled!
- 3 kudos
- 5287 Views
- 2 replies
- 1 kudos
Resolved! Databricks data engineer associate exam
Hi everyone,I'm preparing for the Databricks Data Engineer Associate certification. On the Databricks website, they list the following self-paced courses available in Databricks Academy for exam preparation:Data Ingestion with Delta LakeDeploy Worklo...
- 5287 Views
- 2 replies
- 1 kudos
- 1 kudos
Thanks for sharing this useful discussion on the Databricks Data Engineer Associate exam. I’m also preparing for this certification and found many helpful resources and insights here. If anyone else is currently studying or has already cleared it, fe...
- 1 kudos
- 4248 Views
- 2 replies
- 0 kudos
From Partitioning to Liquid Clustering
We had some delta tables that where previously partitioned on year, month, day, and hour. This resulted in quite small partitions and we now switched to liquid clustering.We followed these steps:Remove partitioning by doing REPLACEALTER TABLE --- CLU...
- 4248 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Volker , we are in Private Preview now for a feature that helps you easily convert a table from Partitioning to Liquid Clustering. Here is the User Guide.
- 0 kudos
- 4625 Views
- 2 replies
- 0 kudos
Resolved! Pathway advice on how to Data Engineer Associate
Hi everyone,I am new to this community and I am a BI/Data Engineer by trade in Microsoft Azure/On prem context. I want some advice on how to be a certified Data Engineer Associate in Databiricks. The training, lesson or courses to be eligible for tak...
- 4625 Views
- 2 replies
- 0 kudos
- 0 kudos
Great post, very helpful insights on the Data Engineer Associate pathway I’m also preparing for this exam and found similar resources really useful for understanding the core concepts and practice approach. Thanks for sharing! If anyone else has add...
- 0 kudos
- 530 Views
- 3 replies
- 0 kudos
Resolved! How to update alias for catalogs
Greetings,Is there a way to create an alias for a Databricks catalog? Current catalog name: trainingDesired alias: development_training The goal is that users connecting to either name should see the same schemas, tables, and data
- 530 Views
- 3 replies
- 0 kudos
- 0 kudos
I have a use case where I need to rename a catalog without impacting existing pipelines and notebooks, as the current catalog name is referenced across multiple applications. Instead of coordinating with multiple teams to update it everywhere, I was ...
- 0 kudos
- 1469 Views
- 7 replies
- 11 kudos
Resolved! Designing Reliable Data Versioning Strategies in Databricks
Hi everyone,I’m working on a use case where I need to retain 30 days of historical data in a Delta table and use it to build trend reports.I’m looking for the best approach to reliably maintain this historical data while also making it suitable for r...
- 1469 Views
- 7 replies
- 11 kudos
- 11 kudos
Hey @Raj_DB , The TLDR is time travel is great for short-term ops and debugging, but brittle as your primary reporting history, and its cost profile is harder to control and reason about than a purpose-built history table. Docs 1,2 explicitly say De...
- 11 kudos
- 886 Views
- 4 replies
- 5 kudos
Valid init script for installing ODBC Driver 18 for SQL Server to a job cluster
I need to execute stored procedures in my notebook, to do that I created an init script that installs ODBC driver to my job cluster. But the script stops working after some time and I can't figure out why, so the cluster can't start. Can someone send...
- 886 Views
- 4 replies
- 5 kudos
- 5 kudos
You don't need an ODBC driver inside a Databricks job cluster to run stored procedures, and init scripts are fragile enough that they easily break cluster startup. Options: Option 1 (Recommended): Use built-in JDBC / connectors instead of ODBC For S...
- 5 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 572 | |
| 349 | |
| 287 |
