Databricks Community

Table Trigger - Too many logfiles

Hi, we have implemented a job that runs on a trigger of a table update. The job worked perfectly, until the source table now have accumulated too many log files and the job isn't running anymore. Only the error message below:Storage location /abcd/_d...

Networking configuration of Azure Databricks managed storage account

Hi all,I created an Azure Databricks Workspace, and the workspace creates an Azure Databricks managed storage account.The networking configuration of the storage account is "Enabled from all networks".Shall I change it to "Enabled from selected virtu...

Is it possible to use Liquid Clustering on Delta Live Tables / Materialised Views?

Is it possible to use Liquid Clustering on Delta Live Tables? If it is available what is the Python syntax for adding liquid clustering to a Delta Live Table / Materialised view please? 

Voucher for Data Engineer Associate certification

Hi, I am looking for a voucher to complete the Data Engineer Associate certification. Could anyone help me please?

Is there any option in databricks to estimate cost of a query before execution

Hi Team,I want to check if there is any option in data bricks which can help to estimate cost of a query before execution?I mean calculate DBU before actual query execution based on the logical plan? Regards 

Mark Your Calendar: Data Day Texas + AI 2025 on January 25!

Hey Austin Databricks Group,If you're passionate about data and AI, here's an event you won't want to miss: Data Day Texas + AI 2025, happening on Saturday, January 25. This event brings together experts and enthusiasts to share insights, trends, and...

Cloning of Workflow from One env to different env using Job API

Hi Team,One of my team members recently shared one requirement: he wants to migrate the 10 Workflows from the sandbox to the dev environment to run his model in dev env.I wanted to move all these workflows in an automated way and one of the solutions...

Foreign table to delta streaming table

I want to copy a table from a foreign catalog as my streaming table. This is the code I used but I am getting error: Table table_name does not support either micro-batch or continuous scan.; spark.readStream                .table(table_name)         ...

My DABS CLI Deploy call not generating a .tfstate file

Hi Community,I'm running into an issue, when executing Databricks CLI Bundle Deploy I dont get the Terraform State file (.tfstate). I know that I should get one but even when defining the state_apth on my YAML (.yml) DABS file I still do not get it.D...

Unable to Access Databricks Customer Academy Platform

I am currently facing an issue accessing the Databricks Customer Academy platform. When I attempt to visit https://customer-academy.databricks.com, I receive the following error message:"You are not authorized to access https://customer-academy.datab...

Invisible empty spaces when reading .csv files

When importing a .csv file with leading and/or trailing empty spaces around the separators, the output results in strings that appear to be trimmed on the output table or when using .display() but are not actually trimmed.It is possible to identify t...

Can DAB used with Standard Tier Databricks workspace?

How do I use DataBricks SQL query to convert a field value % back into a wildcard?

Hi.  If I've posted to the wrong area, please let me know.I am using SQL to join two tables.  One table has the wildcard '%' stored as text/string/varchar.  I need to join the value of TableA.column1 to TableB.column1 based on the wildcard in the str...

Error: no streaming listener attached to the spark app is the error we are observing post accessing streaming statistics API. Please help us with this issue ASAP. Thanks.

Issue: Spark structured streaming applicationAfter adding the listener jar file in the cluster init script, the listener is working (From what I see in the stdout/log4j logs)But when I try to hit the 'Content-Type: application/json' http://host:port/...