Here's your Data + AI Summit 2024 - Warehousing & Analytics recap as you use intelligent data warehousing to improve performance and increase your organization’s productivity with analytics, dashboards and insights. Keynote: Data Warehouse presente...
I recently updated my Databricks VS Code extension from version 1.4.0 to 2.4.0. After the update, I noticed that:The "Configure Sync Destination" button is missing.The sync destination format has changed to a remote folder instead of the previous set...
The upgrade to Databricks VS Code extension v2.4.0 changed the sync destination system from the previous workspace directory setup, removing the "Configure Sync Destination" button and setting the default to a more rigid remote folder structure (comm...
Hi,We publish listings on the Databricks Marketplace.Several times a week, we receive a lead notification stating: "db sp has installed 'Etsy Dataset'."It’s always the same dataset and the same name, so it’s likely a technical error.Does anyone know ...
Yes, this is a known issue reported by other Databricks Marketplace publishers: receiving frequent lead notifications stating "db sp has installed 'Etsy Dataset'" for the same dataset and name, which suggests an underlying technical error in the lead...
We are trying to schedule updates to streaming tables and materialized views in Azure Databricks that we have defined in dbt.Two options we are considering are `SCHEDULE CRON` and just scheduling `dbt run` commands via CI/CD. The `SCHEDULE CRON` opti...
Your understanding of the trade-offs between using SCHEDULE CRON on streaming tables/materialized views and the dbt run orchestration in Azure Databricks is largely correct, and you’ve identified the two main pain points: orchestration of dependencie...
We recently enabled the preview feature to refresh power bi datasets from data bricks workflow, but when we run the job data changes aren't being applied despite a completed status in refresh history. Could anyone clarify whether this is intended pre...
This is a known issue with the Power BI/Databricks integration preview for refreshing Power BI datasets from Databricks workflows. Even when a job completes and the refresh history shows success, actual data changes—such as newly inserted rows—are of...
I would like to be able to express something like: WHERE columnA = "valueA" OR columnB = "valueB" but on the Widget. I only see the possibility that chaining filters that are processed as AND operator. Am I missing something?
Most widget filter interfaces only support multiple filters using an AND operator, meaning each condition must be true for a row to be included. If you want to use an OR operator, such as filtering where columnA = "valueA" OR columnB = "valueB", most...
Hello communityI am facing an issue when I export a dashboard to an HTML: the problem is that the HTML has white background but the exported tables have black colors, like the example I am showing below Does someone know how to handle this visualizat...
To resolve the issue of dashboard tables exporting to HTML with mismatched colors—specifically black table backgrounds against a white HTML page—review and adjust your dashboard’s CSS or formatting settings before export. The problem typically occurs...
Hi all, I need to add a shortcut key for R's pip operator (%>%) to my Databricks notebook. I want the operator to be written in my code snippet when I hold down the shortcut keys (shift + ctrl + m). Is there a straightforward way to add such shortcut...
Databricks notebooks do not natively support defining custom keyboard shortcuts to insert specific code snippets like R’s pipe operator () within the notebook editor UI. The list of standard shortcuts available is limited to navigation, cell manageme...
Hi All,How can I maintain 7 years of transactional data in delta table? Can I have log retention of 7 days, but data retention of 7 years?Appreciate your response.Thanks and regards Suhas
Hi @patilsuhasv,Maintaining 7 years of transactional data in a Delta Lake table is technically possible, but it requires careful configuration — and it’s important to understand the recent changes Databricks announced (effective December 2025) regard...
hi everyone, does somebody have experience with connecting PowerBI to SAP Databricks in the BDC? I have quite a few SAP customers interested in that ... many thanks, Andreas
Hi @WiliamRosa ,do you know which feature is missing in SAP Databricks that it doesnt work? As Andreas said- everybody is asking that question.KR Daniel
As part of a function I create df1 and df2 and aim to stack them and output the results. But the results do not display within the function, nor if I output the results and display after.results = df1.unionByName(df2, allowMissingColumns=False)displ...
Hello @CEH! Did any of the suggestions above help resolve the issue?If so, please mark the most helpful reply as the accepted solution. Or, if you found another fix, please share it with the community so others can benefit as well.
Can anyone provide a sample MERGE INTO SQL query for implementing SCD Type 2 in Databricks using Delta Tables?
Here is a simple example using an upstream Delta table with ChangeDataFeed enabled, using table_changes() to get the records with their corresponding operation, this is a 2 step process you need to close out modified or deleted recordsadd new rows (i...
Is it possible to show the full logs of a databricks job? Currently, the logs are skipped with:*** WARNING: max output size exceeded, skipping output. ***However, I don't believe our log files are more than 20 MB. I know you can press the logs button...
Any news on this topic?Have the limits on the notebook-cell-log output been resolved?
Hi everyone,I’m looking to improve the efficiency of developing and debugging Spark jobs within Databricks and wanted to get insights from the community. Spark is incredibly powerful, but as projects grow in complexity, it can become challenging to m...
Developing and debugging Spark jobs in Databricks can be challenging due to the distributed nature of Spark and the volume of data processed. To streamline your workflow:Leverage Notebooks for Iterative Development:Use Databricks notebooks to write a...
I've been trying to run a dbt project (sourced in Azure DevOps) in Databricks Workflows, but I get this error message: CalledProcessError: Command 'b'\nmkdir -p "/tmp/tmp-dbt-run-1124228490001263"\nunexpected_errors="$(cp -a -u "/Workspace/Repos/.in...
The error you encountered when running your dbt project in Databricks Workflows comes from Databricks trying to copy the entire repository, including the virtual environment (venv) folder and its cached bytecode files (__pycache__), into a temporary ...
Hi all,I'm experiencing an intermittent connection issue between Power BI Desktop and our Azure Databricks SQL Warehouse and looking for help troubleshootingError Message:ODBC: ERROR [HY000] [Microsoft][ThriftExtension] (14) Unexpected response from ...
The intermittent ODBC error you’re seeing in Power BI when connecting to Azure Databricks is a recognized issue related to SSL validation interruptions or proxy interference in the Simba ThriftExtension layer. The behavior—random occurrences, tempora...
