Here's your Data + AI Summit 2024 - Warehousing & Analytics recap as you use intelligent data warehousing to improve performance and increase your organization’s productivity with analytics, dashboards and insights. Keynote: Data Warehouse presente...
Hi All,How can I maintain 7 years of transactional data in delta table? Can I have log retention of 7 days, but data retention of 7 years?Appreciate your response.Thanks and regards Suhas
Hey @patilsuhasv , Yes, you absolutely can maintain 7 years of transactional data in a Delta table while having only 7 days of log retention. These are two separate concepts that work independently. Understanding the Difference Log Retention control...
hi everyone, does somebody have experience with connecting PowerBI to SAP Databricks in the BDC? I have quite a few SAP customers interested in that ... many thanks, Andreas
Hi @WiliamRosa ,do you know which feature is missing in SAP Databricks that it doesnt work? As Andreas said- everybody is asking that question.KR Daniel
As part of a function I create df1 and df2 and aim to stack them and output the results. But the results do not display within the function, nor if I output the results and display after.results = df1.unionByName(df2, allowMissingColumns=False)displ...
Hello @CEH! Did any of the suggestions above help resolve the issue?If so, please mark the most helpful reply as the accepted solution. Or, if you found another fix, please share it with the community so others can benefit as well.
Can anyone provide a sample MERGE INTO SQL query for implementing SCD Type 2 in Databricks using Delta Tables?
Here is a simple example using an upstream Delta table with ChangeDataFeed enabled, using table_changes() to get the records with their corresponding operation, this is a 2 step process you need to close out modified or deleted recordsadd new rows (i...
Is it possible to show the full logs of a databricks job? Currently, the logs are skipped with:*** WARNING: max output size exceeded, skipping output. ***However, I don't believe our log files are more than 20 MB. I know you can press the logs button...
Any news on this topic?Have the limits on the notebook-cell-log output been resolved?
Hi everyone,I’m looking to improve the efficiency of developing and debugging Spark jobs within Databricks and wanted to get insights from the community. Spark is incredibly powerful, but as projects grow in complexity, it can become challenging to m...
Developing and debugging Spark jobs in Databricks can be challenging due to the distributed nature of Spark and the volume of data processed. To streamline your workflow:Leverage Notebooks for Iterative Development:Use Databricks notebooks to write a...
I've been trying to run a dbt project (sourced in Azure DevOps) in Databricks Workflows, but I get this error message: CalledProcessError: Command 'b'\nmkdir -p "/tmp/tmp-dbt-run-1124228490001263"\nunexpected_errors="$(cp -a -u "/Workspace/Repos/.in...
The error you encountered when running your dbt project in Databricks Workflows comes from Databricks trying to copy the entire repository, including the virtual environment (venv) folder and its cached bytecode files (__pycache__), into a temporary ...
Hi all,I'm experiencing an intermittent connection issue between Power BI Desktop and our Azure Databricks SQL Warehouse and looking for help troubleshootingError Message:ODBC: ERROR [HY000] [Microsoft][ThriftExtension] (14) Unexpected response from ...
The intermittent ODBC error you’re seeing in Power BI when connecting to Azure Databricks is a recognized issue related to SSL validation interruptions or proxy interference in the Simba ThriftExtension layer. The behavior—random occurrences, tempora...
We are launching our first Databricks Apps based on Streamlit.The App works when simply running the notebook in our workspace, but fails after deployment due to "could not find a valid JAVA_HOME installation" when running in the system environment.We...
Databricks Apps (which use a lightweight, container-based runtime) do not automatically include JVM, best is to use the databricks package to not have dependency issues.
HiI think I’ve found a reproducible bug / or am misunderstanding some syntax / capabilities of Metric Views when joining a calendar scaffold to an SCD2 table.The same SQL query works perfectly, but the Metric View always returns a constant 1 per mont...
Hey @playnicekids , I dig some digging and have come up with some helpful hints/tips to get you past your issue: This behavior is due to how metric view joins are defined and executed. Diagnosis The join in your metric view is a many-to-many te...
Hi there,I'm encountering the following error while attempting to read a file from R2 storage:[FAILED_READ_FILE.NO_HINT] Error while reading file r2:REDACTED_LOCAL_PART@user_id.r2.cloudflarestorage.com/data/20250128_160228_54805_wpkza_e38751cf-969e-...
But the error is from aws and seen when the payload size is incorrectly defined in the contentLength parameter. Caused by: com.amazonaws.SdkClientException: Data read has a different length than the expected: This sounds like a bug, Was this resolved...
Hello,We recently detected an issue in our product deployment with terraform.At some point, we have some java code that creates a schema in "hive_metastore" catalog.Since "hive_metastore" catalog is the default one, there should not be any need to sp...
The exact error reported by Databricks is:[RequestId=f27975cd-7589-4463-8c03-6015893ee133 ErrorClass=INVALID_PARAMETER_VALUE] Invalid input: RPC CreateSchema Field managedcatalog.SchemaInfo.catalog_name: name "" is not a valid name
So, I can't figure out how to do moving average as custom calculation in Databricks dashboard. I'm applying many different filters and the denominator of the metric has to change dynamically based on the chosen filters. So, in this case using `Custom...
Currently, Databricks dashboards do not support applying a moving average “custom calculation” on top of another custom metric that itself is dynamic with respect to the filters. Workarounds Segmented SQL Datasets: Pre-compute the filtered sets (as ...
We are currently building our dashboards in Apache Superset. With the Git integration in Databricks AI/BI Dashboards, the development process has improved a lot. So we are thinking about switch to Databricks AI/BI Dashboard.One pain point in Databric...
Closing the loop here: Update: The PMs are updated by me and are aware of the usecase and this request (it will help hasten priortization). Thanks!
Long story short, I'm not sure if this is an already known problem, but the Auto Stop feature on SQL Warehouses after minutes of inactivity is not working properly.We started using SQL Warehouses more aggressively this December when we scaled up one ...
This is a good catch. Auto termination is something you tend to set and trust it will do the right thing .Wondering if the Databricks team managed to fix this, seeing it has been half a year since the problem was raised.This is also important for ou...
