Here's your Data + AI Summit 2024 - Warehousing & Analytics recap as you use intelligent data warehousing to improve performance and increase your organization’s productivity with analytics, dashboards and insights. Keynote: Data Warehouse presente...
Hi there,I'm encountering the following error while attempting to read a file from R2 storage:[FAILED_READ_FILE.NO_HINT] Error while reading file r2:REDACTED_LOCAL_PART@user_id.r2.cloudflarestorage.com/data/20250128_160228_54805_wpkza_e38751cf-969e-...
But the error is from aws and seen when the payload size is incorrectly defined in the contentLength parameter. Caused by: com.amazonaws.SdkClientException: Data read has a different length than the expected: This sounds like a bug, Was this resolved...
Hello,We recently detected an issue in our product deployment with terraform.At some point, we have some java code that creates a schema in "hive_metastore" catalog.Since "hive_metastore" catalog is the default one, there should not be any need to sp...
The exact error reported by Databricks is:[RequestId=f27975cd-7589-4463-8c03-6015893ee133 ErrorClass=INVALID_PARAMETER_VALUE] Invalid input: RPC CreateSchema Field managedcatalog.SchemaInfo.catalog_name: name "" is not a valid name
We are currently building our dashboards in Apache Superset. With the Git integration in Databricks AI/BI Dashboards, the development process has improved a lot. So we are thinking about switch to Databricks AI/BI Dashboard.One pain point in Databric...
Closing the loop here: Update: The PMs are updated by me and are aware of the usecase and this request (it will help hasten priortization). Thanks!
Long story short, I'm not sure if this is an already known problem, but the Auto Stop feature on SQL Warehouses after minutes of inactivity is not working properly.We started using SQL Warehouses more aggressively this December when we scaled up one ...
This is a good catch. Auto termination is something you tend to set and trust it will do the right thing .Wondering if the Databricks team managed to fix this, seeing it has been half a year since the problem was raised.This is also important for ou...
Certain tables have this configuration enabled, whereas others do not have it.Delta.checkpointPolicy=v2This is affecting the behavior of the ANALYZE commandIf flag is enabled : Table stats are not visible after doing the DESCRIBE command using SINGLE...
Greetings @yshah , here are some helpful hints/tips/tricks to guide you. To access table column statistics when checkpoint V2 is enabled, you can follow these guidelines: Utilize Databricks Runtime 13.3 LTS or Higher: Ensure that you are using Datab...
Is there any official word as to when we can expect Chloropleth or Marker Map visuals in AI/BI Dashboards? I realize Legacy Dashboards are still supported, but it feels uncertain to build new ones with AI/BI Dashboards in GA.
Choropleth Maps - https://docs.databricks.com/aws/en/dashboards/visualizations/maps#choropleth-options
In on-Prem we currently create dashboards which basically provides year over year or quarter over quarter comparison. When the variance is more than certain threshold for a particular data intersection business has the option to add comments, also so...
There is no inbuilt features to built-in “commentary workflow” like your Java app. But you can custom build one. Here are the steps to build one.1. Use Databricks SQL Dashboards or Lakeview Dashboards for your YoY/QoQ variance analysis.2. Create a D...
I know for spark clusters, the number of driver will remain the same and only the number of workers increases when auto scaling. But for sql classic warehouse I'm a bit confused, since it's scaling number of cluster not driver.
SQL Classic Warehouses:Scaling is done by adding or removing entire clusters, not just workers.Each cluster has its own driver and set of workers.So when autoscaling happens, more clusters (each with a driver + workers) are added to handle concurrent...
Problem: Databricks Apps timeout when connecting to SQL Warehouses, but same connection works locally. First Databricks App I've attempted to create. Symptoms:Local Streamlit app works fine with Databricks SQL connectorDeployed Databricks App hangs ...
Hi @johnv, Do you have any error messages you can share, or is it a silent connection timeout? I assume you confirmed that the SQL Warehouse had started up? (they can take several minutes to start up sometimes) If that's not the issue, you could al...
Is there a way to calculate the percentage of total using the Databricks dashboard's custom calculations and also the cumulative percentage? What function would be equivalent to the DAX All and AllSelect functions, but using Databricks' aggregation f...
Hi @genebaldorios ,I don't use databricks dashboards on my project (we are PBI shop), but I guess you need to use AGGREGATE OVER clause with cumulative frame:
I have a Databricks Dashboard that contains several choropleth maps. They all stopped working today. I can see a background image, but no data is represented on them. The dataset seems ok and the visualizations don't show any errors. I don't see ...
Glad to hear you figured it out. Appreciate you updating the thread with the fix, @Unimog!
Hi everyone,I'm trying to query data from our adls2 delta lake using a serverless sql warehouse. We've already set up private connectivity via NCC, but hitting a snag when running queries like:SELECT * FROM delta.`abfss://container@xxx.dfs.core.windo...
Hi, thanks for the detailed explanation.Unfortunately, configuring fs.azure.account.key in the Serverless advanced options didn’t help (I’m sure I wrote it correctly) - still receiving the same error.I saw in some sources around the net that I should...
I am using streaming tables to read from multiple parquet files and create a table in my raw layer.Tech Stack -- dbt-databricksWhile loading I have a column called "ets" in my source which is nothing "milliseconds since Jan 1st 1970." I want this to ...
Hi @singh_tushar_14 ,Since your ets columns already contains integer that represents milliseconds since Jan 1st 1970, can't you just use Spark SQL fuction? You don't need to add anything to 1970-01-01, since that information is already "encoded" in y...
serverless compute:cannot find column index for attribute error We are seeing this error if there are a complex view, the same view runs fine in sql warehousebut break in serverlesss compute onlylooks like serverless compute has issue parse large co...
ok, I will test on latest version 4 and will let you know if any more issues, thanks
Hi All,I've encountered what seems like a very specific bug. For some reason, running a query with the following where clause results in an execution error.where lower(organization_name) like '%trac%'There's no execution error when trying "tra" and i...
@JPan I noticed you had the new SQL editor mode on in your video. For what it's worth, I flicked between both when testing and both were successful. Given that you're finding it's only happening on some fields, do you think this is to do with special...
