Here's your Data + AI Summit 2024 - Warehousing & Analytics recap as you use intelligent data warehousing to improve performance and increase your organization’s productivity with analytics, dashboards and insights. Keynote: Data Warehouse presente...
We are using dbt core and running our transformations in a dbt task. Since a few months ago we have started to see a warning message in our runs. This started happening after an update of dbt core and it seems that Databricks needs to update the def...
Both of you missed the sentence "Databricks needs to update the definition of the generated profiles file."The warning is regarding the structure of the profiles.yml which is generated by Databrick. This is not anything I as a user can change accept ...
Hello,I am new to Databricks and was asked to evaluate the new Apps function as a way to publish dashboards. We do not use Databricks today. We have an Azure SQL database we use as a home grown warehouse. The question is can I connect directly to ...
To connect your Azure SQL database to Databricks Apps, you will need to pull the data from the existing database into Databricks. Databricks Apps are designed to leverage the data and features within the Databricks platform, which means that the data...
Hi ,We are looking for a option to copy tables more than 50 TB to be copied from syanpse to databricks on weekly basis , please suggest of there are any feasible ways for samewe are using connector but it is taking too long to copyhttps://learn.micro...
There is no databricks documentation on this as it is only involved for a very tiny bit:"CREATE TABLE catalog.schema.table USING PARQUET LOCATION 'url_to_the_parquet_files'.All the rest is done in Azure Data Factory, or you can even use the built-in ...
So I did U2M authentication for account-level operations -databricks auth login --host <account-login-url> --account-id <account-id>then I tried to run code-workspaces = account_client.workspaces.list()workspace_obj=account_client.get_workspace_clien...
I am doing using a service principal so my .databrickscfg has -[databricks-demo]client_id = -----client_secret = ----host = https://accounts.cloud.databricks.com/account_id = ----
Hi Folks, I am looking for any documentation on how Databricks Dashboards handle creating tooltips. Specifically I am interested in dashboards created in the SQL Warehouses (not Workspace Clusters, since using libraries like plotly can have custom to...
Thank you for sharing this. Unfortunately it doesn't really contain much information on how to manage the tooltips that pop up when you hover over a bar or line in a chart/figure. That's what I am looking to customize in (or at least remove from) my ...
Hi Team good day.I'd like to check on the pattern where dashboards are developed in PowerBI Desktop which are then published to PowerBI Online, using data from Databricks on AWS. In order to refresh the dashboards in PowerBI Online (whether on schedu...
To refresh dashboards in PowerBI Online using data from Databricks on AWS, an on-premises data gateway is not required. This is different from the scenario with Redshift, where an on-premises data gateway is necessary. For Databricks on AWS, you can ...
Hello everyone,I would like to inquire about the possibility of increasing the default limit of concurrent queries on the cluster which is set to 10.While researching this topic, I noticed that there is no official documentation available regarding t...
Did you get an answer how to set spark.databricks.maxConcurrentRuns and spark.databricks.maxConcurrentQueries parameters in sql warehouse serverless cluster?
We have Azure Databricks - SQL Serverless Warehouse with Classic configuration. The config details shared in below snip. As per the Auto Stop config, the Serverless warehouse should be stopped after 10 min of inactivity. However, when we were monitor...
@Digvijay_11 A dbsql warehouse can be triggered not only by a query from a JDBC/ODBC client but also by an idle session, as sessions are managed within the backend clusters. Clusters must be running to maintain these sessions, even before a query is...
Hello,I'm designing a system that combines a lake-house and a real-time path that federates across multiple databases. The requirements led us to adopt Starburst Galaxy (a cloud-hosted distributed query engine) for query federation across the various...
Hello there!So any updates so far? Also interesting in this solution.
I added a filter with the :name_parameter to the dashboard and it works fine.But if I add a date range picker for the date axis it doesn't do any effect (standalone, without the other filter enabled, it is working). It seems to me that the date pick...
Thanks a lot. I guess this was a case of not seeing the forest for the trees. But still, maybe the could add a tooltip or so for this field.
Hi,As a formal requirement in my project I need to keep original, raw (mainly CSVs and XMLs) files on the lake. Later on they are being ingested into Delta format backed medallion stages, bronze, silver, gold etc.From the audit, operations and discov...
Many thanks for raising it @bobmclaren .@Retired_mod Do you have any updates on the above?
We work with a application that connects to our Databricks serverless SQL warehouse via Databricks JDBC driver. It runs a few thousand SQL select statements everyday, and a small percentage of them will fail with the following error details: java.sql...
We are trying to schedule updates to streaming tables and materialized views in Azure Databricks that we have defined in dbt.Two options we are considering are `SCHEDULE CRON` and just scheduling `dbt run` commands via CI/CD. The `SCHEDULE CRON` opti...
Hi all, I am currently stuck because runing DESCRIBE DETAL in my catalog's tables return an error like:[PARSE_SYNTAX_ERROR] Syntax error at or near 'hive_metastore'. SQLSTATE: 42601 (line 1, pos 21)Why is this not supported in the dasboards but the s...
Both DESCRIBE and DESCRIBE DETAIL are supported by databricks sqlBut DESCRIBE DETAIL is only supported for tables not for viewsError you are getting is due to the SQL statement you have used is not valid and hence cannot be executedPlease check your ...
I'm interested in using the API to list all dashboards, but the API shows legacy dashboards only. Are there any plans to fix this API call to show all dashboards?I'm using this API call: https://docs.databricks.com/api/workspace/dashboards/listAnd th...
I understand the API is valid for legacy dashboards, but I need to include all dashboards, legacy or not.,
