Here's your Data + AI Summit 2024 - Warehousing & Analytics recap as you use intelligent data warehousing to improve performance and increase your organization’s productivity with analytics, dashboards and insights. Keynote: Data Warehouse presente...
I am looking to monitor my SQL Warehouse, especially the 'Running Clusters' metric that is available in the monitoring tab of the warehouse. This shows the start and shut down time as well as the number of running clusters:The issue I have run into i...
Hey @Dave_Nithio To monitor the “Running Clusters” metric for your SQL Warehouse, you can use the Databricks Cluster Events API. This API retrieves a list of events related to cluster activity, such as start and shutdown times, and provides paginated...
Hello All!I have a python script which utilizes the databricks SQL for python package in order to pull a databricks table into a pandas dataframe which is used to create a table in a Spotfire report. The table contains ~1.28 million rows, with 155 co...
Hey @barchiel33 ,After reviewing your context further, I believe the most effective approach would be to set up an automated pipeline within Databricks that periodically extracts data based on the frequency you need (daily, weekly, hourly, etc.), cre...
Hi!I have a multi-page dashboard based on a handful of tables sharing a common run_id field.I created a dropdown filter widget on the first page to filter datasets based on the table's run_id field.It worked fine for this page, but how to reuse it fo...
I have the same problem here. filters are not applied while switching pages
I have a delta table with columns like below format. I am querying Databricks using databricks-jdbc 2.7.1 driver. I want to convert the result set to a nested java object.Order_actualPickupAddress_residential -> stringOrder_pickupRequestDetails -> ar...
Here are some suggestions/ideas to consider: To map hundreds of nested fields from a Databricks Delta table to complex Java objects like the Order object described, consider the following approaches: 1. Using Libraries for Object Mapping Utilize ...
Application connects to Databricks serverless SQL warehouse via Databricks JDBC driver. It executes SQL select statements only. We see small number of statements failed each day with the following error detail: java.sql.SQLException: [Databricks][JDB...
Interesting issue, looks like a Cluster Load issue. Recommend filing a support ticket with Databticks to investigate, in case if you haven't already
What’s the best and seamless way to connect Quick Books data to Databricks? Is there any recommended third-party tool or direct connection using JDBC/ODBC?
Hi Akshay, You can use third-party tools that specialize in integrating with external platforms like Databricks. Tools such as CData, QODBC, or similar database connectors provide JDBC and ODBC drivers for QB. These drivers allow you to query Quick...
In the databricks Tutorial (https://docs.databricks.com/gcp/en/sql/get-started/sample-dashboards) , it says we can right click the widget and choose "Download as CSV". However, in my databricks dashboard, when I right click the widgets, there is no ...
I find the reason is that my Admin didn't enable "SQL result download" option. Now it is solved. Thanks ALL.
Hi everyone,I'm currently setting up access controls in our Databricks development workspace. The goal is to enable business users to explore data and build their SQL skills within the workspace itself (e.g., via SQL editor or notebooks), but prevent...
Hi @teixeire ,To prevent external tools like Power BI or DBeaver from connecting to your SQL Warehouse, one effective approach is to restrict personal access token (PAT) creation for users who should only query data inside the Databricks workspace.Th...
Hi,I am currently installing UCX version 0.58.0 in Non-Unity enabled Databricks Workspace. It installed successfully and the job completes without errors. However, The Dashboard that goanna generate showing "0 Total Tables" and other metrics as 0.Has...
Hi,Still facing the same issue actually, I am trying to automate the UCX installation using bashscript. while running the bashscript the job starts and run successfully with all run jobs. Even all jobs Ran successfully, I am getting 0 tables and 0 va...
Hello All, We want to update our postgres tables from our spark structured streaming workflow on Databricks. We are using foreachbatch utility to write to this sink. I want to understand an optimized way to do this at near real time latency avoidi...
If near real-time latency is critical, focus on optimizing parallel writes with batch updates (Option 1).If you prioritize transactional stability and can tolerate slightly higher latency due to staging, continue refining your temporary table with ...
java.sql.SQLException: [Databricks][JDBCDriver](500051) ERROR processing query/statement. Error Code: 0, SQL state: null, Query: SELECT `ma***, Error message from Server: org.apache.hive.service.cli.HiveSQLException: Error running query: org.apache.s...
The executor does not seem to have enough memory to process the assigned tasks, OOM error.
We work with a application that connects to our Databricks serverless SQL warehouse via Databricks JDBC driver. It runs a few thousand SQL select statements everyday, and a small percentage of them will fail with the following error details: java.sql...
Hi,I am trying to install UCX for Databricks workspace using the Databricks CLI. When I attempt to install a specific older version instead of the latest, I get a dashboard creation error. This seems to be due to the April 7 update, which removed the...
Hello @meghana_tulla! Older versions of UCX may fail to install because they try to create legacy dashboards, which are no longer supported following the April 7 update. To work around this, make sure your Databricks environment is aligned with the l...
Hi Databricks Community,I'm trying to understand which team is responsible for maintaining the Databricks Connector for Looker Studio . We’re currently facing a major performance bottleneck with how this connector operates.Specifically:The connector ...
Thank you for the feedback! Unfortunately there is a limitation in Looker Studio Community Connector API. getData method does not specify aggregation expectations for the data source. Therefore, a connector is expected to retrieve non-aggregated resu...
Hi everyone,We are having an issue using parameters with EXECUTE IMMEDIATE statements, when running on SQL serverless running DBSQL v2025.15 (currently in preview channel):declare or replace var query = "SELECT :PARAMETER_1"; EXECUTE IMMEDIATE query;...
@Louis_Frolio thanks for the detailed explanation, that makes more sense.
