Here's your Data + AI Summit 2024 - Warehousing & Analytics recap as you use intelligent data warehousing to improve performance and increase your organization’s productivity with analytics, dashboards and insights. Keynote: Data Warehouse presente...
I have an XML string in my table and I want to extract a full node from it using SQL functions, without needing to parse it in full i.e. from '<a><b><c>c1</c></b></a>' I want '<b><c>c1</c></b>' (preferably) or '<c>c1</c>'.What little I can find that ...
Thanks very much. I had over-simplified my example perhaps, and got stuck on the idea of using xml functions, particularly because I'm converting existing SQL from another DB, and wasn't being open-minded enough.The actual data has multiple instance...
Hello professionals and seasoned functional consultants,I recently completed the Databricks Data Engineer Associate certification and have also finished multiple courses from Udemy, Databricks learning events, and Alex The Analyst / Analytics Builder...
I'm unable to start my SQL warehouse (Serverless) due to a RESOURCE_EXHAUSTED error.Error message: Clusters are failing to launch. Cluster launch will be retried. Request to create a cluster failed with an exception: RESOURCE_EXHAUSTED: Cannot create...
Hi @sharath007, Just checked internally. This specific RESOURCE_EXHAUSTED: Cannot create the resource, please try again later message for a serverless SQL warehouse normally indicates that the backing serverless compute pool has run out of capacity (...
Hi,We are implementing external dashboard embedding in Azure Databricks and want to avoid using client secrets by leveraging **Azure Managed Identity** with **OAuth token federation** for generating the embedded report token.Following OAuth token fed...
Did you get any information whether this is on their roadmap?I came across this issue last week and the documentation doesn't have anything about this limitation.
Hello, I was wondering if there was sample code indicating how a java program might leverage Databricks Connect to query the table in the Free Edition of Databricks? I would like to use Connect as I am trying to avoid JDBC and its overhead and t...
Hi — welcome to Databricks! Unfortunately, Databricks Connect v2 (DBR 13.3+) does not support Java — it only supports Python, Scala, and R. The legacy v1 did support Java, but it's been deprecated and is end-of-support. That said, here are your optio...
The documentation gives an example of a period to date measure. The intention is to give the sum (in this example) of orders in the current year to date. However, what actually happens is that the measure returns the result for the most recent period...
Hi Ashwin. Thanks for looking into this. Your description of the behaviour and the options matches my understanding, but from my perspective this is a defect: I don't think there's a reasonable interpretation of "period to date" (which is the documen...
Hi - I saw in this dbx post that there is color scale as a column display type. https://docs.databricks.com/aws/en/dashboards/manage/visualizations/tablesHowever I don't see color scale as an option on my dbx. I only see tables for Basic, Image, HTML...
Hi @brianz, @balajij8 is right. You can see from the docs that for table visualisations, the Display type list is Basic, Image, HTML, JSON (no Colour scale). For pivot tables, under Styles → Display type for fields added as Values, the options inclu...
Hello everyone,I am trying to create a cumulative support ticket chart. I need one line representing the cumulative sum of tickets by creation date, and another line for the cumulative sum by resolution date.While it's easy to achieve this by creatin...
Hi @lucca_luna, Great question. The cross-filtering issue you're hitting is because the dashboard treats separate datasets as independent. The workaround is to reshape your data so both date dimensions (creation and resolution) live in a single colum...
How is Cluster auto-scaling in Databricks different from Dynamic Allocation in Yarn
From spark's documentation dynamic resource can be used on more then just YARN. Is it possible to use it in databricks as well and how does serverless work under the hood in these regards?
I followed the official Databricks documentation("https://docs.databricks.com/en/_extras/notebooks/source/mongodb.html")to integrate MongoDB Atlas with Spark by setting up the MongoDB Spark Connector and configuring the connection string in my Datab...
try using single-user cluster or non-isolation cluster instead of shared cluster
I’m currently working with two workspaces – one for DEV and one for PROD.I’m trying to understand how I can keep the Genie/Dashboard functionalities in sync (mirrored) between these two environments. What is the best way to organize this workflow?Ide...
Hi ATN, Good question. Based on what I know, dataset_catalog and dataset_schema in the dashboard DAB resource currently apply to datasets that use SQL queries (i.e., the queryLines property), but they do not resolve for datasets that reference metr...
When I click on a trace in the Databricks MLflow UI, it defaults to opening a "summary" tab. I have a somewhat complex trace, so this summary is getting bloated and I would like to clean it up.Nowhere, as far as I can tell, is the summary tab describ...
Hi @lkt1, I took a quick look and tested it in my workspace to better understand the issue you mentioned. As of today, the Summary tab isn’t user‑configurable. What’s displayed there and how inputs/outputs are collapsed, or truncated, is enti...
I created an external connection to an Oracle Database 12c Standard Edition Release 12.1.0.2.0.The connection is working, I executed successfully a lot of queries with different filters and joins. But I couldn't find a way to make any date filter pus...
Hi @diogofreitaspe, This is a known behavior with how the Lakehouse Federation Oracle connector serializes date and timestamp literals during predicate pushdown. When Databricks pushes the filter down to Oracle, the date value gets rendered as a bare...
Hi everyone,I’m exploring the new Databricks Metric Views (Semantic Layer) and have two questions regarding programmatic management and UI visualization.1. Parser Disparity: spark.sql vs. SQL WarehouseI'm noticing that CREATE OR REPLACE VIEW ... WITH...
Hi @smpa011, METRIC VIEW DDL ON ALL-PURPOSE COMPUTE (spark.sql / %sql) The CREATE VIEW ... WITH METRICS DDL requires Databricks Runtime 17.2 or above. This applies to both SQL warehouses and all-purpose clusters. If your notebook is attached to a clu...
With the new metric views, I am unable to understand the grouping logic in the following setup:I have a table with timestamps and I define dimensions as follows:dimension: - name: timestamp expr: timestamp - name: date expr: DATE(timestam...
Hi @Malthe, This is a nuanced aspect of how metric views resolve window measure dimensions. The key behavior you are seeing comes down to how the metric view engine matches your query's GROUP BY columns to the dimensions defined in the window clause....
