Here's your Data + AI Summit 2024 - Warehousing & Analytics recap as you use intelligent data warehousing to improve performance and increase your organization’s productivity with analytics, dashboards and insights. Keynote: Data Warehouse presente...
I have created a Ai/Bi dashboard on Databricks and set it to refresh every 1 minute. However, the dashboard is not refreshing as expected. Interestingly, when I terminate the warehouse, it triggers the warehouse but still does not refresh per minute ...
Hi @Alberto_Umana ,I have followed the correct approach to create and schedule the dashboard for a 1-minute refresh, but it's not updating every minute as expected.I have attached images for your reference. Please take a look.
Hello Let's say I create a SQL Warehouse on Current Channel (2024.40) and there is a new release (2024.50).Would I need to take some actions(reboot for example) so that my warehouse uses the 2024.50 version or it should run on the 2024.50 whenever th...
Hey @onlyme ,The Channel in Databricks SQL Warehouse has two options:1.Current: This corresponds to the latest stable version released by Databricks and updates automatically.2.Preview: Similar to a beta version, it includes improvements and new feat...
Hello everyone,I created a report using PowerBI Desktop that I successfully connected to Databricks. However, in PBI Service, the visuals are not displayed and I'm asked to edit credentials of the semantic model. When doing so, I get the following er...
Hi @GarCab, Can you try with a PAT token, just to confirm it works? https://learn.microsoft.com/en-us/azure/databricks/partners/bi/power-bi#connect-power-bi-desktop-to-azure-databricks
Hi,I think I have a similar issue to the one in this post, but the answer isn't detailed enough for me.I have a list defined in my first task, which contains the items I want to iterate through [1,2,3,4]. When I use it as Inputs to the For Each frami...
Thank you both very much, I've nailed it . I have accepted Walter_C's answer as solution because Step 2 is what I was missing. Thanks MariuszK as well for your contribution.
Hi!I have a multi-page dashboard based on a handful of tables sharing a common run_id field.I created a dropdown filter widget on the first page to filter datasets based on the table's run_id field.It worked fine for this page, but how to reuse it fo...
Hello @alex456897874, can you try this approach? Create a Global Parameter: Navigate to the Data tab of your dashboard.Define a new parameter based on the run_id field. This parameter will be used globally across all pages. Configure the Filter W...
Do we have same functionality if we use Databricks with Fabric as it provides with Azure?
hi @DEShoaib Are you planing to move Dedicated (T-SQL) pool or Spark code?With Databricks you can replicated all features from Azure Synapse, you have possibility to use PySpark and Databricks SQL. MS Fabric has nice integration with Power BI and eas...
For Databricks SQL connector for python, the list of fields returned by Cursor.columns() is listed in here (like TABLE_CAT, TABLE_SCHEM, TABLE_NAME, COLUMN_NAME). Could someone please share an exhaustive list of fields (including short description ...
All those fields are explained in the doc. For example search with `TABLE_CAT`
I followed the official Databricks documentation("https://docs.databricks.com/en/_extras/notebooks/source/mongodb.html")to integrate MongoDB Atlas with Spark by setting up the MongoDB Spark Connector and configuring the connection string in my Datab...
Hi @vidya_kothavale ,Could you try to change "spark.mongodb.input.uri" to following?spark.read.format("mongodb").option("spark.mongodb.read.connection.uri"
Hi everyone,I hope you're all doing well.I'm experiencing some challenges with Databricks SQL, and I wanted to reach out to see if others have encountered similar issues or have suggestions for troubleshooting. Below is a summary of the problems I'm ...
Hi @Walter_C,Thank you for your input and support regarding the challenges I’ve been experiencing with Databricks SQL.I followed up with support, and they confirmed that these are known issues currently under review. Here’s a summary of the response:...
Dear all,(migrating for an on-premise Oracle ...)The question is in the subject: "What is the equivalent of Oracle's CLOB in Databricks" ?I saw that the "string" type can go up to 50 thousands characters, which is quite good in most of our cases, but...
Hello;Thanks for the answer.For the concatenation itself, it is not an issue.My question is "is Databricks supporting something bigger than the 'string' data-type" ? Thanks
Using workflows, is there a way to obtain the task name from within a task?EX: I have a workflow with a notebook task. From within that notebook task I would like to retrieve the task name so I can use it for a variety of purposes.Currently, we're re...
Hi @EWhitley,Would {{task.name}} help in getting the current task name?https://docs.databricks.com/en/workflows/jobs/parameter-value-references.htmlPass context about job runs into job t
Long story short, I'm not sure if this is an already known problem, but the Auto Stop feature on SQL Warehouses after minutes of inactivity is not working properly.We started using SQL Warehouses more aggressively this December when we scaled up one ...
Is this still being investigated by Databricks? I'm seeing similar behavior that's costing us a lot of money.
We are experiencing the following issues.Description:I encountered an issue while executing a Spark SQL query in Databricks, and it seems to be related to the query optimization phase. The error message suggests an internal bug within Spark or the Sp...
Update:Response from the Databricks Team.SymptomsInternal Error During Spark SQL Phase Optimization.CauseDataBricks PG Engineering team confirmed that this is indeed a bug in CASE WHEN optimization & they are working on the fix for this issue.Resolut...
I have a Python script workflow with 2 tasks: Task A and Task B.When task A has data, this is shared to Task B via createOrReplaceGlobalTempView with no issues.The goal is: When A has no data, skip the Task B and also set the workflow status to "Skip...
To achieve the goal of setting the workflow status to "Skipped" when Task A has no data, you can use the "Run if" conditional task type in Databricks Jobs. This allows you to specify conditionals for later tasks based on the outcome of other tasks.ht...
I have some legacy software that only runs on Windows, but that can be driven via Python. Is it possible to set up compute resources that run Databricks Container Service on a windows base image, so that I can then add this legacy software and work w...
Unfortunately this is not possible, as part of the requirements you need to use an Ubuntu image: https://docs.databricks.com/en/compute/custom-containers.html#option-2-build-your-own-docker-base
