Here's your Data + AI Summit 2024 - Warehousing & Analytics recap as you use intelligent data warehousing to improve performance and increase your organization’s productivity with analytics, dashboards and insights. Keynote: Data Warehouse presente...
Hi Databricks community,I'm new to Databricks and I'm trying to connect Databricks to Azure MySQL using Azure Data Factory.When I run the pipeline, I get the following error: ErrorCode=AzureDatabricksCommandError, Hit an error when running the comma...
Could you provide more details about high level flow or design you are planning? Not understanding completely but you can take a look here how to connect ADF with Databricks via PATs -> https://www.sqlshack.com/integrating-azure-databricks-with-azure...
Hi folks, I am creating a SQL dashboard and want to know if I can re-use datasets within the same dashboard. The screenshot below captures what I would like to do pretty well, but to summarize... I need to run an computationally expensive query and w...
AgreementYes, a feature to support nested datasets in Databricks AI/BI dashboards would seem convenient, e.g. as in both examples in the OP's screenshot.Specific needA use case that comes to mind would be to avoid the maintenance of parallel code acr...
Hi community,we have several databricks workspaces per different teams and in each workspace several use case covered.How can we trace the costs per individual use cost for SQL Warehosue / serverless in case we use one SQL warehouse per workspace to ...
You have two related questions on Databricks SQL Warehouse (serverless) cost attribution and resource sharing across workspaces. Here’s a clear answer based on current Databricks capabilities and industry practice. Tracing Costs per Use Case (SQL War...
I'm trying to deploy a minimal Streamlit app as a Databricks App on a fresh Free/Community workspace and I get:"Compute error - App creation failed unexpectedly. Please remediate by deleting the app."As alternatives, I also tried the API and the CLI ...
Good morning, everyone.The app is working again. I've already tested it and it's working.
Hi,I want to create a visualization where i want to show top 10 playerName with most sum of counter by dayDate. I reached till here. But i don't know how to limit top 10 playerName.Any help will be highly appreciated.
Finally what i did is:create a new data source with is query:WITH daily_totals AS (SELECTevent_date,playerName,SUM(counter) AS total_counterFROM your_tableGROUP BY event_date, playerName),ranked AS (SELECTevent_date,playerName,total_counter,ROW_NUMBE...
Hi, I've created a service principal and setup OAuth - I can successfully call REST APIs to see workspace details. But, when I try to list the genie spaces in a workspace, I get {} as the result. In the UI, I can see two genie spaces I've created. An...
Hi @hch_fiq ,You need setup proper permissions for your service account to see genie space. Look at below Genie space ACL table from documentation. When there's no ACL set you won't see anynting. So, your Service Principal needs at least CAN VIEW/CAN...
Hello all (and, if you are watching these posts, Databricks employees/representatives ),I hope you take this idea into account - I haven't been able to find a place where one can vote on features, but I believe that this idea would make very real im...
Hi @QueryingQuagga , We always appreciate feature requests! If you are working with an account team, you can always pass this information along to them, they have the ability to submit feature requests and vote on existing feature requests as well. I...
I am attempting to create a Vizro dashboard in a databricks notebook. I have validated that the code to generate dashboard works is accessible when running the notebook on my local machine. However, when attempting to compile the dashboard on a datab...
This issue is common when attempting to run interactive dashboards (such as Vizro or Plotly Dash) within Databricks notebooks; many users face the "This site can't be reached" message because the dashboard tries to launch a local web server on an add...
I'm relatively new to Databricks and Spark and have decided to create a Spark cluster with AWS under the free 14 day trial.The JSON to the cluster is as follows:{"data_security_mode": "DATA_SECURITY_MODE_DEDICATED","single_user_name": "me@gmail.com",...
Hi @neesam, while the command is hanging, check the driver logs from the cluster page to see if they provide any clues about what might be causing this issue.
I was trying to install a personalized .whl file located in the "shared" folder but I'm obtaining this error:org.apache.spark.SparkException: Process List(/bin/su, libraries, -c, bash /local_disk0/.ephemeral_nfs/cluster_libraries/python/python_start_...
The error indicates a permissions issue when trying to install your custom .whl file from the "Shared" folder in Databricks using dbx and Spark. Specifically, the error [Errno 1] Operation not permitted suggests the process does not have the required...
I recently updated my Databricks VS Code extension from version 1.4.0 to 2.4.0. After the update, I noticed that:The "Configure Sync Destination" button is missing.The sync destination format has changed to a remote folder instead of the previous set...
The upgrade to Databricks VS Code extension v2.4.0 changed the sync destination system from the previous workspace directory setup, removing the "Configure Sync Destination" button and setting the default to a more rigid remote folder structure (comm...
Hi,We publish listings on the Databricks Marketplace.Several times a week, we receive a lead notification stating: "db sp has installed 'Etsy Dataset'."It’s always the same dataset and the same name, so it’s likely a technical error.Does anyone know ...
Yes, this is a known issue reported by other Databricks Marketplace publishers: receiving frequent lead notifications stating "db sp has installed 'Etsy Dataset'" for the same dataset and name, which suggests an underlying technical error in the lead...
We are trying to schedule updates to streaming tables and materialized views in Azure Databricks that we have defined in dbt.Two options we are considering are `SCHEDULE CRON` and just scheduling `dbt run` commands via CI/CD. The `SCHEDULE CRON` opti...
Your understanding of the trade-offs between using SCHEDULE CRON on streaming tables/materialized views and the dbt run orchestration in Azure Databricks is largely correct, and you’ve identified the two main pain points: orchestration of dependencie...
We recently enabled the preview feature to refresh power bi datasets from data bricks workflow, but when we run the job data changes aren't being applied despite a completed status in refresh history. Could anyone clarify whether this is intended pre...
This is a known issue with the Power BI/Databricks integration preview for refreshing Power BI datasets from Databricks workflows. Even when a job completes and the refresh history shows success, actual data changes—such as newly inserted rows—are of...
I would like to be able to express something like: WHERE columnA = "valueA" OR columnB = "valueB" but on the Widget. I only see the possibility that chaining filters that are processed as AND operator. Am I missing something?
Most widget filter interfaces only support multiple filters using an AND operator, meaning each condition must be true for a row to be included. If you want to use an OR operator, such as filtering where columnA = "valueA" OR columnB = "valueB", most...
