- 4792 Views
- 2 replies
- 1 kudos
Resolved! Deploy dashboard with asset bundle
Hello, I have some questions regarding dashboard development using Asset Bundles.I have been following the procedure for developing dashboards by referring to this page: Databricks CI/CD for Dashboard Developers.Here is the workflow I followed:Create...
- 4792 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Seunghyun, This is a common workflow question when getting started with AI/BI Dashboard deployment through Databricks Asset Bundles. Here is a walkthrough of the recommended approach to maintain a single dashboard and handle ongoing modifications...
- 1 kudos
- 1911 Views
- 3 replies
- 0 kudos
Read Sqlite file in to create delta table/dataframe with live connection
I have a sqlite database that I want to read into databricks to create delta tables/dataframes in Python that I can export to power BI and have a live connection. When there is new data added to my sqlite data base, the changes will need to reflect i...
- 1911 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @lw2, The key to getting a "live connection" end-to-end is replacing the manual CSV export with a scheduled pipeline that writes directly to Delta tables, then connecting Power BI to those Delta tables via DirectQuery. Here is a complete approach....
- 0 kudos
- 2936 Views
- 4 replies
- 0 kudos
Asset Bundles Overriding Existing Jobs (despite different name_prefix)
Hi there, I'm seeing what seems to be unexpected behavior on databricks asset bundle deployment and I'm hoping I can get clarification on this.Basically, what I'm trying to do is to deploy the same asset bundle twice (two different variations), with ...
- 2936 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @ChristianRRL, This behavior comes down to how Databricks Asset Bundles track deployed resources using Terraform state, and specifically where that state is stored locally. HOW BUNDLE STATE TRACKING WORKS When you run "databricks bundle deploy", t...
- 0 kudos
- 1238 Views
- 2 replies
- 1 kudos
Getting Error when connecting azure databricks to azure sql server using lakeflow connect
Hi, Can anyone please let me know how to resolve this error . I am trying to connect azure sql server to azure databricks using lakeflow connect data ingestion. I am able to create the connection but afterwards it gives me error as :Error starting ga...
- 1238 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @RutujaKadam, The error you are seeing, "Error starting gateway compute resources" with a message about VM quota exhaustion, is related to your Azure subscription's vCPU quota rather than a misconfiguration in Databricks itself. Here is what is ha...
- 1 kudos
- 1532 Views
- 2 replies
- 1 kudos
Need for additional flow control (shortcomings with "run if dependencies")
Maybe there is a way to do this that my team can't figure out but we have a process that kind of looks like this:The main focus of this job1 and job2 but in theory let's extend this issue to any number of linear jobs. So the idea behind this is:job2 ...
- 1532 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @kenmyers-8451, You are describing a real gap in how "Run if dependencies" interacts with If/else condition task outcomes, and I want to walk through the current tooling so you can find a workable pattern. UNDERSTANDING THE BEHAVIOR When an If/els...
- 1 kudos
- 1802 Views
- 3 replies
- 0 kudos
Can we add a column comments for a materialized view on Azure Databricks?
I want to understand whether it’s possible to add or update column comments on an existing materialized view in Azure Databricks, and if so, what command should be used—especially when updating comments for multiple columns at once. Here’s my situati...
- 1802 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Smriti2, Yes, you can absolutely add column comments to a materialized view on Azure Databricks. There are two approaches you can use. OPTION 1: ALTER MATERIALIZED VIEW (Recommended for Existing Views) Since your materialized view already exists,...
- 0 kudos
- 2138 Views
- 3 replies
- 1 kudos
Databricks Genie dashboard promote from Genie TEST workspace to another
Hi,We have manually migrated powerBI dashboard components to Databricks Genie TEST workspace.what is the procedure or approaches to promote these Genie TEST workspace components to Databricks Genie PROD workspace?ThanksSubra
- 2138 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Subra2025, There are two separate things to consider here: promoting AI/BI Dashboards and promoting Genie Spaces. They have different promotion paths, so I will cover both. PROMOTING AI/BI DASHBOARDS ACROSS WORKSPACES AI/BI Dashboards have mature...
- 1 kudos
- 671 Views
- 2 replies
- 0 kudos
Data Load from S3 Frankfurt Region to Unity catalog in AWS USWest Region
Hello,I am trying to bring the Parquet data from S3 to the Unity CatalogCurrently I am doing straight forward read and write like this test_data = spark.read.schema(1_billion_data).parquet(s3_path) test_data.repartition(num_cores*2).write.mode("overw...
- 671 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @theunwoke, The 1h45m you are seeing is heavily influenced by cross-region network transfer between eu-central-1 (Frankfurt) and us-west (your workspace region). S3 reads that cross AWS regions go over the public internet backbone, so throughput p...
- 0 kudos
- 1871 Views
- 4 replies
- 0 kudos
how to use rules dynamically in LDP
HiI see there is a way to store rules in table & use them in python while implementing LDPs how to use the generate/ read rules dynamically in SQL way of implementing LDPs. Could you please help me with this#DLT
- 1871 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @IM_01, The feature you are looking for, storing data quality rules in a table and applying them dynamically, is fully supported in Lakeflow Spark Declarative Pipelines (SDP) through the Python API. Unfortunately, there is currently no equivalent ...
- 0 kudos
- 2025 Views
- 3 replies
- 3 kudos
Configure SAS Token for ADLS Access in Databricks Job (Works on Classic Cluster, Fails on Serverless
I am running a Databricks job that reads from a Delta table and writes to an ADLS Gen2 location using a SAS token for authentication.from pyspark.sql import SparkSession spark = SparkSession.builder.getOrCreate() sas_token = dbutils.secrets.get(scop...
- 2025 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @smpa01, The reason this works on a classic cluster but fails on serverless is that serverless compute only supports a very limited set of Spark configuration properties. The fs.azure.sas.* Hadoop configurations you are setting via spark.conf.set ...
- 3 kudos
- 2846 Views
- 8 replies
- 10 kudos
Resolved! How to provide env variables to a pipeline task
Hello,I create a job with one pipeline task through DAB. Now I want to provide a variable to it, but it is dynamic based on the target environment. As pipeline tasks do not support widgets, how can I provide this variable to the pipeline?
- 2846 Views
- 8 replies
- 10 kudos
- 10 kudos
Hi @yit337, Since pipeline tasks in Databricks Jobs do not support widgets the way notebook tasks do, the recommended approach is to use pipeline configuration parameters. These are key-value pairs you set in the pipeline definition, and you can make...
- 10 kudos
- 739 Views
- 2 replies
- 0 kudos
code unit 1 line 1 statement, stored as Query - considered "empty" in Job TaskRun - fails [ADBricks]
Hi colleagues,could someone explain, why a single line statement [CALL ...();] fails (considered "empty" in error message) when invoked by a Job's Task[SQL\Query] run?- as shown on the attached screen:The same statement, stored in (Notebook, Workspac...
- 739 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @ADBricksExplore, This behavior where a CALL statement works from a Notebook or SQL File task but fails with an "empty" error from a SQL Query task comes down to how these task types resolve the SQL content at runtime. UNDERSTANDING THE DIFFERENCE...
- 0 kudos
- 1294 Views
- 3 replies
- 1 kudos
Resolved! Soumitra dutta : What are the essential concepts a newcomer should master first to become productive
Hii Friends,My name is Soumitra Dutta , I'm oxford based an entrepreneur, Author & Photographer at the intersection of visual storytelling and strategic leadership. I want to know to know what foundational concepts or skills do you believe are most ...
- 1294 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @soumitradutta, Welcome to the Databricks Community. Here is a structured learning path that I would recommend for getting productive quickly, organized from foundational to more advanced topics. PHASE 1: PLATFORM FUNDAMENTALS Start here to unders...
- 1 kudos
- 3631 Views
- 5 replies
- 1 kudos
Ingesting data from views
Hi all , I have been looking to create gold tables from views , and also considering to have features of streaming and change data capture .I know in DLT Workflows this is not possible , so I was wondering is there any other way to do the same please...
- 3631 Views
- 5 replies
- 1 kudos
- 1 kudos
Hi @naman0012, There are a few different approaches depending on where these "views" live and what your exact architecture looks like. Let me walk through the options. CLARIFYING THE SCENARIO The key question is: are these database views on an extern...
- 1 kudos
- 1288 Views
- 3 replies
- 0 kudos
Tags Field Doesn't propagate in Delta Share
Hi,My current work requires it to add tags in Databricks tables & views. I see that there is a tag column associated to tables and views to which we can set the TAG using SET TAG command.My requirement is that once we are creating delta shares out of...
- 1288 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @souravroy1990, You are correct that Unity Catalog tags (set via ALTER TABLE SET TAGS or ALTER TABLE ALTER COLUMN SET TAGS) are not propagated to recipients through Delta Sharing. Tags are treated as Unity Catalog governance metadata that lives in...
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
