cancel
Showing results for 
Search instead for 
Did you mean: 
Databricks Platform Discussions
Dive into comprehensive discussions covering various aspects of the Databricks platform. Join the conversation to deepen your understanding and maximize your usage of the Databricks platform.
cancel
Showing results for 
Search instead for 
Did you mean: 

Browse the Community

Data Engineering

Join discussions on data engineering best practices, architectures, and optimization strategies with...

12107 Posts

Data Governance

Join discussions on data governance practices, compliance, and security within the Databricks Commun...

522 Posts

Generative AI

Explore discussions on generative artificial intelligence techniques and applications within the Dat...

364 Posts

Machine Learning

Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithm...

1016 Posts

Warehousing & Analytics

Engage in discussions on data warehousing, analytics, and BI solutions within the Databricks Communi...

681 Posts

Activity in Databricks Platform Discussions

abhishek2k11
by > New Contributor II
  • 196 Views
  • 6 replies
  • 0 kudos

No workspace in Free Edition

Hi, I have been using free edition from some time using my this mail id. But from last 3-4 days I can’t see any workspace. when ever I am logging in I am getting two accounts name and In no workspace is available. When I tried creating another accoun...

  • 196 Views
  • 6 replies
  • 0 kudos
Latest Reply
abhishek2k11
New Contributor II
  • 0 kudos

when I logging I am getting above page. where no workspace space and no way to create a new one

  • 0 kudos
5 More Replies
emanueol
by > New Contributor III
  • 271 Views
  • 5 replies
  • 0 kudos

Foreign catalog to Snowflake

While learning about Databricks foreign catalogsn (I'm on free tier DBX account), seems theres 2 ways creating foreign catalog to Snowflake:via CONNECTION type=snowflake, seems jdbc connection through where DBX pulls all metadata of 1 snowflake datab...

  • 271 Views
  • 5 replies
  • 0 kudos
Latest Reply
SteveOstrowski
Databricks Employee
  • 0 kudos

Hi @emanueol, Your follow-up question is clear, and it is a good distinction to make. Let me address it directly. SHORT ANSWER Yes, when you set up Snowflake Catalog Federation in Databricks, Databricks does use the Snowflake Horizon REST Iceberg cat...

  • 0 kudos
4 More Replies
Malthe
by > Valued Contributor
  • 59 Views
  • 1 replies
  • 0 kudos

Strange metric view window grouping interaction

With the new metric views, I am unable to understand the grouping logic in the following setup:I have a table with timestamps and I define dimensions as follows:dimension: - name: timestamp expr: timestamp - name: date expr: DATE(timestam...

  • 59 Views
  • 1 replies
  • 0 kudos
Latest Reply
SteveOstrowski
Databricks Employee
  • 0 kudos

Hi @Malthe, This is a nuanced aspect of how metric views resolve window measure dimensions. The key behavior you are seeing comes down to how the metric view engine matches your query's GROUP BY columns to the dimensions defined in the window clause....

  • 0 kudos
lw2
by > New Contributor
  • 92 Views
  • 1 replies
  • 0 kudos

Read sqlite file from s3 bucket into databricks, creating delta tables

I have a sqlite database that I want to read into databricks to create delta tables/dataframes in Python that I can export to power BI and have a live connection. When there is new data added to my sqlite data base, the changes will need to reflect i...

  • 92 Views
  • 1 replies
  • 0 kudos
Latest Reply
SteveOstrowski
Databricks Employee
  • 0 kudos

Hi @lw2, The approach you have been using (copy SQLite to local, read with sqlite3, export to CSV, then manually create tables) works as a one-shot load but, as you noticed, it does not give you an easy path to keep things in sync. Below is a streaml...

  • 0 kudos
RobTScot
by > New Contributor
  • 110 Views
  • 3 replies
  • 1 kudos

GUID or concatenated string as a primary key in silver and gold data models

HiWe are in the process of designing and building new silver and gold layers (Star Schema). We will be using Databricks, which is new to the organisation.The silver layer will be modelled using classic 3NF, with SCD 7.The gold layer is a star schema....

  • 110 Views
  • 3 replies
  • 1 kudos
Latest Reply
SteveOstrowski
Databricks Employee
  • 1 kudos

Hi @RobTScot, This is a common design decision in lakehouse data modeling, and the right answer depends on the layer, the tooling, and the downstream consumers. Here is a breakdown of the key considerations. SILVER LAYER (3NF WITH SCD 7) For silver t...

  • 1 kudos
2 More Replies
Charansai
by > New Contributor III
  • 162 Views
  • 3 replies
  • 0 kudos

Notebooks Not Deploying in Development Mode Using Databricks Asset Bundles (Deploying from Workspace

Hi everyone,I’m using Databricks Asset Bundles and running into an issue when deploying to my dev environment in development mode. Even though my bundle includes sync paths and notebook directories, the deployment only creates the .databricks/artifac...

  • 162 Views
  • 3 replies
  • 0 kudos
Latest Reply
SteveOstrowski
Databricks Employee
  • 0 kudos

Hi @Charansai, The behavior you are seeing is actually expected when deploying a bundle from the Databricks Workspace Editor. Here is why. SOURCE-LINKED DEPLOYMENT When you deploy a bundle from within the workspace (as opposed to using the Databricks...

  • 0 kudos
2 More Replies
dtb_usr
by > New Contributor III
  • 248 Views
  • 10 replies
  • 0 kudos

SELECT Permission error when reading materialised views associated with a pipeline

I am having to pass ownership of pipelines to users for them to read materialised views associated with any pipeline otherwise they get a 'User does not have SELECT on table...' error. This is obviously bonkers as any pipeline can only have one owner...

  • 248 Views
  • 10 replies
  • 0 kudos
Latest Reply
SteveOstrowski
Databricks Employee
  • 0 kudos

Hi @dtb_usr, Based on the error message and the fact that the query works in the SQL Editor (which uses a SQL warehouse) but fails on a personal/dedicated cluster in notebooks, this is almost certainly a compute access mode issue rather than a Unity ...

  • 0 kudos
9 More Replies
ZafarJ
by > New Contributor II
  • 71 Views
  • 2 replies
  • 2 kudos

User Management tab not showing

Hi,I created the workspace with my contributor role from the Azure portal. However, while logged in, I cannot find the User Management tab. I am trying to set up Unity Catalog for user administration.How can I access this?Thanks

  • 71 Views
  • 2 replies
  • 2 kudos
Latest Reply
SteveOstrowski
Databricks Employee
  • 2 kudos

Hi @ZafarJ, This is a common point of confusion when getting started with Azure Databricks, and the answer depends on which level of user management you need. WORKSPACE-LEVEL USER MANAGEMENT As a workspace admin, you can manage users directly in your...

  • 2 kudos
1 More Replies
wrosa
by > New Contributor
  • 107 Views
  • 1 replies
  • 0 kudos

Metric Views + Window Functions not supported

Hi @Hubert-Dudek I am getting following error: METRIC_VIEW_WINDOW_FUNCTION_NOT _SUPPORTED The metric view is not allowed to use window function (...) With the following definition:- name: Sales net Total- expr: SUM(MEASURE(`Sales net`)) OVER())Howeve...

  • 107 Views
  • 1 replies
  • 0 kudos
Latest Reply
SteveOstrowski
Databricks Employee
  • 0 kudos

Hi @wrosa, The error you are seeing (METRIC_VIEW_WINDOW_FUNCTION_NOT_SUPPORTED) is expected behavior. Metric views do not allow raw SQL window functions like SUM(...) OVER() directly in a measure's expr definition. This is by design because metric vi...

  • 0 kudos
Seunghyun
by > New Contributor III
  • 187 Views
  • 2 replies
  • 0 kudos

Deploy dashboard with asset bundle

Hello, I have some questions regarding dashboard development using Asset Bundles.I have been following the procedure for developing dashboards by referring to this page: Databricks CI/CD for Dashboard Developers.Here is the workflow I followed:Create...

  • 187 Views
  • 2 replies
  • 0 kudos
Latest Reply
SteveOstrowski
Databricks Employee
  • 0 kudos

Hi @Seunghyun, This is a common workflow question when getting started with AI/BI Dashboard deployment through Databricks Asset Bundles. Here is a walkthrough of the recommended approach to maintain a single dashboard and handle ongoing modifications...

  • 0 kudos
1 More Replies
lw2
by > New Contributor
  • 162 Views
  • 3 replies
  • 0 kudos

Read Sqlite file in to create delta table/dataframe with live connection

I have a sqlite database that I want to read into databricks to create delta tables/dataframes in Python that I can export to power BI and have a live connection. When there is new data added to my sqlite data base, the changes will need to reflect i...

  • 162 Views
  • 3 replies
  • 0 kudos
Latest Reply
SteveOstrowski
Databricks Employee
  • 0 kudos

Hi @lw2, The key to getting a "live connection" end-to-end is replacing the manual CSV export with a scheduled pipeline that writes directly to Delta tables, then connecting Power BI to those Delta tables via DirectQuery. Here is a complete approach....

  • 0 kudos
2 More Replies
ChristianRRL
by > Honored Contributor
  • 240 Views
  • 4 replies
  • 0 kudos

Asset Bundles Overriding Existing Jobs (despite different name_prefix)

Hi there, I'm seeing what seems to be unexpected behavior on databricks asset bundle deployment and I'm hoping I can get clarification on this.Basically, what I'm trying to do is to deploy the same asset bundle twice (two different variations), with ...

ChristianRRL_1-1769815680547.png ChristianRRL_0-1769815600150.png
  • 240 Views
  • 4 replies
  • 0 kudos
Latest Reply
SteveOstrowski
Databricks Employee
  • 0 kudos

Hi @ChristianRRL, This behavior comes down to how Databricks Asset Bundles track deployed resources using Terraform state, and specifically where that state is stored locally. HOW BUNDLE STATE TRACKING WORKS When you run "databricks bundle deploy", t...

  • 0 kudos
3 More Replies
RutujaKadam
by > New Contributor II
  • 155 Views
  • 2 replies
  • 1 kudos

Getting Error when connecting azure databricks to azure sql server using lakeflow connect

Hi, Can anyone please let me know how to resolve this error . I am trying to connect azure sql server to azure databricks using lakeflow connect data ingestion. I am able to create the connection but afterwards it gives me error as :Error starting ga...

  • 155 Views
  • 2 replies
  • 1 kudos
Latest Reply
SteveOstrowski
Databricks Employee
  • 1 kudos

Hi @RutujaKadam, The error you are seeing, "Error starting gateway compute resources" with a message about VM quota exhaustion, is related to your Azure subscription's vCPU quota rather than a misconfiguration in Databricks itself. Here is what is ha...

  • 1 kudos
1 More Replies
Raman_Unifeye
by > Honored Contributor III
  • 202 Views
  • 6 replies
  • 0 kudos

How to restrict Databricks Apps and Vector Search endpoint creation for workspace users

I am looking to restrict all workspace users' access to create Databricks Apps and Vector Search endpoints.I am aware there is no simple toggle, what is the best way to implement it?

  • 202 Views
  • 6 replies
  • 0 kudos
Latest Reply
SteveOstrowski
Databricks Employee
  • 0 kudos

Hi @Raman_Unifeye, You are correct that there is no single toggle to block creation of these resources today. Here is a breakdown of the proactive and detective controls available for each. VECTOR SEARCH ENDPOINTS Vector Search endpoints use access c...

  • 0 kudos
5 More Replies
bts136
by > New Contributor
  • 179 Views
  • 2 replies
  • 1 kudos

Reading Excel files with Spark returns formula values instead of computed values

Hi,I'm seeing inconsistent behavior when reading Excel files using the built-in connector Lakeflow Connector with spark.read.format("excel") (doc: https://docs.databricks.com/aws/en/query/formats/excel). I read an .xlsx file from S3 using this functi...

  • 179 Views
  • 2 replies
  • 1 kudos
Latest Reply
SteveOstrowski
Databricks Employee
  • 1 kudos

Hi @bts136, This behavior is related to how Excel files store formula results internally, and it is something you can work around. BACKGROUND: HOW EXCEL STORES FORMULAS Excel files (.xlsx) store both the formula text and a cached computed result for ...

  • 1 kudos
1 More Replies