Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best prac...
Explore discussions on Databricks administration, deployment strategies, and architectural best prac...
Join discussions on data engineering best practices, architectures, and optimization strategies with...
Join discussions on data governance practices, compliance, and security within the Databricks Commun...
Explore discussions on generative artificial intelligence techniques and applications within the Dat...
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithm...
Engage in discussions on data warehousing, analytics, and BI solutions within the Databricks Communi...
Hi, I have been using free edition from some time using my this mail id. But from last 3-4 days I can’t see any workspace. when ever I am logging in I am getting two accounts name and In no workspace is available. When I tried creating another accoun...
when I logging I am getting above page. where no workspace space and no way to create a new one
While learning about Databricks foreign catalogsn (I'm on free tier DBX account), seems theres 2 ways creating foreign catalog to Snowflake:via CONNECTION type=snowflake, seems jdbc connection through where DBX pulls all metadata of 1 snowflake datab...
Hi @emanueol, Your follow-up question is clear, and it is a good distinction to make. Let me address it directly. SHORT ANSWER Yes, when you set up Snowflake Catalog Federation in Databricks, Databricks does use the Snowflake Horizon REST Iceberg cat...
With the new metric views, I am unable to understand the grouping logic in the following setup:I have a table with timestamps and I define dimensions as follows:dimension: - name: timestamp expr: timestamp - name: date expr: DATE(timestam...
Hi @Malthe, This is a nuanced aspect of how metric views resolve window measure dimensions. The key behavior you are seeing comes down to how the metric view engine matches your query's GROUP BY columns to the dimensions defined in the window clause....
I have a sqlite database that I want to read into databricks to create delta tables/dataframes in Python that I can export to power BI and have a live connection. When there is new data added to my sqlite data base, the changes will need to reflect i...
Hi @lw2, The approach you have been using (copy SQLite to local, read with sqlite3, export to CSV, then manually create tables) works as a one-shot load but, as you noticed, it does not give you an easy path to keep things in sync. Below is a streaml...
HiWe are in the process of designing and building new silver and gold layers (Star Schema). We will be using Databricks, which is new to the organisation.The silver layer will be modelled using classic 3NF, with SCD 7.The gold layer is a star schema....
Hi @RobTScot, This is a common design decision in lakehouse data modeling, and the right answer depends on the layer, the tooling, and the downstream consumers. Here is a breakdown of the key considerations. SILVER LAYER (3NF WITH SCD 7) For silver t...
Hi everyone,I’m using Databricks Asset Bundles and running into an issue when deploying to my dev environment in development mode. Even though my bundle includes sync paths and notebook directories, the deployment only creates the .databricks/artifac...
Hi @Charansai, The behavior you are seeing is actually expected when deploying a bundle from the Databricks Workspace Editor. Here is why. SOURCE-LINKED DEPLOYMENT When you deploy a bundle from within the workspace (as opposed to using the Databricks...
I am having to pass ownership of pipelines to users for them to read materialised views associated with any pipeline otherwise they get a 'User does not have SELECT on table...' error. This is obviously bonkers as any pipeline can only have one owner...
Hi @dtb_usr, Based on the error message and the fact that the query works in the SQL Editor (which uses a SQL warehouse) but fails on a personal/dedicated cluster in notebooks, this is almost certainly a compute access mode issue rather than a Unity ...
Hi,I created the workspace with my contributor role from the Azure portal. However, while logged in, I cannot find the User Management tab. I am trying to set up Unity Catalog for user administration.How can I access this?Thanks
Hi @ZafarJ, This is a common point of confusion when getting started with Azure Databricks, and the answer depends on which level of user management you need. WORKSPACE-LEVEL USER MANAGEMENT As a workspace admin, you can manage users directly in your...
Hi @Hubert-Dudek I am getting following error: METRIC_VIEW_WINDOW_FUNCTION_NOT _SUPPORTED The metric view is not allowed to use window function (...) With the following definition:- name: Sales net Total- expr: SUM(MEASURE(`Sales net`)) OVER())Howeve...
Hi @wrosa, The error you are seeing (METRIC_VIEW_WINDOW_FUNCTION_NOT_SUPPORTED) is expected behavior. Metric views do not allow raw SQL window functions like SUM(...) OVER() directly in a measure's expr definition. This is by design because metric vi...
Hello, I have some questions regarding dashboard development using Asset Bundles.I have been following the procedure for developing dashboards by referring to this page: Databricks CI/CD for Dashboard Developers.Here is the workflow I followed:Create...
Hi @Seunghyun, This is a common workflow question when getting started with AI/BI Dashboard deployment through Databricks Asset Bundles. Here is a walkthrough of the recommended approach to maintain a single dashboard and handle ongoing modifications...
I have a sqlite database that I want to read into databricks to create delta tables/dataframes in Python that I can export to power BI and have a live connection. When there is new data added to my sqlite data base, the changes will need to reflect i...
Hi @lw2, The key to getting a "live connection" end-to-end is replacing the manual CSV export with a scheduled pipeline that writes directly to Delta tables, then connecting Power BI to those Delta tables via DirectQuery. Here is a complete approach....
Hi there, I'm seeing what seems to be unexpected behavior on databricks asset bundle deployment and I'm hoping I can get clarification on this.Basically, what I'm trying to do is to deploy the same asset bundle twice (two different variations), with ...
Hi @ChristianRRL, This behavior comes down to how Databricks Asset Bundles track deployed resources using Terraform state, and specifically where that state is stored locally. HOW BUNDLE STATE TRACKING WORKS When you run "databricks bundle deploy", t...
Hi, Can anyone please let me know how to resolve this error . I am trying to connect azure sql server to azure databricks using lakeflow connect data ingestion. I am able to create the connection but afterwards it gives me error as :Error starting ga...
Hi @RutujaKadam, The error you are seeing, "Error starting gateway compute resources" with a message about VM quota exhaustion, is related to your Azure subscription's vCPU quota rather than a misconfiguration in Databricks itself. Here is what is ha...
I am looking to restrict all workspace users' access to create Databricks Apps and Vector Search endpoints.I am aware there is no simple toggle, what is the best way to implement it?
Hi @Raman_Unifeye, You are correct that there is no single toggle to block creation of these resources today. Here is a breakdown of the proactive and detective controls available for each. VECTOR SEARCH ENDPOINTS Vector Search endpoints use access c...
Hi,I'm seeing inconsistent behavior when reading Excel files using the built-in connector Lakeflow Connector with spark.read.format("excel") (doc: https://docs.databricks.com/aws/en/query/formats/excel). I read an .xlsx file from S3 using this functi...
Hi @bts136, This behavior is related to how Excel files store formula results internally, and it is something you can work around. BACKGROUND: HOW EXCEL STORES FORMULAS Excel files (.xlsx) store both the formula text and a cached computed result for ...
| User | Count |
|---|---|
| 1837 | |
| 1062 | |
| 882 | |
| 768 | |
| 470 |