- 128 Views
- 1 replies
- 0 kudos
Foreign Catalog Table - Missing Most Rows
I have a table pulled from Snowflake via a foreign catalog that's only returning 100 rows in Databricks even though there's billions of records in the source Snowflake table. What steps should I take to diagnose the limitation of rows?
- 128 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @jruhl, When you say it's only returning 100 rows, how are you accessing the data? Are you previewing the data or querying it? You may want to first confirm whether you are hitting an actual federation limit, or whether you are just seeing a small...
- 0 kudos
- 30935 Views
- 16 replies
- 8 kudos
In Azure Databricks Delta Lake not able to see unity catalog databases or tables in the drop down.
I have created an Azure data factory pipeline with a copy data function to copy data from adls path to a delta table .In the delta table drop downs i am able to see only the hive metastore database and tables only but the unity catalog tables are not...
- 30935 Views
- 16 replies
- 8 kudos
- 8 kudos
Hi, Anybody have an update on this? I also use the managed identity and i receive the same error as the last two people mentioned here.
- 8 kudos
- 801 Views
- 4 replies
- 0 kudos
system.ai.python_exec suddenly failing with UNRESOLVED_ROUTINE on Azure (West US) since May 27 ~4pm
Hi all, We're hitting a sudden failure with Databrick's built-in python sandbox system.ai.python_exec function and want to check whether others are seeing the same thing, since it looks region-specific.Environment:Cloud: Azure DatabricksRegion: West ...
- 801 Views
- 4 replies
- 0 kudos
- 0 kudos
Experiencing this issue as well, started some time prior to June 1st. ```USE CATALOG system;USE SCHEMA ai;SELECT python_exec('print("hello")');``` Returns:```[UNRESOLVED_ROUTINE] Cannot resolve routine `python_exec` on search path [`system`.`session`...
- 0 kudos
- 743 Views
- 4 replies
- 0 kudos
Row and column level security on unity catalog in different approaches
I have one task for you which is related to unity catalog I want to implement row and column level security on databricks for few tables please let me know different kind of approaches and methods.We are using azure cloud and databricks witj unity c...
- 743 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi Rupa,One thing worth adding from running both dynamic views and column masking in production - the table above is a good way to think about the mechanics, but in practice the decision usually comes down to how many things are querying the same dat...
- 0 kudos
- 7270 Views
- 3 replies
- 2 kudos
Deprecation of DBFS?
I am using Azure Databricks DBFS. I have stored some python scripts, notebooks in folders in DBFS root and have mounted ADLS Gen2 to DBFS. Currently I do not have Unity Catalog enabled and this was setup 3-4 years ago. I came across this article http...
- 7270 Views
- 3 replies
- 2 kudos
- 2 kudos
In current state, when DBFS is disabled in a workspace, both the root filestore and mount paths become inaccessible.What is deprecated are:DBFS root storage the workspace‑provisioned filestore - /dbfs/FileStore/, /dbfs/root/ etc.DBFS mounts (/dbfs/mn...
- 2 kudos
- 545 Views
- 2 replies
- 0 kudos
Can change a delta table, but not its schema
Is it possible to let a group in databricks to change a delta table, but not the table’s schema in UC? i.e., they can add or remove rows of the delta table, but cannot change the name or the format of the columns.
- 545 Views
- 2 replies
- 0 kudos
- 0 kudos
Granting only MODIFY on the table allows the group to insert, update, and delete rows. MODIFY grants the ability to change the contents of a table without altering its schema or metadata - it does not include schema-altering privileges like ALTER or ...
- 0 kudos
- 230 Views
- 1 replies
- 0 kudos
First time community member working on a project
Hi everyone,Don't know where to put this specific question. I'm working on a reference data mesh implementation to connect and combine datasets to find matching trials for patients with a specific genetic profile. - Do you know anyone that might be i...
- 230 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @steff_horemans, My guess is that the Free Edition is probably not the best place to demonstrate true external sharing. Free Edition is positioned as a serverless-only, quota-limited, non-commercial environment, with one workspace, one metastore, ...
- 0 kudos
- 690 Views
- 1 replies
- 0 kudos
Resolved! Account Prices for Azure
For AWS and GCP, the discounts provided by Databricks can be found in system.billing.account_prices table. How can we find the same in Azure?
- 690 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @nara_shikamaru Azure Databricks is a native service similar to Synapse in Azure unlike others. Databricks generally never sees the final aligned price and hence its unavailable in system tables.You can follow belowAzure Daily Export - You can exp...
- 0 kudos
- 2593 Views
- 2 replies
- 1 kudos
ALL CATALOGS in SHOW SCHEMAS
Hi there,I was checking though the Databricks Runtime 17.0 features while testing:Support ALL CATALOGS in SHOW SCHEMASI faced below issues, is that anticipated or an issue which needs to be fixed?This is how it was before Runtime 17.0 I was expectin...
- 2593 Views
- 2 replies
- 1 kudos
- 1 kudos
HI @creativeIndex, I stumbled upon this while looking for something else. I appreciate it's been a while, but I realise you haven't received a solution yet. Is this still an issue? I just wanted to point out that you should use an asterisk (*) instea...
- 1 kudos
- 28200 Views
- 13 replies
- 10 kudos
Backup Unity Catalog and managed tables
Hi All, Can anyone point me to either documentation or personally tried and tested method of backing up (and restoring) Unity Catalog and its associated managed tables? We're running on Azure and using ADLS Gen2.Regards,Ashley
- 28200 Views
- 13 replies
- 10 kudos
- 10 kudos
This can be via 3rd party tool called commvault : https://documentation.commvault.com/11.42/software/protect_databricks_using_commvault.html
- 10 kudos
- 914 Views
- 4 replies
- 2 kudos
Databricks Genie: Power Users vs. Broad Org Access
Hi everyone,I’m curious if anyone has successfully implemented Databricks Genie (chat/agent) for production use.Currently, we’ve enabled a few Genie instances for power users who are comfortable working with data outside of the data team. However, we...
- 914 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi Julie, thanks for sharing the screenshot. There are a few things we'd recommend to customers when they are starting out on a Genie journey, we defintiely see enterprises creating Genie spaces at scale but it needs some thought and guardrails. Some...
- 2 kudos
- 1854 Views
- 3 replies
- 0 kudos
Resolved! Unity catalog management
How is Unity Catalog managed in real time at an enterprise scale, including workspace-level restrictions, privilege-based ACLs, row- and column-level security, ABAC, and tag-driven governance, and which languages or tools are used to manage the entir...
- 1854 Views
- 3 replies
- 0 kudos
- 0 kudos
Thank you... Which roles ( workspace admin or Metastore Admin) are recommended to securely create and manage Unity Catalog objects (Storage Credentials, External Locations, Catalogs, Schemas, and Delta Sharing)?” and why ?
- 0 kudos
- 11584 Views
- 7 replies
- 10 kudos
Data Access Control in Databricks
Best Practices for Securing Access to Data in DatabricksUnity Catalog is the unified governance solution for Data & AI assets in Databricks and greatly simplifies and centralized data access control. This guide includes best practices for both the st...
- 11584 Views
- 7 replies
- 10 kudos
- 10 kudos
Which roles ( workspace admin or Metastore Admin) are recommended to securely create and manage Unity Catalog objects (Storage Credentials, External Locations, Catalogs, Schemas, and Delta Sharing)?” and why ?
- 10 kudos
- 2299 Views
- 4 replies
- 3 kudos
Resolved! GDPR/CCPA Compliance Delete for PII data
Hi,I am currently designing a PII governance framework to meet CCPA compliance requirements on Databricks. I understand that Databricks provides mechanisms such as VACUUM and Deletion Vectors combined with REORG … APPLY (PURGE) to permanently remove ...
- 2299 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi @abhijit007 , A new Data Classification feature (currently in Public Preview), allows to automatically classify and tag sensitive data in your catalog. It goes through few steps: AI-driven engine scans Unity Catalog tables and detects PII data and...
- 3 kudos
- 2162 Views
- 2 replies
- 3 kudos
Resolved! Difference between RBAC and Unity Catalog
I am new to Databricks and trying to understand the concepts first, before getting into technical deepdive. I am a bit confused between Role Based Access Control and Unity Catalog. Both feature controls access to the data. Can someone provide me the ...
- 2162 Views
- 2 replies
- 3 kudos