Here's your Data + AI Summit 2024 - Data Governance recap as you navigate the explosion of AI, data and tools in efforts to build a flexible and scalable governance framework that spans your entire data and AI estate. Keynote: Evolving Data Governan...
As a way to minimize storage costs, my team and I want to create views instead of tables in the Gold layer.We always try to improve the experience of our users by adding comments to the columns. the problem is that views do not inherit comments from ...
you can use COMMENT ON to set or update the comment on an view-- View-level comment (use TABLE for views)COMMENT ON TABLE catalog.schema.my_view IS 'View for user info';-- Column-level comment on a viewCOMMENT ON COLUMN catalog.schema.my_view.name IS...
Hello! Newbie here, so apologies if this is a super basic question. Trying to figure out if we can connect Purview to an AWS instance of Databricks (vice Azure instance), but I have only seen articles on connecting Azure Databricks to Purview. I r...
@ShaSap wrote:Hello! Newbie here, so apologies if this is a super basic question. Trying to figure out if we can connect Purview to an AWS instance of Databricks (vice Azure instance), but I have only seen articles on connecting Azure Databricks to...
Hi folks,How encryption is handled for UC managed tables on AWS- based Databricks account? I found this post: https://docs.databricks.com/aws/en/security/keys/ , which describes how workspace file system (dbfs) and control panel data is encrypted, bu...
I stored my artifact of model in volume.I can view the object via catalog page.however, when i open from experiment > artifact, i can't access to view my artifact with error failed to fetch.Any idea to solve the problem?
Hello @tana_sakakimiya! Artifacts stored in Volumes can’t be viewed directly in the MLflow experiment UI, as it only supports displaying artifacts saved to DBFS. You can instead access or download artifacts through Catalog Explorer or using mlflow.ar...
Where does the Unity Catalog metastore- metadata actually reside?Is it stored and managed in the Databricks account (control plane)?Or does it get stored in the customer-managed S3 bucket when we create a bucket for Unity Catalog metastore?I want to ...
1. Unity Catalog metadata (schemas, tables, permissions, lineage, etc.) is stored and managed in the Databricks control plane, not in your S3 bucket. This metadata resides in a Databricks-managed database and is not in your customer-managed storage.2...
Hi everyone,I'm running into an issue when requesting permissions on a Unity Catalog table. Specifically, when trying to request SELECT and MODIFY privileges on a specific table, using Request for Access in Unity Catalog. I get the following error:th...
Hi @boitumelodikoko ,Weird, it looks like a bug. MODIFY is a valid permission for table object. You can check it yourself in docs:Unity Catalog privileges and securable objects | Databricks on AWSModify allows user to add, update, and delete data to ...
I'm currently working with Databricks within Azure and encountering an issue with Unity Catalog in a newly created workspace.My colleague initially deployed a Databricks workspace where Unity Catalog was provisioned and enabled by default. I recently...
Hello like szymon suggested just try to check with the admin also as you share vnet injected workspace was it done recently if yes! ThenMight be you were missing two private endpoint of storage account you needed to create 2 private endpoints, both ...
Hi Everyone ,while I am creating sync table in UC, I got an error and failed . then I am not able to recreate the same table.when I checked the catalogue table name is there but not able to open , and I am querying the same in the notebook .it's givi...
@ajai_duraisamy Here are few checks you can do:1. SHOW TABLES IN <catalog>.<schema> LIKE '<table>'; This command will show if the table exists in UC.2. Make sure your compute is UC‑enabled.If you’re on a cluster/SQL Warehouse without UC enabled (or c...
I'm trying to use the Spark-Nvidia integrations in order to train Spark ML models using GPU.While trying to follow these instructions: https://docs.nvidia.com/spark-rapids/user-guide/23.12.1/getting-started/databricks.htmlI could not execute the init...
Since cluster initialization happens before the "libraries" section installs Maven artifacts, the plugin isn’t available at the required time, causing the error. Workaround Strategies 1. Internal Artifactory or DBFS Manual Upload Upload the RAPIDS j...
Is there anyone can provide reference that AWS Databricks comply with FISC (The Center for Financial Industry Information Systems) standard which is a standard for financial industry in Japan?I can't find any official information yet.or if you find i...
Hi @tana_sakakimiya ,As you said, there's no official documentation on databricks regarding this.But when we consider below FAQ from AWS FISC docs page we can see that from audit perspective they will focus mainly on SOC 1, SOC 2 and SOC 3And those a...
Hello,I am trying to deploy a gradio app (app.py) in databricks, but I am having problem accessing data stored in a volume in unity catalog. It seems like that I cannot access the data using path like "/Volumes/catalog/schema.../my_data", which works...
Unlike notebooks, Databricks Apps does not support mounting Unity Catalog volumes and directly reading and writing files. As this code snippet demonstrates, each file needs to be downloaded to the app compute before being able to manipulate it.
Hi everyone,I'm a Cloud Engineer working on a multi-environment Databricks setup (Dev, QA, Prod), and I've received a request from our Data Engineering team that's a bit unconventional — they are asking for access to Production Unity Catalogs from lo...
Hi @Charansai ,That's a great question! In general, granting Dev/QA users direct access to Production catalogs is not considered best practice. The main risks you already mentioned (governance, compliance, and accidental writes) usually outweigh the ...
I gone through resources supported by DABs but cannot find notebooks deployment via DABs. Is it any possible way we can deploy notebooks using DABs? What the alternative approach?
Hi @Akash30307 ,Thank you for your question!You can also deploy notebooks via our Terraform provider.You can find more information on our website.You can also work with databricks_notebook and databricks_notebook_paths data sources.
We are looking to send messages using databricks to an azure service bus topic and would like to connect to the service bus using a managed identity vs a connection string. Is this possible in databricks. The only thing I could find regarding datab...
Is there any update to use managed identity instead of dbmanaged identity?
Problem: Queries on our 100M+ record Iceberg tables were taking 45+ seconds.Solution: Implemented lightweight zonemap indexing that tracks min/max values per file.Quick Implementationdef apply_zonemap_pruning(table_path, predicate_value):# Load zonem...
Hi! @ck7007 @WiliamRosa ,I have a question — why are you actually doing this? I’m not fully familiar with your exact setup (Iceberg), but my understanding is that Iceberg already stores these stats (min/max) in the manifests, and Spark should be able...
