Here's your Data + AI Summit 2024 - Data Governance recap as you navigate the explosion of AI, data and tools in efforts to build a flexible and scalable governance framework that spans your entire data and AI estate. Keynote: Evolving Data Governan...
We are trying to build the history snapshot for data that we are ingested in the delta tables. We are exploring different option like delta history, change data feed and dbt snapshot.This Question is particularly about the change data feed. So I want...
Hello @manjeettanwar10 , The Delta History API allows you to view the logs of all SQL statements executed on your table, including UPDATE, DELETE, and INSERT commands. When used in conjunction with Delta's Time Travel features, it's possible to retri...
Hi!Recently we encountered a problem with how Databricks handles secrets that does not meet our compliance.We need total isolation of users credentials but seems like the admin role in databricks totally breaks that since a person with that role can ...
Hi @ChechuIGZ - The value of the secret is redacted. https://docs.databricks.com/en/security/secrets/redaction.html#secret-redaction. Also remember the following -
Hi all,I need to mask columns in a table for certain users.What is the best way to go about this in Databricks? My workspace is unity catalog enabled.Thanks,Sean
In UC you can use column masking. See here - https://docs.databricks.com/en/data-governance/unity-catalog/row-and-column-filters.html#what-are-column-masks
I am ingesting job-cluster failure notifications that we send to OpsGenie into a delta table to automate the creation and tracking of Jira tickets. The alert notification includes the job run url, which we use to quickly respond to job failures. Ho...
I tracked this issue down to running the notebook on a shared Unity-enabled cluster. When scheduling the notebook to run on a job cluster with single user mode, this issue does not occur.
I have encountered with the Databricks CLI command databricks metastores list. As per our account setup, we have three metastores configured. However, when I run the command, it only returns information for one metastore instead of listing all three....
Hi @jv_v , Good Day! The command you are running gives you Metastore details attached to your workspace. If you are looking to list out all the metastore present at the Account console, you can try running this below CLI command: ------------------...
SpoilerWe have a UC enabled on Databricks where we have two Metastores,Metastore in West Europe regionMetastore in North America regionAnd Databricks workspace.Workspace in West Europe regionAnd we want to have a single Databricks workspace for both ...
Hi, Cross region metastore and workspace (WS) is not possible. It is important to understand the use case and why you have certain data in a region where you do not have a compute (WS) or do not need a compute. Ideally have each region has their own ...
Hello community !Here's my problem : I had a workspace with Unity Catalog enabled, and a catalog stored on an external location (Azure storage account). All my schemas and tables were storage on this storage. However we recently deleted the metastore...
Hi, Looks like you are creating the catalog at the container level. Are there any other UC objects created pointing to that? Is the external location created at the container level? Can you please have a look at this and see if there is overlapping? ...
Hi All,My client has a table with user names and the authorized territory name the user has access to e.g EMEA, APAC etc. The users can also move to a different territory over time and the same will be updated in the user table table. I have multiple...
Hi, Can you please have a look at this example please? https://docs.databricks.com/en/data-governance/unity-catalog/row-and-column-filters.html#mapping-table-examples If you create account groups for each region and then use IS_ACCOUNT_GROUP_MEMBER t...
Hi,I'm currently using Unity Catalog and serving as the metastore admin. However, I've encountered an issue where I cannot select the "Enable Delta sharing to allow a Databricks user to share data outside their organization" checkbox as it appears to...
@jpaladini - could you please let us know if you had a chance to follow the steps to setup Delta Sharing on your workspace - https://docs.databricks.com/en/data-sharing/set-up.html#enable-delta-sharing-for-your-account?
I want to load a Keras model using abfss. As far as I know, this is not possible, and the only way of doing this is by copying the model from ADLS to dbfs.I am concerned about the data governance, as everybody could potentionally modify/delete the mo...
Hi @Retired_mod,Can you please elaborate on these or share related documentation?Use Access Control Lists (ACLs) or Shared Access Signatures (SAS) to restrict access further.Is it possible to use ACLs to control DBFS access?Serialize your Keras model...
Databricks autocomplete is checking the hive_metastore catalog when I enter the schema_name and table_name in the notebook although we have other default catalog set at the workspace level. How to make Databricks autocomplete use that catalog when en...
Hi @Retired_mod ! Thanks for your answer. I forgot to mention that we already have this set up at the cluster level (using spark.databricks.sql.initial.catalog.name variable) besides setting this on the workspace level in the workspace settings but n...
Hi all,Since the last update of Databricks (21.03.2024), I can't find the option to git reset my repo anymore. Where did it go?Thanks in advance for your guidance!
I have tried different browsers and deleting cache despite that I am seeing this giant logo of databricks on the workspace home page and multiple people in our org are having this issue.
Hi,We are rolling out Unity in our organization and while playing around I had set some tags on a test catalog.After removing this catalog, the tag keys I used are still present in our Unity environment and I am unable to find where they are stored t...
Thanks! I will take that to heart and include it on our internal wiki
I'm trying to use bamboolib on the 11.0 recommended DBR. I am using the option "Databricks: Load database table" The databases and tables that are on the unity catalog do not populate the dropdown boxes. How can I fix this?
Hello @k.b.​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
