We are trying to build the history snapshot for data that we are ingested in the delta tables. We are exploring different option like delta history, change data feed and dbt snapshot.This Question is particularly about the change data feed. So I want...
In my organization, we are using Databricks unity catalog and we have a metastore created for our region which holds all of our workspaces. When we created the metastore last year, we set a metastore root location for it (If I remember correctly, met...
If you have a way to reproduce metastore object creation then better to delete this metastore a create a new one with no root metastore bucket. That will also ensure that catalog cannot be created without specifying a location. If you have locked dow...
Digital Marketing Agency NJ - Design StallionDesign Stallion is a web design and digital marketing agency in NJ delivering bespoke custom websites and online marketing solutions including SEO, PPC, and more.
In my experience with digital marketing, leveraging Empowering Growth through Tech Talent Sloboda has been incredibly effective for both product growth and team scaling. Their expertise in tech talent helped us expand our team efficiently and improve...
What is the vision of uc and purview , on government
Unity Catalog is an operational metastore. That means as the metadata/data is changing the changes are reflected in the UC in real time. Whereas, think of Purview as an Enterprise catalog where metadata is fed from UC or many other database systems u...
Hi!Recently we encountered a problem with how Databricks handles secrets that does not meet our compliance.We need total isolation of users credentials but seems like the admin role in databricks totally breaks that since a person with that role can ...
Hi @ChechuIGZ - The value of the secret is redacted. https://docs.databricks.com/en/security/secrets/redaction.html#secret-redaction. Also remember the following -
Message: Hi Databricks Community,I encountered an error while attempting to create a Metastore in Databricks. The error message is as follows:Error: cannot create metastore: This account with id ******************** has reached the limit for Metastor...
Hi, @jv_v; when was this workspace created? Can you please check this article and see that you already do not have a default UC metastore created?
Hi all,I need to mask columns in a table for certain users.What is the best way to go about this in Databricks? My workspace is unity catalog enabled.Thanks,Sean
In UC you can use column masking. See here - https://docs.databricks.com/en/data-governance/unity-catalog/row-and-column-filters.html#what-are-column-masks
I am ingesting job-cluster failure notifications that we send to OpsGenie into a delta table to automate the creation and tracking of Jira tickets. The alert notification includes the job run url, which we use to quickly respond to job failures. Ho...
I tracked this issue down to running the notebook on a shared Unity-enabled cluster. When scheduling the notebook to run on a job cluster with single user mode, this issue does not occur.
I have encountered with the Databricks CLI command databricks metastores list. As per our account setup, we have three metastores configured. However, when I run the command, it only returns information for one metastore instead of listing all three....
Hi @jv_v , Good Day! The command you are running gives you Metastore details attached to your workspace. If you are looking to list out all the metastore present at the Account console, you can try running this below CLI command: ------------------...
Hello,I would like to know if there is direct access to the Databricks query history tables. For compliance issues, I would like to be able to create reports for something like: who has accessed a particular column in a table in the past 6 months. T...
Thanks @josh_melton ! I was wondering right now about this (one day after your post!) since I only found the UI and API in the documentation and was really puzzled that there is no equivalent in unity to the Snowflake query_history table.
SpoilerWe have a UC enabled on Databricks where we have two Metastores,Metastore in West Europe regionMetastore in North America regionAnd Databricks workspace.Workspace in West Europe regionAnd we want to have a single Databricks workspace for both ...
Hi, Cross region metastore and workspace (WS) is not possible. It is important to understand the use case and why you have certain data in a region where you do not have a compute (WS) or do not need a compute. Ideally have each region has their own ...
Hello community !Here's my problem : I had a workspace with Unity Catalog enabled, and a catalog stored on an external location (Azure storage account). All my schemas and tables were storage on this storage. However we recently deleted the metastore...
Hi, Looks like you are creating the catalog at the container level. Are there any other UC objects created pointing to that? Is the external location created at the container level? Can you please have a look at this and see if there is overlapping? ...
I have been using "rdd.flatMap(lambda x:x)" for a while to create lists from columns however after I have changed the cluster to a Shared acess mode (to use unity catalog) I get the following error: py4j.security.Py4JSecurityException: Method public ...
Hi All,My client has a table with user names and the authorized territory name the user has access to e.g EMEA, APAC etc. The users can also move to a different territory over time and the same will be updated in the user table table. I have multiple...
Hi, Can you please have a look at this example please? https://docs.databricks.com/en/data-governance/unity-catalog/row-and-column-filters.html#mapping-table-examples If you create account groups for each region and then use IS_ACCOUNT_GROUP_MEMBER t...
Setup UCX from Databricks web terminal For cases when your desktop or laptop will not support the databricks labs install ucx technical requirements such as Python 3.10, administrative access, python package access or network access but your Databric...
@DouglasMoore thank you for sharing this!
